Compare commits

...
Author SHA1 Message Date
DeeJanuzandClaude Opus 5.5 7160180906 Merge branch 'org-home' into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:17:40 -06:00
DeeJanuzandClaude Opus 5.5 72d9b79d7b Frametop moves to the organization's repo: issues too
The plan changed from two repos to one: DeeJanuz/frametop transfers to the Frametop
organization, so issues and pull requests go to Frametop/frametop as well (report.sh,
keys-report.py, README, the hand recorder page). The consent text keeps its old link, which
GitHub redirects, so its wording doesn't change. pack/README.md says how the move keeps the old
links and the old one-liner working.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:17:40 -06:00
DeeJanuzandClaude Opus 5.5 b5e410b0b9 Merge branch 'org-home' into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:11:45 -06:00
DeeJanuzandClaude Opus 5.5 d9f213b872 Frametop's home is the organization's repo; a stable release in get.sh
New clones, the one-liners (frametop.github.io/frametop), the SteamOS table that releases
check, and FrameDrop's download URLs now point at Frametop/frametop, where CI builds the
releases. Issues and pull requests stay on DeeJanuz/frametop, the upstream it mirrors.
get.sh's menu offers a stable release next to the experimental one (3 and 4). SteamOS 0.4.5
(20261007.6125817) is marked tested with Frametop 0.2.2. pack/README.md says how the two repos
and a release fit together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:11:43 -06:00
DeeJanuzandClaude Opus 5.5 811e15ed9d Merge branch 'calpanel-clicks' into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:30:46 -06:00
DeeJanuzandClaude Opus 5.5 7f55e9328a Gaze calibration: don't wait for ft-eyes' answers
With our tracker, the full calibration asked ft-eyes about each dot (calib-point, up to 3 s)
and for the fit (calib-fit, up to 10 s) with a blocking ask(), which held up the whole gaze
service. Meanwhile the gaze stopped, the helper's 3 s panel lease ran out (the pointer came
back, and a click went to the desktop behind the panel), and an answer slower than EYES_GONE
closed the calibration as if the headset came off.

Checks.ask_eyes sends the command on a socket of its own in ft-gazed's selector, and the answer
(or its deadline, in tick) finishes the dot or the fit. The dot shows its ring full meanwhile;
further clicks do nothing, and a right click doesn't close the panel during the fit.
first-calibration-test.py covers a slow and a silent ft-eyes; the old code fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:30:43 -06:00
DeeJanuzandClaude Opus 5.5 9def5f1762 Gaze calibration panel: a gaze precision or gaze drag press takes the dot
While the panel was up, the helper took only "btn trigger 1" and "gazekey left 1" as
"take this dot". A left button mapped to gaze precision or gaze drag sends "precision|gazedrag
<source> 1" instead, which the panel dropped, so the click did nothing. The panel's answers
move to pointer/helper/calpanel.h, with an offline test (pointer/test/calpanel-test.sh).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:30:43 -06:00
DeeJanuz b521564a68 Merge branch 'report-input' into experimental 2026-10-09 10:03:41 -06:00
DeeJanuzandClaude Opus 5.5 1fdd6f0e78 report.sh: Frametop's keyboard, the Steam menu, and --watch to record a problem
For reports like "windows won't drag while the Steam menu is open" and
"Frametop's keyboard doesn't open", the report now has:

- A section on the keyboard and the Steam menu: the Keyboard setting,
  the relay's devices (a pass-through keyboard keeps ours closed by
  default), ft-textinput and KWin's input method, the desktop's
  QT_IM_MODULE and GTK_IM_MODULE, ft-screens' state and new debug line,
  the SteamVR overlays shown, and the matching log lines.
- --watch [SECONDS]: after the report, it records ft-screens' debug line
  whenever it changes, and the logs, while the user makes it happen.
- A release's version and container (it reported "container dev:
  missing"), the gaze, desktop, eye grabber and Bluetooth units, and the
  newest SteamVR start rather than the log's first.

ft-screens: a "debug" command (Steam menu, Steam in front, our keyboard
shown or aside, mode, lasers, held press and which laser, drags in
progress, where typing goes), and log lines when a drag starts, ends, or
can't start, when Steam comes in front or goes, and when a requested
keyboard doesn't open. The relay logs why a focused text field did or
didn't open the keyboard, once per decision (keys-test covers it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:03:41 -06:00
DeeJanuz 6b7af52e1c Merge branch 'steamos-045-tested' into experimental 2026-10-09 09:31:41 -06:00
DeeJanuzandClaude Opus 5.5 a53ced4f88 SteamOS 0.4.5 is tested (Frametop 0.3.0-exp.4, SteamVR 2.18.2)
Headset tests on 2026-10-09: Bluetooth reconnects, doctor ok, gaze with
our own tracker and SteamOS 0.4's eye-tracker layout, and the desktop
without Steam's autostart and with its own cursor theme. Release
installs stop asking before they install on build 20261007.6125817.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:31:41 -06:00
DeeJanuz 28f91af0d2 Merge branch 'doctor-release-box' into experimental 2026-10-09 09:27:58 -06:00
DeeJanuzandClaude Opus 5.5 e69fe11b2a doctor: in a release, check the release's own container
A release runs in the container its .frametop-release names (BOX, as
scripts/in-box reads it), but doctor.sh looked for the dev build box,
so every release install reported "FAIL container dev" (seen on
0.3.0-exp.4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:27:58 -06:00
DeeJanuz 65acbd9638 Merge branch 'host-build-release' into experimental 2026-10-09 09:26:43 -06:00
DeeJanuzandClaude Opus 5.5 ef1c802e2d Host setup: Frametop's Vibepollo build from its release
The fork's CI published frametop-2.0.0-1 (Vibepollo 2.0.0 with Frametop's
changes, now including the HDR-off guard for SDR display streams). $BuildUrl
points at its sunshine.exe, so a PC needs only host/windows; the first CI
build (a84b6cfc) is upgraded like the hand-built one. Tested on the test PC:
over the first CI build, the setup downloaded the release, matched its
SHA-256, swapped it in and restarted Vibepollo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:26:32 -06:00
DeeJanuz 26de1430b0 Merge branch 'release-notes-channel' into experimental 2026-10-09 09:04:59 -06:00
DeeJanuzandClaude Opus 5.5 e21a2beae0 Release CI: an experimental release's install line says --experimental
get.sh --release asks for a channel and offers stable first, and the
Frametop organization has no stable release yet, so the notes' line
fetched releases/latest and got a 404 unless you picked 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:04:59 -06:00
DeeJanuz 8d6aa459fc Merge branch 'release-notes-url' into experimental 2026-10-09 09:04:07 -06:00
DeeJanuzandClaude Opus 5.5 b93c25eb7e Release notes: get.sh --release from the Frametop organization's Pages
deejanuz.github.io serves DeeJanuz/frametop's main, whose get.sh has no
--release yet ("unknown option: --release"). frametop.github.io serves
Frametop/frametop's experimental, where the releases are built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:04:07 -06:00
DeeJanuz b61aebcf16 Merge branch 'own-one-eye' into experimental 2026-10-09 08:42:41 -06:00
DeeJanuzandClaude Opus 5.5 dba0ef5707 Gaze: with our tracker and one eye tracked, take that eye's blink for both
With Track Dominant Eye Only on, SteamVR judges only that eye's
openness. Our own tracker still finds both pupils, but ft-gazed took the
blinks from SteamVR per eye, so whatever SteamVR reports for the ignored
eye could drop our good reading of it. A blink closes both eyes, so the
tracked eye's now marks both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:42:41 -06:00
DeeJanuz 70c3ad9b1c Merge branch 'steamos-keep' into experimental 2026-10-09 08:36:34 -06:00
DeeJanuzandClaude Opus 5.5 185cb766b1 SteamOS updates: keep the Bluetooth fixes and the eye grabber through them
A SteamOS update deletes every /etc file its keep list
(/usr/lib/rauc/atomic-update-keep.conf) doesn't name. The list keeps
units but not what they run, so after the 0.4.5 update
steamframe-bt-fixups.service and frametop-eyegrab.service failed with
203/EXEC: Bluetooth LE devices stopped reconnecting, and gaze mode fell
back to SteamVR's tracker, which had no calibration.

- Both installers add a drop-in to /etc/atomic-update.conf.d naming
  their file; uninstall removes it.
- update-check.py (doctor.sh) fails when a unit's program is gone and
  warns when it isn't kept.
- ft-gazed's log and Input Settings' Bluetooth page say what to reinstall.
- setup/README no longer says /etc survives updates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:36:34 -06:00
DeeJanuzandClaude Opus 5.5 305dd9bcef Merge branch 'steamos-0.4' into experimental
SteamOS 0.4 (0.4.5) fixes: gaze goes by one eye with Track Dominant Eye
Only; eye-server layouts named by release; update-check covers the new
parts; the desktop no longer autostarts Steam and keeps its own cursor
theme; README note updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:44 -06:00
DeeJanuzandClaude Opus 5.5 90698db6c7 Docs: SteamOS 0.4 is out; one-eye tracking
The README's note said Frametop didn't work on the 0.4 beta. Gaze reads
0.4's eye tracker layout now, and the missing taskbar in #15 wasn't the
beta's doing (0.4.5 has the same KWin and Plasma as 0.3.0), so the note
now says what the update changes and to run doctor.sh after it. The gaze
README covers Track Dominant Eye Only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:03 -06:00
DeeJanuzandClaude Opus 5.5 48a2dcf759 Session: no Steam autostart in the desktop, keep its own cursor theme
The desktop is a KDE session, so it ran /etc/xdg/autostart/steam.desktop.
Steam is already running (the desktop starts from it), so `steam -silent`
only reached that client as a command line it ran (ExecCommandLine in its
console log). SteamOS 0.4 adds -vrdisable -deckard to that entry, for
Desktop Mode, where Plasma starts Steam itself. The session now hides it
with the other two. The marker goes to 2, so desktops that hid the first
two get Steam's copy once, and an entry someone brought back stays.

The Steam client's XCURSOR_THEME=steam reaches the desktop through its
environment, and KWin and the apps prefer it to the desktop's setting.
There was no such theme, so they fell back to Breeze; SteamOS 0.4's
holo-cursors adds one. The session sets the desktop's own theme (Breeze
unless changed in its System Settings) and leaves the size as it was.
ft-screens doesn't draw KWin's cursor, so this shows over remote access
and with the gamescope backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:03 -06:00
DeeJanuzandClaude Opus 5.5 06228725cc SteamOS 0.4: name the eye-server layouts by release, check its new parts
SteamOS 0.4 (0.4.5) is the stable release now, with the eye-server.mmap
layout that the 0.4.3 beta brought (its eye tracker and cv driver differ
from the beta's only in thread priorities and the presence sensor). ft-gaze
and update-check call the layouts "SteamOS 0.3" and "SteamOS 0.4 (+5)"
instead of stable and beta.

update-check also reports when Track Dominant Eye Only is on, warns when
/usr/share/steamos/steamos-cursor.png (the gamescope backend's cursor) is
gone, since SteamOS 0.4's own session moved to /usr/share/holo, and gives
retest hints for steamdeck-kde-presets (its Steam autostart entry changed)
and the new holo-cursors package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:03 -06:00
DeeJanuzandClaude Opus 5.5 d82364ed00 Gaze: go by one eye when SteamVR tracks only the dominant one
SteamOS 0.4 adds Track Dominant Eye Only (SteamVR settings, General,
advanced: steamvr.eyeTrackingDominantEyeOnly with steamvr.dominantEye).
SteamVR's tracker then ignores the other eye. The calibration kept only
samples where it had both eyes, so if that eye reads as lost, no dot
would ever be taken.

gazecal.tracked_eye reads the setting (TrackedEye re-reads it when the
settings file changes). With it on, steady_samples judges the tracked
eye alone (no vergence check), the calibration needs only that eye's
reading and drops the other's and set 2's average, ft-gazed ignores the
other eye (its mmap2 source takes the eye's own reading, and "eyes"
mode needs only that eye calibrated), and the fit check marks the other
eye "not tracked". Nothing changes with the setting off.

Untested in the headset: what SteamVR's mmap reports for the ignored
eye is unknown; gaze/test/one-eye-test.py covers the logic offline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:03 -06:00
DeeJanuzandClaude Opus 5.5 eca95477cd Hands: HANDS_MODELS setting picks ft-hands' model folder
Fine-tuned models (the hand dataset's student models) can't ship in the
repo yet, so a headset that has them sets HANDS_MODELS in
~/.config/frametop.conf instead of passing --models to every launcher
(service, ft-cutouts, recorder, probes). --models still overrides it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:44:28 -06:00
DeeJanuz 52b10478ad Merge branch 'stream-in-box' into experimental 2026-10-08 10:45:56 -06:00
DeeJanuzandClaude Opus 5.5 275529da8f Gaze probe: run ft-gaze through scripts/in-box, not the "dev" container
On a release install there is no "dev" container. in-box picks the release's own container,
and starts it first, as container-up.sh did here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 10:45:56 -06:00
DeeJanuz b727b8b8fe Merge branch 'stream-in-box' into experimental 2026-10-08 10:44:20 -06:00
DeeJanuzandClaude Opus 5.5 46bf562c5f Remote displays: ft-layout runs ft-stream in the release's container, not "dev"
On a release install with no "dev" container, distrobox asked whether to create it, and
adding a display hung until it timed out. run_stream now goes through scripts/in-box, as
gazecheck.py does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 10:44:20 -06:00
DeeJanuz 0ffbf7a718 Merge branch 'get-release-menu' into experimental 2026-10-08 09:53:38 -06:00
DeeJanuzandClaude Opus 5.5 2e293beb3a get.sh: the menu offers the experimental release; releases come from the Frametop org
A third choice installs the newest experimental release, built, so an install in the headset
needs only the short command typed on the virtual keyboard, not --release options. Releases
are looked up in Frametop/frametop, where CI builds them (Depot's runners need an
organization); FRAMETOP_REPO still overrides it, and the branches still clone from
DeeJanuz/frametop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:53:38 -06:00
DeeJanuz 4daa8fea5b Merge branch 'five-dot-panel' into experimental
# Conflicts:
#	gaze/README.md
#	gaze/gazecheck.py
2026-10-08 09:40:14 -06:00
DeeJanuz 78fac8ca0a Merge branch 'fix/0x1f6-pr42' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuz 09cec0c980 Merge branch 'standalone-recorder' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuz 325d3d91a4 Merge branch 'hand-probe' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuz 48a3c0af87 Merge branch 'hands-misread' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuz d7cc82ab09 Merge branch 'lighting-doc-fix' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuzandClaude Opus 5.5 a4a1d8819c Gaze: the five-dot check gets its own wider, see-through panel
The quick check's 16 degree square left four of the five dots (12 degrees left and right,
9 up and down) off the panel, unseen. The five-dot check now shows in a 40 degree 4:3 panel,
see-through like quick, with the full calibration's longer notes. gazecheck.py's PANEL_DEG
mirrors the panel sizes and logs any dot that wouldn't fit. ft-gazectl and ft-gazed take
quickcal, calibrate, fitcheck, and fivecheck to open a check from a shell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:40:00 -06:00
DeeJanuzandClaude Opus 5.5 e2b4eb2a81 Hand recorder: the standalone repo moved to the Frametop org
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:24:39 -06:00
DeeJanuzandClaude Opus 5.5 95938a1f27 Merge branch remote-displays (profiles keep remote displays; quick reset opens the profile) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:22:26 -06:00
DeeJanuzandClaude Opus 5.5 9c0facd0c5 Quick reset opens the profile in use, as Open profile does
ft-layout reset reopens the profile in use (the custom arrangement with an
active profile) with use NAME, so its screens, hidden screens, remote
displays and apps all go back as the profile has them. Without one it runs
apply, as before. Meta+Shift+R and the Reset Screen Layout entry
(ft-layout-reset), the input relay's layout_reset action (a mapped mouse or
controller button), and the reset button on a screen's bar now run it.
Display Settings' Arrange now still only arranges.

Before, every quick reset ran apply, which put the desktop screens back but
left the profile's hidden screens, remote displays and apps alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:22:14 -06:00
DeeJanuzandClaude Opus 5.5 eaaab1ccaf Profiles keep remote displays like apps, and connect them when opened
Saving a profile records the remote displays connected then, with their
places and hidden state, the way it records the apps open then. Opening the
profile (use, open, desktop start), or arranging while it's the profile in
use, connects each of them whose host answers on Vibepollo's Web UI port (at
its address or its dongle's, as its route allows) and puts it back where it
was saved. A host that doesn't answer within 1.5 s is skipped, and its
displays stay disconnected. Like apps, displays the profile doesn't have are
left as they are: a profile connects, but never disconnects.

Before, a profile kept the places of every display that had one, connected
or not, and opening it never reconnected a display.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:22:09 -06:00
DeeJanuzandClaude Opus 5.5 84a354536e install-release.sh: don't take "current" for an old release; pack/trial.sh switches the live install
Found in the runtime trial (2026-10-07): the clean-up after an install loops over
releases/*/, which includes the "current" link to the release just installed, so it removed
that release's container and image, and the services it had just started kept failing until
the release was installed again. Links are skipped now.

pack/trial.sh switches this Frame to a release and back: on saves what the release's
install.sh replaces (the frametop-* units with their drop-ins and .wants links, the launcher
and menu entries, the driver's folder and SteamVR's registry, frametop.conf, the desktop's
shortcuts), moves the drop-ins aside, installs from the unpacked zip with
--no-eye-tracker --no-bluetooth, and restarts SteamVR; desktop restarts the VR desktop from
the release; status shows which container each program runs in; off puts everything back,
restarts SteamVR, and stops the release's container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 23:31:37 -06:00
DeeJanuzandClaude Opus 5.5 14cbfc310e FrameDrop window: a keypad for the password, clicked with the controller's laser
Tested in VR with frame-testbench: laser clicks reach the install window (the password field
takes focus, a checkbox toggles), but SteamVR's keyboard doesn't come up for the field, and
opened with steam://open/keyboard its keys never reach the window (the field stayed empty,
read through accessibility). So the window brings its own keypad under each password field:
letters, digits, Shift, symbols, space, and delete, shown when Steam started the window and
toggled with a Keypad button. Checked on a virtual display with pointer clicks and
accessibility. Also: "Frametop 0.3.0-exp.0, built,:" reads "Frametop 0.3.0-exp.0:".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 23:15:05 -06:00
DeeJanuzandClaude Opus 5.5 40c7606ace Merge branch remote-displays (the host setup installs the CI build of Vibepollo) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 22:18:52 -06:00
DeeJanuzandClaude Opus 5.5 00a377a0fb Host setup: Frametop's Vibepollo build from its CI, with WebRTC
The pinned sunshine.exe is now the frametop/2.0.0 build from Frametop/frametop-vibepollo's
CI (a84b6cfc), configured like stock 2.0.0, WebRTC included; the hand-made build it
replaces had WebRTC off. A PC with that older build gets the new one like a stock install
does, and the original exe stays the one kept for -Undo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 22:18:52 -06:00
DeeJanuzandClaude Opus 5.5 95ccaa30de Merge branch remote-displays (the fork's new address) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:25:15 -06:00
DeeJanuzandClaude Opus 5.5 b655cd7446 Remote displays: the Vibepollo fork moved to the Frametop org
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:25:15 -06:00
DeeJanuzandClaude Opus 5.5 f0df502a23 Hands: misread guards for fine-tuned landmark models
ft-hands --misread-guard 1 (HANDS_MISREAD_GUARD=1) and ft-handreplay --misread-guard. Fine-tuned
landmark models stay sure of a hand when two hands touch, and can read the held hand as the
other side: the two cameras' views then miss by 3-4 cm, the tracker splits one off as a new
hand, the hand-over puts the old hand back onto it, the duplicate check drops the new one, and
round it goes (a contributed session, holding a card with both hands: 51 duplicates and 40
splits in 59 s with the fine-tuned models, 5 and 9 with the stock ones). With the guard, a
reading whose side is more than 0.7 off its established hand's is dropped, and a split-off view
that sits where its hand already is in that camera is dropped instead of starting a new hand:
21 duplicates and 10 splits, both hands tracked as often. Off by default: the stock model's side
calls are noisier, and the first guard costs it tracking on some recordings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 17:38:34 -06:00
DeeJanuzandClaude Opus 5.5 d4558c7187 Hand recorder docs: hand contrast isn't a daylight test
PR #6, recorded at night in a pale room, had hands nearly as faint against the room (1.22x) as
PR #5's sunlit round (1.15x). Review decides daylight from the pictures (sunlit windows, the
time of day); the contrast is supporting evidence only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:17:51 -06:00
DeeJanuzandClaude Opus 5.5 aef18d3601 Hand recorder: press the headset button twice to redo, hold it to stop
The headset button now has three gestures (ButtonGestures), so a session
can run with the window out of reach, as the standalone Hand Recorder's
window is in SteamVR's dashboard:
- a press: Next, pause or resume, as before. It counts once 0.45 s pass
  without a second press.
- two presses: redo, as R.
- a hold of 1.5 s: stop, as Esc, while the button is still down.

Gaps between events come from the kernel's timestamps, so two presses
read in one go still count as two. The panel's key line, the pause
note and the no-hands question say how.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:43:22 -06:00
DeeJanuzandClaude Opus 5.5 983a4654b3 Hand recorder: delete the window before the backend on quit
Python tore the backend down first, so every binding ran once more against null on the way out
and the log filled with "Cannot read property ... of null".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:33:49 -06:00
DeeJanuzandClaude Opus 5.5 85b8717bb8 Hand recorder: a dry run doesn't start the camera broker to measure the light
checkLighting started ft-camd when no ring was live, dry run or not (found porting the window
for the standalone Hand Recorder). Also drops the backend's unused standalone property.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:31:56 -06:00
DeeJanuzandClaude Opus 5.5 dfd1bbca0f Hand recorder: hooks for the standalone Hand Recorder
frametop-hand-recorder (a separate repo, with this one as a submodule) ships hands/ with its own
dashboard window and binaries built for the SteamOS host. A standalone.json at the top of its tree
marks it (takes.standalone()):
- session.py starts ft-hands directly instead of through distrobox, for recording and tracking
- session.json's and the manifest's tool name its build ("ft-handrec <describe> (<name> <version>)")
- the repair hints say to run its install command again (fix_hint)

ft_handrec.py takes --qml and --style for that window, and has toolVersion and standalone for it.
The Makefile passes LDFLAGS to the ft-hands links (its release links them statically), and
test_qml_backend.py checks another window with FT_HANDREC_QML.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:42:00 -06:00
DeeJanuzandClaude Opus 5.5 0318bb72ab ft-handtest --probe: point at the analysis script's place
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:19:58 -06:00
DeeJanuzandClaude Opus 5.5 3519a7a0e1 ft-handtest --probe: dots where the cutouts would land, for measuring them against Room View
A see-through test panel with dots on the wrist, middle knuckle and fingertips of each
tracked hand, one colour per timing: where the cameras saw the hand, moved ahead to now,
and moved ahead to now + the cutouts' lead. Recorded together with the headset view, they
show how far the cutouts land from the hand Room View shows, still and moving, and which
lead fits. Each tick goes to a JSON-lines log. Renderer::Marks draws the dots; Hands::Points
gives the landmarks. ft-screens' cutouts are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:04:52 -06:00
DeeJanuzandClaude Opus 5.5 8117a4fb7e Merge branch hand-perf (hand cutouts without waiting for the GPU) into hand-probe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:00:31 -06:00
DeeJanuzandClaude Opus 5.5 891d84ae2b Hands: name the side cameras as XRService does, and bind their buffers exactly
The side cameras came out swapped on most SteamVR starts, and right on some.
Two guesses combined:

- ft-hands and camcheck.py named each video device by its TrackingCameraInit
  index (0 = slam_left). The index is only the order XRService opens the
  cameras in: on every start logged since 2026-10-04, index 0 was slam_right.
  XRService names each sensor subdev itself ("Found camera 'slam_left': ...
  v4l_subdev=/dev/v4l-subdev30"), and each TrackingCameraInit line says which
  subdev it opened. Name the devices by that; the index order stays the
  fallback for a log without those lines. The capture-pipe fallback had vfe3
  and vfe4 backwards too.

- ft-camd bound each run of XRService's buffers to a device by the sensor
  subdev opened just before it in XRService's fd table. That order changes
  when XRService restarts its cameras (17:09 today: the run after
  og01a1b 4-0060's subdev was video9's). VIDIOC_QUERYBUF on the device names
  the descriptor XRService queued at each index, from any handle, so bind runs
  by that, and split a run holding two cameras' buffers (the upper pair and the
  colour pair came as 32-buffer runs). The order stays the fallback.

Checked on the live XRService: all four tracking cameras "bound by
VIDIOC_QUERYBUF" with 16 buffers each, and ft-hands now reads
slam_left=video13 slam_right=video9. ft-hands' hands-based side check stays
as the safety net.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:22 -06:00
DeeJanuzandClaude Opus 5.5 64c4eec59f Screens: hand cutouts without waiting for the GPU
The cutout composite drew each screen's whole client buffer into a side-by-side buffer on
every tick a hand was in front of it, then waited for the GPU with glFinish (1-6 ms, the
likely cause of the VR frame drops on 2026-10-01). Now:

- A drawn buffer gets a fence and is shown from a later tick once the fence has passed, so
  ft-screens never waits for the GPU (except for a panel's first buffer after a pause, so a
  stale one never shows). The prediction lead goes from 25 to 36 ms for that tick.
- Nothing is drawn when the client frame and the cutouts haven't changed, and SteamVR
  isn't handed the same buffer again.
- When only the cutouts moved, a buffer that holds the same client frame is drawn again
  only around the old and new cutouts (scissored).
- "cutouts state" reports draws, partial draws, unchanged ticks, busy ticks, waits and CPU
  time per second since the last state; ft-handtest prints the same counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 10:14:54 -06:00
78 changed files with 2505 additions and 396 deletions

No files matched your search

+11 -3
View File
@@ -83,9 +83,17 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
run: |
pre=()
[[ $VERSION == *-* ]] && pre=(--prerelease)
pre=() notes=pack/release-notes.md
if [[ $VERSION == *-* ]]; then
pre=(--prerelease)
# get.sh --release asks for a channel and offers stable first: an experimental
# release's install line names its channel.
notes=$RUNNER_TEMP/release-notes.md
sed 's/bash -s -- --release`/bash -s -- --release --experimental`/' pack/release-notes.md >"$notes"
grep -q -- '--release --experimental`' "$notes" ||
echo "::warning::pack/release-notes.md has no get.sh --release line to mark experimental"
fi
gh release create "$GITHUB_REF_NAME" --draft --verify-tag "${pre[@]}" \
--title "Frametop $VERSION" --notes-file pack/release-notes.md --generate-notes \
--title "Frametop $VERSION" --notes-file "$notes" --generate-notes \
framedrop/build/Frametop.zip framedrop/build/frametop.framedrop.json \
framedrop/build/frametop-release.json framedrop/build/SHA256SUMS
+16 -8
View File
@@ -13,11 +13,13 @@ Two settings apps come with it: Frametop Display Settings for the screens, profi
Frametop is an independent project, not made by or affiliated with Valve.
Frametop lives at [Frametop/frametop](https://github.com/Frametop/frametop): code, releases, issues, and pull requests. It moved there from DeeJanuz/frametop on 2026-10-09; the old links and clones still work.
Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions, ideas, and help with your setup.
## Install on the headset
> **Frametop doesn't work on the SteamOS beta right now.** On the beta (SteamOS 0.4.3), gaze mode can't read the eye tracker, and the desktop has started without its taskbar ([#15](https://github.com/DeeJanuz/frametop/issues/15)). Use the stable SteamOS release until this note is gone.
> **SteamOS 0.4:** SteamOS 0.4 moved the eye tracker's data that gaze mode reads. This version of Frametop reads both SteamOS 0.3's and 0.4's, and it's tested on 0.4.5. Run `scripts/doctor.sh` after the update: it says whether the eye tracker's layout is one Frametop knows. It also says whether the update deleted the Bluetooth fixes or our eye tracker's frame grabber, which happens when they were installed by Frametop 0.3.0-exp.3 or older. Reinstall what it names (`setup/bluetooth/install.sh install`, `gaze/tracker/install.sh`). From then on they're kept through updates.
You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space.
@@ -26,10 +28,12 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
3. Run:
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
curl -fsSL https://frametop.github.io/frametop/get.sh | bash
```
It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/DeeJanuz/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/Frametop/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.1 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
@@ -133,7 +137,7 @@ With the displays off, the headset keeps tracking and rendering, so it uses abou
## Known limitations
This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 20260922, SteamVR 2.17.10).
This is an early release, tested on one Steam Frame (SteamOS 0.4.5 build 20261007, SteamVR 2.18.2; before that SteamOS 0.3.0 build 20260922, SteamVR 2.17.10).
- A SteamOS or SteamVR update can break parts of it until Frametop catches up. After an update, run `cd ~/frametop && scripts/doctor.sh` in a terminal. It checks what Frametop needs from SteamOS, and says what changed since the versions you last marked as working and what to try. Once everything works, `scripts/doctor.sh --mark-good` records the versions. If something stops working, please report it.
- The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes.
@@ -157,14 +161,18 @@ In a terminal on the headset, run:
cd ~/frametop && scripts/report.sh
```
This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. It takes up to half a minute, because it also checks gaze mode: it starts the gaze service for a moment to see whether the eye tracker sends. If gaze or its calibration doesn't work, run it while you wear the headset. `scripts/gaze-report.py` prints only the gaze part, with what looks wrong first. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
From a release, start in `~/.local/share/frametop/releases/current` instead of `~/frametop`.
This writes `frametop-report-<date>.txt` with version numbers, service states, settings, Frametop's keyboard and the Steam menu, and recent logs. Bluetooth addresses and the headset's serial number are masked.
If the problem is something you can make happen, like a window that won't drag or a keyboard that doesn't open, run `scripts/report.sh --watch` instead. After the usual report it records for 60 seconds (`--watch 120` for longer) while you make it happen in the headset. It notes when the Steam menu opens and closes, which laser drags what, where typing goes, and when Frametop's keyboard opens or why it doesn't. It takes up to half a minute, because it also checks gaze mode: it starts the gaze service for a moment to see whether the eye tracker sends. If gaze or its calibration doesn't work, run it while you wear the headset. `scripts/gaze-report.py` prints only the gaze part, with what looks wrong first. Then [open an issue](https://github.com/Frametop/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
## Update
Run the same command again. It updates `~/frametop` to the latest of the version you have (or switches, if you pick the other one) and installs it:
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
curl -fsSL https://frametop.github.io/frametop/get.sh | bash
```
Or by hand: `cd ~/frametop && git pull && ./install.sh`.
@@ -174,7 +182,7 @@ Or by hand: `cd ~/frametop && git pull && ./install.sh`.
In a terminal on the headset, run:
```
curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash
curl -fsSL https://frametop.github.io/frametop/uninstall.sh | bash
```
It works in two steps, so it never takes away the keyboard, mouse, or desktop you're using while it runs:
@@ -194,7 +202,7 @@ A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland com
| Folder | What it is |
| --- | --- |
| `get.sh` | The one-line installer: picks stable or experimental, clones or updates the repo, and runs `install.sh`. |
| `get.sh` | The one-line installer: picks stable or experimental, clones or updates the repo, and runs `install.sh`; or installs a built release (`--release`). |
| `install.sh` | The one-step installer. Safe to re-run. |
| `uninstall.sh` | The uninstaller: run it, restart the headset, and run it again. It doesn't need the rest of the repo. |
| `desktops.sh` | Start, stop, and configure the desktop, and install the input relay. |
+2 -2
View File
@@ -200,7 +200,7 @@ A podman container's monitor process (conmon) stays in the cgroup of whatever st
KWin renders with OpenGL through zink on Turnip, Vulkan on the same GPU vrcompositor needs to hit its frame time, and on the Frame that costs CPU too. The nested session started with KWin's defaults: blur and background contrast on (no `[Plugins]` group in its kwinrc) and animations at full length. Blur re-renders what's behind every translucent panel and menu each time it changes, and every animated frame is one more frame for KWin and ft-screens to draw and send. They're off by default in the Frametop desktop. The session script writes them before KWin starts, only where the desktop's own config has no value, once: System Settings deletes a setting put back to KDE's default rather than writing it, so without the marker in `frametoprc` a user who turned blur back on would lose it at the next start. The effect ids (`blur`, `contrast`) are the ones built into KWin 6.2.5 on SteamOS; KWin reads `<id>Enabled` from `[Plugins]`.
The nested session also runs the system's XDG autostart entries, being a KDE session. Discover's update notifier started `plasma-discover --mode update` in it (520 to 620 MB resident and about 9% of a core, plus `flatpak-system-helper` and AppStream downloads), and IBus started a daemon, the kimpanel panel and its GTK extension that nothing can use: KWin hands text input to the one input method it starts (`ft-textinput`), and the session drops `QT_IM_MODULE`, `GTK_IM_MODULE` and `XMODIFIERS`. The session hides both for this desktop only, with `Hidden=true` copies in its own autostart folder. The geoclue demo agent stays: it's what answers apps' location requests to Geoclue outside GNOME, and it costs nothing while idle. Orca's entry only starts in GNOME-family desktops.
The nested session also runs the system's XDG autostart entries, being a KDE session. Discover's update notifier started `plasma-discover --mode update` in it (520 to 620 MB resident and about 9% of a core, plus `flatpak-system-helper` and AppStream downloads), and IBus started a daemon, the kimpanel panel and its GTK extension that nothing can use: KWin hands text input to the one input method it starts (`ft-textinput`), and the session drops `QT_IM_MODULE`, `GTK_IM_MODULE` and `XMODIFIERS`. Steam's entry (`steam -silent`, from steamdeck-kde-presets) reached the running Steam client as a command line it ran (`ExecCommandLine` in its console log), since the desktop starts from Steam; SteamOS 0.4 added `-vrdisable -deckard` to it, for Desktop Mode, where Plasma starts Steam itself. The session hides all three for this desktop only, with `Hidden=true` copies in its own autostart folder. The geoclue demo agent stays: it's what answers apps' location requests to Geoclue outside GNOME, and it costs nothing while idle. Orca's entry only starts in GNOME-family desktops.
Plasma 6.2.5 keeps each panel on a screen number (`lastScreen` in `plasma-org.kde.plasma.desktop-appletsrc`), and the numbers rank the enabled outputs by priority, so 0 is the primary screen. A panel whose number is past the screen count gets no view, and Plasma never moves it: the remap it runs at every start only moves a panel whose number has no desktop, and this desktop keeps a desktop for every output it has seen, spares included. So the taskbar was lost when the number of screens went down, and once it was found saved on a spare output, number 8 of a desktop with three screens ([#18](https://github.com/DeeJanuz/frametop/issues/18)). Before Plasma starts, the session runs `session/fix-panels.py`, which moves any panel numbered past the screen count, with its system tray's containment, to screen 0, keeping its widgets and settings. A panel stays put when screen 0 already has one on that edge, and comes back by itself if the screens do. Before each repair the file is backed up to `<file>.ft-bak.last`. `<file>.ft-bak` keeps it as it was before the first repair and is never overwritten. The repair writes over a moved panel's old screen number, so the backups are the only record of it, and `.ft-bak.last` also keeps everything changed since the first repair. Plasma's scripting can't do this while it runs (`panel.screen` is read-only in 6.2.5), so a lost taskbar comes back at the desktop's next start. `scripts/doctor.sh` and `scripts/report.sh` list the panels and their screens.
@@ -231,7 +231,7 @@ Hiding the screens during a game kept them out of view, but Frametop kept using
On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-steamvr-rel` package), next to KWin, gamescope, and the kernel, so every SteamOS update can bring a new SteamVR too. Frametop survives updates: it lives in the home folder and the `dev` container, the Bluetooth fixes are in `/etc`, which SteamOS keeps across updates, and nothing goes into `/usr`. What an update can break is what Frametop uses from the image. The public OpenVR API is versioned and stays put. The rest is less certain: `IVRIPCResourceManagerClient`, which is newer than the header SteamVR ships; the text `vrcmd --overlays` prints; the eye tracker's shared memory layout; XRService's camera buffers; KWin's nested backend; and behavior Frametop works around, such as the SteamVR Settings page that `ComputeOverlayIntersection` can't find or the scale KWin's nested backend doesn't undo.
`scripts/update-check.py`, which `scripts/doctor.sh` runs, checks what it can directly: that SteamVR still serves every OpenVR interface version the installed programs were built against (read from the binaries), that `vrcmd`'s format still parses, that the eye tracker's shared memory still has a layout ft-gaze knows (stable's, or the 0.4.x beta's, with every field from the timestamp on 5 bytes later), by the same test ft-gaze uses to pick one, and the host files, services, sockets, and driver registration. Behavior can't be checked without someone in the headset, so it records the versions of the packages that matter once things work (`--mark-good`), and after an update names what changed and what to try by hand.
`scripts/update-check.py`, which `scripts/doctor.sh` runs, checks what it can directly: that SteamVR still serves every OpenVR interface version the installed programs were built against (read from the binaries), that `vrcmd`'s format still parses, that the eye tracker's shared memory still has a layout ft-gaze knows (SteamOS 0.3's, or 0.4's, with every field from the timestamp on 5 bytes later), by the same test ft-gaze uses to pick one, and the host files, services, sockets, and driver registration. Behavior can't be checked without someone in the headset, so it records the versions of the packages that matter once things work (`--mark-good`), and after an update names what changed and what to try by hand.
## Approaches we dropped
+3 -1
View File
@@ -6,7 +6,8 @@ A profile is a named layout that also opens apps. It holds:
- where each screen goes, with its size in metres, curve, roll, and pin (what a named layout held before profiles);
- which screens show and which are hidden;
- the apps, one entry per window: on a screen at a place and size, or floating at a pose, size, and scale.
- the apps, one entry per window: on a screen at a place and size, or floating at a pose, size, and scale;
- the remote displays connected when it was saved, each with its place and whether it's hidden ([remote-displays.md](remote-displays.md)). Opening the profile connects them, if their computer answers, and puts them back. Like its apps, it leaves other displays connected.
So a "Work" profile can put three screens around you with a browser, two terminals, and an editor on them, and a "Couch" profile can hide every screen and float one video player in front of you.
@@ -50,5 +51,6 @@ So a "Work" profile can put three screens around you with a browser, two termina
- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name (found the same way as at capture): single-instance and D-Bus-activated apps open their windows from a process that was already running.
- **Default at start.** The session script runs `ft-layout start --wait 90`. That opens the profile in `FT_PROFILE` or `default_profile` (screens, then the apps once ft-floatd is up), or runs `apply --wait` if there's none. Start in profile on the Layout & profiles page sets `default_profile` (`ft-layout default NAME|none`). Plasma's session restore is turned off in the session (`ksmserverrc`: `loginMode=emptySession`).
- **Launcher entries.** Each profile gets `~/.local/share/applications/frametop-profile-<name>.desktop` ("Frametop: Work"), written when it's saved and removed when it's deleted. They show in SteamVR's Launch a program list, the Application Launcher, and KRunner. Running one (`ft-layout open NAME`) switches to that profile if the desktop runs. Otherwise it starts the desktop with `FT_PROFILE` set (`systemd-run`, as `desktops.sh start` does), which overrides `default_profile` for that start. That needs SteamVR to be running.
- **The quick reset.** Meta+Shift+R, the reset button on a screen's bar, the Reset Screen Layout menu entry, and a button mapped to Reset desktop screen layout run `ft-layout reset`: with a profile in use (`active`, and the custom arrangement), that's `use` on it again, so everything goes back as the profile has it. Without one, it's `apply`.
- **The action.** `profile:NAME` in the input relay (it runs `ft-layout use NAME`) for key combinations, mouse buttons, and controller buttons, with or without pointer mode. Input Settings lists one "Open profile NAME" action per profile.
- **Display Settings.** On the Layout & profiles page, the arrangement list has the profiles, which can be renamed and deleted. Open profile and Save as profile… are the page's actions. A profile's apps are listed with where each goes and a button to leave one out, plus which screens it hides. Start in profile picks the one the desktop starts with. The Visibility tab's Screens shown switches hide screens one at a time.
+3 -3
View File
@@ -30,7 +30,7 @@ kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key contrastEna
kwriteconfig6 --file ~/.config/frametop/kdeglobals --group KDE --key AnimationDurationFactor 1
```
Two of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, and IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`), and skips a name you already have a file for. Delete a copy to start that program again.
Three of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`, and Steam (`steam`), which is already running. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`; Steam's was added later and is hidden once on desktops that already had the other two), and skips a name you already have a file for. Delete a copy to start that program again.
Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in `~/.config/frametop-layout.json`. The backend, remote desktop, and pointer settings are in `~/.config/frametop.conf`; `session/frametop.conf.example` lists every key.
@@ -48,7 +48,7 @@ Every screen is an overlay named `frametop.screen.N` with five controls:
- `.curve` bends the screen into a cylinder around you, using your current distance as the radius, or makes it flat again.
- `.roll` rolls the screen when you drag it sideways, like a knob. It snaps level within 2.5°, and scrolling on it turns 5° per notch.
- `.resize`, the tab on the bottom right corner, sets the width. Screens go down to 15 cm wide.
- `.reset`, left of the bar, puts every screen back in its layout around where you are now, like Meta+Shift+R (`ft-layout apply`).
- `.reset`, left of the bar, is the quick reset, like Meta+Shift+R (`ft-layout reset`): with a profile in use it opens that profile again, as Open profile does, and otherwise it puts every screen back in its layout around where you are now.
The controls are sized from both the screen's width and its distance from you, follow the surface of a curved screen, and stay invisible until a laser or the 3D mouse's cursor lands on one or comes within about 1.5 times a button's size of it. While invisible they're still there, fully transparent, so SteamVR's laser can find them. They're translucent until a laser is on them, like SteamVR's own window controls.
@@ -148,7 +148,7 @@ Device rules are saved in `~/.config/frametop-input.json`. `input-settings/insta
## Frametop Display Settings and ft-layout
When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the reset button left of any screen's bar, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout.
When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the reset button left of any screen's bar, the Reset Screen Layout menu entry, or a button mapped to Reset desktop screen layout. With a profile in use, these open it again, the same as Open profile: its screens, hidden screens, remote displays and apps. Arrange now in the app arranges the screens (and the profile's remote displays) without reopening its apps or hiding its hidden screens again.
The desktop's own screen arrangement follows where the screens are around you, whatever their numbers: a screen you see to the left of another is to its left in Plasma too, so the pointer and dragged windows cross straight to it. Screens one above the other stack, and screens pinned to a wrist or your head come last. It's updated at startup, after arranging or saving the layout, and half a second after you let go of a screen you moved. With the headset off there's no head pose to go by, and the arrangement stays as it was.
+10 -5
View File
@@ -341,7 +341,7 @@ So far, Vibepollo 2.0.0's Remote Monitors aren't reliable on the test PC: a stal
- the release rolls back, so the virtual display stays attached and "Disconnect Monitor" does nothing;
- for a normal game (the Mac's session), the per-client identity stays held, and every later composition fails on it.
Removing the virtual display alone, as a Vibepollo restart does, left the layout intact. The fork [DeeJanuz/frametop-vibepollo](https://github.com/DeeJanuz/frametop-vibepollo) (GPL-3.0, like Vibepollo), on branch `fix/windows-remote-monitor-release`, changes two things on Windows:
Removing the virtual display alone, as a Vibepollo restart does, left the layout intact. The fork [Frametop/frametop-vibepollo](https://github.com/Frametop/frametop-vibepollo) (GPL-3.0, like Vibepollo), on branch `fix/windows-remote-monitor-release`, changes two things on Windows:
- The coordinator removes the departing display first and recomposes only if another owned display remains (`retire_before_recompose`, set by the Windows runtime). A normal game's identity is released even when its display is already gone. Linux keeps the old order, which exists so KWin never has zero outputs.
- Composed layout applies run with display recovery off, so a refused layout can't reset the host's arrangement.
@@ -401,7 +401,7 @@ The user decided remote displays are Frametop displays, with the same controls a
- `frametop-layout.json` has a `hosts` list. Each host has its displays: id, paired client, what it streams, label, stream size, fps, bitrate, and a screen's place, width, curve, pin and hidden flag. Each display keeps its screen number (101 and up), so the numbers don't shift when one is removed.
- ft-layout starts the remote displays' streams on every apply and at desktop start, even without a head pose, and places them when there is one. Displays without a place go in a row above the screens.
- `capture` and `save` record their places.
- A profile keeps their places and hidden state by id (`profiles[NAME]["remote"]`), and `use` puts them back.
- A profile keeps the displays connected when it's saved, like the apps open then: their places and hidden state by id (`profiles[NAME]["remote"]`). Opening it (`use`, `open`, desktop start), or arranging (`apply`, Reset Screen Layout) while it's the profile in use, connects each of them whose host answers on Vibepollo's Web UI port, at its address or its dongle's as its route allows, and puts it back where it was saved. A host that doesn't answer is skipped, and its displays stay disconnected. Displays the profile doesn't have stay as they are: a profile connects, but never disconnects (user decision, 2026-10-08).
- `hide`/`show N` take their numbers.
- `ft-layout remote list|monitors|add|set|connect|disconnect|remove`: `add` pairs the display's client with the host's token, and `remove` unpairs it (the host stays). `disconnect` sets the display's `off` flag and stops its stream; apply, profiles and desktop start skip it until `connect`.
- ft-screens starts a host's streams one after another. Three started at once made Vibepollo refuse some ("Another stream operation is still running"), and the captures that started while the Remote Monitor appeared got no picture.
@@ -439,17 +439,23 @@ The user decided remote displays are Frametop displays, with the same controls a
**The PC side: Frametop host setup (2026-10-07, ~17:40):**
- `host/windows/Setup Frametop host.cmd` (it runs `frametop-host-setup.ps1` and asks for admin) does what the test PC got by hand, nothing else of its settings:
1. Vibepollo 2.0.0 with its own installer when it isn't there (downloaded from Nonary/Vibepollo's release, SHA-256 checked first; you click through it).
2. Frametop's build of `sunshine.exe` over the original (kept as `sunshine.exe.2.0.0-original`), SHA-256 checked: from `-FrametopBuild PATH|URL`, the script's `$BuildUrl`, or `sunshine-frametop.exe` next to it. Only over Vibepollo 2.0.0's own exe; another version stops it with a message. Without the build, virtual displays still work, but not the PC's own monitors.
2. Frametop's build of `sunshine.exe` over the original (kept as `sunshine.exe.2.0.0-original`), SHA-256 checked: from `-FrametopBuild PATH|URL`, the script's `$BuildUrl` (the fork's release), or `sunshine-frametop.exe` next to it. Only over Vibepollo 2.0.0's own exe; another version stops it with a message. Without the build, virtual displays still work, but not the PC's own monitors.
3. `remote_monitor_mute_audio = disabled`, `remote_monitor_disconnect_on_client_disconnect = enabled`, `remote_monitor_disconnect_on_stream_end = disabled`. The rest of `sunshine.conf` stays as it is.
4. The Web UI login: keep the one there is, or set one (`sunshine.exe --creds`, the password typed into a hidden prompt and passed to it quoted).
5. Checks: an inbound firewall rule for `sunshine.exe` on every network type (added if missing; Windows puts a Steam Link dongle's network in Public), whether a Steam Link dongle (an adapter "For Valve") is connected, and that the Web UI answers. It ends with what to pick and sign in as on the Frame.
`-Check` only says what it would change, `-SkipLogin` leaves the login, `-Undo` puts back the original exe and the oldest backup of the settings. Backups and a log go to `%ProgramData%\Frametop`.
- Tested on the test PC (admin over SSH, `FRAMETOP_NO_PAUSE=1`): `-Check`; a run with nothing to change (no restart); `-Undo` (the original exe back) and a run with `-FrametopBuild D:\vp-build\src\build\sunshine.exe` (backed up, swapped, Vibepollo restarted, the streams came back on their own). Not tested: a PC without Vibepollo (the download and its installer), setting the login (needs the user at the PC), adding the firewall rule.
- Still to decide: where other PCs get Frametop's build. `$BuildUrl` is empty; publishing it needs the fork's source published with it (GPL-3.0).
- Where other PCs get Frametop's build: the fork's GitHub releases, with the source as the release's tag (GPL-3.0). See below (2026-10-09).
- The Frame side: `install.sh` now builds ft-stream (`stream/build.sh`, step 7) with Remote Displays' menu entry, and `uninstall.sh` offers to delete `~/.local/share/frametop-stream` (the clients' keys, the hosts' tokens and pins) with the other settings.
- Linux hosts (Vibepollo's Arch package): a host setup like this one, later.
- After Vibepollo restarted, a stream checked the dongle in 300 ms, too soon, and went over the network; ft-stream now gives it a second.
**Frametop's build released (2026-10-09):**
- Release [`frametop-2.0.0-1`](https://github.com/Frametop/frametop-vibepollo/releases/tag/frametop-2.0.0-1) of Frametop/frametop-vibepollo has `sunshine.exe` and the Arch package (`pacman -U`), each with its `.sha256`. The fork's CI (`frametop-build.yml`, Depot's runners) builds them from the tag and publishes them; a `frametop-*` tag is what makes a release. The tag is the source.
- The host setup's `$BuildUrl` points at that `sunshine.exe`, so a PC needs only `host/windows`. It replaces the original, the hand-built `2f032252`, and the first CI build (`a84b6cfc`).
- New in it: a Frametop display stream that asks for SDR turns the display's HDR off while it captures it, and back on when the stream ends (with `dd_hdr_option` automatic, Vibepollo's default). The test PC's HDR monitor looked washed out in SDR: Vibepollo's conversion clips at 80 nits while Windows draws SDR content at its SDR white level (240 nits there), and scaling for that left the colours heavily oversaturated. Displays it turned off are listed in `config\frametop_display_hdr.json` until they're back on, so the next start turns them back on after a crash or a forced stop. On the test PC's dev build: HDR off 0.4 s after the stream started, 8-bit capture, back on at the end, off again on a reconnect.
- Tested on the test PC: with the first CI build installed and only `host/windows` in a folder, `-Check` downloaded the release and matched its SHA-256, and the run backed up, swapped and restarted Vibepollo; the Web UI answered.
**Tested in the live desktop (frame-testbench, the 3D mouse through `@ft_pointer_helper`, 2026-10-07):**
- An Explorer window carried by its title bar from the Remote Monitor to the OLED stayed there, and carried back, stayed there too. The log showed one change of screen each way.
- Disconnect, connect: the panel came back where it was, at its width. Connect in a new run with the headset off: placed from screen 1's anchor.
@@ -477,5 +483,4 @@ The user decided remote displays are Frametop displays, with the same controls a
- Which of the test PC's two desk monitors is the real one (the main session)?
- The Mac's chip (Max chips have two video encode engines) and BetterDisplay Pro matter only once the Mac goes past one display.
- Audio: none, the focused display's host, or a fixed one?
- Does a remote display belong to a profile, so switching profiles connects and disconnects streams?
- Remote displays outside the desktop: should they also show over games, where the decoder is shared with Steam Link VR?
+3 -3
View File
@@ -34,7 +34,7 @@ It uses Valve's SteamOS Devkit path: pair once with the headset's devkit service
### The password
FrameDrop has no way to pass a password along, and the zip is the same file for everyone, so the password is typed on the headset, in the window. It stays in the window's memory until the install ends:
FrameDrop has no way to pass a password along, and the zip is the same file for everyone, so the password is typed on the headset, in the window. In VR that means the window's own keypad (shown when Steam starts the window; its Keypad button shows or hides it), whose keys you click with the controller's laser. SteamVR's keyboard doesn't come up for the field by itself, and when it's opened (`steam://open/keyboard`) its keys don't reach the window (tested with frame-testbench, 2026-10-07: the field stayed empty, while laser clicks on the window's checkboxes worked). A Bluetooth keyboard types as usual. The password stays in the window's memory until the install ends:
- The service gets `SUDO_ASKPASS=installer/askpass`. When install.sh's sudo asks, askpass connects to a socket the window keeps in `/run/user/UID/frametop-install` (mode 0700), and the window answers only a process in the install's own service (checked by its peer credentials and cgroup). If it doesn't have the password yet (the window was opened again), it asks you for it, or you skip that part.
- The password is never written to a file, a log, the service's environment, or a command line. The window wipes its copy when the install ends or the window closes. Strings Python and GTK made from it along the way can't be wiped; they go with the process.
@@ -62,11 +62,11 @@ framedrop/build.sh --image REF --version V [--commit SHA] [--channel C] [ZIP_URL
framedrop/build.sh [ZIP_URL] # a test zip
```
This writes `framedrop/build/Frametop.zip` (reproducible), `frametop.framedrop.json` (FrameDrop's manifest with the zip's sha256), and `SHA256SUMS`. A release's zip has the image REF (`podman save`) with its `frametop-release.json` (`pack/release-info.py`) and `install-release.sh`; CI builds it on a tag (`.github/workflows/release.yml`). A test zip has `get.sh` instead. By default, `ZIP_URL` is the release's asset (`releases/download/vV/Frametop.zip`; for a test zip, a `framedrop-installer` release's). Each release carries its manifest, so the button's link can point at the newest stable one: `https://framedropvr.com/install?manifest=https://github.com/DeeJanuz/frametop/releases/latest/download/frametop.framedrop.json` (the exact URL format is FrameDrop's to confirm).
This writes `framedrop/build/Frametop.zip` (reproducible), `frametop.framedrop.json` (FrameDrop's manifest with the zip's sha256), and `SHA256SUMS`. A release's zip has the image REF (`podman save`) with its `frametop-release.json` (`pack/release-info.py`) and `install-release.sh`; CI builds it on a tag (`.github/workflows/release.yml`). A test zip has `get.sh` instead. By default, `ZIP_URL` is the release's asset (`releases/download/vV/Frametop.zip`; for a test zip, a `framedrop-installer` release's). Each release carries its manifest, so the button's link can point at the newest stable one: `https://framedropvr.com/install?manifest=https://github.com/Frametop/frametop/releases/latest/download/frametop.framedrop.json` (the exact URL format is FrameDrop's to confirm).
## Open questions, for a test with FrameDrop on a Windows PC
- Does FrameDrop keep or set the exec bit on `frametop-install.sh`? A zip unpacked on Windows loses it, and without it nothing runs.
- What start command does FrameDrop pick for this zip, and which runtime?
- Does the manifest's `name` become the Devkit Game name? It has to stay free of `-`.
- Can you type the password in the window in VR: does the SteamVR keyboard come up for its password field?
- Typing the password with the window's keypad, launched with Play from the library (frame-testbench reached the window only when started with devkit.sh run, where the systemui overlay hides its lower half).
+2 -2
View File
@@ -40,14 +40,14 @@ files=("$here/installer/frametop-install.sh" "$here/installer/progress.py" "$her
if [ -n "$image" ]; then
[ -n "$version" ] || { echo "--image needs --version" >&2; exit 2; }
commit=${commit:-$(git -C "$repo" rev-parse HEAD)}
url=${1:-https://github.com/DeeJanuz/frametop/releases/download/v$version/Frametop.zip}
url=${1:-https://github.com/Frametop/frametop/releases/download/v$version/Frametop.zip}
echo "saving $image"
podman save -q --format oci-archive -o "$out/frametop-image.tar" "$image"
python3 "$repo/pack/release-info.py" --image-file "$out/frametop-image.tar" --version "$version" \
--commit "$commit" ${channel:+--channel "$channel"} >"$out/frametop-release.json"
files+=("$repo/pack/install-release.sh" "$out/frametop-release.json" "$out/frametop-image.tar")
else
url=${1:-https://github.com/DeeJanuz/frametop/releases/download/framedrop-installer/Frametop.zip}
url=${1:-https://github.com/Frametop/frametop/releases/download/framedrop-installer/Frametop.zip}
files+=("$repo/get.sh")
fi
+84 -9
View File
@@ -199,10 +199,82 @@ class Askpass:
SOCK.unlink(missing_ok=True)
class Keypad(Gtk.Box):
"""Keys to click with the controller's laser, for a password field: in VR, SteamVR's own
keyboard comes up for a Steam title's window but its keys don't reach it (tested
2026-10-07), while clicks on the window's buttons do. A physical keyboard types as usual."""
LETTERS = ("1234567890", "qwertyuiop", "asdfghjkl", "zxcvbnm")
SYMBOLS = ("!@#$%^&*()", "-_=+[]{}\\|", ";:'\",.<>/?", "`~")
def __init__(self, entry):
super().__init__(orientation=Gtk.Orientation.VERTICAL, spacing=4)
self.entry = entry
self.shift = self.symbols = False
self.rows = []
for _ in range(4):
row = Gtk.Box(spacing=4, halign=Gtk.Align.CENTER, homogeneous=True)
self.rows.append(row)
self.append(row)
bottom = Gtk.Box(spacing=4, halign=Gtk.Align.CENTER)
for label, action in (("Shift", self.toggle_shift), ("!#1", self.toggle_symbols),
("Space", lambda *_: self.type(" ")), ("Delete", self.backspace)):
b = Gtk.Button(label=label)
b.set_size_request(110 if label != "Space" else 260, 48)
b.connect("clicked", action)
bottom.append(b)
if label == "!#1":
self.symbols_key = b
self.append(bottom)
self.fill()
def fill(self):
for row, keys in zip(self.rows, self.SYMBOLS if self.symbols else self.LETTERS):
while (child := row.get_first_child()) is not None:
row.remove(child)
for k in keys:
k = k.upper() if self.shift and not self.symbols else k
b = Gtk.Button(label=k)
b.set_size_request(56, 48)
b.connect("clicked", lambda _b, k=k: self.type(k))
row.append(b)
def type(self, text):
self.entry.set_text(self.entry.get_text() + text)
self.entry.set_position(-1)
def backspace(self, *_):
self.entry.set_text(self.entry.get_text()[:-1])
self.entry.set_position(-1)
def toggle_shift(self, *_):
self.shift = not self.shift
self.fill()
def toggle_symbols(self, *_):
self.symbols = not self.symbols
self.symbols_key.set_label("abc" if self.symbols else "!#1")
self.fill()
def keypad_row(entry, *extra):
"""The password field, a button that shows or hides the keypad, and the keypad: shown by
itself when Steam started this (FrameDrop's title, so most likely in VR)."""
keypad = Keypad(entry)
keypad.set_visible("SteamAppId" in os.environ)
toggle = Gtk.Button(label="Keypad")
toggle.connect("clicked", lambda *_: keypad.set_visible(not keypad.get_visible()))
row = Gtk.Box(spacing=8)
entry.set_hexpand(True)
for w in (entry, toggle, *extra):
row.append(w)
return row, keypad
class Window(Adw.ApplicationWindow):
def __init__(self, app):
super().__init__(application=app, title="Frametop")
self.set_default_size(900, 640)
self.set_default_size(900, 860 if "SteamAppId" in os.environ else 640)
self.offset = 0
self.partial = ""
self.askpass = None
@@ -228,7 +300,7 @@ class Window(Adw.ApplicationWindow):
# --- what to install, and the password
def show_choices(self):
what = f"Frametop {release_version()}, built," if RELEASE else "Frametop from GitHub"
what = f"Frametop {release_version()}" if RELEASE else "Frametop from GitHub"
self.detail.set_label(f"This installs {what}: the multi-screen desktop, the 3D mouse, gaze "
"mode, and Frametop's settings apps. Two optional parts need your "
"SteamOS password (sudo):")
@@ -246,18 +318,20 @@ class Window(Adw.ApplicationWindow):
self.go.add_css_class("suggested-action")
self.go.connect("clicked", self.install)
self.pw.connect("activate", self.install)
self.choices = [self.eye, self.bt, self.pw, pw_note, self.error, self.go]
pw_row, keypad = keypad_row(self.pw)
self.choices = [self.eye, self.bt, pw_row, keypad, pw_note, self.error, self.go]
if not password_set():
for c in (self.eye, self.bt):
c.set_active(False)
c.set_sensitive(False)
self.pw.set_visible(False)
pw_row.set_visible(False)
keypad.set_visible(False)
pw_note.set_label("Your user has no password, so sudo can't run and these two can't be "
"installed from here. To set one, run passwd in Konsole; then play "
"Frametop again, or install them later from a terminal "
"(gaze/tracker/install.sh, setup/bluetooth/install.sh).")
for c in (self.eye, self.bt):
c.connect("toggled", lambda *_: self.pw.set_sensitive(self.eye.get_active() or self.bt.get_active()))
c.connect("toggled", lambda *_: pw_row.set_sensitive(self.eye.get_active() or self.bt.get_active()))
for w in self.choices:
self.box.append(w)
@@ -328,16 +402,17 @@ class Window(Adw.ApplicationWindow):
scroll = Gtk.ScrolledWindow(vexpand=True, child=self.text)
# The install wants the password and the window doesn't have it (played again).
self.ask_bar = Gtk.Box(spacing=8, visible=False)
self.ask_pw = Gtk.PasswordEntry(show_peek_icon=True, hexpand=True,
self.ask_pw = Gtk.PasswordEntry(show_peek_icon=True,
placeholder_text="The next step needs your SteamOS password")
ok = Gtk.Button(label="OK")
skip = Gtk.Button(label="Skip that part")
ok.connect("clicked", self.answer_ask)
self.ask_pw.connect("activate", self.answer_ask)
skip.connect("clicked", self.skip_ask)
for w in (self.ask_pw, ok, skip):
self.ask_bar.append(w)
ask_row, ask_keypad = keypad_row(self.ask_pw, ok, skip)
self.ask_bar = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=8, visible=False)
self.ask_bar.append(ask_row)
self.ask_bar.append(ask_keypad)
self.ask_error = Gtk.Label(xalign=0, wrap=True, visible=False)
self.ask_error.add_css_class("error")
+2 -2
View File
@@ -61,8 +61,8 @@ done
section "what an installer needs"
check "home writable" sh -c 'f=$HOME/.cache/frametop-framedrop/.w && : >"$f" && rm "$f" && echo yes'
check "~/frametop visible" sh -c 'ls -d "$HOME/frametop" && git -C "$HOME/frametop" log -1 --format=%h'
have git && check "git ls-remote github" git ls-remote --heads https://github.com/DeeJanuz/frametop.git experimental
have curl && check "curl get.sh" sh -c 'curl -fsSL https://deejanuz.github.io/frametop/get.sh | head -1'
have git && check "git ls-remote github" git ls-remote --heads https://github.com/Frametop/frametop.git experimental
have curl && check "curl get.sh" sh -c 'curl -fsSL https://frametop.github.io/frametop/get.sh | head -1'
have podman && check "podman ps" podman ps --format '{{.Names}}'
have distrobox && check "distrobox list" distrobox list
have systemctl && check "systemctl --user" systemctl --user is-system-running
+3 -2
View File
@@ -26,6 +26,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes:
- **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
- **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
- **One eye:** SteamOS 0.4's Track Dominant Eye Only (SteamVR's settings, General, with Show advanced settings on; `steamvr.eyeTrackingDominantEyeOnly` with `steamvr.dominantEye`) makes SteamVR's tracker ignore the other eye. With SteamVR's tracker, Frametop then goes by that eye alone: the calibration and the checks wait only for it, the gaze is that eye's own reading once it's calibrated (SteamVR's combined gaze before, which follows that eye then), and the fit check shows the other eye as not tracked. The service reads the setting again when SteamVR's settings file changes. Not yet tried in the headset: what SteamVR's shared memory says about the ignored eye wasn't measured.
With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
@@ -34,7 +35,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
- **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow: the middle, and 12 degrees left and right and 9 up and down, in a see-through panel 40 degrees wide (the quick check's 16 degree square left four of them off the panel, unseen). The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A button mapped to gaze precision or gaze drag counts as the left click there (before 2026-10-09 the panel ignored it). Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). The dot's ring shows full while ft-eyes checks it; the service doesn't wait for that answer (before 2026-10-09 it did, up to 3 s a dot, so the pointer could come back over the panel, and a slow answer closed the calibration as if the headset came off). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
- **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there.
@@ -55,7 +56,7 @@ The tracker stops when the headset is off your head. SteamVR also calibrates gaz
`gaze/tracker/` is an eye tracker of our own, because SteamVR's is about 1.5 degrees off after the best correction the gaze service can learn, and what's left is mostly look-to-look noise that no correction on top of its output can remove. Ours processes the eye cameras itself: 0.59 degrees (median) in its best live session against 0.83 for SteamVR's with the probe's correction, and after the headset was taken off and put back without recalibrating, 0.58 once your first clicks had taught it where the headset sat (`tracker/findings.md` has the measurements).
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too).
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too). It also adds it to SteamOS's update keep list (`/etc/atomic-update.conf.d/frametop-eyegrab.conf`): an update deletes `/etc` files that list doesn't name, and without the grabber gaze mode falls back to SteamVR's tracker and its calibration. `scripts/doctor.sh` reports a grabber an update deleted.
- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`.
- `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones are meant for a PC: if you have `frame-job` (a personal tool, not in this repo), `gaze/tracker/.frame-job` sends them there. `lab/py` runs them with that Python (in the dev container on the Frame; on a PC, the same venv from `requirements.txt`, which frame-job's setup makes).
+17 -4
View File
@@ -51,7 +51,10 @@ GUIDE = [
class FitCheck:
def __init__(self):
def __init__(self, ignore=None):
# The eye SteamVR's tracker ignores (0 left, 1 right) with Track Dominant Eye Only on
# (gazecal.tracked_eye), or None: its losses say nothing about the fit.
self.ignore = ignore
self.reset()
def reset(self):
@@ -99,7 +102,8 @@ class FitCheck:
if q and (eye.get("new") or [1, 1])[k]:
self.q[k].append(q[k])
closed = [opens[k] < CLOSED for k in (0, 1)]
if all(closed) or all(self.lost):
judged = [k for k in (0, 1) if k != self.ignore]
if all(closed[k] for k in judged) or all(self.lost[k] for k in judged):
return # a blink: says nothing about the fit
key = (math.floor(hy / CELL), math.floor(hp / CELL))
for k in (0, 1):
@@ -143,6 +147,8 @@ class FitCheck:
# --- Summaries ---
def status(self, k):
if k == self.ignore:
return "not tracked", (0.6, 0.6, 0.6)
if not self.samples:
return "no data", (0.6, 0.6, 0.6)
if self.lost[k]:
@@ -174,8 +180,13 @@ class FitCheck:
return ["Look around slowly (the screen's corners, then down at your keyboard, up, left and right) "
"or press Enter for a guided check."]
out = []
if self.ignore is not None:
out.append(f"SteamVR tracks only your {EYES[1 - self.ignore].lower()} (Track Dominant Eye Only in "
f"SteamVR's settings), so your {EYES[self.ignore].lower()} doesn't count here.")
bad = {}
for k in (0, 1):
if k == self.ignore:
continue
for key, words, _ in REGIONS:
share = self.region_share(k, key)
if share is not None and share >= 0.15:
@@ -197,7 +208,7 @@ class FitCheck:
"face, not one eye's fit.")
continue
k = next(iter(eyes))
other = self.region_share(1 - k, key)
other = self.region_share(1 - k, key) if 1 - k != self.ignore else None
vs = f", the {EYES[1 - k].lower()} {other:.0%}" if other is not None else ""
line = f"{EYES[k]}: lost {eyes[k]:.0%} of the time looking {words}{vs}."
if key == "down":
@@ -211,10 +222,12 @@ class FitCheck:
"eyes.")
out.append(line)
s0, s1 = self.signal(0), self.signal(1)
if s0 is not None and s1 is not None and abs(s0 - s1) > 0.25:
if self.ignore is None and s0 is not None and s1 is not None and abs(s0 - s1) > 0.25:
k = 0 if s0 < s1 else 1
out.append(f"The tracker is less sure of your {EYES[k].lower()} even when it has it "
f"(signal {min(s0, s1):.0%} against {max(s0, s1):.0%}).")
if self.ignore is not None and len(out) == 1:
out.append(f"Your {EYES[1 - self.ignore].lower()} is tracked everywhere you've looked so far.")
if not out:
out.append("Both eyes are tracked everywhere you've looked so far.")
return out
+9 -8
View File
@@ -98,10 +98,11 @@ double NowRaw() {
}
// --- eye-server.mmap (packed, unaligned: read with memcpy) ---
// The SteamOS 0.4.x beta (SteamVR 2.18.2) moved every field from the timestamp on by 5
// bytes (measured 2026-10-04 with the ftdiag scan: timestamp 0x157 -> 0x15c, the vectors
// moved with it; the counter at 0x38 kept its place). Which layout is live is detected at
// runtime (EyeFile::Detect), so one binary serves both generations.
// SteamOS 0.4 (SteamVR 2.18.2; first on the 0.4.3 beta, the same on 0.4.5, the release)
// moved every field from the timestamp on by 5 bytes (measured 2026-10-04 with the ftdiag
// scan: timestamp 0x157 -> 0x15c, the vectors moved with it; the counter at 0x38 kept its
// place). Which layout is live is detected at runtime (EyeFile::Detect), so one binary
// serves both generations.
constexpr size_t kCounter = 0x38; // u32, one per sample
constexpr size_t kTime = 0x157; // f64, CLOCK_MONOTONIC_RAW seconds
constexpr size_t kLeft1 = 0x15f, kRight1 = 0x16b; // set 1: unit vectors, head space
@@ -115,11 +116,11 @@ constexpr size_t kVar1 = 0x177, kVar2 = 0x1b3;
// x, y, right x, y). An eye's pair stops changing while the tracker can't see it.
constexpr size_t kMeas = 0x1d3;
constexpr size_t kNeed = 0x1f3 + 5; // enough for either layout
constexpr size_t kShifts[] = {0, 5}; // the layouts EyeFile::Detect knows: stable, 0.4.x beta
constexpr size_t kShifts[] = {0, 5}; // the layouts EyeFile::Detect knows: SteamOS 0.3, 0.4
struct EyeFile {
// Everything from the timestamp on is read at base + shift: 0 on stable, 5 on the
// 0.4.x beta (see the constants above). known once Detect has seen that layout's
// Everything from the timestamp on is read at base + shift: 0 on SteamOS 0.3, 5 on 0.4
// (see the constants above). known once Detect has seen that layout's
// timestamp tick; before that, reading would yield garbage that still passes
// ReadSample's check.
size_t shift = 0;
@@ -673,7 +674,7 @@ int main(int argc, char **argv) {
if (haveMmap && !eyes.known && (eyes.Detecting() || (writing && now >= nextLayoutCheck))) {
const EyeFile::Detection d = eyes.Detect(now);
if (d == EyeFile::kFound) {
std::fprintf(stderr, "ft-gaze: eye-server.mmap layout: %s\n", eyes.shift ? "beta (+5)" : "stable");
std::fprintf(stderr, "ft-gaze: eye-server.mmap layout: %s\n", eyes.shift ? "SteamOS 0.4 (+5)" : "SteamOS 0.3");
} else if (d == EyeFile::kNone) {
nextLayoutCheck = now + 1.0;
if (!missSince) missSince = now;
+8
View File
@@ -6,6 +6,9 @@
ft-gazectl status the gaze service (ft-gazed): samples, lessons, the correction
ft-gazectl forget drop what the pointer's lessons taught (the calibration stays)
ft-gazectl reload the service reads calibration.json again
ft-gazectl quickcal|calibrate|fitcheck|fivecheck
open that check in the headset panel: the one-dot check, the
full calibration, the headset fit check, the five-dot check
"""
import json
@@ -55,6 +58,11 @@ def main():
print(json.dumps(json.loads(r), indent=1))
except ValueError:
print(r)
elif cmd in ("quickcal", "calibrate", "fitcheck", "fivecheck"):
r = ask("ft_gazed", cmd)
if r is None:
sys.exit("the gaze service (ft-gazed) isn't running")
print(r)
else:
sys.exit(__doc__)
+52 -9
View File
@@ -103,8 +103,11 @@ Control socket: abstract unix datagram "@ft_gazed":
quickcal the one-dot check now (the calibration if there's none)
calibrate the full calibration in the panel
fitcheck the headset fit check in the panel
fivecheck the five-dot check now (it otherwise follows a quick check
that didn't fix the tracker)
calaccept | calquit from the pointer helper while the panel is up: take this dot
now (a left click, Meta+J) | close it (a right click, Meta+K)
now (a left click, Meta+J, or a gaze precision or gaze drag
press) | close it (a right click, Meta+K)
Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever
the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever
@@ -129,7 +132,7 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correction, EyeFallback, # noqa: E402
EyeWeights, Fixation, LiveCorrection, SteamEyeLog)
EyeWeights, Fixation, LiveCorrection, SteamEyeLog, TrackedEye)
from gazecheck import Checks # noqa: E402
REPO = Path(__file__).resolve().parents[1]
@@ -237,6 +240,9 @@ class Service:
self.opens = (deque(maxlen=90), deque(maxlen=90)) # left, right
self.vergence = deque(maxlen=90)
self.fallback = EyeFallback()
# SteamVR's tracker following one eye only (gazecal.tracked_eye): 0 left, 1 right, or None.
self.tracked = TrackedEye()
self.one_eye = self.tracked()
self.lost = [False, False]
self.bad_at = [0.0, 0.0] # sample time an eye was last lost or closed
self.counts = {"samples": 0, "sent": 0, "blinks": 0, "one_eye": 0, "one_eye_used": 0, "lost_left": 0,
@@ -281,7 +287,9 @@ class Service:
return "source"
if self.tracker == "own":
return "own"
return "eyes" if all(self.models[e].samples for e in SIDES) else "source"
# With one eye tracked, that eye's own calibration is enough.
sides = SIDES if self.one_eye is None else (SIDES[self.one_eye],)
return "eyes" if all(self.models[e].samples for e in sides) else "source"
# --- Settings, calibration and lessons ---
@@ -292,6 +300,9 @@ class Service:
auto = ""
if self.tracker_setting == "auto":
auto = " (auto: ours is installed)" if tracker == "own" else " (auto: ours isn't installed)"
if tracker == "steam" and EYEGRAB[1].exists() and not EYEGRAB[0].exists():
auto = (f" (auto: ours lost {EYEGRAB[0]}, which SteamOS updates delete unless it's kept;"
" reinstall it with gaze/tracker/install.sh)")
log(f"tracker {tracker}{auto}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
self.tracker, self.bias = tracker, bias
for w in self.weights.values():
@@ -638,11 +649,17 @@ class Service:
for k in (0, 1):
self.lost[k] = unc[k] > (EYE_FOUND if self.lost[k] else EYE_LOST)
if not down:
self.counts["lost_left"] += self.lost[0]
self.counts["lost_right"] += self.lost[1]
self.counts["lost_left"] += self.lost[0] and self.one_eye != 1
self.counts["lost_right"] += self.lost[1] and self.one_eye != 0
return low
def on_sample(self, s):
one = self.tracked()
if one != self.one_eye:
log("SteamVR now tracks both eyes" if one is None else
f"SteamVR tracks the {SIDES[one]} eye only (Track Dominant Eye Only): going by that eye")
self.one_eye = one
self.fix.reset()
self.checks.on_sample(s)
kind = self.kind
if kind != self.last_kind:
@@ -669,7 +686,8 @@ class Service:
eyes = [(p["hy"], p["hp"]) if "hy" in p else None for p in per]
if not any(eyes):
return
hp, hit = next(e[1] for e in eyes if e), m1.get("hit")
one = self.one_eye
hp, hit = (eyes[one] if one is not None and eyes[one] else next(e for e in eyes if e))[1], m1.get("hit")
self.counts["samples"] += 1
self.last_sample = time.monotonic()
down = hp < KEYBOARD_PITCH and not hit
@@ -677,8 +695,14 @@ class Service:
if down:
self.counts["looking_down"] += 1
return
if own and self.one_eye is not None:
# SteamVR judges only that eye's openness, and a blink closes both: ours still
# sees the other, so it stays in.
low[1 - self.one_eye] = low[self.one_eye]
# Our tracker finds the pupils itself; SteamVR's openness still marks the blinks.
bad = [eyes[k] is None or low[k] or (not own and self.lost[k]) for k in (0, 1)]
if not own and self.one_eye is not None:
bad[1 - self.one_eye] = True # SteamVR ignores that eye
if all(bad):
self.counts["blinks"] += 1
return
@@ -713,11 +737,27 @@ class Service:
self.bad_at[k] = s["t"] # the fallback doesn't learn from these either
return
bad = [low[k] or self.lost[k] for k in (0, 1)]
hy, hp = src["hy"], src["hp"]
eyes = (s["src"].get("mmap2") or {}).get("eyes")
one = self.one_eye
if one is not None:
# SteamVR's combined gaze already follows that eye alone; set 2's averages the
# ignored one in, so mmap2 takes the eye's own reading. No fallback to learn.
if bad[one]:
self.counts["blinks"] += 1
return
if self.source == "mmap2":
if not eyes:
self.counts["dropped"] += 1
return
hy, hp = eyes[one]
fy, fp = self.fix(hy, hp, s["t"], 1.0)
cy, cp = self.correction(self.source, fy, fp)
self.send(s["t"], fy + cy, fp + cp, fy, fp, None)
return
if all(bad):
self.counts["blinks"] += 1
return
hy, hp = src["hy"], src["hp"]
eyes = (s["src"].get("mmap2") or {}).get("eyes")
for k in (0, 1):
if bad[k]:
self.bad_at[k] = s["t"]
@@ -788,7 +828,8 @@ class Service:
elif words[:1] == ["forget"]:
self.forget_lessons()
reply = "ok"
elif words[:1] and words[0] in ("quickcal", "calibrate", "fitcheck", "calaccept", "calquit", "recheck"):
elif words[:1] and words[0] in ("quickcal", "calibrate", "fitcheck", "fivecheck", "calaccept", "calquit",
"recheck"):
reply = self.checks.command(words)
elif words[:1] == ["eyes"] and len(words) == 2:
try:
@@ -904,6 +945,8 @@ class Service:
self.on_control()
elif key.data == "checks":
self.checks.on_readable()
elif key.data == "eyes_reply":
self.checks.on_eyes_reply(key.fileobj)
elif key.data == "panel" and self.checks.panel_proc:
self.checks.read_panel()
elif key.data == "own":
+70 -3
View File
@@ -7,6 +7,7 @@ the tracker has lost the other), EyeWeights (how much each eye counts), and Stea
reports them.
"""
import json
import math
import os
import statistics
@@ -397,6 +398,63 @@ class LiveCorrection:
EYE_LOST = 0.004
EYE_FOUND = 0.0025
# SteamOS 0.4's "Track Dominant Eye Only" (SteamVR's settings, General, advanced): SteamVR's
# tracker ignores the other eye, for someone whose eyes don't look at the same spot. Its
# settings: steamvr.eyeTrackingDominantEyeOnly, and steamvr.dominantEye (0 left, 1 right,
# SteamVR's default). Frametop then goes by that eye alone: a calibration that waits for both
# eyes would never take a dot, and the other eye's reading isn't where the person looks.
STEAMVR_SETTINGS = (Path.home() / ".config" / "openvr" / "config" / "steamvr.vrsettings",
Path.home() / ".steam" / "steam" / "config" / "steamvr.vrsettings")
def tracked_eye(paths=STEAMVR_SETTINGS):
"""The one eye SteamVR's tracker follows (0 left, 1 right), or None for both. The first
of the settings files that exists counts (SteamVR keeps only settings changed from its
defaults, so a missing key is the default)."""
for path in paths:
try:
text = Path(path).read_text()
except OSError:
continue
try:
steamvr = json.loads(text).get("steamvr")
except (ValueError, AttributeError):
return None
if not isinstance(steamvr, dict) or steamvr.get("eyeTrackingDominantEyeOnly") is not True:
return None
return 0 if steamvr.get("dominantEye", 1) == 0 else 1
return None
class TrackedEye:
"""tracked_eye(), read again when SteamVR's settings file changes (looked at no more than
every CHECK seconds), since the setting can change while a service runs."""
CHECK = 2.0
def __init__(self, paths=STEAMVR_SETTINGS):
self.paths = paths
self.eye = None
self.stamp = None
self.checked = None
def __call__(self, now=None):
now = time.monotonic() if now is None else now
if self.checked is not None and now - self.checked < self.CHECK:
return self.eye
self.checked = now
stamp = []
for path in self.paths:
try:
st = os.stat(path)
stamp.append((st.st_mtime_ns, st.st_size))
except OSError:
stamp.append(None)
if stamp != self.stamp:
self.stamp = stamp
self.eye = tracked_eye(self.paths)
return self.eye
class EyeFallback:
"""The gaze from one eye, while the tracker has lost the other.
@@ -640,7 +698,7 @@ def cross_validate(points, mode):
return errs
def steady_samples(samples, vergence_jump=1.5, why=None):
def steady_samples(samples, vergence_jump=1.5, why=None, eye=None):
"""The samples of one look at one spot where the tracker had both eyes: none in a blink
(openness under half its median over the samples), none where it had lost an eye (its
variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the
@@ -648,28 +706,37 @@ def steady_samples(samples, vergence_jump=1.5, why=None):
vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
fraction of one far off), so only a jump away from what it was during this look means
the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.
`eye` (0 left, 1 right; see tracked_eye) judges that eye alone: the other one's loss,
openness and the vergence don't count.
`why`, a dict, gets how many were dropped for each reason: "lost_left", "lost_right",
"lost_both", "blink" and "vergence" (each sample once, for the first that applies)."""
if why is None:
why = {}
def openness(o):
return o[eye] if eye is not None else min(o)
# Openness: a blink is a sharp drop from what it was during this look. Not a fixed
# level: looking down, the upper lids come down with the eyes, and in bright light you
# squint, so the reading can stay under 0.5 for the whole look while the tracker follows
# the eyes fine (a calibration dot at the bottom of the bright round failed that way).
opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
opens = [openness(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12
seen = []
for smp in samples:
m1 = smp["src"].get("mmap1") or {}
o = m1.get("open")
lost = [u > EYE_LOST for u in m1.get("unc") or [0, 0]]
if eye is not None:
lost[1 - eye] = False
# A lost eye's openness reads 0 too, so a lost eye is named before a blink.
key = ("lost_both" if all(lost) else "lost_left" if lost[0] else "lost_right") if any(lost) else \
"blink" if o and min(o) < floor else None
"blink" if o and openness(o) < floor else None
if key:
why[key] = why.get(key, 0) + 1
else:
seen.append(smp)
if eye is not None:
return seen
def vergence(smp):
return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
+148 -27
View File
@@ -14,7 +14,8 @@ directions: look at each one.
on is seen only while the gaze service is awake (gaze mode on, someone wearing it: see
ft-gazed), so it also opens when gaze mode comes on after the service idled.
five the middle and four around it, when the first FIVE_COUNT lessons after a quick check
were all over FIVE_LIMIT degrees off: the quick check didn't fix it.
were all over FIVE_LIMIT degrees off: the quick check didn't fix it. Its panel is
wider than quick's, to hold them (PANEL_DEG).
full the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil
size, and the tracker's error with it, changes with brightness), each the middle and
a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots)
@@ -45,8 +46,10 @@ the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait
it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason:
gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a
click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common
reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
renewed every second; the helper shows it again by itself when that stops).
reason there and in the status. A right click or Meta+K ("calquit") closes the panel. A press
mapped to gaze precision or gaze drag counts as the left click (pointer/helper/calpanel.h). The
pointer hides meanwhile ("calpanel 1", renewed every second; the helper shows it again by itself
when that stops).
Our tracker's first calibration: before it has one, ft-eyes publishes no gaze (it maps pupils
to a gaze only with a calibration), so there's no gaze to hold still. Its calibration runs
@@ -55,6 +58,12 @@ are enough to start it, each dot stands in for the gaze, and a click takes the C
to it. ft-eyes then checks that each pupil was seen and held still in that window (calib-point)
and says why not. Without this, a fresh install could never calibrate our tracker.
The service doesn't wait for ft-eyes' answers to calib-point and calib-fit (ask_eyes): the dot
shows its ring full, and further clicks do nothing until the answer comes, or its deadline
passes; while it fits, a right click doesn't close the panel either. Until 2026-10-09 it waited, up to 3 s a dot and 10 s for the fit, so the gaze stopped,
the helper's panel lease ran out (the pointer came back, and a click went to the desktop
behind the panel), and an answer over EYES_GONE closed the calibration as the headset coming off.
What a capture teaches:
our tracker quick and five: a click ("click T YAW PITCH", like a pointer lesson); full:
calib-start, a calib-point for each dot, calib-fit (its calibration)
@@ -172,6 +181,26 @@ def reject_reason(reply, why):
return text[:24], f"our tracker said: {text}", False
# The panel's sizes for each check (ft-gazepanel.cpp's kQuickDeg, kFiveDeg, kFullDeg): width in
# degrees, and height / width. Every dot of a check must fit its panel (off_panel).
PANEL_DEG = {"quick": (16.0, 1.0), "five": (40.0, 0.75), "full": (64.0, 0.75)}
def off_panel(kind, own):
"""The dots of a check that wouldn't show whole on its panel: the panel's projection
(ft-gazepanel's ToPixel), with a degree to spare for the dot's ring."""
wdeg, aspect = PANEL_DEG[kind]
half = math.tan(math.radians(wdeg / 2))
m = math.tan(math.radians(1.0)) / (2 * half)
out = []
for yaw, pitch, _ in check_dots(kind, own):
x = 0.5 - math.tan(math.radians(yaw)) / (2 * half)
y = 0.5 - math.tan(math.radians(pitch)) / math.cos(math.radians(yaw)) / (2 * half) / aspect
if not (m <= x <= 1 - m and m / aspect <= y <= 1 - m / aspect):
out.append((yaw, pitch))
return out
def spread(points):
"""The median point and the spread around it (1.4826 x the median distance: a standard
deviation that one stray sample can't move far)."""
@@ -239,6 +268,7 @@ class Checks:
self.panel_restart_at = 0.0
self.screens_shown = None
self.last_progress = 0.0
self.asking = None # (socket, deadline, done): a command to ft-eyes waiting for its reply
@property
def active(self):
@@ -324,6 +354,48 @@ class Checks:
if was != (on, headset):
self.svc.update_awake()
def ask_eyes(self, command, timeout, done):
"""A command to ft-eyes whose reply comes later: done(reply) runs from on_eyes_reply, or with
"" after `timeout` (tick), as ask() gives without one. ask() held the whole service up to 3 s
a dot (calib-point) and 10 s at the end (calib-fit): the gaze stopped, and the helper's 3 s
calpanel lease ran out, so the pointer came back and a click went to the desktop behind the
panel. Each command has its own socket, so a late reply can't be taken for the next one."""
self.drop_ask()
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
try:
s.bind("")
s.sendto(command.encode(), EYES)
except OSError:
s.close()
done("")
return
self.sel.register(s, selectors.EVENT_READ, "eyes_reply")
self.asking = (s, time.monotonic() + timeout, done)
def on_eyes_reply(self, sock):
if not self.asking or self.asking[0] is not sock:
return # dropped already (closed, or timed out in this loop)
try:
reply = sock.recv(4096).decode("utf-8", "replace")
except BlockingIOError:
return
except OSError:
reply = ""
done = self.asking[2]
self.drop_ask()
done(reply)
def drop_ask(self):
if not self.asking:
return
s = self.asking[0]
self.asking = None
try:
self.sel.unregister(s)
except (KeyError, ValueError):
pass
s.close()
# --- State ---
def calibrated(self):
@@ -452,7 +524,10 @@ class Checks:
self.screens_shown = (st[2] == "0") if st[1] == "always" else (st[2] == "1")
ask(SCREENS, "hide", 0.5)
self.to_helper("calpanel 1")
self.to_panel(f"show {'full' if kind == 'full' else 'quick'}")
off = off_panel(kind, own)
if off:
log(f"{kind} check: {len(off)} dots off its panel: {off}")
self.to_panel(f"show {kind}")
self.show_dot()
if kind == "quick":
self.last_quick = now
@@ -463,9 +538,10 @@ class Checks:
if time.monotonic() - self.sample_at > 2:
return "error the headset is off or the eye tracker isn't sending"
now = time.monotonic()
ignore = None if self.svc.one_eye is None else 1 - self.svc.one_eye # the eye SteamVR ignores
self.check = {"kind": "fit", "reason": reason, "own": False, "dots": [], "i": 0, "started": now, "shown": now,
"run": [], "accept": False, "done_at": None, "tries": 0, "skipped": 0, "captured": 0,
"points": {}, "fit": FitCheck(), "drawn": {}, "drawn_at": 0.0, "step": None}
"points": {}, "fit": FitCheck(ignore=ignore), "drawn": {}, "drawn_at": 0.0, "step": None}
log(f"fit check: {reason}")
self.to_helper("calpanel 1")
self.to_panel("show fit")
@@ -535,7 +611,8 @@ class Checks:
if self.pending:
self.run_pending()
unc = (s["src"].get("mmap1") or {}).get("unc")
if unc and min(unc) <= EYE_LOST:
one = self.svc.one_eye
if unc and (min(unc) if one is None else unc[one]) <= EYE_LOST:
self.seen_at = time.monotonic()
if self.away and self.back_since is None:
self.back_since = self.seen_at
@@ -543,8 +620,8 @@ class Checks:
if c and c["kind"] == "fit":
c["fit"].feed(s, self.sample_at)
return
if not c or c["done_at"]:
return
if not c or c["done_at"] or self.asking:
return # (asking: ft-eyes has this dot's look, or the fit)
if c["own"]:
src = s["src"].get("own") or {}
if c["blind"]:
@@ -612,9 +689,12 @@ class Checks:
rec.update(eyes=eyes, miss=miss)
t0, t1 = samples[0]["t"], samples[-1]["t"]
if c["kind"] == "full":
reply = ask(EYES, f"calib-point {t0:.6f} {t1:.6f} {yaw:.4f} {pitch:.4f}", 3.0)
rec["reply"] = reply
ok = reply.startswith("ok")
# ft-eyes checks the pupils held still: its answer takes the dot or not (point_done).
# The ring shows it full meanwhile, so the click is seen to have landed.
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} capture 1.00")
self.ask_eyes(f"calib-point {t0:.6f} {t1:.6f} {yaw:.4f} {pitch:.4f}", 3.0,
lambda reply: self.point_done(rec, miss, reply))
return
else:
try:
svc.eyes_sock.sendto(f"click {t1:.6f} {yaw:.4f} {pitch:.4f}".encode(), EYES)
@@ -624,7 +704,8 @@ class Checks:
svc.weights["own"].add(miss)
else:
why = {}
steady = steady_samples(samples, why=why)
one = svc.one_eye
steady = steady_samples(samples, why=why, eye=one)
rec["dropped"] = why
reads = {}
for name in ("action", "mmap1", "mmap2", "left", "right"):
@@ -632,11 +713,20 @@ class Checks:
if "hy" in (smp["src"].get(name) or {})]
if len(pts) >= 15:
reads[name] = (statistics.median(p[0] for p in pts), statistics.median(p[1] for p in pts))
if one is not None:
# SteamVR tracks one eye (gazecal.tracked_eye): the other's reading isn't where
# you look, and set 2's average has it in, so mmap2 is that eye's own (as ft-gazed
# sends it then).
reads.pop(("left", "right")[1 - one], None)
reads.pop("mmap2", None)
if ("left", "right")[one] in reads:
reads["mmap2"] = reads[("left", "right")[one]]
rec["reads"] = reads
main = ("left", "right") if svc.kind == "eyes" else (svc.source,)
main = (("left", "right") if one is None else (("left", "right")[one],)) if svc.kind == "eyes" else (svc.source,)
if not all(n in reads for n in main):
ok = False
rec["reply"] = f"only {len(steady)} of {len(samples)} samples had both eyes"
rec["reply"] = f"only {len(steady)} of {len(samples)} samples had " + (
"both eyes" if one is None else f"your {('left', 'right')[one]} eye")
elif c["kind"] == "full":
for name, (hy, hp) in reads.items():
c["points"].setdefault(name, []).append((hy, hp, yaw - hy, pitch - hp))
@@ -656,9 +746,23 @@ class Checks:
svc.lives[name].add({"time": time.time(), "hy": hy, "hp": hp, "dy": yaw - hy, "dp": pitch - hp,
"wy": 1.0, "wp": 1.0, "how": "check"}, svc.models[name], svc.mode)
rec["miss"] = miss
if svc.kind == "eyes":
if svc.kind == "eyes" and len(miss) == 2: # (one eye tracked: nothing to weigh)
svc.weights["steam"].add(miss)
svc.dirty = True
self.captured(rec, ok)
def point_done(self, rec, miss, reply):
"""ft-eyes' answer to a full calibration dot's calib-point ("" without one)."""
rec["reply"] = reply
ok = reply.startswith("ok")
if ok:
self.svc.weights["own"].add(miss)
self.captured(rec, ok)
def captured(self, rec, ok):
"""A capture is over: the dot is taken, tried again, or skipped."""
c = self.check
yaw, pitch, _ = c["dots"][c["i"]]
if not ok:
short, long, fit = reject_reason(rec.get("reply", "") if c["own"] else None, rec.get("dropped"))
rec["reason"] = long
@@ -670,9 +774,9 @@ class Checks:
if not ok:
c["tries"] += 1
log(f"{c['kind']} check dot {c['i'] + 1}: not taken: {rec['reason']} ({rec.get('reply', '')})")
full = c["kind"] == "full"
full, big = c["kind"] == "full", c["kind"] != "quick" # quick's panel holds only short notes
if c["tries"] >= 2 or not full:
self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if full else f"Skipped: {short}")
self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if big else f"Skipped: {short}")
self.skip()
else:
self.note(f"Not taken: {long}. Look at the dot and click again")
@@ -737,8 +841,15 @@ class Checks:
return
self.full_failed = None
if c["own"]:
reply = ask(EYES, "calib-fit", 10.0)
log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
def fitted(reply):
log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
self.close()
self.to_panel("text Saving the calibration")
c["done_at"] = None # (tick would advance past the last dot again)
c["fitting"] = True
self.ask_eyes("calib-fit", 10.0, fitted)
return
else:
mode = svc.mode if svc.mode != "none" else DEFAULT_MODEL
for name, pts in c["points"].items():
@@ -761,6 +872,7 @@ class Checks:
return
if why:
log(f"{self.check['kind']} check closed: {why}")
self.drop_ask()
self.to_panel("hide")
self.to_helper("calpanel 0")
if self.check["kind"] == "full" and self.screens_shown:
@@ -770,8 +882,8 @@ class Checks:
def quit(self):
c = self.check
if not c:
return
if not c or c.get("fitting"):
return # (fitting: ft-eyes has every dot; the panel closes once it answers)
self.close("quit")
if c["kind"] == "full" and self.calibrated() is False:
# No calibration still: gaze mode can't work, so it goes off until it's turned on again.
@@ -807,9 +919,9 @@ class Checks:
log(f"{words[0]}, asked for while idle: {reply.removeprefix('error ')}")
def command(self, words, queue=True):
"""quickcal, calibrate, calaccept, calquit -> a reply."""
"""quickcal, calibrate, fitcheck, fivecheck, calaccept, calquit -> a reply."""
cmd = words[0]
if cmd in ("quickcal", "calibrate", "fitcheck") and queue and self.svc.waking() and not self.check:
if cmd in ("quickcal", "calibrate", "fitcheck", "fivecheck") and queue and self.svc.waking() and not self.check:
# The tracker isn't running (or only just started): wake it, and do this once it sends.
self.pending = (words, time.monotonic())
self.svc.update_awake()
@@ -820,10 +932,12 @@ class Checks:
return self.start("full", "asked for")
if cmd == "fitcheck":
return self.start("fit", "asked for")
if cmd == "fivecheck":
return self.start("five", "asked for")
if cmd == "calaccept":
if self.check and self.check["kind"] == "fit":
self.check["fit"].toggle_guide(time.monotonic())
elif self.check:
elif self.check and not self.asking: # (asking: this dot's click landed already)
if not self.check["accept"]:
self.check["accept_at"] = time.monotonic()
self.check["accept"] = True
@@ -861,6 +975,13 @@ class Checks:
else:
self.fit_tick(now)
return
if self.asking:
# ft-eyes has a dot's look or the fit: wait for its answer, at most to the deadline.
if now >= self.asking[1]:
done = self.asking[2]
self.drop_ask()
done("")
return
if c["done_at"] and now >= c["done_at"]:
if c.get("closing"):
self.close()
@@ -874,11 +995,11 @@ class Checks:
return
if c["accept"] and c["accept_at"] and now - c["accept_at"] > ACCEPT_WAIT:
# Clicked, but no capture yet (see on_sample): say what it's waiting for.
full = c["kind"] == "full"
big = c["kind"] != "quick"
if now - c.get("gaze_at", 0.0) > 0.5:
self.note("Waiting: the eye tracker isn't sending a gaze" if full else "Waiting: no gaze")
self.note("Waiting: the eye tracker isn't sending a gaze" if big else "Waiting: no gaze")
else:
self.note("Waiting for your gaze to hold still on the dot" if full else "Hold your look still")
self.note("Waiting for your gaze to hold still on the dot" if big else "Hold your look still")
if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
self.close("ignored")
elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
+14 -9
View File
@@ -7,14 +7,17 @@
//
// The panel sits POINTER-like at --distance (1.5 m, about where Frametop's screens are, so
// the eyes converge as they do in use). "quick" is a small square, QUICK_DEG across, for the
// one-dot check; "full" is FULL_DEG across (4:3), with a solid background whose brightness the
// service sets per round (pupil size changes with it, and the tracker's error with it); "fit"
// is FIT_DEG across (4:3), see-through like quick, for the headset fit check: a card per eye
// (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and hints.
// one-dot check; "five" is FIVE_DEG across (4:3), see-through like quick, for the five-dot
// check, whose dots are 12 degrees left and right and 9 up and down; "full" is FULL_DEG
// across (4:3), with a solid background whose brightness the service sets per round (pupil
// size changes with it, and the tracker's error with it); "fit" is FIT_DEG across (4:3),
// see-through like quick, for the headset fit check: a card per eye (tracked or lost, the
// tracker's signal, how much of the last 10 s it was seen) and hints. Every dot a check shows
// must fit its panel: gazecheck.py's PANEL_DEG mirrors these sizes and checks it.
//
// Control socket: abstract unix datagram "@ft_gazepanel" (--socket NAME); a sender with an
// address gets "ok" or "error ...":
// show quick|full|fit the panel, empty, in front of you
// show quick|five|full|fit the panel, empty, in front of you
// hide
// bg <0..1> the background's brightness (full)
// dot <yaw> <pitch> <state> [<progress 0..1>]
@@ -71,7 +74,8 @@ using Clock = std::chrono::steady_clock;
constexpr double kQuickDeg = 16; // QUICK_DEG: the one-dot check's square
constexpr double kFullDeg = 64; // FULL_DEG: the full calibration's width (4:3)
constexpr double kFitDeg = 40; // FIT_DEG: the headset fit check's width (4:3)
constexpr int kQuickPx = 320, kFullW = 1024, kFullH = 768, kFitW = 800, kFitH = 600;
constexpr double kFiveDeg = 40; // FIVE_DEG: the five-dot check's width (4:3), past its dots
constexpr int kQuickPx = 320, kFullW = 1024, kFullH = 768, kFitW = 800, kFitH = 600, kFiveW = 800, kFiveH = 600;
std::atomic<bool> g_stop{false};
// ---------------------------------------------------------------- text (as screens/keyboard.cpp)
@@ -471,9 +475,10 @@ int main(int argc, char **argv) {
if (!std::strncmp(buf, "show ", 5)) {
p.full = !std::strcmp(buf + 5, "full");
p.fit = !std::strcmp(buf + 5, "fit");
p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx;
p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx;
p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg;
const bool five = !std::strcmp(buf + 5, "five");
p.w = p.full ? kFullW : p.fit ? kFitW : five ? kFiveW : kQuickPx;
p.h = p.full ? kFullH : p.fit ? kFitH : five ? kFiveH : kQuickPx;
p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : five ? kFiveDeg : kQuickDeg;
p.title.clear(), p.text.clear(), p.note.clear(), p.dotOn = false, p.state = "off";
p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
place();
+3 -4
View File
@@ -161,11 +161,10 @@ class GazeReader:
env = dict(os.environ)
# The Frametop desktop has its own runtime dir; podman needs the real one.
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
distrobox = Path.home() / ".local" / "bin" / "distrobox"
# ft-gaze quits when its stdin closes, which is the one thing distrobox passes on
# when we go away (even if we're killed).
self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin"], env=env,
# when we go away (even if we're killed). in-box starts the container first, and picks
# the release's own on a release install.
self.proc = subprocess.Popen([str(REPO / "scripts" / "in-box"), str(HELPER), "--watch-stdin"], env=env,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
start_new_session=True)
out = Gio.UnixInputStream.new(self.proc.stdout.fileno(), False)
+60 -8
View File
@@ -6,7 +6,9 @@ its dots. Until 2026-10-05 it couldn't: a fresh install that chose our tracker n
Runs ft-gazed's Service with its sockets renamed and HOME in a temp folder (state and settings
go there), a fake pointer helper (gaze mode on, headset worn), a fake ft-gaze (SteamVR sees both
eyes; "own" is {"ok":0}, as with an uncalibrated ft-eyes), a fake ft-eyes control socket, and no
panel (a stand-in process). Nothing reaches the live gaze service, the pointer helper, ft-eyes,
panel (a stand-in process). The fake ft-eyes also answers late or not at all, as a slow one
does: the service must keep running meanwhile (until 2026-10-09 it waited, up to 3 s a dot and
10 s for the fit, and the helper's 3 s panel lease ran out). Nothing reaches the live gaze service, the pointer helper, ft-eyes,
or SteamVR, so it's safe next to them.
gaze/test/first-calibration-test.py
@@ -43,7 +45,8 @@ gazecheck.SCREENS = f"\0{tag}_screens"
gazecheck.PANEL = f"\0{tag}_panel"
gazed.EYES_SOCKET = gazecheck.EYES = f"\0{tag}_eyes"
gazed.read_settings = lambda: ("own", "auto", "auto", 55.0)
DOTS = 3
gazed.TrackedEye = lambda: lambda now=None: None # both eyes, whatever SteamVR's settings say
DOTS = 4
real_dots = gazecheck.check_dots
gazecheck.check_dots = lambda kind, own: real_dots(kind, own)[:DOTS] # a short calibration
logs = []
@@ -92,8 +95,10 @@ gazed.Service.start_helper = start_helper
gazed.Service.start_eyes = start_eyes
gazecheck.Checks.start_panel = start_panel
# The fake ft-eyes: uncalibrated until calib-fit. "fail" answers the next calib-point with that.
eyes_state = {"cal": None, "points": [], "fail": None}
# The fake ft-eyes: uncalibrated until calib-fit. "fail" answers the next calib-point with that;
# "delay" holds calib-point's and calib-fit's answers that many seconds; "drop" leaves the next
# calib-point unanswered.
eyes_state = {"cal": None, "points": [], "fail": None, "delay": 0.0, "drop": False}
eyes = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
eyes.bind(gazed.EYES_SOCKET)
eyes.settimeout(0.2)
@@ -116,19 +121,31 @@ def eyes_answer():
reply = "ok"
elif w[0] == "calib-point":
eyes_state["points"].append(tuple(map(float, w[1:5])))
if eyes_state["drop"]:
eyes_state["drop"] = False
continue
reply, eyes_state["fail"] = eyes_state["fail"] or "ok 50 50 1.00 1.00", None
elif w[0] == "calib-fit":
eyes_state["cal"] = {"made": "test", "dots": len(eyes_state["points"])}
reply = f"ok {len(eyes_state['points'])} dots"
else:
reply = f"fail unknown command {w[0]}"
if addr:
if addr and w[0] in ("calib-point", "calib-fit") and eyes_state["delay"]:
threading.Timer(eyes_state["delay"], send_late, (reply, addr)).start()
elif addr:
eyes.sendto(reply.encode(), addr)
def send_late(reply, addr):
try:
eyes.sendto(reply.encode(), addr)
except OSError:
pass # the service gave up on it
threading.Thread(target=eyes_answer, daemon=True).start()
helper_state = {"reply": "ok off worn", "heard": []}
helper_state = {"reply": "ok off worn", "heard": [], "calpanel": []} # calpanel: when "calpanel 1" came
helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
helper.bind(gazed.POINTER)
helper.settimeout(0.2)
@@ -146,6 +163,8 @@ def helper_answer():
helper.sendto(helper_state["reply"].encode(), addr)
else:
helper_state["heard"].append(data.decode())
if data == b"calpanel 1":
helper_state["calpanel"].append(time.monotonic())
threading.Thread(target=helper_answer, daemon=True).start()
@@ -225,10 +244,43 @@ ask("calaccept")
check("ft-eyes refusing a dot: its reason reaches the panel's note",
wait(lambda: "left eye in only 3 frames" in check_state().get("note", ""), 2), True)
check("dot 2, second try: taken", take_dot(1), True)
check("dot 3: taken", take_dot(2), True)
# A slow ft-eyes (2.5 s): the service goes on meanwhile, and a second click is ignored.
eyes_state["delay"] = 2.5
wait(lambda: check_state().get("i") == 2 and not check_state().get("done_at"), 3)
time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1)
before = len(eyes_state["points"])
ask("calaccept")
check("dot 3, ft-eyes slow: the service waits for it", wait(lambda: svc.checks.asking is not None, 2), True)
t = time.monotonic()
ask("status")
check("the service still answers meanwhile", time.monotonic() - t < 0.5, True)
ask("calaccept")
check("dot 3: taken once ft-eyes answers", wait(lambda: check_state().get("captured") == 3, 4), True)
check("the helper's panel lease was renewed while ft-eyes took its time",
sum(t < at < t + eyes_state["delay"] for at in helper_state["calpanel"]) >= 2, True)
check("and the calibration is still open (a blocked service took that as the headset off)",
check_state().get("kind"), "full")
check("the second click asked ft-eyes nothing", len(eyes_state["points"]), before + 1)
eyes_state["delay"] = 0.0
# No answer: the dot isn't taken, after calib-point's 3 s.
eyes_state["drop"] = True
wait(lambda: check_state().get("i") == 3 and not check_state().get("done_at"), 3)
time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1)
ask("calaccept")
check("dot 4, no answer from ft-eyes: not taken, and the panel says so",
wait(lambda: "didn't answer" in check_state().get("note", ""), 5), True)
eyes_state["delay"] = 2.5 # the fit too
check("dot 4, second try: taken", take_dot(3) or wait(lambda: check_state().get("captured") == 4, 4), True)
check("while ours fits, the panel stays", wait(lambda: check_state().get("fitting") is True, 3), True)
ask("calquit")
check("and a right click doesn't close it", check_state().get("kind"), "full")
check("all dots: ours fits its calibration (calib-fit)",
wait(lambda: eyes_state["cal"] is not None and not svc.checks.check, 4), True)
wait(lambda: eyes_state["cal"] is not None and not svc.checks.check, 5), True)
gaps = [b - a for a, b in zip(helper_state["calpanel"], helper_state["calpanel"][1:])]
check("the helper's panel lease (3 s) never ran out", bool(gaps) and max(gaps) < 3.0, True)
check("the service sees it calibrated", wait(lambda: svc.checks.calibrated() is True, 4), True)
check("and gaze mode stays on", "gaze off" in helper_state["heard"], False)
check("and no second calibration opens", wait(lambda: svc.checks.check is not None, 3), False)
+1
View File
@@ -34,6 +34,7 @@ gazecheck.SCREENS = f"\0{tag}_screens"
gazecheck.PANEL_PROG = gazed.REPO / "nonexistent-panel" # "isn't built": no panel
gazed.IDLE_AFTER, gazed.WAKE_SETTLE = 1.0, 3.0
gazed.read_settings = lambda: ("steam", "steam", "auto", 55.0)
gazed.TrackedEye = lambda: lambda now=None: None # both eyes, whatever SteamVR's settings say
logs = []
gazed.log = gazecheck.log = lambda msg: logs.append(msg)
+1 -1
View File
@@ -19,7 +19,7 @@ int failures = 0;
} while (0)
// The file as the eye server writes it, one sample at a time, with every field from the
// timestamp on moved by `shift` (0 stable, 5 the 0.4.x beta).
// timestamp on moved by `shift` (0 on SteamOS 0.3, 5 on 0.4).
struct File {
std::vector<uint8_t> bytes = std::vector<uint8_t>(324122); // eye-server.mmap's size
EyeFile eyes;
+175
View File
@@ -0,0 +1,175 @@
#!/usr/bin/env python3
"""Offline test of Track Dominant Eye Only (SteamOS 0.4): SteamVR's tracker ignores one eye,
and the gaze code then goes by the other (gazecal.tracked_eye, steady_samples' `eye`,
fitcheck's `ignore`, ft-gazed's live gaze). Reads only the temporary settings files it writes;
ft-gazed's Service is built without its sockets, so nothing reaches the live gaze service.
gaze/test/one-eye-test.py
"""
import importlib.machinery
import importlib.util
import json
import os
import sys
import tempfile
from collections import deque
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, ".."))
from fitcheck import FitCheck # noqa: E402
from gazecal import ( # noqa: E402
Correction,
EyeFallback,
EyeWeights,
Fixation,
TrackedEye,
steady_samples,
tracked_eye,
)
loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(HERE, "..", "ft-gazed"))
gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
loader.exec_module(gazed)
failures = []
def check(what, got, want):
if got != want:
failures.append(what)
print(f"FAIL {what}: got {got!r}, want {want!r}", flush=True)
tmp = tempfile.mkdtemp(prefix="ft-one-eye-test-")
first, second = os.path.join(tmp, "a.vrsettings"), os.path.join(tmp, "b.vrsettings")
paths = (first, second)
def write(path, steamvr):
with open(path, "w") as f:
f.write(steamvr if isinstance(steamvr, str) else json.dumps({"steamvr": steamvr}, indent=3))
# --- Reading SteamVR's settings ---
check("no settings file: both eyes", tracked_eye(paths), None)
write(second, {"eyeTrackingDominantEyeOnly": True})
check("only the second file: it counts (right, SteamVR's default eye)", tracked_eye(paths), 1)
write(first, {"supersampleScale": 1.0})
check("the first file counts, without the setting: both eyes", tracked_eye(paths), None)
write(first, {"eyeTrackingDominantEyeOnly": True, "dominantEye": 0})
check("dominant eye left", tracked_eye(paths), 0)
write(first, {"eyeTrackingDominantEyeOnly": False, "dominantEye": 0})
check("setting off", tracked_eye(paths), None)
write(first, "{ not json")
check("a broken file: both eyes", tracked_eye(paths), None)
eye = TrackedEye(paths)
write(first, {"eyeTrackingDominantEyeOnly": True, "dominantEye": 1})
check("TrackedEye reads it", eye(now=100.0), 1)
write(first, {"eyeTrackingDominantEyeOnly": True, "dominantEye": 0, "pad": "x" * 10})
check("TrackedEye waits CHECK seconds", eye(now=101.0), 1)
check("then sees the change", eye(now=102.5), 0)
# --- One look at a dot: the left eye lost all along (what SteamVR's tracker may report for
# the eye it ignores), the right seen, and the vergence jumping with the lost eye ---
look = [{"t": i / 90, "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.02, 0.001], "open": [0.0, 0.8],
"lr": 2.8 if i % 2 else 9.0}}} for i in range(40)]
why = {}
check("both eyes judged: nothing kept", len(steady_samples(look, why=why)), 0)
check("both eyes judged: why", why, {"lost_left": 40})
why = {}
check("right eye only: all kept", len(steady_samples(look, why=why, eye=1)), 40)
check("right eye only: nothing dropped", why, {})
why = {}
check("left eye only: nothing kept", len(steady_samples(look, why=why, eye=0)), 0)
blink = [dict(s, src={"mmap1": dict(s["src"]["mmap1"], open=[0.0, 0.05])}) if 10 <= i < 15 else s
for i, s in enumerate(look)]
why = {}
check("right eye only: its blinks still drop", len(steady_samples(blink, why=why, eye=1)), 35)
check("right eye only: as blinks", why, {"blink": 5})
# --- The headset fit check ---
fit = FitCheck(ignore=0)
for i in range(400):
fit.feed({"src": {"mmap1": {"hy": 0.0, "hp": 0.0, "unc": [0.02, 0.001], "open": [0.0, 0.8]}}}, i / 90)
check("fit: the ignored eye's card", fit.status(0)[0], "not tracked")
check("fit: the tracked eye's card", fit.status(1)[0], "tracking")
hints = fit.hints()
check("fit: says which eye counts", hints[0].startswith("SteamVR tracks only your right eye"), True)
check("fit: no losses blamed on the ignored eye", any("Left eye: lost" in h for h in hints), False)
check("fit: the tracked eye is fine", hints[-1], "Your right eye is tracked everywhere you've looked so far.")
both = FitCheck()
for i in range(400):
both.feed({"src": {"mmap1": {"hy": 0.0, "hp": 0.0, "unc": [0.02, 0.001], "open": [0.0, 0.8]}}}, i / 90)
check("fit, both eyes judged: the left eye is lost", both.status(0)[0], "LOST")
# --- ft-gazed's live gaze: the parts of Service the samples go through, no sockets ---
def service(one, source="mmap1"):
svc = gazed.Service.__new__(gazed.Service)
svc.override, svc.tracker, svc.source, svc.one_eye = None, "steam", source, one
svc.models = {name: Correction() for name in gazed.SOURCES}
svc.counts = dict.fromkeys(("samples", "sent", "blinks", "one_eye", "one_eye_used", "lost_left", "lost_right",
"looking_down", "dropped"), 0)
svc.opens, svc.vergence = (deque(maxlen=90), deque(maxlen=90)), deque(maxlen=90)
svc.lost, svc.bad_at, svc.fallback = [False, False], [0.0, 0.0], EyeFallback()
svc.fix, svc.last_sample = Fixation(radius=1.0), 0.0
svc.correction = lambda name, hy, hp: (0.0, 0.0)
svc.sent = []
svc.send = lambda t, hy, hp, rhy, rhp, eyes: svc.sent.append((hy, hp))
return svc
def feed(svc, n=90):
for i in range(n):
svc.on_source_sample({"t": i / 90, "src": {
"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.02, 0.001], "open": [0.0, 0.8]},
"mmap2": {"hy": 3.0, "hp": 2.0, "eyes": [[9.0, 9.0], [5.0, 2.0]]}}})
svc = service(None, "mmap2")
feed(svc)
check("mmap2, both eyes judged: a lost eye and no fallback yet drop the gaze", svc.sent, [])
svc = service(1, "mmap2")
feed(svc)
check("mmap2, right eye only: its own reading goes out", (len(svc.sent), svc.sent[-1] if svc.sent else None), (90, (5.0, 2.0)))
check("mmap2, right eye only: no blinks counted", svc.counts["blinks"], 0)
svc = service(0, "mmap1")
feed(svc)
check("left eye only, and it's lost: nothing goes out", (svc.sent, svc.counts["blinks"]), ([], 90))
svc = service(1)
check("no calibration: the source, as a whole", svc.kind, "source")
svc.models["right"].samples = 9
check("right eye only and calibrated: each eye's own", svc.kind, "eyes")
svc.one_eye = None
check("both eyes judged: the left needs a calibration too", svc.kind, "source")
# Our own tracker sees both eyes whatever SteamVR tracks; only the blinks come from SteamVR.
def feed_own(svc, n=90, right_open=0.8):
for i in range(n):
svc.on_eyes_sample({"t": i / 90, "src": {
"mmap1": {"unc": [0.02, 0.001], "open": [0.0, right_open]},
"own": {"hy": 5.0, "hp": 2.0, "eyes": [[4.0, 2.0], [6.0, 2.0]]}}}, True)
svc = service(1)
svc.tracker, svc.weights = "own", {"own": EyeWeights()}
check("own tracker, right eye only: our tracker", svc.kind, "own")
feed_own(svc)
check("own tracker, right eye only: SteamVR's closed left doesn't drop our left",
(len(svc.sent), svc.counts["one_eye"], svc.counts["blinks"]), (90, 0, 0))
svc = service(1)
svc.tracker, svc.weights = "own", {"own": EyeWeights()}
feed_own(svc, right_open=0.0)
check("own tracker, right eye only: its blink drops both", (svc.sent, svc.counts["blinks"]), ([], 90))
svc = service(None)
svc.tracker, svc.weights = "own", {"own": EyeWeights()}
feed_own(svc)
check("own tracker, both eyes judged: SteamVR's closed left still drops it", svc.counts["one_eye"], 90)
for p in paths:
if os.path.exists(p):
os.remove(p)
os.rmdir(tmp)
print("FAILED: " + ", ".join(failures) if failures else "all passed")
sys.exit(1 if failures else 0)
+4
View File
@@ -0,0 +1,4 @@
# Installed to /etc/atomic-update.conf.d/frametop-eyegrab.conf by gaze/tracker/install.sh.
# A SteamOS update deletes every /etc file its keep list (/usr/lib/rauc/atomic-update-keep.conf)
# doesn't name. That list keeps the unit, but not the program it runs.
/etc/frametop/ft-eyegrab
+3 -3
View File
@@ -111,9 +111,9 @@ reading only.
The file is 324,122 bytes. Only bytes 0x0-0x1f3 are used; the rest is zero. It's packed and
unaligned, so read it with memcpy. Offsets are also in `~/frametop/gaze/ft-gaze.cpp`.
On the SteamOS 0.4.x beta (SteamVR 2.18.2) every field from 0x157 on sits 5 bytes later, and the
counter stays at 0x38 (measured 2026-10-04, PR #26). The offsets below are stable's; ft-gaze detects
which layout is live.
On SteamOS 0.4 (SteamVR 2.18.2: the 0.4.3 beta, and 0.4.5, the release) every field from 0x157 on
sits 5 bytes later, and the counter stays at 0x38 (measured 2026-10-04, PR #26). The offsets below
are SteamOS 0.3's; ft-gaze detects which layout is live.
| Offset | What |
| --- | --- |
+4 -2
View File
@@ -5,7 +5,8 @@
# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when ours is the
# tracker in use (GAZE_TRACKER=auto, the default, picks it once this is installed) or the gaze
# probe uses it. install.sh offers this after gaze mode.
# Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit: it asks for
# Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's), the unit, and the entry that
# keeps the binary through SteamOS updates (/etc/atomic-update.conf.d): it asks for
# the password in the terminal, on the Frame or from a PC, or runs SUDO_ASKPASS when that's set
# (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env).
# Usage: gaze/tracker/install.sh [install|uninstall|status|log [lines]]
@@ -27,6 +28,7 @@ case ${1:-install} in
sudo_run "set -e
install -D -m 0755 -o root -g root $src/build/ft-eyegrab /etc/frametop/ft-eyegrab
install -D -m 0644 -o root -g root /tmp/$unit /etc/systemd/system/$unit
install -D -m 0644 -o root -g root $src/atomic-update.conf /etc/atomic-update.conf.d/frametop-eyegrab.conf
rm -f /tmp/$unit
systemctl daemon-reload
systemctl enable $unit
@@ -36,7 +38,7 @@ echo \"$unit: \$(systemctl is-active $unit)\""
;;
uninstall)
sudo_run "systemctl disable --now $unit 2>/dev/null
rm -f /etc/systemd/system/$unit /etc/frametop/ft-eyegrab
rm -f /etc/systemd/system/$unit /etc/frametop/ft-eyegrab /etc/atomic-update.conf.d/frametop-eyegrab.conf
rmdir /etc/frametop 2>/dev/null; systemctl daemon-reload; echo removed" ;;
status) on_frame "systemctl is-active $unit; ls -l /dev/shm/frametop-eyes-cams 2>/dev/null" || true ;;
log) on_frame "journalctl -u $unit --no-pager -o cat -n ${2:-20}" ;;
+19 -7
View File
@@ -2,11 +2,11 @@
# Frametop's one-line installer. In a terminal on the Steam Frame (Konsole in the desktop, or
# over SSH):
#
# curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
# curl -fsSL https://frametop.github.io/frametop/get.sh | bash
#
# It asks which version to install, clones the repo into ~/frametop (or updates the clone
# that's there), and runs its install.sh. Run it again to update, or to switch versions.
# With --release it installs a release instead: Frametop built, in one file. It downloads the
# Its third and fourth choices, or --release, install a release instead: Frametop built, in one file. It downloads the
# release's Frametop.zip from GitHub (about 1.1 GB: the newest stable release, or with
# --experimental the newest of any), unpacks it in ~/.cache/frametop/release, and runs its
# install-release.sh, which checks this SteamOS build against the releases' SteamOS table and
@@ -34,11 +34,14 @@ usage: get.sh [--stable | --experimental | --branch NAME] [--dir DIR] [--clone-o
get.sh --release [--stable | --experimental | --version V | --zip FILE|URL]
[--any-steamos] [--dir DIR] [--clone-only] [--yes] [--no-eye-tracker]
[--no-bluetooth | --bluetooth]
piped: curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- [options]
piped: curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- [options]
EOF
}
SLUG=${FRAMETOP_REPO:-DeeJanuz/frametop} # FRAMETOP_REPO: another repo's releases, such as a fork's
# Frametop lives in the Frametop organization's repo: the branches clone from it, and its CI
# runners (Depot, which need an organization) build the releases. It moved there from
# DeeJanuz/frametop on 2026-10-09; GitHub redirects clones made from the old name.
SLUG=${FRAMETOP_REPO:-Frametop/frametop} # FRAMETOP_REPO: another repo's releases, such as a fork's
# release_zip CHANNEL VERSION: the URL of a release's Frametop.zip on GitHub.
release_zip() {
@@ -111,7 +114,7 @@ install_release() {
# Everything happens in main, called on the last line, so a download cut short runs nothing.
main() {
local repo=https://github.com/DeeJanuz/frametop.git dir= branch= clone_only=0
local repo=https://github.com/Frametop/frametop.git dir= branch= clone_only=0
local yes=0 tty=0 current= def answer release=0 zip= want= any=0
local pass=()
while [ $# -gt 0 ]; do
@@ -140,6 +143,7 @@ main() {
echo "releases come from the stable or experimental list, not a branch" >&2
return 2
fi
local dir_arg=$dir # the menu's release choice puts releases in their own default place
if [ -z "$dir" ] && [ "$release" = 0 ]; then
dir=$HOME/frametop
fi
@@ -176,12 +180,20 @@ main() {
echo "Which version of Frametop?"
echo " 1) stable: the main branch, tested releases"
echo " 2) experimental: the newest features, less tested"
if [ "$release" = 0 ]; then
echo " 3) stable release: built, nothing to compile (a 1.1 GB download)"
echo " 4) experimental release: built, nothing to compile (a 1.1 GB download)"
fi
[ -n "$current" ] && echo "(installed now: $current)"
read -r -p "Choose 1 or 2 [$([ "$def" = main ] && echo 1 || echo 2)]: " answer </dev/tty || answer=
read -r -p "Choose 1$([ "$release" = 0 ] && echo ", 2, 3, or 4" || echo " or 2") [$([ "$def" = main ] && echo 1 || echo 2)]: " \
answer </dev/tty || answer=
case ${answer:-$def} in
1|main|s*) branch=main ;;
2|experimental|e*) branch=experimental ;;
*) echo "not 1 or 2: $answer" >&2; return 2 ;;
3|4) [ "$release" = 0 ] || { echo "not 1 or 2: $answer" >&2; return 2; }
release=1 dir=$dir_arg
branch=$([ "$answer" = 3 ] && echo main || echo experimental) ;;
*) echo "not one of the choices: $answer" >&2; return 2 ;;
esac
fi
fi
+5 -5
View File
@@ -14,7 +14,7 @@ For now the recorder runs inside Frametop's desktop, so these steps install Fram
## Before you start
- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/DeeJanuz/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/Frametop/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
- You need a Steam Frame on the stable SteamOS release (not the beta), an internet connection, and a keyboard (Bluetooth, or the on-screen one).
- You need a `sudo` password. If you've never set one, run `passwd` in Konsole first.
- Recordings are several gigabytes per round, and uploading one needs about the same again free while it runs. `df -h ~` shows your free space.
@@ -27,7 +27,7 @@ For now the recorder runs inside Frametop's desktop, so these steps install Fram
## 1. Install Frametop
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --stable
curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --stable
```
This clones Frametop into `~/frametop` and runs its installer. The first run downloads 1–2 GB. The installer asks a few questions (gaze mode, the eye tracker, the Bluetooth fixes); the defaults are fine. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, this updates it.
@@ -49,7 +49,7 @@ Open Frametop Hand Recorder from the desktop's application menu. It walks you th
## Update
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --stable
curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --stable
~/frametop/hands/rec/install.sh
```
@@ -61,8 +61,8 @@ Run the second command after SteamVR has restarted, as in the install.
~/frametop/hands/rec/install.sh uninstall
```
This removes the menu entry. With your `sudo` password, it also takes back the camera broker's permission to read the cameras. If you also installed Frametop's live hand tracking, the camera broker keeps that permission, because live hand tracking still uses it. Your recordings stay in `~/.local/share/frametop/hands/contrib`; delete that folder to remove them. To remove Frametop as well, follow [Uninstall](https://github.com/DeeJanuz/frametop#uninstall) in the README.
This removes the menu entry. With your `sudo` password, it also takes back the camera broker's permission to read the cameras. If you also installed Frametop's live hand tracking, the camera broker keeps that permission, because live hand tracking still uses it. Your recordings stay in `~/.local/share/frametop/hands/contrib`; delete that folder to remove them. To remove Frametop as well, follow [Uninstall](https://github.com/Frametop/frametop#uninstall) in the README.
## Help
Ask in the [Frametop Discord](https://discord.gg/W3X9f7z3Bc), the [Frametop issues](https://github.com/DeeJanuz/frametop/issues), or the dataset's [discussion page](https://huggingface.co/datasets/DeeJanuz/frametop-hands/discussions). All three are public.
Ask in the [Frametop Discord](https://discord.gg/W3X9f7z3Bc), the [Frametop issues](https://github.com/Frametop/frametop/issues), or the dataset's [discussion page](https://huggingface.co/datasets/DeeJanuz/frametop-hands/discussions). All three are public.
+5 -3
View File
@@ -10,6 +10,8 @@ CFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter
CXXFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter -Wno-psabi
CXXFLAGS += -std=c++17 -fopenmp -I$(NCNN)/include/ncnn
LDLIBS = $(NCNN)/lib/libncnn.a -ljsoncpp -fopenmp -lpthread
# The standalone Hand Recorder's release build passes LDFLAGS=-static (it runs on the host).
LDFLAGS ?=
CAMD = camd/camd.c camd/tp.c camd/xrcams.c
TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp track/sides.cpp
@@ -24,15 +26,15 @@ build/ft-camd: $(CAMD) camd/tp.h camd/xrcams.h camd/fhring.h
build/ft-hands: track/main.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
@mkdir -p build
$(CXX) $(CXXFLAGS) -o $@ track/main.cpp $(TRACK) $(LDLIBS)
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ track/main.cpp $(TRACK) $(LDLIBS)
build/ft-handreplay: track/replay.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
@mkdir -p build
$(CXX) $(CXXFLAGS) -o $@ track/replay.cpp $(TRACK) $(LDLIBS)
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ track/replay.cpp $(TRACK) $(LDLIBS)
build/sides-test: tests/sides_test.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
@mkdir -p build
$(CXX) $(CXXFLAGS) -o $@ tests/sides_test.cpp $(TRACK) $(LDLIBS)
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ tests/sides_test.cpp $(TRACK) $(LDLIBS)
check: build/sides-test
build/sides-test
+3 -2
View File
@@ -40,6 +40,7 @@ Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides
- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (`sides.json`), so recordings are labelled right. The example config said `0` until 2026-10-05; `scripts/conf-migrate.sh` (run by `install.sh` and `hands/rec/install.sh`) turns that untouched line into `auto`.
- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
- `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
- `HANDS_MODELS` (unset: `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to run fine-tuned models, such as the ones trained on the hand dataset, without passing options to every launcher. Those folders usually lack the `*-int8` files, so `--int8` won't load them.
- `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
The pointer helper's `POINTER_HANDS` and `POINTER_PINCH_*`/`POINTER_GRIP_*` settings are in "Pinches and grips in the pointer" below.
@@ -87,7 +88,7 @@ Options:
It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. The service starts it again, and it attaches to the new buffers.
**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands now catches this by itself (`track/sides.h`, `HANDS_SWAP_SIDES=auto`):
**Which camera is which:** video9 is `slam_right` (sensor `og01a1b 4-0036`), video13 is `slam_left` (`4-0060`), video6 is `upper_left` and video7 is `upper_right`; without the colour module the side pair is on video0 (`slam_right`) and video3. XRService says so in its log: `Found camera 'slam_left': ... v4l_subdev=/dev/v4l-subdev30` names each sensor, and each `TrackingCameraInit` line gives the device and subdev it opened. ft-hands and `camcheck.py` name the devices that way. Until 2026-10-05 they named them by the `TrackingCameraInit` index instead, which is only the order XRService opens them in (index 0 was `slam_right` on every start logged), and ft-camd bound each buffer queue to a device by the order of XRService's file descriptors, which changes when XRService restarts its cameras. The two mistakes made the side names come out swapped on most starts and right on some. ft-camd now asks each device which buffers it holds (`VIDIOC_QUERYBUF` names the descriptor XRService queued at each index) and only falls back to the order if that fails. With the names swapped, every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands still checks by itself, in case (`track/sides.h`, `HANDS_SWAP_SIDES=auto`):
- Whenever a hand's landmarks are found in two cameras at once (one of them a side camera), it intersects the rays through the 21 landmarks twice: once with the calibrations as named, once with the two side cameras exchanged. The same hand seen the right way meets within a few mm, in front of both cameras and as far away as its size says. The wrong way misses by centimetres or meets behind a camera.
- With the names wrong, the tracker never gets such pairs on its own: it hands the hand over to where the wrong calibration puts it and finds nothing there. So 5 times a second while undecided, the check places a tracked hand in 3D under the other naming and runs the landmark model where that puts it in the other side camera.
- It decides after 10 votes one way and none the other, or 20 with at most a fifth the other way, over at least 1 s. That takes about 1-2 s of hands in view. If the names are backwards, it exchanges them; the tracked views move with their images. Then it checks once more, more strictly.
@@ -264,7 +265,7 @@ Guesses, not tested:
## Known issues
- **The side cameras can come out swapped.** ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and some XRService restarts reverse it. ft-hands corrects it from the hands (`HANDS_SWAP_SIDES=auto`, the default). Until it has seen about 1-2 s of hands in both namings' reach, the cutouts may sit beside the hands. ft-camd itself still can't tell.
- **The side cameras could come out swapped** (fixed 2026-10-05, see "Which camera is which"). If they still do, ft-hands corrects it from the hands (`HANDS_SWAP_SIDES=auto`, the default): until it has seen about 1-2 s of hands in both namings' reach, the cutouts may sit beside the hands. ft-camd's log says `bound by VIDIOC_QUERYBUF` for each camera when its buffers were matched exactly.
- **The colour cameras can't be used while the headset is worn.** The colour module then writes only a half-size image into the top-left quarter of its buffers, and ft-camd drops those frames. So the service runs the mono cameras only, and tracking in bright light, where the mono cameras see dark hands, doesn't get the colour pair's help.
- **The colour calibration mapping isn't settled.** Which colour camera is `passthrough_left` (`HANDS_COLOR_LEFT`) and how the module's crop applies (`HANDS_COLOR_CROP`) still need `tools/check_color.py` on a recording with a lit, textured view.
- **Depth when one camera loses the hand.** A hand seen in one camera drifts 10% per update toward the one-camera depth guess (`kMonoDepthGain`, 0.1, in `track/tracker.cpp`). In the 2026-09-30 replays that was worse than keeping the last distance (see "3D" above). A smaller gain, such as 0.02, is the next thing to try.
+32 -10
View File
@@ -3,7 +3,7 @@
ft-hands see them all?
The Frame has four mono IR tracking cameras: the side pair slam_left and slam_right
(/dev/video9 and /dev/video13) and the upper pair (/dev/video6 and /dev/video7). With the
(/dev/video13 and /dev/video9) and the upper pair (/dev/video6 and /dev/video7). With the
Arcturus colour module attached, SteamVR's XRService loads an FPGA image ("VCINT") onto the
module whenever it opens the cameras (at start and after every wake). When that load fails
(seen 2026-10-02 17:02, after a sleep), XRService runs only the two side cameras, the IR
@@ -40,7 +40,7 @@ import time
LOG_DIR = os.path.expanduser("~/.local/share/Steam/logs")
LOG_LINK = os.path.join(LOG_DIR, "xrservice.txt")
SIDE_NODES = (9, 13) # slam_left, slam_right (TrackingCameraInit index 0 and 1)
SIDE_NODES = (9, 13) # the side pair: slam_right on video9, slam_left on video13 (see camera_map)
UPPER_NODES = (6, 7) # the upper pair (index 2 and 3)
TRACKING = 4
@@ -55,15 +55,20 @@ USER_TEXT = ("The headset's upper cameras and IR light are off. SteamVR couldn't
ANSI = re.compile(r"\x1b\[[0-9;]*m")
STAMP = re.compile(r"^\w{3} \w{3} \d{2} \d{4} (\d{2}:\d{2}:\d{2})\.\d+ (\w+): ?(.*)$")
# Lines worth reading; anything else is skipped before the regexes (the log grows by MBs a day).
KEYS = ("FPGA", "VCINT", "Created", "TrackingCameraInit", "Closing tracking camera", "Streaming",
KEYS = ("FPGA", "VCINT", "Created", "TrackingCameraInit", "Found camera", "Closing tracking camera", "Streaming",
"systemd suspend", "systemd resume", "XRService logging to", "Exiting XRService", "ISP ")
# XRService's numbering of its tracking cameras (the TrackingCameraInit index): ft-hands names
# them this way too (track/main.cpp, cameras_from_xrservice_log).
# The tracking cameras' names. XRService says which sensor subdev each name is ("Found camera
# 'slam_left': ... v4l_subdev=/dev/v4l-subdev30", once per instance) and which subdev and video
# device each TrackingCameraInit index opened. The index is only the order it opens them in: on
# every start logged since 2026-10-04, index 0 was slam_right. A log without the "Found camera"
# lines falls back to this order. ft-hands names them the same way (track/main.cpp,
# cameras_from_xrservice_log).
NAMES = ("slam_left", "slam_right", "upper_left", "upper_right")
RE_FOUND = re.compile(r"Found camera '(\w+)': interface=\S+ v4l_subdev=(\S+)")
RE_PASSTHRU = re.compile(r"Passthrough connected but FPGA is (\S+) - loading VCINT")
RE_INTERLEAVE = re.compile(r"Upper cameras FPGA interleaving support: (\d)")
RE_TASKS = re.compile(r"Created (\d+) tasks \((\d+) tracking, (\d+) passthrough\)")
RE_INIT = re.compile(r"TrackingCameraInit: index: (\d+)\. video device: /dev/video(\d+)")
RE_INIT = re.compile(r"TrackingCameraInit: index: (\d+)\. video device: /dev/video(\d+)(?:\. v4l subdevice: (\S+))?")
RE_STREAM = re.compile(r"Streaming resumed \(FPGA: (\S+), VC interleaving: (\w+)\)")
RE_STATE = re.compile(r"FPGA state check: (\S+)")
# Without the colour module XRService runs the side cameras through the ISP, as NV12 on other
@@ -85,13 +90,15 @@ class LogState:
self.closed_at = ""
self.episode = None
self.nodes = {} # TrackingCameraInit index -> /dev/videoN, from the whole log
self.subdevs = {} # TrackingCameraInit index -> its sensor subdev, from the whole log
self.found = {} # sensor subdev -> camera name ("Found camera" lines)
self.failures = [] # [(time, line)]: every VCINT failure in this log
self.lines = 0
def _new_episode(self, t):
self.closed = False
self.episode = {"start": t, "fpga_before": "", "vcint": "", "interleave": None, "tasks": None,
"inits": {}, "stream": "", "isp": None, "failure": "", "evidence": []}
"inits": {}, "init_subdevs": {}, "stream": "", "isp": None, "failure": "", "evidence": []}
if self.closed_at:
self.episode["evidence"].append(self.closed_at)
@@ -164,12 +171,18 @@ class LogState:
ep["tasks"] = tuple(int(v) for v in m.groups())
ep["evidence"].append(short)
return
m = RE_FOUND.search(text)
if m:
self.found[m.group(2)] = m.group(1)
return
m = RE_INIT.search(text)
if m:
ep = self._ep(t)
idx, node = int(m.group(1)), int(m.group(2))
ep["inits"][idx] = node
self.nodes[idx] = node
if m.group(3):
ep["init_subdevs"][idx] = self.subdevs[idx] = m.group(3)
ep["evidence"].append(short)
return
m = RE_STREAM.search(text)
@@ -198,9 +211,18 @@ class LogState:
def camera_map(self):
"""{calibration name: /dev/videoN's N} from the latest camera start's TrackingCameraInit
lines (the whole log's when that start has none yet)."""
inits = (self.episode or {}).get("inits") or self.nodes
return {NAMES[i]: node for i, node in sorted(inits.items()) if 0 <= i < len(NAMES)}
lines (the whole log's when that start has none yet): each one's subdev named by the
"Found camera" lines, else by the index (see NAMES)."""
ep = self.episode or {}
inits, subdevs = (ep.get("inits"), ep.get("init_subdevs")) if ep.get("inits") else (self.nodes, self.subdevs)
out = {}
for i, node in sorted(inits.items()):
name = self.found.get(subdevs.get(i))
if name not in NAMES:
name = NAMES[i] if 0 <= i < len(NAMES) else None
if name:
out[name] = node
return out
def tracking_nodes(self):
got = tuple(self.nodes[i] for i in range(TRACKING) if i in self.nodes)
+4 -3
View File
@@ -264,11 +264,11 @@ static void setup_camera(cam_t *c, xr_camera_t *cam, int pidfd)
xr_slugify(cam->sensor, sensor, sizeof(sensor));
snprintf(c->slug, sizeof(c->slug), "%.20s_video%d", sensor, cam->node);
bool own = false;
bool own = false, exact = false;
for (int g = 0; g < xr.ngroups; g++)
if (xr.groups[g].cam == cam && group_fits(&xr.groups[g], cam, c->need))
own = true;
own = true, exact = exact || xr.groups[g].exact;
char model[16];
model_of(cam->sensor, model, sizeof(model));
@@ -318,7 +318,8 @@ static void setup_camera(cam_t *c, xr_camera_t *cam, int pidfd)
}
printf(" %-24s %-12s %ux%u pitch %u, %d candidate buffers%s\n", c->slug, cam->path,
c->lay.width, c->lay.height, c->lay.pitch, c->nslots, own ? "" : " (shared run)");
c->lay.width, c->lay.height, c->lay.pitch, c->nslots,
!own ? " (shared run)" : exact ? ", bound by VIDIOC_QUERYBUF" : ", bound by open order");
}
/* ------------------------------------------------------ index -> buffer */
+98 -5
View File
@@ -9,8 +9,11 @@
* - The V4L2 nodes and sensor subdevs it holds open come from /proc/<pid>/fd.
* - Each node's geometry comes from VIDIOC_G_FMT on our own handle.
* - Each node is traced back to its sensor through MEDIA_IOC_G_TOPOLOGY.
* - Buffers are split into queues by allocation order: XRService opens a
* sensor subdev, then allocates that camera's buffers.
* - Buffers are split into runs by allocation order, and each run is bound to
* its camera by VIDIOC_QUERYBUF on the camera's node, which names the
* descriptor XRService queued at each index. Where that fails, the sensor
* subdev opened just before the run decides (XRService opens a sensor's
* subdev, then allocates its buffers), which isn't always right.
*/
#define _GNU_SOURCE
@@ -457,6 +460,38 @@ static bool scan_xr_fds(pid_t pid, char *err, size_t errn)
/* ------------------------------------------------------ camera discovery */
/*
* Which of XRService's buffers each V4L2 index of a camera holds. vb2 lets any handle query a
* queue's buffers, and for a DMABUF buffer it returns the descriptor its owner last queued it
* with: that descriptor's number in XRService's fd table, the same numbers scan_xr_fds reads
* from /proc. XRService queues each index with the same buffer every time.
*/
static void query_buffers(int fd, xr_camera_t *c, bool mplane)
{
c->nqbuf = 0;
for (int i = 0; i < XR_MAX_RUNBUFS; i++) {
struct v4l2_buffer b;
struct v4l2_plane planes[VIDEO_MAX_PLANES];
memset(&b, 0, sizeof(b));
memset(planes, 0, sizeof(planes));
b.index = (unsigned)i;
b.type = mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE : V4L2_BUF_TYPE_VIDEO_CAPTURE;
if (mplane) {
b.m.planes = planes;
b.length = VIDEO_MAX_PLANES;
}
if (ioctl(fd, VIDIOC_QUERYBUF, &b) < 0 || b.memory != V4L2_MEMORY_DMABUF)
break; /* EINVAL past the last index */
c->qbuf_xfd[c->nqbuf++] = mplane ? planes[0].m.fd : b.m.fd;
}
}
static void probe_cameras(xr_state_t *st)
{
int seen[64];
@@ -521,6 +556,8 @@ static void probe_cameras(xr_state_t *st)
c->planesize[0] = fmt.fmt.pix.sizeimage;
}
query_buffers(fd, c, fmt.type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
struct stat sb;
if (fstat(fd, &sb) == 0) {
@@ -630,6 +667,46 @@ const char *xr_fmt_name(xr_fmt_t f)
/* ------------------------------------------------------- buffer grouping */
/* The camera whose VIDIOC_QUERYBUF names this XRService descriptor, or NULL. */
static xr_camera_t *qbuf_owner(xr_state_t *st, int xfd)
{
for (int c = 0; c < st->ncameras; c++)
for (int k = 0; k < st->cameras[c].nqbuf; k++)
if (st->cameras[c].qbuf_xfd[k] == xfd)
return &st->cameras[c];
return NULL;
}
/*
* A run can hold two cameras' queues when nothing between them in the fd table ends it (the
* upper pair, both 640x480). Where VIDIOC_QUERYBUF says so, cut it where the owner changes.
*/
static void split_groups(xr_state_t *st)
{
for (int i = 0; i < st->ngroups && st->ngroups < XR_MAX_GROUPS; i++) {
xr_group_t *g = &st->groups[i];
xr_camera_t *first = qbuf_owner(st, g->buf[0].xfd);
int cut = -1;
for (int b = 1; first && b < g->nbufs && cut < 0; b++) {
xr_camera_t *o = qbuf_owner(st, g->buf[b].xfd);
if (o && o != first)
cut = b;
}
if (cut < 0)
continue;
xr_group_t *t = &st->groups[st->ngroups++];
*t = *g;
t->nbufs = g->nbufs - cut;
memmove(t->buf, g->buf + cut, (size_t)t->nbufs * sizeof(t->buf[0]));
g->nbufs = cut;
}
}
/*
* XRService allocates one udmabuf per plane, plane 0 then plane 1, a whole
* queue at a time right after opening the sensor's subdev. Plane 1 matches
@@ -703,6 +780,8 @@ static void build_groups(xr_state_t *st)
i++; /* consume the plane 1 descriptor */
}
split_groups(st);
int keep = 0;
for (int i = 0; i < st->ngroups; i++)
@@ -712,7 +791,21 @@ static void build_groups(xr_state_t *st)
st->ngroups = keep;
/*
* Bind each run to a camera. The sensor marker alone can be wrong: XRService
* Bind each run to a camera: exactly where a camera's VIDIOC_QUERYBUF names the run's first
* buffer (query_buffers). The two side cameras have the same format, so for them nothing
* else is sure.
*/
for (int i = 0; i < st->ngroups; i++)
for (int c = 0; c < st->ncameras && !st->groups[i].cam; c++)
for (int k = 0; k < st->cameras[c].nqbuf; k++)
if (st->cameras[c].qbuf_xfd[k] == st->groups[i].buf[0].xfd) {
st->groups[i].cam = &st->cameras[c];
st->groups[i].exact = true;
break;
}
/*
* The rest by the sensor marker and sizes. The sensor marker alone can be wrong: XRService
* sometimes opens another sensor's subdev (e.g. the idle color camera)
* between an upper camera's subdev and its buffers, and two upper cameras
* can resolve to the same sensor name. So a marker match must also fit the
@@ -792,9 +885,9 @@ void xr_print(const xr_state_t *st, FILE *f)
const xr_group_t *g = &st->groups[i];
fprintf(f, " queue %d: %d buffers plane0=%zu plane1=%zu fds %d..%d sensor '%s' -> %s\n",
fprintf(f, " queue %d: %d buffers plane0=%zu plane1=%zu fds %d..%d sensor '%s' -> %s%s\n",
i, g->nbufs, g->planesize[0], g->planesize[1],
g->buf[0].xfd, g->buf[g->nbufs - 1].xfd1, g->sensor,
g->cam ? g->cam->path : "(unbound)");
g->cam ? g->cam->path : "(unbound)", g->exact ? " (VIDIOC_QUERYBUF)" : "");
}
}
+3
View File
@@ -32,6 +32,8 @@ typedef struct {
uint32_t pixfmt;
char sensor[XR_SENSOR_LEN]; /* media entity name of the sensor */
const char *role;
int nqbuf; /* V4L2 indices VIDIOC_QUERYBUF named */
int qbuf_xfd[XR_MAX_RUNBUFS]; /* index -> its plane 0 fd in XRService */
} xr_camera_t;
typedef struct {
@@ -48,6 +50,7 @@ typedef struct {
xr_bufref_t buf[XR_MAX_RUNBUFS];
char sensor[XR_SENSOR_LEN]; /* from the preceding sensor subdev */
xr_camera_t *cam;
bool exact; /* cam is from VIDIOC_QUERYBUF, not the order */
} xr_group_t;
typedef struct {
+19 -4
View File
@@ -23,6 +23,17 @@ The plan this belongs to is `~/Desktop/Projects/frame-hands/notes/hands-plan.md`
Every part runs in the dev container, as ft-hands and Input Settings do. The host Python isn't used: it lacks PySide6 and zstd. `setup/dev-container.sh` gains `zstd`.
### The standalone Hand Recorder
[frametop-hand-recorder](https://github.com/Frametop/frametop-hand-recorder) is the recorder for people without Frametop. It pins this repo as a submodule and ships the parts above with its own window, a Qt Quick Controls `main.qml` sized for SteamVR's dashboard. Its release builds the binaries for the SteamOS host: ft-camd and ft-hands statically (`make LDFLAGS=-static`), and ft-handpanel against SteamVR's `libopenvr_api`. Everything runs on the host, the Python from a venv (`ft_handrec.py --qml ITS_QML --style Basic`).
Its tarball keeps this tree's layout (`hands/build/`, `hands/rec/`, `hands/models/`), so the paths here don't change. A `standalone.json` at the top marks it (`takes.standalone()`):
- `session.py` starts ft-hands directly, not through distrobox.
- session.json's `tool` and the manifest's are `ft-handrec <describe> (frametop-hand-recorder <version>)`.
- The repair hints say to run its install command again.
The recordings go to the same `~/.local/share/frametop/hands/contrib`, so either recorder shows both's sessions.
## Processes during a session
- **ft-camd** publishes the camera ring. If it isn't running, the session starts it as the transient user unit `frametop-handrec-camd.service`, the way `hands/ft-cutouts` starts `frametop-cutouts-camd.service` (needs `hands/build/ft-camd` with capabilities: `hands/run.sh caps`). An ft-camd already running from ft-cutouts or ft-handsctl is used as it is.
@@ -283,7 +294,7 @@ Before section 7: "Put on both controllers and tighten the straps". Before secti
- **Hands seen:** from the hands file. A hand counts as seen if its flags match the side and the file is fresh (publish within 0.3 s). Prompts with `hands` set show the `hands` chips. If an asked-for hand is lost for more than 1.5 s, the note says "I can't see your left hand: bring it into view".
- **Controller tracking:** in sections 8 and 9, `devices` is polled once a second. A result other than 200 for more than 1 s says "The left controller lost tracking: turn your palm slightly toward you". Each such stretch goes into `prompts.jsonl` as `feedback` with `"controller": {...}`.
- **Lighting, measured:** the checklist page measures the light when it opens, starting ft-camd (`frametop-handrec-camd.service`) if nothing runs it; the window stops it again on quit if it started it. The mean of every mono camera's `mean` and `dark_mean` from the ring (`hands/tools/ring.py` layout; struct only, no numpy) is compared with the person's earlier sessions. If it matches an earlier round's within 15%, the window says so before starting.
- **Lighting label:** "Measured by the cameras" is the default. `ambient_ir`, the mono cameras' mean `dark_mean` (the room's infrared), labels the round `daylight` from 6.0 and `indoor` below (`session.classify_lighting`). Lamps and LEDs give off hardly any infrared, so a dim room and a lit one read about the same (1.8 by one lamp, 2.2 in a lamp-lit room) and the cameras can't tell them apart; the person can pick dim, room or daylight instead (`source: "picked"`). The 6.0 threshold was a guess, and the first daylight round (dataset PR #5, 2026-10-05, a room with big sunlit windows) read 2.38 and was labelled `indoor`: the windows are a small part of each picture. So the checklist now asks people to pick Daylight when sunlight comes in, and the maintainer corrects the label in review (`hub_review.py lighting`) when hands stand out far less than in lamp light (PR #5: hands a median 1.15x as bright as their surroundings, against 1.5-1.7x).
- **Lighting label:** "Measured by the cameras" is the default. `ambient_ir`, the mono cameras' mean `dark_mean` (the room's infrared), labels the round `daylight` from 6.0 and `indoor` below (`session.classify_lighting`). Lamps and LEDs give off hardly any infrared, so a dim room and a lit one read about the same (1.8 by one lamp, 2.2 in a lamp-lit room) and the cameras can't tell them apart; the person can pick dim, room or daylight instead (`source: "picked"`). The 6.0 threshold was a guess, and the first daylight round (dataset PR #5, 2026-10-05, a room with big sunlit windows) read 2.38 and was labelled `indoor`: the windows are a small part of each picture. So the checklist now asks people to pick Daylight when sunlight comes in, and the maintainer corrects the label in review (`hub_review.py lighting`) from what the pictures show: windows lit by the sun, the time of day. Hands that stand out little from the room (PR #5: a median 1.15x as bright as their surroundings, against 1.5-1.7x in some lamp-lit rounds) go with daylight, but a pale room at night read 1.22 (PR #6), so that alone doesn't decide it.
### Camera check
@@ -296,11 +307,15 @@ On 2026-10-02 a whole session showed "I can't see your hands": after the headset
### Controls
- The window has Start, a big Next (while a step waits; its hint is the panel's), Pause/Resume, Redo step, Skip section and Stop. While it has focus: Space is Next, P pauses or resumes, R redoes, S skips the section, Esc stops. The panel's bottom line and the window list them. The window also shows the step's picture, diagram, countdown and "Hold".
- **The headset's button.** The Frame has a click button on its right side, for use without controllers: `KEY_SELECT` (353) on the evdev device `gpio-keys`. While a step waits (the welcome, a section's intro, a step's ready screen) a press is Next; during a countdown or hold (and auto mode's timed screens) it pauses; while paused it resumes. The session finds the device in `/proc/bus/input/devices` by name and its KEY bitmap (`event3` on the maintainer's Frame) and reads `input_event` structs with plain `struct` (no python-evdev), from a thread. Only key-downs count (value 1: releases and autorepeat, value 2, don't), and presses closer than 0.3 s count once.
- **The headset's button.** The Frame has a click button on its right side, for use without controllers: `KEY_SELECT` (353) on the evdev device `gpio-keys`. It has three gestures (`ButtonGestures`), so a session can run with the window out of reach (the standalone Hand Recorder's window is in SteamVR's dashboard, closed while recording):
- **A press.** While a step waits (the welcome, a section's intro, a step's ready screen, the no-hands question) it's Next; during a countdown or hold (and auto mode's timed screens) it pauses; while paused it resumes. It counts once 0.45 s have passed without a second press, so Next and pause come that much after the press.
- **Two presses**, the second down within 0.45 s of the first one's release: redo, as R. It counts at the second press. Where R does nothing (nothing recorded yet, a section's intro), neither does this.
- **A hold** of 1.5 s: stop, as Esc. It counts after 1.5 s, while the button is still down.
- The session finds the device in `/proc/bus/input/devices` by name and its KEY bitmap (`event3` on the maintainer's Frame) and reads `input_event` structs with plain `struct` (no python-evdev), from a thread. Downs (value 1) and ups (value 0) count; autorepeat (value 2) doesn't. The gaps between events come from the kernel's timestamps, so a double press read in one go still counts as two. A release and a press closer than 30 ms are one press (gpio-keys debounces too).
- It's read, never grabbed (`EVIOCGRAB`). Frametop's input relay (`input/input-relay.py`), ft-powerd, SteamVR and gamescope read the same device, and the relay remaps its volume keys (the experimental branch's `docs/hazards.md`). A grab would take the volume keys from the relay.
- `steamos` is in the `input` group, so it needs no sudo. The dev container sees the host's `/dev/input` and `/proc/bus/input/devices`, and the group carries over (checked 2026-10-02), so it works from the window there and from `session.py` on the host. If the device is missing or can't be opened, the session logs it, keeps trying every 5 s, and the hints don't mention the button.
- **Not known yet (needs the headset):** what SteamVR and gamescope do with the same press. They read it too, so it may also click whatever is under the head or gaze pointer in VR, or open something. Check on the first session; if it does, the fix is on their side or a different button, not a grab.
- **The Next hint follows what's there.** No mouse connected: "Ready? Press the button on the right side of the headset" (the window's Next can't be clicked, and Space needs the window focused). A mouse: "Ready? Press Space, click Next, or press the headset button". No button: "Ready? Press Space or click Next". The panel's bottom line leads with "Headset button: next, pause". A mouse is a device in `/proc/bus/input/devices` with EV_REL, REL_X and REL_Y that isn't made in software: uinput devices (Frametop's virtual mouse, frame-voice's keyboard) sit under `/devices/virtual/input` or on the virtual bus (6), and are left out; Bluetooth mice come through uhid, under `/devices/virtual/misc/uhid`, and count. It's looked at again at each step, so a mouse plugged in mid-session counts from the next step.
- **Not known yet (needs the headset):** what SteamVR and gamescope do with the same press. They read it too, so it may also click whatever is under the head or gaze pointer in VR, or open something, and a hold or a double press may mean something else to them. Check on the first session; if it does, the fix is on their side or a different button, not a grab.
- **The Next hint follows what's there.** No mouse connected: "Ready? Press the button on the right side of the headset" (the window's Next can't be clicked, and Space needs the window focused). A mouse: "Ready? Press Space, click Next, or press the headset button". No button: "Ready? Press Space or click Next". The panel's bottom line leads with "Headset button: press for next or pause · press twice to redo · hold to stop", then the window's keys by letter. With the button, the pause note and the no-hands question say how to resume or stop with it too. A mouse is a device in `/proc/bus/input/devices` with EV_REL, REL_X and REL_Y that isn't made in software: uinput devices (Frametop's virtual mouse, frame-voice's keyboard) sit under `/devices/virtual/input` or on the virtual bus (6), and are left out; Bluetooth mice come through uhid, under `/devices/virtual/misc/uhid`, and count. It's looked at again at each step, so a mouse plugged in mid-session counts from the next step.
- **R (redo).** During a step's countdown or hold: that step starts again from its ready screen. At a step's ready screen: the step before it (in this section) goes again. Either way a `redo` event marks the range of the try being redone, so its labels are skipped; the sets stay, to delete in review if wanted.
- A pause stops the take's recording and starts it again on resume as the next part of the same take (`sets-2.bin`, and so on). takes.py reads all parts in order. A pause while a step waits only shows "Paused"; a Next pressed while paused doesn't count.
+21 -5
View File
@@ -24,7 +24,9 @@ Everything lives under ~/.local/share/frametop/hands/contrib (--base). Nothing i
unless the person presses Upload; while CONSENT.md or UPLOAD.md is a draft, Upload stays off
unless FT_HANDREC_ALLOW_UPLOAD=1 (the maintainer's rehearsal against a test repo, picked with
FT_HANDREC_DATASET). --hub-dry-run does everything but the network calls.
Launch with hands/rec/ft-handrec (host wrapper).
Launch with hands/rec/ft-handrec (host wrapper). The standalone Hand Recorder
(github.com/Frametop/frametop-hand-recorder) runs this backend on the host, from a venv, with
its own window for SteamVR's dashboard: --qml its main.qml, --style Basic.
"""
import argparse
import datetime
@@ -43,6 +45,7 @@ from PySide6.QtGui import QColor, QDesktopServices, QFont, QGuiApplication, QIco
from PySide6.QtQml import QQmlApplicationEngine
from PySide6.QtQuick import QQuickImageProvider
from PySide6.QtQuickControls2 import QQuickStyle
import shiboken6
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
@@ -292,6 +295,11 @@ class Backend(QObject):
def consentText(self):
return read_text(CONSENT_PATH) or "CONSENT.md is missing."
@Property(str, constant=True)
def toolVersion(self):
"""What session.json records as the tool (takes.tool_version), for the window to show."""
return takes.tool_version()
@Property(str, constant=True)
def consentVersion(self):
return consent_version()
@@ -389,7 +397,8 @@ class Backend(QObject):
measured = ""
try:
ring = mod.ring_lighting()
if ring is None and not self.sessionActive:
# A dry run starts nothing, the camera broker included.
if ring is None and not self.sessionActive and not self._session_options.get("dry_run"):
if mod.start_camd():
self._camd_started = True
time.sleep(1.0) # the first near-black frames
@@ -1173,6 +1182,9 @@ def main():
help="test: Upload checks the export and says what it would send, with no network calls")
ap.add_argument("--ignore-cameras", action="store_true",
help="start sessions even if the camera check (hands/camcheck.py) finds the upper cameras off")
ap.add_argument("--qml", default=os.path.join(HERE, "main.qml"),
help="the window (default: this folder's main.qml, which needs Kirigami)")
ap.add_argument("--style", default="org.kde.desktop", help="the Qt Quick Controls style")
a, qt_args = ap.parse_known_args()
app = QGuiApplication([sys.argv[0]] + qt_args)
app.setApplicationName("ft-handrec")
@@ -1180,7 +1192,7 @@ def main():
app.setDesktopFileName("ft-handrec")
if not QIcon.themeName():
QIcon.setThemeName("breeze")
QQuickStyle.setStyle("org.kde.desktop")
QQuickStyle.setStyle(a.style)
store = takes.Store(a.base)
engine = QQmlApplicationEngine()
engine.addImageProvider("frames", FrameProvider(store))
@@ -1195,7 +1207,7 @@ def main():
app.aboutToQuit.connect(backend.shutdown)
engine.rootContext().setContextProperty("backend", backend)
engine.rootContext().setContextProperty("startPage", a.page)
engine.load(QUrl.fromLocalFile(os.path.join(HERE, "main.qml")))
engine.load(QUrl.fromLocalFile(os.path.abspath(a.qml)))
if not engine.rootObjects():
sys.exit(1)
@@ -1208,7 +1220,11 @@ def main():
signal.signal(signal.SIGINT, on_signal)
tick = QTimer(interval=500, timeout=lambda: None)
tick.start()
sys.exit(app.exec())
code = app.exec()
# The window goes before the backend: otherwise its bindings run again against a deleted
# backend as Python tears down ("Cannot read property ... of null", one per binding).
shiboken6.delete(engine)
sys.exit(code)
if __name__ == "__main__":
+4 -2
View File
@@ -487,7 +487,8 @@ Kirigami.ApplicationWindow {
+ "Each step waits until you're ready: press the button on the right side of the headset, "
+ "or Space or Next in this window. A 3-2-1 "
+ "countdown follows, then hold the pose until the bar runs out. Nothing is recorded while "
+ "a step waits. The headset button also pauses and resumes a recording. In this window P "
+ "a step waits. The headset button also pauses and resumes a recording; press it twice "
+ "to record a step again, or hold it to stop. In this window P "
+ "pauses, R records the last step again, S skips a section and Esc stops.\n\n"
+ "Nothing leaves the headset. Afterwards you watch the takes in Review, delete anything "
+ "you don't want to share, and only then export."
@@ -920,7 +921,8 @@ Kirigami.ApplicationWindow {
opacity: 0.7
text: "The instructions appear in the headset. "
+ (sessionView.st.button ? "The button on the right side of the headset: "
+ (sessionView.stepMode ? "next, " : "") + "pause or resume. " : "")
+ (sessionView.stepMode ? "next, " : "") + "pause or resume; twice: record "
+ "the last step again; hold: stop. " : "")
+ "While this window has focus: "
+ (sessionView.stepMode ? "Space: next · " : "")
+ "P: pause or resume · R: record the last step again · S: skip section · Esc: stop. "
+153 -59
View File
@@ -51,6 +51,7 @@ sys.path.insert(0, HANDS)
import camcheck # noqa: E402 (hands/camcheck.py: are all four mono cameras running?)
sys.path.insert(0, HERE)
import sides # noqa: E402 (hands/rec/sides.py: which side camera is which)
import takes # noqa: E402 (hands/rec/takes.py: the version, and the standalone recorder's build info)
FT_HANDS = os.path.join(HANDS, "build", "ft-hands")
FT_CAMD = os.path.join(HANDS, "build", "ft-camd")
PANEL_BIN = os.path.join(HERE, "build", "ft-handpanel")
@@ -60,6 +61,9 @@ BASE_DIR = os.path.expanduser("~/.local/share/frametop/hands/contrib")
PANEL_SOCKET = "ft_handpanel"
CAMD_UNIT = "frametop-handrec-camd.service"
HANDS_UNIT = "frametop-handrec-hands.service"
# The standalone Hand Recorder (takes.standalone()): its binaries run on the host, so ft-hands
# starts directly rather than in the dev container, and repairs mean its install command.
STANDALONE = takes.standalone()
RECORD_HZ = 10
SIDES_READ_S = 0.5 # how often the tracking ft-hands' side camera decision is read
@@ -72,15 +76,19 @@ MIN_FREE = 1.5e9 # stop the session before the disk fills
COUNTDOWN_S = 3 # step mode: the 3-2-1 before each step, recorded
FIRST_SET_S = 3.0 # step mode: how long the hold may wait for its recording's first set
RESUME_HINT = "Paused. Resume: P in the Hand recorder window"
RESUME_HINT_BUTTON = "Paused. Resume: press the headset button, or P in the Hand recorder window"
READY_TEXT = "Ready? Press Space or click Next"
# With the headset's button: it leads when no mouse is connected (the window's Next can't be clicked).
READY_BUTTON = "Ready? Press the button on the right side of the headset"
READY_BUTTON_MOUSE = "Ready? Press Space, click Next, or press the headset button"
KEYS_STEP = "Hand recorder window: Space next \u00b7 P pause \u00b7 R redo \u00b7 S skip section \u00b7 Esc stop"
KEYS_AUTO = "Hand recorder window: P pause \u00b7 R redo \u00b7 S skip section \u00b7 Esc stop"
KEYS_STEP_BUTTON = ("Headset button: next, pause \u00b7 Window: Space next \u00b7 P pause \u00b7 R redo "
"\u00b7 S skip \u00b7 Esc stop")
KEYS_AUTO_BUTTON = "Headset button: pause \u00b7 Window: P pause \u00b7 R redo \u00b7 S skip section \u00b7 Esc stop"
# The headset button: a press, two (redo) and a hold (stop): ButtonGestures. The window's keys follow
# by letter: the window lists what they do.
KEYS_STEP_BUTTON = ("Headset button: press for next or pause \u00b7 press twice to redo \u00b7 hold to stop "
"\u00b7 Window keys: Space P R S Esc")
KEYS_AUTO_BUTTON = ("Headset button: press to pause \u00b7 press twice to redo \u00b7 hold to stop "
"\u00b7 Window keys: P R S Esc")
# The early no-hands stop (DESIGN.md, "Camera check"): the first step of this section has both
# hands up. If the live tracker publishes through its hold and never sees a hand, the session
# stops that step and asks: try again, or stop. Only when the tracker published in at least
@@ -93,6 +101,8 @@ NO_HANDS_TITLE = "I can't see your hands"
NO_HANDS_TEXT = "The hand tracker didn't see either of your hands during that whole step."
NO_HANDS_RETRY = ("Try again: hold both hands up in front of you, about 40 cm away. To stop instead: "
"Esc or Stop in the Hand recorder window.")
NO_HANDS_RETRY_BUTTON = ("Try again: hold both hands up in front of you, about 40 cm away, and press the headset "
"button. To stop instead: hold the button, or Esc or Stop in the Hand recorder window.")
def mono_ns():
@@ -131,6 +141,13 @@ def host_command(*cmd):
exe] + list(cmd)
def fix_hint(frametop):
"""How to repair a missing or broken part: Frametop's command, or the standalone install's."""
if STANDALONE:
return STANDALONE.get("reinstall") or "run the Hand Recorder's install command again"
return frametop
def host_path(path):
"""A host file, from the container (/run/host) or the host itself."""
for p in ("/run/host" + path, path):
@@ -310,8 +327,9 @@ def classify_lighting(ring):
Nor does a sunlit room reliably: the first daylight round (dataset PR #5, big sunlit windows)
read 2.38, since the windows are a small part of each picture and the mean barely moves. So
"indoor" means "no strong daylight on the cameras", and the window asks people to pick
daylight themselves. What did show it there: hands only ~1.15x as bright as their surroundings
(1.5-1.7x in lamp-lit rooms), which needs hands in view, so it's measured on the dataset side."""
daylight themselves. Review corrects a missed one from the pictures (sunlit windows, the time
of day): hands standing out little from the room goes with daylight but also with pale rooms
at night, so it isn't a test either."""
ir = ambient_ir(ring)
if ir is None:
return ""
@@ -374,10 +392,10 @@ def start_camd(path=None, log=lambda line: None, stop=lambda: False, timeout=15.
if ring_alive(path):
return False
if not os.access(FT_CAMD, os.X_OK):
raise RuntimeError("ft-camd isn't built: hands/build.sh")
raise RuntimeError("ft-camd isn't built: " + fix_hint("hands/build.sh"))
caps = subprocess.run(["getcap", FT_CAMD], capture_output=True, text=True) if shutil.which("getcap") else None
if caps is not None and "cap_sys_ptrace" not in caps.stdout:
raise RuntimeError("ft-camd needs its capabilities: hands/run.sh caps (asks for sudo)")
raise RuntimeError("ft-camd needs its capabilities: " + fix_hint("hands/run.sh caps (asks for sudo)"))
started = start_unit(CAMD_UNIT, "the camera broker", [FT_CAMD, "--status", "60"], log)
end = time.monotonic() + timeout
while time.monotonic() < end:
@@ -619,7 +637,7 @@ class Recorder:
"--sides", "auto" if swap is None else "1" if swap else "0"]
if ring:
argv += ["--ring", ring]
if not in_container():
if not in_container() and not STANDALONE:
argv = [os.path.expanduser("~/.local/bin/distrobox"), "enter", "dev", "--"] + argv
self.proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=log_file, stderr=log_file)
self.started_ns = mono_ns()
@@ -678,7 +696,9 @@ EV_KEY, EV_REL = 0x01, 0x02
REL_X, REL_Y = 0x00, 0x01
KEY_SELECT = 353
BUTTON_DEVICE = "gpio-keys"
BUTTON_DEBOUNCE_S = 0.3 # presses closer than this count once
BUTTON_DOUBLE_S = 0.45 # a second press this soon after the first one's release: a double press
BUTTON_HOLD_S = 1.5 # held down this long: a hold
BUTTON_BOUNCE_S = 0.03 # a release and press closer than this are one press (gpio-keys debounces too)
def parse_input_devices(text):
@@ -758,29 +778,84 @@ def mouse_connected(path=INPUT_DEVICES):
return any(real_mouse(d) for d in read_input_devices(path))
def button_presses(data):
"""KEY_SELECT key-downs in a run of input_event structs (value 1; releases and autorepeat
are left out). Returns (how many, the bytes left over after the last whole event)."""
n, usable = 0, len(data) - len(data) % INPUT_EVENT.size
def button_events(data):
"""KEY_SELECT downs and ups in a run of input_event structs: [(True for a down or False for
an up, the event's time in seconds), ...] (autorepeat, value 2, is left out). Returns (them,
the bytes left over after the last whole event)."""
out, usable = [], len(data) - len(data) % INPUT_EVENT.size
for off in range(0, usable, INPUT_EVENT.size):
_, _, etype, code, value = INPUT_EVENT.unpack_from(data, off)
if etype == EV_KEY and code == KEY_SELECT and value == 1:
n += 1
return n, data[usable:]
sec, usec, etype, code, value = INPUT_EVENT.unpack_from(data, off)
if etype == EV_KEY and code == KEY_SELECT and value in (0, 1):
out.append((value == 1, sec + usec / 1e6))
return out, data[usable:]
class ButtonGestures:
"""The headset button's downs and ups, with their times, as gestures:
"press": one short press, reported once the double-press time has passed without a second;
"double": two presses, the second down within double_s of the first one's release, reported
at the second down;
"hold": held down for hold_s, reported while still down.
A release and a press closer than bounce_s are one press. Each call returns the gestures that
are due, in order; tick() is called now and then to report the ones that are due by time."""
def __init__(self, double_s=BUTTON_DOUBLE_S, hold_s=BUTTON_HOLD_S, bounce_s=BUTTON_BOUNCE_S):
self.double_s, self.hold_s, self.bounce_s = double_s, hold_s, bounce_s
self.down_at = None # when the button went down, while it's down
self.done = False # the press that's down was already reported (a double, a hold)
self.released = None # when a short press was let go, until it's reported or doubled
self._up = None # the last release, for the bounce: (time, down_at, done, released before it)
def down(self, t):
if self.down_at is not None: # a missed release
return []
if self._up and t - self._up[0] < self.bounce_s: # a bounce: still the same press
_, self.down_at, self.done, self.released = self._up
self._up = None
return []
out = self.tick(t)
self.down_at, self.done = t, False
if self.released is not None: # within the double-press time (else tick reported it)
self.released, self.done = None, True
out.append("double")
return out
def up(self, t):
if self.down_at is None:
return []
out = self.tick(t)
self._up = (t, self.down_at, self.done, self.released)
if not self.done:
self.released = t
self.down_at, self.done = None, False
return out
def tick(self, t):
if self.down_at is not None and not self.done and t - self.down_at >= self.hold_s:
self.done = True
return ["hold"]
if self.released is not None and t - self.released > self.double_s:
self.released = None
return ["press"]
return []
def waiting(self):
"""Something is due by time alone: a hold, or a press that may still become a double."""
return (self.down_at is not None and not self.done) or self.released is not None
class ButtonReader:
"""Reads the headset button on a thread and calls on_press() per press, debounced. path:
an event device or, for testing, a FIFO carrying input_event structs. It's opened read-only
and shared; if it can't be opened (no device, no permission, /dev/input not reachable in a
container) it says so in the log and tries again now and then."""
"""Reads the headset button on a thread and calls on_gesture("press" | "double" | "hold")
(ButtonGestures). path: an event device or, for testing, a FIFO carrying input_event
structs. It's opened read-only and shared; if it can't be opened (no device, no permission,
/dev/input not reachable in a container) it says so in the log and tries again now and then."""
def __init__(self, path, on_press, log=None, debounce_s=BUTTON_DEBOUNCE_S):
self.path, self.on_press, self.log = path, on_press, log or (lambda s: None)
self.debounce_s = debounce_s
def __init__(self, path, on_gesture, log=None, double_s=BUTTON_DOUBLE_S, hold_s=BUTTON_HOLD_S,
bounce_s=BUTTON_BOUNCE_S):
self.path, self.on_gesture, self.log = path, on_gesture, log or (lambda s: None)
self.gestures = ButtonGestures(double_s, hold_s, bounce_s)
self.ok = False # opened at least once
self._stop = threading.Event()
self._last = -1e9
self._thread = threading.Thread(target=self._run, name="handrec-button", daemon=True)
def start(self):
@@ -791,11 +866,9 @@ class ButtonReader:
self._stop.set()
self._thread.join(2)
def _press(self, n):
now = time.monotonic()
if n and now - self._last >= self.debounce_s:
self._last = now
self.on_press()
def _report(self, gestures):
for g in gestures:
self.on_gesture(g)
def _run(self):
failed = False
@@ -814,8 +887,9 @@ class ButtonReader:
rest = b""
try:
while not self._stop.is_set():
r, _, _ = select.select([fd], [], [], 0.2)
r, _, _ = select.select([fd], [], [], 0.02 if self.gestures.waiting() else 0.2)
if not r:
self._report(self.gestures.tick(time.monotonic()))
continue
try:
data = os.read(fd, INPUT_EVENT.size * 64)
@@ -824,8 +898,13 @@ class ButtonReader:
if not data: # a FIFO's writer left: open it again
self._stop.wait(0.2)
break
n, rest = button_presses(rest + data)
self._press(n)
events, rest = button_events(rest + data)
# The kernel's times (the realtime clock) keep the gaps between events that
# came in one read; the last one is taken as now.
now = time.monotonic()
for is_down, t in events:
at = now - min(max(events[-1][1] - t, 0.0), 2.0)
self._report(self.gestures.down(at) if is_down else self.gestures.up(at))
except OSError as e: # the device went away
self.log("headset button: %s: %s" % (self.path, e.strerror))
self._stop.wait(2)
@@ -1122,15 +1201,6 @@ class _Redo(Exception):
pass
def _git_describe():
try:
r = subprocess.run(["git", "-C", REPO, "describe", "--always", "--dirty", "--tags"],
capture_output=True, text=True, timeout=5)
return r.stdout.strip() or "unknown"
except (OSError, subprocess.TimeoutExpired):
return "unknown"
def _os_version():
path = "/run/host/etc/os-release" if in_container() else "/etc/os-release"
try:
@@ -1198,8 +1268,8 @@ def camera_text(result):
return camcheck.USER_TEXT
if camcheck.is_ring_short(result):
return ("The headset's tracking cameras are all running, but the recorder can't read some of them (%s). "
"Close the Hand Recorder, run ~/frametop/hands/rec/install.sh again, and open it again. If that "
"doesn't help, ask in the Frametop Discord." % result.get("reason", ""))
"Close the Hand Recorder, %s, and open it again. If that doesn't help, ask in the Frametop "
"Discord." % (result.get("reason", ""), fix_hint("run ~/frametop/hands/rec/install.sh again")))
return ("Not all of the headset's tracking cameras are running (%s). Restart SteamVR, or restart the "
"headset if that doesn't fix it." % result.get("reason", ""))
@@ -1302,8 +1372,25 @@ class Session:
self._want[key] = value
self._wake.set()
def button_gesture(self, gesture):
"""The headset's button (ButtonGestures). A press: Next while a step waits, pause during
a countdown or hold (and auto mode's timed screens), resume while paused. A double press:
redo, as R. A hold: stop, as Esc."""
if gesture == "press":
self.button_press()
elif gesture == "double":
if self._status["state"] in ("done", "stopped", "error") or not self._status.get("can_redo"):
return
self.redo()
self._log("headset button (redo)")
elif gesture == "hold":
if self._status["state"] in ("done", "stopped", "error"):
return
self._log("headset button (stop)")
self.stop(wait=0)
def button_press(self):
"""The headset's button: Next while a step waits, pause during a countdown or hold
"""The headset button's press: Next while a step waits, pause during a countdown or hold
(and auto mode's timed screens), resume while paused."""
with self._lock:
paused = self._want["pause"]
@@ -1449,7 +1536,7 @@ class Session:
self._ring_path = ring_path
if not self.dry_run:
if not os.access(FT_HANDS, os.X_OK):
raise _Fail("ft-hands isn't built: hands/build.sh")
raise _Fail("ft-hands isn't built: " + fix_hint("hands/build.sh"))
self._ensure_ring(ring_path)
if not tracker_running():
if self.start_processes:
@@ -1467,7 +1554,7 @@ class Session:
pass
removed = self._write_calibration() + self._write_device()
self._session_json = {
"schema": 1, "tool": "ft-handrec " + _git_describe(), "started": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
"schema": 1, "tool": takes.tool_version(), "started": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
"contributor": self.profile.get("contributor", ""),
"lighting": lighting_record(self.lighting_choice, lighting),
"checklist": self.checklist,
@@ -1512,7 +1599,8 @@ class Session:
if not path:
self._log("headset button: no %s device with KEY_SELECT in %s" % (BUTTON_DEVICE, INPUT_DEVICES))
return
self._button = ButtonReader(path, self.button_press, log=self._log, debounce_s=BUTTON_DEBOUNCE_S).start()
self._button = ButtonReader(path, self.button_gesture, log=self._log, double_s=BUTTON_DOUBLE_S,
hold_s=BUTTON_HOLD_S, bounce_s=BUTTON_BOUNCE_S).start()
end = time.monotonic() + 0.5 # opened in a moment, or it isn't reachable
while not self._button.ok and time.monotonic() < end:
time.sleep(0.01)
@@ -1604,11 +1692,12 @@ class Session:
raise _Stop()
def _start_tracker(self):
up = os.path.join(REPO, "scripts", "container-up.sh")
if os.access(up, os.X_OK):
subprocess.run(host_command(up), capture_output=True, timeout=120)
argv = [os.path.expanduser("~/.local/bin/distrobox"), "enter", "dev", "--", FT_HANDS,
"--no-gestures", "--status", "0"]
argv = [FT_HANDS, "--no-gestures", "--status", "0"]
if not STANDALONE: # built in the dev container: it runs there
up = os.path.join(REPO, "scripts", "container-up.sh")
if os.access(up, os.X_OK):
subprocess.run(host_command(up), capture_output=True, timeout=120)
argv = [os.path.expanduser("~/.local/bin/distrobox"), "enter", "dev", "--"] + argv
if self.ring:
argv += ["--ring", self.ring]
self._start_unit(HANDS_UNIT, "hand tracking for feedback", argv)
@@ -1631,7 +1720,7 @@ class Session:
self._log("using the ft-handpanel that's running")
return
if not os.access(self.panel_bin, os.X_OK):
raise _Fail("ft-handpanel isn't built: hands/rec/build.sh")
raise _Fail("ft-handpanel isn't built: " + fix_hint("hands/rec/build.sh"))
self._panel_proc = subprocess.Popen([self.panel_bin, "--watch-stdin"], stdin=subprocess.PIPE,
stdout=self._log_file, stderr=self._log_file)
end = time.monotonic() + 15
@@ -1747,9 +1836,10 @@ class Session:
if self._recording:
self._stop_recording()
self._event("pause")
hint = RESUME_HINT_BUTTON if self._button_ok() else RESUME_HINT
self._panel.cmd("paused on")
self._panel.set("note", "note " + RESUME_HINT)
self._emit(state="paused", note=RESUME_HINT)
self._panel.set("note", "note " + hint)
self._emit(state="paused", note=hint)
self._log("paused")
def _unpause(self, record=True):
@@ -1935,10 +2025,13 @@ class Session:
for e in items)))
self._status.update(strip=items, cue=cue)
def _button_ok(self):
return bool(self._button and self._button.ok)
def _hints(self, action=True):
"""The Next hint (with action) and the key line for what's there now: the headset button
leads when no mouse is connected. Looked at again for each step, so a mouse plugged in counts."""
button = bool(self._button and self._button.ok)
button = self._button_ok()
mouse = mouse_connected(self.input_devices)
text = (READY_BUTTON_MOUSE if mouse else READY_BUTTON) if button else READY_TEXT
keys = (KEYS_AUTO_BUTTON if self.auto else KEYS_STEP_BUTTON) if button else (KEYS_AUTO if self.auto else KEYS_STEP)
@@ -2244,7 +2337,7 @@ class Session:
for line in (cam or {}).get("evidence", []):
self._log(" " + line)
text = "%s|%s|%s" % (NO_HANDS_TEXT, cam_text or "The camera check found nothing wrong (%s)." % summary,
NO_HANDS_RETRY)
NO_HANDS_RETRY_BUTTON if self._button_ok() else NO_HANDS_RETRY)
self._stop_note = "Stopped: no hands were seen in the first step. Camera check: %s." % summary
if cam_text:
self._stop_note += " " + cam_text
@@ -2497,7 +2590,8 @@ def main():
help="test: press Next by itself after S seconds of waiting (real time)")
ap.add_argument("--poses", help="the pose pictures' folder, with poses.json (default hands/rec/poses)")
ap.add_argument("--no-headset-button", action="store_true",
help="don't read the headset's button (gpio-keys KEY_SELECT: Next, pause, resume)")
help="don't read the headset's button (gpio-keys KEY_SELECT: a press is Next, pause or resume; "
"two are redo; a hold is stop)")
ap.add_argument("--button-device", metavar="PATH",
help="test: read the button from this event device or FIFO of input_event structs (also in a dry run)")
ap.add_argument("--quick", action="store_true",
+4 -3
View File
@@ -1,8 +1,9 @@
"""Which side camera is which, in recordings: the rules every reader shares.
ft-camd tells slam_left's buffers from slam_right's by the order XRService allocated them, and
some XRService starts reverse it: then each side camera's images carry the other's name. A
tracking ft-hands tells from the hands (hands/track/sides.h, HANDS_SWAP_SIDES=auto) and publishes
Before 2026-10-05, ft-hands named the side cameras by XRService's start-up order and ft-camd bound
their buffers by XRService's descriptor order, so on most starts each side camera's images carried
the other's name (hands/README.md, "Which camera is which"). Both are exact now, and a
tracking ft-hands still tells from the hands (hands/track/sides.h, HANDS_SWAP_SIDES=auto) and publishes
what it found in /run/user/UID/frametop-hands/sides.json (read_live). Two things are recorded:
swapped whether ft-camd's naming was backwards during the recording (the truth);
+19 -1
View File
@@ -83,8 +83,26 @@ def find_zstd():
return next((p for p in ZSTD_PATHS if os.access(p, os.X_OK)), None)
def standalone(path=None):
"""The standalone Hand Recorder's build info, or None in a Frametop checkout. Its release
(github.com/Frametop/frametop-hand-recorder) ships this repo's tree with standalone.json at
the top: {"name", "version", "frametop": this repo's git describe, "reinstall": how to repair
an install}. Its binaries are built for the SteamOS host, so nothing runs in the dev container."""
try:
with open(path or os.path.join(REPO, "standalone.json")) as f:
info = json.load(f)
except (OSError, ValueError):
return None
return info if isinstance(info, dict) else None
def tool_version():
"""ft-handrec plus the checkout's git describe, for session.json and the manifest."""
"""ft-handrec plus the checkout's git describe, for session.json and the manifest. The
standalone Hand Recorder has no checkout: the describe it was built from, and its version."""
info = standalone()
if info:
return "ft-handrec %s (%s %s)" % (info.get("frametop") or "unknown", info.get("name") or "standalone",
info.get("version") or "unknown")
try:
out = subprocess.run(["git", "-C", REPO, "describe", "--always", "--dirty", "--tags"],
capture_output=True, text=True, timeout=5)
+4 -2
View File
@@ -2,7 +2,8 @@
"""main.qml against ft_handrec.Backend: every backend.name(...) the window calls is a slot, and
every backend.name it reads is a property or a slot. A method that lost its @Slot shows up in
QML only as "is not a function" when its button is pressed (2026-10-03: Export did nothing).
Needs PySide6 (the dev container); skipped without it.
Needs PySide6 (the dev container); skipped without it. FT_HANDREC_QML=PATH checks another
window against the same backend (the standalone Hand Recorder's, in its CI).
python3 hands/rec/tests/test_qml_backend.py
"""
@@ -14,6 +15,7 @@ import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
REC = os.path.dirname(HERE)
sys.path.insert(0, REC)
QML = os.environ.get("FT_HANDREC_QML") or os.path.join(REC, "main.qml")
class QmlBackendTest(unittest.TestCase):
@@ -25,7 +27,7 @@ class QmlBackendTest(unittest.TestCase):
meta = self.meta = ft_handrec.Backend.staticMetaObject
self.slots = {bytes(meta.method(i).name()).decode() for i in range(meta.methodCount())}
self.props = {meta.property(i).name() for i in range(meta.propertyCount())}
with open(os.path.join(REC, "main.qml")) as f:
with open(QML) as f:
self.qml = f.read()
def test_calls_are_slots(self):
+154 -17
View File
@@ -458,8 +458,15 @@ B: MSC=10
"""
def event(etype, code, value):
return session.INPUT_EVENT.pack(1, 2, etype, code, value)
def event(etype, code, value, t=1.000002):
return session.INPUT_EVENT.pack(int(t), round(t % 1 * 1e6), etype, code, value)
def taps(*times):
"""Presses, each 50 ms down, at these times (the kernel's clock), as one write."""
return b"".join(event(session.EV_KEY, session.KEY_SELECT, 1, t) + event(0, 0, 0, t)
+ event(session.EV_KEY, session.KEY_SELECT, 0, t + 0.05) + event(0, 0, 0, t + 0.05)
for t in times)
PRESS = event(session.EV_KEY, session.KEY_SELECT, 1) + event(0, 0, 0)
@@ -479,35 +486,74 @@ class InputTest(unittest.TestCase):
# another gpio-keys without KEY_SELECT isn't the button
self.assertIsNone(session.find_button(session.parse_input_devices(DEVICES.replace("200000000", "0"))))
def test_presses(self):
data = (PRESS + event(session.EV_KEY, session.KEY_SELECT, 2) * 3 # autorepeat: not presses
def test_events(self):
data = (PRESS + event(session.EV_KEY, session.KEY_SELECT, 2) * 3 # autorepeat: left out
+ RELEASE + event(session.EV_KEY, 115, 1) + PRESS) # another key
self.assertEqual(session.button_presses(data), (2, b""))
n, rest = session.button_presses(PRESS + PRESS[:10])
self.assertEqual((n, rest), (1, PRESS[:10])) # half an event waits for the rest
self.assertEqual(session.button_presses(rest + PRESS[10:])[0], 1)
self.assertEqual(session.button_events(data), ([(True, 1.000002), (False, 1.000002), (True, 1.000002)], b""))
out, rest = session.button_events(PRESS + PRESS[:10])
self.assertEqual((out, rest), ([(True, 1.000002)], PRESS[:10])) # half an event waits for the rest
self.assertEqual(session.button_events(rest + PRESS[10:])[0], [(True, 1.000002)])
self.assertEqual([t for _, t in session.button_events(taps(5.25))[0]], [5.25, 5.3])
def test_gestures(self):
def run(steps, **kw):
"""steps: (time, "down" | "up" | "tick"). The gestures, with the time each came."""
g, out = session.ButtonGestures(double_s=0.4, hold_s=1.5, bounce_s=0.05, **kw), []
for t, what in steps:
out += [(t, x) for x in getattr(g, what)(t)]
return out
# a press comes once the double-press time has passed
self.assertEqual(run([(0, "down"), (0.1, "up"), (0.3, "tick"), (0.6, "tick")]), [(0.6, "press")])
# two: a double, at the second down, and nothing at its release
self.assertEqual(run([(0, "down"), (0.1, "up"), (0.4, "down"), (0.5, "up"), (2, "tick")]),
[(0.4, "double")])
# the second too late: two presses (the first reported when the second goes down)
self.assertEqual(run([(0, "down"), (0.1, "up"), (0.6, "down"), (0.7, "up"), (1.2, "tick")]),
[(0.6, "press"), (1.2, "press")])
# a hold: while still down, and nothing at the release
self.assertEqual(run([(0, "down"), (1.0, "tick"), (1.5, "tick"), (3, "up"), (4, "tick")]),
[(1.5, "hold")])
# a hold noticed only at the release (a slow read) still counts
self.assertEqual(run([(0, "down"), (2, "up"), (3, "tick")]), [(2, "hold")])
# a bounce (up and down 20 ms apart) is one press
self.assertEqual(run([(0, "down"), (0.1, "up"), (0.12, "down"), (0.2, "up"), (1, "tick")]),
[(1, "press")])
# a press, then a hold soon after: a double (the hold counts from a fresh down)
self.assertEqual(run([(0, "down"), (0.1, "up"), (0.3, "down"), (2, "tick"), (2.1, "up")]),
[(0.3, "double")])
# a release with no down, and two downs: left alone
self.assertEqual(run([(0, "up"), (1, "down"), (1.1, "down"), (1.2, "up"), (2, "tick")]),
[(2, "press")])
def test_reader_fifo(self):
tmp = tempfile.mkdtemp(prefix="handrec-button-test-")
try:
fifo = os.path.join(tmp, "button")
os.mkfifo(fifo)
presses = []
reader = session.ButtonReader(fifo, lambda: presses.append(time.monotonic()), debounce_s=0.3).start()
got = []
reader = session.ButtonReader(fifo, got.append, double_s=0.15, hold_s=0.5).start()
with open(fifo, "wb", buffering=0) as w:
w.write(PRESS + RELEASE)
w.write(PRESS[:7]) # a press split across writes, a bounce
time.sleep(0.05)
w.write(PRESS + RELEASE) # a press
time.sleep(0.3)
w.write(PRESS[:7]) # two, the first split across writes
time.sleep(0.02)
w.write(PRESS[7:] + RELEASE)
time.sleep(0.4)
time.sleep(0.07)
w.write(PRESS + RELEASE)
time.sleep(0.3)
w.write(taps(10, 10.1)) # two in one read: the kernel's times tell them apart
time.sleep(0.3)
w.write(PRESS) # held
time.sleep(0.7)
self.assertEqual(got, ["press", "double", "double", "hold"]) # the hold came before the release
w.write(RELEASE)
time.sleep(0.2)
with open(fifo, "wb", buffering=0) as w: # the writer comes back: read again
time.sleep(0.4)
w.write(PRESS)
w.write(PRESS + RELEASE)
time.sleep(0.3)
reader.stop()
self.assertEqual(len(presses), 3)
self.assertEqual(got, ["press", "double", "double", "hold", "press"])
self.assertTrue(reader.ok)
finally:
shutil.rmtree(tmp, ignore_errors=True)
@@ -533,7 +579,7 @@ class ButtonSessionTest(SessionBase):
writer = os.open(fifo, os.O_RDWR) # kept open, so the reader never sees the end
press = lambda: os.write(writer, PRESS + RELEASE)
try:
with mock.patch.object(session, "BUTTON_DEBOUNCE_S", 0.0):
with mock.patch.object(session, "BUTTON_DOUBLE_S", 0.1):
s = self.session(speed=4, button_device=fifo)
s.input_devices = procfile
s.start()
@@ -563,6 +609,41 @@ class ButtonSessionTest(SessionBase):
finally:
os.close(writer)
def test_redo_and_stop(self):
"""Two presses redo the step, a hold stops the session; the hints say so."""
fifo = os.path.join(self.tmp, "button")
os.mkfifo(fifo)
writer = os.open(fifo, os.O_RDWR)
press = lambda: os.write(writer, PRESS + RELEASE)
try:
with mock.patch.multiple(session, BUTTON_DOUBLE_S=0.15, BUTTON_HOLD_S=0.4):
s = self.session(speed=2, button_device=fifo)
s.start()
self.waiting(s, "starting")
press()
self.waiting(s, "intro")
press()
self.waiting(s, "ready", "Fist.")
press()
self.wait_for(s, lambda st: st["state"] == "running", "the hold")
os.write(writer, taps(1, 1.1)) # twice: redo
self.waiting(s, "ready", "Fist.")
time.sleep(0.1) # (sooner would be a bounce of the last)
press()
self.wait_for(s, lambda st: st["state"] == "running", "the hold again")
press() # pause
st = self.wait_for(s, lambda st: st["state"] == "paused", "paused")
self.assertEqual(st["note"], session.RESUME_HINT_BUTTON)
os.write(writer, PRESS) # held: stop
s.join(20)
os.write(writer, RELEASE)
self.assertEqual(s.state, "stopped")
names = [e["event"] for e in self.events(s)]
self.assertEqual(names, ["take", "ready", "prompt", "redo", "wait", "ready", "prompt", "pause", "end"])
self.assertIn("panel: keys " + session.KEYS_STEP_BUTTON, self.panel)
finally:
os.close(writer)
def test_no_button(self):
s = self.session(next_after=0.0, button=False, button_device="/nonexistent")
s.start()
@@ -630,5 +711,61 @@ class ManyPartsTest(unittest.TestCase):
self.assertEqual(r.summary["sets"], 120 + 30)
class StandaloneTest(unittest.TestCase):
"""The standalone Hand Recorder's tree (standalone.json at the top): ft-hands runs on the
host, the version comes from the build info, and repairs point at its install command."""
INFO = {"name": "frametop-hand-recorder", "version": "0.1.0", "frametop": "4ba49af",
"reinstall": "run the install command again"}
def test_build_info(self):
with tempfile.TemporaryDirectory() as d:
path = os.path.join(d, "standalone.json")
self.assertIsNone(takes.standalone(path))
with open(path, "w") as f:
json.dump(self.INFO, f)
self.assertEqual(takes.standalone(path), self.INFO)
with open(path, "w") as f:
f.write("[1]")
self.assertIsNone(takes.standalone(path))
with mock.patch.object(takes, "standalone", return_value=self.INFO):
self.assertEqual(takes.tool_version(), "ft-handrec 4ba49af (frametop-hand-recorder 0.1.0)")
def recorder_argv(self, standalone):
with tempfile.TemporaryDirectory() as d, mock.patch.object(session, "STANDALONE", standalone), \
mock.patch.object(session, "in_container", return_value=False), \
mock.patch.object(session.subprocess, "Popen") as popen:
session.Recorder(d, 1, 10, None, None)
return popen.call_args[0][0]
def tracker_argv(self, standalone):
fake = mock.Mock(ring=None)
with mock.patch.object(session, "STANDALONE", standalone), \
mock.patch.object(session.subprocess, "run") as run:
session.Session._start_tracker(fake)
return fake._start_unit.call_args[0][2], run
def test_ft_hands_on_the_host(self):
self.assertEqual(self.recorder_argv(self.INFO)[0], session.FT_HANDS)
argv, run = self.tracker_argv(self.INFO)
self.assertEqual(argv[0], session.FT_HANDS)
self.assertIn("--no-gestures", argv)
run.assert_not_called() # no container to bring up
def test_ft_hands_in_the_dev_container(self):
self.assertTrue(self.recorder_argv(None)[0].endswith("distrobox"))
argv, _ = self.tracker_argv(None)
self.assertEqual(argv[:4], [os.path.expanduser("~/.local/bin/distrobox"), "enter", "dev", "--"])
self.assertEqual(argv[4], session.FT_HANDS)
def test_fix_hint(self):
with mock.patch.object(session, "STANDALONE", self.INFO):
self.assertEqual(session.fix_hint("hands/build.sh"), "run the install command again")
text = session.camera_text({"status": "degraded", "reason": "2 of 4", "ring_missing": ["upper_left"]})
with mock.patch.object(session, "STANDALONE", None):
self.assertEqual(session.fix_hint("hands/build.sh"), "hands/build.sh")
self.assertIn("run the install command again", text)
self.assertNotIn("~/frametop", text)
if __name__ == "__main__":
unittest.main()
+18
View File
@@ -148,6 +148,24 @@ class SyntheticLogs(unittest.TestCase):
st = state(START + GOOD_OPEN)
self.assertEqual(st.camera_map(), {"slam_left": 9, "slam_right": 13, "upper_left": 6, "upper_right": 7})
def test_camera_map_by_found_camera(self):
"""XRService's own naming (2026-10-05's log): index 0 is slam_right, by its subdev."""
found = [L("10:00:00", "DeckardCaptureSource: Found camera '%s': interface=msm_csiphy%d v4l_subdev=/dev/v4l-subdev%d "
"driver=/sys/bus/i2c/drivers/%s" % f) for f in (
("slam_left", 0, 30, "og01a1b/4-0060"), ("slam_right", 1, 31, "og01a1b/4-0036"),
("upper_left", 0, 32, "og0ve10/5-0060"), ("upper_right", 1, 33, "og0ve10/5-003e"))]
inits = [L("10:00:03", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: /dev/v4l-subdev%d"
% f) for f in ((0, 9, 31), (1, 13, 30), (2, 6, 32), (3, 7, 33))]
st = state(START + found + GOOD_OPEN[:-4] + inits)
self.assertEqual(st.camera_map(), {"slam_right": 9, "slam_left": 13, "upper_left": 6, "upper_right": 7})
# without the colour module the sides are on video0 and video3, named the same way
unplug = [L("10:30:00", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: /dev/v4l-subdev%d"
% f) for f in ((0, 0, 31), (1, 3, 30))]
st = state(START + found + GOOD_OPEN[:-4] + inits + CLOSE + unplug)
self.assertEqual(st.camera_map(), {"slam_right": 0, "slam_left": 3})
# a new instance forgets the old one's names: back to the index order
self.assertEqual(state(START + found + START + GOOD_OPEN).camera_map()["slam_left"], 9)
def test_module_unplugged(self):
st = state(START + GOOD_OPEN + UNPLUG)
self.assertEqual(st.verdict()[0], "ok")
+4 -4
View File
@@ -14,9 +14,9 @@ line: {"pair", "verdict": "named"|"swapped"|"unknown", "named", "swapped", "matc
ft-hands decides the side cameras' naming by itself (HANDS_SWAP_SIDES=auto, track/sides.h);
this is the independent check, from the scene rather than hands.
ft-camd tells the two side cameras' buffers apart by the order XRService allocated them,
and after some XRService restarts that order puts each camera's images under the other's
name. The tracker then sees every hand in one camera only, at the wrong depth. This
Before 2026-10-05 the side cameras' images often carried each other's names (hands/README.md,
"Which camera is which"); recordings from then may still. The tracker then sees every hand in
one camera only, at the wrong depth. This
matches features between the two images and measures how close each pair's rays pass
with the factory calibration, once as named and once swapped: true matches meet in
front of both cameras only under the right naming.
@@ -34,7 +34,7 @@ from tools.show_set import index, read_set # noqa: E402
from tools import calib # noqa: E402
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right', # with the colour module
PIPES = {'msm_vfe3_video0': 'slam_right', 'msm_vfe4_video0': 'slam_left', # with the colour module
'msm_vfe2_video0': 'upper_left', 'msm_vfe2_video1': 'upper_right'}
+52 -20
View File
@@ -22,9 +22,10 @@
// own clock, so they're placed on the mono cameras' by when they were dequeued, less the
// mono cameras' measured delay.
//
// Which side camera is which (--sides, HANDS_SWAP_SIDES): ft-camd tells slam_left's buffers from
// slam_right's by XRService's allocation order, which some XRService starts reverse. auto (the
// default) tells from the hands it tracks (track/sides.h): once it's sure, it exchanges the two
// Which side camera is which (--sides, HANDS_SWAP_SIDES): the names come from XRService's log
// (cameras_from_xrservice_log) and ft-camd matches each camera's buffers exactly (VIDIOC_QUERYBUF),
// so they should be right; before 2026-10-05 they were often swapped. auto (the default) still
// tells from the hands it tracks (track/sides.h): once it's sure, it exchanges the two
// cameras if they're backwards (the tracked views move with their images), and checks once
// more. 0 and 1 force the naming (1: exchanged; --swap-sides is --sides 1); it still checks,
// and if the hands disagree it warns and publishes what the hands say as the truth ("swapped"),
@@ -36,7 +37,9 @@
// options override both): HANDS_SWAP_SIDES (auto, 0 or 1), HANDS_CPUS (as --cpus),
// HANDS_CAMERAS, HANDS_BRIGHT, HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT (which
// colour camera is passthrough_left: color_video0 or color_video3), HANDS_COLOR_CROP
// (subtract or none: tools/check_color.py tells both).
// (subtract or none: tools/check_color.py tells both), HANDS_MISREAD_GUARD (0 or 1: the
// tracker's guards for fine-tuned landmark models, Tracker::set_misread_guard), HANDS_MODELS (as
// --models: a folder with palm.ncnn.* and hand.ncnn.*, e.g. fine-tuned ones).
#include "io.h"
#include "pinch.h"
#include "record.h"
@@ -47,6 +50,7 @@
#include <sys/stat.h>
#include <unistd.h>
#include <algorithm>
#include <cmath>
#include <ctime>
#include <memory>
@@ -64,31 +68,53 @@ namespace {
volatile std::sig_atomic_t g_stop = 0, g_record = 0;
// Which calibrated camera each video device carries. XRService numbers its tracking cameras
// (index 0 to 3: slam_left, slam_right, upper_left, upper_right) and logs the device each one
// opened ("TrackingCameraInit: index: 0. video device: /dev/video9"). The devices depend on
// the colour module: with it, the side cameras are on vfe3 and vfe4 and the upper pair on
// vfe2; without it, XRService runs the side cameras through the ISP on vfe0 and vfe1, and the
// upper pair on vfe3 and vfe4. So the running XRService's log decides; when it can't be read,
// the capture pipes as they are with the module. {} if the log has no cameras.
// Which calibrated camera each video device carries. XRService names each tracking camera's
// sensor subdev once per instance ("Found camera 'slam_left': interface=msm_csiphy0
// v4l_subdev=/dev/v4l-subdev30 ...") and logs the device and subdev each index opened at every
// camera start ("TrackingCameraInit: index: 0. video device: /dev/video9. v4l subdevice:
// /dev/v4l-subdev31"). The index is only the order it opens them in: on every start logged since
// 2026-10-04, index 0 was slam_right. So the subdev names the device; a log without "Found
// camera" lines falls back to the index order (slam_left, slam_right, upper_left, upper_right).
// The devices depend on the colour module: with it, the side cameras are on vfe3 (slam_right)
// and vfe4 (slam_left) and the upper pair on vfe2; without it, XRService runs the side cameras
// through the ISP on vfe0 and vfe1, and the upper pair on vfe3 and vfe4. So the running
// XRService's log decides; when it can't be read, the capture pipes as they are with the module.
// {} if the log has no cameras.
std::map<int, std::string> cameras_from_xrservice_log() {
static const char *const names[] = {"slam_left", "slam_right", "upper_left", "upper_right"};
const char *home = std::getenv("HOME");
std::ifstream in(std::string(home ? home : "") + "/.local/share/Steam/logs/xrservice.txt");
const std::string key = "TrackingCameraInit: index: ";
std::map<int, int> node_of; // index -> N of /dev/videoN, from the latest camera start
const std::string key = "TrackingCameraInit: index: ", found_key = "Found camera '";
std::map<int, std::pair<int, std::string>> init; // index -> (N of /dev/videoN, subdev), latest start
std::map<std::string, std::string> name_of; // subdev -> camera name
std::string line;
while (std::getline(in, line)) {
if (line.find("XRService logging to") != std::string::npos) node_of.clear();
if (line.find("XRService logging to") != std::string::npos) init.clear(), name_of.clear();
if (const auto at = line.find(found_key); at != std::string::npos) {
const auto name_end = line.find('\'', at + found_key.size());
const auto sub = line.find("v4l_subdev=", at);
if (name_end != std::string::npos && sub != std::string::npos) {
const auto sub_end = line.find_first_of(" \t\r", sub + 11);
name_of[line.substr(sub + 11, sub_end == std::string::npos ? std::string::npos : sub_end - sub - 11)] =
line.substr(at + found_key.size(), name_end - at - found_key.size());
}
continue;
}
const auto at = line.find(key);
int index = -1, node = -1;
char subdev[64] = "";
if (at != std::string::npos &&
std::sscanf(line.c_str() + at + key.size(), "%d. video device: /dev/video%d", &index, &node) == 2 &&
std::sscanf(line.c_str() + at + key.size(), "%d. video device: /dev/video%d. v4l subdevice: %63s", &index,
&node, subdev) >= 2 &&
index >= 0 && index < 4)
node_of[index] = node;
init[index] = {node, subdev};
}
std::map<int, std::string> out;
for (auto &[index, node] : node_of) out[node] = names[index];
for (auto &[index, ns] : init) {
const auto it = name_of.find(ns.second);
const bool known = it != name_of.end() && std::find(std::begin(names), std::end(names), it->second) != std::end(names);
out[ns.first] = known ? it->second : names[index];
}
return out;
}
@@ -97,8 +123,8 @@ const char *camera_for_pipe(int node) {
std::snprintf(path, sizeof path, "/sys/class/video4linux/video%d/name", node);
std::ifstream f(path);
f.getline(name, sizeof name);
if (!std::strcmp(name, "msm_vfe3_video0")) return "slam_left";
if (!std::strcmp(name, "msm_vfe4_video0")) return "slam_right";
if (!std::strcmp(name, "msm_vfe3_video0")) return "slam_right";
if (!std::strcmp(name, "msm_vfe4_video0")) return "slam_left";
if (!std::strcmp(name, "msm_vfe2_video0")) return "upper_left";
if (!std::strcmp(name, "msm_vfe2_video1")) return "upper_right";
return nullptr;
@@ -215,6 +241,7 @@ int main(int argc, char **argv) {
// latency 9.6 against 14.1 ms, and the compositor's late frames and CPU/GPU time didn't change.
std::vector<int> cpus = {5, 6, 7};
if (const auto c = parse_cpus(setting("HANDS_CPUS").c_str()); !c.empty()) cpus = c;
if (const std::string m = setting("HANDS_MODELS"); !m.empty()) models = m;
// Which side camera is which (see the top): auto, 0 or 1, and where that came from
std::string sides_mode = setting("HANDS_SWAP_SIDES"), sides_from = "config";
if (sides_mode.empty()) sides_mode = "auto", sides_from = "default";
@@ -232,6 +259,7 @@ int main(int argc, char **argv) {
// dim recording), but makes the landmarks jitter, so they get plain crops.
Contrast palm_contrast, hand_contrast{Contrast::None};
double keep_presence = 0.5; // landmark presence a tracked view needs to stay
bool misread_guard = setting("HANDS_MISREAD_GUARD") == "1";
PinchParams pinch_params;
GripParams grip_params;
bool gesture_log = false; // what the pinch and grip detectors measure, 10 times a second
@@ -262,6 +290,7 @@ int main(int argc, char **argv) {
else if (a == "--record-for" && more) record_for = std::atof(argv[++i]);
else if (a == "--record-hz" && more) record_hz = std::max(0.0, std::atof(argv[++i]));
else if (a == "--keep-presence" && more) keep_presence = std::atof(argv[++i]);
else if (a == "--misread-guard" && more) misread_guard = std::string(argv[++i]) == "1";
else if (a == "--cams" && more) cams_arg = argv[++i];
else if (a == "--bright" && more) bright_arg = argv[++i];
else if (a == "--bright-on" && more) light.on = std::atof(argv[++i]);
@@ -282,6 +311,7 @@ int main(int argc, char **argv) {
" [--sides auto|0|1] (auto: tell from the hands which side camera is which; 1: exchange them,\n"
" as --swap-sides; 0: as ft-camd names them)\n"
" [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
" [--misread-guard 0|1] (0; 1 for fine-tuned landmark models: see Tracker::set_misread_guard)\n"
" [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
" [--color-left color_video0|color_video3] [--color-crop subtract|none]\n"
" [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (1: off)\n"
@@ -294,7 +324,8 @@ int main(int argc, char **argv) {
"camera's newest dark frame, as <name>_dk; with --with-color, the color cameras' as color_video<N>.\n"
"auto picks the cameras by the light (see the top of track/main.cpp).\n"
"Settings in ~/.config/frametop.conf: HANDS_SWAP_SIDES=auto|0|1, HANDS_CPUS=5,6,7, HANDS_CAMERAS, HANDS_BRIGHT,\n"
"HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT, HANDS_COLOR_CROP (FT_<name> overrides).\n",
"HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT, HANDS_COLOR_CROP, HANDS_MISREAD_GUARD=0|1, HANDS_MODELS=DIR\n"
"(FT_<name> overrides).\n",
argv[0]);
return a == "--help" ? 0 : 1;
}
@@ -466,6 +497,7 @@ int main(int argc, char **argv) {
Pool pool(threads, cpus);
Tracker tracker(used, nets, pool);
tracker.set_keep_presence(keep_presence);
tracker.set_misread_guard(misread_guard);
std::map<std::string, std::vector<uint8_t>> pixels;
std::map<std::string, uint64_t> lit_seen; // per colour camera: the frame last counted for the light
const uint64_t start = mono_ns();
+4
View File
@@ -14,6 +14,7 @@
// --timeline: per processed set, a line per hand (time, id, side, views, wrist) and per view
// (hand, camera, presence, next crop, set index).
// --keep-presence P: landmark presence a tracked view needs to stay (default 0.5, as new ones).
// --misread-guard: the tracker's guards for fine-tuned landmark models (Tracker::set_misread_guard).
// --pinch-begin M, --pinch-end M, --pinch-triangulated, --pinch-palm-down MAX: the pinch detector (track/pinch.h);
// the timeline gets its begin/end/lost events and both distance measures per set.
// --grip-begin R, --grip-end R: the grip detector (a closed hand; track/pinch.h); the timeline
@@ -86,6 +87,7 @@ int main(int argc, char **argv) {
bool cost = false;
Contrast palm_contrast, hand_contrast{Contrast::None}; // as ft-hands's
double keep_presence = 0.5; // landmark presence a tracked view needs to stay
bool misread_guard = false;
PinchParams pinch_params;
GripParams grip_params;
std::string use = "mono", color_left = "color_video0", color_crop = "subtract", sides = "file";
@@ -102,6 +104,7 @@ int main(int argc, char **argv) {
else if (a == "--models" && more) models = argv[++i];
else if (a == "--cost") cost = true;
else if (a == "--keep-presence" && more) keep_presence = std::atof(argv[++i]);
else if (a == "--misread-guard") misread_guard = true;
else if (a == "--cams" && more) use = argv[++i];
else if (a == "--sides" && more) sides = argv[++i];
else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]);
@@ -183,6 +186,7 @@ int main(int argc, char **argv) {
return n == "slam_left" ? "slam_right" : n == "slam_right" ? "slam_left" : n;
};
tracker.set_keep_presence(keep_presence);
tracker.set_misread_guard(misread_guard);
FILE *dp = depth.empty() ? nullptr : std::fopen(depth.c_str(), "w");
FILE *pp = poses.empty() ? nullptr : std::fopen(poses.c_str(), "w");
if (dp)
+24 -4
View File
@@ -223,8 +223,16 @@ bool Tracker::hand_3d(Hand &hand, std::vector<View *> views, int64_t t_ns) {
if (residual > 0.03 || size_misfit({views.begin(), views.end()}, pts) < 0) {
View *best = *std::max_element(views.begin(), views.end(),
[](View *a, View *b) { return a->lm.presence < b->lm.presence; });
for (View *v : views)
if (v != best) v->hand = -1;
// a view that disagrees but sits where this hand already is in its camera is this hand
// misread: drop it (-2), and the hand-over gives a fresh crop next frame
for (View *v : views) {
if (v == best) continue;
v->hand = -1;
if (!misread_guard_ || !hand.has_pts) continue;
double z;
const V2 at = v->cam->project(hand.pts[9], &z);
if (z > 0 && norm(at - palm_centre(v->lm)) < 0.5 * hand_size(v->lm)) v->hand = -2;
}
++stats.splits;
return hand_3d(hand, {best}, t_ns);
}
@@ -416,7 +424,14 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
if (int(chosen.size()) > hand_budget_) chosen.resize(hand_budget_);
run_landmarks(images, chosen);
std::vector<View *> kept;
for (View *v : chosen)
for (View *v : chosen) {
if (misread_guard_ && v->has_lm) { // see set_misread_guard
const auto h = hands_.find(v->hand);
if (h != hands_.end() && h->second.frames >= 5 && std::fabs(v->lm.right - h->second.right_score) > 0.7) {
++stats.lost;
continue;
}
}
if (v->lm.presence >= (v->frames > 0 ? keep_presence_ : min_presence_)) {
v->roi = v->lm.next_roi();
++v->frames;
@@ -424,6 +439,7 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
} else {
++(v->frames > 0 ? stats.lost : stats.handoff_miss);
}
}
// the same hand twice in one camera: keep the more confident
std::sort(kept.begin(), kept.end(), [](View *a, View *b) { return a->lm.presence > b->lm.presence; });
std::vector<View> next;
@@ -537,7 +553,11 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
++it;
}
}
// views split off by a failed triangulation start over as new hands next frame
// views split off by a failed triangulation start over as new hands next frame, unless they
// were this hand misread (-2, the misread guard: dropped)
const size_t before = views_.size();
views_.erase(std::remove_if(views_.begin(), views_.end(), [](const View &v) { return v.hand == -2; }), views_.end());
stats.lost += int(before - views_.size());
for (View &v : views_)
if (v.hand <= 0) {
v.hand = next_id_++;
+8
View File
@@ -93,6 +93,13 @@ public:
// bright rooms the camera exposes for the room, the hands come out dim, and presence
// dips under 0.5 for a frame at a time.
void set_keep_presence(double p) { keep_presence_ = p; }
// Misread guards, for fine-tuned landmark models (frame-hands' students): they stay sure of a
// hand when two hands touch and can read the held hand as the other side, which made a split /
// hand-over / duplicate loop. On: a reading of an established hand (5+ frames) whose side is
// more than 0.7 off the hand's own is dropped (the hand-over crops it afresh next frame), and a
// view split off where its hand already is in that camera is dropped instead of starting a new
// hand. Off by default: the stock model's side is noisier and the first guard costs it tracking.
void set_misread_guard(bool on) { misread_guard_ = on; }
// One view's 3D hand: each landmark along its ray, as far as how big the palm looks says
// for a hand `scale` times the model's (Hand::scale). False if the palm is degenerate.
static bool single_view(const Camera &cam, const Landmarks &lm, double scale, V3 out[21]);
@@ -130,6 +137,7 @@ private:
Pool &pool_;
int max_views_, hand_budget_ = 4, search_budget_ = 3;
double min_presence_ = 0.5, keep_presence_ = 0.5;
bool misread_guard_ = false;
std::vector<View> views_;
std::map<int, Hand> hands_;
std::vector<Tile> tiles_;
+16 -7
View File
@@ -4,10 +4,11 @@
#
# 1. Vibepollo 2.0.0 (github.com/Nonary/Vibepollo), installed with its own installer if it
# isn't here (checked against its SHA-256 first).
# 2. Frametop's build of Vibepollo's sunshine.exe (github.com/DeeJanuz/frametop-vibepollo):
# 2. Frametop's build of Vibepollo's sunshine.exe (github.com/Frametop/frametop-vibepollo):
# it can stream any of your monitors (not only the main one), keeps your monitor layout
# when a virtual display goes away, and doesn't stall the Web UI. Checked against its
# SHA-256; the original is kept as sunshine.exe.2.0.0-original.
# when a virtual display goes away, turns an HDR monitor's HDR off while Frametop streams
# it (back on after), and doesn't stall the Web UI. Downloaded from its release and
# checked against its SHA-256; the original is kept as sunshine.exe.2.0.0-original.
# 3. Settings Frametop needs (the rest of Vibepollo's settings stay as they are): remote
# displays carry the PC's sound, and a virtual display goes away when Frametop
# disconnects it, but stays through a dropped stream.
@@ -33,9 +34,14 @@ $VibepolloVersion = "2.0.0"
$SetupUrl = "https://github.com/Nonary/Vibepollo/releases/download/2.0.0/VibepolloSetup-v2.0.0.exe"
$SetupSha = "7B3500EC0C774644CE5A435A48F61C046C48494D0F18B67AFA0B3561931794B7"
$OriginalSha = "2CC018FD92DDB4D3748D91D8DA25316909ED45DB3710D1FF278BD73D51EB00C1" # its sunshine.exe
# Frametop's build: frametop-vibepollo branch frametop/2.0.0-remote-monitor-fix at 2f032252.
$BuildSha = "B5B7D2E7353454AEA6D895D0B68DE235E4CC581684C9DD44F8EFAF2E872F517F"
$BuildUrl = "" # not published yet
# Frametop's build: Frametop/frametop-vibepollo release frametop-2.0.0-1 (its CI, from the tag).
$BuildSha = "57EECCF9CA6ECEC4F0C5AEBBB27AAFE711A700C9E0986601344A52682856E94E"
$BuildUrl = "https://github.com/Frametop/frametop-vibepollo/releases/download/frametop-2.0.0-1/sunshine.exe"
# Earlier Frametop builds, replaced by this one like the original is.
$OlderBuildShas = @(
"B5B7D2E7353454AEA6D895D0B68DE235E4CC581684C9DD44F8EFAF2E872F517F" # 2f032252, built by hand without WebRTC
"6AF4F34503C7F6F84D1F6967D9FCEFAF2BD8145B34044C992B5E26A614E37A91" # a84b6cfc, CI, before the HDR-off guard
)
$Settings = [ordered]@{
"remote_monitor_mute_audio" = "disabled"
"remote_monitor_disconnect_on_client_disconnect" = "enabled"
@@ -194,6 +200,9 @@ function Main {
if (-not $swap) {
Say "Frametop's build of Vibepollo isn't available here, so this PC can't stream its own monitors yet (virtual displays work)."
}
} elseif ($OlderBuildShas -contains $now) {
$swap = Get-Build
if (-not $swap) { Say "An older Frametop build of Vibepollo is installed, and the new one isn't available here; keeping it." }
} else {
$v = (Get-Item $exe).VersionInfo.ProductVersion
throw "This is Vibepollo $v, and Frametop's build is for $VibepolloVersion. Install Vibepollo $VibepolloVersion ($SetupUrl), then run this again."
@@ -243,7 +252,7 @@ function Main {
Stop-Vibepollo
try {
if ($swap) {
if (-not (Test-Path $orig)) { Copy-Item $exe $orig }
if (-not (Test-Path $orig) -and $now -eq $OriginalSha) { Copy-Item $exe $orig }
Copy-Item $swap $exe -Force
Say "Installed Frametop's build of Vibepollo (the original is $orig)."
}
+6 -1
View File
@@ -1235,7 +1235,12 @@ class Backend(QObject):
@Slot()
def applyBluetoothFixes(self):
if not os.path.exists("/run/host/etc/steamframe/bt-fixups.sh") and not os.path.exists("/etc/steamframe/bt-fixups.sh"):
self.message.emit("Bluetooth fixes aren't installed (setup/bluetooth/install.sh)", True)
# The unit outlives the script: SteamOS updates keep /etc's units, not what they run.
if any(os.path.exists(r + "/etc/systemd/system/steamframe-bt-fixups.service") for r in ("/run/host", "")):
self.message.emit("A SteamOS update deleted the Bluetooth fixes: reinstall them with "
"setup/bluetooth/install.sh install", True)
else:
self.message.emit("Bluetooth fixes aren't installed (setup/bluetooth/install.sh)", True)
return
result = host("pkexec", "/etc/steamframe/bt-fixups.sh")
if result.returncode == 0:
+18 -5
View File
@@ -27,7 +27,7 @@ look; held, the pointer stops there and your head steers it (it stays put in you
the release clicks; held still for half a second, it's a real press that your head drags
("gazekey left|right 1|0" to the helper; by default Meta+J and Meta+K, DEFAULT_KEY_BINDINGS),
gaze_quickcal = the gaze service's one-dot check ("quickcal" to @ft_gazed), sens_up, sens_down,
layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens,
layout_reset = open the profile in use again, or put the desktop screens back in their layout (ft-layout reset), screens_toggle = hide or show the desktop screens,
keyboard_toggle = open or close Frametop's keyboard, float_toggle = float the desktop window under the
pointer (else the active one) in VR, or put it back if it floats, dock_all = put every floating
window back (both to ft-floatd, @frametop_float), spin_next and spin_prev = turn every panel in the
@@ -91,7 +91,7 @@ default: only while no pass-through keyboard is connected; a program's uinput ke
doesn't count), "button" (only the keyboard_toggle action opens it), or "never"
(keyboard_toggle does nothing either). With "vr_keyboard_persist" (the default), it stays
open when the text field loses focus, until its Close key, keyboard_toggle, or a layout reset
(ft-layout apply) closes it.
(ft-layout reset) closes it.
Volume keys, from every device that has them (the headset's own buttons included),
are handled here: wpctl steps the default output. Nothing else may see a volume key,
@@ -658,7 +658,7 @@ class Pointer:
pass # ft-screens not running
elif name == "layout_reset":
# Runs a few seconds and borrows the pointer; ft-layout refuses a second copy.
subprocess.Popen([FT_LAYOUT, "apply"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
subprocess.Popen([FT_LAYOUT, "reset"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, start_new_session=True)
log("layout reset")
elif name in ("sens_up", "sens_down"):
@@ -932,9 +932,22 @@ def main():
if not focused:
if not state["rules"].get("vr_keyboard_persist", True):
vr_keyboard("hide") # ft-screens closes it only if it opened it for a text field
elif mode == "always" or (mode == "no_keyboard" and not any(
n.candidate and n.is_keyboard and n.role == "passthrough" and not n.uinput for n in nodes.values())):
return
keyboards = sorted({n.name for n in nodes.values()
if n.candidate and n.is_keyboard and n.role == "passthrough" and not n.uinput})
if mode == "always" or (mode == "no_keyboard" and not keyboards):
vr_keyboard("show")
why = "asking ft-screens to open Frametop's keyboard"
elif mode == "no_keyboard":
why = (f"not opening Frametop's keyboard: a keyboard is connected ({', '.join(keyboards)}), "
"and the Keyboard setting opens it only without one")
else:
why = f"not opening Frametop's keyboard (Keyboard setting: {mode})"
# For bug reports (scripts/report.sh): once per decision, again after 30 s.
now = time.monotonic()
if why != state.get("text_field_said") or now - state.get("text_field_said_at", 0.0) > 30:
log(f"text field focused: {why}")
state["text_field_said"], state["text_field_said_at"] = why, now
def do_action(action, value, now, source="mouse"):
"""A mapped mouse or controller button, or key combination (pointer mode only, but
+35 -1
View File
@@ -93,9 +93,11 @@ class NoDevice:
relay.Virtual = NoDevice
relay.Volume.key = lambda self, fd, code, value, now: VOLUME.append((code, value))
relay.log = lambda *args: None # the relay's own log
LOG = [] # the relay's own log
relay.log = lambda *args: LOG.append(" ".join(map(str, args)))
bindings = {"now": None} # the rules' key_bindings; None: the relay's defaults
devices = {"roles": {}, "buttons": {}} # the rules' "devices" roles and per-device "buttons"
vr_keyboard = {"mode": None} # the rules' vr_keyboard (Frametop's keyboard); None: the default
MOUSE_ID = "usb:0003:0004:test mouse" # the fake mouse's id (Node.id)
@@ -103,6 +105,8 @@ def read_rules(path=None):
rules = {"devices": {i: {"role": r} for i, r in devices["roles"].items()},
"buttons": {i: dict(b) for i, b in devices["buttons"].items()}, "controller_buttons": {}}
rules["key_bindings"] = dict(relay.DEFAULT_KEY_BINDINGS if bindings["now"] is None else bindings["now"])
if vr_keyboard["mode"]:
rules["vr_keyboard"] = vr_keyboard["mode"]
return rules
@@ -413,6 +417,36 @@ def tests():
check("steam_menu, pause_toggle and commands work without pointer mode",
(relay.needs_pointer("steam_menu"), relay.needs_pointer("pause_toggle"), relay.needs_pointer("command:ls")),
(False, False, False))
# A text field on the desktop got focus (ft-textinput): Frametop's keyboard by the Keyboard
# setting, and a log line saying why, once per decision (scripts/report.sh reads them).
def text_field():
c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
c.sendto(b"textfield 1", f"\0{tag}_relay")
time.sleep(0.1)
return [m for m in typed() if m.startswith("vrkeyboard")]
def said():
got = [m for m in LOG if m.startswith("text field focused")]
LOG.clear()
return got
typed(), said()
check("text field, a keyboard connected (default setting): no keyboard", text_field(), [])
check("...and the log says why", said(), [("text field focused: not opening Frametop's keyboard: a keyboard is "
"connected (test keyboard), and the Keyboard setting opens it only "
"without one")])
check("the same again: not logged twice", (text_field(), said()), ([], []))
vr_keyboard["mode"] = "always"
use(None)
check("text field, setting always: it opens", text_field(), ["vrkeyboard show"])
check("...logged", said(), ["text field focused: asking ft-screens to open Frametop's keyboard"])
vr_keyboard["mode"] = "never"
use(None)
check("text field, setting never: no keyboard, logged",
(text_field(), said()), ([], ["text field focused: not opening Frametop's keyboard (Keyboard setting: never)"]))
vr_keyboard["mode"] = None
use(None)
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
shutil.rmtree(OUT, ignore_errors=True)
os._exit(1 if failures else 0)
+1 -1
View File
@@ -1,4 +1,4 @@
#!/bin/bash
# Reset Screen Layout's own program. Steam lists menu entries by program, so entries that
# shared ft-layout (with different arguments) launched as one.
exec "$(dirname "$(readlink -f "$0")")/ft-layout" apply "$@"
exec "$(dirname "$(readlink -f "$0")")/ft-layout" reset "$@"
+77 -13
View File
@@ -50,8 +50,10 @@ you face (yaw only), like a recenter. It lives in ~/.config/frametop-layout.json
"off": true, disconnected: no stream, no panel
"pos": ..., "face": ..., "roll": 0}]}]} as a screen's
Remote displays are screens 101 and up, in the order they're listed. Without a place of their
own they go in a row above the screens. A profile keeps their places and hidden state by id
(profiles[NAME]["remote"]).
own they go in a row above the screens. A profile keeps the ones connected when it was saved,
like its apps: their places and hidden state by id (profiles[NAME]["remote"]). Opening it, or
arranging while it's the one in use, connects each whose host answers and puts it there; the
others stay as they are.
Custom positions: x right, y up, -z forward from the head, in metres; face = the
direction you look to see the screen's front straight on, in degrees, relative to your
heading; roll = the panel turned about its front, counterclockwise as you see it.
@@ -62,6 +64,9 @@ is top left, then left to right.
Usage (on the Frame host; Frametop Display Settings calls it too):
ft-layout apply [--wait SECONDS] arrange every screen; --wait is for desktop start:
wait for the screens, skip if "auto" is off
ft-layout reset a quick reset (Meta+Shift+R, a screen's reset button, a mapped
button): open the profile in use again, as use NAME does, or
else apply
ft-layout capture save the current arrangement as the custom layout
ft-layout save NAME save it as a named layout too, and use that; with the
desktop's apps and hidden screens, as a profile
@@ -97,6 +102,7 @@ Usage (on the Frame host; Frametop Display Settings calls it too):
ft-layout remote disconnect ID... stop their streams and panels until connected again
ft-layout remote remove ID stops it and unpairs its client (the host stays)
"""
import concurrent.futures
import fcntl
import json
import math
@@ -130,6 +136,7 @@ DEFAULTS = {"auto": True, "mode": "preset",
REMOTE_FIRST = 101 # ft-screens' remote screens (screens/remote.h)
REMOTE_MAX = 16
REMOTE_PIXELS_PER_METRE = 2400 # a new remote display's width in VR (5120 px: 2.1 m), at least 1 m
WEB_UI_PORT = 47990 # Vibepollo's Web UI: a host answers there when it's up
STREAM = os.path.join(REPO, "stream", "build", "ft-stream")
@@ -788,6 +795,50 @@ def place_remote(sock, n, d, t, eye, heading):
sock.ask(f"{'conceal' if d.get('hidden') else 'reveal'} {n}")
def host_answers(host, timeout=1.5):
"""Whether a host's Vibepollo answers, at its address or its Steam Link dongle's (as its
route allows)."""
route = host.get("route", "auto")
addresses = ([] if route == "dongle" else [host.get("address")]) + \
([] if route == "network" else list(host.get("direct", [])))
for address in filter(None, addresses):
try:
with socket.create_connection((address, WEB_UI_PORT), timeout=timeout):
return True
except OSError:
pass
return False
def profile_remotes(layout, name):
"""Remote displays the profile has a place for go there; the others stay where they are."""
remote = layout.get("profiles", {}).get(name, {}).get("remote", {})
for _, _, d in remote_displays(layout):
if d.get("id") in remote:
for k in SPATIAL:
d.pop(k, None)
d.update(json.loads(json.dumps({k: v for k, v in remote[d["id"]].items() if k in SPATIAL})))
def open_remotes(layout, name):
"""Connect the remote displays profile `name` was saved with (in the layout: apply starts
their streams), each whose host answers now. A host that doesn't is skipped, and its
displays stay disconnected."""
remote = layout.get("profiles", {}).get(name, {}).get("remote", {})
wanted = [(h, d) for _, h, d in remote_displays(layout) if d.get("id") in remote and d.get("off")]
hosts = list({id(h): h for h, _ in wanted}.values())
if not hosts:
return
with concurrent.futures.ThreadPoolExecutor(len(hosts)) as pool:
up = dict(zip(map(id, hosts), pool.map(host_answers, hosts)))
for h, d in wanted:
if up[id(h)]:
d.pop("off", None)
log(f"remote display {d['id']}: connecting (profile {name!r})")
else:
log(f"remote display {d['id']}: {h.get('name') or h['address']} isn't answering; skipped")
def start_remotes(sock, layout, only=None):
"""Start the remote displays' streams (or just the ids in `only`; ft-screens leaves one
running with the same settings alone), and stop the streams of ones no longer listed or
@@ -848,10 +899,12 @@ def in_container():
def run_stream(*args, timeout=60):
"""ft-stream (it runs in the dev container, as ft-screens does)."""
"""ft-stream, in the container ft-screens runs in: "dev" for a clone, the release's own for a
release (scripts/in-box). A missing "dev" made distrobox ask whether to create it, and the
question hung until the timeout."""
cmd = [STREAM, *args]
if not in_container():
cmd = [os.path.expanduser("~/.local/bin/distrobox"), "enter", "dev", "--", *cmd]
cmd = [os.path.join(REPO, "scripts", "in-box"), *cmd]
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
return r.returncode, r.stdout, r.stderr
@@ -1103,7 +1156,9 @@ def save_named(layout, name):
if not screens:
raise RuntimeError("nothing to save: no arrangement captured")
layout.setdefault("layouts", {})[name] = [{k: s[k] for k in SPATIAL if k in s} for s in screens]
remote = {d["id"]: {k: d[k] for k in SPATIAL if k in d} for _, _, d in remote_displays(layout) if "pos" in d}
# The remote displays connected now, like the apps open now.
remote = {d["id"]: {k: d[k] for k in SPATIAL if k in d} for _, _, d in remote_displays(layout)
if not d.get("off") and "pos" in d}
profile = layout.setdefault("profiles", {}).setdefault(name, {})
if remote:
profile["remote"] = remote
@@ -1133,13 +1188,7 @@ def use_named(layout, name):
elif "pos" not in screens[i]:
screens[i].update({"pos": list(preset[i]["pos"]), "face": list(preset[i]["face"]),
"roll": preset[i]["roll"]})
# Remote displays the profile has a place for go there; the others stay where they are.
remote = layout.get("profiles", {}).get(name, {}).get("remote", {})
for _, _, d in remote_displays(layout):
if d.get("id") in remote:
for k in SPATIAL:
d.pop(k, None)
d.update(json.loads(json.dumps({k: v for k, v in remote[d["id"]].items() if k in SPATIAL})))
profile_remotes(layout, name)
layout["mode"], layout["active"] = "custom", name
@@ -1194,8 +1243,10 @@ def capture_profile(layout, name):
hidden = [i + 1 for i in range(screen_count(layout)) if screen_entry(layout, i).get("hidden")]
profile = layout.setdefault("profiles", {}).setdefault(name, {})
profile["hidden"] = hidden
remote = profile.get("remote", {}) # the ones connected (save_named)
for _, _, d in remote_displays(layout):
profile.setdefault("remote", {}).setdefault(d["id"], {})["hidden"] = bool(d.get("hidden"))
if d["id"] in remote:
remote[d["id"]]["hidden"] = bool(d.get("hidden"))
reply = ask_float("windows")
if reply and reply.startswith("ok "):
profile["windows"] = json.loads(reply[3:])
@@ -1531,6 +1582,7 @@ def main(argv):
return main([argv[0], "apply"] + argv[2:])
use_named(layout, name)
use_hidden(layout, name)
open_remotes(layout, name)
save_layout(layout)
log(f"starting in profile {name!r}")
wait = float(argv[argv.index("--wait") + 1]) if "--wait" in argv else 60
@@ -1547,6 +1599,10 @@ def main(argv):
else:
log(f"kwin: {last}")
open_apps(name, wait=90) # ft-floatd starts with Plasma
elif cmd == "reset":
layout = load_layout()
name = layout.get("active") if layout.get("mode") == "custom" else None
return main([argv[0], "use", name] if name in layout.get("layouts", {}) else [argv[0], "apply"])
elif cmd == "remote":
remote_command(argv[2:])
elif cmd in ("pin", "unpin") and len(argv) >= 3:
@@ -1577,6 +1633,13 @@ def main(argv):
except RuntimeError as e:
log(f"visibility: {e}")
else:
layout = load_layout()
name = layout.get("active") if layout.get("mode") == "custom" else None
if name in layout.get("profiles", {}):
# Arranged as the profile in use: its remote displays too.
profile_remotes(layout, name)
open_remotes(layout, name)
save_layout(layout)
apply(wait)
if not wait:
kwin_follow()
@@ -1609,6 +1672,7 @@ def main(argv):
layout = load_layout()
use_named(layout, argv[2])
use_hidden(layout, argv[2])
open_remotes(layout, argv[2])
save_layout(layout)
log(f"using layout {argv[2]!r}")
try:
+8
View File
@@ -194,6 +194,14 @@ prerelease when the tag has a `-` (`v0.3.0-exp.1`); someone publishes it. A
manual run keeps the zip as an artifact for a week. The repo needs Depot's
GitHub app for the runner label to work.
The repo moved from DeeJanuz/frametop to the Frametop organization on
2026-10-09, because Depot's runners need an organization. GitHub redirects the
old repo URLs; the old Pages one-liner (`deejanuz.github.io/frametop/get.sh`)
is kept by DeeJanuz/deejanuz.github.io, whose `frametop/get.sh` and
`frametop/uninstall.sh` hand over to `frametop.github.io/frametop`. A release:
merge into experimental (and main for a stable one, by merge commit), push,
tag, and publish the draft.
## The longer arc
The image only pays off when it makes installing Frametop easier, so none of
+2 -1
View File
@@ -24,7 +24,7 @@
set -euo pipefail
here=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
TABLE_URL=${FRAMETOP_STEAMOS_TABLE:-https://raw.githubusercontent.com/DeeJanuz/frametop/main/pack/steamos.json}
TABLE_URL=${FRAMETOP_STEAMOS_TABLE:-https://raw.githubusercontent.com/Frametop/frametop/main/pack/steamos.json}
# check_release RELEASE_JSON TABLE_JSON BUILD_ID: is the release usable, and is it for this
# SteamOS build? Prints seven lines: status (tested, untested, or broken), version, commit,
@@ -219,6 +219,7 @@ main() {
ln -sfn "$version" "$base/current"
for d in "$base"/*/; do
d=${d%/}
[ -L "$d" ] && continue # "current" itself, a link to the release just installed
[ -f "$d/.frametop-release" ] || continue
case ${d##*/} in "$version"|"$prev") continue ;; esac
old_image=$(sed -n 's/^IMAGE=//p' "$d/.frametop-release")
+1 -1
View File
@@ -4,7 +4,7 @@ On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1
- **With [FrameDrop](https://framedropvr.com) on a PC:** drop Frametop.zip on your paired headset. In the headset, open Frametop in your library and press Play.
- **On the headset:** unpack Frametop.zip (in the desktop's Dolphin or Konsole) and run `Frametop/frametop-install.sh`.
- **In a terminal on the headset:** `curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --release`
- **In a terminal on the headset:** `curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --release`
A window asks what to install, and your SteamOS password for the two optional parts that need it (our own eye tracker and the Bluetooth fixes); the password is only used for this install. Nothing is compiled on the headset. Restart SteamVR once afterwards.
+8
View File
@@ -8,6 +8,14 @@
"steamvr": "2.17.10",
"frametop": "0.2.1",
"date": "2026-10-05"
},
{
"build": "20261007.6125817",
"version": "0.4.5",
"branch": "stable",
"steamvr": "2.18.2",
"frametop": "0.2.2",
"date": "2026-10-09"
}
],
"broken": []
Executable
+110
View File
@@ -0,0 +1,110 @@
#!/usr/bin/env bash
# The runtime trial (HEADSET-TESTS.md, "Frametop from the image"): switch this Frame's Frametop
# to a release, installed from its Frametop.zip, and back to what was there.
# pack/trial.sh on DIR # DIR: the unpacked zip's Frametop folder (install-release.sh in it)
# pack/trial.sh desktop # restart the VR desktop from the release (ft-screens in its container)
# pack/trial.sh status # what runs where
# pack/trial.sh off # put it all back
# on saves what the release's install.sh replaces to ~/.local/state/frametop-trial/saved: the
# frametop-* user units with their drop-ins and .wants links, the launcher and menu entries, the
# SteamVR driver's folder and registry, frametop.conf, and the desktop's shortcuts file. It
# moves the drop-ins aside (they would override the release's units), runs the release's
# install-release.sh --yes --no-eye-tracker --no-bluetooth (no sudo: the installed eye grabber
# stays), and restarts SteamVR so it loads the release's driver and services.
# off removes those files and puts the saved ones back, stops the release's container, and
# restarts SteamVR. The release stays in ~/.local/share/frametop/releases for another try.
# Restarting SteamVR closes everything open in VR.
set -euo pipefail
shopt -s nullglob
self=$(readlink -f "$0")
state=$HOME/.local/state/frametop-trial
saved=$state/saved
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
# What install.sh changes (pack/README.md, Releases), relative to the home folder, as globs.
paths() {
cat <<'EOF'
.config/systemd/user/frametop-*.service
.config/systemd/user/frametop-*.service.d
.config/systemd/user/*.wants/frametop-*
.local/share/applications/deckard-nested-desktop.desktop
.local/share/applications/native-deckard-nested-desktop.desktop
.local/share/applications/ft-*.desktop
.local/share/applications/frametop-profile-*.desktop
.local/share/frametop/ft_pointer
.config/openvr/openvrpaths.vrpath
.config/frametop.conf
.config/frametop/kglobalshortcutsrc
EOF
}
restart_steamvr() {
echo "restarting SteamVR"
systemctl --user restart steamvr.service
for _ in $(seq 60); do
systemctl --user -q is-active frametop-pointer.service && break
sleep 1
done
}
case ${1:-status} in
on)
src=$(cd "${2:?usage: pack/trial.sh on DIR (the Frametop folder of the unpacked zip)}" && pwd)
[ -x "$src/install-release.sh" ] || { echo "$src/install-release.sh isn't there" >&2; exit 1; }
[ -e "$saved" ] && { echo "a trial is on already: pack/trial.sh off first" >&2; exit 1; }
mkdir -p "$saved"
cd "$HOME"
while read -r glob; do
for p in $glob; do
[ -e "$p" ] || [ -L "$p" ] || continue # a plain name stays as is when it's missing
cp -a --parents "$p" "$saved/"
done
done < <(paths)
echo "saved $(find "$saved" -mindepth 1 -not -type d | wc -l) files in $saved"
rm -rf .config/systemd/user/frametop-*.service.d
systemctl --user daemon-reload
"$src/install-release.sh" --yes --no-eye-tracker --no-bluetooth
restart_steamvr
"$self" status ;;
desktop)
release=$HOME/.local/share/frametop/releases/current
[ -e "$saved" ] && [ -x "$release/desktops.sh" ] || { echo "no trial on" >&2; exit 1; }
"$release/desktops.sh" restart ;;
off)
[ -d "$saved" ] || { echo "no trial on (nothing saved in $saved)" >&2; exit 1; }
cd "$HOME"
while read -r glob; do
for p in $glob; do rm -rf "$p"; done
done < <(paths)
cp -a "$saved/." "$HOME/"
systemctl --user daemon-reload
restart_steamvr # first: the release's programs run in its container until SteamVR stops them
pkill -x ft-screens || true # the VR desktop from the release, if it's still up
for box in $(podman ps --format '{{.Names}}' | grep -E '^frametop-[0-9a-f]{12}$' || true); do
echo "stopping $box"
podman stop -t 5 "$box" >/dev/null || echo "couldn't stop $box (podman stop $box)" >&2
done
mv "$saved" "$state/restored-$(date +%Y%m%d-%H%M%S)"
echo "back as before; restart the VR desktop from Launch a program to leave the release's"
"$self" status ;;
status)
[ -d "$saved" ] && echo "trial: ON (saved in $saved)" || echo "trial: off"
for u in frametop-input-relay frametop-pointer frametop-power frametop-gaze frametop-desktop; do
printf '%-22s %-9s %s\n' "$u" "$(systemctl --user is-active $u.service)" \
"$(systemctl --user show -P ExecStart $u.service | grep -o 'argv\[\]=[^;]*' | cut -c8- | cut -c1-110)"
done
for name in ft-pointer ft-powerd ft-gaze ft-eyes ft-screens; do
if [ "$name" = ft-eyes ]; then pids=$(pgrep -f "[/]ft-eyes " || true); else pids=$(pgrep -x "$name" || true); fi
for pid in $pids; do
id=$(grep -o 'libpod-[0-9a-f]\{12\}' "/proc/$pid/cgroup" 2>/dev/null | head -1 | cut -c8-) || id=
box=host
[ -n "$id" ] && box=$(podman ps --filter "id=$id" --format '{{.Names}}' 2>/dev/null || echo "$id")
printf '%-11s pid %-7s in %-22s %s\n' "$name" "$pid" "$box" \
"$(tr '\0' ' ' </proc/$pid/cmdline 2>/dev/null | grep -o '/home/[^ ]*' | head -1 || true)"
done
done
readlink -f "$HOME/.local/share/frametop/releases/current" 2>/dev/null | sed 's/^/release: /' || true ;;
*) sed -n '2,6p' "$0" >&2; exit 2 ;;
esac
+32
View File
@@ -0,0 +1,32 @@
// The gaze calibration panel's answers (see ft-pointer.cpp's top): while ft-gazed says the panel
// is up, a press from the relay answers it instead of clicking. Here so it can be tested without
// SteamVR (pointer/test/calpanel-test.sh).
//
// Accept (take this dot): the left button's press ("btn trigger 1", whatever mouse button,
// controller button or key combination is mapped to the left action), Meta+J ("gazekey left 1"),
// and a gaze precision or gaze drag press ("precision|gazedrag <source> 1"): those are the left
// button for someone who mapped it to one, and were dropped until 2026-10-09, so the panel
// never took a dot from them. Quit: the right button ("btn b 1") and Meta+K ("gazekey right 1").
// Their releases, and the other buttons, do nothing while the panel is up.
#pragma once
#include <cstdio>
#include <cstring>
enum class CalPanelAnswer { None, Accept, Quit, Ignore };
inline CalPanelAnswer calPanelAnswer(const char *msg) {
char source[16];
int value;
if (!std::strncmp(msg, "btn trigger 1", 13) || !std::strncmp(msg, "gazekey left 1", 14))
return CalPanelAnswer::Accept;
if ((std::sscanf(msg, "precision %15s %d", source, &value) == 2 ||
std::sscanf(msg, "gazedrag %15s %d", source, &value) == 2) && value == 1)
return CalPanelAnswer::Accept;
if (!std::strncmp(msg, "btn b 1", 7) || !std::strncmp(msg, "gazekey right 1", 15))
return CalPanelAnswer::Quit;
if (!std::strncmp(msg, "btn ", 4) || !std::strncmp(msg, "gazekey ", 8) ||
!std::strncmp(msg, "precision ", 10) || !std::strncmp(msg, "gazedrag ", 9))
return CalPanelAnswer::Ignore;
return CalPanelAnswer::None;
}
+8 -8
View File
@@ -227,8 +227,9 @@
// drags again from there. Without gaze mode they work from wherever the pointer is.
// The gaze calibration panel (gaze/panel/ft-gazepanel, run by the gaze service): while
// ft-gazed says it's up ("calpanel 1", renewed every second; it lapses 3 s after the last),
// the dot hides and a press answers the panel instead of clicking: a left click or gaze_left
// sends "calaccept" to @ft_gazed (take this dot now), a right click or gaze_right "calquit".
// the dot hides and a press answers the panel instead of clicking: a left click, gaze_left, or
// a gaze_precision or gaze_drag press sends "calaccept" to @ft_gazed (take this dot now), a right
// click or gaze_right "calquit" (calpanel.h).
// POINTER_ROLE (right, left, or stylus): the hand role our device takes while connected. A
// Frame controller in your hand counts as used through its touch sensors and takes its hand's
// role back, and then no click lands (see "no hand role" in the main loop): with a controller
@@ -316,6 +317,7 @@
// POINTER_ROLE (right): gaze precision and keyboard clicks, above.
#include <openvr.h>
#include "calpanel.h"
#include "vrbuttons.h"
#include "vrmath.h"
@@ -1485,14 +1487,12 @@ int main() {
}
}
if (Clock::now() < calPanelUntil) {
const bool accept = !std::strncmp(buf, "btn trigger 1", 13) || !std::strncmp(buf, "gazekey left 1", 14);
const bool quit = !std::strncmp(buf, "btn b 1", 7) || !std::strncmp(buf, "gazekey right 1", 15);
if (accept || quit) {
SendTo(out, "ft_gazed", accept ? "calaccept" : "calquit");
const CalPanelAnswer answer = calPanelAnswer(buf);
if (answer == CalPanelAnswer::Accept || answer == CalPanelAnswer::Quit) {
SendTo(out, "ft_gazed", answer == CalPanelAnswer::Accept ? "calaccept" : "calquit");
continue;
}
if (!std::strncmp(buf, "btn ", 4) || !std::strncmp(buf, "gazekey ", 8) ||
!std::strncmp(buf, "precision ", 10) || !std::strncmp(buf, "gazedrag ", 9))
if (answer == CalPanelAnswer::Ignore)
continue; // their releases, and the other buttons: nothing to click now
}
{
+49
View File
@@ -0,0 +1,49 @@
// Offline test of the gaze calibration panel's answers (pointer/helper/calpanel.h): which relay
// messages take the dot, close the panel, or do nothing while it's up. Needs no SteamVR.
//
// pointer/test/calpanel-test.sh
#include "../helper/calpanel.h"
#include <cstdio>
int main() {
struct Case {
const char *msg;
CalPanelAnswer want;
} cases[] = {
{"btn trigger 1", CalPanelAnswer::Accept},
{"gazekey left 1", CalPanelAnswer::Accept},
{"precision mouse 1", CalPanelAnswer::Accept},
{"precision keyboard 1", CalPanelAnswer::Accept},
{"gazedrag mouse 1", CalPanelAnswer::Accept},
{"gazedrag keyboard 1", CalPanelAnswer::Accept},
{"btn b 1", CalPanelAnswer::Quit},
{"gazekey right 1", CalPanelAnswer::Quit},
{"btn trigger 0", CalPanelAnswer::Ignore},
{"btn b 0", CalPanelAnswer::Ignore},
{"btn a 1", CalPanelAnswer::Ignore}, // the relay's laser claim
{"btn x 1", CalPanelAnswer::Ignore},
{"btn system 1", CalPanelAnswer::Ignore},
{"gazekey left 0", CalPanelAnswer::Ignore},
{"gazekey right 0", CalPanelAnswer::Ignore},
{"precision mouse 0", CalPanelAnswer::Ignore},
{"gazedrag mouse 0", CalPanelAnswer::Ignore},
{"move 0.1000 -0.2000", CalPanelAnswer::None},
{"show", CalPanelAnswer::None},
{"recenter", CalPanelAnswer::None},
{"scroll 0 1", CalPanelAnswer::None},
{"typing", CalPanelAnswer::None},
};
const char *names[] = {"none", "accept", "quit", "ignore"};
int failed = 0;
for (const Case &c : cases) {
const CalPanelAnswer got = calPanelAnswer(c.msg);
if (got != c.want) {
std::printf("FAIL \"%s\": %s, want %s\n", c.msg, names[int(got)], names[int(c.want)]);
++failed;
}
}
std::printf("%s: %d of %zu cases\n", failed ? "FAILED" : "ok", int(sizeof cases / sizeof cases[0]) - failed,
sizeof cases / sizeof cases[0]);
return failed ? 1 : 0;
}
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# Offline test of the gaze calibration panel's answers (pointer/test/calpanel-test.cpp): builds
# it in the dev container and runs it there. Nothing reaches SteamVR, so it's safe next to it.
#
# pointer/test/calpanel-test.sh
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C pointer 'set -e; mkdir -p test/build
g++ -std=c++17 -O2 -Wall -o test/build/calpanel-test test/calpanel-test.cpp
test/build/calpanel-test'
+16 -2
View File
@@ -686,9 +686,15 @@ static void keyboard_command(struct server *s, const char *what, char *reply, in
return (void)snprintf(reply, size, "ok closed");
}
if (open) return (void)snprintf(reply, size, "ok open");
if (!ft_vr_screens_shown()) return (void)snprintf(reply, size, "ok screens hidden");
if (!ft_vr_screens_shown()) {
wlr_log(WLR_INFO, "keyboard not opened (%s): the screens are hidden", toggle ? "button" : "text field");
return (void)snprintf(reply, size, "ok screens hidden");
}
const int screen = focused_screen(s);
if (screen < 0 || !ft_vr_keyboard_show(screen)) return (void)snprintf(reply, size, "error not shown");
if (screen < 0 || !ft_vr_keyboard_show(screen)) {
wlr_log(WLR_INFO, "keyboard not opened (%s) for screen %d", toggle ? "button" : "text field", screen + 1);
return (void)snprintf(reply, size, "error not shown");
}
wlr_log(WLR_INFO, "keyboard open for screen %d (%s)", screen + 1, toggle ? "button" : "text field");
s->kb_screen = screen;
s->kb_auto = !toggle;
@@ -831,6 +837,14 @@ static int control_readable(int fd, uint32_t mask, void *data) {
if (strcmp(buf + 6, "-") != 0 && strncmp(buf + 6, "frametop.", 9) != 0) s->keys_clicked = false;
len = sizeof from;
continue;
} else if (strcmp(buf, "debug") == 0) {
// ft-screens' side of scripts/report.sh's debug line; vr.cpp adds SteamVR's.
char vr[1024] = "ok vr=off";
if (s->vr) ft_vr_command("debug", vr, sizeof vr);
snprintf(reply, sizeof reply, "%s kb_screen=%d kb_auto=%d typing=%s focused_screen=%d pointer_screen=%d",
vr, s->kb_screen + 1, s->kb_auto ? 1 : 0,
s->key_remote >= 0 ? "remote" : s->keys_desktop ? "desktop" : s->keys_clicked ? "desktop (hidden)" : "steam",
focused_screen(s) + 1, s->pointer_focus ? s->pointer_focus->index + 1 : 0);
} else if (!ft_remote_command(buf, reply, sizeof reply)) {
ft_vr_command(buf, reply, sizeof reply);
}
+212 -27
View File
@@ -88,7 +88,7 @@ bool Hands::Read() {
const Mat head = HeadAt(captureNs_);
// each hand's palm in the room, and its velocity from the last time it was seen
ids_.clear();
ids_.clear(), basePts_.clear();
std::vector<int> owners; // the hand each capsule belongs to, in file order
for (uint32_t k = 0; k < nhands; ++k) {
const fh_hand_t &h = copy.hands[k];
@@ -97,6 +97,13 @@ bool Hands::Read() {
const int idx = int(ids_.size());
ids_.push_back(id);
owners.insert(owners.end(), std::min<uint32_t>(h.ncapsules, kMaxCapsules), idx);
HandPoints hp{id, (h.flags & FH_HAND_RIGHT) != 0, {}};
bool finite = true;
for (int j = 0; j < 21; ++j) {
for (float v : pts[j]) finite = finite && std::isfinite(v) && std::fabs(v) < 10;
Apply(head, pts[j], hp.p[j]);
}
if (finite) basePts_.push_back(hp);
double palm[3] = {0, 0, 0};
bool ok = true;
for (int j : {0, 5, 9, 13, 17}) {
@@ -162,7 +169,7 @@ bool Hands::Update(const Mat &head, int64_t nowNs) {
history_.push_back({nowNs, head});
while (!history_.empty() && nowNs - history_.front().ns > kHistoryNs) history_.erase(history_.begin());
Read();
if (nowNs - publishNs_ > kStaleNs) base_.clear(), owner_.clear();
if (nowNs - publishNs_ > kStaleNs) base_.clear(), owner_.clear(), basePts_.clear();
// move each hand ahead to when this frame will be on the displays; a slow hand's
// velocity is mostly tracking noise, so it fades out below kStillSpeed
const double ahead = std::clamp((nowNs + leadNs_ - captureNs_) / 1e9, 0.0, kMaxAhead);
@@ -182,6 +189,21 @@ bool Hands::Update(const Mat &head, int64_t nowNs) {
return !world_.empty();
}
void Hands::Points(int64_t nowNs, bool predict, double leadMs, std::vector<HandPoints> &out) const {
out = basePts_;
if (!predict) return;
const double ahead = std::clamp((nowNs + leadMs * 1e6 - captureNs_) / 1e9, 0.0, kMaxAhead);
for (HandPoints &hp : out) {
const auto m = motion_.find(hp.id);
if (m == motion_.end()) continue;
const double *v = m->second.v;
const double speed = std::sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2]);
const double gain = std::clamp((speed - kStillSpeed) / kStillSpeed, 0.0, 1.0); // as Update
for (auto &p : hp.p)
for (int i = 0; i < 3; ++i) p[i] += float(v[i] * gain * ahead);
}
}
void EyePositions(const Mat &head, double out[2][3]) {
const vr::EVREye eyes[2] = {vr::Eye_Left, vr::Eye_Right};
for (int e = 0; e < 2; ++e) {
@@ -279,6 +301,9 @@ PFNEGLCREATEIMAGEKHRPROC pCreateImage;
PFNEGLDESTROYIMAGEKHRPROC pDestroyImage;
PFNGLEGLIMAGETARGETTEXTURE2DOESPROC pImageTargetTexture;
PFNGLEGLIMAGETARGETRENDERBUFFERSTORAGEOESPROC pImageTargetRenderbuffer;
PFNEGLCREATESYNCKHRPROC pCreateSync;
PFNEGLDESTROYSYNCKHRPROC pDestroySync;
PFNEGLCLIENTWAITSYNCKHRPROC pClientWaitSync;
const char *kVertex = R"(
attribute vec2 pos; // the unit square
@@ -314,6 +339,20 @@ void main() {
gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0 - smoothstep(rad - feather, rad + feather, d));
})";
// A probe dot: the colour inside, a dark ring around it so it reads on any background.
const char *kMark = R"(
precision highp float;
uniform vec2 c;
uniform float r;
uniform vec3 color;
varying vec2 px;
void main() {
float d = length(px - c);
float a = 1.0 - smoothstep(r - 0.75, r + 0.75, d);
if (a <= 0.0) discard;
gl_FragColor = vec4(d > r - 2.0 ? vec3(0.0) : color, a);
})";
unsigned Shader(GLenum type, const char *src) {
const GLuint s = glCreateShader(type);
glShaderSource(s, 1, &src, nullptr);
@@ -400,7 +439,11 @@ bool Renderer::Init(const std::vector<uint64_t> &modifiers, std::function<void(c
pImageTargetTexture = reinterpret_cast<PFNGLEGLIMAGETARGETTEXTURE2DOESPROC>(eglGetProcAddress("glEGLImageTargetTexture2DOES"));
pImageTargetRenderbuffer = reinterpret_cast<PFNGLEGLIMAGETARGETRENDERBUFFERSTORAGEOESPROC>(
eglGetProcAddress("glEGLImageTargetRenderbufferStorageOES"));
if (!gbm_ || !pGetPlatformDisplay || !pCreateImage || !pImageTargetTexture || !pImageTargetRenderbuffer) {
pCreateSync = reinterpret_cast<PFNEGLCREATESYNCKHRPROC>(eglGetProcAddress("eglCreateSyncKHR"));
pDestroySync = reinterpret_cast<PFNEGLDESTROYSYNCKHRPROC>(eglGetProcAddress("eglDestroySyncKHR"));
pClientWaitSync = reinterpret_cast<PFNEGLCLIENTWAITSYNCKHRPROC>(eglGetProcAddress("eglClientWaitSyncKHR"));
if (!gbm_ || !pGetPlatformDisplay || !pCreateImage || !pImageTargetTexture || !pImageTargetRenderbuffer ||
!pCreateSync || !pDestroySync || !pClientWaitSync) {
std::fprintf(stderr, "handcut: GBM or EGL extensions missing\n");
return false;
}
@@ -415,7 +458,8 @@ bool Renderer::Init(const std::vector<uint64_t> &modifiers, std::function<void(c
ctx_ = ctx;
copyProg_ = Program(kCopy);
cutProg_ = Program(kCut);
if (!copyProg_ || !cutProg_) return false;
markProg_ = Program(kMark);
if (!copyProg_ || !cutProg_ || !markProg_) return false;
const float quad[] = {0, 0, 1, 0, 0, 1, 1, 1};
glGenBuffers(1, &vbo_);
glBindBuffer(GL_ARRAY_BUFFER, vbo_);
@@ -451,6 +495,9 @@ void Renderer::Forget(const void *key) {
glDeleteTextures(1, &it->second.tex);
pDestroyImage(EGLDisplay(dpy_), EGLImageKHR(it->second.image));
imported_.erase(it);
for (auto &[k, r] : rings_) // a new buffer at the same address isn't this one
for (Output &o : r.out)
if (o.key == key) o.drawn = false;
}
bool Renderer::MakeOutput(Output &o, int w, int h) {
@@ -489,6 +536,7 @@ bool Renderer::MakeOutput(Output &o, int w, int h) {
void Renderer::FreeOutput(Output &o) {
if (o.bo && released_) released_(&o);
if (o.fence) pDestroySync(EGLDisplay(dpy_), EGLSyncKHR(o.fence));
if (o.fbo) glDeleteFramebuffers(1, &o.fbo);
if (o.rb) glDeleteRenderbuffers(1, &o.rb);
if (o.image) pDestroyImage(EGLDisplay(dpy_), EGLImageKHR(o.image));
@@ -505,26 +553,68 @@ void Renderer::DropPanel(int panel) {
rings_.erase(it);
}
const Output *Renderer::Composite(int panel, const void *key, const ft_dmabuf &src, const std::vector<Capsule2D> eyes[2]) {
if (!ready_) return nullptr;
const auto t0 = std::chrono::steady_clock::now();
const int w = src.width, h = src.height;
Ring &ring = rings_[panel];
if (ring.w != w || ring.h != h) {
for (Output &old : ring.out) FreeOutput(old);
ring.w = w, ring.h = h, ring.next = 0;
}
Output &o = ring.out[ring.next];
if (!o.bo && !MakeOutput(o, 2 * w, h)) return nullptr;
const GLuint tex = Texture(key, src);
if (!tex) return nullptr;
ring.next = (ring.next + 1) % 3;
namespace {
// The pixels a cutout's quad covers (see Draw), as x0 y0 x1 y1 in the eye's half.
void Bounds(const Capsule2D &c, float b[4]) {
const float feather = std::max(1.5f, 0.15f * std::min(c.ra, c.rb));
const float r = std::max(c.ra, c.rb) + feather;
b[0] = std::min(c.ax, c.bx) - r, b[1] = std::min(c.ay, c.by) - r;
b[2] = std::max(c.ax, c.bx) + r, b[3] = std::max(c.ay, c.by) + r;
}
// Within a quarter pixel: the same picture.
bool SameSpots(const std::vector<Capsule2D> a[2], const std::vector<Capsule2D> b[2]) {
for (int e = 0; e < 2; ++e) {
if (a[e].size() != b[e].size()) return false;
for (size_t i = 0; i < a[e].size(); ++i) {
const Capsule2D &p = a[e][i], &q = b[e][i];
for (float d : {p.ax - q.ax, p.ay - q.ay, p.bx - q.bx, p.by - q.by, p.ra - q.ra, p.rb - q.rb})
if (std::fabs(d) > 0.25f) return false;
}
}
return true;
}
int64_t SteadyNs() {
return std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();
}
} // namespace
bool Renderer::Passed(Output &o, int64_t timeoutNs) {
if (!o.fence) return true;
const EGLint r = pClientWaitSync(EGLDisplay(dpy_), EGLSyncKHR(o.fence), 0, EGLTimeKHR(timeoutNs));
if (r == EGL_TIMEOUT_EXPIRED_KHR) return false;
pDestroySync(EGLDisplay(dpy_), EGLSyncKHR(o.fence)); // passed, or failed: don't wait on it again
o.fence = nullptr;
return true;
}
// Draws one buffer. Partial: the buffer holds this client frame already, with o.spots cut
// out, so each eye is drawn again only inside the box around those and the new cutouts.
void Renderer::Draw(Output &o, unsigned tex, int w, int h, const std::vector<Capsule2D> eyes[2], bool partial) {
glBindFramebuffer(GL_FRAMEBUFFER, o.fbo);
glBindBuffer(GL_ARRAY_BUFFER, vbo_);
glEnableVertexAttribArray(0);
glVertexAttribPointer(0, 2, GL_FLOAT, GL_FALSE, 0, nullptr);
for (int e = 0; e < 2; ++e) {
if (partial) {
float box[4] = {1e9f, 1e9f, -1e9f, -1e9f}, b[4];
const std::vector<Capsule2D> *lists[2] = {&o.spots[e], &eyes[e]};
for (const std::vector<Capsule2D> *list : lists)
for (const Capsule2D &c : *list) {
Bounds(c, b);
box[0] = std::min(box[0], b[0]), box[1] = std::min(box[1], b[1]);
box[2] = std::max(box[2], b[2]), box[3] = std::max(box[3], b[3]);
}
// Window y is the buffer's row, the same way down as the cutouts' y (see kVertex).
const int x0 = std::clamp(int(std::floor(box[0])) - 1, 0, w), y0 = std::clamp(int(std::floor(box[1])) - 1, 0, h);
const int x1 = std::clamp(int(std::ceil(box[2])) + 1, 0, w), y1 = std::clamp(int(std::ceil(box[3])) + 1, 0, h);
if (x1 <= x0 || y1 <= y0) continue; // no cutout in this eye, then or now
glEnable(GL_SCISSOR_TEST);
glScissor(e * w + x0, y0, x1 - x0, y1 - y0);
}
glViewport(e * w, 0, w, h);
glDisable(GL_BLEND);
glUseProgram(copyProg_);
@@ -543,22 +633,117 @@ const Output *Renderer::Composite(int panel, const void *key, const ft_dmabuf &s
uB = glGetUniformLocation(cutProg_, "b"), uR = glGetUniformLocation(cutProg_, "r"),
uF = glGetUniformLocation(cutProg_, "feather");
for (const Capsule2D &c : eyes[e]) {
const float feather = std::max(1.5f, 0.15f * std::min(c.ra, c.rb));
const float r = std::max(c.ra, c.rb) + feather;
glUniform4f(uRect, std::min(c.ax, c.bx) - r, std::min(c.ay, c.by) - r, std::max(c.ax, c.bx) + r,
std::max(c.ay, c.by) + r);
float b[4];
Bounds(c, b);
glUniform4f(uRect, b[0], b[1], b[2], b[3]);
glUniform2f(uA, c.ax, c.ay);
glUniform2f(uB, c.bx, c.by);
glUniform2f(uR, c.ra, c.rb);
glUniform1f(uF, feather);
glUniform1f(uF, std::max(1.5f, 0.15f * std::min(c.ra, c.rb)));
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
}
glDisable(GL_SCISSOR_TEST);
}
glDisable(GL_BLEND);
// SteamVR reads the buffer from another process and GPU queue; make sure it's done.
glFinish();
lastMs_ = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count();
return &o;
}
const Output *Renderer::Composite(int panel, const void *key, uint64_t serial, const ft_dmabuf &src,
const std::vector<Capsule2D> eyes[2]) {
if (!ready_) return nullptr;
const int64_t t0 = SteadyNs();
const int w = src.width, h = src.height;
Ring &ring = rings_[panel];
if (ring.w != w || ring.h != h) {
for (Output &old : ring.out) FreeOutput(old);
ring.w = w, ring.h = h, ring.shown = ring.before = ring.drawing = -1;
}
// After a pause the panel showed its client buffer, so nothing of ours is on it.
if (t0 - ring.lastCall > 30'000'000) ring.shown = ring.before = -1;
ring.lastCall = t0;
auto promote = [&ring] {
ring.before = ring.shown, ring.shown = ring.drawing, ring.drawing = -1;
};
if (ring.drawing >= 0 && Passed(ring.out[ring.drawing], 0)) promote();
const int newest = ring.drawing >= 0 ? ring.drawing : ring.shown;
const Output *n = newest >= 0 ? &ring.out[newest] : nullptr;
if (n && n->key == key && n->serial == serial && SameSpots(n->spots, eyes)) {
++stats_.same;
} else if (ring.drawing >= 0) {
++stats_.busy; // drawn on a later tick, from what's current then
} else {
int i = 0;
while (i == ring.shown || i == ring.before) ++i;
Output &o = ring.out[i];
if (!o.bo && !MakeOutput(o, 2 * w, h)) return nullptr;
const GLuint tex = Texture(key, src);
if (!tex) return nullptr;
const bool partial = o.drawn && o.key == key && o.serial == serial;
Draw(o, tex, w, h, eyes, partial);
o.fence = pCreateSync(EGLDisplay(dpy_), EGL_SYNC_FENCE_KHR, nullptr);
glFlush();
if (!o.fence) glFinish(); // no fence: wait here, as before
o.key = key, o.serial = serial, o.drawn = true;
for (int e = 0; e < 2; ++e) o.spots[e] = eyes[e];
ring.drawing = i;
++stats_.draws, stats_.partial += partial;
}
// Nothing of ours to show yet: wait for this one rather than show none.
if (ring.shown < 0 && ring.drawing >= 0) {
++stats_.waits;
if (Passed(ring.out[ring.drawing], 50'000'000)) promote();
}
lastMs_ = (SteadyNs() - t0) / 1e6;
stats_.cpuMs += lastMs_, stats_.worstMs = std::max(stats_.worstMs, lastMs_);
return ring.shown >= 0 ? &ring.out[ring.shown] : nullptr;
}
const Output *Renderer::Marks(int panel, int w, int h, const std::vector<Mark> eyes[2]) {
if (!ready_) return nullptr;
Ring &ring = rings_[panel];
if (ring.w != w || ring.h != h) {
for (Output &old : ring.out) FreeOutput(old);
ring.w = w, ring.h = h, ring.shown = ring.before = ring.drawing = -1;
}
auto promote = [&ring] {
ring.before = ring.shown, ring.shown = ring.drawing, ring.drawing = -1;
};
if (ring.drawing >= 0 && Passed(ring.out[ring.drawing], 0)) promote();
if (ring.drawing < 0) {
int i = 0;
while (i == ring.shown || i == ring.before) ++i;
Output &o = ring.out[i];
if (!o.bo && !MakeOutput(o, 2 * w, h)) return nullptr;
glBindFramebuffer(GL_FRAMEBUFFER, o.fbo);
glViewport(0, 0, 2 * w, h);
glClearColor(0, 0, 0, 0);
glClear(GL_COLOR_BUFFER_BIT);
glBindBuffer(GL_ARRAY_BUFFER, vbo_);
glEnableVertexAttribArray(0);
glVertexAttribPointer(0, 2, GL_FLOAT, GL_FALSE, 0, nullptr);
glDisable(GL_BLEND);
glUseProgram(markProg_);
glUniform2f(glGetUniformLocation(markProg_, "size"), float(w), float(h));
const GLint uRect = glGetUniformLocation(markProg_, "rect"), uC = glGetUniformLocation(markProg_, "c"),
uR = glGetUniformLocation(markProg_, "r"), uColor = glGetUniformLocation(markProg_, "color");
for (int e = 0; e < 2; ++e) {
glViewport(e * w, 0, w, h);
for (const Mark &m : eyes[e]) {
glUniform4f(uRect, m.x - m.r - 1, m.y - m.r - 1, m.x + m.r + 1, m.y + m.r + 1);
glUniform2f(uC, m.x, m.y);
glUniform1f(uR, m.r);
glUniform3f(uColor, m.rgb[0], m.rgb[1], m.rgb[2]);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
}
}
o.fence = pCreateSync(EGLDisplay(dpy_), EGL_SYNC_FENCE_KHR, nullptr);
glFlush();
if (!o.fence) glFinish();
o.key = nullptr, o.drawn = false; // holds no client frame
ring.drawing = i;
}
if (ring.shown < 0 && ring.drawing >= 0 && Passed(ring.out[ring.drawing], 50'000'000)) promote();
return ring.shown >= 0 ? &ring.out[ring.shown] : nullptr;
}
} // namespace handcut
+64 -7
View File
@@ -58,6 +58,17 @@ public:
bool predicting() const { return predict_; }
double leadMs() const { return leadNs_ / 1e6; }
// Each hand's 21 landmarks in the room, for ft-handtest --probe: as the cameras saw them
// (predict false), or moved ahead along the hand's velocity to now + leadMs, as the
// capsules are. Empty while no fresh hands are known (as capsules()).
struct HandPoints {
uint32_t id;
bool right;
float p[21][3];
};
void Points(int64_t nowNs, bool predict, double leadMs, std::vector<HandPoints> &out) const;
int64_t captureNs() const { return captureNs_; }
private:
bool Read();
Mat HeadAt(int64_t ns) const;
@@ -67,10 +78,11 @@ private:
std::vector<Capsule> base_; // the capsules at capture time, in the room
std::vector<int> owner_; // each capsule's hand (index into ids_), or -1
std::vector<uint32_t> ids_; // the hands in the file
std::vector<HandPoints> basePts_; // their landmarks at capture time, in the room
std::map<uint32_t, Motion> motion_;
std::vector<Capsule> world_; // base_, moved ahead
bool predict_ = true;
int64_t leadNs_ = 25'000'000;
int64_t leadNs_ = 36'000'000; // 25 ms to the displays, plus the tick a cutout buffer waits for its fence
int fd_ = -1;
const void *map_ = nullptr;
uint64_t seq_ = 0;
@@ -91,6 +103,28 @@ struct Output {
void *bo = nullptr;
unsigned fbo = 0, rb = 0;
void *image = nullptr;
// What's drawn in it: the client buffer and its frame, and the cutouts (a later draw
// with the same frame only redraws around the old and new cutouts).
void *fence = nullptr; // the GPU is still drawing it
const void *key = nullptr;
uint64_t serial = 0;
bool drawn = false;
std::vector<Capsule2D> spots[2];
};
// A dot for ft-handtest --probe: centre and radius in pixels of the eye's half, colour.
struct Mark {
float x, y, r;
float rgb[3];
};
// Composite's counts since the last TakeStats.
struct CutStats {
int draws = 0, partial = 0; // buffers drawn, of them only around the cutouts
int same = 0; // nothing changed: the newest buffer stays
int busy = 0; // the GPU hadn't finished the last one: drawn next tick
int waits = 0; // a panel's first buffer, waited for
double cpuMs = 0, worstMs = 0;
};
class Renderer {
@@ -99,31 +133,54 @@ public:
// modifiers: what SteamVR takes for DRM_FORMAT_ABGR8888, the outputs' format.
// released: an output is about to be freed (drop its SteamVR import).
bool Init(const std::vector<uint64_t> &modifiers, std::function<void(const Output *)> released);
// Draw client buffer `src` (identified by `key`) into the next output buffer of
// panel `panel`, both eyes, cutting out `eyes`. Returns that buffer, or null.
const Output *Composite(int panel, const void *key, const ft_dmabuf &src, const std::vector<Capsule2D> eyes[2]);
// Draw client buffer `src` (identified by `key`; `serial` counts its frames) into a
// buffer of panel `panel`, both eyes, cutting out `eyes`. Returns the newest buffer the
// GPU has finished, or null.
//
// It doesn't wait for the GPU: a buffer is drawn, fenced, and returned from a later call
// once the fence has passed, so what SteamVR shows is a tick behind. Each panel has three
// buffers: the one shown, the one shown before it (SteamVR may still be reading it), and
// the one being drawn. Nothing is drawn when the frame and the cutouts are what the newest
// buffer has, and when only the cutouts moved, a buffer that holds the same client frame
// is drawn again only around them. The first call after a pause (no call for 30 ms, about
// 3 ticks: the panel showed its client buffer meanwhile) waits for its buffer, so a stale
// one never shows.
const Output *Composite(int panel, const void *key, uint64_t serial, const ft_dmabuf &src,
const std::vector<Capsule2D> eyes[2]);
// ft-handtest --probe: a transparent w x h buffer per eye (side by side) with only the
// dots in it, drawn every call, fenced as Composite's. Returns the newest finished one.
const Output *Marks(int panel, int w, int h, const std::vector<Mark> eyes[2]);
// A client buffer is going away.
void Forget(const void *key);
// A panel is gone: drop its outputs.
void DropPanel(int panel);
// How long the last Composite took, ms (it waits for the GPU).
// How long the last Composite took on the CPU, ms.
double lastMs() const { return lastMs_; }
CutStats TakeStats() { CutStats s = stats_; stats_ = {}; return s; }
private:
unsigned Texture(const void *key, const ft_dmabuf &src);
bool MakeOutput(Output &o, int w, int h);
void FreeOutput(Output &o);
bool Passed(Output &o, int64_t timeoutNs);
void Draw(Output &o, unsigned tex, int w, int h, const std::vector<Capsule2D> eyes[2], bool partial);
bool ready_ = false;
int drm_ = -1;
void *gbm_ = nullptr, *dpy_ = nullptr, *ctx_ = nullptr;
unsigned copyProg_ = 0, cutProg_ = 0, vbo_ = 0;
unsigned copyProg_ = 0, cutProg_ = 0, markProg_ = 0, vbo_ = 0;
std::vector<uint64_t> modifiers_;
std::function<void(const Output *)> released_;
struct Imported { void *image; unsigned tex; };
std::map<const void *, Imported> imported_;
struct Ring { Output out[3]; int next = 0; int w = 0, h = 0; };
struct Ring {
Output out[3];
int shown = -1, before = -1, drawing = -1; // indices into out
int w = 0, h = 0;
int64_t lastCall = 0; // steady clock ns
};
std::map<int, Ring> rings_;
double lastMs_ = 0;
CutStats stats_;
};
} // namespace handcut
+150 -10
View File
@@ -20,6 +20,7 @@
#include <cstdlib>
#include <cstring>
#include <map>
#include <string>
#include <thread>
namespace {
@@ -77,14 +78,142 @@ bool TestPattern(int drm, int w, int h, gbm_bo **out, ft_dmabuf *b) {
return true;
}
// --probe: where the cutouts would land, against the hand Room View shows. The panel is
// see-through, with dots on the wrist, middle knuckle and fingertips of each tracked hand,
// one colour per timing: magenta where the cameras saw the hand, cyan moved ahead to now,
// green moved ahead to now + lead (what the cutouts use). White dots mark the panel's
// corners. Record the headset view meanwhile and compare (frame-hands/probes/
// probe_video.py), or look: which colour sits on your fingertips, still and moving?
// Each tick goes to the log as a JSON line, after a header line with the panel and eyes.
struct Variant {
const char *name;
bool predict;
double leadMs;
float rgb[3];
};
const int kProbePoints[] = {0, 9, 4, 8, 12, 16, 20};
void Matrix(FILE *f, const handcut::Mat &m) {
std::fprintf(f, "[");
for (int r = 0; r < 3; ++r)
for (int c = 0; c < 4; ++c) std::fprintf(f, "%s%.6f", r || c ? "," : "", m.m[r][c]);
std::fprintf(f, "]");
}
void Point(FILE *f, const float p[3]) { std::fprintf(f, "[%.5f,%.5f,%.5f]", p[0], p[1], p[2]); }
int RunProbe(vr::VROverlayHandle_t ov, const handcut::Panel &panel, handcut::Renderer &renderer,
std::map<const void *, vr::SharedTextureHandle_t> &imports, double seconds, double leadMs, double dotMm,
const char *logPath) {
handcut::Hands hands;
if (leadMs < 0) leadMs = hands.leadMs();
const Variant variants[] = {{"seen", false, 0, {1, 0, 1}}, {"now", true, 0, {0, 1, 1}}, {"lead", true, leadMs, {0, 1, 0}}};
FILE *log = std::fopen(logPath, "w");
if (!log) return std::perror(logPath), 1;
std::fprintf(log, "{\"probe\":1,\"panel\":{\"pose\":");
Matrix(log, panel.pose);
std::fprintf(log, ",\"width\":%.4f,\"height\":%.4f,\"px\":[%d,%d]},\"eyes\":[", panel.width, panel.height,
panel.pxWidth, panel.pxHeight);
for (vr::EVREye e : {vr::Eye_Left, vr::Eye_Right}) {
Matrix(log, vr::VRSystem()->GetEyeToHeadTransform(e));
std::fprintf(log, e == vr::Eye_Left ? "," : "],");
}
std::fprintf(log, "\"points\":[0,9,4,8,12,16,20],\"dot_mm\":%.2f,\"variants\":[", dotMm);
for (size_t k = 0; k < 3; ++k)
std::fprintf(log, "%s{\"name\":\"%s\",\"predict\":%s,\"lead_ms\":%.1f,\"rgb\":[%.0f,%.0f,%.0f]}", k ? "," : "",
variants[k].name, variants[k].predict ? "true" : "false", variants[k].leadMs,
variants[k].rgb[0] * 255, variants[k].rgb[1] * 255, variants[k].rgb[2] * 255);
std::fprintf(log, "]}\n");
vr::VROverlay()->SetOverlayFlag(ov, vr::VROverlayFlags_IgnoreTextureAlpha, false);
vr::VROverlay()->SetOverlayFlag(ov, vr::VROverlayFlags_SideBySide_Parallel, true);
vr::SharedTextureHandle_t shown = 0;
vr::Texture_t tex = {&shown, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
std::printf("probe: magenta = as seen, cyan = now, green = now + %.0f ms; log %s\n", leadMs, logPath);
vr::TrackedDevicePose_t poses[vr::k_unMaxTrackedDeviceCount];
const int64_t start = MonoNs();
int64_t lastReport = start;
int ticks = 0, withHands = 0;
std::vector<handcut::Hands::HandPoints> pts;
while (!g_stop && (seconds <= 0 || (MonoNs() - start) / 1e9 < seconds)) {
const auto tick = std::chrono::steady_clock::now();
vr::VRSystem()->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, poses, vr::k_unMaxTrackedDeviceCount);
const auto &head = poses[vr::k_unTrackedDeviceIndex_Hmd].mDeviceToAbsoluteTracking;
const int64_t now = MonoNs();
hands.Update(head, now);
double eyes[2][3];
handcut::EyePositions(head, eyes);
std::vector<handcut::Mark> marks[2];
const float W = float(panel.pxWidth), H = float(panel.pxHeight);
for (int e = 0; e < 2; ++e)
for (float x : {14.f, W - 14}) for (float y : {14.f, H - 14}) marks[e].push_back({x, y, 6, {1, 1, 1}});
std::fprintf(log, "{\"t\":%lld,\"cap\":%lld,\"head\":", (long long)now, (long long)hands.captureNs());
Matrix(log, head);
std::fprintf(log, ",\"hands\":[");
bool any = false;
for (size_t v = 0; v < 3; ++v) {
hands.Points(now, variants[v].predict, variants[v].leadMs, pts);
for (size_t h = 0; h < pts.size(); ++h) {
std::fprintf(log, "%s{\"v\":%zu,\"id\":%u,\"right\":%d,\"p\":[", any ? "," : "", v, pts[h].id, pts[h].right);
any = true;
for (size_t j = 0; j < sizeof kProbePoints / sizeof *kProbePoints; ++j) {
const float *p = pts[h].p[kProbePoints[j]];
if (j) std::fputc(',', log);
Point(log, p);
const handcut::Capsule c{{p[0], p[1], p[2]}, {p[0], p[1], p[2]}, float(dotMm / 1000), float(dotMm / 1000)};
std::vector<handcut::Capsule2D> on[2];
if (!handcut::Project(panel, {c}, eyes, on)) continue;
for (int e = 0; e < 2; ++e)
for (const auto &d : on[e])
marks[e].push_back({d.ax, d.ay, std::max(4.f, d.ra), {variants[v].rgb[0], variants[v].rgb[1], variants[v].rgb[2]}});
}
std::fprintf(log, "]}");
}
}
std::fprintf(log, "]}\n");
withHands += any;
const handcut::Output *out = renderer.Marks(0, panel.pxWidth, panel.pxHeight, marks);
if (out) {
auto it = imports.find(out);
if (it == imports.end()) it = imports.emplace(out, Import(out->buf)).first;
if (it->second && shown != it->second) {
const bool first = !shown;
shown = it->second;
vr::VROverlay()->SetOverlayTexture(ov, &tex);
if (first) vr::VROverlay()->ShowOverlay(ov);
}
}
++ticks;
if (now - lastReport > 2'000'000'000) {
std::printf("%.0f s: %d ticks, %d with hands\n", (now - start) / 1e9, ticks, withHands);
std::fflush(stdout);
std::fflush(log);
lastReport = now, ticks = withHands = 0;
}
std::this_thread::sleep_until(tick + std::chrono::microseconds(11111));
}
std::fclose(log);
return 0;
}
} // namespace
int main(int argc, char **argv) {
double distance = 0.8, width = 1.0, seconds = 0;
for (int i = 1; i + 1 < argc; i += 2) {
if (!std::strcmp(argv[i], "--distance")) distance = std::atof(argv[i + 1]);
else if (!std::strcmp(argv[i], "--width")) width = std::atof(argv[i + 1]);
else if (!std::strcmp(argv[i], "--seconds")) seconds = std::atof(argv[i + 1]);
double distance = 0.8, width = 1.0, seconds = 0, leadMs = -1, dotMm = 3;
bool probe = false;
std::string logPath = "/tmp/handprobe-" + std::to_string(time(nullptr)) + ".jsonl";
for (int i = 1; i < argc; ++i) {
const bool more = i + 1 < argc;
if (!std::strcmp(argv[i], "--probe")) probe = true;
else if (!std::strcmp(argv[i], "--distance") && more) distance = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--width") && more) width = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--seconds") && more) seconds = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--lead") && more) leadMs = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--dot-mm") && more) dotMm = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--log") && more) logPath = argv[++i];
else return std::fprintf(stderr, "usage: ft-handtest [--distance m] [--width m] [--seconds s] "
"[--probe [--lead ms] [--dot-mm mm] [--log FILE]]\n"), 2;
}
signal(SIGINT, Stop);
signal(SIGTERM, Stop);
@@ -137,6 +266,16 @@ int main(int argc, char **argv) {
P.m[2][3] = float(hm.m[2][3] - std::cos(yaw) * distance);
panel.width = width, panel.height = width * H / W, panel.curve = 0, panel.pxWidth = W, panel.pxHeight = H;
vr::VROverlay()->SetOverlayTransformAbsolute(ov, vr::TrackingUniverseStanding, &P);
if (probe) {
const int rc = RunProbe(ov, panel, renderer, imports, seconds, leadMs, dotMm, logPath.c_str());
vr::VROverlay()->DestroyOverlay(ov);
for (auto &[k, h] : imports)
if (h) vr::VRIPCResourceManager()->UnrefResource(h);
imports.clear();
vr::VRIPCResourceManager()->UnrefResource(plain);
vr::VR_Shutdown();
return rc;
}
vr::VROverlay()->SetOverlayFlag(ov, vr::VROverlayFlags_IgnoreTextureAlpha, true);
vr::SharedTextureHandle_t shown = plain;
vr::Texture_t tex = {&shown, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
@@ -163,7 +302,7 @@ int main(int argc, char **argv) {
handcut::EyePositions(head, eyes);
cut = handcut::Project(panel, hands.capsules(), eyes, eyes2d);
}
const handcut::Output *out = cut ? renderer.Composite(0, bo, client, eyes2d) : nullptr;
const handcut::Output *out = cut ? renderer.Composite(0, bo, 1, client, eyes2d) : nullptr;
if (out) {
auto it = imports.find(out);
if (it == imports.end()) {
@@ -177,8 +316,7 @@ int main(int argc, char **argv) {
vr::VROverlay()->SetOverlayFlag(ov, vr::VROverlayFlags_SideBySide_Parallel, true);
cutting = true;
}
shown = it->second;
vr::VROverlay()->SetOverlayTexture(ov, &tex);
if (shown != it->second) shown = it->second, vr::VROverlay()->SetOverlayTexture(ov, &tex);
ms += renderer.lastMs(), worst = std::max(worst, renderer.lastMs());
++cutFrames;
caps2d += eyes2d[0].size() + eyes2d[1].size();
@@ -192,10 +330,12 @@ int main(int argc, char **argv) {
}
++frames;
if (now - lastReport > 2'000'000'000) {
const handcut::CutStats st = renderer.TakeStats();
std::printf("%.0f s: %d ticks, %d with a cutout (%.1f capsules per eye), composite %.2f ms avg %.2f ms worst, "
"%zu hand capsules known\n",
"%zu hand capsules known; %d draws (%d partial), %d same, %d busy\n",
(now - start) / 1e9, frames, cutFrames, cutFrames ? caps2d / 2.0 / cutFrames : 0.0,
cutFrames ? ms / cutFrames : 0.0, worst, hands.capsules().size());
cutFrames ? ms / cutFrames : 0.0, worst, hands.capsules().size(), st.draws, st.partial, st.same,
st.busy);
std::fflush(stdout);
lastReport = now, frames = cutFrames = 0, ms = worst = 0, caps2d = 0;
}
+71 -13
View File
@@ -11,8 +11,8 @@
// its centre (it snaps level within kRollSnap), or scroll on it for kRollStep steps.
// - a resize tab on the bottom right corner: drag it to set the width (the height
// follows the screen's resolution).
// - a reset button left of the bar: every screen back in its layout, around where you
// are now (`ft-layout apply`, like Meta+Shift+R).
// - a reset button left of the bar: the profile in use opened again, or every screen back
// in its layout, around where you are now (`ft-layout reset`, like Meta+Shift+R).
// The controls are translucent, like SteamVR's own, and brighten under a laser. They
// are invisible until a laser (a controller's, or the 3D mouse's) lands on or passes very close to
// one of them (UpdateControls).
@@ -288,7 +288,9 @@ struct Screen {
const void *key = nullptr; // the client buffer on it now, and its dmabuf (for cutouts)
ft_dmabuf buf{};
vr::SharedTextureHandle_t plain = 0; // that buffer's SteamVR import
uint64_t frames = 0; // client frames presented (a cutout buffer's "serial")
bool cutting = false; // showing a cutout buffer (side by side) instead
vr::SharedTextureHandle_t cutShown = 0; // ...this one
double chrome = 0.3; // the bar's width; the other controls follow it (ChromeSize)
double grip = 0.04; // the corner tab's and the round buttons' size
// A floating window's panel (see the top): the window's rectangle in the buffer, its
@@ -327,6 +329,28 @@ struct Screen {
}
};
std::map<int, Screen> g_screens;
// For the log and the debug line (scripts/report.sh): which screen, which drag, which laser.
int ScreenIndexOf(const Screen &s) {
for (const auto &[index, t] : g_screens)
if (&t == &s) return index;
return -1;
}
const char *DragName(Drag d) {
switch (d) {
case Drag::Move: return "move";
case Drag::Resize: return "resize";
case Drag::Roll: return "roll";
default: return "none";
}
}
std::string DeviceLabel(vr::TrackedDeviceIndex_t i) {
if (i == kNone) return "none";
if (vr::VRSystem()->GetTrackedDeviceClass(i) == vr::TrackedDeviceClass_Controller && !IsHandController(i))
return "3D mouse";
const std::string hand = HandName(i);
return hand == "none" ? "device " + std::to_string(i) : hand + " controller";
}
std::map<const void *, vr::SharedTextureHandle_t> g_imports;
// Hand cutouts (see the top and handcut.h).
@@ -1124,7 +1148,11 @@ vr::TrackedDeviceIndex_t WristOnLaser(const Screen &s, const Mat &d, const Mat &
void StartDrag(Screen &s, Drag mode, vr::TrackedDeviceIndex_t dev) {
Mat d, p;
if (dev == kNone || !DevicePose(dev, &d) || !ScreenPose(s, &p)) return;
if (dev == kNone || !DevicePose(dev, &d) || !ScreenPose(s, &p)) {
std::printf("screen %d: %s not started: no pose for the %s or the screen\n", ScreenIndexOf(s) + 1,
DragName(mode), DeviceLabel(dev).c_str());
return;
}
s.pinTarget = kNone;
if (s.pinned != kNone && mode == Drag::Move) {
// Carried freely; let go, it goes back on the same wrist (unless disarmed).
@@ -1148,11 +1176,15 @@ void StartDrag(Screen &s, Drag mode, vr::TrackedDeviceIndex_t dev) {
Mat l;
if (!LaserPose(dev, &l) || !RollLaserAngle(s, l, &s.rollAngle)) s.drag = Drag::None, s.dragDevice = kNone;
}
if (s.drag != Drag::None)
std::printf("screen %d: %s by the %s%s\n", ScreenIndexOf(s) + 1, DragName(mode), DeviceLabel(dev).c_str(),
vr::VROverlay()->IsDashboardVisible() ? " (Steam menu open)" : "");
ApplyAlpha(s);
}
// Stop moving where it is (a command took over).
void EndDrag(Screen &s) {
if (s.drag != Drag::None) std::printf("screen %d: %s ended\n", ScreenIndexOf(s) + 1, DragName(s.drag));
s.drag = Drag::None;
s.dragDevice = kNone;
s.pinTarget = s.onWrist = kNone;
@@ -1925,7 +1957,7 @@ void StopCutting(Screen &s) {
vr::VROverlay()->SetOverlayFlag(s.overlay, vr::VROverlayFlags_SideBySide_Parallel, false);
vr::VROverlay()->SetOverlayFlag(s.overlay, vr::VROverlayFlags_IgnoreTextureAlpha, true);
if (s.plain) SetScreenTexture(s, s.plain);
s.cutting = false;
s.cutting = false, s.cutShown = 0;
}
// Each tick: for each visible screen with a hand in front of it (for either eye), draw its
@@ -1946,7 +1978,7 @@ void UpdateCutouts() {
bool cut = hands && s.visible && !s.floating && s.key && s.width > 0 && ScreenPose(s, &p) &&
handcut::Project({p, s.metres, s.heightMetres(), s.curve, s.width, s.height}, g_hands.capsules(),
eyes, spots);
const handcut::Output *out = cut && CutterReady() ? g_cutter.Composite(i, s.key, s.buf, spots) : nullptr;
const handcut::Output *out = cut && CutterReady() ? g_cutter.Composite(i, s.key, s.frames, s.buf, spots) : nullptr;
const vr::SharedTextureHandle_t h = out ? ImportCutout(out) : 0;
if (!h) {
StopCutting(s);
@@ -1957,7 +1989,7 @@ void UpdateCutouts() {
vr::VROverlay()->SetOverlayFlag(s.overlay, vr::VROverlayFlags_SideBySide_Parallel, true);
s.cutting = true;
}
SetScreenTexture(s, h);
if (h != s.cutShown) SetScreenTexture(s, h), s.cutShown = h; // the same buffer stays: no new frame for SteamVR
}
}
@@ -1982,6 +2014,8 @@ void UpdateSteamInFront() {
const bool front = SteamInFront();
if (front == g_steamInFront) return;
g_steamInFront = front;
std::printf("Steam %s%s\n", front ? "in front (the Steam menu or Steam's keyboard)" : "out of the way",
front && keyboard::Shown() ? ": our keyboard steps aside" : !front && g_keyboardAside ? ": our keyboard comes back" : "");
if (front && keyboard::Shown()) {
g_asidePose = keyboard::Pose();
keyboard::Hide();
@@ -2277,7 +2311,7 @@ bool ft_vr_screen_present(int index, const void *key, const struct ft_dmabuf *b)
PlaceChrome(s); // the height changed
std::printf("screen %d: %dx%d\n", index + 1, s.width, s.height);
}
s.key = key, s.buf = *b, s.plain = it->second;
s.key = key, s.buf = *b, s.plain = it->second, ++s.frames;
// While cutting, the next tick draws the new buffer with the cutouts (never floating).
if (!s.cutting) SetScreenTexture(s, it->second);
vr::SharedTextureHandle_t handle = it->second;
@@ -2429,14 +2463,14 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
}
}
}
// The reset button: every screen back in the layout, around where you are now
// (`ft-layout apply`, like Meta+Shift+R; it refuses a second copy).
// The reset button: the profile in use opened again, or every screen back in the layout,
// around where you are now (`ft-layout reset`, like Meta+Shift+R; it refuses a second copy).
while (s.resetButton != vr::k_ulOverlayHandleInvalid &&
vr::VROverlay()->PollNextOverlayEvent(s.resetButton, &ev, sizeof ev)) {
hover(6);
if (ev.eventType == vr::VREvent_MouseButtonDown && ev.data.mouse.button == vr::VRMouseButton_Left) {
std::printf("screen %d: reset the layout\n", index + 1);
RunLayout("apply");
RunLayout("reset");
} else if (ev.eventType == vr::VREvent_MouseButtonUp) {
EndDragsBy(ev.trackedDeviceIndex);
ReleaseAwayBy(ev.trackedDeviceIndex, ev.data.mouse.button, handle, data);
@@ -2775,9 +2809,18 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
g_hands.SetPrediction(g_hands.predicting(), ms);
else if (std::strcmp(word, "state") != 0)
return (void)std::snprintf(reply, size, "error cutouts on|off|state|predict on|off|lead <ms>");
std::snprintf(reply, size, "ok %s %s %.2f ms, predict %s lead %.0f ms", g_cutouts ? "on" : "off",
g_cutterState > 0 ? "ready" : g_cutterState < 0 ? "unavailable" : "idle", g_cutter.lastMs(),
g_hands.predicting() ? "on" : "off", g_hands.leadMs());
// Composite's counts since the last state, per second.
static auto since = Clock::now();
const double dt = std::max(1e-3, std::chrono::duration<double>(Clock::now() - since).count());
since = Clock::now();
const handcut::CutStats c = g_cutter.TakeStats();
const int calls = c.draws + c.same + c.busy;
std::snprintf(reply, size,
"ok %s %s %.2f ms, predict %s lead %.0f ms; per s: %.1f draws (%.1f partial), %.1f same, %.1f busy, "
"%.1f waits; CPU %.2f ms avg %.2f worst",
g_cutouts ? "on" : "off", g_cutterState > 0 ? "ready" : g_cutterState < 0 ? "unavailable" : "idle",
g_cutter.lastMs(), g_hands.predicting() ? "on" : "off", g_hands.leadMs(), c.draws / dt,
c.partial / dt, c.same / dt, c.busy / dt, c.waits / dt, calls ? c.cpuMs / calls : 0.0, c.worstMs);
} else if (int x0, y0, w0, h0, t0; std::sscanf(cmd, "float %d %lf %d %d %d %d %d", &n, &w, &x0, &y0, &w0, &h0, &t0) == 7) {
Screen *s = Find(n);
if (!s || !s->floating) return (void)std::snprintf(reply, size, "error no floating window panel %d", n);
@@ -2827,6 +2870,21 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
std::snprintf(reply, size, "ok");
} else if (std::sscanf(cmd, "spin %15s", word) == 1) {
SpinCommand(word, reply, size);
} else if (std::strcmp(cmd, "debug") == 0) {
// One line for scripts/report.sh: what decides whether a laser drags and the keyboard shows.
std::string drags;
for (const auto &[index, s] : g_screens)
if (s.drag != Drag::None)
drags += (drags.empty() ? "" : ",") + std::to_string(index + 1) + ":" + DragName(s.drag) + ":" +
DeviceLabel(s.dragDevice);
std::snprintf(reply, size,
"ok dashboard=%d steam_front=%d keyboard=%s mode=%s manual=%d lasers=%s game=%d paused=%d "
"press=%#x by=%s on=%d catcher=%d drags=%s",
vr::VROverlay()->IsDashboardVisible() ? 1 : 0, SteamInFront() ? 1 : 0,
keyboard::Shown() ? "shown" : g_keyboardAside ? "aside" : "hidden", ModeName(), g_manual ? 1 : 0,
LasersName(), g_gameRunning ? 1 : 0, g_paused ? 1 : 0, g_press.buttons,
DeviceLabel(g_press.device).c_str(), g_press.screen + 1, g_catcherShown ? 1 : 0,
drags.empty() ? "-" : drags.c_str());
} else if (std::strncmp(cmd, "state", 5) == 0) {
std::snprintf(reply, size, "ok %s %d %.0f %s %.0f %s %d %s", ModeName(), g_manual ? 1 : 0, g_wristAngle,
g_gestureHand.c_str(), g_gestureAngle, LasersName(), g_gameRunning ? 1 : 0,
+5 -1
View File
@@ -44,7 +44,11 @@ else
fi
check "SteamOS" on_home '. /etc/os-release; echo "$PRETTY_NAME $VERSION_ID build $BUILD_ID, $(uname -m)"'
check "distrobox" on_home 'test -x ~/.local/bin/distrobox && ~/.local/bin/distrobox version | head -1'
check "container $FRAME_BOX" on_home "podman ps -a --filter name=^$FRAME_BOX\$ --format '{{.Image}} {{.Status}}' | grep ."
# A release runs in its own container, which its .frametop-release names (as scripts/in-box
# reads it); a checkout builds and runs in FRAME_BOX.
box=$FRAME_BOX
[ "$FRAME_RELEASE" = 1 ] && box=${FRAMETOP_BOX:-$(sed -n 's/^BOX=//p' "$REPO_ROOT/.frametop-release" | tail -1)}
check "container ${box:-(none named)}" on_home "podman ps -a --filter name=^$box\$ --format '{{.Image}} {{.Status}}' | grep ."
check "free space in ~" on_home "df -h ~ | awk 'NR==2{print \$4\" free\"}'"
if check "repo on the Frame" on_frame 'pwd'; then
# A taskbar saved on a screen the desktop doesn't have is hidden; the desktop's next start
+1 -1
View File
@@ -204,7 +204,7 @@ def main():
with open(path, "w") as f:
f.write(text)
print(f"Wrote {path}")
print("Attach it to https://github.com/DeeJanuz/frametop/issues/2 with what you saw at each step.")
print("Attach it to https://github.com/Frametop/frametop/issues/2 with what you saw at each step.")
if __name__ == "__main__":
+124 -8
View File
@@ -1,11 +1,23 @@
#!/usr/bin/env bash
# Collect what a Frametop bug report needs into one text file: versions, service states,
# settings, recent logs, and the gaze report (scripts/gaze-report.py). Bluetooth addresses and
# the headset's serial number are masked.
# Usage: scripts/report.sh (in a terminal on the Frame, or from a PC over SSH)
# settings, Frametop's keyboard and the Steam menu, recent logs, and the gaze report
# (scripts/gaze-report.py). Bluetooth addresses and the headset's serial number are masked.
# Usage: scripts/report.sh [--watch [SECONDS]] (in a terminal on the Frame, or from a PC over SSH)
# --watch then record SECONDS (default 60) while you make the problem happen: the
# desktop's state (the Steam menu, our keyboard, which laser drags what, where
# typing goes) whenever it changes, and the logs from that time
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
. "$root/scripts/_env.sh"
watch=0
case ${1:-} in
"") ;;
--watch) watch=${2:-60}
[[ $watch =~ ^[0-9]+$ ]] && [ "$watch" -ge 5 ] && [ "$watch" -le 600 ] ||
{ echo "--watch takes 5 to 600 seconds" >&2; exit 2; } ;;
-h|--help) sed -n '2,8p' "$0"; exit 0 ;;
*) echo "usage: $0 [--watch [SECONDS]]" >&2; exit 2 ;;
esac
out=$root/frametop-report-$(date +%Y%m%d-%H%M%S).txt
echo "Collecting (up to half a minute: the gaze service is woken to see whether the eye tracker sends)..."
@@ -14,18 +26,45 @@ repo=$1
export XDG_RUNTIME_DIR=/run/user/$(id -u) DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus
logs=~/.local/share/Steam/logs
section() { printf '\n===== %s\n' "$*"; }
# ask SOCKET COMMAND: a request to one of Frametop's abstract datagram sockets; prints the reply.
ask() {
python3 -I - "$1" "$2" <<'PY' 2>&1
import socket, sys
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
s.bind("")
s.settimeout(2)
try:
s.sendto(sys.argv[2].encode(), "\0" + sys.argv[1])
print(s.recv(65536).decode())
except OSError as e:
print(f"no answer from @{sys.argv[1]} ({e})")
PY
}
section Versions
grep -E '^(PRETTY_NAME|VERSION_ID|BUILD_ID|VARIANT_ID)=' /etc/os-release
grep -m1 -o 'vrcompositor [0-9.]* startup' $logs/vrcompositor.txt 2>/dev/null || echo "SteamVR: not found"
git -C "$repo" log -1 --format='Frametop %h (%cd)' --date=short 2>/dev/null || echo "Frametop: not a git checkout"
# The log keeps earlier runs: the last start is this SteamVR.
grep -a -o 'vrcompositor [0-9.]* startup' $logs/vrcompositor.txt 2>/dev/null | tail -n 1 | grep . || echo "SteamVR: not found"
if [ -f "$repo/.frametop-release" ]; then
rel() { sed -n "s/^$1=//p" "$repo/.frametop-release" | tail -1; }
echo "Frametop release $(rel VERSION) ($(rel COMMIT | cut -c1-7), $(rel CHANNEL)) in $repo"
box=$(rel BOX)
else
git -C "$repo" log -1 --format='Frametop %h (%cd)' --date=short 2>/dev/null || echo "Frametop: not a git checkout"
git -C "$repo" branch --show-current 2>/dev/null | sed 's/^/branch: /'
box=${FRAME_BOX:-dev}
fi
~/.local/bin/distrobox version 2>/dev/null || echo "distrobox: not installed"
podman container inspect -f 'dev container: running={{.State.Running}} image={{.ImageName}}' dev 2>/dev/null || echo "dev container: missing"
podman container inspect -f "container $box: running={{.State.Running}} image={{.ImageName}}" "$box" 2>/dev/null ||
echo "container $box: missing"
section Services
for u in frametop-input-relay frametop-pointer frametop-power; do
for u in frametop-input-relay frametop-pointer frametop-power frametop-gaze frametop-desktop; do
echo "$u: $(systemctl --user is-enabled $u 2>/dev/null) / $(systemctl --user is-active $u 2>/dev/null)"
done
for u in frametop-eyegrab steamframe-bt-fixups; do
echo "$u (system): $(systemctl is-enabled $u 2>/dev/null) / $(systemctl is-active $u 2>/dev/null)"
done
echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)"
echo "paused for VR games: $(cat /run/user/$(id -u)/frametop-pause.json 2>/dev/null || echo 'no (never paused since boot)')"
# SteamVR's config folder: a terminal in the desktop has the session's own (docs/design.md).
@@ -64,6 +103,46 @@ r = subprocess.run(["qdbus6", "org.kde.plasmashell", "/PlasmaShell", "org.kde.Pl
print("live panels:", (r.stdout or r.stderr).strip())
PY
section "Frametop's keyboard and the Steam menu"
python3 - "$(ask frametop_relay devices)" <<'PY' 2>&1
import json, os, sys
try:
rules = json.load(open(os.path.expanduser("~/.config/frametop-input.json")))
except (OSError, ValueError):
rules = {}
print("Keyboard setting (Input Settings > Keyboard):", rules.get("vr_keyboard", "no_keyboard (the default)"),
"| keep it open:", rules.get("vr_keyboard_persist", True))
print("the relay's devices (with the default setting, a pass-through keyboard keeps ours closed):")
try:
d = json.loads(sys.argv[1])
except ValueError:
sys.exit(print(" " + sys.argv[1].strip()))
print(f" pointer mode: {d.get('pointer_mode')}, paused: {d.get('paused')}")
for n in d.get("nodes", []):
print(" " + json.dumps(n))
PY
echo "ft-textinput (tells the relay about text fields): $(pgrep -f '[f]t-textinput' >/dev/null && echo running || echo 'not running')"
echo "the desktop's KWin input method: $(pgrep -a kwin_wayland | grep -o -- '--inputmethod [^ ]*' | head -n 1 | grep . || echo 'none (the keyboard never opens for text fields)')"
for p in $(pgrep -x plasmashell); do
env=$(tr '\0' '\n' < /proc/$p/environ 2>/dev/null) || continue
echo "$env" | grep -q '^XDG_CONFIG_HOME=.*/frametop$' || continue
for v in QT_IM_MODULE GTK_IM_MODULE; do # xim here: Qt and GTK apps never ask for the keyboard
printf "the desktop's %s: %s\n" $v "$(echo "$env" | sed -n "s/^$v=//p" | grep . || echo unset)"
done
done
echo "ft-screens debug: $(ask ft_screens debug)"
echo "ft-screens state (mode, manual, wrist, gesture hand and angle, lasers, game, in games): $(ask ft_screens state)"
echo "ft-screens pause: $(ask ft_screens 'pause state'), concealed: $(ask ft_screens concealed)"
echo "SteamVR overlays shown now:"
XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 timeout 10 \
/opt/steamvr/bin/linuxarm64/vrcmd --overlays 2>/dev/null |
awk '/^\x27/ { show = $0 !~ /not_visible/ } show' | sed 's/^/ /' | head -n 80
echo "keyboard, drag and Steam menu lines from ft-screens (last 40):"
grep -a -E 'keyboard|Steam (in front|out of the way)|: (move|resize|roll)( by| ended| not started)' \
/tmp/frametop-screens.log 2>/dev/null | tail -n 40 | sed 's/^/ /'
echo "text fields, from the relay (last 20):"
journalctl --user -u frametop-input-relay --no-pager -o short 2>/dev/null | grep 'text field' | tail -n 20 | sed 's/^/ /'
section "Input relay (last 60 lines)"
journalctl --user -u frametop-input-relay -n 60 --no-pager -o short 2>/dev/null
section "Pointer helper (last 60 lines)"
@@ -81,6 +160,43 @@ section "SteamVR server: Frametop and errors (last 60 lines)"
grep -a -i -E 'ft_pointer|frametop|\[error\]' $logs/vrserver.txt 2>/dev/null | tail -n 60
EOF
if [ "$watch" -gt 0 ]; then
echo
echo "Now make the problem happen within $watch seconds: for example, open the Steam menu, try to"
echo "drag a window by its bar, and click into a text field. Recording..."
on_frame_script "$watch" >> "$out" 2>&1 <<'EOF' || true
seconds=$1
printf '\n===== Watched for %s s from %s\n' "$seconds" "$(date '+%F %T')"
since=$(date '+%F %T')
start=$(wc -l < /tmp/frametop-screens.log 2>/dev/null || echo 0)
python3 -I - "$seconds" <<'PY'
import socket, sys, time
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
s.bind("")
s.settimeout(1)
end, last = time.monotonic() + float(sys.argv[1]), None
print("ft-screens' debug line whenever it changed:")
while time.monotonic() < end:
try:
s.sendto(b"debug", "\0ft_screens")
line = s.recv(4096).decode()
except OSError as e:
line = f"no answer ({e})"
if line != last:
print(time.strftime("%H:%M:%S") + f"{time.time() % 1:.2f}"[1:], line, flush=True)
last = line
time.sleep(0.2)
PY
printf '\n===== ft-screens log while watching\n'
tail -n +"$((start + 1))" /tmp/frametop-screens.log 2>/dev/null | tail -n 300
for u in frametop-input-relay frametop-pointer; do
printf '\n===== %s while watching\n' "$u"
journalctl --user -u $u --since "$since" --no-pager -o short 2>/dev/null | tail -n 200
done
EOF
echo "Recorded."
fi
sed -i -E 's/([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}/xx:xx:xx:xx:xx:xx/g; s/cv\.[A-Z0-9]{8,}/cv.<serial>/g' "$out"
echo "wrote $out"
echo "Attach it to an issue at https://github.com/DeeJanuz/frametop/issues, with what you did, what you expected, and what happened."
echo "Attach it to an issue at https://github.com/Frametop/frametop/issues, with what you did, what you expected, and what happened."
+64 -3
View File
@@ -41,6 +41,9 @@ VRPATHS = os.path.join(HOME, ".config/openvr/openvrpaths.vrpath")
STRAY_VRPATHS = os.path.join(HOME, ".config/frametop/openvr/openvrpaths.vrpath")
BACKLIGHT = "/sys/class/backlight/ae94000.dsi.0/brightness" # as in power/ft-powerd.cpp
EYE_MMAP = "/dev/shm/eye-server.mmap"
# SteamVR's settings, where SteamOS 0.4's Track Dominant Eye Only lives (as gaze/gazecal.py reads it).
STEAMVR_SETTINGS = (os.path.join(HOME, ".config/openvr/config/steamvr.vrsettings"),
os.path.join(HOME, ".steam/steam/config/steamvr.vrsettings"))
VRSERVER = "http://127.0.0.1:27062" # its web socket: input/vrws.py
PAUSE_STATE = f"/run/user/{os.getuid()}/frametop-pause.json" # input/game_pause.py
HOST_GLIBC = (2, 39) # the newest the pointer driver may need (pointer/driver/build.sh)
@@ -48,7 +51,8 @@ HOST_GLIBC = (2, 39) # the newest the pointer driver may need (pointer/driver/b
# Packages in the OS image that Frametop depends on, and what to try by hand when one changes.
PACKAGES = {
"deckard-steamvr-rel": "the 3D mouse on the dashboard, on SteamVR Settings, and on a SteamVR "
"window's grab bar; picking up a controller; mapped controller buttons; gaze",
"window's grab bar; picking up a controller; mapped controller buttons; gaze, "
"and a gaze calibration with Track Dominant Eye Only on",
"kwin": "clicks near the far edge of a screen whose scale isn't 1; every screen comes back "
"after a desktop restart; floating a window; no blur behind the taskbar's menus",
"plasma-workspace": "the taskbar and panels after a desktop restart; no DiscoverNotifier or "
@@ -57,7 +61,10 @@ PACKAGES = {
"the registry stops and comes back after a desktop restart",
"gamescope": "the headset's volume buttons with nothing focused; typing goes where you last clicked",
"bluez": "a Bluetooth mouse reconnecting after it sleeps",
"steamdeck-kde-presets": "Launch a program -> Native Desktop opens SteamOS's own desktop",
"steamdeck-kde-presets": "Launch a program -> Native Desktop opens SteamOS's own desktop; starting "
"the desktop sends Steam no command line (no ExecCommandLine in "
"~/.local/share/Steam/logs/console_log.txt)",
"holo-cursors": "the desktop keeps its Breeze cursor (over remote access, and with the gamescope backend)",
}
KERNEL_HINT = "display power (ft-powerd) and hand tracking"
@@ -75,7 +82,7 @@ UNITS = {
# layouts its EyeFile::Detect knows: everything from the timestamp on moved by a shift.
EYE_COUNTER, EYE_TIME, EYE_NEED = 0x38, 0x157, 0x1F3 + 5
EYE_SET1 = (0x15F, 0x16B) # set 1 left, right
EYE_LAYOUTS = {0: "stable", 5: "the 0.4.x beta's (+5)"}
EYE_LAYOUTS = {0: "SteamOS 0.3's", 5: "SteamOS 0.4's (+5)"}
# Runs in a child process, so a SteamVR that hangs can't hang the check.
OPENVR_PROBE = r"""
@@ -185,6 +192,8 @@ def check_host():
("/etc/profile.d/flatpak.sh", "warn", "Flatpak apps may open Discover instead of starting"),
(STOCK_LAUNCHER, "warn", "SteamOS's Desktop launcher entry is gone or renamed, so Frametop's "
"copy may not replace it, and there's no Native Desktop"),
("/usr/share/steamos/steamos-cursor.png", "warn",
"the gamescope backend has no cursor image (SteamOS 0.4's own session moved to /usr/share/holo)"),
]
missing = [n for n in needed if not os.path.exists(n[0])]
for path, state, effect in missing:
@@ -312,6 +321,36 @@ def installed_binaries():
return found
# Frametop's root files in /etc, by what installs them. A SteamOS update deletes every /etc file
# its keep list (/usr/lib/rauc/atomic-update-keep.conf) doesn't name: it keeps the units, but not
# what they run, unless a drop-in in /etc/atomic-update.conf.d names that too.
ROOT_PARTS = [
("Bluetooth fixes", "/etc/systemd/system/steamframe-bt-fixups.service", "/etc/steamframe/bt-fixups.sh",
"/etc/atomic-update.conf.d/frametop-bluetooth.conf", "setup/bluetooth/install.sh install"),
("eye tracker's frame grabber", "/etc/systemd/system/frametop-eyegrab.service", "/etc/frametop/ft-eyegrab",
"/etc/atomic-update.conf.d/frametop-eyegrab.conf", "gaze/tracker/install.sh"),
]
def check_root_files():
for label, unit, program, keep, install in ROOT_PARTS:
if not os.path.exists(unit):
continue
try:
with open(keep) as f:
kept = program in (line.strip() for line in f)
except OSError:
kept = False
if not os.path.exists(program):
report("FAIL", label, f"{program} is gone (a SteamOS update deletes it unless it's kept); "
f"reinstall with {install}")
elif not kept:
report("warn", label, f"the next SteamOS update will delete {program}; "
f"reinstall with {install}, which keeps it")
else:
report("ok", label, "installed, and kept through SteamOS updates")
def check_openvr():
needs = {} # interface version: programs built against it
for path in installed_binaries():
@@ -471,8 +510,29 @@ def check_vr_socket():
"(input/vrws.py: the message format changed?)")
def tracked_eye():
"""gaze/gazecal.py's tracked_eye: the one eye SteamVR's tracker follows with Track Dominant
Eye Only on (0 left, 1 right), or None."""
for path in STEAMVR_SETTINGS:
try:
with open(path) as f:
steamvr = json.load(f).get("steamvr")
except OSError:
continue
except (ValueError, AttributeError):
return None
if not isinstance(steamvr, dict) or steamvr.get("eyeTrackingDominantEyeOnly") is not True:
return None
return 0 if steamvr.get("dominantEye", 1) == 0 else 1
return None
def check_eye_tracker():
gaze = systemctl("is-enabled", "frametop-gaze") == "enabled"
one = tracked_eye()
if one is not None:
report("ok", "eye tracking", f"SteamVR tracks your {('left', 'right')[one]} eye only (Track Dominant "
"Eye Only), and Frametop's gaze goes by that eye")
try:
with open(EYE_MMAP, "rb") as f:
m = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
@@ -522,6 +582,7 @@ def main():
versions = current_versions()
check_versions(versions)
check_host()
check_root_files()
steamvr_up = run("pgrep", "-x", "vrserver")[0] == 0
if not steamvr_up:
report("skip", "SteamVR checks", "SteamVR isn't running")
+23 -4
View File
@@ -178,6 +178,16 @@ export XDG_CONFIG_HOME=$HOME/.config/frametop
export XDG_STATE_HOME=$HOME/.local/state/frametop
mkdir -p "$XDG_CONFIG_HOME" "$XDG_STATE_HOME"
# The cursor: the Steam client's XCURSOR_THEME=steam comes along in the environment, and
# KWin and the apps take it over this desktop's own setting. SteamOS had no theme of that
# name, so they fell back to Breeze; SteamOS 0.4 has one (holo-cursors: Steam's arrow, the
# rest Breeze Light). The theme is this desktop's own (System Settings, Breeze unless
# changed there); the size stays what it was, unless one is set there too.
cursor_theme=$(kreadconfig6 --file kcminputrc --group Mouse --key cursorTheme)
cursor_size=$(kreadconfig6 --file kcminputrc --group Mouse --key cursorSize)
export XCURSOR_THEME=${cursor_theme:-breeze_cursors}
[ -z "$cursor_size" ] || export XCURSOR_SIZE=$cursor_size
# Remote desktop over VNC: session/remote-desktop.sh captures the desktop with
# krdp on 127.0.0.1, and session/vnc-bridge.sh re-serves its primary screen over VNC. krdpserver runs from the container, so KWin can't
# match it to an installed app. KWin's permission check for screencast and fake
@@ -273,15 +283,24 @@ fi
# 600 MB and a share of a core, with Flatpak's helper and AppStream behind it); updates
# come with SteamOS and from Discover in the stock desktop. IBus can't reach the
# desktop's apps: KWin's input method is ft-textinput, and the session drops the
# variables that point apps at IBus or XIM. Deleting the copy brings an entry back.
if [ "$(kreadconfig6 --file "$frametoprc" --group Defaults --key autostart)" != 1 ]; then
for entry in org.kde.discover.notifier ibus; do
# variables that point apps at IBus or XIM. Steam is already running (the desktop starts
# from it), so its entry's `steam -silent` only reaches that client as a command line it
# runs; SteamOS 0.4 adds -vrdisable -deckard to it, meant for Desktop Mode's own Steam.
# Deleting the copy brings an entry back. The marker is the last list done, so an entry
# added later is hidden once on desktops that did the first list, and none comes back.
case $(kreadconfig6 --file "$frametoprc" --group Defaults --key autostart) in
2) hide= ;;
1) hide=steam ;;
*) hide="org.kde.discover.notifier ibus steam" ;;
esac
if [ -n "$hide" ]; then
for entry in $hide; do
src=/etc/xdg/autostart/$entry.desktop dst=$XDG_CONFIG_HOME/autostart/$entry.desktop
[ -r "$src" ] && [ ! -e "$dst" ] || continue
mkdir -p "$(dirname "$dst")"
sed '/^\[Desktop Entry\]$/a Hidden=true' "$src" > "$dst"
done
kwriteconfig6 --file "$frametoprc" --group Defaults --key autostart 1
kwriteconfig6 --file "$frametoprc" --group Defaults --key autostart 2
fi
# Profiles reopen apps (docs/profiles.md), so Plasma's own session restore stays off here;
+1
View File
@@ -48,6 +48,7 @@ HANDS_BRIGHT_ON=40 # hand tracking, auto: the colour frames' mean bright
HANDS_BRIGHT_OFF=25 # ... and back to the mono cameras under this
HANDS_COLOR_LEFT=color_video0 # hand tracking: which colour camera is passthrough_left (hands/tools/check_color.py tells)
HANDS_COLOR_CROP=subtract # hand tracking: how the colour calibration's crop applies, subtract | none (check_color.py tells)
#HANDS_MODELS= # hand tracking: a folder of palm.ncnn.* and hand.ncnn.* to use instead of the stock models (hands/models/ncnn)
POINTER_HANDS=0 # 1 = pinches click (on the release; hold and move to nudge the pointer first) and grips press and drag, with hand tracking
POINTER_PINCH_GAIN=0.5 # hands: the pointer moves this times the pinching hand's angle (precision)
POINTER_PINCH_DEADZONE=1.5 # hands: degrees the pinching hand moves before the pointer does (a tap's jitter)
+13 -1
View File
@@ -32,6 +32,7 @@ Both settings reset whenever bluetoothd restarts or the Frame reboots, so they h
| --- | --- |
| `/etc/steamframe/bt-fixups.sh` | Turns controller privacy off, then sets the `ADDRESS_RESOLUTION` flags (`0x6`) on every bonded LE device that has an identity key |
| `/etc/systemd/system/steamframe-bt-fixups.service` | Runs the script after every Bluetooth start |
| `/etc/atomic-update.conf.d/frametop-bluetooth.conf` | Keeps the script through SteamOS updates |
The service runs after Bluetooth has started and never makes Bluetooth wait for it, because SteamOS's `set-bluetooth-mac-address.service`, which gives the Bluetooth chip its address, needs `bluetooth.service` to finish starting first. SteamOS's own files, including `main.conf`, are left untouched, so system updates won't conflict.
@@ -54,7 +55,7 @@ The install writes to `/etc`, so it needs `sudo` and the `steamos` user's passwo
setup/bluetooth/install.sh install
```
It copies the files above into place (`/etc` survives SteamOS updates) and enables the service. `./install.sh` offers this same step.
It copies the files above into place and enables the service. A SteamOS update deletes every `/etc` file that SteamOS's keep list (`/usr/lib/rauc/atomic-update-keep.conf`) doesn't name. That list keeps the service but not the script, so the third file adds the script to it. `./install.sh` offers this same step.
### Step 3: pair your mouse or keyboard
@@ -88,6 +89,17 @@ The real test is to reboot the Frame, then move or click the device. It should r
Run `setup/bluetooth/install.sh run` again, then wake the device. Check that its address appears in the service log. The script only flags devices that have an identity key: look for `[IdentityResolvingKey]` in `/var/lib/bluetooth/<controller>/<device>/info` (readable as root).
#### Devices stopped reconnecting after a SteamOS update
Fixes installed before the keep entry existed lose their script in the update, and the service fails with `Unable to locate executable '/etc/steamframe/bt-fixups.sh'`. `scripts/doctor.sh` reports it too. Install again, which adds the keep entry, then start the service:
```
setup/bluetooth/install.sh install
sudo systemctl restart steamframe-bt-fixups.service
```
The update keeps a copy of the deleted files in `/etc/previous`.
#### No Bluetooth at all after a boot
Check the controller:
+4
View File
@@ -0,0 +1,4 @@
# Installed to /etc/atomic-update.conf.d/frametop-bluetooth.conf by setup/bluetooth/install.sh.
# A SteamOS update deletes every /etc file its keep list (/usr/lib/rauc/atomic-update-keep.conf)
# doesn't name. That list keeps the unit, but not the script it runs.
/etc/steamframe/bt-fixups.sh
+3 -1
View File
@@ -18,6 +18,7 @@ case ${1:-install} in
sudo_run "set -e
install -D -m 0755 -o root -g root $src/bt-fixups.sh /etc/steamframe/bt-fixups.sh
install -D -m 0644 -o root -g root $src/steamframe-bt-fixups.service /etc/systemd/system/steamframe-bt-fixups.service
install -D -m 0644 -o root -g root $src/atomic-update.conf /etc/atomic-update.conf.d/frametop-bluetooth.conf
rm -f /etc/systemd/system/bluetooth.service.d/steamframe.conf
rmdir /etc/systemd/system/bluetooth.service.d 2>/dev/null || true
systemctl daemon-reload
@@ -25,7 +26,8 @@ systemctl enable steamframe-bt-fixups.service
echo installed" ;;
uninstall)
sudo_run "systemctl disable steamframe-bt-fixups.service 2>/dev/null
rm -f /etc/systemd/system/steamframe-bt-fixups.service /etc/systemd/system/bluetooth.service.d/steamframe.conf /etc/steamframe/bt-fixups.sh
rm -f /etc/systemd/system/steamframe-bt-fixups.service /etc/systemd/system/bluetooth.service.d/steamframe.conf /etc/steamframe/bt-fixups.sh \
/etc/atomic-update.conf.d/frametop-bluetooth.conf
rmdir /etc/systemd/system/bluetooth.service.d /etc/steamframe 2>/dev/null; systemctl daemon-reload; echo removed" ;;
run) sudo_run "/etc/steamframe/bt-fixups.sh" ;;
*) echo "usage: $0 [install|uninstall|run]" >&2; exit 2 ;;
+7 -6
View File
@@ -2,7 +2,7 @@
# Uninstall Frametop from the Steam Frame. In a terminal on the headset (Konsole in the desktop,
# or over SSH):
#
# curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash
# curl -fsSL https://frametop.github.io/frametop/uninstall.sh | bash
#
# or ~/frametop/uninstall.sh. It doesn't use the rest of the repo, so it also works when
# ~/frametop is gone or broken.
@@ -28,7 +28,7 @@ shopt -s nullglob
usage() {
cat <<'EOF'
usage: uninstall.sh [--dir DIR] [--dry-run]
piped: curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash -s -- [options]
piped: curl -fsSL https://frametop.github.io/frametop/uninstall.sh | bash -s -- [options]
EOF
}
@@ -41,9 +41,10 @@ driver=$HOME/.local/share/frametop/ft_pointer
releases=$HOME/.local/share/frametop/releases # pack/install-release.sh
vrpathreg=/opt/steamvr/bin/linuxarm64/vrpathreg
handsctl=$HOME/.local/bin/ft-handsctl
eyegrab_files=(/etc/systemd/system/frametop-eyegrab.service /etc/frametop/ft-eyegrab)
eyegrab_files=(/etc/systemd/system/frametop-eyegrab.service /etc/frametop/ft-eyegrab
/etc/atomic-update.conf.d/frametop-eyegrab.conf)
bt_files=(/etc/systemd/system/steamframe-bt-fixups.service /etc/systemd/system/bluetooth.service.d/steamframe.conf
/etc/steamframe/bt-fixups.sh)
/etc/steamframe/bt-fixups.sh /etc/atomic-update.conf.d/frametop-bluetooth.conf)
step() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
run() { # run a command, or with --dry-run, show it
@@ -184,9 +185,9 @@ main() {
echo "($repo) and, if you want, your settings and the build container:"
echo
if [ "$repo" = "$HOME/frametop" ]; then
echo " curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash"
echo " curl -fsSL https://frametop.github.io/frametop/uninstall.sh | bash"
else
echo " curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash -s -- --dir $(printf %q "$repo")"
echo " curl -fsSL https://frametop.github.io/frametop/uninstall.sh | bash -s -- --dir $(printf %q "$repo")"
fi
echo
if ask "Restart the headset now? This closes everything open, in VR and on the desktop." n; then