Compare commits

...
127 Commits
Author SHA1 Message Date
DeeJanuzandClaude Opus 5.5 24a86dc026 Merge branch 'experimental' into main: Frametop 0.2.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:17:40 -06:00
DeeJanuzandClaude Opus 5.5 7160180906 Merge branch 'org-home' into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:17:40 -06:00
DeeJanuzandClaude Opus 5.5 72d9b79d7b Frametop moves to the organization's repo: issues too
The plan changed from two repos to one: DeeJanuz/frametop transfers to the Frametop
organization, so issues and pull requests go to Frametop/frametop as well (report.sh,
keys-report.py, README, the hand recorder page). The consent text keeps its old link, which
GitHub redirects, so its wording doesn't change. pack/README.md says how the move keeps the old
links and the old one-liner working.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:17:40 -06:00
DeeJanuzandClaude Opus 5.5 194eed71c9 Merge branch 'experimental' into main: Frametop 0.2.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:11:49 -06:00
DeeJanuzandClaude Opus 5.5 b5e410b0b9 Merge branch 'org-home' into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:11:45 -06:00
DeeJanuzandClaude Opus 5.5 d9f213b872 Frametop's home is the organization's repo; a stable release in get.sh
New clones, the one-liners (frametop.github.io/frametop), the SteamOS table that releases
check, and FrameDrop's download URLs now point at Frametop/frametop, where CI builds the
releases. Issues and pull requests stay on DeeJanuz/frametop, the upstream it mirrors.
get.sh's menu offers a stable release next to the experimental one (3 and 4). SteamOS 0.4.5
(20261007.6125817) is marked tested with Frametop 0.2.2. pack/README.md says how the two repos
and a release fit together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:11:43 -06:00
DeeJanuzandClaude Opus 5.5 811e15ed9d Merge branch 'calpanel-clicks' into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:30:46 -06:00
DeeJanuzandClaude Opus 5.5 7f55e9328a Gaze calibration: don't wait for ft-eyes' answers
With our tracker, the full calibration asked ft-eyes about each dot (calib-point, up to 3 s)
and for the fit (calib-fit, up to 10 s) with a blocking ask(), which held up the whole gaze
service. Meanwhile the gaze stopped, the helper's 3 s panel lease ran out (the pointer came
back, and a click went to the desktop behind the panel), and an answer slower than EYES_GONE
closed the calibration as if the headset came off.

Checks.ask_eyes sends the command on a socket of its own in ft-gazed's selector, and the answer
(or its deadline, in tick) finishes the dot or the fit. The dot shows its ring full meanwhile;
further clicks do nothing, and a right click doesn't close the panel during the fit.
first-calibration-test.py covers a slow and a silent ft-eyes; the old code fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:30:43 -06:00
DeeJanuzandClaude Opus 5.5 9def5f1762 Gaze calibration panel: a gaze precision or gaze drag press takes the dot
While the panel was up, the helper took only "btn trigger 1" and "gazekey left 1" as
"take this dot". A left button mapped to gaze precision or gaze drag sends "precision|gazedrag
<source> 1" instead, which the panel dropped, so the click did nothing. The panel's answers
move to pointer/helper/calpanel.h, with an offline test (pointer/test/calpanel-test.sh).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:30:43 -06:00
DeeJanuz b521564a68 Merge branch 'report-input' into experimental 2026-10-09 10:03:41 -06:00
DeeJanuzandClaude Opus 5.5 1fdd6f0e78 report.sh: Frametop's keyboard, the Steam menu, and --watch to record a problem
For reports like "windows won't drag while the Steam menu is open" and
"Frametop's keyboard doesn't open", the report now has:

- A section on the keyboard and the Steam menu: the Keyboard setting,
  the relay's devices (a pass-through keyboard keeps ours closed by
  default), ft-textinput and KWin's input method, the desktop's
  QT_IM_MODULE and GTK_IM_MODULE, ft-screens' state and new debug line,
  the SteamVR overlays shown, and the matching log lines.
- --watch [SECONDS]: after the report, it records ft-screens' debug line
  whenever it changes, and the logs, while the user makes it happen.
- A release's version and container (it reported "container dev:
  missing"), the gaze, desktop, eye grabber and Bluetooth units, and the
  newest SteamVR start rather than the log's first.

ft-screens: a "debug" command (Steam menu, Steam in front, our keyboard
shown or aside, mode, lasers, held press and which laser, drags in
progress, where typing goes), and log lines when a drag starts, ends, or
can't start, when Steam comes in front or goes, and when a requested
keyboard doesn't open. The relay logs why a focused text field did or
didn't open the keyboard, once per decision (keys-test covers it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:03:41 -06:00
DeeJanuz 6b7af52e1c Merge branch 'steamos-045-tested' into experimental 2026-10-09 09:31:41 -06:00
DeeJanuzandClaude Opus 5.5 a53ced4f88 SteamOS 0.4.5 is tested (Frametop 0.3.0-exp.4, SteamVR 2.18.2)
Headset tests on 2026-10-09: Bluetooth reconnects, doctor ok, gaze with
our own tracker and SteamOS 0.4's eye-tracker layout, and the desktop
without Steam's autostart and with its own cursor theme. Release
installs stop asking before they install on build 20261007.6125817.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:31:41 -06:00
DeeJanuz 28f91af0d2 Merge branch 'doctor-release-box' into experimental 2026-10-09 09:27:58 -06:00
DeeJanuzandClaude Opus 5.5 e69fe11b2a doctor: in a release, check the release's own container
A release runs in the container its .frametop-release names (BOX, as
scripts/in-box reads it), but doctor.sh looked for the dev build box,
so every release install reported "FAIL container dev" (seen on
0.3.0-exp.4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:27:58 -06:00
DeeJanuz 65acbd9638 Merge branch 'host-build-release' into experimental 2026-10-09 09:26:43 -06:00
DeeJanuzandClaude Opus 5.5 ef1c802e2d Host setup: Frametop's Vibepollo build from its release
The fork's CI published frametop-2.0.0-1 (Vibepollo 2.0.0 with Frametop's
changes, now including the HDR-off guard for SDR display streams). $BuildUrl
points at its sunshine.exe, so a PC needs only host/windows; the first CI
build (a84b6cfc) is upgraded like the hand-built one. Tested on the test PC:
over the first CI build, the setup downloaded the release, matched its
SHA-256, swapped it in and restarted Vibepollo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:26:32 -06:00
DeeJanuz 26de1430b0 Merge branch 'release-notes-channel' into experimental 2026-10-09 09:04:59 -06:00
DeeJanuzandClaude Opus 5.5 e21a2beae0 Release CI: an experimental release's install line says --experimental
get.sh --release asks for a channel and offers stable first, and the
Frametop organization has no stable release yet, so the notes' line
fetched releases/latest and got a 404 unless you picked 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:04:59 -06:00
DeeJanuz 8d6aa459fc Merge branch 'release-notes-url' into experimental 2026-10-09 09:04:07 -06:00
DeeJanuzandClaude Opus 5.5 b93c25eb7e Release notes: get.sh --release from the Frametop organization's Pages
deejanuz.github.io serves DeeJanuz/frametop's main, whose get.sh has no
--release yet ("unknown option: --release"). frametop.github.io serves
Frametop/frametop's experimental, where the releases are built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:04:07 -06:00
DeeJanuz b61aebcf16 Merge branch 'own-one-eye' into experimental 2026-10-09 08:42:41 -06:00
DeeJanuzandClaude Opus 5.5 dba0ef5707 Gaze: with our tracker and one eye tracked, take that eye's blink for both
With Track Dominant Eye Only on, SteamVR judges only that eye's
openness. Our own tracker still finds both pupils, but ft-gazed took the
blinks from SteamVR per eye, so whatever SteamVR reports for the ignored
eye could drop our good reading of it. A blink closes both eyes, so the
tracked eye's now marks both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:42:41 -06:00
DeeJanuz 70c3ad9b1c Merge branch 'steamos-keep' into experimental 2026-10-09 08:36:34 -06:00
DeeJanuzandClaude Opus 5.5 185cb766b1 SteamOS updates: keep the Bluetooth fixes and the eye grabber through them
A SteamOS update deletes every /etc file its keep list
(/usr/lib/rauc/atomic-update-keep.conf) doesn't name. The list keeps
units but not what they run, so after the 0.4.5 update
steamframe-bt-fixups.service and frametop-eyegrab.service failed with
203/EXEC: Bluetooth LE devices stopped reconnecting, and gaze mode fell
back to SteamVR's tracker, which had no calibration.

- Both installers add a drop-in to /etc/atomic-update.conf.d naming
  their file; uninstall removes it.
- update-check.py (doctor.sh) fails when a unit's program is gone and
  warns when it isn't kept.
- ft-gazed's log and Input Settings' Bluetooth page say what to reinstall.
- setup/README no longer says /etc survives updates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:36:34 -06:00
DeeJanuzandClaude Opus 5.5 305dd9bcef Merge branch 'steamos-0.4' into experimental
SteamOS 0.4 (0.4.5) fixes: gaze goes by one eye with Track Dominant Eye
Only; eye-server layouts named by release; update-check covers the new
parts; the desktop no longer autostarts Steam and keeps its own cursor
theme; README note updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:44 -06:00
DeeJanuzandClaude Opus 5.5 90698db6c7 Docs: SteamOS 0.4 is out; one-eye tracking
The README's note said Frametop didn't work on the 0.4 beta. Gaze reads
0.4's eye tracker layout now, and the missing taskbar in #15 wasn't the
beta's doing (0.4.5 has the same KWin and Plasma as 0.3.0), so the note
now says what the update changes and to run doctor.sh after it. The gaze
README covers Track Dominant Eye Only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:03 -06:00
DeeJanuzandClaude Opus 5.5 48a2dcf759 Session: no Steam autostart in the desktop, keep its own cursor theme
The desktop is a KDE session, so it ran /etc/xdg/autostart/steam.desktop.
Steam is already running (the desktop starts from it), so `steam -silent`
only reached that client as a command line it ran (ExecCommandLine in its
console log). SteamOS 0.4 adds -vrdisable -deckard to that entry, for
Desktop Mode, where Plasma starts Steam itself. The session now hides it
with the other two. The marker goes to 2, so desktops that hid the first
two get Steam's copy once, and an entry someone brought back stays.

The Steam client's XCURSOR_THEME=steam reaches the desktop through its
environment, and KWin and the apps prefer it to the desktop's setting.
There was no such theme, so they fell back to Breeze; SteamOS 0.4's
holo-cursors adds one. The session sets the desktop's own theme (Breeze
unless changed in its System Settings) and leaves the size as it was.
ft-screens doesn't draw KWin's cursor, so this shows over remote access
and with the gamescope backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:03 -06:00
DeeJanuzandClaude Opus 5.5 06228725cc SteamOS 0.4: name the eye-server layouts by release, check its new parts
SteamOS 0.4 (0.4.5) is the stable release now, with the eye-server.mmap
layout that the 0.4.3 beta brought (its eye tracker and cv driver differ
from the beta's only in thread priorities and the presence sensor). ft-gaze
and update-check call the layouts "SteamOS 0.3" and "SteamOS 0.4 (+5)"
instead of stable and beta.

update-check also reports when Track Dominant Eye Only is on, warns when
/usr/share/steamos/steamos-cursor.png (the gamescope backend's cursor) is
gone, since SteamOS 0.4's own session moved to /usr/share/holo, and gives
retest hints for steamdeck-kde-presets (its Steam autostart entry changed)
and the new holo-cursors package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:03 -06:00
DeeJanuzandClaude Opus 5.5 d82364ed00 Gaze: go by one eye when SteamVR tracks only the dominant one
SteamOS 0.4 adds Track Dominant Eye Only (SteamVR settings, General,
advanced: steamvr.eyeTrackingDominantEyeOnly with steamvr.dominantEye).
SteamVR's tracker then ignores the other eye. The calibration kept only
samples where it had both eyes, so if that eye reads as lost, no dot
would ever be taken.

gazecal.tracked_eye reads the setting (TrackedEye re-reads it when the
settings file changes). With it on, steady_samples judges the tracked
eye alone (no vergence check), the calibration needs only that eye's
reading and drops the other's and set 2's average, ft-gazed ignores the
other eye (its mmap2 source takes the eye's own reading, and "eyes"
mode needs only that eye calibrated), and the fit check marks the other
eye "not tracked". Nothing changes with the setting off.

Untested in the headset: what SteamVR's mmap reports for the ignored
eye is unknown; gaze/test/one-eye-test.py covers the logic offline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:33:03 -06:00
DeeJanuzandClaude Opus 5.5 eca95477cd Hands: HANDS_MODELS setting picks ft-hands' model folder
Fine-tuned models (the hand dataset's student models) can't ship in the
repo yet, so a headset that has them sets HANDS_MODELS in
~/.config/frametop.conf instead of passing --models to every launcher
(service, ft-cutouts, recorder, probes). --models still overrides it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:44:28 -06:00
DeeJanuz 52b10478ad Merge branch 'stream-in-box' into experimental 2026-10-08 10:45:56 -06:00
DeeJanuzandClaude Opus 5.5 275529da8f Gaze probe: run ft-gaze through scripts/in-box, not the "dev" container
On a release install there is no "dev" container. in-box picks the release's own container,
and starts it first, as container-up.sh did here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 10:45:56 -06:00
DeeJanuz b727b8b8fe Merge branch 'stream-in-box' into experimental 2026-10-08 10:44:20 -06:00
DeeJanuzandClaude Opus 5.5 46bf562c5f Remote displays: ft-layout runs ft-stream in the release's container, not "dev"
On a release install with no "dev" container, distrobox asked whether to create it, and
adding a display hung until it timed out. run_stream now goes through scripts/in-box, as
gazecheck.py does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 10:44:20 -06:00
DeeJanuz 0ffbf7a718 Merge branch 'get-release-menu' into experimental 2026-10-08 09:53:38 -06:00
DeeJanuzandClaude Opus 5.5 2e293beb3a get.sh: the menu offers the experimental release; releases come from the Frametop org
A third choice installs the newest experimental release, built, so an install in the headset
needs only the short command typed on the virtual keyboard, not --release options. Releases
are looked up in Frametop/frametop, where CI builds them (Depot's runners need an
organization); FRAMETOP_REPO still overrides it, and the branches still clone from
DeeJanuz/frametop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:53:38 -06:00
DeeJanuz 4daa8fea5b Merge branch 'five-dot-panel' into experimental
# Conflicts:
#	gaze/README.md
#	gaze/gazecheck.py
2026-10-08 09:40:14 -06:00
DeeJanuz 78fac8ca0a Merge branch 'fix/0x1f6-pr42' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuz 09cec0c980 Merge branch 'standalone-recorder' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuz 325d3d91a4 Merge branch 'hand-probe' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuz 48a3c0af87 Merge branch 'hands-misread' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuz d7cc82ab09 Merge branch 'lighting-doc-fix' into experimental 2026-10-08 09:40:02 -06:00
DeeJanuzandClaude Opus 5.5 a4a1d8819c Gaze: the five-dot check gets its own wider, see-through panel
The quick check's 16 degree square left four of the five dots (12 degrees left and right,
9 up and down) off the panel, unseen. The five-dot check now shows in a 40 degree 4:3 panel,
see-through like quick, with the full calibration's longer notes. gazecheck.py's PANEL_DEG
mirrors the panel sizes and logs any dot that wouldn't fit. ft-gazectl and ft-gazed take
quickcal, calibrate, fitcheck, and fivecheck to open a check from a shell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:40:00 -06:00
DeeJanuzandClaude Opus 5.5 95938a1f27 Merge branch remote-displays (profiles keep remote displays; quick reset opens the profile) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:22:26 -06:00
DeeJanuzandClaude Opus 5.5 9c0facd0c5 Quick reset opens the profile in use, as Open profile does
ft-layout reset reopens the profile in use (the custom arrangement with an
active profile) with use NAME, so its screens, hidden screens, remote
displays and apps all go back as the profile has them. Without one it runs
apply, as before. Meta+Shift+R and the Reset Screen Layout entry
(ft-layout-reset), the input relay's layout_reset action (a mapped mouse or
controller button), and the reset button on a screen's bar now run it.
Display Settings' Arrange now still only arranges.

Before, every quick reset ran apply, which put the desktop screens back but
left the profile's hidden screens, remote displays and apps alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:22:14 -06:00
DeeJanuzandClaude Opus 5.5 eaaab1ccaf Profiles keep remote displays like apps, and connect them when opened
Saving a profile records the remote displays connected then, with their
places and hidden state, the way it records the apps open then. Opening the
profile (use, open, desktop start), or arranging while it's the profile in
use, connects each of them whose host answers on Vibepollo's Web UI port (at
its address or its dongle's, as its route allows) and puts it back where it
was saved. A host that doesn't answer within 1.5 s is skipped, and its
displays stay disconnected. Like apps, displays the profile doesn't have are
left as they are: a profile connects, but never disconnects.

Before, a profile kept the places of every display that had one, connected
or not, and opening it never reconnected a display.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:22:09 -06:00
DeeJanuzandClaude Opus 5.5 84a354536e install-release.sh: don't take "current" for an old release; pack/trial.sh switches the live install
Found in the runtime trial (2026-10-07): the clean-up after an install loops over
releases/*/, which includes the "current" link to the release just installed, so it removed
that release's container and image, and the services it had just started kept failing until
the release was installed again. Links are skipped now.

pack/trial.sh switches this Frame to a release and back: on saves what the release's
install.sh replaces (the frametop-* units with their drop-ins and .wants links, the launcher
and menu entries, the driver's folder and SteamVR's registry, frametop.conf, the desktop's
shortcuts), moves the drop-ins aside, installs from the unpacked zip with
--no-eye-tracker --no-bluetooth, and restarts SteamVR; desktop restarts the VR desktop from
the release; status shows which container each program runs in; off puts everything back,
restarts SteamVR, and stops the release's container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 23:31:37 -06:00
DeeJanuzandClaude Opus 5.5 14cbfc310e FrameDrop window: a keypad for the password, clicked with the controller's laser
Tested in VR with frame-testbench: laser clicks reach the install window (the password field
takes focus, a checkbox toggles), but SteamVR's keyboard doesn't come up for the field, and
opened with steam://open/keyboard its keys never reach the window (the field stayed empty,
read through accessibility). So the window brings its own keypad under each password field:
letters, digits, Shift, symbols, space, and delete, shown when Steam started the window and
toggled with a Keypad button. Checked on a virtual display with pointer clicks and
accessibility. Also: "Frametop 0.3.0-exp.0, built,:" reads "Frametop 0.3.0-exp.0:".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 23:15:05 -06:00
DeeJanuzandClaude Opus 5.5 40c7606ace Merge branch remote-displays (the host setup installs the CI build of Vibepollo) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 22:18:52 -06:00
DeeJanuzandClaude Opus 5.5 00a377a0fb Host setup: Frametop's Vibepollo build from its CI, with WebRTC
The pinned sunshine.exe is now the frametop/2.0.0 build from Frametop/frametop-vibepollo's
CI (a84b6cfc), configured like stock 2.0.0, WebRTC included; the hand-made build it
replaces had WebRTC off. A PC with that older build gets the new one like a stock install
does, and the original exe stays the one kept for -Undo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 22:18:52 -06:00
DeeJanuzandClaude Opus 5.5 8a88fbea05 Release CI: the FrameDrop manifest names this repo's release; get.sh --release takes FRAMETOP_REPO
So a fork's test release (Frametop/frametop, where Depot's runners are set up) points at its
own zip, and get.sh --release can install from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:44:43 -06:00
DeeJanuzandClaude Opus 5.5 9411148596 Releases are one file: Frametop.zip on a GitHub release, built on Depot, no registry
User decision (2026-10-07): people download one file from GitHub, installed through FrameDrop
or unpacked on the headset, holding everything, with no container registry.

Frametop.zip (framedrop/build.sh --image, about 1.1 GB) holds the image as an OCI archive
(podman save), frametop-release.json (pack/release-info.py: version, commit, the archive's
sha256, the image's ID, and the SteamOS table), the install window, and
pack/install-release.sh. That script checks this SteamOS build (against main's steamos.json
when it can fetch it, else the release's copy), checks the archive's sha256, loads it and
checks the ID, copies the release's tree out and runs its install.sh, then keeps the release
before and removes older ones. pack/release-box.sh checks the image's ID before making the
container. The install window runs install-release.sh when the zip has one (get.sh in a test
zip), and reaches systemd through the user's real bus when it's opened from a VR desktop.

get.sh --release now downloads the zip from GitHub (newest stable, --experimental,
--version V, or --zip FILE|URL), resumable and per release, and runs its installer; a
damaged download (exit status 3) is fetched again next time. The registry release list and
its writer are gone.

.github/workflows/release.yml replaces image.yml: on a v* tag or by hand only, on
depot-ubuntu-24.04-arm-4, it builds the image with podman, runs the test gate in it, checks
what a release installs, builds the zip, and makes a draft release (prerelease for a tag
with a "-") with the zip, its FrameDrop manifest, frametop-release.json, and SHA256SUMS.
pack/release-notes.md is the draft's install text.

Tested on the Frame: the release tests (24 checks), a real zip from the local image installed
with get.sh --release --zip --clone-only (sha256, load, ID, copy), a damaged archive (exit 3),
and release-box.sh refusing an image with another ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:34:57 -06:00
DeeJanuzandClaude Opus 5.5 95ccaa30de Merge branch remote-displays (the fork's new address) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:25:15 -06:00
DeeJanuzandClaude Opus 5.5 b655cd7446 Remote displays: the Vibepollo fork moved to the Frametop org
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:25:15 -06:00
DeeJanuzandClaude Opus 5.5 0e506aaf81 Release test: read the old list with Path
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:11:45 -06:00
DeeJanuzandClaude Opus 5.5 fcafd43679 FrameDrop: install a release, and take the password for sudo in the window
The install window now asks first: our own eye tracker (on by default) and the Bluetooth fixes,
which both need sudo, and the SteamOS password for them, checked with sudo -v. A user without
a password (SteamOS starts without one) is told how to set one, and both parts are left out.
Then it starts the install service itself, with the zip's own get.sh, and with
--release --manifest frametop-releases.json when framedrop/build.sh --releases put a release
list in the zip.

The password stays in the window's memory until the install ends. The service gets
SUDO_ASKPASS=askpass, which asks the window over a socket in a 0700 folder in XDG_RUNTIME_DIR;
the window answers only processes in the install's service (peer credentials and cgroup), and
asks again on the headset when it was reopened without it. Never in a file, a log, the
service's environment, or a command line. Tested on the Frame with sudo -A true: a process in
the service gets it, one outside gets nothing, and an ask without it waits for the window.

install.sh and get.sh get --no-eye-tracker, for an unticked eye tracker with the Bluetooth
fixes ticked. pack/README.md describes the runtime and releases, and notes that releases and
the ft wrapper's installed mode pin the image two ways, to merge before this ships.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:10:09 -06:00
DeeJanuzandClaude Opus 5.5 4764c21548 Eye tracker build: the image's small venv names the locked venv's purelib, and a venv without numpy fails the build
site.getsitepackages()[0] of a venv with system site-packages is /usr/local/lib64's, so the
.pth named a folder without numpy, and the build only printed an empty version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:02:20 -06:00
DeeJanuzandClaude Opus 5.5 eae2f0b754 Releases: get.sh --release installs Frametop built, from an image, by SteamOS build
get.sh --release reads a release list (frametop.release/v1: releases/stable.json or
experimental.json on Frametop's page, or --manifest), picks the newest release tested on this
SteamOS build, else the newest not known to break on it (asking first), and refuses a build
every release breaks on (--any-steamos overrides). It pulls the release's image by digest,
copies the image's /src/frametop out to ~/.local/share/frametop/releases/VERSION with a
.frametop-release (version, image, commit, channel, container), and runs that copy's
install.sh. The release before stays, for going back; older ones go, with their containers
and images.

In a release tree (FRAME_RELEASE in _env.sh), install.sh builds nothing: it installs the
distrobox the image brings (pack/build/distrobox) and makes the release's own container from
its image (pack/release-box.sh, named after the digest, so installing a release never stops
the one running). scripts/in-box then runs the programs there. install.sh also gets
--bluetooth, to install the Bluetooth fixes without asking.

pack/steamos.json is the SteamOS table: builds tested, and builds that break releases from
"from" up to "fixed_in". pack/release-manifest.py writes release lists with it, for CI. The
pick logic and the lists are tested offline (pack/test/release-test.py, in the gate).
uninstall.sh removes releases, their containers, and their images in its second step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:00:30 -06:00
DeeJanuzandClaude Opus 5.5 38bdfd856a Merge branch framedrop (the FrameDrop install proof of concept, #25) into fix/0x1f6-pr42
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:55:21 -06:00
DeeJanuzandClaude Opus 5.5 6dd9fb1ad8 One launch switch: scripts/in-box runs a program in the container it was built for
Every place an installed Frametop started a program with `distrobox enter dev --` now goes
through scripts/in-box: the pointer and power units, ft-gazed (ft-gaze, ft-eyes), the gaze
check's panel, the desktop's ft-screens, the remote desktop's krdp and VNC, and the three
settings apps. in-box starts the container in a scope of its own (container-up.sh, which now
takes the container's name) and becomes `distrobox enter BOX --`, so callers behave as before.

BOX is "dev" for a source install. A release names its own container in .frametop-release,
so the same tree runs from an image without editing nine call sites. FRAMETOP_BOX overrides
both. Development tools (frame.sh, the headless test, the gaze probe and lab) and hand
tracking, which install.sh doesn't install, keep using the dev container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:54:13 -06:00
DeeJanuzandClaude Opus 5.5 6dc4521905 Image: remote displays, our eye tracker, distrobox, and SteamVR's library path
The image now builds everything install.sh installs:
- ft-stream (remote displays) with moonlight's build dependencies, its OpenVR from
  scripts/openvr.sh like the other components;
- ft-eyegrab and ft-eyes' build/venv. In the image that venv is a small one that sees the
  locked venv's numpy and OpenCV (uv.lock has requirements.txt's versions), not a copy;
- distrobox 1.8.2.5, by commit, for the release installer to run the image with;
- /opt/steamvr -> /run/host/opt/steamvr, as setup/dev-container.sh makes in the dev
  container, so the programs load SteamVR's libopenvr_api on the Frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:52:27 -06:00
DeeJanuzandClaude Opus 5.5 96452bbee5 Merge experimental (remote displays, gaze report, PR #45 and #46) into fix/0x1f6-pr42
screens/build.sh conflicted: experimental added remote.c to ft-screens, and this branch
links OpenVR through scripts/openvr.sh. Both kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:51:03 -06:00
DeeJanuzandClaude Opus 5.5 b6908737c4 Merge branch remote-displays (other computers' monitors as Frametop screens) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:15:28 -06:00
DeeJanuzandClaude Opus 5.5 f6a9845996 install: build ft-stream and add Frametop Remote Displays
install.sh's desktop step builds ft-stream and installs Remote Displays' menu entry.
uninstall.sh removes the entry and offers to delete the remote displays' pairings and
sign-ins with the other settings. docs/reference.md describes the app and the PC setup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:15:16 -06:00
DeeJanuzandClaude Opus 5.5 050be96bf0 host/windows: Setup Frametop host, the PC side of remote displays
"Setup Frametop host.cmd" runs frametop-host-setup.ps1 as admin. It installs Vibepollo
2.0.0 when it's missing (the release's hash checked), swaps in Frametop's build of its
sunshine.exe (the stock one kept, other versions refused), sets the three settings
Frametop needs (sound from remote displays, and a virtual display that goes when Frametop
disconnects it but survives a dropped stream), keeps or sets the Web UI login, checks the
firewall on every network type and looks for a Steam Link dongle. -Check shows what it
would change, -Undo puts the original exe and settings back. The build's download URL is
empty until Frametop's Vibepollo build has a release: -FrametopBuild takes a file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:15:16 -06:00
DeeJanuzandClaude Opus 5.5 7dfa04aab2 Frametop Remote Displays: the app for remote displays and their computers
remote-displays/ is a Kirigami app (in the dev container, like Display Settings). It finds
Vibepollo computers over mDNS, marks a computer's address on the Frame's hotspot as its
dongle, and signs in with the Web UI login: it keeps a token with only the scopes
Frametop uses and pins the certificate's key, never the password. Each computer's card
shows whether it answers, a Connected switch for all of its displays and its connection
(auto, network or dongle only, once a dongle is known). Its displays (its monitors, or a
virtual one at any size) each have Connected and Shown switches, their stream's
resolution, rate and bitrate, and their width in VR. Display Settings' Screens page opens
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:15:16 -06:00
DeeJanuzandClaude Opus 5.5 2a675b9121 Remote displays: other computers' monitors as Frametop screens
A remote display is a screen of ft-screens whose picture comes from another computer,
streamed by Vibepollo with Moonlight's protocol. ft-screens (screens/remote.c) runs one
ft-stream per remote screen (101 and up) over a socket pair: ft-stream (stream/, GPLv3:
moonlight-common-c and moonlight-embedded's libgamestream, pinned) decodes on the iris
decoder into a ring of three RGBA buffers that ft-screens shows as a panel with a
screen's controls, layout, profiles, curve, pins and hand cutouts. Mouse, keys and scroll
go back to the host; a button's release goes through the stream it was pressed on, as
Vibepollo wants, and a press carried onto another remote panel moves the pointer there.
A panel out of sight gets 10 fps; the host's sound plays from one stream per host.

ft-stream signs in with an API token and the host's pinned public key
(stream/host.cpp), and chooses its path: a Steam Link dongle on the Frame's hotspot when
it answers, or the network (the layout's "route": auto, network or dongle). ft-layout
gets "remote" commands (list, add, connect, disconnect, host route and direct), starts
the connected displays with the desktop and keeps disconnected ones parked. ft-gaze and
ft-pointer know the remote panels. ft-screens now shuts its VR side down before its
streams and waits for SteamVR before exiting (a vrcompositor crash at quit, in standby).

docs/remote-displays.md has the design, the spikes and the test results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:15:16 -06:00
DeeJanuzandClaude Opus 5.5 f0df502a23 Hands: misread guards for fine-tuned landmark models
ft-hands --misread-guard 1 (HANDS_MISREAD_GUARD=1) and ft-handreplay --misread-guard. Fine-tuned
landmark models stay sure of a hand when two hands touch, and can read the held hand as the
other side: the two cameras' views then miss by 3-4 cm, the tracker splits one off as a new
hand, the hand-over puts the old hand back onto it, the duplicate check drops the new one, and
round it goes (a contributed session, holding a card with both hands: 51 duplicates and 40
splits in 59 s with the fine-tuned models, 5 and 9 with the stock ones). With the guard, a
reading whose side is more than 0.7 off its established hand's is dropped, and a split-off view
that sits where its hand already is in that camera is dropped instead of starting a new hand:
21 duplicates and 10 splits, both hands tracked as often. Off by default: the stock model's side
calls are noisier, and the first guard costs it tracking on some recordings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 17:38:34 -06:00
DeeJanuzandClaude Opus 5.5 e402b134ad Merge branch gaze-report (a gaze report that says what looks wrong first) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:51:10 -06:00
DeeJanuzandClaude Opus 5.5 083fa89f52 Report: a gaze report that says what looks wrong first
scripts/gaze-report.py checks what gaze mode and its calibration need and
lists findings before the details: the gaze service's unit, checkout and
builds (missing or older than their sources), the frame grabber installed vs
built, SteamVR's eye tracker (process, eye-server.mmap layout via
update-check.py, eyetracking.txt folded), our tracker's shared files,
calibration and ft-eyes status, the service's status and the pointer
helper's gaze mode, a live check that wakes an idle service for up to 20 s
and counts samples, a summary of checks.jsonl runs with why dots weren't
taken, the gaze settings and mappings, and the gaze lines of each log.
Errors in the service's log since its last start become findings too.

scripts/report.sh runs it in place of its own gaze sections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:51:08 -06:00
DeeJanuzandClaude Opus 5.5 eb13cfc5dc Merge branch fix/dkiiv-pr45 (our fixes for PR #45) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:30:16 -06:00
DeeJanuzandClaude Opus 5.5 c40bb645f2 doctor: checks that don't need the repo work before the first sync
From a PC, every check ran in the Frame's copy of the repo, which the first sync creates.
Before that, each one failed on its cd or carried on past it: SteamOS and the taskbar
check said ok with a cd error, and distrobox, the container, and free space failed (#44).
They now run in the home folder, and the taskbar check waits for the repo, with a line
saying what copies it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:30:09 -06:00
DeeJanuzandClaude Opus 5.5 195fb1e441 sync: make the Frame's repo dir in rsync's own connection
The separate ssh mkdir meant a second SSH login for every sync, and every build and
install step from a PC syncs first. rsync's --rsync-path runs the mkdir before the remote
rsync, in the same connection, and still works with rsync older than 3.2.3 on the PC. The
mkdir now uses the same home-relative path as the rsync destination.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:29:21 -06:00
DeeJanuzandClaude Opus 5.5 f6fe1455fc Merge PR #46 from SaberMage/fix/chrome-hitbox: ft-screens: Make the hit area of each control equal to its texture size
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:21:02 -06:00
DeeJanuzandClaude Opus 5.5 2052dff70d Merge PR #45 from dkiiv/fix/sync-mkdir-parent: sync: create the Frame's repo dir before the first rsync
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:21:02 -06:00
SaberMageandClaude Opus 5.5 0d7ed2d75c ft-screens: the controls take hits only where they're drawn
SteamVR's laser and ComputeOverlayIntersection size an overlay's hit area
from its mouse scale, not its texture. At the default 1x1 every control
was hit as a square as tall as it is wide, so the grab bar (a 256x24
texture) caught clicks meant for the bottom tenth or so of the screen
above it.

MakeChrome now sets each control's mouse scale to its texture size. The
controls only read button and scroll events, never the mouse position,
so nothing else changes. Measured on the Frame (SteamOS 0.3.0 build
20260922) with a 0.2 m wide 256x24 overlay: a hit area 199 mm tall at
the default scale, 18 mm at 256x24 (the bar is drawn 18.8 mm tall); an
intersection mask doesn't change it. The pointer helper's measure command
gave the live grab bars square hit areas (0.197 x 0.197 m, 0.212 x
0.212 m) before the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019u6dFg7wXBokn1TWD7Cg6X
2026-10-06 23:33:10 -07:00
dkiiv a94af27131 sync: create the Frame's repo dir before the first rsync
rsync creates only the last component of the destination path, and a
fresh Frame has no ~/dev, so the first sync from a PC always failed:

  rsync: [Receiver] mkdir "/home/steamos/dev/frametop" failed: No such file or directory

Create FRAME_REPO over SSH before rsyncing. Fixes #44.
2026-10-07 01:02:39 +00:00
DeeJanuzandClaude Opus 5.5 d4558c7187 Hand recorder docs: hand contrast isn't a daylight test
PR #6, recorded at night in a pale room, had hands nearly as faint against the room (1.22x) as
PR #5's sunlit round (1.15x). Review decides daylight from the pictures (sunlit windows, the
time of day); the contrast is supporting evidence only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:17:51 -06:00
DeeJanuzandClaude Opus 5.5 221d01db32 Merge experimental (first-calibration-test race fix) into fix/0x1f6-pr42
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:34:56 -06:00
DeeJanuzandClaude Opus 5.5 282b038a3c Merge branch gaze-test-race (first-calibration-test no longer races ft-eyes against the fake ft-gaze) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:34:52 -06:00
DeeJanuzandClaude Opus 5.5 63bcea49a6 Gaze test: wait for the fake ft-gaze's first sample before the quick-check refusal
first-calibration-test.py failed about 1 run in 4, on main as well as
experimental: ft-eyes' "not calibrated" can arrive before the fake
ft-gaze has sent a sample, and with no eyes seen yet start() refuses the
quick check with "the headset is off" before it gets to "not calibrated".
The test now waits for the eyes to be seen first. 8 of 8 runs pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:34:52 -06:00
DeeJanuzandClaude Opus 5.5 9b11162b61 Merge branch lighting-prompt (the recorder asks for Daylight when sunlight comes in) into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:26:42 -06:00
DeeJanuzandClaude Opus 5.5 e2f51477a3 Hand recorder: ask for Daylight when sunlight comes in; the cameras can't tell it
The first daylight round (dataset PR #5, a room with big sunlit windows) read 2.38 ambient IR
and was labelled indoor: the windows are a small part of each picture, so the dark frames' mean
barely moves. The checklist no longer says the cameras tell daylight themselves, and the docs
say how review corrects the label (hub_review.py lighting) from how much hands stand out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:19:30 -06:00
DeeJanuzandClaude Opus 5.5 525c77e60c Tests in the image get the host's datagram queue length (512, not 10)
A container's own network namespace starts with net.unix.max_dgram_qlen
at the kernel's 10; systemd sets 512 on the host. The input relay sends
without blocking and drops what a full queue refuses, so in the image
keys-test.py saw a starting relay's release burst lose its last five
mouse-button releases. ft runs and CI's test step now set 512 and keep
their own namespace, so tests can't collide with a live desktop's
sockets. pack/design.md adds the queue length to what the runtime on the
Frame must keep (the host namespace has it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:06:09 -06:00
DeeJanuzandClaude Opus 5.5 64d3a9eb9c CI smoke: test this run's image, through docker and a registry in the job
The runner has podman as well as docker, so ft picked podman, which
pulled the last image published to GHCR: the wrapper smoke tested that
image, not the one the run built (on the fork, ghcr.io/0x1f6/frametop:
pack-frametop-image). On a branch nothing is in GHCR, so ft update failed.
FT_ENGINE now picks the engine, the job sets it to docker, and ft update
pulls this run's image from a registry started inside the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:58:34 -06:00
DeeJanuzandClaude Opus 5.5 d1430d20e8 Docs: what the image does now, the open runtime question, and the plan
The README and AGENTS.md no longer say Frametop can run from the image
or that the units become one-liners: no installer uses it yet. AGENTS.md
keeps the rules that hold now (pin every input, one build recipe, tests
green in the image) and drops FT_FRAME. pack/README.md fixes the Python
and OpenVR descriptions and the CI section, and ends with the plan: a
headset trial of the runtime, a release that builds the image and the
host payload from one commit, then get.sh and FrameDrop installing it.
pack/design.md keeps the device findings for OpenVR clients, lists what
each program needs from the host, and compares a distrobox from the image
with Quadlet units.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:57:20 -06:00
DeeJanuzandClaude Opus 5.5 6825f80096 .dockerignore: patterns at any depth, and .worktrees
Unlike .gitignore, .dockerignore patterns only match at the top, so a
local build sent every nested build/ and __pycache__/, any stray eye-camera
frame dumps (*.raw, *.pgm, biometric per .gitignore), and all of
.worktrees/ into the build context. CI builds from a clean checkout, so
published images weren't affected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:54:17 -06:00
DeeJanuzandClaude Opus 5.5 8086350e06 ft: no Frame runtime mode yet; unique container names; no implicit :latest
FT_FRAME=1 mounted too little for the programs to work: no /dev/dri,
/dev/input, writable /sys, host groups, home config, or XDG_RUNTIME_DIR.
How they run on the Frame is an open decision (pack/design.md keeps the
device findings). Runs now get no host access beyond the repo mount.

- Containers are named frametop-<program>-<pid>: with one name per
  program, a second python3 run removed the first.
- A reference with neither tag nor digest is :latest too, and is refused.
- ft update writes the pin whole or not at all.
- The help no longer says update refreshes the wrapper (deferred).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:54:17 -06:00
DeeJanuzandClaude Opus 5.5 fb6ef934a4 Image: built by the build scripts, on Fedora's python3, every download checked
The binaries come from screens/, pointer/, gaze/, and power/build.sh
(FRAME_IN_BOX=1), so the image and the Frame build them one way.

The venv now sits on Fedora's python3 with the system site-packages.
With uv's own CPython 3.12 first on PATH, python3 couldn't import the dnf
PySide6 (built for Fedora's 3.14), so the settings apps and the hand
recorder couldn't start, and their tests only skipped. The base image is
pinned by digest, so its python3 is frozen too. The layer checks that
PySide6, numpy, and cv2 import together.

uv and libopenvr_api.so are checked against pinned sha256 sums; the
headers and stb_truetype come through the build scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:54:08 -06:00
DeeJanuzandClaude Opus 5.5 54ec0853aa Build scripts: one pinned OpenVR recipe, and they run inside the image too
scripts/openvr.sh is the one place for the OpenVR SDK the programs build
against: the v2.15.6 headers, fetched into build/include and checked by
sha256, and the API library. ft-pointer, the ft_pointer driver, and
ft-powerd now build against those headers like ft-screens and ft-gaze,
instead of the 2.1.0 copy in SteamVR's samples, which has no public tag
an image could pin. The library stays SteamVR's own; OPENVR_LIB links
another copy (the image has no SteamVR), and the rpath still names
SteamVR's folder first.

FRAME_IN_BOX=1 tells the scripts they already run in the build container,
so pack/Containerfile can run them instead of keeping a second copy of
every compile line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:54:08 -06:00
DeeJanuzandClaude Opus 5.5 8fc8cf2063 just test: a failing Python suite fails the gate; relay-buttons-test joins test-c
python3 "$t" && echo inside the for loop slipped past set -e, so a
failing input, hands, or gaze suite still passed the gate. Each suite now
reports ok or FAILED and any failure fails the run. test-python first
checks that python3 imports PySide6, numpy, and cv2 together: in the
image's first build it couldn't import PySide6, and the Qt tests only
skipped. test_fix_panels.py ran twice (directly and under pytest); pytest
keeps it. relay-buttons-test.c (from experimental) runs with the other
header-only C test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:52:00 -06:00
DeeJanuzandClaude Opus 5.5 b15dce6a0a CI: a lowercase image name, so the workflow runs under DeeJanuz/frametop
github.repository keeps the owner's capitals, and image names must be
lowercase: under DeeJanuz every run failed at the first docker run. The
workflow_dispatch choice of a self-hosted runner goes too (that runner
and its setup script exist only on the fork), and scripts/** now
triggers a build, since the build scripts the image runs live there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:51:35 -06:00
DeeJanuzandClaude Opus 5.5 b52df75bc0 Merge branch experimental into fix/0x1f6-pr42 (PR #42 on top of current experimental)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:48:52 -06:00
DeeJanuzandClaude Opus 5.5 0318bb72ab ft-handtest --probe: point at the analysis script's place
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:19:58 -06:00
DeeJanuzandClaude Opus 5.5 3519a7a0e1 ft-handtest --probe: dots where the cutouts would land, for measuring them against Room View
A see-through test panel with dots on the wrist, middle knuckle and fingertips of each
tracked hand, one colour per timing: where the cameras saw the hand, moved ahead to now,
and moved ahead to now + the cutouts' lead. Recorded together with the headset view, they
show how far the cutouts land from the hand Room View shows, still and moving, and which
lead fits. Each tick goes to a JSON-lines log. Renderer::Marks draws the dots; Hands::Points
gives the landmarks. ft-screens' cutouts are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:04:52 -06:00
DeeJanuzandClaude Opus 5.5 8117a4fb7e Merge branch hand-perf (hand cutouts without waiting for the GPU) into hand-probe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:00:31 -06:00
0x1f6 18e436d7b4 CI: publish (ghcr push + ft artifact) only from main and version tags
Feature branches are validated, not published — the registry holds only
states a user would actually run. Branch validation still covers the
full build/test/smoke chain against the local image.
2026-10-06 02:30:12 +02:00
0x1f6 bb88599085 CI: a newer push to the same ref cancels the superseded run 2026-10-06 02:28:54 +02:00
0x1f6 302df6c2da Image: ~900 MB lighter — uv cache, langpacks, docs, wallpapers, mypy out
Measured on the current image (3.29 GB): /root/.cache/uv held 266 MB
after sync (--no-cache now); glibc-langpack-* plus their /usr/lib and
/usr/share locale data ~460 MB (only en stays); /usr/share doc/man/info
+ wallpapers ~150 MB. mypy moved from the dev group to a types group
that the image does not install (uv sync --frozen --group tracker);
pytest and ruff stay because CI tests inside the product environment.
uv.lock relocked for the group change.
2026-10-06 02:23:39 +02:00
0x1f6 8298754437 Image: drop git; test-bash lists scripts with find, not git
The runtime image needs no git: builds use the copied sources, the
installed mode has no repo at all, and report.sh falls back gracefully
outside a checkout. test-bash now lists shipped shell scripts with find
— git ls-files broke silently in CI (dubious ownership on a mounted
checkout, for-loop word list swallowed the failure) and only tested ft.
A count guard fails the gate on a suspiciously short list instead.
2026-10-06 02:18:44 +02:00
0x1f6 d6f6f0fe7f CI: upload-artifact v4.6.2 → v7.0.1 (Node 24 native, no deprecation warning) 2026-10-06 02:14:09 +02:00
0x1f6 491202acfc CI: one job, five steps — the image is built once and never re-pulled
Separate jobs meant every stage re-pulled the just-pushed image from
GHCR (~1.5 GB each). One runner keeps it in the local Docker cache:
build (no push) → test → smoke → push → artifact. The registry only
ever receives a green image, and the artifact's image-digest.txt names
a digest that exists because the push preceded it.
2026-10-06 02:13:26 +02:00
0x1f6 b234abc245 CI: smoke stops rerunning the suites; just test delegates to the strict recipes
The smoke job ran 'just test' beside the new test job — the same suites
twice, once lenient, once strict. Smoke now checks only the artifact
(binaries, venv) and the wrapper paths; 'just test' is a thin alias for
test-python test-c test-bash, so there is exactly one definition of
what passing means.
2026-10-06 02:08:24 +02:00
0x1f6 f8f33c5663 CI: publish the ft wrapper as a checksummed artifact
An artifact job (needs build+test+smoke) extracts /src/frametop/ft from
the image the run built, verifies it byte-for-byte against the checkout,
and uploads it with its sha256 and the image's RepoDigest. The artifact,
the image, and the commit are thereby provably one thing — and install.sh
can later verify the wrapper it installs against ft.sha256.
2026-10-06 02:06:10 +02:00
0x1f6 b33c5df41f CI smoke: capture the published reference before unsetting FT_IMAGE
The unset landed before the sandbox published-file was written, so
update pulled the empty reference (podman: 'repository name must have
at least one component', exit 125). The wrapper was never at fault.
2026-10-06 02:00:39 +02:00
0x1f6 06288ff20c ft: defer the wrapper-from-image refresh and the baked update default
The hard wrapper/image pairing broke the smoke job under Docker (exit
125, 'repository name must have at least one component' right after the
digest pin). Deferred until the smoke job can say exactly where it
fails; ft update stays the simple, locally verified pull + digest pin,
and the published reference must be configured explicitly again (no
silent fallback). The smoke job drops the pair cmp accordingly.
2026-10-06 01:54:20 +02:00
0x1f6 7854d6ab9d CI: a strict test gate (test-python, test-c, test-bash) beside the smoke job
just test-python runs every Python suite strictly (hands and gaze
included — all pass off-device); test-c compiles the controller-click
unit test with -Werror in the image (sides_test stays with the hands
ncnn build, documented); test-bash syntax-gates every shipped shell
script. The new test job runs inside the image the build pushed, next
to smoke: smoke asks if the artifact is sound, test asks if the code
is right. controller-click-test.c #undefs NDEBUG so a release build
still tests. hands/** joins the CI trigger paths.
2026-10-06 01:48:34 +02:00
DeeJanuzandClaude Opus 5.5 891d84ae2b Hands: name the side cameras as XRService does, and bind their buffers exactly
The side cameras came out swapped on most SteamVR starts, and right on some.
Two guesses combined:

- ft-hands and camcheck.py named each video device by its TrackingCameraInit
  index (0 = slam_left). The index is only the order XRService opens the
  cameras in: on every start logged since 2026-10-04, index 0 was slam_right.
  XRService names each sensor subdev itself ("Found camera 'slam_left': ...
  v4l_subdev=/dev/v4l-subdev30"), and each TrackingCameraInit line says which
  subdev it opened. Name the devices by that; the index order stays the
  fallback for a log without those lines. The capture-pipe fallback had vfe3
  and vfe4 backwards too.

- ft-camd bound each run of XRService's buffers to a device by the sensor
  subdev opened just before it in XRService's fd table. That order changes
  when XRService restarts its cameras (17:09 today: the run after
  og01a1b 4-0060's subdev was video9's). VIDIOC_QUERYBUF on the device names
  the descriptor XRService queued at each index, from any handle, so bind runs
  by that, and split a run holding two cameras' buffers (the upper pair and the
  colour pair came as 32-buffer runs). The order stays the fallback.

Checked on the live XRService: all four tracking cameras "bound by
VIDIOC_QUERYBUF" with 16 buffers each, and ft-hands now reads
slam_left=video13 slam_right=video9. ft-hands' hands-based side check stays
as the safety net.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:22 -06:00
0x1f6 434b4aadc9 CI smoke: unset FT_IMAGE in the installed-mode section — the override shadowed the pin, so the digest and :latest checks tested nothing 2026-10-06 01:43:46 +02:00
0x1f6 ed6caf6ff0 ft update: refresh the installed wrapper from the image; CI verifies the pair
ft update now also copies /src/frametop/ft out of the pulled (digest-
pinned) image over the installed wrapper, backing the old copy up as
ft.previous — wrapper and image are one artifact, built from the same
commit by CI, and they cannot drift apart on a device. The update
source falls back to the channel baked into the wrapper
(FT_DEFAULT_PUBLISHED) instead of failing when install.sh has not
written a config yet.

CI smoke: after the sandbox update, the refreshed wrapper must be
byte-identical to the checkout's ft — the pair test.
2026-10-06 01:40:21 +02:00
0x1f6 5fa557afbb CI: the smoke job exercises the ft wrapper; wrapper changes trigger CI
The wrapper is the counterpart of the image — every device run goes
through it. The smoke job now runs it against the image the build
pushed: repo-mode program run, installed-mode update with digest-pin
verification, and the :latest refusal. ft joins the CI trigger paths.
2026-10-06 01:37:42 +02:00
0x1f6 a0a5468c59 ft: the Frame mount matrix, validated on device for ft-powerd
Device findings (SteamOS 0.4.3, SteamVR 2.18.2, containerized ft-powerd
connected to the running vrserver, verified in vrserver.txt):

- OpenVR's path registry (~/.config/openvr) must be visible at both the
  container HOME and the absolute /home/steamos paths it references;
  the referenced dirs (Steam config/logs, the ft_pointer driver dir)
  are mounted at the same paths.
- SteamVR's IPC control file lives in /tmp: a private /tmp namespace
  makes VR_Init fail with Init_Internal (124).
- Rootless podman maps container root to the desktop user, so a plain
  run writes host files as steamos and needs no --user; forcing
  --user 1000 breaks the subuid mapping instead.
- HOME must be set explicitly: podman derives it from the workdir,
  which is /src/frametop.
- ft-powerd binds the abstract socket @ft_powerd: second instances
  refuse cleanly (single-instance by interface, not by accident).
2026-10-06 01:34:35 +02:00
0x1f6 e984418b25 CI: merge the doc exclusion into paths (paths-ignore cannot combine with paths) 2026-10-06 01:28:29 +02:00
0x1f6 77f7cc82be CI: doc-only changes no longer build the image 2026-10-06 01:25:12 +02:00
0x1f6 897488ff02 ft: :latest stays legal in development, illegal only for deployments
The refusal applies to the pinned image and the update source — the two
references a headset depends on. Repo mode never second-guesses FT_IMAGE,
so a developer can run :latest locally if they want. Wording in the
wrapper header and pack/README aligned ("No :latest on a headset").
2026-10-06 01:21:41 +02:00
0x1f6 17f3a2c55c ft: digest pinning instead of :latest; ft clean for the shared podman store
On a Frame, frametop's rootless podman store is shared with Valve's
lepton-* containers. ft clean removes only frametop-* containers and
frametop-named images; store-wide podman commands are now explicitly
documented as off-limits.

Installed mode refuses to run without a pinned reference, and :latest is
rejected for both the pinned image and the update source. ft update pins
the pulled digest to ~/.config/frametop/image, so bug reports name an
exact image and rollback is a one-file edit. The published reference is
version-tag-only and must be configured (install.sh will write it).

pack/README: shared-store rules, digest-pinning workflow, storage note
(the image replaces the toolchain, net smaller), network path for pulls;
GHCR visibility flip documented. design.md: tagging open decision
narrowed to cadence.
2026-10-06 01:18:56 +02:00
0x1f6 72e5ad4489 ft: dev subcommand and stable container names; packaging docs
ft gains the dev/runtime split: build, shell, and test live behind
'ft dev' so an installed copy refuses them, programs run under stable
container names (frametop-<program>, leftovers replaced), and the
macOS bash 3.2 empty-array edge is handled.

pack/design.md is the packaging rationale — the image as the product,
what it solves, why OCI and not uv alone or Flatpak, how the wrapper,
tests, and the Frame runtime fit in, and the open decisions. The
README gets a short Packaging section pointing there, AGENTS.md gets
the working rules for pack/ (pin every input, ft as the single
integration point, build and test through the image, host
dependencies explicit), and pack/README.md follows the ft dev
renaming.
2026-10-06 01:11:04 +02:00
0x1f6 e88ae95c0c ft: two homes — repo checkout and installed (~/.local/bin)
Repo mode (pack/Containerfile beside the script) keeps building and testing
from the sources and mounts them at /src/frametop. Installed mode (the copy
install.sh will put in ~/.local/bin, since the SteamOS root is read-only) runs
the image's own copy: no repo on the device needed. The image reference
resolves  > ~/.config/frametop/image (written by install.sh, so
installs pin what was installed) > the published image in installed mode,
the local build in repo mode. update always pulls the published image.
2026-10-06 01:11:04 +02:00
0x1f6 2ce81c553a actions: bump the docker actions to their node-24 majors (checkout v7, setup-buildx v4, login v4, metadata v6, build-push v7) — silences the runner's node-20 deprecation 2026-10-06 01:11:04 +02:00
0x1f6 332633c919 smoke: pull the tag the build pushed and log in first (fresh GHCR packages are private) 2026-10-06 01:11:04 +02:00
0x1f6 d2a9e2bbc9 CI: point buildx at pack/Containerfile (it defaults to ./Dockerfile) 2026-10-06 01:11:04 +02:00
0x1f6 318a4a1682 runs-on: the env context is not allowed there — inline the runner choice 2026-10-06 01:11:04 +02:00
0x1f6 60095b02bf workflow_dispatch with a local-runner option
The cloud arm64 runner stays the default on push. For on-demand testing while
it sits in the queue, workflow_dispatch takes a 'local' choice: both jobs then
run on a self-hosted runner labeled ft-arm64 — an ephemeral actions-runner
container in OrbStack (native arm64, same architecture the Frame runs), set up
by a local script that stays out of the repo. The runner fetches a single-use
registration token per start; the PAT stays in the macOS Keychain and only
ever scopes this one repository. The workflow triggers on push and
workflow_dispatch only, never pull_request — that is what keeps a self-hosted
runner safe on a public repo.
2026-10-06 01:11:04 +02:00
0x1f6 ea47422872 Frametop as an OCI image: pack/Containerfile, the ft wrapper, just, CI
The container becomes an artifact instead of a recipe: the base toolbox image
is pinned by digest, Python dependencies come from the committed uv.lock
(uv-managed CPython, clean venv at /opt/frametop/venv, no system-site-packages),
and OpenVR's header and libopenvr_api come from the same pinned public tag the
Frame-side build scripts pin. The native binaries compile inside the image the
same way the build.sh scripts compile them; hand tracking stays deferred as in
install.sh.

- ft: one wrapper for everything that runs (build, shell, run, test, update);
  Frame mounts (ipc=host, /run/user, /opt/steamvr) are designed behind
  FT_FRAME=1 until validated on the device (pack/README.md)
- justfile: runner for the in-image actions (test, lint as report, check)
- CI: native arm64 runner builds and pushes ghcr.io/0x1f6/frametop and runs
  the suites inside the built image as a smoke job; actions pinned to SHAs

Validated locally (docker, aarch64): full build, all six binaries, and
./ft test green (unittest suites, check scripts, 11 pytest tests).
2026-10-06 01:11:04 +02:00
DeeJanuzandClaude Opus 5.5 06f2dd63c1 FrameDrop: proof of concept for installing Frametop from FrameDrop (#25)
FrameDrop sideloads a Linux zip as a Steam Devkit Game. Frametop can't be
copied over as an app, so the zip holds a small installer: playing it runs
get.sh --yes in a transient user service (Steam reaps the title's process
tree on quit) and shows progress in a GTK window.

- probe/probe.sh records what a Devkit Game can reach. On SteamOS 0.3.0 it
  runs on the host with git, podman, systemd and GTK, even with the SLR4
  compat tool set; inside SLR4, flatpak-spawn --host reaches the host.
- devkit.sh registers a folder the way FrameDrop does (Valve's devkit-utils),
  for testing without a PC.
- build.sh makes a reproducible Frametop.zip and the FrameDrop manifest.

Nothing is published. Open: whether FrameDrop keeps the exec bit, and its
start command and runtime choice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 21:19:30 -06:00
DeeJanuzandClaude Opus 5.5 64c4eec59f Screens: hand cutouts without waiting for the GPU
The cutout composite drew each screen's whole client buffer into a side-by-side buffer on
every tick a hand was in front of it, then waited for the GPU with glFinish (1-6 ms, the
likely cause of the VR frame drops on 2026-10-01). Now:

- A drawn buffer gets a fence and is shown from a later tick once the fence has passed, so
  ft-screens never waits for the GPU (except for a panel's first buffer after a pause, so a
  stale one never shows). The prediction lead goes from 25 to 36 ms for that tick.
- Nothing is drawn when the client frame and the cutouts haven't changed, and SteamVR
  isn't handed the same buffer again.
- When only the cutouts moved, a buffer that holds the same client frame is drawn again
  only around the old and new cutouts (scissored).
- "cutouts state" reports draws, partial draws, unchanged ticks, busy ticks, waits and CPU
  time per second since the last state; ft-handtest prints the same counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 10:14:54 -06:00
132 changed files with 12419 additions and 431 deletions

No files matched your search

+18
View File
@@ -0,0 +1,18 @@
# Keep the build context small: what .gitignore ignores, and regenerated files.
# Unlike .gitignore, patterns here only match at the top unless they start
# with **/, so the ones that can appear in any folder do.
.git/
.worktrees/
.venv/
**/build/
**/target/
**/captures/
**/__pycache__/
**/*.pyc
# Eye-camera frame dumps are biometric (see .gitignore): never in an image.
**/*.raw
**/*.pgm
**/.env
**/.env.*
frametop-report-*.txt
.frame-job.d/
+99
View File
@@ -0,0 +1,99 @@
# Build Frametop's release on Depot's arm64 runners (the Frame is aarch64): the image from
# pack/Containerfile, the test gate inside it, then Frametop.zip (framedrop/build.sh), the
# image with its installer: FrameDrop installs it from a PC, or you unpack it on the headset
# and run Frametop/frametop-install.sh, or get.sh --release downloads it (pack/README.md,
# Releases). It's about 1.1 GB, under GitHub's 2 GB a file.
#
# A v* tag makes a draft GitHub release with the zip, its FrameDrop manifest, its
# frametop-release.json, and SHA256SUMS (a prerelease when the tag has a "-", like
# v0.3.0-exp.1). Nothing is public until someone publishes the draft. A manual run keeps the
# zip as the run's artifact for a week.
#
# podman, as on the Frame: the zip holds podman save's archive, which the headset loads with
# podman. Not on pushes or pull requests: Depot's runners are paid, and a fork's pull request
# would run on them. Actions are pinned to commit SHAs (the version in the comment).
name: release
on:
push:
tags: ["v*"]
workflow_dispatch:
permissions:
contents: write # the draft release
concurrency:
group: release-${{ github.ref }}
jobs:
release:
runs-on: depot-ubuntu-24.04-arm-4
timeout-minutes: 90
env:
FT_ENGINE: podman
steps:
# actions/checkout@v7.0.1
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
- name: version
run: |
if [ "$GITHUB_REF_TYPE" = tag ]; then v=${GITHUB_REF_NAME#v}; else v=0.0.0-ci.${GITHUB_SHA::7}; fi
echo "VERSION=$v" >> "$GITHUB_ENV"
echo "FT_IMAGE=localhost/frametop:$v" >> "$GITHUB_ENV"
- name: build the image
run: ./ft dev build
# The unit gate: Python (strict), C, and shell, inside the image, so what passes here is
# what the zip installs.
- name: test (python, c, bash)
run: ./ft dev test
- name: smoke (what a release installs from the image)
run: |
podman run --rm --entrypoint sh "$FT_IMAGE" -c '
set -e
for p in ft-screens ft-pointer ft-powerd ft-gaze ft-gazepanel ft-stream ft-eyegrab; do
test -x /opt/frametop/bin/$p
done
test -f /src/frametop/pointer/driver/build/driver_ft_pointer.so
test -x /src/frametop/pack/build/distrobox/install
/src/frametop/gaze/tracker/build/venv/bin/python -c "import numpy, cv2"
python3 -c "import PySide6"'
# The FrameDrop manifest names the zip's URL in this repo's release (a fork's, in a fork).
- name: Frametop.zip
run: |
url=()
[ "$GITHUB_REF_TYPE" = tag ] &&
url=("https://github.com/$GITHUB_REPOSITORY/releases/download/$GITHUB_REF_NAME/Frametop.zip")
framedrop/build.sh --image "$FT_IMAGE" --version "$VERSION" --commit "$GITHUB_SHA" "${url[@]}"
- name: keep the zip (manual runs)
if: github.ref_type != 'tag'
# actions/upload-artifact@v7.0.2
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
with:
name: Frametop-${{ env.VERSION }}
path: framedrop/build/
retention-days: 7
- name: draft release (tags)
if: github.ref_type == 'tag'
env:
GH_TOKEN: ${{ github.token }}
run: |
pre=() notes=pack/release-notes.md
if [[ $VERSION == *-* ]]; then
pre=(--prerelease)
# get.sh --release asks for a channel and offers stable first: an experimental
# release's install line names its channel.
notes=$RUNNER_TEMP/release-notes.md
sed 's/bash -s -- --release`/bash -s -- --release --experimental`/' pack/release-notes.md >"$notes"
grep -q -- '--release --experimental`' "$notes" ||
echo "::warning::pack/release-notes.md has no get.sh --release line to mark experimental"
fi
gh release create "$GITHUB_REF_NAME" --draft --verify-tag "${pre[@]}" \
--title "Frametop $VERSION" --notes-file "$notes" --generate-notes \
framedrop/build/Frametop.zip framedrop/build/frametop.framedrop.json \
framedrop/build/frametop-release.json framedrop/build/SHA256SUMS
+1
View File
@@ -0,0 +1 @@
3.14
+8
View File
@@ -33,6 +33,14 @@ A Steam Frame is someone's personal headset, and they may be wearing it while yo
A SteamOS update replaces SteamVR, KWin, and gamescope with the rest of the OS image. When a change starts depending on something from the image (a host file, an OpenVR interface outside the bundled header, an undocumented layout or output format, a SteamVR or KWin quirk), add a check for it to `scripts/update-check.py`, or a retest hint for its package there. [docs/design.md](docs/design.md) has the background.
## The image (`pack/`)
`pack/Containerfile` builds Frametop as an OCI image, the groundwork for installing without building on the headset; no installer uses it yet. [pack/design.md](pack/design.md) explains why and what is open, and [pack/README.md](pack/README.md) documents the specifics. Rules:
- **Pin every input.** Base images by digest, Python via `uv.lock` (commit the lock, never a bare `uv pip install`), downloads by tag or commit and sha256, CI actions by commit SHA.
- **One build recipe.** The image runs the components' own `build.sh` scripts (`FRAME_IN_BOX=1`), and the OpenVR SDK they build against is pinned in `scripts/openvr.sh`. Change a build there, not in the Containerfile, and keep the Containerfile buildable on arm64, the only architecture the Frame has.
- **Keep the tests green in the image.** `./ft dev test` runs `just test` inside it, and CI runs the same recipes. A new offline test that needs no headset goes in the `justfile` too.
## Names
User-facing names are "Frametop", "Frametop Display Settings", and "Frametop Input Settings". Programs and files use the `ft-` / `ft_` prefix (`ft-screens`, `ft-pointer`, `ft-layout`, the `ft_pointer` driver); config, units, and overlay keys use `frametop`. Program names must stay within 15 characters: Linux truncates process names there, and the scripts find programs with `pgrep -x` / `pkill -x`.
+20 -8
View File
@@ -13,11 +13,13 @@ Two settings apps come with it: Frametop Display Settings for the screens, profi
Frametop is an independent project, not made by or affiliated with Valve.
Frametop lives at [Frametop/frametop](https://github.com/Frametop/frametop): code, releases, issues, and pull requests. It moved there from DeeJanuz/frametop on 2026-10-09; the old links and clones still work.
Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions, ideas, and help with your setup.
## Install on the headset
> **Frametop doesn't work on the SteamOS beta right now.** On the beta (SteamOS 0.4.3), gaze mode can't read the eye tracker, and the desktop has started without its taskbar ([#15](https://github.com/DeeJanuz/frametop/issues/15)). Use the stable SteamOS release until this note is gone.
> **SteamOS 0.4:** SteamOS 0.4 moved the eye tracker's data that gaze mode reads. This version of Frametop reads both SteamOS 0.3's and 0.4's, and it's tested on 0.4.5. Run `scripts/doctor.sh` after the update: it says whether the eye tracker's layout is one Frametop knows. It also says whether the update deleted the Bluetooth fixes or our eye tracker's frame grabber, which happens when they were installed by Frametop 0.3.0-exp.3 or older. Reinstall what it names (`setup/bluetooth/install.sh install`, `gaze/tracker/install.sh`). From then on they're kept through updates.
You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space.
@@ -26,10 +28,12 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
3. Run:
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
curl -fsSL https://frametop.github.io/frametop/get.sh | bash
```
It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/DeeJanuz/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/Frametop/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.1 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
@@ -133,7 +137,7 @@ With the displays off, the headset keeps tracking and rendering, so it uses abou
## Known limitations
This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 20260922, SteamVR 2.17.10).
This is an early release, tested on one Steam Frame (SteamOS 0.4.5 build 20261007, SteamVR 2.18.2; before that SteamOS 0.3.0 build 20260922, SteamVR 2.17.10).
- A SteamOS or SteamVR update can break parts of it until Frametop catches up. After an update, run `cd ~/frametop && scripts/doctor.sh` in a terminal. It checks what Frametop needs from SteamOS, and says what changed since the versions you last marked as working and what to try. Once everything works, `scripts/doctor.sh --mark-good` records the versions. If something stops working, please report it.
- The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes.
@@ -157,14 +161,18 @@ In a terminal on the headset, run:
cd ~/frametop && scripts/report.sh
```
This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
From a release, start in `~/.local/share/frametop/releases/current` instead of `~/frametop`.
This writes `frametop-report-<date>.txt` with version numbers, service states, settings, Frametop's keyboard and the Steam menu, and recent logs. Bluetooth addresses and the headset's serial number are masked.
If the problem is something you can make happen, like a window that won't drag or a keyboard that doesn't open, run `scripts/report.sh --watch` instead. After the usual report it records for 60 seconds (`--watch 120` for longer) while you make it happen in the headset. It notes when the Steam menu opens and closes, which laser drags what, where typing goes, and when Frametop's keyboard opens or why it doesn't. It takes up to half a minute, because it also checks gaze mode: it starts the gaze service for a moment to see whether the eye tracker sends. If gaze or its calibration doesn't work, run it while you wear the headset. `scripts/gaze-report.py` prints only the gaze part, with what looks wrong first. Then [open an issue](https://github.com/Frametop/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
## Update
Run the same command again. It updates `~/frametop` to the latest of the version you have (or switches, if you pick the other one) and installs it:
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
curl -fsSL https://frametop.github.io/frametop/get.sh | bash
```
Or by hand: `cd ~/frametop && git pull && ./install.sh`.
@@ -174,7 +182,7 @@ Or by hand: `cd ~/frametop && git pull && ./install.sh`.
In a terminal on the headset, run:
```
curl -fsSL https://deejanuz.github.io/frametop/uninstall.sh | bash
curl -fsSL https://frametop.github.io/frametop/uninstall.sh | bash
```
It works in two steps, so it never takes away the keyboard, mouse, or desktop you're using while it runs:
@@ -194,7 +202,7 @@ A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland com
| Folder | What it is |
| --- | --- |
| `get.sh` | The one-line installer: picks stable or experimental, clones or updates the repo, and runs `install.sh`. |
| `get.sh` | The one-line installer: picks stable or experimental, clones or updates the repo, and runs `install.sh`; or installs a built release (`--release`). |
| `install.sh` | The one-step installer. Safe to re-run. |
| `uninstall.sh` | The uninstaller: run it, restart the headset, and run it again. It doesn't need the rest of the repo. |
| `desktops.sh` | Start, stop, and configure the desktop, and install the input relay. |
@@ -213,6 +221,10 @@ A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland com
| `setup/` | The build container and the Bluetooth fixes. See [setup/README.md](setup/README.md). |
| `scripts/` | Helpers the installers use. They run commands locally on the Frame, or over SSH from a PC. |
## Packaging
`pack/` builds Frametop as an OCI image: the toolchain, the native binaries, and the locked Python environment (via [uv](https://docs.astral.sh/uv/)), built and tested by GitHub Actions. It is the groundwork for installing Frametop without building anything on the headset. No installer uses it yet. For development, `./ft dev build` builds the image and `./ft dev test` runs the tests inside it. [pack/design.md](pack/design.md) explains why an image and what is still open, and [pack/README.md](pack/README.md) documents the image itself.
## Developing from a PC
The scripts also work from a Linux or WSL PC over SSH, which is easier for editing code. On the Frame they use the local checkout; on a PC they sync the repo to `~/dev/frametop` on the Frame and run there.
+2 -3
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# Launch Frametop Display Settings from a Plasma session on the Frame host.
# The app runs in the dev container (PySide6 and Kirigami come from Fedora there).
# The app runs in Frametop's container (PySide6 and Kirigami come from Fedora there).
# podman needs the real XDG_RUNTIME_DIR and the real user bus (to reach systemd for
# the container's cgroup; the Frametop session runs on a private bus from
# dbus-run-session). The session's Wayland socket and bus go to the app itself.
@@ -10,8 +10,7 @@ case $wl in /*) ;; *) wl="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/$wl" ;; esac
session_bus=${DBUS_SESSION_BUS_ADDRESS:-}
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
"$here/../scripts/container-up.sh"
exec "$HOME/.local/bin/distrobox" enter dev -- env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
exec "$here/../scripts/in-box" env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
QT_QPA_PLATFORM="wayland;xcb" \
python3 "$here/ft_display_settings.py" "$@"
+10
View File
@@ -16,6 +16,8 @@ the dev container:
saved from where the screens are, with a preview; arrange now; save the current
arrangement under a name; rename and delete; arrange automatically when the
desktop starts.
- Remote displays (other computers' monitors as screens) have their own app, Frametop
Remote Displays (remote-displays/); a button opens it.
- Power: how long the headset can go unused before ft-powerd turns its displays off
(DISPLAY_OFF_MIN; the service's state comes from its control socket, @ft_powerd),
and whether the Frame stays awake while plugged in, which is Steam's own setting
@@ -54,6 +56,7 @@ SCREEN_RESOLUTIONS = [(1920, 1080, ""), (2560, 1440, ""), (3840, 2160, "4K"), (2
(2560, 1600, "16:10"), (1080, 1920, "portrait"), (1440, 2560, "portrait"),
(2160, 3840, "portrait 4K")]
FT_SCREENS = "\0ft_screens"
REMOTE_DISPLAYS = os.path.join(HERE, "..", "remote-displays", "ft_remote_displays.py")
FT_POWERD = "\0ft_powerd"
# Steam's default for "When Plugged In and Idle -> Sleep after", to go back to when
# nothing was saved.
@@ -658,6 +661,13 @@ class Backend(QObject):
def capture(self):
self._run("Saving the current arrangement", "capture")
# --- remote displays: their own app ---
@Slot()
def openRemoteDisplays(self):
"""Frametop Remote Displays (we're in the dev container already, as it runs)."""
if not QProcess.startDetached(sys.executable, [os.path.abspath(REMOTE_DISPLAYS)]):
self.message.emit("Couldn't start Frametop Remote Displays", True)
# --- ft-layout on the host ---
def _run(self, label, *args):
if self._proc is not None:
+7
View File
@@ -180,6 +180,13 @@ Kirigami.ApplicationWindow {
icon.name: "view-visible"
onTriggered: backend.toggleScreens()
},
Kirigami.Action {
visible: spage.md
text: "Remote displays"
icon.name: "network-workgroup"
tooltip: "Other computers' monitors as screens: Frametop Remote Displays"
onTriggered: backend.openRemoteDisplays()
},
Kirigami.Action {
visible: backend.desktopRunning
text: "Restart desktop"
+4 -2
View File
@@ -42,6 +42,8 @@ Wherever ft-screens needs to know where a laser points (showing the controls, th
`ComputeOverlayIntersection` ignores `SetOverlayIntersectionMask`, and a control can't be allowed to cover part of its screen, so the resize tab sits entirely outside the corner.
What SteamVR hits isn't the texture's shape but the mouse scale's: an overlay is as tall, for SteamVR's laser and `ComputeOverlayIntersection`, as its width times the mouse scale's height over its width, and the default scale is 1 × 1. With it, the grab bar (a 256 × 24 texture) took hits in a square as tall as the bar is wide, so it caught clicks meant for the bottom tenth or so of the screen above it. Measured with a 0.2 m wide 256 × 24 overlay: a hit area 199 mm tall at the default scale, 18 mm at 256 × 24 (the bar itself is 18.8 mm), and no change from an intersection mask. `MakeChrome` sets each control's mouse scale to its texture size.
### Pinning
Pinning started as "bring the screen to your wrist", which doesn't work for big screens, because their centre is far from the edge you bring close. It became aiming: while a screen is carried, the line from the carrying device to its bar is tested against the other hand controllers. Crossing a controller's 6 cm ring arms the pin (leaving past 9 cm, so it doesn't flicker), and crossing it again disarms it. The pin happens on release, with the screen's pose at that moment, so you can arm it and then turn the screen. An earlier version pinned the moment the laser touched the wrist, which left the screen at whatever angle the carrying hand had while pointing there.
@@ -198,7 +200,7 @@ A podman container's monitor process (conmon) stays in the cgroup of whatever st
KWin renders with OpenGL through zink on Turnip, Vulkan on the same GPU vrcompositor needs to hit its frame time, and on the Frame that costs CPU too. The nested session started with KWin's defaults: blur and background contrast on (no `[Plugins]` group in its kwinrc) and animations at full length. Blur re-renders what's behind every translucent panel and menu each time it changes, and every animated frame is one more frame for KWin and ft-screens to draw and send. They're off by default in the Frametop desktop. The session script writes them before KWin starts, only where the desktop's own config has no value, once: System Settings deletes a setting put back to KDE's default rather than writing it, so without the marker in `frametoprc` a user who turned blur back on would lose it at the next start. The effect ids (`blur`, `contrast`) are the ones built into KWin 6.2.5 on SteamOS; KWin reads `<id>Enabled` from `[Plugins]`.
The nested session also runs the system's XDG autostart entries, being a KDE session. Discover's update notifier started `plasma-discover --mode update` in it (520 to 620 MB resident and about 9% of a core, plus `flatpak-system-helper` and AppStream downloads), and IBus started a daemon, the kimpanel panel and its GTK extension that nothing can use: KWin hands text input to the one input method it starts (`ft-textinput`), and the session drops `QT_IM_MODULE`, `GTK_IM_MODULE` and `XMODIFIERS`. The session hides both for this desktop only, with `Hidden=true` copies in its own autostart folder. The geoclue demo agent stays: it's what answers apps' location requests to Geoclue outside GNOME, and it costs nothing while idle. Orca's entry only starts in GNOME-family desktops.
The nested session also runs the system's XDG autostart entries, being a KDE session. Discover's update notifier started `plasma-discover --mode update` in it (520 to 620 MB resident and about 9% of a core, plus `flatpak-system-helper` and AppStream downloads), and IBus started a daemon, the kimpanel panel and its GTK extension that nothing can use: KWin hands text input to the one input method it starts (`ft-textinput`), and the session drops `QT_IM_MODULE`, `GTK_IM_MODULE` and `XMODIFIERS`. Steam's entry (`steam -silent`, from steamdeck-kde-presets) reached the running Steam client as a command line it ran (`ExecCommandLine` in its console log), since the desktop starts from Steam; SteamOS 0.4 added `-vrdisable -deckard` to it, for Desktop Mode, where Plasma starts Steam itself. The session hides all three for this desktop only, with `Hidden=true` copies in its own autostart folder. The geoclue demo agent stays: it's what answers apps' location requests to Geoclue outside GNOME, and it costs nothing while idle. Orca's entry only starts in GNOME-family desktops.
Plasma 6.2.5 keeps each panel on a screen number (`lastScreen` in `plasma-org.kde.plasma.desktop-appletsrc`), and the numbers rank the enabled outputs by priority, so 0 is the primary screen. A panel whose number is past the screen count gets no view, and Plasma never moves it: the remap it runs at every start only moves a panel whose number has no desktop, and this desktop keeps a desktop for every output it has seen, spares included. So the taskbar was lost when the number of screens went down, and once it was found saved on a spare output, number 8 of a desktop with three screens ([#18](https://github.com/DeeJanuz/frametop/issues/18)). Before Plasma starts, the session runs `session/fix-panels.py`, which moves any panel numbered past the screen count, with its system tray's containment, to screen 0, keeping its widgets and settings. A panel stays put when screen 0 already has one on that edge, and comes back by itself if the screens do. Before each repair the file is backed up to `<file>.ft-bak.last`. `<file>.ft-bak` keeps it as it was before the first repair and is never overwritten. The repair writes over a moved panel's old screen number, so the backups are the only record of it, and `.ft-bak.last` also keeps everything changed since the first repair. Plasma's scripting can't do this while it runs (`panel.screen` is read-only in 6.2.5), so a lost taskbar comes back at the desktop's next start. `scripts/doctor.sh` and `scripts/report.sh` list the panels and their screens.
@@ -229,7 +231,7 @@ Hiding the screens during a game kept them out of view, but Frametop kept using
On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-steamvr-rel` package), next to KWin, gamescope, and the kernel, so every SteamOS update can bring a new SteamVR too. Frametop survives updates: it lives in the home folder and the `dev` container, the Bluetooth fixes are in `/etc`, which SteamOS keeps across updates, and nothing goes into `/usr`. What an update can break is what Frametop uses from the image. The public OpenVR API is versioned and stays put. The rest is less certain: `IVRIPCResourceManagerClient`, which is newer than the header SteamVR ships; the text `vrcmd --overlays` prints; the eye tracker's shared memory layout; XRService's camera buffers; KWin's nested backend; and behavior Frametop works around, such as the SteamVR Settings page that `ComputeOverlayIntersection` can't find or the scale KWin's nested backend doesn't undo.
`scripts/update-check.py`, which `scripts/doctor.sh` runs, checks what it can directly: that SteamVR still serves every OpenVR interface version the installed programs were built against (read from the binaries), that `vrcmd`'s format still parses, that the eye tracker's shared memory still has a layout ft-gaze knows (stable's, or the 0.4.x beta's, with every field from the timestamp on 5 bytes later), by the same test ft-gaze uses to pick one, and the host files, services, sockets, and driver registration. Behavior can't be checked without someone in the headset, so it records the versions of the packages that matter once things work (`--mark-good`), and after an update names what changed and what to try by hand.
`scripts/update-check.py`, which `scripts/doctor.sh` runs, checks what it can directly: that SteamVR still serves every OpenVR interface version the installed programs were built against (read from the binaries), that `vrcmd`'s format still parses, that the eye tracker's shared memory still has a layout ft-gaze knows (SteamOS 0.3's, or 0.4's, with every field from the timestamp on 5 bytes later), by the same test ft-gaze uses to pick one, and the host files, services, sockets, and driver registration. Behavior can't be checked without someone in the headset, so it records the versions of the packages that matter once things work (`--mark-good`), and after an update names what changed and what to try by hand.
## Approaches we dropped
+3 -1
View File
@@ -6,7 +6,8 @@ A profile is a named layout that also opens apps. It holds:
- where each screen goes, with its size in metres, curve, roll, and pin (what a named layout held before profiles);
- which screens show and which are hidden;
- the apps, one entry per window: on a screen at a place and size, or floating at a pose, size, and scale.
- the apps, one entry per window: on a screen at a place and size, or floating at a pose, size, and scale;
- the remote displays connected when it was saved, each with its place and whether it's hidden ([remote-displays.md](remote-displays.md)). Opening the profile connects them, if their computer answers, and puts them back. Like its apps, it leaves other displays connected.
So a "Work" profile can put three screens around you with a browser, two terminals, and an editor on them, and a "Couch" profile can hide every screen and float one video player in front of you.
@@ -50,5 +51,6 @@ So a "Work" profile can put three screens around you with a browser, two termina
- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name (found the same way as at capture): single-instance and D-Bus-activated apps open their windows from a process that was already running.
- **Default at start.** The session script runs `ft-layout start --wait 90`. That opens the profile in `FT_PROFILE` or `default_profile` (screens, then the apps once ft-floatd is up), or runs `apply --wait` if there's none. Start in profile on the Layout & profiles page sets `default_profile` (`ft-layout default NAME|none`). Plasma's session restore is turned off in the session (`ksmserverrc`: `loginMode=emptySession`).
- **Launcher entries.** Each profile gets `~/.local/share/applications/frametop-profile-<name>.desktop` ("Frametop: Work"), written when it's saved and removed when it's deleted. They show in SteamVR's Launch a program list, the Application Launcher, and KRunner. Running one (`ft-layout open NAME`) switches to that profile if the desktop runs. Otherwise it starts the desktop with `FT_PROFILE` set (`systemd-run`, as `desktops.sh start` does), which overrides `default_profile` for that start. That needs SteamVR to be running.
- **The quick reset.** Meta+Shift+R, the reset button on a screen's bar, the Reset Screen Layout menu entry, and a button mapped to Reset desktop screen layout run `ft-layout reset`: with a profile in use (`active`, and the custom arrangement), that's `use` on it again, so everything goes back as the profile has it. Without one, it's `apply`.
- **The action.** `profile:NAME` in the input relay (it runs `ft-layout use NAME`) for key combinations, mouse buttons, and controller buttons, with or without pointer mode. Input Settings lists one "Open profile NAME" action per profile.
- **Display Settings.** On the Layout & profiles page, the arrangement list has the profiles, which can be renamed and deleted. Open profile and Save as profile… are the page's actions. A profile's apps are listed with where each goes and a button to leave one out, plus which screens it hides. Start in profile picks the one the desktop starts with. The Visibility tab's Screens shown switches hide screens one at a time.
+15 -3
View File
@@ -30,7 +30,7 @@ kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key contrastEna
kwriteconfig6 --file ~/.config/frametop/kdeglobals --group KDE --key AnimationDurationFactor 1
```
Two of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, and IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`), and skips a name you already have a file for. Delete a copy to start that program again.
Three of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`, and Steam (`steam`), which is already running. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`; Steam's was added later and is hidden once on desktops that already had the other two), and skips a name you already have a file for. Delete a copy to start that program again.
Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in `~/.config/frametop-layout.json`. The backend, remote desktop, and pointer settings are in `~/.config/frametop.conf`; `session/frametop.conf.example` lists every key.
@@ -48,7 +48,7 @@ Every screen is an overlay named `frametop.screen.N` with five controls:
- `.curve` bends the screen into a cylinder around you, using your current distance as the radius, or makes it flat again.
- `.roll` rolls the screen when you drag it sideways, like a knob. It snaps level within 2.5°, and scrolling on it turns 5° per notch.
- `.resize`, the tab on the bottom right corner, sets the width. Screens go down to 15 cm wide.
- `.reset`, left of the bar, puts every screen back in its layout around where you are now, like Meta+Shift+R (`ft-layout apply`).
- `.reset`, left of the bar, is the quick reset, like Meta+Shift+R (`ft-layout reset`): with a profile in use it opens that profile again, as Open profile does, and otherwise it puts every screen back in its layout around where you are now.
The controls are sized from both the screen's width and its distance from you, follow the surface of a curved screen, and stay invisible until a laser or the 3D mouse's cursor lands on one or comes within about 1.5 times a button's size of it. While invisible they're still there, fully transparent, so SteamVR's laser can find them. They're translucent until a laser is on them, like SteamVR's own window controls.
@@ -148,7 +148,7 @@ Device rules are saved in `~/.config/frametop-input.json`. `input-settings/insta
## Frametop Display Settings and ft-layout
When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the reset button left of any screen's bar, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout.
When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the reset button left of any screen's bar, the Reset Screen Layout menu entry, or a button mapped to Reset desktop screen layout. With a profile in use, these open it again, the same as Open profile: its screens, hidden screens, remote displays and apps. Arrange now in the app arranges the screens (and the profile's remote displays) without reopening its apps or hiding its hidden screens again.
The desktop's own screen arrangement follows where the screens are around you, whatever their numbers: a screen you see to the left of another is to its left in Plasma too, so the pointer and dragged windows cross straight to it. Screens one above the other stack, and screens pinned to a wrist or your head come last. It's updated at startup, after arranging or saving the layout, and half a second after you let go of a screen you moved. With the headset off there's no head pose to go by, and the arrangement stays as it was.
@@ -179,6 +179,18 @@ display-settings/install.sh # menu entries and the Meta+Shift+R and Meta+Shift+H
The layout is stored relative to your head when it's applied. `/run/user/<uid>/frametop-layout.log`, in the host's runtime directory (not the nested desktop's `/run/user/<uid>/frametop`), has the run from the last desktop start and ft-screens' layout runs after it.
## Frametop Remote Displays
Other computers' monitors as Frametop screens (ft-screens only), streamed from Vibepollo with Moonlight's protocol. Frametop Remote Displays (`remote-displays/`, also opened by the Remote displays button on Display Settings' Screens page) finds Vibepollo computers on the network, signs in to one with its Web UI login (it keeps a narrow API token, not the password, and pins the host's certificate), shows whether it answers, and adds its displays: its monitors, or a virtual one at any size. A computer with a Steam Link dongle on the Frame's hotspot streams over it (Connection: auto, network or dongle only); the others use the network. Each display has a Connected switch (and the host one for all of its displays), Shown, its stream's resolution, frame rate and bitrate, and its width in VR. A disconnected display keeps its settings, and within the same desktop run, its place. In VR each one is a panel with a screen's controls, and profiles keep where they are. See [remote-displays.md](remote-displays.md).
```
layout/ft-layout remote list # the remote displays and their streams' state
layout/ft-layout remote connect|disconnect ID...
remote-displays/install.sh # its menu entry
```
On the PC, `host/windows/Setup Frametop host.cmd` sets it up for Frametop: Vibepollo 2.0.0 (installed if missing), Frametop's build of its `sunshine.exe`, the settings Frametop needs, the Web UI login you sign in with from the Frame, and a firewall check (`-Check` to see what it would change, `-Undo` to put things back).
## Floating windows
A desktop window can float in VR as a panel of its own, away from the screens. Meta+Shift+F floats the window under the pointer (or the active one, over the wallpaper), or puts it back on its screen if it floats. So do Float in VR in every window's menu (Alt+F3; Back to Desktop on a floating one), the button left of Close in its title bar, and a mouse button, controller button, or key combination mapped to Float window in VR in Frametop Input Settings; Put all floating windows back is mappable too. Launch as Standalone, in an app's right-click menu in the Application Launcher or the taskbar, starts the app with its first window floating, where that app last floated or in front of you. [floating-windows.md](floating-windows.md) explains how it works.
+486
View File
@@ -0,0 +1,486 @@
# Remote displays (exploration)
Status: exploration. Spike S1 is done; nothing in Frametop has changed yet. Branch `remote-displays`, written 2026-10-06.
The idea: show desktops streamed from other machines as Frametop panels that behave like the native screens. They get placement, curve, pinning, layouts, lasers, gaze, and attention-based frame rates. The target is up to 5 remote displays from 2 hosts, with no more headset overhead than 5 native screens.
## What the hardware gives us
These were checked on the Frame (SteamOS kernel 6.18, SM8650 / Snapdragon 8 Gen 3, Mesa 26.3 Turnip on Adreno 750).
- **Decoder.** `/dev/video22` (`/dev/video-dec0`) is `qcom-iris-decoder`, Qualcomm's downstream iris driver (`drivers/media/platform/qcom/vcodec/iris`). It takes H.264, HEVC, and VP9 up to 8192x8192. AV1 isn't supported. The capture queue can produce `Q08C` (NV12 in Qualcomm's UBWC compressed layout), linear NV12, and NV21. It also lists RGBA (`AB24`, `QC24`), but those write a 10-bit UBWC YUV picture (S1), so the decoder has no usable RGB output.
- **10-bit works on Valve's kernel.** Steam Link VR's client (`vrlink.txt`, `SVLCodecV4L2`) decodes into `Q10C`, 10-bit UBWC, at 1152x4608. That is both eyes stacked in one frame, so it uses one decode session. Valve drives the decoder through the raw V4L2 stateful API, as `vr-recorder` does for the encoder.
- **Decoder budget.** The driver carries `MAX_SESSION_COUNT`, `MAX_MBPF`, and `MAX_MBPS` capability tables. For SM8650 the upstream values are 16 sessions, 278,528 macroblocks per frame, and 7,776,000 macroblocks per second (one 8K stream at 60 fps). A session that would go over the budget is refused when it starts. The downstream numbers on Valve's kernel are still to be confirmed by spike S2.
- **SteamVR imports YUV but doesn't convert it.** `IVRIPCResourceManagerClient::GetDmabufModifiers(VRApplication_Overlay, …)` returns LINEAR and `0x0500000000000001` (`DRM_FORMAT_MOD_QCOM_COMPRESSED`) for NV12, P010, and the RGB formats. The probe is in `~/.cache/remote-displays-spike/modprobe.cpp`. `DmabufAttributes_t` takes multiple planes. This is the call ft-screens already makes for KWin's buffers (`ft_vr_screen_present`, `vr.cpp:2080`). The import works, but vrcompositor samples the planes as they are: red shows Cr, green Y, blue Cb (S1). `DmabufAttributes_t` has no colour-space fields to change that.
So a decoded frame needs one GPU pass before SteamVR sees it: NV12 to RGBA, about 0.13 ms of GPU time for a 3440x1440 frame at full clock. It is the same kind of pass as the hand cutouts' (`screens/handcut.cpp`), and native screens pay for one of the same size: KWin's composite of each output.
## Is the decoder the bottleneck?
Mostly no. Macroblocks per second against the 7,776,000 budget, at 60 fps:
| Displays | Share of the decoder |
|---|---|
| 5x 1920x1080 | 31% |
| 5x 2560x1440 | 56% |
| The current layout (3440x1440 + 2x 1440x1920) | 32% |
| 5x 2560x1440 at 90 fps | 83% |
| 4x 3840x2160 | 100% |
| 5x 3840x2160 | 125%, refused |
| Steam Link VR's own stream (1152x4608 at 120 Hz) | about 32% |
Five 1440p displays fit at full rate with room left. Five 4K displays only fit at about 48 fps or less. That number is what gets declared at session start; actual load is lower, because hosts send frames only when the screen changes (see "Keeping the decoder under budget").
### The planned setup
Two hosts that share the same two desk monitors: a Mac (14-inch MacBook Pro) with those two plus its built-in display, and the test PC (Windows, RTX 5090) with the two. Windows and Linux hosts run Vibepollo, which streams real displays and creates virtual ones on demand. The Mac runs Sunshine and is limited to one display for now (see "Host side"). At 60 fps:
| Display | Share of the decoder |
|---|---|
| Super ultrawide 5120x1440 (5160x1440 as given; same load within 0.2%) | 22.2% |
| Built-in 3024x1964 (native pixels) | 17.9% |
| Ultrawide 3440x1440 | 14.9% |
| Mac, one display | 14.9% to 22.2% |
| The test PC, ultrawide + super ultrawide | 37.1% |
| Both hosts today (three displays) | at most 59.4% |
| Five: the three above plus two 3440x1440 virtual displays on the test PC | at most 89.2% |
| The original five (all three Mac displays + the test PC's two) | 92.2% |
Every combination fits under the driver's limit at 60 fps. The original five stay as the worst case for spike S2, so the Mac can grow past one display later without a new budget. These are declared numbers, with every display changing every frame at once. With damage-driven hosts the real load is far lower.
Notes:
- The 5120-wide display needs HEVC. H.264 hardware encoders stop at 4096 pixels wide, and the iris decoder takes HEVC up to 8192. HEVC for every stream keeps it simple.
- Streams don't need more than 60 fps. The Frame's display runs at 90 Hz, and the MacBook's 120 Hz ProMotion would double the built-in display's load for nothing.
- A stream can be smaller than its display, because the host scales before encoding. A panel in VR covers far fewer headset pixels than the display has; a 1 m wide panel at 1 m spans about 53°, which is roughly 1,000 headset pixels across (estimate; the Frame's pixels per degree haven't been measured here). The built-in display at 2268x1473 instead of 3024x1964 would cost 10% instead of 18%. That's the lever when Steam Link VR also needs the decoder, or when a virtual display is made larger.
- The desk monitors are shared through input switching. A monitor switched to the other machine may disappear from the first one, depending on the monitor and the cable. Real-display streams only work for monitors that host currently sees. In the headset, virtual displays avoid the question.
A remote display should cost less than a native one everywhere else:
| Per display | Native screen | Remote display |
|---|---|---|
| Apps | Run on the Frame's CPU | Run on the host |
| Composition | KWin draws each output on the Adreno GPU | One GPU pass per new frame turns the decoder's NV12 into RGBA |
| Hand-off to SteamVR | XRGB dmabuf, zero copy | RGBA UBWC dmabuf, zero copy |
| Sampled by vrcompositor | 4 bytes per pixel | The same |
| New work | — | Network receive and reassembly, V4L2 queueing |
The new costs are CPU for receiving packets, plus the decoder's and Wi-Fi radio's power and heat. Heat matters because the SoC slows down when it gets hot. Those are what the spikes need to measure.
## Keeping the decoder under budget
1. **Damage-driven hosts.** Sunshine and its forks send a new frame only when the captured screen changes, plus duplicates down to `minimum_fps_target` (default half the stream rate, settable to 1). With `minimum_fps_target = 1`, a static display costs about one decoded frame a second, the same idea as a native screen with no damage. A display playing video costs full rate, as a native video screen does.
2. **Out of sight means 1 frame per second, whatever the host sends.** A display outside your view updates once a second, even when the host is sending 60 fps from a game. See "Displays out of sight" below for how. Concealed panels, and every panel while the desktop is paused for a game, disconnect fully after a grace period. That frees the host's encoder and the network.
3. **An admission budget.** Before connecting, Frametop adds up the declared load of every stream: width/16 × height/16 × fps. If a new stream would go past the driver's limit, minus whatever else is decoding, Frametop lowers settings before connecting: first fps, then resolution, starting with the displays that get the least attention. Today's three displays come to at most 59%, which fits next to Steam Link VR's stream (about 32%). Five displays (89-92%) fit alone but not next to it, so a streamed PC game with five remote panels up would push them down to about 45 fps (or some lower and others higher).
4. **No B-frames, low-latency decode.** Sunshine doesn't send B-frames, so every decoded frame can be shown at once. The decoder gets a small capture pool (6-8 buffers). Valve's client found the standard `DISPLAY_DELAY` controls unsupported on this driver (`EINVAL` in `vrlink.txt`). They aren't needed: S1 showed that with `Q08C` each frame comes out as soon as it's decoded (linear NV12 holds two more).
5. **Tiling only as a fallback.** Valve tiles both eyes into one frame because they always change together. Desktops don't: one busy display would make the whole canvas decode at full rate, and a hidden tile can't be skipped. Per-display streams keep each display's cost proportional to its own changes. Tiling makes sense only for a host with many small, mostly static displays, and Sunshine can't capture a spanning desktop on Windows anyway.
The Moonlight protocol can't change resolution, fps, or bitrate mid-stream; that takes a reconnect. With stock hosts, attention changes therefore act on the client: in what gets decoded (2), not in what the host sends.
## Displays out of sight
Remote panels follow the native screens' attention rules (`UpdateAttention`, `vr.cpp:855`):
| Attention | Native screen | Remote display |
|---|---|---|
| Focused (within 12° of where you look) | Full rate | Full rate |
| In view (within 60°) | 15 Hz, or full rate while it plays video | Full rate. Every frame has to be decoded anyway (below), and the host sends only what changed |
| Hidden (out of view) | 1 Hz | 1 Hz, even when the host sends 60 |
The "video" signal comes for free: a damage-driven host sends frames only when its screen changes, so frames arriving faster than 10 a second for 8 frames in a row mean video, matching `VIDEO_COMMITS` and `VIDEO_HZ` in `compositor.c`.
The stream's fps is ours to choose. A game running at 144 Hz on the host is captured and sent at the fps the stream asked for (60 at most), so 144 never reaches the decoder.
**Why 1 Hz can't just skip frames.** Each frame in a stream is coded as changes to the frame before. Decoding frame 60 needs frames 1 to 59. Throwing away 59 of every 60 frames breaks the chain, and the next frame decodes as garbage. Valve's client works the same way: on a stall it asks the host for a new full frame (an "IFrame" in `vrlink.txt`).
**With stock hosts (Moonlight protocol): keyframe sampling.** While a panel is hidden, ft-stream:
1. drops every incoming frame before the decoder (moonlight-common-c's decode callback gets a `DECODE_UNIT` with `frameType`; return `DR_OK` without queueing);
2. calls `LiRequestIdrFrame()` once a second, and decodes and shows only the IDR frame that comes back, which needs no earlier frames;
3. when the panel comes back into view, requests one more IDR and goes back to decoding everything. The last frame, at most a second old, stays up until it arrives, about one round trip plus one frame later.
What that saves and what it doesn't, for a hidden display whose host sends 60 fps:
| Cost | Saved? |
|---|---|
| Decoder work | About 59 of 60 frames. One IDR costs a bit more to decode than one change frame |
| Panel updates in vrcompositor | 59 of 60 |
| Network traffic | No. The host keeps sending 60 fps, plus one larger IDR a second. A 5120x1440 IDR is a few hundred KB (estimate), so a few Mbit/s extra |
| Frame CPU to receive, reassemble, and repair packets | No. moonlight-common-c still handles every packet. Patching it (it's GPL, and so is ft-stream) to discard hidden video packets early would cut most of this |
| Host encoder | No |
**With a host that takes an fps change mid-stream: real 1 Hz.** If the host can be told "this display is out of sight, send 1 fps", the hidden display costs almost nothing anywhere. That's network, Frame CPU, decoder, and host encoder alike, and no IDRs are needed. The 1 Hz frames are ordinary change frames against the one a second earlier. This needs our own streamer, or a Sunshine fork with a control message that changes the capture rate. Owning a host streamer was rejected for its maintenance cost (see "Alternatives considered"), so this stays a possible upstream contribution. Vibepollo is the likeliest place to propose it: it already extends the protocol, since its own Moonlight fork sends a VRR pacing request that changes how the host captures.
Spike S3 checks the stock-host version: whether Sunshine honours an IDR request every second, how big and how late those IDRs are, and what receiving a hidden 60 fps stream costs the Frame's CPU.
## Proposed architecture
Frametop maintains only the headset side. The hosts run existing, separately maintained streamers that speak the Moonlight protocol: Vibepollo on Windows and Linux, Sunshine on macOS (see "Host side").
```
host machine: Vibepollo (Windows, Linux) / Sunshine (macOS), one stream per display
│ RTSP + RTP over UDP, ENet control
▼
ft-stream (one process per remote display, on the Frame, GPLv3)
moonlight-common-c (protocol) + pairing (moonlight-embedded's libgamestream)
iris decoder session: OUTPUT = HEVC, CAPTURE = Q08C, VIDIOC_EXPBUF
GLES pass: Q08C → ring of 3 RGBA buffers (EGL YUV hints = the stream's colour space)
│ unix socket, SCM_RIGHTS
▼
ft-screens (existing, MIT)
remote screen type: ImportDmabuf once per ring buffer,
SetOverlayTexture per frame, buffer returned to ft-stream when replaced
panel input → ft-stream → LiSendMousePositionEvent / LiSendKeyboardEvent / …
```
**Why one process per display.** moonlight-common-c keeps global state, so it runs one stream per process. It and libgamestream are GPLv3 while Frametop is MIT, and a helper that talks over a socket keeps the licences apart (this isn't legal advice). ft-stream lives in its own directory with its own licence file. Separate processes also mean a stalled stream or a decoder reset can't freeze the native screens. Each process pairs as its own client, which Vibepollo needs anyway: it ties each Remote Monitor to the client that opened it, one role per client, so every display needs its own client certificate and pairing.
**Pairing with a host token (chosen 2026-10-07).** Since every display is its own client, PIN pairing would mean one PIN and one round of permission clicks per display. Vibepollo has neither a multi-use PIN (its one-time PINs pair one client each) nor a way to pair several certificates at once. It does have scoped API tokens for its Web UI API (`POST /api/token`, sent as `Authorization: Bearer`). So the user creates one token on the host, limited to submitting pairing PINs (`POST /api/pin`), listing clients and setting their permissions (`GET /api/clients/list`, `POST /api/clients/update`), listing the host's monitors (`GET /api/display-devices`), and placing Remote Monitors (`GET`/`PUT /api/clients/display-layout`). A script on the host makes it from the Web UI login (`make-frametop-token`), and the user enters it once in Frametop's "add host" dialog. From then on, Frametop makes a client for each new display, pairs it by sending its own PIN, and grants it launch, mouse, and keyboard (Vibepollo gives a new client only list and view). It sizes Remote Monitors like the host's real monitors, and unpairs a display's client with that client's own certificate when the display is removed. The token can't change the host's settings and can be revoked in the Web UI. It can change any client's permissions, though, so Frametop keeps it like a password (a file only the user can read). Vibepollo's client update replaces the whole client record, so Frametop always sends every field.
**Why ft-stream converts.** ft-screens then gets RGBA dmabufs, as it does from KWin. Each decoder buffer goes back to the decoder right after the pass, so the decoder's pool doesn't depend on what SteamVR still holds. A GPU fault in one stream stays in its own process. ft-stream asks the host for BT.709 limited range (Moonlight's colour space and range settings) and gives the converter the same as EGL hints. S1 showed all four matrix and range combinations come out right when the hints match the stream.
**Opening a display.** For a host's main session, ft-stream launches the host's desktop app as Moonlight does. For a Vibepollo virtual display it launches the synthetic "Remote Monitor" app (id `2147483505` in `remote_session.h`), and the requested stream size and fps become the virtual display's mode. When the stream drops, Vibepollo keeps that display and its windows by default (`remote_monitor_disconnect_on_stream_end = false`), and ft-stream reconnects with "Resume" (id `2147483501`). So a concealed panel can disconnect fully without the host rearranging its windows.
**The socket protocol**, roughly:
- ft-stream → ft-screens: `buffers` (count, width, height, DRM format, modifier, offset and pitch, with the dmabuf fds of the RGBA ring) after each (re)configuration; `frame i` when ring buffer i holds a new picture (sent once the GPU is done, so no fence is needed); `title`, `state` (connecting, live, waiting for a keyframe, lost).
- ft-screens → ft-stream: `release i` when buffer i is no longer on screen; `attention focused|view|hidden|concealed`; input events.
**In ft-screens**, the explorer's map gives the seams:
- Remote screens get `g_screens` entries through a `MakePanel` variant, with indices out of KWin's first-free-slot range (`compositor.c:279`) and the overlay keys `frametop.remote.N`. That brings visibility, attention, lasers, controls, spin, pinning, and hand cutouts.
- Frames enter through an eventfd on the wl event loop and go through the same `ft_vr_screen_present` path, keyed by ring buffer, so the `g_imports` cache hits: 3 imports per display.
- The pointer helper needs the new prefix in `FramePanel` (`ft-pointer.cpp:485`). The `screens` reply and `get N` need to list remote screens so gaze hit-testing sees them (`ft-gaze.cpp:302`).
- `frametop-layout.json` gets a separate `hosts` list, since the `screens` array also sets KWin's output count. The user adds a host once, and its displays come as a bundle: Frametop pairs a client per display, starts their streams when the desktop starts, and stops them with it. Each host entry has the address, the token's file, and its displays. Each display has which one it is (the main session or a Remote Monitor), its own stream settings (size, fps, bitrate, and later codec and HDR), and the usual place, width, curve, and pin. ft-layout, profiles, and Display Settings learn the new list; Display Settings shows a host with its displays under it, each with its own settings (user decision, 2026-10-07).
**Input.** `handle_vr_event` (`compositor.c:333`) branches on the screen type. A remote screen sends:
- pointer motion as `LiSendMousePositionEvent(x, y, w, h)` in stream pixels;
- buttons as `LiSendMouseButtonEvent`;
- scroll as `LiSendHighResScrollEvent`, which matches the existing notches × 120.
A drag can cross panels, for example a window dragged from one of the host's displays to another. SteamVR sends a held button's moves only to the panel where the press began, with coordinates off that panel once the laser leaves it. The panel under the laser gets nothing (S3 measured this). So while a button is held on a remote screen, ft-screens hit-tests the laser (`ComputeOverlayIntersection`) against the host's other remote screens and sends the position to the ft-stream of the screen it hits. Moves that land on no panel are dropped, never clamped: clamping pins the host's cursor to the first display's edge. The host's input is shared across its sessions, so Windows sees one drag. But the release goes through the stream the press went through: Vibepollo takes a mouse button's release only from the client that pressed it (`mouse_press_owner` in its `src/input.cpp`). Sent through the display under the laser, it was dropped, and the window stayed on the pointer (found in the headset, 2026-10-07; `g_pressed_on` in `remote.c`).
Keyboard focus follows the last panel clicked. When it's a remote one, `send_key` and `relay_button` send evdev codes to its ft-stream, which maps them to Windows virtual-key codes (moonlight-qt has the table). Sunshine on macOS maps the Windows key to Cmd and Alt to Option. The host draws its own cursor into the video, so it lags the laser by one round trip.
**Won't work across the boundary:** drag and drop, the clipboard (the Moonlight protocol has none), floating windows, and KWin window rules. A remote display is a picture of another machine's monitor.
## Host side
**Decision for v1 (2026-10-07): real monitors only.** Vibepollo 2.0.0's Remote Monitors broke the test PC's monitor layout every time one was removed (see "S3 results so far"), while streaming a real monitor changes nothing on the host. Frametop can make extra screens in the headset itself, and the feature is mostly for controlling other machines, so v1 streams only a host's real monitors; virtual displays wait until the Vibepollo bugs are fixed. One host program streams one real monitor (a second client launching the main app joins the display already streaming), so each extra real monitor needs its own host instance: its own config file and ports (`port` 100 apart) and `output_name` set to that monitor. On the test PC that's Vibepollo for the primary (it also serves the Steam Deck and the Mac) and a plain Sunshine instance for the LC34G55T. Plain Sunshine has no scoped API tokens, so a Sunshine instance is paired once by PIN; it has a single Frametop client anyway. Frametop's host bundle becomes a list of instances, one per real monitor.
The requirement: each host streams its real displays, and virtual displays it creates on demand. Frametop doesn't maintain a host streamer; it uses existing ones and ships setup notes or scripts for them. State checked 2026-10-06.
| Host OS | Streamer | Displays |
|---|---|---|
| Windows | Vibepollo | 1 main session (real or virtual) + up to 4 virtual Remote Monitors |
| Linux (Arch, CachyOS; beta) | Vibepollo | The same, with at most 4 virtual displays at once |
| macOS | Sunshine | 1, the Mac's main display, for now |
### Windows and Linux: Vibepollo
Vibepollo 2.0.0 (2026-09-30) is a Sunshine fork. One install gives:
- **A main session.** As on any Sunshine host, this is a real display (`virtual_display_mode = disabled`, picked by `output_name`) or a virtual display created for the client (`per_client`, the default on Windows 11 and Linux).
- **Up to four Remote Monitors** (`max_client_vdds = 4` in `remote_session.h`). Each is a virtual display at the size and rate its client asks for, streamed and captured on its own (`capture_plan` in `remote_session.cpp`). They don't need a main session: with nothing running, the app list still offers Remote Monitor. They are always virtual and can't stream a real display.
That covers virtual displays completely, up to five from one host. Real displays are the limit: only the main session shows one, so one Vibepollo install streams one real display. A second real display at the same time needs a second host instance with its own config file, port, and `output_name`. On Windows that could be a plain Sunshine instance next to Vibepollo (untested). On Linux, Vibepollo expects to be the only host install on the machine. For the test PC this means one desk monitor streams as the main session and further displays are virtual, unless S3 shows a second instance works. Virtual displays also avoid the input-switching question (see "The planned setup").
Settings Frametop's setup notes change from the defaults:
- `virtual_display_layout`: the default `exclusive` turns off the host's other monitors while a virtual main session streams. `extended` keeps them on; `extended_isolated` also stops the host's own mouse from wandering onto the virtual display. Remote Monitors don't use it: Vibepollo always adds them next to the monitors already on (`apply_remote_monitor_composition` in `nvhttp.cpp`).
- `minimum_fps_target = 1`, so a static display costs about one frame a second.
- `remote_monitor_mute_audio = true`, unless that display should carry audio.
Both platforms send frames only when the screen changes. On Linux the virtual displays use presentation-driven capture: sparse changes are captured at once, and faster ones are coalesced to the stream's fps. The virtual outputs have no cursor plane, so the cursor is in the video, as with every Moonlight host.
PyroWave, Vibepollo's wavelet codec, isn't usable here. It decodes on the client's GPU and needs hundreds of Mbit/s over wired LAN. Frametop uses HEVC.
**Windows (the test PC).** The chosen setup (2026-10-06): one desk monitor streams as the main session, a real display (`virtual_display_mode = disabled`, `output_name` set to it). The other desk monitor is replaced by a Remote Monitor, a virtual display at that monitor's size, which Vibepollo releases when Frametop ends the connection (`remote_monitor_disconnect_on_client_disconnect = true`). A dropped connection keeps it (`remote_monitor_disconnect_on_stream_end = false`), so a Wi-Fi drop or a panel that disconnects while out of sight doesn't move its windows; ft-stream releases it explicitly with "Disconnect Monitor" (id `2147483502`) when the remote display is removed or Frametop exits. Each Remote Monitor's place next to the real monitors is set per client in the Web UI. GeForce allows 8 concurrent NVENC sessions per system. Installing Vibepollo and its display driver needs an admin; the test PC's `maptrainer` account isn't one.
How it's set up on the test PC (2026-10-06). Vibepollo installed over Apollo in `C:\Program Files\Apollo` (service `ApolloService`) and kept Apollo's settings, pairings, and Web UI login. The existing clients (a Steam Deck and a Mac) launch "Desktop" or "Steam Big Picture", which stream a virtual display with the other monitors turned off (`dd_configuration_option = ensure_only_display`). Frametop leaves them alone and gets its own app instead:
- "Frametop primary display": `"display-output": ""` streams the real primary monitor, and `"dd-configuration-option": "disabled"` leaves the other monitors as they are. ft-stream launches this app for the main session. These are the fields the Web UI writes for "use my own display" (`process.cpp` reads them).
- `minimum_fps_target = 1` globally, which Remote Monitors use. "Desktop" and "Steam Big Picture" keep 120 as per-app `config-overrides`, so the Deck and the Mac stream as before.
- `remote_monitor_mute_audio`, `remote_monitor_disconnect_on_client_disconnect` on, `remote_monitor_disconnect_on_stream_end` off.
The config folder is admin-only, so the change is a script for the user to run (`D:\remote-displays\vibepollo-setup\run-setup.cmd`, with a backup and an optional Web UI password reset). It ran on 2026-10-07. Vibepollo rewrites apps.json each time it starts and adds its built-in "Remote Input" and "Remote Monitor" apps, so any later change must start from the live file.
**Linux (beta).** x86_64 only: Arch Linux or CachyOS, KDE Plasma 6 on Wayland started by SDDM or Plasma Login Manager, Linux 6.16 or newer with matching headers, and a GPU with hardware H.264 encode (tested on NVIDIA and modern AMD). Virtual displays come from Vibepollo's own DKMS module, `vibeshine_drm`, which has four connectors, so at most four virtual displays exist at once. Streaming before login needs NVIDIA. Other distributions aren't covered; plain Sunshine still streams real displays there, without virtual ones.
**Risk.** Vibepollo is new, moves fast, and has one maintainer, and its README says about 99% of its code is AI-generated. Frametop depends only on its Moonlight-protocol behaviour and the Remote Monitor app ids, so on Windows Apollo (virtual displays through the SudoVDA driver) or plain Sunshine (real displays) stay as fallbacks.
### macOS: Sunshine, one display for now
Vibepollo has no macOS build. Sunshine (v2026.914, labelled experimental on macOS) is the only Moonlight-protocol host for the Mac, so the Mac streams one display: its main display, from one Sunshine instance. The reasons:
- Since v2026.906, absolute mouse input only reaches the main display (Sunshine #5733). Any other display would show but couldn't be clicked.
- Nobody reports running several Sunshine instances on one Mac yet.
- Sunshine can't create virtual displays on macOS.
The main display is the one with the menu bar (System Settings → Displays). Capture is at backing pixel size, so the built-in display streams at 3024x1964. Closing the lid removes the built-in display, so a stream of it ends. Capture stalls if the display sleeps mid-stream (#5509).
One display can still reach the encoder's limit. Sunshine forces VideoToolbox's low-latency mode, which roughly halves throughput (#5814); an M4 Pro managed 0.36-0.46 Gpix/s in that mode. The super ultrawide at 60 fps is 0.44 Gpix/s, so full-motion video on it may need a lower fps or a smaller stream. The other two displays need 0.30 and 0.36 Gpix/s.
**What would lift the limit.** The mouse fix is libvirtualhid PR #145 ("target configured mouse viewport", open), which Sunshine's draft PR #5739 pulls in. Once it ships, spike S4 tries one Sunshine instance per display (own config file, `port` about 100 apart, `output_name`) and BetterDisplay virtual displays switched on by `global_prep_cmd`. Open PR #5817 (ScreenCaptureKit + OBS's VideoToolbox encoder) may raise the encoder limit. Helping #145 along upstream is the one Mac contribution worth making.
### What stock hosts cost us
| Limit | Effect | What can be done |
|---|---|---|
| A stream's fps is fixed at connect | Out-of-sight panels save decoder work through keyframe sampling, but the host keeps encoding and sending 60 fps | Patch ft-stream's copy of moonlight-common-c to drop hidden video packets early (saves Frame CPU). A "change rate" control message needs a host upstream; Vibepollo is the likeliest |
| One stream per display | Five ft-stream processes, connections, and client pairings on the Frame | Measure the CPU (S3) |
| One real display per Vibepollo install | A second real display needs a second host instance | Use virtual displays; S3 tries a second instance on Windows |
| Cursor drawn into the video | Lags the laser by a round trip; each mouse move over a still page becomes a video frame | Accept |
| Mac: one display | Sunshine's mouse reaches only the main display, and there are no virtual displays | Upstream fix in progress, then S4 |
| Mac encoder | Full-motion video on the super ultrawide at 60 fps is at the limit | Lower fps or size for that stream |
What Frametop maintains: ft-stream (pairing, decoder, keyframe sampling, input mapping, Remote Monitor launch and resume; the protocol itself is moonlight-common-c's), the remote screen type in ft-screens, the layout and settings changes, and host setup notes or scripts.
## Alternatives considered
- **Our own host streamer (ft-host)** on macOS, Windows, and Linux: capture and encode each display at a rate the headset can change mid-stream, real 1 Hz for hidden panels, a separate cursor, one connection per host, virtual displays built in. It solves every limit in the table above, but it means maintaining capture, encoding, transport, input, pairing, and virtual displays on three operating systems. Rejected for a free project (2026-10-06).
- **Moonlight-qt in a window on a Frametop screen.** Nothing to build, and a quick way to check that a host and pairing work. But each frame goes decoder → Moonlight's GL renderer → KWin → ft-screens: two GPU passes per display where ft-stream needs one. Hidden panels can't drop to 1 Hz, and FFmpeg's v4l2m2m path on this device hasn't been tried (its encoder segfaults). Useful as a baseline, not as the feature.
- **RDP (FreeRDP client, RDPGFX).** The best multi-monitor design on paper: one session, up to 16 monitors as separate surfaces, damage rectangles. But Windows' RDP host takes over the login session (the PC's own monitors lock) and runs at 30 fps by default, macOS has no RDP host, and FreeRDP decodes H.264 on the CPU or through VA-API, which the Frame lacks.
- **Parsec, Steam Remote Play, RustDesk, NoMachine, Selkies, Apple Screen Sharing.** No aarch64 Linux client with hardware decode, no per-monitor streams, a closed protocol, or a Mac-only client.
## Spikes before any Frametop change
Spikes on the Frame run through `frame-job --local` as a standalone test overlay, with the headset on or with frame-testbench holding its worn state and pose. None of them touches the live desktop. S3 and S4 also need the hosts set up.
| # | Question | Pass |
|---|---|---|
| S1 (done) | Does a decoded `Q08C` buffer show in a SteamVR overlay with correct colours? Test pattern clip, BT.709 limited range, then full range | Correct colours; under 3% of a core; no extra missed frames (`~/.cache/frametop-perf/drops`); the decoder's real output delay |
| S2 | Concurrency: the original five, the worst case (2x 5120x1440, 2x 3440x1440, 3024x1964), fed from files in real time at 60 fps | Admitted (92% declared); decode time per frame; SoC temperature and clocks over 10 minutes |
| S3 | Live streams from the test PC (Vibepollo): the main session on a real desk monitor plus two Remote Monitors, through moonlight-common-c + the S1 decoder; then keyframe sampling on a hidden 60 fps stream | CPU per stream at real bitrates; glass-to-glass latency; time to picture after `LiRequestIdrFrame()`; the host honours one IDR request a second; IDR size; Frame CPU for a hidden stream; each Remote Monitor's mouse lands on its own display; Resume after a dropped stream keeps the windows; whether a second host instance can stream the other real monitor |
| S4 | The Mac: one Sunshine instance on the main display. Once the mouse fix ships: one instance per display, then a BetterDisplay virtual display | Mouse and keyboard work; encode rate with full-motion video on the super ultrawide; `minimum_fps_target = 1` keeps a static display near 1 fps; later, all three stream at once with the mouse on the right display |
| S5 | The baseline: 5 native screens, one playing video, four static | The CPU, GPU time (DRM fdinfo), temperature, and missed-frame numbers that the remote version must match |
### S1 results (2026-10-06)
S1 passes, with one change to the plan: a GPU pass between the decoder and SteamVR.
`stream/spike/ft-dectest` (built by `stream/build.sh`) decodes a clip with the V4L2 stateful API and hands each picture to SteamVR with `ImportDmabuf`, as the remote screen type would. Clips: a colour test pattern (`stream/spike/pattern.py`) with a moving box, 10 s at 60 fps, encoded by NVENC HEVC on the test PC (`-preset p1 -tune ull -bf 0`, as Sunshine-style streaming), in BT.709 and BT.601, limited and full range. The in-headset runs used frame-testbench to hold the worn state and a still pose, with nobody wearing the headset.
**Decoding.**
| | 3440x1440, Q08C | 3024x1964, Q08C | 3440x1440, linear NV12 |
|---|---|---|---|
| Decoder's buffer (coded size) | 3456x1440, 7,557,120 bytes | 3072x1984, 9,191,424 bytes | 3456x1440, 7,467,008 bytes |
| SteamVR import | Works (NV12 + `QCOM_COMPRESSED`) | Works | Works (NV12, linear) |
| Decode time, steady | 2.1 ms avg, 2.7 ms worst | 2.6 ms avg, 3.3 ms worst | 2.2 ms avg, 2.8 ms worst |
| First picture after | 1 input frame | 1 input frame | 3 input frames |
| CPU at 60 fps, no conversion | 1.2% of a core | 1.5% of a core | 1.4% of a core |
- The buffer layout from `msm_media_info.h` (per plane: UBWC metadata, then pixels, each 4 KiB aligned; Y stride aligned to 128, heights to 32) matches the driver's size to the byte, so the import offsets are right. The visible size comes from `G_SELECTION`; the decoder pads the coded size.
- Q08C has no extra delay: each frame comes out as soon as it's decoded. Linear NV12 holds two more frames, so the remote screen type uses Q08C.
- The CPU figure is ft-dectest's own process (file reading, V4L2 queueing, the overlay call). Network receive is S3's.
**Colours.** `stream/spike/s1-colours.py` shows the clip head-locked, switches the panel between the decoded video and an RGB copy of the pattern every 2.5 s, grabs the headset view (`/dev/video99`) in both states, and compares each colour patch.
- Decoder buffers straight into SteamVR come out wrong. vrcompositor doesn't convert NV12: red carries Cr, green Y, blue Cb. 100% red (Y 63, Cb 102, Cr 240 in BT.709 limited range) showed as 239/62/102, and greys as a dull magenta.
- The decoder's RGBA formats don't help. It accepts `AB24` and `QC24` but writes a 10-bit UBWC YUV picture into them: 10,027,008 bytes used, exactly TP10 UBWC at 3456x1440, starting with UBWC metadata. It also reports their `bytesperline` in pixels.
- A GPU pass gets them right (`ft-dectest --convert 709|601 --range tv|pc`). GLES samples the decoder's buffer through an external texture with the matrix and range as EGL hints (`EGL_YUV_COLOR_SPACE_HINT_EXT`, `EGL_SAMPLE_RANGE_HINT_EXT`). It draws into a ring of 3 RGBA buffers (UBWC, `QCOM_COMPRESSED`) that SteamVR imported once, set up as in `screens/handcut.cpp`.
| Clip, converted with its own matrix and range | Mean error | Worst patch |
|---|---|---|
| 3440x1440, BT.709 limited | 1.4 | 4.0 |
| 3440x1440, BT.709 full | 1.2 | 4.0 |
| 3440x1440, BT.601 limited | 0.2 | 2.4 |
| 3440x1440, BT.601 full | 0.3 | 2.9 |
| 3024x1964, BT.709 limited | 1.3 | 4.0 |
Errors are on the 0-255 scale, over 39-40 patches (27 for 3024x1964, where less of the panel is in view). The near-black steps (0-30) and near-white steps (225-255) all stay apart, so nothing is crushed or clipped. BT.709 sits about 3 low in red throughout, probably a rounding difference between ffmpeg's and Mesa's coefficients. That isn't visible.
**Cost of the pass**, 3440x1440 at 60 fps:
| Measure | Result |
|---|---|
| GPU work | About 118,000 cycles a frame: 0.13 ms at the 903 MHz top clock. With the headset idle, the GPU ran at about 230 MHz and was busy 0.51 ms a frame (DRM fdinfo) |
| Time until the GPU is done | 1.1-1.4 ms avg, 2-6 ms worst |
| CPU, whole ft-dectest process | 2.2-3.5% of a core, up from 1.2% |
| Missed compositor frames, 30 s at 90 Hz | 0 of 2,701 with the stream shown, 0 of 2,700 idle |
- The CPU is at the 3% pass line. The increase is the GL driver plus the `glFinish` wait; ft-stream should try waiting on a sync file in its poll loop instead.
- The five-display worst case (89% of the decoder) is about 360 converted 3440x1440-sized frames a second. That's about 5% of the GPU at top clock, and S2 measures it.
- frame-testbench held the pose still and nobody wore the headset, so tracking and eye-tracking load may differ from a real session. S5 rechecks missed frames with the headset worn.
The benchmark is S5 against the same scene on remote displays: total Frame CPU (Frametop + ft-stream), GPU time, missed frames, and SoC temperature all at or below native.
### S3 results so far (2026-10-07)
`stream/spike/ft-streamtest.cpp` is a Moonlight client for one display: moonlight-common-c and libgamestream from moonlight-embedded (pinned in `stream/build.sh`), S1's decoder and GPU pass (`spike/iris.h`), and a SteamVR overlay. It pairs through the host's API token, launches the primary app or a Remote Monitor, and reports every 2 s. Tests ran from the test PC over the LAN, with the headset held by frame-testbench.
**The primary display (5120x1440 at 60 fps, HEVC, 50 Mbit/s asked).** It works end to end. The picture in the headset is right, colours included. The test PC's primary is an HDR monitor, and Vibepollo sends it as SDR Rec. 709, as asked.
| | Shown | Hidden (keyframe sampling) |
|---|---|---|
| Frames received | 60 fps | 60 fps |
| Frames decoded and shown | 60 fps | 1 fps |
| Network | 12-14 Mbit/s | the same |
| ft-streamtest CPU | 4.2-4.6% of a core | 1.5-1.7% of a core |
- Time from a frame's first packet to the panel: 5.1-5.4 ms on average (worst about 12 ms). Of that, decoding takes 3.7 ms, and receiving the whole frame 0.1 ms. The host reports 3.2 ms from capture to encoded, and half the round trip is 1.5-2.5 ms. So a frame reaches the panel about 10 ms after the host captures it, before vrcompositor shows it. Glass to glass isn't measured yet.
- Connecting took 225 ms, and the first picture showed 0.5 s after the launch.
- Keyframe sampling: the host answered all 14 IDR requests, one a second. Each IDR arrived 18 ms after the request (worst 25 ms), reached the panel at 25 ms (worst 32 ms), and was about 81 KB for this desktop. Going back to full rate took 17 ms.
- No packets were lost.
- The test PC's desktop has an animated wallpaper, so the stream ran at 60 fps throughout, though Vibepollo applied a 0.5 fps floor. A still wallpaper would let an idle display drop to about one frame a second.
**Vibepollo bug: a stale display slot blocks Remote Monitors.** The first Remote Monitor launch failed with 503, "The composed display topology did not apply", for 40 s of retries. Vibepollo still held a display slot for the Mac's earlier "Desktop" session (`/api/clients/display-layout`: the Mac as a client node, its runtime "retryable" with the lease held), though that session's virtual display was gone. Composing the layout includes every held slot, and the Mac's can't be resolved to a device, so every composition fails. The Mac's session had been cut by a Vibepollo restart and reconnected before it ended. The failed launch still created frametop-2's virtual display, and "Disconnect Monitor" then reported success without removing it, because the launch never finished. A Vibepollo restart clears both. ft-stream must detect this state, a 503 that persists, and report it rather than retry forever. It's worth reporting upstream with the steps that cause it.
**Remote Monitor (frametop-2, 3440x1440 at 60 fps), after a Vibepollo restart.** The launch took 2.3 s (Vibepollo makes the virtual display, then answers), and the first picture came 0.9 s after connecting. The picture was right. An empty, still monitor dropped to 1 fps at once: 0.1 Mbit/s, 0.6% of a core, about 4-5 ms from first packet to panel. Windows moved a window it remembered for that spot onto the new monitor, so a Remote Monitor can take windows off the real screens without being asked.
**Vibepollo bug: a dropped Remote Monitor reset the host's real monitor layout.** The test killed ft-streamtest to imitate a dropped connection. Vibepollo kept the Remote Monitor for Resume, as configured, and recomposed the display layout. Windows refused it (`SetDisplayConfig`: ERROR_INVALID_PARAMETER, "failed to move device ... to new origin" for the monitor above the primary). Vibepollo's recovery (`SDC_USE_DATABASE_CURRENT`, a topology jog, `CDS_RESET`) then left Windows with a different layout: the monitor above became the primary, beside the old primary. "Disconnect Monitor" again reported success and did nothing. A Vibepollo restart was needed again.
One more defect that may have contributed: `GET /api/clients/display-layout` rebuilds Vibepollo's record of the real monitors without their positions or modes (every monitor at 0,0, 1920x1080; `refresh_remote_display_physical_baseline` in `confighttp.cpp`). A layout composed from that record stacks the monitors on one another. Launching a Remote Monitor reads the real positions again (`refresh_remote_monitor_baseline` in `nvhttp.cpp`), and the last such call came before the launch here, so this isn't proven to be the cause. Frametop must not call that endpoint until it's fixed.
So far, Vibepollo 2.0.0's Remote Monitors aren't reliable on the test PC: a stale slot blocks them, a drop can rearrange the host's real monitors, and "Disconnect Monitor" can't release a monitor whose ownership was lost. These go upstream with logs before Frametop depends on them. Further Remote Monitor tests on the test PC need the user's go-ahead, since they can move the user's own screens.
**The cause, and a fix (frametop-vibepollo, 2026-10-07).** All three Remote Monitor failures come from one call. When a display role ends, Vibepollo's coordinator (`src/remote_display_topology.cpp`) first recomposes the topology without the departing display, and only then removes it. On the test PC, Windows refuses that composed `SetDisplayConfig`. Then:
- libdisplaydevice's automatic recovery (`SDC_USE_DATABASE_CURRENT`, a topology jog, `CDS_RESET`) rearranges the real monitors;
- the release rolls back, so the virtual display stays attached and "Disconnect Monitor" does nothing;
- for a normal game (the Mac's session), the per-client identity stays held, and every later composition fails on it.
Removing the virtual display alone, as a Vibepollo restart does, left the layout intact. The fork [Frametop/frametop-vibepollo](https://github.com/Frametop/frametop-vibepollo) (GPL-3.0, like Vibepollo), on branch `fix/windows-remote-monitor-release`, changes two things on Windows:
- The coordinator removes the departing display first and recomposes only if another owned display remains (`retire_before_recompose`, set by the Windows runtime). A normal game's identity is released even when its display is already gone. Linux keeps the old order, which exists so KWin never has zero outputs.
- Composed layout applies run with display recovery off, so a refused layout can't reset the host's arrangement.
Four new unit tests cover it, and the coordinator's 40 tests pass. The plan is to offer the fix upstream once it's proven on the test PC. Frametop stays MIT: it only talks to the host over the network.
**The fix works on the test PC (2026-10-07).** The dev build, `2.0.0` plus the fix, was built on the test PC with MSYS2 (UCRT64, as the CI does, without WebRTC, drivers or packaging; `D:\vp-build`). It links only Windows system DLLs, so it drops into the install as a plain exe swap (`D:\vp-build\deploy.ps1`; the original is kept as `sunshine.exe.2.0.0-original`, and `-Restore` puts it back). Three Remote Monitor cycles left the real layout exactly as it was, with no layout re-apply and no `SetDisplayConfig` errors in Vibepollo's log: a clean end, a killed client, and a relaunch after the kill. Before the fix, the clean end reset the layout twice in a row. Restarting Vibepollo for the deploy also ended a leftover Mac session cleanly and brought the real monitors back.
A dropped connection removes the Remote Monitor at once (Vibepollo saw the disconnect within 3 s), even with `remote_monitor_disconnect_on_stream_end = disabled`. With `remote_monitor_disconnect_on_client_disconnect = enabled`, a lost connection counts as a client disconnect. Keeping a monitor and its windows across a Wi-Fi drop would need that setting off, and Frametop releasing monitors explicitly with "Disconnect Monitor". That's still to test.
The development loop is an incremental Ninja build on the test PC (only changed files recompile), the exe swap through the admin SSH login, and `ft-streamtest`: a few minutes per change. Pure logic gets unit tests on the Frame.
**Real displays, several at once (frametop-vibepollo, 2026-10-07).** A new hidden control, "Frametop display" (id 2147483521), streams one of the host's existing displays, named by the launch argument `frametopDisplay` (a device id from `/api/display-devices`). The session captures that output exactly, the way a Remote Monitor captures its virtual display, but nothing is created and the layout is never touched. Each client holds one such stream, and the main app stays free for the Deck and the Mac. On the test PC:
- both real monitors streamed at once (OLED at 5120x1440, LC34G55T at 3440x1440), each to its own client, at about 0.6% of a core each while idle;
- a real display and a Remote Monitor streamed side by side, and the layout was unchanged afterwards.
With that, Frametop no longer needs the "Frametop primary display" app; every real display is captured the same way.
**Vibepollo bug: one idle HTTPS client froze the API for everyone.** While any stream ran, every other client's HTTPS requests (serverinfo, pairing, launch) waited until it ended, so a second display couldn't start. The stock 2.0.0 build did the same. A stack dump (MSYS2 gdb attached to the service) showed the HTTPS server's only I/O thread blocked in a synchronous TLS shutdown: `SunshineHTTPS`'s destructor (`nvhttp.h`) sends close_notify and then waits for the client's. libgamestream leaves its connection idle in curl's cache (it forbids reuse only on FreeBSD), so the wait lasted as long as the streaming client process. Fixed on both sides:
- the fork marks the client's close_notify as received before shutting down, so nothing waits (a deliberately idle client no longer delays another client's request: 0.13 s);
- `ft-streamtest` drops libgamestream's curl handle after every request, which ft-stream must do too.
**The 3D mouse, and dragging windows between displays (2026-10-07).** The setup had three panels from the test PC: the OLED and the LC34G55T as Frametop displays, and a 2560x1440 Remote Monitor. frame-testbench held the headset, and the 3D mouse was driven through `@ft_pointer_helper`. The 3D mouse moved and clicked the test PC's cursor on every panel.
The first version clamped every move to its own panel. A drag from the Remote Monitor to the OLED left the window below the Remote Monitor's bottom edge, mostly off screen. Removing the Remote Monitor brought it back to the OLED. Drags between monitors that touch in Windows' layout seemed to work, but only because the cursor, pinned at the shared edge, left the window hanging across it. `--input-log` showed why. With the button held, SteamVR kept sending the moves to the panel where the press began (for example -213,2093 on the 2560x1440 panel), and sent the panel under the laser no events at all. The release also went to the first panel, off its edge.
The spike's fix is the routing described under "Input", with one difference: each panel is its own process. The panel that got the press publishes the laser's device in a small shared file (`/dev/shm/frametop-streamtest-drag`). Every other panel hit-tests that device's ray against itself and moves the host's cursor while the ray hits it. When hovering, the panel's own hit test matched SteamVR's coordinates exactly, so the ray origin and the UV orientation (bottom-left origin, like the mouse events) are right. With the fix, three drags worked:
- Remote Monitor to OLED;
- OLED to Remote Monitor;
- Remote Monitor to LC34G55T.
In each, the window followed the laser across panels and stayed where it was dropped. The release still went through the first panel's connection and landed right every time.
Windows' "Remember window locations based on monitor connection" moves windows on its own. When a Remote Monitor appears, Windows puts back windows it remembers there, and when it goes, they move to a real monitor.
## In ft-screens (2026-10-07)
The user decided remote displays are Frametop displays, with the same controls as the desktop's screens and their places saved in profiles. The spike's standalone overlays are replaced.
**What's built (branch `remote-displays`, uncommitted):**
- `stream/ft-stream.cpp` is the per-display helper. It pairs, launches and decodes, and converts into a ring of three RGBA buffers. It hands their dmabufs to ft-screens once, then says which buffer holds each new picture. The protocol is in its header comment: frames one way, then release, attention, pointer, keys and blur the other way. `stream/host.cpp` holds the pairing and launch code ft-streamtest had.
- `screens/remote.c` starts one ft-stream per remote screen with a socket pair, restarts a stream that ends (2 s, then backing off to 30 s), shows its frames through `ft_vr_screen_present`, and sends it the panel's input and attention. Commands: `remote <N> start <client> <host> <app> <W>x<H> <fps> <kbit/s> <metres> [label]` (`ok restored` when its panel is back where it was, below), `remote <N> stop`, `remote <N> info` (the stream's whole state, such as `lost can't connect`) and `remotes`. Remote screens are numbered from 101, so every other command (`place`, `width`, `curve`, `pin`, `get`, `conceal`) works on them as it is. `screens` leaves them out, because ft-layout, Display Settings and ft-floatd count KWin's outputs with it.
- `vr.cpp` gives a remote screen a screen's panel (`frametop.remote.N`) with its controls. A press on a remote screen dragged onto another one is routed there (`UpdateRemoteDrag`). SteamVR keeps sending the moves to the panel the press began on, as if its surface went on past its edges. So every tick the pressing laser is hit-tested against all the remote screens, and the nearest one it meets takes the moves. On the panel the press began on, SteamVR's own moves count only while that panel is the nearest. Before this (2026-10-07), that panel's carried-on surface passed in front of or behind the other panel, and its moves pulled the host's pointer back: drags across worked only some of the time. The release comes up where the host's pointer is, through the stream the press went down on (Vibepollo takes a release only from the client that pressed).
- `compositor.c` sends a remote screen's pointer events to its stream. A click on a remote screen takes the typing there (input relay keys and our VR keyboard), and leaving releases the keys it held (`blur`).
- `--beside` runs a second ft-screens next to the desktop for tests. It shows remote screens only and sends nothing to the input relay, ft-floatd or ft-layout.
- ft-pointer counts `frametop.remote.` panels as screens, and ft-gaze hit-tests them too (`remotes`, then `get N`). Without that, ft-gazed dropped looks below the keyboard pitch (-20°) on a low remote panel, as if you were looking at the keyboard.
**Tested through frame-testbench, with a `--beside` instance:**
- the OLED and a Remote Monitor as panels;
- Win+R, `notepad` and Enter typed through ft-screens' key path started Notepad on the test PC;
- Notepad dragged from the OLED panel to the Remote Monitor's stayed where it was dropped;
- the grab bar moves a remote panel;
- stopping the instance released the Remote Monitor.
**SteamVR's overlay limit.** SteamVR allows 128 overlays in the whole system (`k_unMaxOverlayCount`), its own included. Each Frametop panel takes six or seven with its controls. The desktop's floating-window slots made all of theirs at start: eight empty slots held 56. With three screens, a third remote screen got `VROverlayError_OverlayLimitExceeded`. A floating window's panel and controls are now made when a window floats on it, and destroyed when it docks (`EnsureFloatPanel`, `DropFloatPanel`). This needs a test on the live desktop. The controls of the other panels stay up, invisible until a laser comes near, because SteamVR's laser hover is what brings them in.
**Layouts, profiles and settings (2026-10-07, same branch):**
- `frametop-layout.json` has a `hosts` list. Each host has its displays: id, paired client, what it streams, label, stream size, fps, bitrate, and a screen's place, width, curve, pin and hidden flag. Each display keeps its screen number (101 and up), so the numbers don't shift when one is removed.
- ft-layout starts the remote displays' streams on every apply and at desktop start, even without a head pose, and places them when there is one. Displays without a place go in a row above the screens.
- `capture` and `save` record their places.
- A profile keeps the displays connected when it's saved, like the apps open then: their places and hidden state by id (`profiles[NAME]["remote"]`). Opening it (`use`, `open`, desktop start), or arranging (`apply`, Reset Screen Layout) while it's the profile in use, connects each of them whose host answers on Vibepollo's Web UI port, at its address or its dongle's as its route allows, and puts it back where it was saved. A host that doesn't answer is skipped, and its displays stay disconnected. Displays the profile doesn't have stay as they are: a profile connects, but never disconnects (user decision, 2026-10-08).
- `hide`/`show N` take their numbers.
- `ft-layout remote list|monitors|add|set|connect|disconnect|remove`: `add` pairs the display's client with the host's token, and `remove` unpairs it (the host stays). `disconnect` sets the display's `off` flag and stops its stream; apply, profiles and desktop start skip it until `connect`.
- ft-screens starts a host's streams one after another. Three started at once made Vibepollo refuse some ("Another stream operation is still running"), and the captures that started while the Remote Monitor appeared got no picture.
- Frametop Remote Displays (`remote-displays/`) is the app for them. It was a Display Settings tab at first; the user wanted an app of its own for the connections. Display Settings' Screens page has a button that opens it.
- Add a host with its token (written to `~/.local/share/frametop-stream/hosts/ADDRESS.token`, mode 0600). Whether it answers on its Web UI port (47990) is checked every 15 s.
- Add a display: one of the host's monitors from its API, or a virtual one.
- Per display: Connected, Shown, stream size, fps and bitrate (the stream starts over), width in VR, remove. Each host has a Connected switch for all of its displays. A lost stream shows why (`remote N info`).
- A disconnected display comes back where it was. ft-screens keeps a stopped remote panel's place, width, curve, pin and hidden state for the rest of its run (`Parked` in `vr.cpp`), and the next start of that screen restores them and replies `ok restored`. Otherwise ft-layout places it from the head, or with the headset off, from where the last arrangement was made, worked out from screen 1's place (`remote_anchor`).
- A vrcompositor crash: ft-screens freed a remote panel's texture imports while the panel still showed one. That happened at quit, at `remote stop`, and when a stream started over. vrcompositor crashed drawing it as the headset left standby. `ft_vr_forget` now clears a panel's texture before its import goes, and remote panels are destroyed before their buffers.
**vrcompositor crashed again at ft-screens quit (15:42, SIGBUS), and at 14:34.** Both times remote screens were up and the headset was in standby. SteamVR logs "leaving standby" within 2 ms of ft-screens disconnecting, so the compositor wakes in the middle of our cleanup, and it drew a texture that was already freed. The crash took the gamescope session and Steam down with it. The restart at 15:39 under the same conditions didn't crash, so it's a race. Hardened, not yet verified: ft-screens now tears SteamVR down first, while every buffer the panels show still exists (before KWin and the streams go). It clears the panels' textures, destroys the overlays, waits 200 ms, and only then releases the imports; nothing calls SteamVR after `VR_Shutdown`. Until that's checked, restart the desktop only with the compositor awake.
**Frozen panels, out-of-sight rate, sound (2026-10-07, after the user's test):**
- The OLED froze while a video played on it: frames came in at 59 fps and none were shown. ft-stream's message loop asked ft-screens' socket once more after the queue ran dry, to see whether it had closed, and lost whatever came in between. A lost `release` kept one of the three ring buffers from ft-stream for good, and after three nothing could be shown. Every pointer move is a message, so drags made it likely. (A lost button or key release would have stuck on the host the same way.) Fixed; ft-stream also takes the ring back if none of it comes back for 2 s.
- Out of sight, a stream used to decode only keyframes, one asked for each second. The user found that too aggressive. Every frame is now decoded, and 10 a second shown (`kHiddenFps`); coming back into view is immediate, with no keyframe requests (those are big frames, and the link already loses some).
- ft-stream ignored SIGTERM, ft-screens' death signal included: posix_spawn passed on ft-screens' blocked signals (its event loop takes SIGTERM, SIGINT and SIGCHLD through a signalfd). ft-screens now spawns it with none blocked, and ft-stream clears its mask too.
- Sound: ft-stream plays the host's sound (Opus, then PipeWire's PulseAudio server through libpulse-simple, about 40 ms queued). One stream per host plays it, the one holding `$XDG_RUNTIME_DIR/frametop-audio-HOST.lock`; the others try every 3 s, so another takes over when it ends. The host keeps playing its sound too. The test PC sent none: `remote_monitor_mute_audio = enabled` (our setup script) mutes the monitor and display roles. Turning it off needs an admin change on the test PC and a Vibepollo restart.
**Where the lost frames went, and the dongle (2026-10-07, ~16:40):**
- With the streams on the home network, about one frame every 2 s (all three together) was unrecoverable: bursts of 5 to 14 packets of one frame lost, beyond what FEC (Vibepollo's default 20%) repairs, and each loss then waited for a keyframe.
- Not on the Frame: UDP `RcvbufErrors` and `InErrors` stayed 0, wlan0 rx drops 0, the driver's misc drops +6 in 90 s. Not on the test PC: I225-V outbound discards and errors 0. The Frame's link was fine (-28 dBm, 2.1 Gbit/s, power save off). So the router loses them, as `~/.cache/wifitest` found for Steam Link VR (2026-10-01).
- Same burst test (vrsim, 50 Mbit/s at 60 fps, 20 s) from the test PC: through the router 0.33% of packets and 21 of 1200 frames lost (worst frame 21 ms); over the test PC's Steam Link dongle on the Frame's hotspot (the PC on the Frame's hotspot, the Frame at 10.35.78.1), none (worst 5.6 ms).
- So a host can go over its dongle. Its entry in `frametop-layout.json` has `direct` (the dongle's address) and `route`: `auto` (the default: the dongle when its port 47989 answers within 300 ms, else the network), `network` or `dongle` (only: while it's down, the stream is "lost the dongle link is down" and ft-screens tries again). ft-stream reads them itself, so ft-screens passes the host's own address as before. Its state says which way it went (`live via dongle|network`, `remote N info`). Remote Displays has a Connection setting per host, the dongle's address, and Find, which looks for the host among the hotspot's clients (`/proc/net/arp`, device `wlanap`) with the same `<uniqueid>` in its serverinfo. `ft-layout remote host NAME route=... direct=...` saves it and starts the host's running streams over in place.
- On the dongle, none of the three streams lost a frame in 2 to 3 minutes each, all at 60 fps.
- Sound: `remote_monitor_mute_audio = disabled` on the test PC (backup `D:\remote-displays\vibepollo-setup\sunshine.conf.before-audio-20261007-163556`, ApolloService restarted). One stream plays it (`Remote display: ADDRESS` in PipeWire, through SteamOS's spatial filter chain to the headset's speakers). That stream uses about 5% of a core more than the others (240-sample `pa_simple_write`s; batching them would help).
- Seen on the way: a stream that started while Vibepollo restarted chose the network (the dongle didn't answer yet) and then got no video, only sound; starting it over fixed it. Starting one display's stream once ended another's ("lost", back 5 s later). Vibepollo takes about 5 s to answer serverinfo, so every stream start waits that long.
**Adding a computer: sign in instead of carrying a token (2026-10-07, ~17:00):**
- Before, a host's API token came from `make-frametop-token` run on the host (its Web UI login typed there), and the file had to reach the Frame by hand. Now Remote Displays does it. Add computer lists the Vibepollo and Sunshine computers that announce `_nvstream._tcp` over mDNS (`avahi-browse -rpt` on the host; a computer also seen on the hotspot, `wlanap`, has a dongle, and that address is kept as its `direct`), or takes an address. You sign in once with the host's Web UI user name and password. Frametop posts them (HTTP Basic) to `/api/token` for a token with only what it uses: `/api/pin` POST, `/api/clients/list` GET, `/api/clients/update` POST and `/api/display-devices` GET (not `/api/clients/display-layout`, which the old script allowed). It keeps the token, mode 0600, and forgets the password. The sign-in goes over the dongle when there is one. Then Add displays lists the host's monitors, all ticked but those already added, and a virtual display.
- Hosts without a dongle use the network, and their card says so, with Find a dongle; the Connection choices only show once a dongle is known. A laptop without one was the test: it's on the network and not on the hotspot.
- The Web UI's certificate is self-signed. Its public key is pinned at the first sign-in (`hosts/ADDRESS.pin`, `sha256//...` as curl takes it): ft-stream's API calls set `CURLOPT_PINNEDPUBLICKEY`, and a later sign-in refuses another key ("If Vibepollo was installed again, remove the host and add it again"). The test PC's key is the same on both paths. Checked: the right pin works, a wrong one fails. The test PC's existing token got its pin too.
- Checked without the password: discovery found the test PC (its LAN address and its dongle on the hotspot, marked added) and a laptop; a wrong password says "Wrong user name or password". A real sign-in is the user's to try (the password never goes through chat).
- Sign in again (on a host card) makes a new token; the old one stays in the host's Web UI under API Tokens until revoked there (Frametop's token can't revoke tokens).
**The PC side: Frametop host setup (2026-10-07, ~17:40):**
- `host/windows/Setup Frametop host.cmd` (it runs `frametop-host-setup.ps1` and asks for admin) does what the test PC got by hand, nothing else of its settings:
1. Vibepollo 2.0.0 with its own installer when it isn't there (downloaded from Nonary/Vibepollo's release, SHA-256 checked first; you click through it).
2. Frametop's build of `sunshine.exe` over the original (kept as `sunshine.exe.2.0.0-original`), SHA-256 checked: from `-FrametopBuild PATH|URL`, the script's `$BuildUrl` (the fork's release), or `sunshine-frametop.exe` next to it. Only over Vibepollo 2.0.0's own exe; another version stops it with a message. Without the build, virtual displays still work, but not the PC's own monitors.
3. `remote_monitor_mute_audio = disabled`, `remote_monitor_disconnect_on_client_disconnect = enabled`, `remote_monitor_disconnect_on_stream_end = disabled`. The rest of `sunshine.conf` stays as it is.
4. The Web UI login: keep the one there is, or set one (`sunshine.exe --creds`, the password typed into a hidden prompt and passed to it quoted).
5. Checks: an inbound firewall rule for `sunshine.exe` on every network type (added if missing; Windows puts a Steam Link dongle's network in Public), whether a Steam Link dongle (an adapter "For Valve") is connected, and that the Web UI answers. It ends with what to pick and sign in as on the Frame.
`-Check` only says what it would change, `-SkipLogin` leaves the login, `-Undo` puts back the original exe and the oldest backup of the settings. Backups and a log go to `%ProgramData%\Frametop`.
- Tested on the test PC (admin over SSH, `FRAMETOP_NO_PAUSE=1`): `-Check`; a run with nothing to change (no restart); `-Undo` (the original exe back) and a run with `-FrametopBuild D:\vp-build\src\build\sunshine.exe` (backed up, swapped, Vibepollo restarted, the streams came back on their own). Not tested: a PC without Vibepollo (the download and its installer), setting the login (needs the user at the PC), adding the firewall rule.
- Where other PCs get Frametop's build: the fork's GitHub releases, with the source as the release's tag (GPL-3.0). See below (2026-10-09).
- The Frame side: `install.sh` now builds ft-stream (`stream/build.sh`, step 7) with Remote Displays' menu entry, and `uninstall.sh` offers to delete `~/.local/share/frametop-stream` (the clients' keys, the hosts' tokens and pins) with the other settings.
- Linux hosts (Vibepollo's Arch package): a host setup like this one, later.
- After Vibepollo restarted, a stream checked the dongle in 300 ms, too soon, and went over the network; ft-stream now gives it a second.
**Frametop's build released (2026-10-09):**
- Release [`frametop-2.0.0-1`](https://github.com/Frametop/frametop-vibepollo/releases/tag/frametop-2.0.0-1) of Frametop/frametop-vibepollo has `sunshine.exe` and the Arch package (`pacman -U`), each with its `.sha256`. The fork's CI (`frametop-build.yml`, Depot's runners) builds them from the tag and publishes them; a `frametop-*` tag is what makes a release. The tag is the source.
- The host setup's `$BuildUrl` points at that `sunshine.exe`, so a PC needs only `host/windows`. It replaces the original, the hand-built `2f032252`, and the first CI build (`a84b6cfc`).
- New in it: a Frametop display stream that asks for SDR turns the display's HDR off while it captures it, and back on when the stream ends (with `dd_hdr_option` automatic, Vibepollo's default). The test PC's HDR monitor looked washed out in SDR: Vibepollo's conversion clips at 80 nits while Windows draws SDR content at its SDR white level (240 nits there), and scaling for that left the colours heavily oversaturated. Displays it turned off are listed in `config\frametop_display_hdr.json` until they're back on, so the next start turns them back on after a crash or a forced stop. On the test PC's dev build: HDR off 0.4 s after the stream started, 8-bit capture, back on at the end, off again on a reconnect.
- Tested on the test PC: with the first CI build installed and only `host/windows` in a folder, `-Check` downloaded the release and matched its SHA-256, and the run backed up, swapped and restarted Vibepollo; the Web UI answered.
**Tested in the live desktop (frame-testbench, the 3D mouse through `@ft_pointer_helper`, 2026-10-07):**
- An Explorer window carried by its title bar from the Remote Monitor to the OLED stayed there, and carried back, stayed there too. The log showed one change of screen each way.
- Disconnect, connect: the panel came back where it was, at its width. Connect in a new run with the headset off: placed from screen 1's anchor.
- Windows rescales a window that moves between displays with different scaling, so the point you held moves on the window. A second drag from the same spot can land on the address bar instead of the title bar. That's Windows, not the routing.
**Still to do:**
- A headset test of all of it in the live desktop:
- the row above the screens;
- moving a remote display and saving a profile, then `use` putting it back;
- adding a display from Remote Displays;
- drags across with a controller, and gaze on the remote panels.
- The shutdown hardening above, verified: restart the desktop with remote screens up and the headset in standby (a crash takes the gamescope session down, so only with the user's OK).
- The floating windows' panels made on demand, tested live.
- Sound in the headset, by ear: one copy, in time with the picture.
- Over the network, a lost frame still waits for a keyframe. Reference frame invalidation (moonlight-common-c's `CAPABILITY_REFERENCE_FRAME_INVALIDATION_HEVC`) would recover without one, if the decoder copes.
- Find the dongle again if its address changes (the hotspot's DHCP), and a host whose own address changes (the token and pin files are named by it).
- "Approve on the PC" instead of the password (our Vibepollo fork), later if wanted.
- A per-display mouse mode (relative, for games).
- A picture for a stream that's connecting or lost.
- The installer doesn't build ft-stream yet (`stream/build.sh`), and `uninstall.sh` leaves `~/.local/share/frametop-stream` (the client keys and host tokens).
- For the test, the pointer and gaze services run this branch's builds through drop-ins (`~/.config/systemd/user/frametop-{pointer,gaze}.service.d/remote-displays-worktree.conf`; `gaze/tracker/build` here links to the installed tracker's). Remove them when this is merged and installed.
## Open questions
- Which of the test PC's two desk monitors is the real one (the main session)?
- The Mac's chip (Max chips have two video encode engines) and BetterDisplay Pro matter only once the Mac goes past one display.
- Audio: none, the focused display's host, or a fixed one?
- Remote displays outside the desktop: should they also show over games, where the decoder is shared with Steam Link VR?
+72
View File
@@ -0,0 +1,72 @@
# Install with FrameDrop (proof of concept)
[FrameDrop](https://framedropvr.com) is a Windows app that sideloads onto a Steam Frame: it copies a build to the headset and adds it to the Steam library. Issue #25 asks for an "Install with FrameDrop" button. Frametop isn't an app FrameDrop can copy over as is: it installs user services, a SteamVR driver, and a container, and two optional parts need sudo. So FrameDrop installs a small installer instead. Playing "Frametop" from the library opens a window that asks what to install, and your password for the parts that need it, then installs and shows its progress. A release's Frametop.zip (about 1.1 GB, built by CI on a tag, see [pack/README.md](../pack/README.md), Releases) carries Frametop built, as an image, and installs it with `install-release.sh`: nothing compiles on the headset and nothing else downloads. The same zip works unpacked on the headset. A test zip (a few KB) clones Frametop with `get.sh` instead.
Nothing here is published yet: no release, no button.
## How FrameDrop installs a Linux zip
It uses Valve's SteamOS Devkit path: pair once with the headset's devkit service, then rsync the unpacked zip into `~/devkit-game/<name>` over SSH, and register it with Steam as a Devkit Game with a start command. `devkit.sh` here makes the same calls with Valve's devkit-utils, so all of this can be tested on the Frame without a PC.
## What the probe found (SteamOS 0.3.0, build 20260922.6101926)
`probe/probe.sh`, started as a Devkit Game, recorded:
- Devkit titles run on the host, not in a container, as user `steamos`, from a process tree that Steam's reaper owns. This held even with the compat tool set to `SteamLinuxRuntime_4-arm64`: Steam recorded the mapping and still ran it on the host.
- On the host, everything the installer needs works: git, curl to GitHub, podman (sees the `dev` container), `systemctl --user`, `systemd-run --user`, and GTK 4 with libadwaita.
- A GTK window opens in gamescope (an X11 window on `:1`, drawn through gamescope's Vulkan WSI).
- Steam puts its overlay in `LD_PRELOAD` and Steam runtime paths in `LD_LIBRARY_PATH` and `PATH`. Every host tool prints a preload error unless they're cleared.
- Inside the Steam Linux Runtime 4 container (started by hand with its `run` script), there's no git, podman, systemctl, or GTK, but `flatpak-spawn --host` runs commands on the host.
- Steam refuses Devkit Game names with a `-` ("missing/invalid arguments").
- Steam doesn't make the start command executable: without the exec bit, the title exits in a second and nothing runs.
## The installer
`installer/frametop-install.sh` is the start command.
1. In the container, it starts itself again on the host with `flatpak-spawn --host`.
2. It clears Steam's preload and library paths, and opens `installer/progress.py` (GTK 4 and libadwaita).
3. The window asks which optional parts to install: our own eye tracker (on by default) and the Bluetooth fixes. Both need sudo, so it asks for your SteamOS password and checks it with `sudo -v`. If your user has no password (SteamOS starts without one), it says how to set one and leaves both out.
4. It runs the zip's `install-release.sh --yes` (a test zip: `get.sh --yes`) in a transient user service, `frametop-framedrop-install`. The service is used because Steam ends the title's whole process tree when it's quit, and starts it with an OOM score of 900. Opened from a VR desktop (the zip unpacked by hand), it still uses the user's real bus and runtime folder for the service.
5. It follows the service's log, shows the steps as a progress bar, and reports the result. Closing it leaves the install running. Playing the title again reattaches.
`--yes` keeps the version that's installed, or installs stable, and skips the SteamVR restart. The window says to restart SteamVR.
### The password
FrameDrop has no way to pass a password along, and the zip is the same file for everyone, so the password is typed on the headset, in the window. In VR that means the window's own keypad (shown when Steam starts the window; its Keypad button shows or hides it), whose keys you click with the controller's laser. SteamVR's keyboard doesn't come up for the field by itself, and when it's opened (`steam://open/keyboard`) its keys don't reach the window (tested with frame-testbench, 2026-10-07: the field stayed empty, while laser clicks on the window's checkboxes worked). A Bluetooth keyboard types as usual. The password stays in the window's memory until the install ends:
- The service gets `SUDO_ASKPASS=installer/askpass`. When install.sh's sudo asks, askpass connects to a socket the window keeps in `/run/user/UID/frametop-install` (mode 0700), and the window answers only a process in the install's own service (checked by its peer credentials and cgroup). If it doesn't have the password yet (the window was opened again), it asks you for it, or you skip that part.
- The password is never written to a file, a log, the service's environment, or a command line. The window wipes its copy when the install ends or the window closes. Strings Python and GTK made from it along the way can't be wiped; they go with the process.
- With the window closed there's nobody to answer: sudo fails, install.sh says which parts it skipped, and playing Frametop again finishes them.
`--dry-run` unpacks into `~/.cache/frametop-framedrop/dry-run` and stops there without installing (`install-release.sh --unpack-only`, or `get.sh --clone-only`).
## Try it on the Frame
```
framedrop/build.sh # a test zip; or --image localhost/frametop:local --version 0.3.0-dev.1
unzip -q framedrop/build/Frametop.zip -d /tmp/fd
framedrop/devkit.sh add FrametopTest /tmp/fd/Frametop "./frametop-install.sh --dry-run"
framedrop/devkit.sh run FrametopTest # or Play it from the Steam library
framedrop/devkit.sh remove FrametopTest
framedrop/devkit.sh add FrametopProbe framedrop/probe "./probe.sh native" # the probe
```
The probe writes `~/.cache/frametop-framedrop/probe-native.log`, and the installer writes `~/.cache/frametop-framedrop/install.log`.
## Build the download
```
framedrop/build.sh --image REF --version V [--commit SHA] [--channel C] [ZIP_URL] # a release
framedrop/build.sh [ZIP_URL] # a test zip
```
This writes `framedrop/build/Frametop.zip` (reproducible), `frametop.framedrop.json` (FrameDrop's manifest with the zip's sha256), and `SHA256SUMS`. A release's zip has the image REF (`podman save`) with its `frametop-release.json` (`pack/release-info.py`) and `install-release.sh`; CI builds it on a tag (`.github/workflows/release.yml`). A test zip has `get.sh` instead. By default, `ZIP_URL` is the release's asset (`releases/download/vV/Frametop.zip`; for a test zip, a `framedrop-installer` release's). Each release carries its manifest, so the button's link can point at the newest stable one: `https://framedropvr.com/install?manifest=https://github.com/Frametop/frametop/releases/latest/download/frametop.framedrop.json` (the exact URL format is FrameDrop's to confirm).
## Open questions, for a test with FrameDrop on a Windows PC
- Does FrameDrop keep or set the exec bit on `frametop-install.sh`? A zip unpacked on Windows loses it, and without it nothing runs.
- What start command does FrameDrop pick for this zip, and which runtime?
- Does the manifest's `name` become the Devkit Game name? It has to stay free of `-`.
- Typing the password with the window's keypad, launched with Play from the library (frame-testbench reached the window only when started with devkit.sh run, where the systemui overlay hides its lower half).
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# Build Frametop's download: framedrop/build/Frametop.zip, a "Frametop" folder that FrameDrop
# installs from a PC, or that you unpack on the headset and run (frametop-install.sh). Also
# framedrop/build/frametop.framedrop.json, the manifest an "Install with FrameDrop" button
# points at, and SHA256SUMS. Same files in, same zip out.
#
# Usage: framedrop/build.sh --image REF --version V [--commit SHA] [--channel C] [ZIP_URL]
# framedrop/build.sh [ZIP_URL]
# --image REF a release: the image REF (built from this checkout, pack/Containerfile) goes
# in the zip as frametop-image.tar with frametop-release.json, and the
# installer installs it, built (pack/install-release.sh). About 1.1 GB.
# --version V the release's version (0.3.0, or 0.3.0-exp.1 for an experimental one)
# --commit SHA the commit it was built from (default: this checkout's HEAD)
# --channel C stable or experimental (default: experimental if V has a "-")
# Without --image, a few KB: the installer clones Frametop from GitHub (get.sh), for
# testing the installer itself.
# ZIP_URL where the zip will be downloaded from (default: the release's asset, or for
# a test zip, the framedrop-installer release)
set -euo pipefail
here=$(cd "$(dirname "$0")" && pwd)
repo=$(cd "$here/.." && pwd)
image= version= commit= channel=
while [ $# -gt 0 ]; do
case $1 in
--image) image=${2:?--image needs an image}; shift ;;
--version) version=${2:?--version needs a version}; shift ;;
--commit) commit=${2:?--commit needs a commit}; shift ;;
--channel) channel=${2:?--channel needs stable or experimental}; shift ;;
-*) echo "unknown option: $1" >&2; exit 2 ;;
*) break ;;
esac
shift
done
out=$here/build
rm -rf "$out"
mkdir -p "$out"
files=("$here/installer/frametop-install.sh" "$here/installer/progress.py" "$here/installer/askpass")
if [ -n "$image" ]; then
[ -n "$version" ] || { echo "--image needs --version" >&2; exit 2; }
commit=${commit:-$(git -C "$repo" rev-parse HEAD)}
url=${1:-https://github.com/Frametop/frametop/releases/download/v$version/Frametop.zip}
echo "saving $image"
podman save -q --format oci-archive -o "$out/frametop-image.tar" "$image"
python3 "$repo/pack/release-info.py" --image-file "$out/frametop-image.tar" --version "$version" \
--commit "$commit" ${channel:+--channel "$channel"} >"$out/frametop-release.json"
files+=("$repo/pack/install-release.sh" "$out/frametop-release.json" "$out/frametop-image.tar")
else
url=${1:-https://github.com/Frametop/frametop/releases/download/framedrop-installer/Frametop.zip}
files+=("$repo/get.sh")
fi
python3 - "$out/Frametop.zip" "${files[@]}" <<'EOF'
import os, shutil, sys, zipfile
dest, *files = sys.argv[1:]
with zipfile.ZipFile(dest, "w", allowZip64=True) as z:
for path in files:
name = os.path.basename(path)
info = zipfile.ZipInfo(f"Frametop/{name}", date_time=(2026, 1, 1, 0, 0, 0))
info.create_system = 3 # unix, so the permissions below count
info.external_attr = (0o100755 if os.access(path, os.X_OK) else 0o100644) << 16
# The image's layers are compressed already.
info.compress_type = zipfile.ZIP_STORED if name.endswith(".tar") else zipfile.ZIP_DEFLATED
info.file_size = os.path.getsize(path)
with open(path, "rb") as src, z.open(info, "w", force_zip64=True) as dst:
shutil.copyfileobj(src, dst, 1 << 20)
EOF
rm -f "$out/frametop-image.tar"
sha=$(sha256sum "$out/Frametop.zip" | cut -d' ' -f1)
python3 - "$url" "$sha" >"$out/frametop.framedrop.json" <<'EOF'
import json, sys
url, sha = sys.argv[1:]
print(json.dumps({
"schema": "framedrop.install/v1",
"name": "Frametop",
"files": [{"url": url, "sha256": sha}],
}, indent=2))
EOF
(cd "$out" && sha256sum Frametop.zip frametop.framedrop.json ${image:+frametop-release.json} >SHA256SUMS)
echo "built $out/Frametop.zip ($(du -h "$out/Frametop.zip" | cut -f1), sha256 $sha)"
echo " $out/frametop.framedrop.json, $out/SHA256SUMS"
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env bash
# Do on the Frame what FrameDrop does from a PC: copy a folder into ~/devkit-game/NAME and
# register it with Steam as a "Devkit Game" (Valve's devkit-utils, the same calls FrameDrop and
# the SteamOS Devkit Client make over SSH). For testing the FrameDrop installer and the probe
# without a PC. Steam must be running, and Developer Mode on.
#
# Usage: devkit.sh add NAME DIR COMMAND [--compat TOOL]
# devkit.sh run NAME # start it, as the library's Play button does
# devkit.sh remove NAME # delete the folder and the Steam entry
# devkit.sh list
#
# NAME can't contain "-": Steam answers "missing/invalid arguments". COMMAND is relative to
# the folder, like FrameDrop's start command ("./probe.sh native"). --compat sets the runtime
# (SteamLinuxRuntime_4-arm64, say); on SteamOS 0.3.0 Steam recorded it but still ran the
# title on the host. There's no environment option: this Steam ignores the devkit env file.
#
# devkit-utils is Valve's (LGPL, gitlab.steamos.cloud/devkit/steamos-devkit). FrameDrop and
# the devkit client copy it to ~/devkit-utils; if that isn't there, it's fetched at a pinned
# commit into ~/.cache/frametop-framedrop.
set -euo pipefail
devkit_rev=a00ceb7d91ea44a0c3e714a91a06417d6e5cdb33
cache=$HOME/.cache/frametop-framedrop
usage() { sed -n '7,10p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; }
utils() {
if [ -e "$HOME/devkit-utils/steam-client-create-shortcut" ]; then
echo "$HOME/devkit-utils"; return
fi
if [ ! -e "$cache/devkit-utils/steam-client-create-shortcut" ]; then
echo "fetching devkit-utils ($devkit_rev)" >&2
local tmp
tmp=$(mktemp -d)
git -C "$tmp" init -q
git -C "$tmp" fetch -q --depth 1 https://gitlab.steamos.cloud/devkit/steamos-devkit.git "$devkit_rev"
git -C "$tmp" checkout -q FETCH_HEAD
mkdir -p "$cache"
rm -rf "$cache/devkit-utils"
cp -r "$tmp/client/devkit-utils" "$cache/devkit-utils"
rm -rf "$tmp"
fi
echo "$cache/devkit-utils"
}
[ $# -ge 1 ] || usage
cmd=$1; shift
case $cmd in
add)
[ $# -ge 3 ] || usage
name=$1 src=$2 start=$3; shift 3
case $name in *-*) echo "NAME can't contain '-' (Steam refuses it)" >&2; exit 2 ;; esac
compat=
while [ $# -gt 0 ]; do
case $1 in
--compat) compat=${2:?}; shift ;;
*) usage ;;
esac
shift
done
u=$(utils)
dir=$(python3 "$u/steamos-prepare-upload" --gameid "$name" | python3 -c 'import json,sys; print(json.load(sys.stdin)["directory"])')
# FrameDrop rsyncs the unpacked zip; --delete as a clean upload would.
rsync -a --delete "$src/" "$dir/"
parms=$(python3 - "$name" "$dir" "$start" "$compat" <<'EOF'
import json, sys
name, directory, start, compat = sys.argv[1:]
print(json.dumps({
"gameid": name, "directory": directory,
"argv": [start], "env": {},
"settings": {"steam_play": "0", "compat_tool": compat},
"force_appid": "", "lepton_args": "",
}))
EOF
)
res=$(python3 "$u/steam-client-create-shortcut" --parms "$parms" | tail -1)
echo "Steam: $res"
case $res in *'"error"'*) exit 1 ;; esac
echo "registered $name ($dir)"
;;
run)
[ $# -eq 1 ] || usage
python3 "$(utils)/steam-devkit-rpc" run-game "gameid=$1" | tail -1
echo
;;
remove)
[ $# -eq 1 ] || usage
python3 "$(utils)/steamos-delete" --delete-title "$1"
rm -f "$HOME/devkit-game/$1"-*.json
;;
list)
python3 "$(utils)/steam-devkit-rpc" list-shortcuts
;;
*) usage ;;
esac
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/python3
"""SUDO_ASKPASS for the FrameDrop install: sudo runs this for the password, and it asks the
install window (progress.py) for it, over the window's socket in XDG_RUNTIME_DIR. The window
answers only programs in the install's own service, and asks you on the headset if it doesn't
have the password yet. The password goes to sudo on stdout and nowhere else.
With the window closed there's nobody to ask: this fails, so sudo does, and install.sh says
which parts it skipped. Playing Frametop again reopens the window and the install.
"""
import os
import socket
import sys
path = os.environ.get("FRAMETOP_ASKPASS_SOCKET", f"/run/user/{os.getuid()}/frametop-install/askpass")
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.settimeout(600) # time for you to type it, if the window has to ask
try:
s.connect(path) # connecting is the question: the window answers, then closes
answer = bytearray()
while chunk := s.recv(4096):
answer += chunk
except OSError as e:
print(f"askpass: the install window isn't answering ({e.strerror or e}): play Frametop again",
file=sys.stderr)
sys.exit(1)
if not answer:
sys.exit(1) # you cancelled
sys.stdout.buffer.write(bytes(answer) + b"\n")
answer[:] = bytes(len(answer))
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
# Frametop's installer, in a release's Frametop.zip: the start command of the "Frametop" title
# FrameDrop puts in the Steam library, or what you run after unpacking the zip on the headset.
# It opens the install window (progress.py), which asks what to install, and your password
# for the parts that need sudo, then installs Frametop (or updates it) in a service of its own
# and shows its progress. A release's zip installs its own image, built
# (install-release.sh); a test zip clones Frametop from GitHub (get.sh).
#
# Usage: frametop-install.sh [--dry-run]
# --dry-run unpack into ~/.cache/frametop-framedrop/dry-run and stop there, without
# installing, for testing this flow
set -u
here=$(cd "$(dirname "$0")" && pwd)
self=$here/$(basename "$0")
# A Linux title can be started in the Steam Linux Runtime container, which has no git, podman,
# systemctl, or GTK. Start this again on the host. (SteamOS 0.3.0 runs devkit titles on the
# host anyway; this is for when FrameDrop or Steam picks the runtime.)
if [ -e /run/pressure-vessel ]; then
exec flatpak-spawn --host --watch-bus --env=DISPLAY="${DISPLAY-}" \
--env=XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR-}" --env=SteamAppId="${SteamAppId-}" \
--env=ENABLE_GAMESCOPE_WSI="${ENABLE_GAMESCOPE_WSI-}" /usr/bin/bash "$self" "$@"
fi
# Steam adds its overlay and runtime libraries to every child; host tools don't want them.
unset LD_PRELOAD LD_LIBRARY_PATH
export PATH=/usr/local/bin:/usr/bin:/bin
exec /usr/bin/python3 "$here/progress.py" "$here" "$@"
+543
View File
@@ -0,0 +1,543 @@
#!/usr/bin/env python3
"""The FrameDrop installer's window: what to install, your password for the parts that need it,
then the install's progress.
Usage: progress.py DIR [--dry-run]
DIR the installer's folder. A release's Frametop.zip has install-release.sh, the
image (frametop-image.tar), and frametop-release.json: it installs that, built.
A test zip has get.sh instead, which clones Frametop from GitHub.
--dry-run unpack into ~/.cache/frametop-framedrop/dry-run and stop there, without
installing, for testing this flow
The install runs in a user service of its own (UNIT): Steam ends the title's whole process
tree when it's quit, and starts it with a high OOM score. Closing the window doesn't stop the
install; playing the title again reattaches to it.
Our eye tracker and the Bluetooth fixes need sudo. The window asks for your password on the
headset, checks it with sudo, and keeps it in this process's memory until the install ends.
sudo in the install gets it through askpass (SUDO_ASKPASS), from a socket in a folder only you
can open, and the window answers only programs in the install's service. It's never written to
a file, a log, the service's environment, or a command line.
"""
import json
import os
import pwd
import re
import socket
import struct
import subprocess
import sys
import threading
from pathlib import Path
import gi
gi.require_version("Gtk", "4.0")
gi.require_version("Adw", "1")
from gi.repository import Adw, GLib, Gtk, Pango # noqa: E402
HERE = Path(sys.argv[1]).resolve() if len(sys.argv) > 1 else Path(__file__).resolve().parent
DRY_RUN = "--dry-run" in sys.argv[2:]
UNIT = "frametop-framedrop-install"
HOME = Path.home()
STATE = HOME / ".cache" / "frametop-framedrop"
LOG = STATE / "install.log"
RELEASE = (HERE / "install-release.sh").exists() and (HERE / "frametop-image.tar").exists()
SOCK_DIR = Path(f"/run/user/{os.getuid()}/frametop-install")
SOCK = SOCK_DIR / "askpass"
ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
# get.sh and install.sh mark each step with "== N/10 what it is"
STEP = re.compile(r"^== (\d+)/(\d+) (.*)$")
DONE_TEXT = ("Frametop is installed. Restart SteamVR once, or reboot the headset, so it loads "
"Frametop's driver. Then open Launch a program, then Desktop.")
def host_env():
"""The user's real runtime folder and bus, for systemctl and systemd-run: a window opened
from a VR desktop's Dolphin or Konsole has that session's own. GTK keeps the session's."""
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
env["DBUS_SESSION_BUS_ADDRESS"] = f"unix:path=/run/user/{os.getuid()}/bus"
return env
def unit_state():
out = subprocess.run(["systemctl", "--user", "show", "-P", "ActiveState", UNIT],
capture_output=True, text=True, env=host_env()).stdout.strip()
return out or "inactive"
def release_version():
try:
return json.loads((HERE / "frametop-release.json").read_text())["version"]
except (OSError, ValueError, KeyError, TypeError):
return ""
def password_set():
"""Does this user have a password sudo can take? SteamOS starts without one."""
user = pwd.getpwuid(os.getuid()).pw_name
out = subprocess.run(["passwd", "-S", user], capture_output=True, text=True).stdout.split()
return len(out) < 2 or out[1] == "P" # P: usable; NP: none; L: locked
def check_password(pw):
"""Does sudo take it? Checked with cached credentials ignored, and forgotten after."""
try:
r = subprocess.run(["sudo", "-S", "-k", "-v", "-p", ""], input=bytes(pw) + b"\n",
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=30)
except (OSError, subprocess.TimeoutExpired):
return False
subprocess.run(["sudo", "-k"], stdin=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
return r.returncode == 0
def in_unit(pid):
"""Is this process in the install's service?"""
try:
with open(f"/proc/{pid}/cgroup") as f:
return any(line.rstrip("\n").endswith(f"/{UNIT}.service") for line in f)
except OSError:
return False
def install_command(eye_tracker, bluetooth):
if RELEASE:
args = [str(HERE / "install-release.sh"), "--yes"]
if DRY_RUN:
args += ["--unpack-only", "--dir", str(STATE / "dry-run")]
else:
args = [str(HERE / "get.sh"), "--yes"]
if DRY_RUN:
args += ["--clone-only", "--dir", str(STATE / "dry-run")]
if not eye_tracker:
args.append("--no-eye-tracker")
if bluetooth:
args.append("--bluetooth")
return args
def start_unit(args, askpass):
env = host_env()
subprocess.run(["systemctl", "--user", "stop", UNIT], stderr=subprocess.DEVNULL, env=env)
subprocess.run(["systemctl", "--user", "reset-failed", UNIT], stderr=subprocess.DEVNULL, env=env)
STATE.mkdir(parents=True, exist_ok=True)
LOG.write_bytes(b"")
cmd = ["systemd-run", "--user", f"--unit={UNIT}", "--description=Frametop install (FrameDrop)",
"--property=Type=oneshot", "--property=RemainAfterExit=yes",
f"--property=StandardOutput=truncate:{LOG}", "--property=StandardError=inherit",
f"--setenv=HOME={HOME}", f"--setenv=PATH={os.environ.get('PATH', '/usr/bin:/bin')}",
"--setenv=TERM=dumb", f"--working-directory={HOME}", "--quiet", "--no-block"]
if askpass:
cmd += [f"--setenv=SUDO_ASKPASS={HERE / 'askpass'}", f"--setenv=FRAMETOP_ASKPASS_SOCKET={SOCK}"]
cmd += ["/usr/bin/bash"] + args
return subprocess.run(cmd, env=env).returncode == 0
class Askpass:
"""The socket askpass asks: answers programs in the install's service with the password,
or holds them while the window asks you for it."""
def __init__(self, on_ask):
self.on_ask = on_ask
self.password = None # bytearray, while the install runs
self.waiting = []
SOCK_DIR.mkdir(mode=0o700, exist_ok=True)
st = SOCK_DIR.stat()
if st.st_uid != os.getuid():
raise OSError(f"{SOCK_DIR} isn't yours")
os.chmod(SOCK_DIR, 0o700)
SOCK.unlink(missing_ok=True)
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
self.sock.bind(str(SOCK))
self.sock.listen(4)
self.sock.setblocking(False)
self.watch = GLib.io_add_watch(GLib.IOChannel.unix_new(self.sock.fileno()), GLib.PRIORITY_DEFAULT,
GLib.IO_IN, self.accept)
def accept(self, *_):
try:
conn, _ = self.sock.accept()
except OSError:
return True
pid, uid, _ = struct.unpack("3i", conn.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED,
struct.calcsize("3i")))
if uid != os.getuid() or not in_unit(pid):
conn.close()
return True
if self.password:
self.answer(conn)
else:
self.waiting.append(conn)
self.on_ask()
return True
def answer(self, conn):
try:
conn.setblocking(True)
conn.sendall(bytes(self.password) if self.password else b"")
except OSError:
pass
conn.close()
def give(self, pw):
self.password = pw
while self.waiting:
self.answer(self.waiting.pop())
def refuse(self):
while self.waiting:
self.waiting.pop().close()
def close(self):
if self.password:
self.password[:] = bytes(len(self.password))
self.password = None
self.refuse()
GLib.source_remove(self.watch)
self.sock.close()
SOCK.unlink(missing_ok=True)
class Keypad(Gtk.Box):
"""Keys to click with the controller's laser, for a password field: in VR, SteamVR's own
keyboard comes up for a Steam title's window but its keys don't reach it (tested
2026-10-07), while clicks on the window's buttons do. A physical keyboard types as usual."""
LETTERS = ("1234567890", "qwertyuiop", "asdfghjkl", "zxcvbnm")
SYMBOLS = ("!@#$%^&*()", "-_=+[]{}\\|", ";:'\",.<>/?", "`~")
def __init__(self, entry):
super().__init__(orientation=Gtk.Orientation.VERTICAL, spacing=4)
self.entry = entry
self.shift = self.symbols = False
self.rows = []
for _ in range(4):
row = Gtk.Box(spacing=4, halign=Gtk.Align.CENTER, homogeneous=True)
self.rows.append(row)
self.append(row)
bottom = Gtk.Box(spacing=4, halign=Gtk.Align.CENTER)
for label, action in (("Shift", self.toggle_shift), ("!#1", self.toggle_symbols),
("Space", lambda *_: self.type(" ")), ("Delete", self.backspace)):
b = Gtk.Button(label=label)
b.set_size_request(110 if label != "Space" else 260, 48)
b.connect("clicked", action)
bottom.append(b)
if label == "!#1":
self.symbols_key = b
self.append(bottom)
self.fill()
def fill(self):
for row, keys in zip(self.rows, self.SYMBOLS if self.symbols else self.LETTERS):
while (child := row.get_first_child()) is not None:
row.remove(child)
for k in keys:
k = k.upper() if self.shift and not self.symbols else k
b = Gtk.Button(label=k)
b.set_size_request(56, 48)
b.connect("clicked", lambda _b, k=k: self.type(k))
row.append(b)
def type(self, text):
self.entry.set_text(self.entry.get_text() + text)
self.entry.set_position(-1)
def backspace(self, *_):
self.entry.set_text(self.entry.get_text()[:-1])
self.entry.set_position(-1)
def toggle_shift(self, *_):
self.shift = not self.shift
self.fill()
def toggle_symbols(self, *_):
self.symbols = not self.symbols
self.symbols_key.set_label("abc" if self.symbols else "!#1")
self.fill()
def keypad_row(entry, *extra):
"""The password field, a button that shows or hides the keypad, and the keypad: shown by
itself when Steam started this (FrameDrop's title, so most likely in VR)."""
keypad = Keypad(entry)
keypad.set_visible("SteamAppId" in os.environ)
toggle = Gtk.Button(label="Keypad")
toggle.connect("clicked", lambda *_: keypad.set_visible(not keypad.get_visible()))
row = Gtk.Box(spacing=8)
entry.set_hexpand(True)
for w in (entry, toggle, *extra):
row.append(w)
return row, keypad
class Window(Adw.ApplicationWindow):
def __init__(self, app):
super().__init__(application=app, title="Frametop")
self.set_default_size(900, 860 if "SteamAppId" in os.environ else 640)
self.offset = 0
self.partial = ""
self.askpass = None
self.checking = False
self.status = Gtk.Label(label="Install Frametop", xalign=0, wrap=True)
self.status.add_css_class("title-2")
self.detail = Gtk.Label(xalign=0, wrap=True)
self.box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=12,
margin_top=18, margin_bottom=18, margin_start=18, margin_end=18)
self.box.append(self.status)
self.box.append(self.detail)
view = Adw.ToolbarView(content=self.box)
view.add_top_bar(Adw.HeaderBar())
self.set_content(view)
self.connect("close-request", self.closing)
if unit_state() == "activating":
self.show_progress() # playing the title again: the install is still going
else:
self.show_choices()
# --- what to install, and the password
def show_choices(self):
what = f"Frametop {release_version()}" if RELEASE else "Frametop from GitHub"
self.detail.set_label(f"This installs {what}: the multi-screen desktop, the 3D mouse, gaze "
"mode, and Frametop's settings apps. Two optional parts need your "
"SteamOS password (sudo):")
self.eye = Gtk.CheckButton(label="Our own eye tracker for gaze mode (more accurate than SteamVR's)",
active=True)
self.bt = Gtk.CheckButton(label="Bluetooth fixes (LE mice and keyboards, like the Swiftpoint Z3, "
"reconnect after they sleep)")
self.pw = Gtk.PasswordEntry(show_peek_icon=True, placeholder_text="Your SteamOS password")
pw_note = Gtk.Label(label="It's used only for this install, and isn't saved anywhere.", xalign=0,
wrap=True)
pw_note.add_css_class("dim-label")
self.error = Gtk.Label(xalign=0, wrap=True, visible=False)
self.error.add_css_class("error")
self.go = Gtk.Button(label="Install", halign=Gtk.Align.END)
self.go.add_css_class("suggested-action")
self.go.connect("clicked", self.install)
self.pw.connect("activate", self.install)
pw_row, keypad = keypad_row(self.pw)
self.choices = [self.eye, self.bt, pw_row, keypad, pw_note, self.error, self.go]
if not password_set():
for c in (self.eye, self.bt):
c.set_active(False)
c.set_sensitive(False)
pw_row.set_visible(False)
keypad.set_visible(False)
pw_note.set_label("Your user has no password, so sudo can't run and these two can't be "
"installed from here. To set one, run passwd in Konsole; then play "
"Frametop again, or install them later from a terminal "
"(gaze/tracker/install.sh, setup/bluetooth/install.sh).")
for c in (self.eye, self.bt):
c.connect("toggled", lambda *_: pw_row.set_sensitive(self.eye.get_active() or self.bt.get_active()))
for w in self.choices:
self.box.append(w)
def install(self, *_):
if self.checking:
return
needs = self.eye.get_active() or self.bt.get_active()
if not needs:
return self.begin(None)
text = self.pw.get_text()
if not text:
return self.say("Type your password, or untick the parts that need it.")
pw = bytearray(text.encode())
self.pw.set_text("")
self.checking = True
self.go.set_sensitive(False)
self.say("Checking the password...")
def check():
ok = check_password(pw)
GLib.idle_add(checked, ok)
def checked(ok):
self.checking = False
self.go.set_sensitive(True)
if ok:
self.begin(pw)
else:
pw[:] = bytes(len(pw))
self.say("sudo didn't take that password. Try again, or untick the parts that need it.")
return False
threading.Thread(target=check, daemon=True).start()
def say(self, text):
self.error.set_label(text)
self.error.set_visible(True)
def begin(self, pw):
args = install_command(self.eye.get_active(), self.bt.get_active())
if pw is not None:
try:
self.askpass = Askpass(self.ask_again)
except OSError as e:
pw[:] = bytes(len(pw))
return self.say(f"Couldn't make the password's socket: {e}")
self.askpass.give(pw)
if not start_unit(args, pw is not None):
if self.askpass:
self.askpass.close()
self.askpass = None
return self.say("Couldn't start the install service (systemd-run).")
for w in self.choices:
self.box.remove(w)
self.show_progress()
# --- progress
def show_progress(self):
self.status.set_label("Installing Frametop")
self.detail.set_label("Starting...")
self.bar = Gtk.ProgressBar()
self.bar.pulse()
self.text = Gtk.TextView(editable=False, cursor_visible=False, monospace=True,
wrap_mode=Pango.WrapMode.WORD_CHAR)
self.text.set_left_margin(8)
self.text.set_right_margin(8)
scroll = Gtk.ScrolledWindow(vexpand=True, child=self.text)
# The install wants the password and the window doesn't have it (played again).
self.ask_pw = Gtk.PasswordEntry(show_peek_icon=True,
placeholder_text="The next step needs your SteamOS password")
ok = Gtk.Button(label="OK")
skip = Gtk.Button(label="Skip that part")
ok.connect("clicked", self.answer_ask)
self.ask_pw.connect("activate", self.answer_ask)
skip.connect("clicked", self.skip_ask)
ask_row, ask_keypad = keypad_row(self.ask_pw, ok, skip)
self.ask_bar = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=8, visible=False)
self.ask_bar.append(ask_row)
self.ask_bar.append(ask_keypad)
self.ask_error = Gtk.Label(xalign=0, wrap=True, visible=False)
self.ask_error.add_css_class("error")
keep = " Keep it open until the install is done: it gives the steps that need it your password." \
if self.askpass else ""
self.note = Gtk.Label(label="You can close this window: the install keeps going. Play Frametop "
"again to come back to it." + keep, xalign=0, wrap=True)
self.note.add_css_class("dim-label")
close = Gtk.Button(label="Close", halign=Gtk.Align.END)
close.connect("clicked", lambda *_: self.close())
for w in (self.bar, scroll, self.ask_bar, self.ask_error, self.note, close):
self.box.append(w)
if self.askpass is None:
try:
self.askpass = Askpass(self.ask_again)
except OSError:
self.askpass = None # askpass then fails, and install.sh skips those parts
GLib.timeout_add(500, self.tick)
self.tick()
def ask_again(self):
if hasattr(self, "ask_bar"):
self.ask_bar.set_visible(True)
self.ask_pw.grab_focus()
def answer_ask(self, *_):
text = self.ask_pw.get_text()
if not text or self.checking:
return
pw = bytearray(text.encode())
self.ask_pw.set_text("")
self.checking = True
def check():
GLib.idle_add(checked, check_password(pw))
def checked(ok):
self.checking = False
if ok and self.askpass:
self.askpass.give(pw)
self.ask_bar.set_visible(False)
self.ask_error.set_visible(False)
else:
pw[:] = bytes(len(pw))
self.ask_error.set_label("sudo didn't take that password.")
self.ask_error.set_visible(True)
return False
threading.Thread(target=check, daemon=True).start()
def skip_ask(self, *_):
if self.askpass:
self.askpass.refuse()
self.ask_bar.set_visible(False)
self.ask_error.set_visible(False)
def add_lines(self, lines):
buf = self.text.get_buffer()
for line in lines:
m = STEP.match(line)
if m:
n, total, what = int(m[1]), int(m[2]), m[3]
self.bar.set_fraction(max(0, n - 1) / total)
self.detail.set_label(f"Step {max(n, 1)} of {total}: {what}")
buf.insert(buf.get_end_iter(), line + "\n")
# Keep the view at the newest line.
end = buf.create_mark(None, buf.get_end_iter(), False)
self.text.scroll_mark_onscreen(end)
buf.delete_mark(end)
def tick(self):
try:
with open(LOG, "rb") as f:
f.seek(self.offset)
data = f.read()
self.offset += len(data)
except FileNotFoundError:
data = b""
if data:
text = self.partial + ANSI.sub("", data.decode("utf-8", "replace")).replace("\r", "\n")
*lines, self.partial = text.split("\n")
self.add_lines(lines)
state = unit_state()
if state == "activating":
if self.bar.get_fraction() == 0:
self.bar.pulse()
return True
if self.partial:
self.add_lines([self.partial])
self.partial = ""
self.forget()
self.note.set_visible(False)
self.ask_bar.set_visible(False)
if state == "active":
self.status.set_label("Done")
self.detail.set_label(DONE_TEXT)
self.bar.set_fraction(1)
else:
self.status.set_label("The install stopped")
self.detail.set_label("The log above says why. Play Frametop again to try again.")
return False
def forget(self):
if self.askpass:
self.askpass.close()
self.askpass = None
def closing(self, *_):
self.forget()
return False
def main():
app = Adw.Application(application_id="io.github.deejanuz.FrametopInstall")
def activate(a):
# Played again while the window is open: show that one.
win = a.get_active_window() or Window(a)
win.present()
app.connect("activate", activate)
app.run([])
if __name__ == "__main__":
main()
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env bash
# FrameDrop proof of concept, step 1: run as a Steam "Devkit Game" (what FrameDrop installs a
# Linux zip as) and record what an installer started from Steam could use: host or Steam Linux
# Runtime container, podman, the user's systemd, git, the network, GTK, and a window on screen.
# Register and start it: framedrop/devkit.sh add FrametopProbe framedrop/probe "./probe.sh native"
# framedrop/devkit.sh run FrametopProbe
# Usage: probe.sh LABEL
# Writes ~/.cache/frametop-framedrop/probe-LABEL.log, replacing the previous one.
label=${1:-unlabeled}
out=$HOME/.cache/frametop-framedrop
mkdir -p "$out"
log=$out/probe-$label.log
exec >"$log" 2>&1
here=$(cd "$(dirname "$0")" && pwd)
section() { printf '\n== %s\n' "$1"; }
have() { command -v "$1" >/dev/null 2>&1; }
check() { # check NAME CMD...: one line, ok or failed with the exit status
local name=$1; shift
if out_=$(timeout 20 "$@" 2>&1); then echo "ok $name: $(echo "$out_" | head -1)"
else echo "FAILED $name (exit $?): $(echo "$out_" | head -2 | tr '\n' ' ')"; fi
}
section "when, who, where"
date -Is
echo "label=$label uid=$(id -u) user=$(id -un) pwd=$PWD here=$here"
echo "args: $*"
grep -E '^(ID|VARIANT_ID|VERSION_ID|BUILD_ID|PRETTY_NAME)=' /etc/os-release
section "container?"
for p in /run/pressure-vessel /.flatpak-info /run/host/os-release; do
[ -e "$p" ] && echo "present: $p" || echo "absent: $p"
done
echo "container=${container-} PRESSURE_VESSEL_RUNTIME=${PRESSURE_VESSEL_RUNTIME-}"
[ -e /run/host/os-release ] && grep -E '^(ID|VARIANT_ID)=' /run/host/os-release
section "environment"
env | grep -E '^(PATH|HOME|XDG_[A-Z_]+|DISPLAY|WAYLAND_DISPLAY|DBUS_SESSION_BUS_ADDRESS|SteamAppId|SteamGameId|STEAM_COMPAT_[A-Z_]+|LD_LIBRARY_PATH|LD_PRELOAD|SDL_[A-Z_]+|GDK_BACKEND|ENABLE_[A-Z_]+|PRESSURE_VESSEL_[A-Z_]+)=' | sort
echo "process tree:"
pid=$$
for _ in 1 2 3 4 5 6 7 8; do
[ "$pid" -le 1 ] 2>/dev/null && break
printf ' %s %s\n' "$pid" "$(tr '\0' ' ' </proc/"$pid"/cmdline 2>/dev/null | cut -c1-200)"
pid=$(awk '/^PPid:/{print $2}' /proc/"$pid"/status 2>/dev/null)
done
# Steam adds its overlay (LD_PRELOAD) and runtime libraries to every child. An installer has
# to drop them before running host tools, as the checks below do.
unset LD_PRELOAD
LD_LIBRARY_PATH=$(printf %s "${LD_LIBRARY_PATH-}" | tr ':' '\n' | grep -v -e '/Steam/' -e '^$' -e 'x86_64' -e 'i386' | paste -sd:)
[ -n "$LD_LIBRARY_PATH" ] && export LD_LIBRARY_PATH || unset LD_LIBRARY_PATH
PATH=$(printf %s "$PATH" | tr ':' '\n' | grep -v '/Steam/' | paste -sd:)
echo "cleaned: PATH=$PATH LD_LIBRARY_PATH=${LD_LIBRARY_PATH-}"
section "tools"
for t in bash git curl python3 podman distrobox systemctl systemd-run busctl gdbus flatpak-spawn \
steam-runtime-launch-client konsole zenity kdialog; do
printf '%-28s %s\n' "$t" "$(command -v "$t" || echo -)"
done
section "what an installer needs"
check "home writable" sh -c 'f=$HOME/.cache/frametop-framedrop/.w && : >"$f" && rm "$f" && echo yes'
check "~/frametop visible" sh -c 'ls -d "$HOME/frametop" && git -C "$HOME/frametop" log -1 --format=%h'
have git && check "git ls-remote github" git ls-remote --heads https://github.com/Frametop/frametop.git experimental
have curl && check "curl get.sh" sh -c 'curl -fsSL https://frametop.github.io/frametop/get.sh | head -1'
have podman && check "podman ps" podman ps --format '{{.Names}}'
have distrobox && check "distrobox list" distrobox list
have systemctl && check "systemctl --user" systemctl --user is-system-running
have systemctl && check "user units visible" systemctl --user is-enabled frametop-input-relay.service
have python3 && check "python3 gi Gtk 4" python3 -c 'import gi; gi.require_version("Gtk","4.0"); gi.require_version("Adw","1"); from gi.repository import Gtk, Adw; print(Gtk.get_major_version(), Gtk.get_minor_version())'
section "escape to the host (needed if this is a container)"
# A transient user unit runs in the host's user manager, outside any container.
if have systemd-run; then
rm -f "$out/escape-$label"
check "systemd-run --user" systemd-run --user --wait --collect --quiet -- \
sh -c "grep -E '^(ID|VARIANT_ID)=' /etc/os-release > '$out/escape-$label'; command -v podman git >> '$out/escape-$label'"
[ -s "$out/escape-$label" ] && sed 's/^/ host says: /' "$out/escape-$label"
fi
if have busctl; then
check "busctl --user (systemd1)" busctl --user get-property org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager Version
fi
section "window"
# A plain GTK 4 window for 20 s. Watch for it on the Frame; "window: mapped" means GTK showed it.
if have python3; then
timeout 40 python3 - <<'EOF'
import sys
try:
import gi
gi.require_version("Gtk", "4.0")
from gi.repository import Gtk, GLib
except Exception as e:
print("window: no GTK 4:", e); sys.exit(0)
app = Gtk.Application(application_id="io.github.deejanuz.FrametopProbe")
def on_activate(app):
w = Gtk.ApplicationWindow(application=app, title="Frametop installer probe")
w.set_default_size(640, 240)
w.set_child(Gtk.Label(label="Frametop installer probe\n\nIf you can read this, a FrameDrop install can show progress.\nThis window closes in 20 seconds."))
w.connect("map", lambda *_: print("window: mapped", flush=True))
w.present()
GLib.timeout_add_seconds(20, app.quit)
app.connect("activate", on_activate)
print("window: display", Gtk.Widget.get_display(Gtk.Label()) if hasattr(Gtk.Widget, "get_display") else "?", flush=True)
app.run([])
print("window: closed")
EOF
echo "window exit: $?"
else
echo "window: no python3"
fi
section "done"
date -Is
Executable
+202
View File
@@ -0,0 +1,202 @@
#!/usr/bin/env bash
# ft — build and run Frametop out of its OCI image (pack/Containerfile).
#
# The image is the product: one container that holds the frozen toolchain, the
# native binaries, and the locked Python environment. Everything Frametop runs
# goes through this wrapper, so the integration points (mounts, IPC, devices)
# live in exactly one place.
#
# The wrapper works from two homes:
# Repo mode the script sits in a checkout (pack/Containerfile beside it):
# development commands build from the sources, and runs mount
# the repo at /src/frametop
# Installed the script sits in ~/.local/bin (put there by install.sh):
# programs run from the image's own copy, no repo needed
#
# Image reference, in order: $FT_IMAGE, then ~/.config/frametop/image — which
# install.sh writes and `ft update` keeps digest-pinned. A moving :latest tag
# is refused for what gets deployed (the pinned image and the update source):
# what runs on a headset must be reproducible and rollback-able. Local
# development is free to use any tag, :latest included.
#
# Usage:
# ft <prog> [args] [both] run a program from /opt/frametop
# ft update [both] pull the published image, pin its digest
# ft clean [both] remove frametop's leftover containers and
# unused frametop images (see below)
# Development commands live behind "ft dev" (they need a repo checkout):
# ft dev build [repo] build the image from the repo
# ft dev shell [repo] interactive shell in the image
# ft dev test [repo] run the Python test suites inside the image
#
# About `ft clean` and the shared podman store: on SteamOS, frametop's
# containers share one rootless podman store with Valve's (named
# lepton-<context>). `ft clean` touches ONLY objects that are provably
# frametop's: containers named frametop-*, and images whose reference
# contains a /frametop or frametop: component. Everything else — every
# lepton-* container, every dangling image, every cache layer — is left
# alone. Never run `podman rm -a`, `podman rmi -a`, or `podman system prune`
# on a Frame: those are store-wide and would delete Valve's containers.
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
[ -f "$root/pack/Containerfile" ] && repo_mode=1 || repo_mode=0
config_dir=${XDG_CONFIG_HOME:-$HOME/.config}/frametop
ref_file=$config_dir/image # installed: the pinned image reference
published_file=$config_dir/published # installed: where updates come from
reject_latest() { # reject_latest <what> <ref>
# A reference with neither a tag nor a digest means :latest too. The tag is
# after the last "/", so a registry port (host:5000/...) isn't one.
local name=${2%%@*} latest=0
case ${name##*/} in
*:latest) latest=1 ;;
*:*) ;;
*) [[ $2 == *@sha256:* ]] || latest=1 ;;
esac
if [ "$latest" = 1 ]; then
echo "ft: $2 uses the moving :latest tag — refused for $1." >&2
echo " Pin a version tag or a digest (repo@sha256:...) instead," >&2
echo " e.g. in $published_file or $ref_file." >&2
exit 2
fi
}
# The reference updates pull from: $FT_IMAGE_PUBLISHED, then the config file.
# It must be configured and versioned — there is no silent :latest default,
# and there is no default that appears when nothing is configured.
# (Deferred: refreshing the installed wrapper from the image, so wrapper and
# image ship as one artifact. The wiring broke CI under Docker; revisit after
# the smoke job tells us exactly where.)
default_published() {
local ref
if [ -n "${FT_IMAGE_PUBLISHED:-}" ]; then ref=$FT_IMAGE_PUBLISHED
elif [ -r "$published_file" ]; then ref=$(<"$published_file")
else
echo "ft update: no published reference configured." >&2
echo " Write a versioned reference (never :latest) to:" >&2
echo " $published_file" >&2
echo " e.g. ghcr.io/0x1f6/frametop:v0.2.1 — install.sh does this." >&2
exit 2
fi
reject_latest "the update source" "$ref"
echo "$ref"
}
# The reference programs run: $FT_IMAGE, then the pinned file. Installed mode
# refuses to run without a pin — a headset must never end up on :latest by
# accident, and must always know exactly what image it is running.
default_image() {
if [ "$repo_mode" = 1 ]; then echo "frametop:local"; return; fi
local ref
if [ -r "$ref_file" ] && [ -n "$(<"$ref_file")" ]; then ref=$(<"$ref_file")
elif [ -n "${FT_IMAGE:-}" ]; then echo "$FT_IMAGE"; return
else
echo "ft: no image pinned. Nothing has been installed yet." >&2
echo " install.sh writes the pinned reference to:" >&2
echo " $ref_file" >&2
exit 2
fi
reject_latest "the pinned image" "$ref"
echo "$ref"
}
image=${FT_IMAGE:-}
published=${FT_IMAGE_PUBLISHED:-}
# FT_ENGINE picks one when both are installed (CI builds with docker, and
# its runner has podman too).
engine=${FT_ENGINE:-$(command -v podman || command -v docker || true)}
[ -n "$engine" ] || { echo "ft: need podman or docker on PATH" >&2; exit 1; }
# Repo mode mounts the checkout; installed mode uses the image's own copy.
mounts=()
[ "$repo_mode" = 1 ] && mounts+=(-v "$root":/src/frametop)
# A container's own network namespace starts with the kernel's datagram queue
# of 10; systemd sets 512 on the host. The input relay sends without blocking
# and drops what a full queue refuses, so at 10 its burst of key and button
# releases loses the last ones (keys-test.py caught this).
mounts+=(--sysctl net.unix.max_dgram_qlen=512)
# How the programs run on the Frame (mounts, devices, groups, namespaces) is
# not decided yet: see "The runtime on the Frame" in pack/design.md. Runs here
# get no access to the host beyond the repo mount.
run_in_image() {
[ -n "$image" ] || image=$(default_image)
# frametop-<program>-<pid>: podman ps names the program, two runs of one
# program (python3 a.py, python3 b.py) don't replace each other, and
# ft clean finds the leftovers of crashed runs by the prefix.
local name
name=frametop-$(printf '%s' "${1##*/}" | tr -c 'a-zA-Z0-9._-' '-')-$$
exec "$engine" run --rm --name "$name" ${mounts[@]+"${mounts[@]}"} \
--workdir /src/frametop "$image" "$@"
}
need_repo() {
[ "$repo_mode" = 1 ] || { echo "ft $1: only in a repo checkout (this copy runs installed)" >&2; exit 2; }
}
update() {
[ -n "$published" ] || published=$(default_published)
reject_latest "the update source" "$published"
"$engine" pull "$published"
# Pin what was pulled, by digest, so every later run and any bug report
# names exactly this image — and rollback is editing one file.
local digest
if digest=$("$engine" inspect --format '{{index .RepoDigests 0}}' "$published" 2>/dev/null) && [ -n "$digest" ]; then
mkdir -p "$config_dir"
# Written whole or not at all: an empty pin would stop every run.
printf '%s\n' "$digest" > "$ref_file.new" && mv "$ref_file.new" "$ref_file"
echo "ft update: pinned $digest"
else
echo "ft update: could not read a digest for $published — keeping $image." >&2
fi
}
clean() {
# Frametop's leftovers, and nothing else. See the header: the podman store
# is shared with Valve's lepton-* containers, so this only deletes objects
# whose name proves they are ours.
local found=0 obj
while IFS= read -r obj; do
[ -n "$obj" ] || continue
found=1
echo "removing container $obj"
"$engine" rm -f "$obj" >/dev/null
done < <("$engine" ps -a --format '{{.Names}}' 2>/dev/null | grep '^frametop-' || true)
# Images: only references that name frametop, and only ones no container
# uses (podman rmi refuses those anyway — the guard is belt and braces).
while IFS= read -r obj; do
[ -n "$obj" ] || continue
case $obj in *frametop*) ;; *) continue ;; esac
found=1
echo "removing image $obj"
"$engine" rmi "$obj" >/dev/null 2>&1 \
|| echo " (in use or already gone — left in place)" >&2
done < <("$engine" images --format '{{.Repository}}:{{.Tag}} {{.ID}}' 2>/dev/null \
| grep -E '(^|[ /])frametop(:| |$)' | awk '{print $1}' || true)
[ "$found" = 1 ] || echo "nothing of frametop's to clean"
[ "$found" = 1 ] || exit 0
}
case "${1:-help}" in
dev)
shift
image=${FT_IMAGE:-frametop:local} # dev always works against the local build
case "${1:-help}" in
build) need_repo build
exec "$engine" build -f "$root/pack/Containerfile" -t "$image" "$root" ;;
shell) need_repo shell
exec "$engine" run --rm -it ${mounts[@]+"${mounts[@]}"} --workdir /src/frametop "$image" bash ;;
test) need_repo test; run_in_image just test ;;
help|-h|--help|"") sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' ;;
*) echo "ft dev: unknown command: $1 (build, shell, test)" >&2; exit 2 ;;
esac ;;
update) update ;;
clean) clean ;;
help|-h|--help)
sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
[ -z "${1:-}" ] || exit 0 ;;
*) [ -z "${1:-}" ] && { echo "ft: no command given" >&2; exit 2; }
run_in_image "$@" ;;
esac
+4 -2
View File
@@ -16,6 +16,7 @@ gaze/tracker/install.sh # our own eye tracker's frame grabber (asks for su
gaze/build.sh # build ft-gaze and the panel by hand
gaze/probe/install.sh # development: build, and add Frametop Gaze Probe to the app menu
gaze/probe/ft-gazeprobe --screen 1
scripts/gaze-report.py # why gaze or its calibration doesn't work, with what looks wrong first
```
## Gaze pointer
@@ -25,6 +26,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes:
- **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
- **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
- **One eye:** SteamOS 0.4's Track Dominant Eye Only (SteamVR's settings, General, with Show advanced settings on; `steamvr.eyeTrackingDominantEyeOnly` with `steamvr.dominantEye`) makes SteamVR's tracker ignore the other eye. With SteamVR's tracker, Frametop then goes by that eye alone: the calibration and the checks wait only for it, the gaze is that eye's own reading once it's calibrated (SteamVR's combined gaze before, which follows that eye then), and the fit check shows the other eye as not tracked. The service reads the setting again when SteamVR's settings file changes. Not yet tried in the headset: what SteamVR's shared memory says about the ignored eye wasn't measured.
With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
@@ -33,7 +35,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
- **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow: the middle, and 12 degrees left and right and 9 up and down, in a see-through panel 40 degrees wide (the quick check's 16 degree square left four of them off the panel, unseen). The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A button mapped to gaze precision or gaze drag counts as the left click there (before 2026-10-09 the panel ignored it). Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). The dot's ring shows full while ft-eyes checks it; the service doesn't wait for that answer (before 2026-10-09 it did, up to 3 s a dot, so the pointer could come back over the panel, and a slow answer closed the calibration as if the headset came off). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
- **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there.
@@ -54,7 +56,7 @@ The tracker stops when the headset is off your head. SteamVR also calibrates gaz
`gaze/tracker/` is an eye tracker of our own, because SteamVR's is about 1.5 degrees off after the best correction the gaze service can learn, and what's left is mostly look-to-look noise that no correction on top of its output can remove. Ours processes the eye cameras itself: 0.59 degrees (median) in its best live session against 0.83 for SteamVR's with the probe's correction, and after the headset was taken off and put back without recalibrating, 0.58 once your first clicks had taught it where the headset sat (`tracker/findings.md` has the measurements).
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too).
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too). It also adds it to SteamOS's update keep list (`/etc/atomic-update.conf.d/frametop-eyegrab.conf`): an update deletes `/etc` files that list doesn't name, and without the grabber gaze mode falls back to SteamVR's tracker and its calibration. `scripts/doctor.sh` reports a grabber an update deleted.
- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`.
- `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones are meant for a PC: if you have `frame-job` (a personal tool, not in this repo), `gaze/tracker/.frame-job` sends them there. `lab/py` runs them with that Python (in the dev container on the Frame; on a PC, the same venv from `requirements.txt`, which frame-job's setup makes).
+4 -5
View File
@@ -3,18 +3,17 @@
# (gaze/build/; they also run there). The panel draws its text with stb_truetype (public
# domain, one header, pinned as in screens/build.sh).
# The eye tracking API (IVRInput::GetEyeTrackingDataRelativeToNow) is newer than the header
# shipped with SteamVR's samples, so this uses the pinned public header ft-screens fetches.
# shipped with SteamVR's samples, so this uses the pinned public header (scripts/openvr.sh).
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C gaze 'set -e; mkdir -p build/include
openvr=v2.15.6
[ -f build/include/openvr-$openvr ] || { curl -fsSL "https://raw.githubusercontent.com/ValveSoftware/openvr/$openvr/headers/openvr.h" -o build/include/openvr.h && touch build/include/openvr-$openvr; }
. ../scripts/openvr.sh
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include -I../pointer/common \
-o build/ft-gaze ft-gaze.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
-o build/ft-gaze ft-gaze.cpp $OPENVR_LIBS -lpthread
stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
[ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
-o build/ft-gazepanel panel/ft-gazepanel.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 \
-o build/ft-gazepanel panel/ft-gazepanel.cpp $OPENVR_LIBS \
$(pkg-config --libs gbm libdrm) -lpthread
echo "built build/ft-gaze build/ft-gazepanel"'
+17 -4
View File
@@ -51,7 +51,10 @@ GUIDE = [
class FitCheck:
def __init__(self):
def __init__(self, ignore=None):
# The eye SteamVR's tracker ignores (0 left, 1 right) with Track Dominant Eye Only on
# (gazecal.tracked_eye), or None: its losses say nothing about the fit.
self.ignore = ignore
self.reset()
def reset(self):
@@ -99,7 +102,8 @@ class FitCheck:
if q and (eye.get("new") or [1, 1])[k]:
self.q[k].append(q[k])
closed = [opens[k] < CLOSED for k in (0, 1)]
if all(closed) or all(self.lost):
judged = [k for k in (0, 1) if k != self.ignore]
if all(closed[k] for k in judged) or all(self.lost[k] for k in judged):
return # a blink: says nothing about the fit
key = (math.floor(hy / CELL), math.floor(hp / CELL))
for k in (0, 1):
@@ -143,6 +147,8 @@ class FitCheck:
# --- Summaries ---
def status(self, k):
if k == self.ignore:
return "not tracked", (0.6, 0.6, 0.6)
if not self.samples:
return "no data", (0.6, 0.6, 0.6)
if self.lost[k]:
@@ -174,8 +180,13 @@ class FitCheck:
return ["Look around slowly (the screen's corners, then down at your keyboard, up, left and right) "
"or press Enter for a guided check."]
out = []
if self.ignore is not None:
out.append(f"SteamVR tracks only your {EYES[1 - self.ignore].lower()} (Track Dominant Eye Only in "
f"SteamVR's settings), so your {EYES[self.ignore].lower()} doesn't count here.")
bad = {}
for k in (0, 1):
if k == self.ignore:
continue
for key, words, _ in REGIONS:
share = self.region_share(k, key)
if share is not None and share >= 0.15:
@@ -197,7 +208,7 @@ class FitCheck:
"face, not one eye's fit.")
continue
k = next(iter(eyes))
other = self.region_share(1 - k, key)
other = self.region_share(1 - k, key) if 1 - k != self.ignore else None
vs = f", the {EYES[1 - k].lower()} {other:.0%}" if other is not None else ""
line = f"{EYES[k]}: lost {eyes[k]:.0%} of the time looking {words}{vs}."
if key == "down":
@@ -211,10 +222,12 @@ class FitCheck:
"eyes.")
out.append(line)
s0, s1 = self.signal(0), self.signal(1)
if s0 is not None and s1 is not None and abs(s0 - s1) > 0.25:
if self.ignore is None and s0 is not None and s1 is not None and abs(s0 - s1) > 0.25:
k = 0 if s0 < s1 else 1
out.append(f"The tracker is less sure of your {EYES[k].lower()} even when it has it "
f"(signal {min(s0, s1):.0%} against {max(s0, s1):.0%}).")
if self.ignore is not None and len(out) == 1:
out.append(f"Your {EYES[1 - self.ignore].lower()} is tracked everywhere you've looked so far.")
if not out:
out.append("Both eyes are tracked everywhere you've looked so far.")
return out
+2 -2
View File
@@ -8,8 +8,8 @@ PartOf=steamvr.service
Requisite=steamvr.service
[Service]
# Host Python; it runs ft-gaze in the dev container (distrobox enter), which quits when
# the service's pipe to it closes.
# Host Python; it runs ft-gaze in the container it was built in (scripts/in-box), which
# quits when the service's pipe to it closes.
ExecStart=/usr/bin/python3 @REPO@/gaze/ft-gazed
Restart=on-failure
RestartSec=3
+36 -21
View File
@@ -56,8 +56,8 @@
// The mmap samples are in head space, 17 ms or so old when they appear, so each is turned
// into the room with the head pose at its own timestamp, from a short pose history.
//
// Screens come from ft-screens (@ft_screens: "screens", "get N"), refreshed 4 times a
// second in the background. A curved screen is a cylinder toward its front (see OnSurface
// Screens come from ft-screens (@ft_screens: "screens", "remotes" for other machines'
// displays, "get N"), refreshed 4 times a second in the background. A curved screen is a cylinder toward its front (see OnSurface
// in screens/vr.cpp).
#include <openvr.h>
@@ -98,10 +98,11 @@ double NowRaw() {
}
// --- eye-server.mmap (packed, unaligned: read with memcpy) ---
// The SteamOS 0.4.x beta (SteamVR 2.18.2) moved every field from the timestamp on by 5
// bytes (measured 2026-10-04 with the ftdiag scan: timestamp 0x157 -> 0x15c, the vectors
// moved with it; the counter at 0x38 kept its place). Which layout is live is detected at
// runtime (EyeFile::Detect), so one binary serves both generations.
// SteamOS 0.4 (SteamVR 2.18.2; first on the 0.4.3 beta, the same on 0.4.5, the release)
// moved every field from the timestamp on by 5 bytes (measured 2026-10-04 with the ftdiag
// scan: timestamp 0x157 -> 0x15c, the vectors moved with it; the counter at 0x38 kept its
// place). Which layout is live is detected at runtime (EyeFile::Detect), so one binary
// serves both generations.
constexpr size_t kCounter = 0x38; // u32, one per sample
constexpr size_t kTime = 0x157; // f64, CLOCK_MONOTONIC_RAW seconds
constexpr size_t kLeft1 = 0x15f, kRight1 = 0x16b; // set 1: unit vectors, head space
@@ -115,11 +116,11 @@ constexpr size_t kVar1 = 0x177, kVar2 = 0x1b3;
// x, y, right x, y). An eye's pair stops changing while the tracker can't see it.
constexpr size_t kMeas = 0x1d3;
constexpr size_t kNeed = 0x1f3 + 5; // enough for either layout
constexpr size_t kShifts[] = {0, 5}; // the layouts EyeFile::Detect knows: stable, 0.4.x beta
constexpr size_t kShifts[] = {0, 5}; // the layouts EyeFile::Detect knows: SteamOS 0.3, 0.4
struct EyeFile {
// Everything from the timestamp on is read at base + shift: 0 on stable, 5 on the
// 0.4.x beta (see the constants above). known once Detect has seen that layout's
// Everything from the timestamp on is read at base + shift: 0 on SteamOS 0.3, 5 on 0.4
// (see the constants above). known once Detect has seen that layout's
// timestamp tick; before that, reading would yield garbage that still passes
// ReadSample's check.
size_t shift = 0;
@@ -367,18 +368,32 @@ private:
Screen s;
if (std::sscanf(p, " %d:%dx%d:%lf%n", &s.index, &s.wpx, &s.hpx, &s.metres, &used) != 4) break;
p += used;
// "ok x y z xx xy xz yx yy yz zx zy zz width height curve hand"
const std::string g = Ask(fd, "get " + std::to_string(s.index));
double v[15];
if (std::sscanf(g.c_str(), "ok %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf", &v[0], &v[1],
&v[2], &v[3], &v[4], &v[5], &v[6], &v[7], &v[8], &v[9], &v[10], &v[11], &v[12], &v[13],
&v[14]) != 15)
continue;
s.c = {v[0], v[1], v[2]};
s.b = {{v[3], v[4], v[5]}, {v[6], v[7], v[8]}, {v[9], v[10], v[11]}};
s.metres = v[12], s.height = v[13], s.curve = v[14];
out.push_back(s);
if (Place(fd, s)) out.push_back(s);
}
// Other machines' displays (remote.c): "ok <count> <index>:<client>:<state>:<w>x<h> ..."
const std::string remotes = Ask(fd, "remotes");
if (remotes.rfind("ok ", 0) != 0) return;
p = remotes.c_str() + 3;
if (std::sscanf(p, "%d%n", &count, &used) != 1) return;
p += used;
for (int k = 0; k < count; ++k) {
Screen s;
if (std::sscanf(p, " %d:%*[^:]:%*[^:]:%dx%d%n", &s.index, &s.wpx, &s.hpx, &used) != 3) break;
p += used;
if (s.wpx > 0 && s.hpx > 0 && Place(fd, s)) out.push_back(s);
}
}
static bool Place(int fd, Screen &s) {
// "ok x y z xx xy xz yx yy yz zx zy zz width height curve hand"
const std::string g = Ask(fd, "get " + std::to_string(s.index));
double v[15];
if (std::sscanf(g.c_str(), "ok %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf", &v[0], &v[1], &v[2],
&v[3], &v[4], &v[5], &v[6], &v[7], &v[8], &v[9], &v[10], &v[11], &v[12], &v[13], &v[14]) != 15)
return false;
s.c = {v[0], v[1], v[2]};
s.b = {{v[3], v[4], v[5]}, {v[6], v[7], v[8]}, {v[9], v[10], v[11]}};
s.metres = v[12], s.height = v[13], s.curve = v[14];
return true;
}
std::thread thread_;
@@ -659,7 +674,7 @@ int main(int argc, char **argv) {
if (haveMmap && !eyes.known && (eyes.Detecting() || (writing && now >= nextLayoutCheck))) {
const EyeFile::Detection d = eyes.Detect(now);
if (d == EyeFile::kFound) {
std::fprintf(stderr, "ft-gaze: eye-server.mmap layout: %s\n", eyes.shift ? "beta (+5)" : "stable");
std::fprintf(stderr, "ft-gaze: eye-server.mmap layout: %s\n", eyes.shift ? "SteamOS 0.4 (+5)" : "SteamOS 0.3");
} else if (d == EyeFile::kNone) {
nextLayoutCheck = now + 1.0;
if (!missSince) missSince = now;
+8
View File
@@ -6,6 +6,9 @@
ft-gazectl status the gaze service (ft-gazed): samples, lessons, the correction
ft-gazectl forget drop what the pointer's lessons taught (the calibration stays)
ft-gazectl reload the service reads calibration.json again
ft-gazectl quickcal|calibrate|fitcheck|fivecheck
open that check in the headset panel: the one-dot check, the
full calibration, the headset fit check, the five-dot check
"""
import json
@@ -55,6 +58,11 @@ def main():
print(json.dumps(json.loads(r), indent=1))
except ValueError:
print(r)
elif cmd in ("quickcal", "calibrate", "fitcheck", "fivecheck"):
r = ask("ft_gazed", cmd)
if r is None:
sys.exit("the gaze service (ft-gazed) isn't running")
print(r)
else:
sys.exit(__doc__)
+60 -20
View File
@@ -103,8 +103,11 @@ Control socket: abstract unix datagram "@ft_gazed":
quickcal the one-dot check now (the calibration if there's none)
calibrate the full calibration in the panel
fitcheck the headset fit check in the panel
fivecheck the five-dot check now (it otherwise follows a quick check
that didn't fix the tracker)
calaccept | calquit from the pointer helper while the panel is up: take this dot
now (a left click, Meta+J) | close it (a right click, Meta+K)
now (a left click, Meta+J, or a gaze precision or gaze drag
press) | close it (a right click, Meta+K)
Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever
the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever
@@ -129,11 +132,12 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correction, EyeFallback, # noqa: E402
EyeWeights, Fixation, LiveCorrection, SteamEyeLog)
EyeWeights, Fixation, LiveCorrection, SteamEyeLog, TrackedEye)
from gazecheck import Checks # noqa: E402
REPO = Path(__file__).resolve().parents[1]
HELPER = REPO / "gaze" / "build" / "ft-gaze"
IN_BOX = REPO / "scripts" / "in-box" # runs a program in the container it was built in
ME = "\0ft_gazed"
POINTER = "\0ft_pointer_helper"
EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes" # our own tracker
@@ -236,6 +240,9 @@ class Service:
self.opens = (deque(maxlen=90), deque(maxlen=90)) # left, right
self.vergence = deque(maxlen=90)
self.fallback = EyeFallback()
# SteamVR's tracker following one eye only (gazecal.tracked_eye): 0 left, 1 right, or None.
self.tracked = TrackedEye()
self.one_eye = self.tracked()
self.lost = [False, False]
self.bad_at = [0.0, 0.0] # sample time an eye was last lost or closed
self.counts = {"samples": 0, "sent": 0, "blinks": 0, "one_eye": 0, "one_eye_used": 0, "lost_left": 0,
@@ -280,7 +287,9 @@ class Service:
return "source"
if self.tracker == "own":
return "own"
return "eyes" if all(self.models[e].samples for e in SIDES) else "source"
# With one eye tracked, that eye's own calibration is enough.
sides = SIDES if self.one_eye is None else (SIDES[self.one_eye],)
return "eyes" if all(self.models[e].samples for e in sides) else "source"
# --- Settings, calibration and lessons ---
@@ -291,6 +300,9 @@ class Service:
auto = ""
if self.tracker_setting == "auto":
auto = " (auto: ours is installed)" if tracker == "own" else " (auto: ours isn't installed)"
if tracker == "steam" and EYEGRAB[1].exists() and not EYEGRAB[0].exists():
auto = (f" (auto: ours lost {EYEGRAB[0]}, which SteamOS updates delete unless it's kept;"
" reinstall it with gaze/tracker/install.sh)")
log(f"tracker {tracker}{auto}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
self.tracker, self.bias = tracker, bias
for w in self.weights.values():
@@ -499,13 +511,11 @@ class Service:
return
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}" # podman needs the real one
subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
distrobox = Path.home() / ".local" / "bin" / "distrobox"
# ft-gaze quits when its stdin closes: the one thing distrobox passes on.
sources = self.wanted_sources()
self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin",
"--sources", sources], env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, start_new_session=True)
self.proc = subprocess.Popen([str(IN_BOX), str(HELPER), "--watch-stdin", "--sources", sources], env=env,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
start_new_session=True)
self.proc_sources = sources
os.set_blocking(self.proc.stdout.fileno(), False)
os.set_blocking(self.proc.stderr.fileno(), False)
@@ -539,7 +549,7 @@ class Service:
return (self.tracker == "own" and not self.override and self.awake) or time.monotonic() < self.eyes_until
def start_eyes(self):
"""Our own tracker, in the dev container, with build/venv's numpy and OpenCV. Like
"""Our own tracker, in the container, with build/venv's numpy and OpenCV. Like
ft-gaze, it quits when its stdin closes."""
if not EYES_PYTHON.exists():
log(f"ft-eyes isn't built: run {REPO}/gaze/tracker/build.sh")
@@ -547,11 +557,9 @@ class Service:
return
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
distrobox = Path.home() / ".local" / "bin" / "distrobox"
self.eyes_proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(EYES_PYTHON), str(EYES_PROG), "-v",
"--watch-stdin"], env=env, stdin=subprocess.PIPE,
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, start_new_session=True)
self.eyes_proc = subprocess.Popen([str(IN_BOX), str(EYES_PYTHON), str(EYES_PROG), "-v", "--watch-stdin"],
env=env, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE, start_new_session=True)
os.set_blocking(self.eyes_proc.stderr.fileno(), False)
self.sel.register(self.eyes_proc.stderr, selectors.EVENT_READ, "eyes")
log("ft-eyes started" + ("" if EYES_CAMS.exists() else
@@ -641,11 +649,17 @@ class Service:
for k in (0, 1):
self.lost[k] = unc[k] > (EYE_FOUND if self.lost[k] else EYE_LOST)
if not down:
self.counts["lost_left"] += self.lost[0]
self.counts["lost_right"] += self.lost[1]
self.counts["lost_left"] += self.lost[0] and self.one_eye != 1
self.counts["lost_right"] += self.lost[1] and self.one_eye != 0
return low
def on_sample(self, s):
one = self.tracked()
if one != self.one_eye:
log("SteamVR now tracks both eyes" if one is None else
f"SteamVR tracks the {SIDES[one]} eye only (Track Dominant Eye Only): going by that eye")
self.one_eye = one
self.fix.reset()
self.checks.on_sample(s)
kind = self.kind
if kind != self.last_kind:
@@ -672,7 +686,8 @@ class Service:
eyes = [(p["hy"], p["hp"]) if "hy" in p else None for p in per]
if not any(eyes):
return
hp, hit = next(e[1] for e in eyes if e), m1.get("hit")
one = self.one_eye
hp, hit = (eyes[one] if one is not None and eyes[one] else next(e for e in eyes if e))[1], m1.get("hit")
self.counts["samples"] += 1
self.last_sample = time.monotonic()
down = hp < KEYBOARD_PITCH and not hit
@@ -680,8 +695,14 @@ class Service:
if down:
self.counts["looking_down"] += 1
return
if own and self.one_eye is not None:
# SteamVR judges only that eye's openness, and a blink closes both: ours still
# sees the other, so it stays in.
low[1 - self.one_eye] = low[self.one_eye]
# Our tracker finds the pupils itself; SteamVR's openness still marks the blinks.
bad = [eyes[k] is None or low[k] or (not own and self.lost[k]) for k in (0, 1)]
if not own and self.one_eye is not None:
bad[1 - self.one_eye] = True # SteamVR ignores that eye
if all(bad):
self.counts["blinks"] += 1
return
@@ -716,11 +737,27 @@ class Service:
self.bad_at[k] = s["t"] # the fallback doesn't learn from these either
return
bad = [low[k] or self.lost[k] for k in (0, 1)]
hy, hp = src["hy"], src["hp"]
eyes = (s["src"].get("mmap2") or {}).get("eyes")
one = self.one_eye
if one is not None:
# SteamVR's combined gaze already follows that eye alone; set 2's averages the
# ignored one in, so mmap2 takes the eye's own reading. No fallback to learn.
if bad[one]:
self.counts["blinks"] += 1
return
if self.source == "mmap2":
if not eyes:
self.counts["dropped"] += 1
return
hy, hp = eyes[one]
fy, fp = self.fix(hy, hp, s["t"], 1.0)
cy, cp = self.correction(self.source, fy, fp)
self.send(s["t"], fy + cy, fp + cp, fy, fp, None)
return
if all(bad):
self.counts["blinks"] += 1
return
hy, hp = src["hy"], src["hp"]
eyes = (s["src"].get("mmap2") or {}).get("eyes")
for k in (0, 1):
if bad[k]:
self.bad_at[k] = s["t"]
@@ -791,7 +828,8 @@ class Service:
elif words[:1] == ["forget"]:
self.forget_lessons()
reply = "ok"
elif words[:1] and words[0] in ("quickcal", "calibrate", "fitcheck", "calaccept", "calquit", "recheck"):
elif words[:1] and words[0] in ("quickcal", "calibrate", "fitcheck", "fivecheck", "calaccept", "calquit",
"recheck"):
reply = self.checks.command(words)
elif words[:1] == ["eyes"] and len(words) == 2:
try:
@@ -907,6 +945,8 @@ class Service:
self.on_control()
elif key.data == "checks":
self.checks.on_readable()
elif key.data == "eyes_reply":
self.checks.on_eyes_reply(key.fileobj)
elif key.data == "panel" and self.checks.panel_proc:
self.checks.read_panel()
elif key.data == "own":
+70 -3
View File
@@ -7,6 +7,7 @@ the tracker has lost the other), EyeWeights (how much each eye counts), and Stea
reports them.
"""
import json
import math
import os
import statistics
@@ -397,6 +398,63 @@ class LiveCorrection:
EYE_LOST = 0.004
EYE_FOUND = 0.0025
# SteamOS 0.4's "Track Dominant Eye Only" (SteamVR's settings, General, advanced): SteamVR's
# tracker ignores the other eye, for someone whose eyes don't look at the same spot. Its
# settings: steamvr.eyeTrackingDominantEyeOnly, and steamvr.dominantEye (0 left, 1 right,
# SteamVR's default). Frametop then goes by that eye alone: a calibration that waits for both
# eyes would never take a dot, and the other eye's reading isn't where the person looks.
STEAMVR_SETTINGS = (Path.home() / ".config" / "openvr" / "config" / "steamvr.vrsettings",
Path.home() / ".steam" / "steam" / "config" / "steamvr.vrsettings")
def tracked_eye(paths=STEAMVR_SETTINGS):
"""The one eye SteamVR's tracker follows (0 left, 1 right), or None for both. The first
of the settings files that exists counts (SteamVR keeps only settings changed from its
defaults, so a missing key is the default)."""
for path in paths:
try:
text = Path(path).read_text()
except OSError:
continue
try:
steamvr = json.loads(text).get("steamvr")
except (ValueError, AttributeError):
return None
if not isinstance(steamvr, dict) or steamvr.get("eyeTrackingDominantEyeOnly") is not True:
return None
return 0 if steamvr.get("dominantEye", 1) == 0 else 1
return None
class TrackedEye:
"""tracked_eye(), read again when SteamVR's settings file changes (looked at no more than
every CHECK seconds), since the setting can change while a service runs."""
CHECK = 2.0
def __init__(self, paths=STEAMVR_SETTINGS):
self.paths = paths
self.eye = None
self.stamp = None
self.checked = None
def __call__(self, now=None):
now = time.monotonic() if now is None else now
if self.checked is not None and now - self.checked < self.CHECK:
return self.eye
self.checked = now
stamp = []
for path in self.paths:
try:
st = os.stat(path)
stamp.append((st.st_mtime_ns, st.st_size))
except OSError:
stamp.append(None)
if stamp != self.stamp:
self.stamp = stamp
self.eye = tracked_eye(self.paths)
return self.eye
class EyeFallback:
"""The gaze from one eye, while the tracker has lost the other.
@@ -640,7 +698,7 @@ def cross_validate(points, mode):
return errs
def steady_samples(samples, vergence_jump=1.5, why=None):
def steady_samples(samples, vergence_jump=1.5, why=None, eye=None):
"""The samples of one look at one spot where the tracker had both eyes: none in a blink
(openness under half its median over the samples), none where it had lost an eye (its
variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the
@@ -648,28 +706,37 @@ def steady_samples(samples, vergence_jump=1.5, why=None):
vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
fraction of one far off), so only a jump away from what it was during this look means
the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.
`eye` (0 left, 1 right; see tracked_eye) judges that eye alone: the other one's loss,
openness and the vergence don't count.
`why`, a dict, gets how many were dropped for each reason: "lost_left", "lost_right",
"lost_both", "blink" and "vergence" (each sample once, for the first that applies)."""
if why is None:
why = {}
def openness(o):
return o[eye] if eye is not None else min(o)
# Openness: a blink is a sharp drop from what it was during this look. Not a fixed
# level: looking down, the upper lids come down with the eyes, and in bright light you
# squint, so the reading can stay under 0.5 for the whole look while the tracker follows
# the eyes fine (a calibration dot at the bottom of the bright round failed that way).
opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
opens = [openness(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12
seen = []
for smp in samples:
m1 = smp["src"].get("mmap1") or {}
o = m1.get("open")
lost = [u > EYE_LOST for u in m1.get("unc") or [0, 0]]
if eye is not None:
lost[1 - eye] = False
# A lost eye's openness reads 0 too, so a lost eye is named before a blink.
key = ("lost_both" if all(lost) else "lost_left" if lost[0] else "lost_right") if any(lost) else \
"blink" if o and min(o) < floor else None
"blink" if o and openness(o) < floor else None
if key:
why[key] = why.get(key, 0) + 1
else:
seen.append(smp)
if eye is not None:
return seen
def vergence(smp):
return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
+149 -29
View File
@@ -14,7 +14,8 @@ directions: look at each one.
on is seen only while the gaze service is awake (gaze mode on, someone wearing it: see
ft-gazed), so it also opens when gaze mode comes on after the service idled.
five the middle and four around it, when the first FIVE_COUNT lessons after a quick check
were all over FIVE_LIMIT degrees off: the quick check didn't fix it.
were all over FIVE_LIMIT degrees off: the quick check didn't fix it. Its panel is
wider than quick's, to hold them (PANEL_DEG).
full the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil
size, and the tracker's error with it, changes with brightness), each the middle and
a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots)
@@ -45,8 +46,10 @@ the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait
it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason:
gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a
click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common
reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
renewed every second; the helper shows it again by itself when that stops).
reason there and in the status. A right click or Meta+K ("calquit") closes the panel. A press
mapped to gaze precision or gaze drag counts as the left click (pointer/helper/calpanel.h). The
pointer hides meanwhile ("calpanel 1", renewed every second; the helper shows it again by itself
when that stops).
Our tracker's first calibration: before it has one, ft-eyes publishes no gaze (it maps pupils
to a gaze only with a calibration), so there's no gaze to hold still. Its calibration runs
@@ -55,6 +58,12 @@ are enough to start it, each dot stands in for the gaze, and a click takes the C
to it. ft-eyes then checks that each pupil was seen and held still in that window (calib-point)
and says why not. Without this, a fresh install could never calibrate our tracker.
The service doesn't wait for ft-eyes' answers to calib-point and calib-fit (ask_eyes): the dot
shows its ring full, and further clicks do nothing until the answer comes, or its deadline
passes; while it fits, a right click doesn't close the panel either. Until 2026-10-09 it waited, up to 3 s a dot and 10 s for the fit, so the gaze stopped,
the helper's panel lease ran out (the pointer came back, and a click went to the desktop
behind the panel), and an answer over EYES_GONE closed the calibration as the headset coming off.
What a capture teaches:
our tracker quick and five: a click ("click T YAW PITCH", like a pointer lesson); full:
calib-start, a calib-point for each dot, calib-fit (its calibration)
@@ -172,6 +181,26 @@ def reject_reason(reply, why):
return text[:24], f"our tracker said: {text}", False
# The panel's sizes for each check (ft-gazepanel.cpp's kQuickDeg, kFiveDeg, kFullDeg): width in
# degrees, and height / width. Every dot of a check must fit its panel (off_panel).
PANEL_DEG = {"quick": (16.0, 1.0), "five": (40.0, 0.75), "full": (64.0, 0.75)}
def off_panel(kind, own):
"""The dots of a check that wouldn't show whole on its panel: the panel's projection
(ft-gazepanel's ToPixel), with a degree to spare for the dot's ring."""
wdeg, aspect = PANEL_DEG[kind]
half = math.tan(math.radians(wdeg / 2))
m = math.tan(math.radians(1.0)) / (2 * half)
out = []
for yaw, pitch, _ in check_dots(kind, own):
x = 0.5 - math.tan(math.radians(yaw)) / (2 * half)
y = 0.5 - math.tan(math.radians(pitch)) / math.cos(math.radians(yaw)) / (2 * half) / aspect
if not (m <= x <= 1 - m and m / aspect <= y <= 1 - m / aspect):
out.append((yaw, pitch))
return out
def spread(points):
"""The median point and the spread around it (1.4826 x the median distance: a standard
deviation that one stray sample can't move far)."""
@@ -239,6 +268,7 @@ class Checks:
self.panel_restart_at = 0.0
self.screens_shown = None
self.last_progress = 0.0
self.asking = None # (socket, deadline, done): a command to ft-eyes waiting for its reply
@property
def active(self):
@@ -253,8 +283,7 @@ class Checks:
return
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
distrobox = Path.home() / ".local" / "bin" / "distrobox"
self.panel_proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(PANEL_PROG), "--watch-stdin"],
self.panel_proc = subprocess.Popen([str(REPO / "scripts" / "in-box"), str(PANEL_PROG), "--watch-stdin"],
env=env, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE, start_new_session=True)
os.set_blocking(self.panel_proc.stderr.fileno(), False)
@@ -325,6 +354,48 @@ class Checks:
if was != (on, headset):
self.svc.update_awake()
def ask_eyes(self, command, timeout, done):
"""A command to ft-eyes whose reply comes later: done(reply) runs from on_eyes_reply, or with
"" after `timeout` (tick), as ask() gives without one. ask() held the whole service up to 3 s
a dot (calib-point) and 10 s at the end (calib-fit): the gaze stopped, and the helper's 3 s
calpanel lease ran out, so the pointer came back and a click went to the desktop behind the
panel. Each command has its own socket, so a late reply can't be taken for the next one."""
self.drop_ask()
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
try:
s.bind("")
s.sendto(command.encode(), EYES)
except OSError:
s.close()
done("")
return
self.sel.register(s, selectors.EVENT_READ, "eyes_reply")
self.asking = (s, time.monotonic() + timeout, done)
def on_eyes_reply(self, sock):
if not self.asking or self.asking[0] is not sock:
return # dropped already (closed, or timed out in this loop)
try:
reply = sock.recv(4096).decode("utf-8", "replace")
except BlockingIOError:
return
except OSError:
reply = ""
done = self.asking[2]
self.drop_ask()
done(reply)
def drop_ask(self):
if not self.asking:
return
s = self.asking[0]
self.asking = None
try:
self.sel.unregister(s)
except (KeyError, ValueError):
pass
s.close()
# --- State ---
def calibrated(self):
@@ -453,7 +524,10 @@ class Checks:
self.screens_shown = (st[2] == "0") if st[1] == "always" else (st[2] == "1")
ask(SCREENS, "hide", 0.5)
self.to_helper("calpanel 1")
self.to_panel(f"show {'full' if kind == 'full' else 'quick'}")
off = off_panel(kind, own)
if off:
log(f"{kind} check: {len(off)} dots off its panel: {off}")
self.to_panel(f"show {kind}")
self.show_dot()
if kind == "quick":
self.last_quick = now
@@ -464,9 +538,10 @@ class Checks:
if time.monotonic() - self.sample_at > 2:
return "error the headset is off or the eye tracker isn't sending"
now = time.monotonic()
ignore = None if self.svc.one_eye is None else 1 - self.svc.one_eye # the eye SteamVR ignores
self.check = {"kind": "fit", "reason": reason, "own": False, "dots": [], "i": 0, "started": now, "shown": now,
"run": [], "accept": False, "done_at": None, "tries": 0, "skipped": 0, "captured": 0,
"points": {}, "fit": FitCheck(), "drawn": {}, "drawn_at": 0.0, "step": None}
"points": {}, "fit": FitCheck(ignore=ignore), "drawn": {}, "drawn_at": 0.0, "step": None}
log(f"fit check: {reason}")
self.to_helper("calpanel 1")
self.to_panel("show fit")
@@ -536,7 +611,8 @@ class Checks:
if self.pending:
self.run_pending()
unc = (s["src"].get("mmap1") or {}).get("unc")
if unc and min(unc) <= EYE_LOST:
one = self.svc.one_eye
if unc and (min(unc) if one is None else unc[one]) <= EYE_LOST:
self.seen_at = time.monotonic()
if self.away and self.back_since is None:
self.back_since = self.seen_at
@@ -544,8 +620,8 @@ class Checks:
if c and c["kind"] == "fit":
c["fit"].feed(s, self.sample_at)
return
if not c or c["done_at"]:
return
if not c or c["done_at"] or self.asking:
return # (asking: ft-eyes has this dot's look, or the fit)
if c["own"]:
src = s["src"].get("own") or {}
if c["blind"]:
@@ -613,9 +689,12 @@ class Checks:
rec.update(eyes=eyes, miss=miss)
t0, t1 = samples[0]["t"], samples[-1]["t"]
if c["kind"] == "full":
reply = ask(EYES, f"calib-point {t0:.6f} {t1:.6f} {yaw:.4f} {pitch:.4f}", 3.0)
rec["reply"] = reply
ok = reply.startswith("ok")
# ft-eyes checks the pupils held still: its answer takes the dot or not (point_done).
# The ring shows it full meanwhile, so the click is seen to have landed.
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} capture 1.00")
self.ask_eyes(f"calib-point {t0:.6f} {t1:.6f} {yaw:.4f} {pitch:.4f}", 3.0,
lambda reply: self.point_done(rec, miss, reply))
return
else:
try:
svc.eyes_sock.sendto(f"click {t1:.6f} {yaw:.4f} {pitch:.4f}".encode(), EYES)
@@ -625,7 +704,8 @@ class Checks:
svc.weights["own"].add(miss)
else:
why = {}
steady = steady_samples(samples, why=why)
one = svc.one_eye
steady = steady_samples(samples, why=why, eye=one)
rec["dropped"] = why
reads = {}
for name in ("action", "mmap1", "mmap2", "left", "right"):
@@ -633,11 +713,20 @@ class Checks:
if "hy" in (smp["src"].get(name) or {})]
if len(pts) >= 15:
reads[name] = (statistics.median(p[0] for p in pts), statistics.median(p[1] for p in pts))
if one is not None:
# SteamVR tracks one eye (gazecal.tracked_eye): the other's reading isn't where
# you look, and set 2's average has it in, so mmap2 is that eye's own (as ft-gazed
# sends it then).
reads.pop(("left", "right")[1 - one], None)
reads.pop("mmap2", None)
if ("left", "right")[one] in reads:
reads["mmap2"] = reads[("left", "right")[one]]
rec["reads"] = reads
main = ("left", "right") if svc.kind == "eyes" else (svc.source,)
main = (("left", "right") if one is None else (("left", "right")[one],)) if svc.kind == "eyes" else (svc.source,)
if not all(n in reads for n in main):
ok = False
rec["reply"] = f"only {len(steady)} of {len(samples)} samples had both eyes"
rec["reply"] = f"only {len(steady)} of {len(samples)} samples had " + (
"both eyes" if one is None else f"your {('left', 'right')[one]} eye")
elif c["kind"] == "full":
for name, (hy, hp) in reads.items():
c["points"].setdefault(name, []).append((hy, hp, yaw - hy, pitch - hp))
@@ -657,9 +746,23 @@ class Checks:
svc.lives[name].add({"time": time.time(), "hy": hy, "hp": hp, "dy": yaw - hy, "dp": pitch - hp,
"wy": 1.0, "wp": 1.0, "how": "check"}, svc.models[name], svc.mode)
rec["miss"] = miss
if svc.kind == "eyes":
if svc.kind == "eyes" and len(miss) == 2: # (one eye tracked: nothing to weigh)
svc.weights["steam"].add(miss)
svc.dirty = True
self.captured(rec, ok)
def point_done(self, rec, miss, reply):
"""ft-eyes' answer to a full calibration dot's calib-point ("" without one)."""
rec["reply"] = reply
ok = reply.startswith("ok")
if ok:
self.svc.weights["own"].add(miss)
self.captured(rec, ok)
def captured(self, rec, ok):
"""A capture is over: the dot is taken, tried again, or skipped."""
c = self.check
yaw, pitch, _ = c["dots"][c["i"]]
if not ok:
short, long, fit = reject_reason(rec.get("reply", "") if c["own"] else None, rec.get("dropped"))
rec["reason"] = long
@@ -671,9 +774,9 @@ class Checks:
if not ok:
c["tries"] += 1
log(f"{c['kind']} check dot {c['i'] + 1}: not taken: {rec['reason']} ({rec.get('reply', '')})")
full = c["kind"] == "full"
full, big = c["kind"] == "full", c["kind"] != "quick" # quick's panel holds only short notes
if c["tries"] >= 2 or not full:
self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if full else f"Skipped: {short}")
self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if big else f"Skipped: {short}")
self.skip()
else:
self.note(f"Not taken: {long}. Look at the dot and click again")
@@ -738,8 +841,15 @@ class Checks:
return
self.full_failed = None
if c["own"]:
reply = ask(EYES, "calib-fit", 10.0)
log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
def fitted(reply):
log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
self.close()
self.to_panel("text Saving the calibration")
c["done_at"] = None # (tick would advance past the last dot again)
c["fitting"] = True
self.ask_eyes("calib-fit", 10.0, fitted)
return
else:
mode = svc.mode if svc.mode != "none" else DEFAULT_MODEL
for name, pts in c["points"].items():
@@ -762,6 +872,7 @@ class Checks:
return
if why:
log(f"{self.check['kind']} check closed: {why}")
self.drop_ask()
self.to_panel("hide")
self.to_helper("calpanel 0")
if self.check["kind"] == "full" and self.screens_shown:
@@ -771,8 +882,8 @@ class Checks:
def quit(self):
c = self.check
if not c:
return
if not c or c.get("fitting"):
return # (fitting: ft-eyes has every dot; the panel closes once it answers)
self.close("quit")
if c["kind"] == "full" and self.calibrated() is False:
# No calibration still: gaze mode can't work, so it goes off until it's turned on again.
@@ -808,9 +919,9 @@ class Checks:
log(f"{words[0]}, asked for while idle: {reply.removeprefix('error ')}")
def command(self, words, queue=True):
"""quickcal, calibrate, calaccept, calquit -> a reply."""
"""quickcal, calibrate, fitcheck, fivecheck, calaccept, calquit -> a reply."""
cmd = words[0]
if cmd in ("quickcal", "calibrate", "fitcheck") and queue and self.svc.waking() and not self.check:
if cmd in ("quickcal", "calibrate", "fitcheck", "fivecheck") and queue and self.svc.waking() and not self.check:
# The tracker isn't running (or only just started): wake it, and do this once it sends.
self.pending = (words, time.monotonic())
self.svc.update_awake()
@@ -821,10 +932,12 @@ class Checks:
return self.start("full", "asked for")
if cmd == "fitcheck":
return self.start("fit", "asked for")
if cmd == "fivecheck":
return self.start("five", "asked for")
if cmd == "calaccept":
if self.check and self.check["kind"] == "fit":
self.check["fit"].toggle_guide(time.monotonic())
elif self.check:
elif self.check and not self.asking: # (asking: this dot's click landed already)
if not self.check["accept"]:
self.check["accept_at"] = time.monotonic()
self.check["accept"] = True
@@ -862,6 +975,13 @@ class Checks:
else:
self.fit_tick(now)
return
if self.asking:
# ft-eyes has a dot's look or the fit: wait for its answer, at most to the deadline.
if now >= self.asking[1]:
done = self.asking[2]
self.drop_ask()
done("")
return
if c["done_at"] and now >= c["done_at"]:
if c.get("closing"):
self.close()
@@ -875,11 +995,11 @@ class Checks:
return
if c["accept"] and c["accept_at"] and now - c["accept_at"] > ACCEPT_WAIT:
# Clicked, but no capture yet (see on_sample): say what it's waiting for.
full = c["kind"] == "full"
big = c["kind"] != "quick"
if now - c.get("gaze_at", 0.0) > 0.5:
self.note("Waiting: the eye tracker isn't sending a gaze" if full else "Waiting: no gaze")
self.note("Waiting: the eye tracker isn't sending a gaze" if big else "Waiting: no gaze")
else:
self.note("Waiting for your gaze to hold still on the dot" if full else "Hold your look still")
self.note("Waiting for your gaze to hold still on the dot" if big else "Hold your look still")
if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
self.close("ignored")
elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
+14 -9
View File
@@ -7,14 +7,17 @@
//
// The panel sits POINTER-like at --distance (1.5 m, about where Frametop's screens are, so
// the eyes converge as they do in use). "quick" is a small square, QUICK_DEG across, for the
// one-dot check; "full" is FULL_DEG across (4:3), with a solid background whose brightness the
// service sets per round (pupil size changes with it, and the tracker's error with it); "fit"
// is FIT_DEG across (4:3), see-through like quick, for the headset fit check: a card per eye
// (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and hints.
// one-dot check; "five" is FIVE_DEG across (4:3), see-through like quick, for the five-dot
// check, whose dots are 12 degrees left and right and 9 up and down; "full" is FULL_DEG
// across (4:3), with a solid background whose brightness the service sets per round (pupil
// size changes with it, and the tracker's error with it); "fit" is FIT_DEG across (4:3),
// see-through like quick, for the headset fit check: a card per eye (tracked or lost, the
// tracker's signal, how much of the last 10 s it was seen) and hints. Every dot a check shows
// must fit its panel: gazecheck.py's PANEL_DEG mirrors these sizes and checks it.
//
// Control socket: abstract unix datagram "@ft_gazepanel" (--socket NAME); a sender with an
// address gets "ok" or "error ...":
// show quick|full|fit the panel, empty, in front of you
// show quick|five|full|fit the panel, empty, in front of you
// hide
// bg <0..1> the background's brightness (full)
// dot <yaw> <pitch> <state> [<progress 0..1>]
@@ -71,7 +74,8 @@ using Clock = std::chrono::steady_clock;
constexpr double kQuickDeg = 16; // QUICK_DEG: the one-dot check's square
constexpr double kFullDeg = 64; // FULL_DEG: the full calibration's width (4:3)
constexpr double kFitDeg = 40; // FIT_DEG: the headset fit check's width (4:3)
constexpr int kQuickPx = 320, kFullW = 1024, kFullH = 768, kFitW = 800, kFitH = 600;
constexpr double kFiveDeg = 40; // FIVE_DEG: the five-dot check's width (4:3), past its dots
constexpr int kQuickPx = 320, kFullW = 1024, kFullH = 768, kFitW = 800, kFitH = 600, kFiveW = 800, kFiveH = 600;
std::atomic<bool> g_stop{false};
// ---------------------------------------------------------------- text (as screens/keyboard.cpp)
@@ -471,9 +475,10 @@ int main(int argc, char **argv) {
if (!std::strncmp(buf, "show ", 5)) {
p.full = !std::strcmp(buf + 5, "full");
p.fit = !std::strcmp(buf + 5, "fit");
p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx;
p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx;
p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg;
const bool five = !std::strcmp(buf + 5, "five");
p.w = p.full ? kFullW : p.fit ? kFitW : five ? kFiveW : kQuickPx;
p.h = p.full ? kFullH : p.fit ? kFitH : five ? kFiveH : kQuickPx;
p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : five ? kFiveDeg : kQuickDeg;
p.title.clear(), p.text.clear(), p.note.clear(), p.dotOn = false, p.state = "off";
p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
place();
+3 -4
View File
@@ -161,11 +161,10 @@ class GazeReader:
env = dict(os.environ)
# The Frametop desktop has its own runtime dir; podman needs the real one.
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
distrobox = Path.home() / ".local" / "bin" / "distrobox"
# ft-gaze quits when its stdin closes, which is the one thing distrobox passes on
# when we go away (even if we're killed).
self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin"], env=env,
# when we go away (even if we're killed). in-box starts the container first, and picks
# the release's own on a release install.
self.proc = subprocess.Popen([str(REPO / "scripts" / "in-box"), str(HELPER), "--watch-stdin"], env=env,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
start_new_session=True)
out = Gio.UnixInputStream.new(self.proc.stdout.fileno(), False)
+1 -1
View File
@@ -9,7 +9,7 @@ frame="$root/scripts/frame.sh"
unit=frametop-gaze.service
case ${1:-status} in
install)
"$root/gaze/build.sh"
[ "$FRAME_RELEASE" = 1 ] || "$root/gaze/build.sh"
fill_template "$root/gaze/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit"
on_frame "chmod +x gaze/ft-gazed gaze/ft-gazectl"
"$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable $unit
+63 -8
View File
@@ -6,7 +6,9 @@ its dots. Until 2026-10-05 it couldn't: a fresh install that chose our tracker n
Runs ft-gazed's Service with its sockets renamed and HOME in a temp folder (state and settings
go there), a fake pointer helper (gaze mode on, headset worn), a fake ft-gaze (SteamVR sees both
eyes; "own" is {"ok":0}, as with an uncalibrated ft-eyes), a fake ft-eyes control socket, and no
panel (a stand-in process). Nothing reaches the live gaze service, the pointer helper, ft-eyes,
panel (a stand-in process). The fake ft-eyes also answers late or not at all, as a slow one
does: the service must keep running meanwhile (until 2026-10-09 it waited, up to 3 s a dot and
10 s for the fit, and the helper's 3 s panel lease ran out). Nothing reaches the live gaze service, the pointer helper, ft-eyes,
or SteamVR, so it's safe next to them.
gaze/test/first-calibration-test.py
@@ -43,7 +45,8 @@ gazecheck.SCREENS = f"\0{tag}_screens"
gazecheck.PANEL = f"\0{tag}_panel"
gazed.EYES_SOCKET = gazecheck.EYES = f"\0{tag}_eyes"
gazed.read_settings = lambda: ("own", "auto", "auto", 55.0)
DOTS = 3
gazed.TrackedEye = lambda: lambda now=None: None # both eyes, whatever SteamVR's settings say
DOTS = 4
real_dots = gazecheck.check_dots
gazecheck.check_dots = lambda kind, own: real_dots(kind, own)[:DOTS] # a short calibration
logs = []
@@ -92,8 +95,10 @@ gazed.Service.start_helper = start_helper
gazed.Service.start_eyes = start_eyes
gazecheck.Checks.start_panel = start_panel
# The fake ft-eyes: uncalibrated until calib-fit. "fail" answers the next calib-point with that.
eyes_state = {"cal": None, "points": [], "fail": None}
# The fake ft-eyes: uncalibrated until calib-fit. "fail" answers the next calib-point with that;
# "delay" holds calib-point's and calib-fit's answers that many seconds; "drop" leaves the next
# calib-point unanswered.
eyes_state = {"cal": None, "points": [], "fail": None, "delay": 0.0, "drop": False}
eyes = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
eyes.bind(gazed.EYES_SOCKET)
eyes.settimeout(0.2)
@@ -116,19 +121,31 @@ def eyes_answer():
reply = "ok"
elif w[0] == "calib-point":
eyes_state["points"].append(tuple(map(float, w[1:5])))
if eyes_state["drop"]:
eyes_state["drop"] = False
continue
reply, eyes_state["fail"] = eyes_state["fail"] or "ok 50 50 1.00 1.00", None
elif w[0] == "calib-fit":
eyes_state["cal"] = {"made": "test", "dots": len(eyes_state["points"])}
reply = f"ok {len(eyes_state['points'])} dots"
else:
reply = f"fail unknown command {w[0]}"
if addr:
if addr and w[0] in ("calib-point", "calib-fit") and eyes_state["delay"]:
threading.Timer(eyes_state["delay"], send_late, (reply, addr)).start()
elif addr:
eyes.sendto(reply.encode(), addr)
def send_late(reply, addr):
try:
eyes.sendto(reply.encode(), addr)
except OSError:
pass # the service gave up on it
threading.Thread(target=eyes_answer, daemon=True).start()
helper_state = {"reply": "ok off worn", "heard": []}
helper_state = {"reply": "ok off worn", "heard": [], "calpanel": []} # calpanel: when "calpanel 1" came
helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
helper.bind(gazed.POINTER)
helper.settimeout(0.2)
@@ -146,6 +163,8 @@ def helper_answer():
helper.sendto(helper_state["reply"].encode(), addr)
else:
helper_state["heard"].append(data.decode())
if data == b"calpanel 1":
helper_state["calpanel"].append(time.monotonic())
threading.Thread(target=helper_answer, daemon=True).start()
@@ -195,6 +214,9 @@ def take_dot(i):
check("gaze mode off, Gaze page open (wake): ours runs, uncalibrated", ask("wake 60"), "ok")
check("the service knows ours has no calibration", wait(lambda: svc.checks.calibrated() is False, 6), True)
# ft-eyes' status can come before the fake ft-gaze's first sample, and without one the refusal
# below is "the headset is off" instead (failed about 1 run in 4 until 2026-10-06).
check("SteamVR sees the eyes (the fake ft-gaze is sending)", wait(svc.checks.eyes_seen, 6), True)
check("a quick check is refused while ours has no calibration", svc.checks.start("quick", "test"),
"error our tracker isn't calibrated yet: use Calibrate")
helper_state["reply"] = "ok on worn"
@@ -222,10 +244,43 @@ ask("calaccept")
check("ft-eyes refusing a dot: its reason reaches the panel's note",
wait(lambda: "left eye in only 3 frames" in check_state().get("note", ""), 2), True)
check("dot 2, second try: taken", take_dot(1), True)
check("dot 3: taken", take_dot(2), True)
# A slow ft-eyes (2.5 s): the service goes on meanwhile, and a second click is ignored.
eyes_state["delay"] = 2.5
wait(lambda: check_state().get("i") == 2 and not check_state().get("done_at"), 3)
time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1)
before = len(eyes_state["points"])
ask("calaccept")
check("dot 3, ft-eyes slow: the service waits for it", wait(lambda: svc.checks.asking is not None, 2), True)
t = time.monotonic()
ask("status")
check("the service still answers meanwhile", time.monotonic() - t < 0.5, True)
ask("calaccept")
check("dot 3: taken once ft-eyes answers", wait(lambda: check_state().get("captured") == 3, 4), True)
check("the helper's panel lease was renewed while ft-eyes took its time",
sum(t < at < t + eyes_state["delay"] for at in helper_state["calpanel"]) >= 2, True)
check("and the calibration is still open (a blocked service took that as the headset off)",
check_state().get("kind"), "full")
check("the second click asked ft-eyes nothing", len(eyes_state["points"]), before + 1)
eyes_state["delay"] = 0.0
# No answer: the dot isn't taken, after calib-point's 3 s.
eyes_state["drop"] = True
wait(lambda: check_state().get("i") == 3 and not check_state().get("done_at"), 3)
time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1)
ask("calaccept")
check("dot 4, no answer from ft-eyes: not taken, and the panel says so",
wait(lambda: "didn't answer" in check_state().get("note", ""), 5), True)
eyes_state["delay"] = 2.5 # the fit too
check("dot 4, second try: taken", take_dot(3) or wait(lambda: check_state().get("captured") == 4, 4), True)
check("while ours fits, the panel stays", wait(lambda: check_state().get("fitting") is True, 3), True)
ask("calquit")
check("and a right click doesn't close it", check_state().get("kind"), "full")
check("all dots: ours fits its calibration (calib-fit)",
wait(lambda: eyes_state["cal"] is not None and not svc.checks.check, 4), True)
wait(lambda: eyes_state["cal"] is not None and not svc.checks.check, 5), True)
gaps = [b - a for a, b in zip(helper_state["calpanel"], helper_state["calpanel"][1:])]
check("the helper's panel lease (3 s) never ran out", bool(gaps) and max(gaps) < 3.0, True)
check("the service sees it calibrated", wait(lambda: svc.checks.calibrated() is True, 4), True)
check("and gaze mode stays on", "gaze off" in helper_state["heard"], False)
check("and no second calibration opens", wait(lambda: svc.checks.check is not None, 3), False)
+1
View File
@@ -34,6 +34,7 @@ gazecheck.SCREENS = f"\0{tag}_screens"
gazecheck.PANEL_PROG = gazed.REPO / "nonexistent-panel" # "isn't built": no panel
gazed.IDLE_AFTER, gazed.WAKE_SETTLE = 1.0, 3.0
gazed.read_settings = lambda: ("steam", "steam", "auto", 55.0)
gazed.TrackedEye = lambda: lambda now=None: None # both eyes, whatever SteamVR's settings say
logs = []
gazed.log = gazecheck.log = lambda msg: logs.append(msg)
+1 -1
View File
@@ -19,7 +19,7 @@ int failures = 0;
} while (0)
// The file as the eye server writes it, one sample at a time, with every field from the
// timestamp on moved by `shift` (0 stable, 5 the 0.4.x beta).
// timestamp on moved by `shift` (0 on SteamOS 0.3, 5 on 0.4).
struct File {
std::vector<uint8_t> bytes = std::vector<uint8_t>(324122); // eye-server.mmap's size
EyeFile eyes;
+175
View File
@@ -0,0 +1,175 @@
#!/usr/bin/env python3
"""Offline test of Track Dominant Eye Only (SteamOS 0.4): SteamVR's tracker ignores one eye,
and the gaze code then goes by the other (gazecal.tracked_eye, steady_samples' `eye`,
fitcheck's `ignore`, ft-gazed's live gaze). Reads only the temporary settings files it writes;
ft-gazed's Service is built without its sockets, so nothing reaches the live gaze service.
gaze/test/one-eye-test.py
"""
import importlib.machinery
import importlib.util
import json
import os
import sys
import tempfile
from collections import deque
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, ".."))
from fitcheck import FitCheck # noqa: E402
from gazecal import ( # noqa: E402
Correction,
EyeFallback,
EyeWeights,
Fixation,
TrackedEye,
steady_samples,
tracked_eye,
)
loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(HERE, "..", "ft-gazed"))
gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
loader.exec_module(gazed)
failures = []
def check(what, got, want):
if got != want:
failures.append(what)
print(f"FAIL {what}: got {got!r}, want {want!r}", flush=True)
tmp = tempfile.mkdtemp(prefix="ft-one-eye-test-")
first, second = os.path.join(tmp, "a.vrsettings"), os.path.join(tmp, "b.vrsettings")
paths = (first, second)
def write(path, steamvr):
with open(path, "w") as f:
f.write(steamvr if isinstance(steamvr, str) else json.dumps({"steamvr": steamvr}, indent=3))
# --- Reading SteamVR's settings ---
check("no settings file: both eyes", tracked_eye(paths), None)
write(second, {"eyeTrackingDominantEyeOnly": True})
check("only the second file: it counts (right, SteamVR's default eye)", tracked_eye(paths), 1)
write(first, {"supersampleScale": 1.0})
check("the first file counts, without the setting: both eyes", tracked_eye(paths), None)
write(first, {"eyeTrackingDominantEyeOnly": True, "dominantEye": 0})
check("dominant eye left", tracked_eye(paths), 0)
write(first, {"eyeTrackingDominantEyeOnly": False, "dominantEye": 0})
check("setting off", tracked_eye(paths), None)
write(first, "{ not json")
check("a broken file: both eyes", tracked_eye(paths), None)
eye = TrackedEye(paths)
write(first, {"eyeTrackingDominantEyeOnly": True, "dominantEye": 1})
check("TrackedEye reads it", eye(now=100.0), 1)
write(first, {"eyeTrackingDominantEyeOnly": True, "dominantEye": 0, "pad": "x" * 10})
check("TrackedEye waits CHECK seconds", eye(now=101.0), 1)
check("then sees the change", eye(now=102.5), 0)
# --- One look at a dot: the left eye lost all along (what SteamVR's tracker may report for
# the eye it ignores), the right seen, and the vergence jumping with the lost eye ---
look = [{"t": i / 90, "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.02, 0.001], "open": [0.0, 0.8],
"lr": 2.8 if i % 2 else 9.0}}} for i in range(40)]
why = {}
check("both eyes judged: nothing kept", len(steady_samples(look, why=why)), 0)
check("both eyes judged: why", why, {"lost_left": 40})
why = {}
check("right eye only: all kept", len(steady_samples(look, why=why, eye=1)), 40)
check("right eye only: nothing dropped", why, {})
why = {}
check("left eye only: nothing kept", len(steady_samples(look, why=why, eye=0)), 0)
blink = [dict(s, src={"mmap1": dict(s["src"]["mmap1"], open=[0.0, 0.05])}) if 10 <= i < 15 else s
for i, s in enumerate(look)]
why = {}
check("right eye only: its blinks still drop", len(steady_samples(blink, why=why, eye=1)), 35)
check("right eye only: as blinks", why, {"blink": 5})
# --- The headset fit check ---
fit = FitCheck(ignore=0)
for i in range(400):
fit.feed({"src": {"mmap1": {"hy": 0.0, "hp": 0.0, "unc": [0.02, 0.001], "open": [0.0, 0.8]}}}, i / 90)
check("fit: the ignored eye's card", fit.status(0)[0], "not tracked")
check("fit: the tracked eye's card", fit.status(1)[0], "tracking")
hints = fit.hints()
check("fit: says which eye counts", hints[0].startswith("SteamVR tracks only your right eye"), True)
check("fit: no losses blamed on the ignored eye", any("Left eye: lost" in h for h in hints), False)
check("fit: the tracked eye is fine", hints[-1], "Your right eye is tracked everywhere you've looked so far.")
both = FitCheck()
for i in range(400):
both.feed({"src": {"mmap1": {"hy": 0.0, "hp": 0.0, "unc": [0.02, 0.001], "open": [0.0, 0.8]}}}, i / 90)
check("fit, both eyes judged: the left eye is lost", both.status(0)[0], "LOST")
# --- ft-gazed's live gaze: the parts of Service the samples go through, no sockets ---
def service(one, source="mmap1"):
svc = gazed.Service.__new__(gazed.Service)
svc.override, svc.tracker, svc.source, svc.one_eye = None, "steam", source, one
svc.models = {name: Correction() for name in gazed.SOURCES}
svc.counts = dict.fromkeys(("samples", "sent", "blinks", "one_eye", "one_eye_used", "lost_left", "lost_right",
"looking_down", "dropped"), 0)
svc.opens, svc.vergence = (deque(maxlen=90), deque(maxlen=90)), deque(maxlen=90)
svc.lost, svc.bad_at, svc.fallback = [False, False], [0.0, 0.0], EyeFallback()
svc.fix, svc.last_sample = Fixation(radius=1.0), 0.0
svc.correction = lambda name, hy, hp: (0.0, 0.0)
svc.sent = []
svc.send = lambda t, hy, hp, rhy, rhp, eyes: svc.sent.append((hy, hp))
return svc
def feed(svc, n=90):
for i in range(n):
svc.on_source_sample({"t": i / 90, "src": {
"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.02, 0.001], "open": [0.0, 0.8]},
"mmap2": {"hy": 3.0, "hp": 2.0, "eyes": [[9.0, 9.0], [5.0, 2.0]]}}})
svc = service(None, "mmap2")
feed(svc)
check("mmap2, both eyes judged: a lost eye and no fallback yet drop the gaze", svc.sent, [])
svc = service(1, "mmap2")
feed(svc)
check("mmap2, right eye only: its own reading goes out", (len(svc.sent), svc.sent[-1] if svc.sent else None), (90, (5.0, 2.0)))
check("mmap2, right eye only: no blinks counted", svc.counts["blinks"], 0)
svc = service(0, "mmap1")
feed(svc)
check("left eye only, and it's lost: nothing goes out", (svc.sent, svc.counts["blinks"]), ([], 90))
svc = service(1)
check("no calibration: the source, as a whole", svc.kind, "source")
svc.models["right"].samples = 9
check("right eye only and calibrated: each eye's own", svc.kind, "eyes")
svc.one_eye = None
check("both eyes judged: the left needs a calibration too", svc.kind, "source")
# Our own tracker sees both eyes whatever SteamVR tracks; only the blinks come from SteamVR.
def feed_own(svc, n=90, right_open=0.8):
for i in range(n):
svc.on_eyes_sample({"t": i / 90, "src": {
"mmap1": {"unc": [0.02, 0.001], "open": [0.0, right_open]},
"own": {"hy": 5.0, "hp": 2.0, "eyes": [[4.0, 2.0], [6.0, 2.0]]}}}, True)
svc = service(1)
svc.tracker, svc.weights = "own", {"own": EyeWeights()}
check("own tracker, right eye only: our tracker", svc.kind, "own")
feed_own(svc)
check("own tracker, right eye only: SteamVR's closed left doesn't drop our left",
(len(svc.sent), svc.counts["one_eye"], svc.counts["blinks"]), (90, 0, 0))
svc = service(1)
svc.tracker, svc.weights = "own", {"own": EyeWeights()}
feed_own(svc, right_open=0.0)
check("own tracker, right eye only: its blink drops both", (svc.sent, svc.counts["blinks"]), ([], 90))
svc = service(None)
svc.tracker, svc.weights = "own", {"own": EyeWeights()}
feed_own(svc)
check("own tracker, both eyes judged: SteamVR's closed left still drops it", svc.counts["one_eye"], 90)
for p in paths:
if os.path.exists(p):
os.remove(p)
os.rmdir(tmp)
print("FAILED: " + ", ".join(failures) if failures else "all passed")
sys.exit(1 if failures else 0)
+4
View File
@@ -0,0 +1,4 @@
# Installed to /etc/atomic-update.conf.d/frametop-eyegrab.conf by gaze/tracker/install.sh.
# A SteamOS update deletes every /etc file its keep list (/usr/lib/rauc/atomic-update-keep.conf)
# doesn't name. That list keeps the unit, but not the program it runs.
/etc/frametop/ft-eyegrab
+11 -3
View File
@@ -4,7 +4,9 @@
# (frametop-eyegrab.service, gaze/tracker/install.sh), so this checks it
# only needs glibc symbols the SteamOS host has (2.39; the container has 2.43).
# build/venv Python with numpy and OpenCV (requirements.txt) for ft-eyes and lab/,
# remade when requirements.txt changes.
# remade when requirements.txt changes. In the image (pack/Containerfile)
# they're in its locked venv already (uv.lock has the same versions), so
# build/venv is a small venv that sees that one's packages, not a copy.
# Usage: gaze/tracker/build.sh
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
@@ -14,10 +16,16 @@ gcc -std=gnu11 -O2 -Wall -Wextra -pthread -o build/ft-eyegrab ft-eyegrab.c
max=$(objdump -T build/ft-eyegrab | grep -oE "GLIBC_[0-9.]+" | sort -uV | tail -1)
echo "built build/ft-eyegrab, newest glibc symbol: $max"
[ "$(printf "%s\n" "$max" GLIBC_2.39 | sort -V | tail -1)" = GLIBC_2.39 ] || { echo "needs newer glibc than the host has" >&2; exit 1; }
if ! cmp -s requirements.txt build/venv/requirements.done; then
if [ "${FRAME_IN_BOX:-0}" = 1 ] && [ -x /opt/frametop/venv/bin/python ]; then
rm -rf build/venv
python3 -m venv --without-pip build/venv
/opt/frametop/venv/bin/python -c "import sysconfig; print(sysconfig.get_path(\"purelib\"))" \
>"$(build/venv/bin/python -c "import sysconfig; print(sysconfig.get_path(\"purelib\"))")/frametop-image.pth"
elif ! cmp -s requirements.txt build/venv/requirements.done; then
rm -rf build/venv
python3 -m venv build/venv
build/venv/bin/pip install -q --disable-pip-version-check -r requirements.txt
cp requirements.txt build/venv/requirements.done
fi
echo "build/venv: $(build/venv/bin/python -c "import numpy, cv2; print(\"numpy\", numpy.__version__, \"opencv\", cv2.__version__)")"'
v=$(build/venv/bin/python -c "import numpy, cv2; print(\"numpy\", numpy.__version__, \"opencv\", cv2.__version__)")
echo "build/venv: $v"'
+3 -3
View File
@@ -111,9 +111,9 @@ reading only.
The file is 324,122 bytes. Only bytes 0x0-0x1f3 are used; the rest is zero. It's packed and
unaligned, so read it with memcpy. Offsets are also in `~/frametop/gaze/ft-gaze.cpp`.
On the SteamOS 0.4.x beta (SteamVR 2.18.2) every field from 0x157 on sits 5 bytes later, and the
counter stays at 0x38 (measured 2026-10-04, PR #26). The offsets below are stable's; ft-gaze detects
which layout is live.
On SteamOS 0.4 (SteamVR 2.18.2: the 0.4.3 beta, and 0.4.5, the release) every field from 0x157 on
sits 5 bytes later, and the counter stays at 0x38 (measured 2026-10-04, PR #26). The offsets below
are SteamOS 0.3's; ft-gaze detects which layout is live.
| Offset | What |
| --- | --- |
+5 -3
View File
@@ -5,7 +5,8 @@
# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when ours is the
# tracker in use (GAZE_TRACKER=auto, the default, picks it once this is installed) or the gaze
# probe uses it. install.sh offers this after gaze mode.
# Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit: it asks for
# Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's), the unit, and the entry that
# keeps the binary through SteamOS updates (/etc/atomic-update.conf.d): it asks for
# the password in the terminal, on the Frame or from a PC, or runs SUDO_ASKPASS when that's set
# (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env).
# Usage: gaze/tracker/install.sh [install|uninstall|status|log [lines]]
@@ -20,13 +21,14 @@ sudo_run() { frame_sudo "$1"; }
case ${1:-install} in
install)
"$root/gaze/tracker/build.sh"
[ "$FRAME_RELEASE" = 1 ] || "$root/gaze/tracker/build.sh"
ids=$(on_frame 'echo "$(id -u):$(id -g)"')
fill_template "$root/gaze/tracker/$unit" | sed "s|@UID@|${ids%:*}|g; s|@GID@|${ids#*:}|g" |
on_frame "cat > /tmp/$unit"
sudo_run "set -e
install -D -m 0755 -o root -g root $src/build/ft-eyegrab /etc/frametop/ft-eyegrab
install -D -m 0644 -o root -g root /tmp/$unit /etc/systemd/system/$unit
install -D -m 0644 -o root -g root $src/atomic-update.conf /etc/atomic-update.conf.d/frametop-eyegrab.conf
rm -f /tmp/$unit
systemctl daemon-reload
systemctl enable $unit
@@ -36,7 +38,7 @@ echo \"$unit: \$(systemctl is-active $unit)\""
;;
uninstall)
sudo_run "systemctl disable --now $unit 2>/dev/null
rm -f /etc/systemd/system/$unit /etc/frametop/ft-eyegrab
rm -f /etc/systemd/system/$unit /etc/frametop/ft-eyegrab /etc/atomic-update.conf.d/frametop-eyegrab.conf
rmdir /etc/frametop 2>/dev/null; systemctl daemon-reload; echo removed" ;;
status) on_frame "systemctl is-active $unit; ls -l /dev/shm/frametop-eyes-cams 2>/dev/null" || true ;;
log) on_frame "journalctl -u $unit --no-pager -o cat -n ${2:-20}" ;;
+137 -15
View File
@@ -2,31 +2,120 @@
# Frametop's one-line installer. In a terminal on the Steam Frame (Konsole in the desktop, or
# over SSH):
#
# curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
# curl -fsSL https://frametop.github.io/frametop/get.sh | bash
#
# It asks which version to install, clones the repo into ~/frametop (or updates the clone
# that's there), and runs its install.sh. Run it again to update, or to switch versions.
# Its third and fourth choices, or --release, install a release instead: Frametop built, in one file. It downloads the
# release's Frametop.zip from GitHub (about 1.1 GB: the newest stable release, or with
# --experimental the newest of any), unpacks it in ~/.cache/frametop/release, and runs its
# install-release.sh, which checks this SteamOS build against the releases' SteamOS table and
# installs without building anything (pack/README.md, Releases). The same zip installs with
# FrameDrop from a PC, or unpacked by hand on the headset.
# Options (piped, they go after "bash -s --"):
# --stable the main branch: tested releases (the default for a new install)
# --experimental the experimental branch: the newest features, less tested
# --branch NAME another branch, such as a fix to test before it's released
# --dir DIR where the repo goes (default ~/frametop)
# --clone-only get or update the repo, but don't run install.sh
# --yes, --no-bluetooth passed to install.sh (--yes also answers this script's question:
# the version already there, or stable)
# --dir DIR where the repo goes (default ~/frametop; for --release, where the
# releases go, default ~/.local/share/frametop/releases)
# --clone-only get or update the repo (or unpack the release), but don't run install.sh
# --release install a release from GitHub instead of cloning the repo
# --version V with --release: that release (tag vV), not the newest
# --zip FILE|URL with --release: this Frametop.zip instead of GitHub's newest
# --any-steamos with --release: install even on a SteamOS build the release breaks on
# --yes, --no-eye-tracker, --no-bluetooth, --bluetooth passed to install.sh (--yes also
# answers this script's questions: the version already there, or stable)
set -euo pipefail
usage() {
cat <<'EOF'
usage: get.sh [--stable | --experimental | --branch NAME] [--dir DIR] [--clone-only] [--yes] [--no-bluetooth]
piped: curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- [options]
usage: get.sh [--stable | --experimental | --branch NAME] [--dir DIR] [--clone-only] [--yes]
[--no-eye-tracker] [--no-bluetooth | --bluetooth]
get.sh --release [--stable | --experimental | --version V | --zip FILE|URL]
[--any-steamos] [--dir DIR] [--clone-only] [--yes] [--no-eye-tracker]
[--no-bluetooth | --bluetooth]
piped: curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- [options]
EOF
}
# Frametop lives in the Frametop organization's repo: the branches clone from it, and its CI
# runners (Depot, which need an organization) build the releases. It moved there from
# DeeJanuz/frametop on 2026-10-09; GitHub redirects clones made from the old name.
SLUG=${FRAMETOP_REPO:-Frametop/frametop} # FRAMETOP_REPO: another repo's releases, such as a fork's
# release_zip CHANNEL VERSION: the URL of a release's Frametop.zip on GitHub.
release_zip() {
if [ -n "$2" ]; then
echo "https://github.com/$SLUG/releases/download/v$2/Frametop.zip"
elif [ "$1" = stable ]; then
echo "https://github.com/$SLUG/releases/latest/download/Frametop.zip" # newest non-prerelease
else
curl -fsSL "https://api.github.com/repos/$SLUG/releases?per_page=30" | python3 -c '
import json, sys
for r in json.load(sys.stdin): # newest first
for a in r.get("assets", []):
if a.get("name") == "Frametop.zip" and not r.get("draft"):
print(a["browser_download_url"])
sys.exit(0)
sys.exit(1)'
fi
}
# install_release CHANNEL VERSION ZIP DIR CLONE_ONLY ANY_STEAMOS -- INSTALL_ARGS...
install_release() {
local channel=$1 version=$2 zip=$3 dir=$4 clone_only=$5 any=$6 cache=$HOME/.cache/frametop/release
shift 7
local args=("$@")
[ -n "$dir" ] && args+=(--dir "$dir")
[ "$clone_only" = 1 ] && args+=(--unpack-only)
[ "$any" = 1 ] && args+=(--any-steamos)
mkdir -p "$cache"
local url= file loc key status=0
case $zip in
https://*|'')
url=$zip
if [ -z "$url" ]; then
url=$(release_zip "$channel" "$version") ||
{ echo "couldn't find a Frametop release on GitHub (or GitHub can't be reached)" >&2; return 1; }
fi
# "latest" names a different file after each release: resume only the same release's.
if [[ $url == */releases/latest/download/* ]]; then
loc=$(curl -fsSI --proto '=https' "$url" | tr -d '\r' | sed -n 's/^[Ll]ocation: //p' | head -1)
[[ $loc == https://github.com/* ]] && url=$loc
fi
key=$(printf %s "$url" | sha256sum | cut -c1-16)
file=$cache/Frametop-$key.zip
find "$cache" -maxdepth 1 -name 'Frametop-*.zip*' ! -name "Frametop-$key.zip*" -delete
if [ ! -f "$file" ]; then
echo "Downloading $url (about 1.1 GB; if it stops, run this again to resume)"
curl -fL --proto '=https' -C - -o "$file.part" "$url" ||
{ echo "the download didn't finish: run this again to resume it" >&2; return 1; }
mv "$file.part" "$file"
fi ;;
*://*) echo "the zip has to come over https: $zip" >&2; return 1 ;;
*) file=$zip; [ -f "$file" ] || { echo "no file $file" >&2; return 1; } ;;
esac
echo "Unpacking $file"
rm -rf "$cache/unpacked"
if ! unzip -q "$file" -d "$cache/unpacked"; then
[ -n "$url" ] && rm -f "$file"
echo "$file didn't unpack: run this again to download it again" >&2
return 1
fi
"$cache/unpacked/Frametop/install-release.sh" "${args[@]}" || status=$?
rm -rf "$cache/unpacked"
if [ "$status" = 3 ] && [ -n "$url" ]; then
rm -f "$file" # damaged: the next run downloads it again
fi
[ "$status" = 0 ] || return "$status"
# Loaded into podman and copied out: the download isn't needed any more.
[ "$clone_only" = 1 ] || rm -rf "$cache"
}
# Everything happens in main, called on the last line, so a download cut short runs nothing.
main() {
local repo=https://github.com/DeeJanuz/frametop.git dir=$HOME/frametop branch= clone_only=0
local yes=0 tty=0 current= def answer
local repo=https://github.com/Frametop/frametop.git dir= branch= clone_only=0
local yes=0 tty=0 current= def answer release=0 zip= want= any=0
local pass=()
while [ $# -gt 0 ]; do
case $1 in
@@ -35,13 +124,29 @@ main() {
--branch) branch=${2:?--branch needs a branch name}; shift ;;
--dir) dir=${2:?--dir needs a folder}; shift ;;
--clone-only) clone_only=1 ;;
--release) release=1 ;;
--zip) zip=${2:?--zip needs a file or URL}; shift ;;
--version) want=${2:?--version needs a version}; shift ;;
--any-steamos) any=1 ;;
--yes) yes=1; pass+=("$1") ;;
--no-bluetooth) pass+=("$1") ;;
--no-bluetooth|--bluetooth|--no-eye-tracker) pass+=("$1") ;;
-h|--help) usage; return 0 ;;
*) echo "unknown option: $1" >&2; usage >&2; return 2 ;;
esac
shift
done
if [ "$release" = 0 ] && { [ -n "$zip" ] || [ -n "$want" ] || [ "$any" = 1 ]; }; then
echo "--zip, --version, and --any-steamos go with --release" >&2
return 2
fi
if [ "$release" = 1 ] && [ -n "$branch" ] && [ "$branch" != main ] && [ "$branch" != experimental ]; then
echo "releases come from the stable or experimental list, not a branch" >&2
return 2
fi
local dir_arg=$dir # the menu's release choice puts releases in their own default place
if [ -z "$dir" ] && [ "$release" = 0 ]; then
dir=$HOME/frametop
fi
if ! { grep -qx 'ID=steamos' /etc/os-release && grep -qE '^VARIANT_ID="?vr"?$' /etc/os-release; } 2>/dev/null; then
echo "Frametop installs on a Steam Frame (SteamOS, VR variant). Run this in a terminal on the headset." >&2
@@ -54,16 +159,19 @@ main() {
return 1
fi
if [ -e "$dir/.git" ]; then
if [ "$release" = 0 ] && [ -e "$dir/.git" ]; then
git -C "$dir" remote get-url origin 2>/dev/null | grep -qi 'frametop' ||
{ echo "$dir is a git repo, but not Frametop's. Pick another folder with --dir." >&2; return 1; }
current=$(git -C "$dir" branch --show-current)
elif [ -e "$dir" ]; then
elif [ "$release" = 0 ] && [ -e "$dir" ]; then
echo "$dir is there and isn't Frametop's repo. Move it, or pick another folder with --dir." >&2
return 1
elif [ "$release" = 1 ] && [ -f "${dir:-$HOME/.local/share/frametop/releases}/current/.frametop-release" ]; then
current=$(sed -n 's/^CHANNEL=//p' "${dir:-$HOME/.local/share/frametop/releases}/current/.frametop-release")
[ "$current" = stable ] && current=main
fi
if [ -z "$branch" ]; then
if [ -z "$branch" ] && { [ "$release" = 0 ] || { [ -z "$zip" ] && [ -z "$want" ]; }; }; then
def=main
[ "$current" = experimental ] && def=experimental
if [ "$yes" = 1 ]; then
@@ -72,16 +180,30 @@ main() {
echo "Which version of Frametop?"
echo " 1) stable: the main branch, tested releases"
echo " 2) experimental: the newest features, less tested"
if [ "$release" = 0 ]; then
echo " 3) stable release: built, nothing to compile (a 1.1 GB download)"
echo " 4) experimental release: built, nothing to compile (a 1.1 GB download)"
fi
[ -n "$current" ] && echo "(installed now: $current)"
read -r -p "Choose 1 or 2 [$([ "$def" = main ] && echo 1 || echo 2)]: " answer </dev/tty || answer=
read -r -p "Choose 1$([ "$release" = 0 ] && echo ", 2, 3, or 4" || echo " or 2") [$([ "$def" = main ] && echo 1 || echo 2)]: " \
answer </dev/tty || answer=
case ${answer:-$def} in
1|main|s*) branch=main ;;
2|experimental|e*) branch=experimental ;;
*) echo "not 1 or 2: $answer" >&2; return 2 ;;
3|4) [ "$release" = 0 ] || { echo "not 1 or 2: $answer" >&2; return 2; }
release=1 dir=$dir_arg
branch=$([ "$answer" = 3 ] && echo main || echo experimental) ;;
*) echo "not one of the choices: $answer" >&2; return 2 ;;
esac
fi
fi
if [ "$release" = 1 ]; then
install_release "$([ "$branch" = experimental ] && echo experimental || echo stable)" "$want" "$zip" "$dir" \
"$clone_only" "$any" -- ${pass[@]+"${pass[@]}"}
return
fi
if ! git ls-remote --exit-code --heads "$repo" "$branch" >/dev/null; then
echo "Frametop has no branch called $branch (or GitHub can't be reached)." >&2
return 1
+5 -5
View File
@@ -14,7 +14,7 @@ For now the recorder runs inside Frametop's desktop, so these steps install Fram
## Before you start
- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/DeeJanuz/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/Frametop/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
- You need a Steam Frame on the stable SteamOS release (not the beta), an internet connection, and a keyboard (Bluetooth, or the on-screen one).
- You need a `sudo` password. If you've never set one, run `passwd` in Konsole first.
- Recordings are several gigabytes per round, and uploading one needs about the same again free while it runs. `df -h ~` shows your free space.
@@ -27,7 +27,7 @@ For now the recorder runs inside Frametop's desktop, so these steps install Fram
## 1. Install Frametop
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --stable
curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --stable
```
This clones Frametop into `~/frametop` and runs its installer. The first run downloads 1–2 GB. The installer asks a few questions (gaze mode, the eye tracker, the Bluetooth fixes); the defaults are fine. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, this updates it.
@@ -49,7 +49,7 @@ Open Frametop Hand Recorder from the desktop's application menu. It walks you th
## Update
```
curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- --stable
curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --stable
~/frametop/hands/rec/install.sh
```
@@ -61,8 +61,8 @@ Run the second command after SteamVR has restarted, as in the install.
~/frametop/hands/rec/install.sh uninstall
```
This removes the menu entry. With your `sudo` password, it also takes back the camera broker's permission to read the cameras. If you also installed Frametop's live hand tracking, the camera broker keeps that permission, because live hand tracking still uses it. Your recordings stay in `~/.local/share/frametop/hands/contrib`; delete that folder to remove them. To remove Frametop as well, follow [Uninstall](https://github.com/DeeJanuz/frametop#uninstall) in the README.
This removes the menu entry. With your `sudo` password, it also takes back the camera broker's permission to read the cameras. If you also installed Frametop's live hand tracking, the camera broker keeps that permission, because live hand tracking still uses it. Your recordings stay in `~/.local/share/frametop/hands/contrib`; delete that folder to remove them. To remove Frametop as well, follow [Uninstall](https://github.com/Frametop/frametop#uninstall) in the README.
## Help
Ask in the [Frametop Discord](https://discord.gg/W3X9f7z3Bc), the [Frametop issues](https://github.com/DeeJanuz/frametop/issues), or the dataset's [discussion page](https://huggingface.co/datasets/DeeJanuz/frametop-hands/discussions). All three are public.
Ask in the [Frametop Discord](https://discord.gg/W3X9f7z3Bc), the [Frametop issues](https://github.com/Frametop/frametop/issues), or the dataset's [discussion page](https://huggingface.co/datasets/DeeJanuz/frametop-hands/discussions). All three are public.
+3 -2
View File
@@ -40,6 +40,7 @@ Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides
- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (`sides.json`), so recordings are labelled right. The example config said `0` until 2026-10-05; `scripts/conf-migrate.sh` (run by `install.sh` and `hands/rec/install.sh`) turns that untouched line into `auto`.
- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
- `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
- `HANDS_MODELS` (unset: `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to run fine-tuned models, such as the ones trained on the hand dataset, without passing options to every launcher. Those folders usually lack the `*-int8` files, so `--int8` won't load them.
- `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
The pointer helper's `POINTER_HANDS` and `POINTER_PINCH_*`/`POINTER_GRIP_*` settings are in "Pinches and grips in the pointer" below.
@@ -87,7 +88,7 @@ Options:
It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. The service starts it again, and it attaches to the new buffers.
**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands now catches this by itself (`track/sides.h`, `HANDS_SWAP_SIDES=auto`):
**Which camera is which:** video9 is `slam_right` (sensor `og01a1b 4-0036`), video13 is `slam_left` (`4-0060`), video6 is `upper_left` and video7 is `upper_right`; without the colour module the side pair is on video0 (`slam_right`) and video3. XRService says so in its log: `Found camera 'slam_left': ... v4l_subdev=/dev/v4l-subdev30` names each sensor, and each `TrackingCameraInit` line gives the device and subdev it opened. ft-hands and `camcheck.py` name the devices that way. Until 2026-10-05 they named them by the `TrackingCameraInit` index instead, which is only the order XRService opens them in (index 0 was `slam_right` on every start logged), and ft-camd bound each buffer queue to a device by the order of XRService's file descriptors, which changes when XRService restarts its cameras. The two mistakes made the side names come out swapped on most starts and right on some. ft-camd now asks each device which buffers it holds (`VIDIOC_QUERYBUF` names the descriptor XRService queued at each index) and only falls back to the order if that fails. With the names swapped, every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands still checks by itself, in case (`track/sides.h`, `HANDS_SWAP_SIDES=auto`):
- Whenever a hand's landmarks are found in two cameras at once (one of them a side camera), it intersects the rays through the 21 landmarks twice: once with the calibrations as named, once with the two side cameras exchanged. The same hand seen the right way meets within a few mm, in front of both cameras and as far away as its size says. The wrong way misses by centimetres or meets behind a camera.
- With the names wrong, the tracker never gets such pairs on its own: it hands the hand over to where the wrong calibration puts it and finds nothing there. So 5 times a second while undecided, the check places a tracked hand in 3D under the other naming and runs the landmark model where that puts it in the other side camera.
- It decides after 10 votes one way and none the other, or 20 with at most a fifth the other way, over at least 1 s. That takes about 1-2 s of hands in view. If the names are backwards, it exchanges them; the tracked views move with their images. Then it checks once more, more strictly.
@@ -264,7 +265,7 @@ Guesses, not tested:
## Known issues
- **The side cameras can come out swapped.** ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and some XRService restarts reverse it. ft-hands corrects it from the hands (`HANDS_SWAP_SIDES=auto`, the default). Until it has seen about 1-2 s of hands in both namings' reach, the cutouts may sit beside the hands. ft-camd itself still can't tell.
- **The side cameras could come out swapped** (fixed 2026-10-05, see "Which camera is which"). If they still do, ft-hands corrects it from the hands (`HANDS_SWAP_SIDES=auto`, the default): until it has seen about 1-2 s of hands in both namings' reach, the cutouts may sit beside the hands. ft-camd's log says `bound by VIDIOC_QUERYBUF` for each camera when its buffers were matched exactly.
- **The colour cameras can't be used while the headset is worn.** The colour module then writes only a half-size image into the top-left quarter of its buffers, and ft-camd drops those frames. So the service runs the mono cameras only, and tracking in bright light, where the mono cameras see dark hands, doesn't get the colour pair's help.
- **The colour calibration mapping isn't settled.** Which colour camera is `passthrough_left` (`HANDS_COLOR_LEFT`) and how the module's crop applies (`HANDS_COLOR_CROP`) still need `tools/check_color.py` on a recording with a lit, textured view.
- **Depth when one camera loses the hand.** A hand seen in one camera drifts 10% per update toward the one-camera depth guess (`kMonoDepthGain`, 0.1, in `track/tracker.cpp`). In the 2026-09-30 replays that was worse than keeping the last distance (see "3D" above). A smaller gain, such as 0.02, is the next thing to try.
+32 -10
View File
@@ -3,7 +3,7 @@
ft-hands see them all?
The Frame has four mono IR tracking cameras: the side pair slam_left and slam_right
(/dev/video9 and /dev/video13) and the upper pair (/dev/video6 and /dev/video7). With the
(/dev/video13 and /dev/video9) and the upper pair (/dev/video6 and /dev/video7). With the
Arcturus colour module attached, SteamVR's XRService loads an FPGA image ("VCINT") onto the
module whenever it opens the cameras (at start and after every wake). When that load fails
(seen 2026-10-02 17:02, after a sleep), XRService runs only the two side cameras, the IR
@@ -40,7 +40,7 @@ import time
LOG_DIR = os.path.expanduser("~/.local/share/Steam/logs")
LOG_LINK = os.path.join(LOG_DIR, "xrservice.txt")
SIDE_NODES = (9, 13) # slam_left, slam_right (TrackingCameraInit index 0 and 1)
SIDE_NODES = (9, 13) # the side pair: slam_right on video9, slam_left on video13 (see camera_map)
UPPER_NODES = (6, 7) # the upper pair (index 2 and 3)
TRACKING = 4
@@ -55,15 +55,20 @@ USER_TEXT = ("The headset's upper cameras and IR light are off. SteamVR couldn't
ANSI = re.compile(r"\x1b\[[0-9;]*m")
STAMP = re.compile(r"^\w{3} \w{3} \d{2} \d{4} (\d{2}:\d{2}:\d{2})\.\d+ (\w+): ?(.*)$")
# Lines worth reading; anything else is skipped before the regexes (the log grows by MBs a day).
KEYS = ("FPGA", "VCINT", "Created", "TrackingCameraInit", "Closing tracking camera", "Streaming",
KEYS = ("FPGA", "VCINT", "Created", "TrackingCameraInit", "Found camera", "Closing tracking camera", "Streaming",
"systemd suspend", "systemd resume", "XRService logging to", "Exiting XRService", "ISP ")
# XRService's numbering of its tracking cameras (the TrackingCameraInit index): ft-hands names
# them this way too (track/main.cpp, cameras_from_xrservice_log).
# The tracking cameras' names. XRService says which sensor subdev each name is ("Found camera
# 'slam_left': ... v4l_subdev=/dev/v4l-subdev30", once per instance) and which subdev and video
# device each TrackingCameraInit index opened. The index is only the order it opens them in: on
# every start logged since 2026-10-04, index 0 was slam_right. A log without the "Found camera"
# lines falls back to this order. ft-hands names them the same way (track/main.cpp,
# cameras_from_xrservice_log).
NAMES = ("slam_left", "slam_right", "upper_left", "upper_right")
RE_FOUND = re.compile(r"Found camera '(\w+)': interface=\S+ v4l_subdev=(\S+)")
RE_PASSTHRU = re.compile(r"Passthrough connected but FPGA is (\S+) - loading VCINT")
RE_INTERLEAVE = re.compile(r"Upper cameras FPGA interleaving support: (\d)")
RE_TASKS = re.compile(r"Created (\d+) tasks \((\d+) tracking, (\d+) passthrough\)")
RE_INIT = re.compile(r"TrackingCameraInit: index: (\d+)\. video device: /dev/video(\d+)")
RE_INIT = re.compile(r"TrackingCameraInit: index: (\d+)\. video device: /dev/video(\d+)(?:\. v4l subdevice: (\S+))?")
RE_STREAM = re.compile(r"Streaming resumed \(FPGA: (\S+), VC interleaving: (\w+)\)")
RE_STATE = re.compile(r"FPGA state check: (\S+)")
# Without the colour module XRService runs the side cameras through the ISP, as NV12 on other
@@ -85,13 +90,15 @@ class LogState:
self.closed_at = ""
self.episode = None
self.nodes = {} # TrackingCameraInit index -> /dev/videoN, from the whole log
self.subdevs = {} # TrackingCameraInit index -> its sensor subdev, from the whole log
self.found = {} # sensor subdev -> camera name ("Found camera" lines)
self.failures = [] # [(time, line)]: every VCINT failure in this log
self.lines = 0
def _new_episode(self, t):
self.closed = False
self.episode = {"start": t, "fpga_before": "", "vcint": "", "interleave": None, "tasks": None,
"inits": {}, "stream": "", "isp": None, "failure": "", "evidence": []}
"inits": {}, "init_subdevs": {}, "stream": "", "isp": None, "failure": "", "evidence": []}
if self.closed_at:
self.episode["evidence"].append(self.closed_at)
@@ -164,12 +171,18 @@ class LogState:
ep["tasks"] = tuple(int(v) for v in m.groups())
ep["evidence"].append(short)
return
m = RE_FOUND.search(text)
if m:
self.found[m.group(2)] = m.group(1)
return
m = RE_INIT.search(text)
if m:
ep = self._ep(t)
idx, node = int(m.group(1)), int(m.group(2))
ep["inits"][idx] = node
self.nodes[idx] = node
if m.group(3):
ep["init_subdevs"][idx] = self.subdevs[idx] = m.group(3)
ep["evidence"].append(short)
return
m = RE_STREAM.search(text)
@@ -198,9 +211,18 @@ class LogState:
def camera_map(self):
"""{calibration name: /dev/videoN's N} from the latest camera start's TrackingCameraInit
lines (the whole log's when that start has none yet)."""
inits = (self.episode or {}).get("inits") or self.nodes
return {NAMES[i]: node for i, node in sorted(inits.items()) if 0 <= i < len(NAMES)}
lines (the whole log's when that start has none yet): each one's subdev named by the
"Found camera" lines, else by the index (see NAMES)."""
ep = self.episode or {}
inits, subdevs = (ep.get("inits"), ep.get("init_subdevs")) if ep.get("inits") else (self.nodes, self.subdevs)
out = {}
for i, node in sorted(inits.items()):
name = self.found.get(subdevs.get(i))
if name not in NAMES:
name = NAMES[i] if 0 <= i < len(NAMES) else None
if name:
out[name] = node
return out
def tracking_nodes(self):
got = tuple(self.nodes[i] for i in range(TRACKING) if i in self.nodes)
+4 -3
View File
@@ -264,11 +264,11 @@ static void setup_camera(cam_t *c, xr_camera_t *cam, int pidfd)
xr_slugify(cam->sensor, sensor, sizeof(sensor));
snprintf(c->slug, sizeof(c->slug), "%.20s_video%d", sensor, cam->node);
bool own = false;
bool own = false, exact = false;
for (int g = 0; g < xr.ngroups; g++)
if (xr.groups[g].cam == cam && group_fits(&xr.groups[g], cam, c->need))
own = true;
own = true, exact = exact || xr.groups[g].exact;
char model[16];
model_of(cam->sensor, model, sizeof(model));
@@ -318,7 +318,8 @@ static void setup_camera(cam_t *c, xr_camera_t *cam, int pidfd)
}
printf(" %-24s %-12s %ux%u pitch %u, %d candidate buffers%s\n", c->slug, cam->path,
c->lay.width, c->lay.height, c->lay.pitch, c->nslots, own ? "" : " (shared run)");
c->lay.width, c->lay.height, c->lay.pitch, c->nslots,
!own ? " (shared run)" : exact ? ", bound by VIDIOC_QUERYBUF" : ", bound by open order");
}
/* ------------------------------------------------------ index -> buffer */
+98 -5
View File
@@ -9,8 +9,11 @@
* - The V4L2 nodes and sensor subdevs it holds open come from /proc/<pid>/fd.
* - Each node's geometry comes from VIDIOC_G_FMT on our own handle.
* - Each node is traced back to its sensor through MEDIA_IOC_G_TOPOLOGY.
* - Buffers are split into queues by allocation order: XRService opens a
* sensor subdev, then allocates that camera's buffers.
* - Buffers are split into runs by allocation order, and each run is bound to
* its camera by VIDIOC_QUERYBUF on the camera's node, which names the
* descriptor XRService queued at each index. Where that fails, the sensor
* subdev opened just before the run decides (XRService opens a sensor's
* subdev, then allocates its buffers), which isn't always right.
*/
#define _GNU_SOURCE
@@ -457,6 +460,38 @@ static bool scan_xr_fds(pid_t pid, char *err, size_t errn)
/* ------------------------------------------------------ camera discovery */
/*
* Which of XRService's buffers each V4L2 index of a camera holds. vb2 lets any handle query a
* queue's buffers, and for a DMABUF buffer it returns the descriptor its owner last queued it
* with: that descriptor's number in XRService's fd table, the same numbers scan_xr_fds reads
* from /proc. XRService queues each index with the same buffer every time.
*/
static void query_buffers(int fd, xr_camera_t *c, bool mplane)
{
c->nqbuf = 0;
for (int i = 0; i < XR_MAX_RUNBUFS; i++) {
struct v4l2_buffer b;
struct v4l2_plane planes[VIDEO_MAX_PLANES];
memset(&b, 0, sizeof(b));
memset(planes, 0, sizeof(planes));
b.index = (unsigned)i;
b.type = mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE : V4L2_BUF_TYPE_VIDEO_CAPTURE;
if (mplane) {
b.m.planes = planes;
b.length = VIDEO_MAX_PLANES;
}
if (ioctl(fd, VIDIOC_QUERYBUF, &b) < 0 || b.memory != V4L2_MEMORY_DMABUF)
break; /* EINVAL past the last index */
c->qbuf_xfd[c->nqbuf++] = mplane ? planes[0].m.fd : b.m.fd;
}
}
static void probe_cameras(xr_state_t *st)
{
int seen[64];
@@ -521,6 +556,8 @@ static void probe_cameras(xr_state_t *st)
c->planesize[0] = fmt.fmt.pix.sizeimage;
}
query_buffers(fd, c, fmt.type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
struct stat sb;
if (fstat(fd, &sb) == 0) {
@@ -630,6 +667,46 @@ const char *xr_fmt_name(xr_fmt_t f)
/* ------------------------------------------------------- buffer grouping */
/* The camera whose VIDIOC_QUERYBUF names this XRService descriptor, or NULL. */
static xr_camera_t *qbuf_owner(xr_state_t *st, int xfd)
{
for (int c = 0; c < st->ncameras; c++)
for (int k = 0; k < st->cameras[c].nqbuf; k++)
if (st->cameras[c].qbuf_xfd[k] == xfd)
return &st->cameras[c];
return NULL;
}
/*
* A run can hold two cameras' queues when nothing between them in the fd table ends it (the
* upper pair, both 640x480). Where VIDIOC_QUERYBUF says so, cut it where the owner changes.
*/
static void split_groups(xr_state_t *st)
{
for (int i = 0; i < st->ngroups && st->ngroups < XR_MAX_GROUPS; i++) {
xr_group_t *g = &st->groups[i];
xr_camera_t *first = qbuf_owner(st, g->buf[0].xfd);
int cut = -1;
for (int b = 1; first && b < g->nbufs && cut < 0; b++) {
xr_camera_t *o = qbuf_owner(st, g->buf[b].xfd);
if (o && o != first)
cut = b;
}
if (cut < 0)
continue;
xr_group_t *t = &st->groups[st->ngroups++];
*t = *g;
t->nbufs = g->nbufs - cut;
memmove(t->buf, g->buf + cut, (size_t)t->nbufs * sizeof(t->buf[0]));
g->nbufs = cut;
}
}
/*
* XRService allocates one udmabuf per plane, plane 0 then plane 1, a whole
* queue at a time right after opening the sensor's subdev. Plane 1 matches
@@ -703,6 +780,8 @@ static void build_groups(xr_state_t *st)
i++; /* consume the plane 1 descriptor */
}
split_groups(st);
int keep = 0;
for (int i = 0; i < st->ngroups; i++)
@@ -712,7 +791,21 @@ static void build_groups(xr_state_t *st)
st->ngroups = keep;
/*
* Bind each run to a camera. The sensor marker alone can be wrong: XRService
* Bind each run to a camera: exactly where a camera's VIDIOC_QUERYBUF names the run's first
* buffer (query_buffers). The two side cameras have the same format, so for them nothing
* else is sure.
*/
for (int i = 0; i < st->ngroups; i++)
for (int c = 0; c < st->ncameras && !st->groups[i].cam; c++)
for (int k = 0; k < st->cameras[c].nqbuf; k++)
if (st->cameras[c].qbuf_xfd[k] == st->groups[i].buf[0].xfd) {
st->groups[i].cam = &st->cameras[c];
st->groups[i].exact = true;
break;
}
/*
* The rest by the sensor marker and sizes. The sensor marker alone can be wrong: XRService
* sometimes opens another sensor's subdev (e.g. the idle color camera)
* between an upper camera's subdev and its buffers, and two upper cameras
* can resolve to the same sensor name. So a marker match must also fit the
@@ -792,9 +885,9 @@ void xr_print(const xr_state_t *st, FILE *f)
const xr_group_t *g = &st->groups[i];
fprintf(f, " queue %d: %d buffers plane0=%zu plane1=%zu fds %d..%d sensor '%s' -> %s\n",
fprintf(f, " queue %d: %d buffers plane0=%zu plane1=%zu fds %d..%d sensor '%s' -> %s%s\n",
i, g->nbufs, g->planesize[0], g->planesize[1],
g->buf[0].xfd, g->buf[g->nbufs - 1].xfd1, g->sensor,
g->cam ? g->cam->path : "(unbound)");
g->cam ? g->cam->path : "(unbound)", g->exact ? " (VIDIOC_QUERYBUF)" : "");
}
}
+3
View File
@@ -32,6 +32,8 @@ typedef struct {
uint32_t pixfmt;
char sensor[XR_SENSOR_LEN]; /* media entity name of the sensor */
const char *role;
int nqbuf; /* V4L2 indices VIDIOC_QUERYBUF named */
int qbuf_xfd[XR_MAX_RUNBUFS]; /* index -> its plane 0 fd in XRService */
} xr_camera_t;
typedef struct {
@@ -48,6 +50,7 @@ typedef struct {
xr_bufref_t buf[XR_MAX_RUNBUFS];
char sensor[XR_SENSOR_LEN]; /* from the preceding sensor subdev */
xr_camera_t *cam;
bool exact; /* cam is from VIDIOC_QUERYBUF, not the order */
} xr_group_t;
typedef struct {
+1 -1
View File
@@ -294,7 +294,7 @@ Before section 7: "Put on both controllers and tighten the straps". Before secti
- **Hands seen:** from the hands file. A hand counts as seen if its flags match the side and the file is fresh (publish within 0.3 s). Prompts with `hands` set show the `hands` chips. If an asked-for hand is lost for more than 1.5 s, the note says "I can't see your left hand: bring it into view".
- **Controller tracking:** in sections 8 and 9, `devices` is polled once a second. A result other than 200 for more than 1 s says "The left controller lost tracking: turn your palm slightly toward you". Each such stretch goes into `prompts.jsonl` as `feedback` with `"controller": {...}`.
- **Lighting, measured:** the checklist page measures the light when it opens, starting ft-camd (`frametop-handrec-camd.service`) if nothing runs it; the window stops it again on quit if it started it. The mean of every mono camera's `mean` and `dark_mean` from the ring (`hands/tools/ring.py` layout; struct only, no numpy) is compared with the person's earlier sessions. If it matches an earlier round's within 15%, the window says so before starting.
- **Lighting label:** "Measured by the cameras" is the default. `ambient_ir`, the mono cameras' mean `dark_mean` (the room's infrared), labels the round `daylight` from 6.0 and `indoor` below (`session.classify_lighting`). Lamps and LEDs give off hardly any infrared, so a dim room and a lit one read about the same (1.8 by one lamp, 2.2 in a lamp-lit room) and the cameras can't tell them apart; the person can pick dim, room or daylight instead (`source: "picked"`). The 6.0 threshold is a guess until a daylight round is measured.
- **Lighting label:** "Measured by the cameras" is the default. `ambient_ir`, the mono cameras' mean `dark_mean` (the room's infrared), labels the round `daylight` from 6.0 and `indoor` below (`session.classify_lighting`). Lamps and LEDs give off hardly any infrared, so a dim room and a lit one read about the same (1.8 by one lamp, 2.2 in a lamp-lit room) and the cameras can't tell them apart; the person can pick dim, room or daylight instead (`source: "picked"`). The 6.0 threshold was a guess, and the first daylight round (dataset PR #5, 2026-10-05, a room with big sunlit windows) read 2.38 and was labelled `indoor`: the windows are a small part of each picture. So the checklist now asks people to pick Daylight when sunlight comes in, and the maintainer corrects the label in review (`hub_review.py lighting`) from what the pictures show: windows lit by the sun, the time of day. Hands that stand out little from the room (PR #5: a median 1.15x as bright as their surroundings, against 1.5-1.7x in some lamp-lit rounds) go with daylight, but a pale room at night read 1.22 (PR #6), so that alone doesn't decide it.
### Camera check
+2 -1
View File
@@ -362,7 +362,8 @@ Kirigami.ApplicationWindow {
wrapMode: Text.Wrap
opacity: 0.7
text: "Each round in a different light helps the most: dim, a normal room, daylight. The cameras "
+ "tell daylight from indoor light themselves; to say dim or a normal room, pick it here."
+ "can't tell daylight on their own (a sunlit room has measured as indoor light), so if "
+ "sunlight comes into the room, pick Daylight here; dim or a normal room the same way."
}
Kirigami.InlineMessage {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+8 -2
View File
@@ -309,7 +309,7 @@ def similar_lighting(base_dir, lighting):
return None
DAYLIGHT_IR = 6.0 # ambient IR (the mono cameras' mean dark_mean) from which it's daylight
DAYLIGHT_IR = 6.0 # ambient IR (the mono cameras' mean dark_mean) from which it's daylight; see classify_lighting
def ambient_ir(ring):
@@ -323,7 +323,13 @@ def ambient_ir(ring):
def classify_lighting(ring):
""""daylight" or "indoor" from the room's infrared light, "" if it can't tell. Sunlight
carries a lot of infrared; lamps and LEDs hardly any, so a dim room and a bright one read
about the same (2026-10: 1.8 by one lamp, 2.2 in a lamp-lit room) and aren't told apart."""
about the same (2026-10: 1.8 by one lamp, 2.2 in a lamp-lit room) and aren't told apart.
Nor does a sunlit room reliably: the first daylight round (dataset PR #5, big sunlit windows)
read 2.38, since the windows are a small part of each picture and the mean barely moves. So
"indoor" means "no strong daylight on the cameras", and the window asks people to pick
daylight themselves. Review corrects a missed one from the pictures (sunlit windows, the time
of day): hands standing out little from the room goes with daylight but also with pale rooms
at night, so it isn't a test either."""
ir = ambient_ir(ring)
if ir is None:
return ""
+4 -3
View File
@@ -1,8 +1,9 @@
"""Which side camera is which, in recordings: the rules every reader shares.
ft-camd tells slam_left's buffers from slam_right's by the order XRService allocated them, and
some XRService starts reverse it: then each side camera's images carry the other's name. A
tracking ft-hands tells from the hands (hands/track/sides.h, HANDS_SWAP_SIDES=auto) and publishes
Before 2026-10-05, ft-hands named the side cameras by XRService's start-up order and ft-camd bound
their buffers by XRService's descriptor order, so on most starts each side camera's images carried
the other's name (hands/README.md, "Which camera is which"). Both are exact now, and a
tracking ft-hands still tells from the hands (hands/track/sides.h, HANDS_SWAP_SIDES=auto) and publishes
what it found in /run/user/UID/frametop-hands/sides.json (read_live). Two things are recorded:
swapped whether ft-camd's naming was backwards during the recording (the truth);
+18
View File
@@ -148,6 +148,24 @@ class SyntheticLogs(unittest.TestCase):
st = state(START + GOOD_OPEN)
self.assertEqual(st.camera_map(), {"slam_left": 9, "slam_right": 13, "upper_left": 6, "upper_right": 7})
def test_camera_map_by_found_camera(self):
"""XRService's own naming (2026-10-05's log): index 0 is slam_right, by its subdev."""
found = [L("10:00:00", "DeckardCaptureSource: Found camera '%s': interface=msm_csiphy%d v4l_subdev=/dev/v4l-subdev%d "
"driver=/sys/bus/i2c/drivers/%s" % f) for f in (
("slam_left", 0, 30, "og01a1b/4-0060"), ("slam_right", 1, 31, "og01a1b/4-0036"),
("upper_left", 0, 32, "og0ve10/5-0060"), ("upper_right", 1, 33, "og0ve10/5-003e"))]
inits = [L("10:00:03", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: /dev/v4l-subdev%d"
% f) for f in ((0, 9, 31), (1, 13, 30), (2, 6, 32), (3, 7, 33))]
st = state(START + found + GOOD_OPEN[:-4] + inits)
self.assertEqual(st.camera_map(), {"slam_right": 9, "slam_left": 13, "upper_left": 6, "upper_right": 7})
# without the colour module the sides are on video0 and video3, named the same way
unplug = [L("10:30:00", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: /dev/v4l-subdev%d"
% f) for f in ((0, 0, 31), (1, 3, 30))]
st = state(START + found + GOOD_OPEN[:-4] + inits + CLOSE + unplug)
self.assertEqual(st.camera_map(), {"slam_right": 0, "slam_left": 3})
# a new instance forgets the old one's names: back to the index order
self.assertEqual(state(START + found + START + GOOD_OPEN).camera_map()["slam_left"], 9)
def test_module_unplugged(self):
st = state(START + GOOD_OPEN + UNPLUG)
self.assertEqual(st.verdict()[0], "ok")
+4 -4
View File
@@ -14,9 +14,9 @@ line: {"pair", "verdict": "named"|"swapped"|"unknown", "named", "swapped", "matc
ft-hands decides the side cameras' naming by itself (HANDS_SWAP_SIDES=auto, track/sides.h);
this is the independent check, from the scene rather than hands.
ft-camd tells the two side cameras' buffers apart by the order XRService allocated them,
and after some XRService restarts that order puts each camera's images under the other's
name. The tracker then sees every hand in one camera only, at the wrong depth. This
Before 2026-10-05 the side cameras' images often carried each other's names (hands/README.md,
"Which camera is which"); recordings from then may still. The tracker then sees every hand in
one camera only, at the wrong depth. This
matches features between the two images and measures how close each pair's rays pass
with the factory calibration, once as named and once swapped: true matches meet in
front of both cameras only under the right naming.
@@ -34,7 +34,7 @@ from tools.show_set import index, read_set # noqa: E402
from tools import calib # noqa: E402
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right', # with the colour module
PIPES = {'msm_vfe3_video0': 'slam_right', 'msm_vfe4_video0': 'slam_left', # with the colour module
'msm_vfe2_video0': 'upper_left', 'msm_vfe2_video1': 'upper_right'}
+52 -20
View File
@@ -22,9 +22,10 @@
// own clock, so they're placed on the mono cameras' by when they were dequeued, less the
// mono cameras' measured delay.
//
// Which side camera is which (--sides, HANDS_SWAP_SIDES): ft-camd tells slam_left's buffers from
// slam_right's by XRService's allocation order, which some XRService starts reverse. auto (the
// default) tells from the hands it tracks (track/sides.h): once it's sure, it exchanges the two
// Which side camera is which (--sides, HANDS_SWAP_SIDES): the names come from XRService's log
// (cameras_from_xrservice_log) and ft-camd matches each camera's buffers exactly (VIDIOC_QUERYBUF),
// so they should be right; before 2026-10-05 they were often swapped. auto (the default) still
// tells from the hands it tracks (track/sides.h): once it's sure, it exchanges the two
// cameras if they're backwards (the tracked views move with their images), and checks once
// more. 0 and 1 force the naming (1: exchanged; --swap-sides is --sides 1); it still checks,
// and if the hands disagree it warns and publishes what the hands say as the truth ("swapped"),
@@ -36,7 +37,9 @@
// options override both): HANDS_SWAP_SIDES (auto, 0 or 1), HANDS_CPUS (as --cpus),
// HANDS_CAMERAS, HANDS_BRIGHT, HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT (which
// colour camera is passthrough_left: color_video0 or color_video3), HANDS_COLOR_CROP
// (subtract or none: tools/check_color.py tells both).
// (subtract or none: tools/check_color.py tells both), HANDS_MISREAD_GUARD (0 or 1: the
// tracker's guards for fine-tuned landmark models, Tracker::set_misread_guard), HANDS_MODELS (as
// --models: a folder with palm.ncnn.* and hand.ncnn.*, e.g. fine-tuned ones).
#include "io.h"
#include "pinch.h"
#include "record.h"
@@ -47,6 +50,7 @@
#include <sys/stat.h>
#include <unistd.h>
#include <algorithm>
#include <cmath>
#include <ctime>
#include <memory>
@@ -64,31 +68,53 @@ namespace {
volatile std::sig_atomic_t g_stop = 0, g_record = 0;
// Which calibrated camera each video device carries. XRService numbers its tracking cameras
// (index 0 to 3: slam_left, slam_right, upper_left, upper_right) and logs the device each one
// opened ("TrackingCameraInit: index: 0. video device: /dev/video9"). The devices depend on
// the colour module: with it, the side cameras are on vfe3 and vfe4 and the upper pair on
// vfe2; without it, XRService runs the side cameras through the ISP on vfe0 and vfe1, and the
// upper pair on vfe3 and vfe4. So the running XRService's log decides; when it can't be read,
// the capture pipes as they are with the module. {} if the log has no cameras.
// Which calibrated camera each video device carries. XRService names each tracking camera's
// sensor subdev once per instance ("Found camera 'slam_left': interface=msm_csiphy0
// v4l_subdev=/dev/v4l-subdev30 ...") and logs the device and subdev each index opened at every
// camera start ("TrackingCameraInit: index: 0. video device: /dev/video9. v4l subdevice:
// /dev/v4l-subdev31"). The index is only the order it opens them in: on every start logged since
// 2026-10-04, index 0 was slam_right. So the subdev names the device; a log without "Found
// camera" lines falls back to the index order (slam_left, slam_right, upper_left, upper_right).
// The devices depend on the colour module: with it, the side cameras are on vfe3 (slam_right)
// and vfe4 (slam_left) and the upper pair on vfe2; without it, XRService runs the side cameras
// through the ISP on vfe0 and vfe1, and the upper pair on vfe3 and vfe4. So the running
// XRService's log decides; when it can't be read, the capture pipes as they are with the module.
// {} if the log has no cameras.
std::map<int, std::string> cameras_from_xrservice_log() {
static const char *const names[] = {"slam_left", "slam_right", "upper_left", "upper_right"};
const char *home = std::getenv("HOME");
std::ifstream in(std::string(home ? home : "") + "/.local/share/Steam/logs/xrservice.txt");
const std::string key = "TrackingCameraInit: index: ";
std::map<int, int> node_of; // index -> N of /dev/videoN, from the latest camera start
const std::string key = "TrackingCameraInit: index: ", found_key = "Found camera '";
std::map<int, std::pair<int, std::string>> init; // index -> (N of /dev/videoN, subdev), latest start
std::map<std::string, std::string> name_of; // subdev -> camera name
std::string line;
while (std::getline(in, line)) {
if (line.find("XRService logging to") != std::string::npos) node_of.clear();
if (line.find("XRService logging to") != std::string::npos) init.clear(), name_of.clear();
if (const auto at = line.find(found_key); at != std::string::npos) {
const auto name_end = line.find('\'', at + found_key.size());
const auto sub = line.find("v4l_subdev=", at);
if (name_end != std::string::npos && sub != std::string::npos) {
const auto sub_end = line.find_first_of(" \t\r", sub + 11);
name_of[line.substr(sub + 11, sub_end == std::string::npos ? std::string::npos : sub_end - sub - 11)] =
line.substr(at + found_key.size(), name_end - at - found_key.size());
}
continue;
}
const auto at = line.find(key);
int index = -1, node = -1;
char subdev[64] = "";
if (at != std::string::npos &&
std::sscanf(line.c_str() + at + key.size(), "%d. video device: /dev/video%d", &index, &node) == 2 &&
std::sscanf(line.c_str() + at + key.size(), "%d. video device: /dev/video%d. v4l subdevice: %63s", &index,
&node, subdev) >= 2 &&
index >= 0 && index < 4)
node_of[index] = node;
init[index] = {node, subdev};
}
std::map<int, std::string> out;
for (auto &[index, node] : node_of) out[node] = names[index];
for (auto &[index, ns] : init) {
const auto it = name_of.find(ns.second);
const bool known = it != name_of.end() && std::find(std::begin(names), std::end(names), it->second) != std::end(names);
out[ns.first] = known ? it->second : names[index];
}
return out;
}
@@ -97,8 +123,8 @@ const char *camera_for_pipe(int node) {
std::snprintf(path, sizeof path, "/sys/class/video4linux/video%d/name", node);
std::ifstream f(path);
f.getline(name, sizeof name);
if (!std::strcmp(name, "msm_vfe3_video0")) return "slam_left";
if (!std::strcmp(name, "msm_vfe4_video0")) return "slam_right";
if (!std::strcmp(name, "msm_vfe3_video0")) return "slam_right";
if (!std::strcmp(name, "msm_vfe4_video0")) return "slam_left";
if (!std::strcmp(name, "msm_vfe2_video0")) return "upper_left";
if (!std::strcmp(name, "msm_vfe2_video1")) return "upper_right";
return nullptr;
@@ -215,6 +241,7 @@ int main(int argc, char **argv) {
// latency 9.6 against 14.1 ms, and the compositor's late frames and CPU/GPU time didn't change.
std::vector<int> cpus = {5, 6, 7};
if (const auto c = parse_cpus(setting("HANDS_CPUS").c_str()); !c.empty()) cpus = c;
if (const std::string m = setting("HANDS_MODELS"); !m.empty()) models = m;
// Which side camera is which (see the top): auto, 0 or 1, and where that came from
std::string sides_mode = setting("HANDS_SWAP_SIDES"), sides_from = "config";
if (sides_mode.empty()) sides_mode = "auto", sides_from = "default";
@@ -232,6 +259,7 @@ int main(int argc, char **argv) {
// dim recording), but makes the landmarks jitter, so they get plain crops.
Contrast palm_contrast, hand_contrast{Contrast::None};
double keep_presence = 0.5; // landmark presence a tracked view needs to stay
bool misread_guard = setting("HANDS_MISREAD_GUARD") == "1";
PinchParams pinch_params;
GripParams grip_params;
bool gesture_log = false; // what the pinch and grip detectors measure, 10 times a second
@@ -262,6 +290,7 @@ int main(int argc, char **argv) {
else if (a == "--record-for" && more) record_for = std::atof(argv[++i]);
else if (a == "--record-hz" && more) record_hz = std::max(0.0, std::atof(argv[++i]));
else if (a == "--keep-presence" && more) keep_presence = std::atof(argv[++i]);
else if (a == "--misread-guard" && more) misread_guard = std::string(argv[++i]) == "1";
else if (a == "--cams" && more) cams_arg = argv[++i];
else if (a == "--bright" && more) bright_arg = argv[++i];
else if (a == "--bright-on" && more) light.on = std::atof(argv[++i]);
@@ -282,6 +311,7 @@ int main(int argc, char **argv) {
" [--sides auto|0|1] (auto: tell from the hands which side camera is which; 1: exchange them,\n"
" as --swap-sides; 0: as ft-camd names them)\n"
" [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
" [--misread-guard 0|1] (0; 1 for fine-tuned landmark models: see Tracker::set_misread_guard)\n"
" [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
" [--color-left color_video0|color_video3] [--color-crop subtract|none]\n"
" [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (1: off)\n"
@@ -294,7 +324,8 @@ int main(int argc, char **argv) {
"camera's newest dark frame, as <name>_dk; with --with-color, the color cameras' as color_video<N>.\n"
"auto picks the cameras by the light (see the top of track/main.cpp).\n"
"Settings in ~/.config/frametop.conf: HANDS_SWAP_SIDES=auto|0|1, HANDS_CPUS=5,6,7, HANDS_CAMERAS, HANDS_BRIGHT,\n"
"HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT, HANDS_COLOR_CROP (FT_<name> overrides).\n",
"HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT, HANDS_COLOR_CROP, HANDS_MISREAD_GUARD=0|1, HANDS_MODELS=DIR\n"
"(FT_<name> overrides).\n",
argv[0]);
return a == "--help" ? 0 : 1;
}
@@ -466,6 +497,7 @@ int main(int argc, char **argv) {
Pool pool(threads, cpus);
Tracker tracker(used, nets, pool);
tracker.set_keep_presence(keep_presence);
tracker.set_misread_guard(misread_guard);
std::map<std::string, std::vector<uint8_t>> pixels;
std::map<std::string, uint64_t> lit_seen; // per colour camera: the frame last counted for the light
const uint64_t start = mono_ns();
+4
View File
@@ -14,6 +14,7 @@
// --timeline: per processed set, a line per hand (time, id, side, views, wrist) and per view
// (hand, camera, presence, next crop, set index).
// --keep-presence P: landmark presence a tracked view needs to stay (default 0.5, as new ones).
// --misread-guard: the tracker's guards for fine-tuned landmark models (Tracker::set_misread_guard).
// --pinch-begin M, --pinch-end M, --pinch-triangulated, --pinch-palm-down MAX: the pinch detector (track/pinch.h);
// the timeline gets its begin/end/lost events and both distance measures per set.
// --grip-begin R, --grip-end R: the grip detector (a closed hand; track/pinch.h); the timeline
@@ -86,6 +87,7 @@ int main(int argc, char **argv) {
bool cost = false;
Contrast palm_contrast, hand_contrast{Contrast::None}; // as ft-hands's
double keep_presence = 0.5; // landmark presence a tracked view needs to stay
bool misread_guard = false;
PinchParams pinch_params;
GripParams grip_params;
std::string use = "mono", color_left = "color_video0", color_crop = "subtract", sides = "file";
@@ -102,6 +104,7 @@ int main(int argc, char **argv) {
else if (a == "--models" && more) models = argv[++i];
else if (a == "--cost") cost = true;
else if (a == "--keep-presence" && more) keep_presence = std::atof(argv[++i]);
else if (a == "--misread-guard") misread_guard = true;
else if (a == "--cams" && more) use = argv[++i];
else if (a == "--sides" && more) sides = argv[++i];
else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]);
@@ -183,6 +186,7 @@ int main(int argc, char **argv) {
return n == "slam_left" ? "slam_right" : n == "slam_right" ? "slam_left" : n;
};
tracker.set_keep_presence(keep_presence);
tracker.set_misread_guard(misread_guard);
FILE *dp = depth.empty() ? nullptr : std::fopen(depth.c_str(), "w");
FILE *pp = poses.empty() ? nullptr : std::fopen(poses.c_str(), "w");
if (dp)
+24 -4
View File
@@ -223,8 +223,16 @@ bool Tracker::hand_3d(Hand &hand, std::vector<View *> views, int64_t t_ns) {
if (residual > 0.03 || size_misfit({views.begin(), views.end()}, pts) < 0) {
View *best = *std::max_element(views.begin(), views.end(),
[](View *a, View *b) { return a->lm.presence < b->lm.presence; });
for (View *v : views)
if (v != best) v->hand = -1;
// a view that disagrees but sits where this hand already is in its camera is this hand
// misread: drop it (-2), and the hand-over gives a fresh crop next frame
for (View *v : views) {
if (v == best) continue;
v->hand = -1;
if (!misread_guard_ || !hand.has_pts) continue;
double z;
const V2 at = v->cam->project(hand.pts[9], &z);
if (z > 0 && norm(at - palm_centre(v->lm)) < 0.5 * hand_size(v->lm)) v->hand = -2;
}
++stats.splits;
return hand_3d(hand, {best}, t_ns);
}
@@ -416,7 +424,14 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
if (int(chosen.size()) > hand_budget_) chosen.resize(hand_budget_);
run_landmarks(images, chosen);
std::vector<View *> kept;
for (View *v : chosen)
for (View *v : chosen) {
if (misread_guard_ && v->has_lm) { // see set_misread_guard
const auto h = hands_.find(v->hand);
if (h != hands_.end() && h->second.frames >= 5 && std::fabs(v->lm.right - h->second.right_score) > 0.7) {
++stats.lost;
continue;
}
}
if (v->lm.presence >= (v->frames > 0 ? keep_presence_ : min_presence_)) {
v->roi = v->lm.next_roi();
++v->frames;
@@ -424,6 +439,7 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
} else {
++(v->frames > 0 ? stats.lost : stats.handoff_miss);
}
}
// the same hand twice in one camera: keep the more confident
std::sort(kept.begin(), kept.end(), [](View *a, View *b) { return a->lm.presence > b->lm.presence; });
std::vector<View> next;
@@ -537,7 +553,11 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
++it;
}
}
// views split off by a failed triangulation start over as new hands next frame
// views split off by a failed triangulation start over as new hands next frame, unless they
// were this hand misread (-2, the misread guard: dropped)
const size_t before = views_.size();
views_.erase(std::remove_if(views_.begin(), views_.end(), [](const View &v) { return v.hand == -2; }), views_.end());
stats.lost += int(before - views_.size());
for (View &v : views_)
if (v.hand <= 0) {
v.hand = next_id_++;
+8
View File
@@ -93,6 +93,13 @@ public:
// bright rooms the camera exposes for the room, the hands come out dim, and presence
// dips under 0.5 for a frame at a time.
void set_keep_presence(double p) { keep_presence_ = p; }
// Misread guards, for fine-tuned landmark models (frame-hands' students): they stay sure of a
// hand when two hands touch and can read the held hand as the other side, which made a split /
// hand-over / duplicate loop. On: a reading of an established hand (5+ frames) whose side is
// more than 0.7 off the hand's own is dropped (the hand-over crops it afresh next frame), and a
// view split off where its hand already is in that camera is dropped instead of starting a new
// hand. Off by default: the stock model's side is noisier and the first guard costs it tracking.
void set_misread_guard(bool on) { misread_guard_ = on; }
// One view's 3D hand: each landmark along its ray, as far as how big the palm looks says
// for a hand `scale` times the model's (Hand::scale). False if the palm is degenerate.
static bool single_view(const Camera &cam, const Landmarks &lm, double scale, V3 out[21]);
@@ -130,6 +137,7 @@ private:
Pool &pool_;
int max_views_, hand_budget_ = 4, search_budget_ = 3;
double min_presence_ = 0.5, keep_presence_ = 0.5;
bool misread_guard_ = false;
std::vector<View> views_;
std::map<int, Hand> hands_;
std::vector<Tile> tiles_;
+4
View File
@@ -0,0 +1,4 @@
@echo off
rem Frametop host setup: makes this PC a host for Frametop's remote displays (frametop-host-setup.ps1).
rem It asks Windows for admin.
powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0frametop-host-setup.ps1" %*
+327
View File
@@ -0,0 +1,327 @@
# Frametop host setup for Windows: makes this PC a host for Frametop's remote displays, its
# monitors shown as screens on a Steam Frame (docs/remote-displays.md). Run
# "Setup Frametop host.cmd"; it asks Windows for admin.
#
# 1. Vibepollo 2.0.0 (github.com/Nonary/Vibepollo), installed with its own installer if it
# isn't here (checked against its SHA-256 first).
# 2. Frametop's build of Vibepollo's sunshine.exe (github.com/Frametop/frametop-vibepollo):
# it can stream any of your monitors (not only the main one), keeps your monitor layout
# when a virtual display goes away, turns an HDR monitor's HDR off while Frametop streams
# it (back on after), and doesn't stall the Web UI. Downloaded from its release and
# checked against its SHA-256; the original is kept as sunshine.exe.2.0.0-original.
# 3. Settings Frametop needs (the rest of Vibepollo's settings stay as they are): remote
# displays carry the PC's sound, and a virtual display goes away when Frametop
# disconnects it, but stays through a dropped stream.
# 4. The Web UI login you sign in with once from the Steam Frame: keep the one there is, or
# set one. The password is typed here and never shown or saved.
# 5. Checks: Vibepollo's firewall rule covers every network type (a Steam Link dongle's
# network counts as Public), whether a Steam Link dongle is plugged in, and that the Web
# UI answers.
#
# Options:
# -FrametopBuild PATH|URL where Frametop's sunshine.exe comes from (default: BuildUrl
# below, else sunshine-frametop.exe next to this script)
# -SkipLogin leave the Web UI login alone
# -Check only say what would change
# -Undo put back the original sunshine.exe and the settings from before
# A log goes to %ProgramData%\Frametop\host-setup.log, backups to %ProgramData%\Frametop\backup-*.
param([string]$FrametopBuild = "", [switch]$SkipLogin, [switch]$Check, [switch]$Undo)
$ErrorActionPreference = "Stop"
$ProgressPreference = "SilentlyContinue"
$VibepolloVersion = "2.0.0"
$SetupUrl = "https://github.com/Nonary/Vibepollo/releases/download/2.0.0/VibepolloSetup-v2.0.0.exe"
$SetupSha = "7B3500EC0C774644CE5A435A48F61C046C48494D0F18B67AFA0B3561931794B7"
$OriginalSha = "2CC018FD92DDB4D3748D91D8DA25316909ED45DB3710D1FF278BD73D51EB00C1" # its sunshine.exe
# Frametop's build: Frametop/frametop-vibepollo release frametop-2.0.0-1 (its CI, from the tag).
$BuildSha = "57EECCF9CA6ECEC4F0C5AEBBB27AAFE711A700C9E0986601344A52682856E94E"
$BuildUrl = "https://github.com/Frametop/frametop-vibepollo/releases/download/frametop-2.0.0-1/sunshine.exe"
# Earlier Frametop builds, replaced by this one like the original is.
$OlderBuildShas = @(
"B5B7D2E7353454AEA6D895D0B68DE235E4CC581684C9DD44F8EFAF2E872F517F" # 2f032252, built by hand without WebRTC
"6AF4F34503C7F6F84D1F6967D9FCEFAF2BD8145B34044C992B5E26A614E37A91" # a84b6cfc, CI, before the HDR-off guard
)
$Settings = [ordered]@{
"remote_monitor_mute_audio" = "disabled"
"remote_monitor_disconnect_on_client_disconnect" = "enabled"
"remote_monitor_disconnect_on_stream_end" = "disabled"
}
$Service = "ApolloService" # Vibepollo keeps Apollo's service name
$Data = Join-Path $env:ProgramData "Frametop"
$Here = if ($PSScriptRoot) { $PSScriptRoot } else { (Get-Location).Path }
$me = [Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()
if (-not $me.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {
$pass = @()
foreach ($k in $PSBoundParameters.Keys) {
$v = $PSBoundParameters[$k]
if ($v -is [switch]) { if ($v) { $pass += "-$k" } } else { $pass += "-$k `"$v`"" }
}
Start-Process powershell -Verb RunAs -ArgumentList ("-NoProfile -ExecutionPolicy Bypass -File `"$PSCommandPath`" " + ($pass -join " "))
exit
}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
New-Item -ItemType Directory -Force $Data | Out-Null
$Log = Join-Path $Data "host-setup.log"
function Say([string]$text) {
Write-Host $text
Add-Content -Path $Log -Value ((Get-Date -Format "yyyy-MM-dd HH:mm:ss") + " " + $text)
}
function Sha([string]$path) { (Get-FileHash $path -Algorithm SHA256).Hash }
function Plain($secure) {
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
try { [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) } finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
}
# One argument for a Windows program, quoted so it reads back exactly this string
# (Windows PowerShell 5 mangles arguments that contain double quotes).
function Quote([string]$s) {
$r = '"'
$slashes = 0
foreach ($c in $s.ToCharArray()) {
if ($c -eq '\') { $slashes++; continue }
if ($c -eq '"') { $r += ('\' * (2 * $slashes + 1)) + '"' } else { $r += ('\' * $slashes) + $c }
$slashes = 0
}
$r + ('\' * (2 * $slashes)) + '"'
}
function Find-Vibepollo {
$keys = "HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*", "HKLM:\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*"
$entry = Get-ItemProperty $keys -ErrorAction SilentlyContinue | Where-Object { $_.DisplayName -eq "Vibepollo" -and $_.InstallLocation } | Select-Object -First 1
$dir = if ($entry) { $entry.InstallLocation.TrimEnd('\') } else { "C:\Program Files\Apollo" }
if ((Test-Path "$dir\sunshine.exe") -and (Get-Service $Service -ErrorAction SilentlyContinue)) { return $dir }
return $null
}
function Stop-Vibepollo {
Stop-Service $Service
for ($i = 0; $i -lt 30 -and (Get-Process sunshine -ErrorAction SilentlyContinue); $i++) { Start-Sleep -Milliseconds 500 }
}
# sunshine.conf: "key = value" lines. Returns the lines with $Settings applied.
function Set-Settings([string[]]$lines) {
$out = New-Object System.Collections.Generic.List[string]
$done = @{}
foreach ($line in $lines) {
$key = ($line -split "=", 2)[0].Trim()
if ($Settings.Contains($key)) {
if (-not $done[$key]) { $out.Add("$key = $($Settings[$key])"); $done[$key] = $true }
} else {
$out.Add($line)
}
}
foreach ($key in $Settings.Keys) { if (-not $done[$key]) { $out.Add("$key = $($Settings[$key])") } }
return , $out.ToArray()
}
function Get-Build {
$from = $FrametopBuild
if (-not $from) { $from = if ($BuildUrl) { $BuildUrl } else { Join-Path $Here "sunshine-frametop.exe" } }
if ($from -match "^https?://") {
$file = Join-Path $env:TEMP "sunshine-frametop.exe"
Say "Downloading Frametop's build of Vibepollo..."
Invoke-WebRequest -UseBasicParsing -Uri $from -OutFile $file
} else {
$file = $from
}
if (-not (Test-Path $file)) { return $null }
if ((Sha $file) -ne $BuildSha) { throw "$file isn't Frametop's build of Vibepollo $VibepolloVersion (its SHA-256 doesn't match)." }
return $file
}
function Wait-WebUi {
Add-Type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class FrametopTrustLocal : ICertificatePolicy {
public bool CheckValidationResult(ServicePoint s, X509Certificate c, WebRequest r, int p) { return true; }
}
"@ -ErrorAction SilentlyContinue
$old = [Net.ServicePointManager]::CertificatePolicy
[Net.ServicePointManager]::CertificatePolicy = New-Object FrametopTrustLocal # 127.0.0.1 only
try {
for ($i = 0; $i -lt 40; $i++) {
try { Invoke-WebRequest -UseBasicParsing -TimeoutSec 2 "https://127.0.0.1:47990/" | Out-Null; return $true } catch { Start-Sleep 1 }
}
return $false
} finally {
[Net.ServicePointManager]::CertificatePolicy = $old
}
}
function Main {
Say "== Frametop host setup$(if ($Check) { ' (check only)' })$(if ($Undo) { ' (undo)' })"
$dir = Find-Vibepollo
$exe = if ($dir) { "$dir\sunshine.exe" } else { $null }
$orig = if ($dir) { "$dir\sunshine.exe.$VibepolloVersion-original" } else { $null }
if ($Undo) {
if (-not $dir) { throw "Vibepollo isn't installed here." }
$backup = Get-ChildItem $Data -Directory -Filter "backup-*" -ErrorAction SilentlyContinue | Sort-Object Name | Select-Object -First 1
Stop-Vibepollo
if (Test-Path $orig) { Copy-Item $orig $exe -Force; Say "Put back the original sunshine.exe." }
if ($backup -and (Test-Path "$($backup.FullName)\sunshine.conf")) {
Copy-Item "$($backup.FullName)\sunshine.conf" "$dir\config\sunshine.conf" -Force
Say "Put back the settings from $($backup.FullName)."
}
Start-Service $Service
Say "Done. (The Web UI login stays as it is.)"
return
}
# 1. Vibepollo
if (-not $dir) {
if ($Check) { Say "Would install Vibepollo $VibepolloVersion."; return }
$setup = Join-Path $env:TEMP "VibepolloSetup-v$VibepolloVersion.exe"
Say "Downloading Vibepollo $VibepolloVersion..."
Invoke-WebRequest -UseBasicParsing -Uri $SetupUrl -OutFile $setup
if ((Sha $setup) -ne $SetupSha) { throw "The Vibepollo installer didn't match its SHA-256; not running it." }
Say "Running Vibepollo's installer: follow its steps, then come back here."
Start-Process $setup -Wait
$dir = Find-Vibepollo
if (-not $dir) { throw "Vibepollo doesn't seem to be installed. Run this again once it is." }
$exe = "$dir\sunshine.exe"
$orig = "$dir\sunshine.exe.$VibepolloVersion-original"
}
Say "Vibepollo: $dir"
# 2. Frametop's build of sunshine.exe
$now = Sha $exe
$swap = $null
if ($now -eq $BuildSha) {
Say "Frametop's build of Vibepollo is already installed."
} elseif ($now -eq $OriginalSha) {
$swap = Get-Build
if (-not $swap) {
Say "Frametop's build of Vibepollo isn't available here, so this PC can't stream its own monitors yet (virtual displays work)."
}
} elseif ($OlderBuildShas -contains $now) {
$swap = Get-Build
if (-not $swap) { Say "An older Frametop build of Vibepollo is installed, and the new one isn't available here; keeping it." }
} else {
$v = (Get-Item $exe).VersionInfo.ProductVersion
throw "This is Vibepollo $v, and Frametop's build is for $VibepolloVersion. Install Vibepollo $VibepolloVersion ($SetupUrl), then run this again."
}
# 3. Settings
$conf = "$dir\config\sunshine.conf"
$lines = if (Test-Path $conf) { [IO.File]::ReadAllLines($conf) } else { @() }
$new = Set-Settings $lines
$changed = ($new -join "`n") -ne ($lines -join "`n")
# 4. Login
$state = "$dir\config\sunshine_state.json"
$user = $null
if (Test-Path $state) { try { $user = (Get-Content $state -Raw | ConvertFrom-Json).username } catch { } }
$password = $null
if (-not $SkipLogin -and -not $Check) {
if ($user) {
$answer = Read-Host "The Web UI login is '$user'. Keep it? You'll sign in with it once from the Steam Frame. (Y/n)"
$setLogin = $answer -match "^[nN]"
} else {
Write-Host "Vibepollo has no Web UI login yet. Make one: you'll sign in with it once from the Steam Frame."
$setLogin = $true
}
if ($setLogin) {
$typed = Read-Host "User name$(if ($user) { " (Enter keeps '$user')" })"
if ($typed) { $user = $typed }
while (-not $user) { $user = Read-Host "User name" }
while ($true) {
$a = Plain (Read-Host "Password" -AsSecureString)
$b = Plain (Read-Host "Same password again" -AsSecureString)
if ($a.Length -ge 4 -and $a -ceq $b) { $password = $a; break }
Write-Host "They don't match, or it's shorter than 4 characters. Try again."
}
Remove-Variable a, b
}
}
if ($Check) {
Say "Would $(if ($swap) { 'install' } else { 'not change' }) sunshine.exe, $(if ($changed) { 'change' } else { 'not change' }) the settings."
} elseif ($swap -or $changed -or $password) {
$backup = Join-Path $Data ("backup-" + (Get-Date -Format "yyyyMMdd-HHmmss"))
New-Item -ItemType Directory $backup | Out-Null
foreach ($f in "$dir\config\sunshine.conf", "$dir\config\apps.json") { if (Test-Path $f) { Copy-Item $f $backup } }
Say "Backed up the settings to $backup."
Say "Stopping Vibepollo..."
Stop-Vibepollo
try {
if ($swap) {
if (-not (Test-Path $orig) -and $now -eq $OriginalSha) { Copy-Item $exe $orig }
Copy-Item $swap $exe -Force
Say "Installed Frametop's build of Vibepollo (the original is $orig)."
}
if ($changed) {
[IO.File]::WriteAllLines($conf, [string[]]$new)
Say "Set: $(($Settings.Keys | ForEach-Object { "$_ = $($Settings[$_])" }) -join ', ')."
}
if ($password) {
$info = New-Object Diagnostics.ProcessStartInfo
$info.FileName = $exe
$info.WorkingDirectory = $dir
$info.Arguments = "--creds " + (Quote $user) + " " + (Quote $password)
$info.UseShellExecute = $false
$info.RedirectStandardOutput = $true
$info.RedirectStandardError = $true
$proc = [Diagnostics.Process]::Start($info)
$info.Arguments = ""
$proc.StandardOutput.ReadToEnd() | Out-Null
$proc.StandardError.ReadToEnd() | Out-Null
$proc.WaitForExit()
Remove-Variable password
if ($proc.ExitCode -ne 0) { throw "Setting the Web UI login failed (sunshine.exe --creds: exit code $($proc.ExitCode))." }
Say "Web UI login set: '$user'."
}
} finally {
Start-Service $Service
Say "Vibepollo is running again."
}
}
# 5. Checks
$rules = @(Get-NetFirewallApplicationFilter -ErrorAction SilentlyContinue |
Where-Object { [Environment]::ExpandEnvironmentVariables($_.Program) -ieq $exe } |
Get-NetFirewallRule | Where-Object { $_.Enabled -eq "True" -and $_.Direction -eq "Inbound" -and $_.Action -eq "Allow" })
$everywhere = $rules | Where-Object { $_.Profile -eq "Any" -or ($_.Profile -match "Private" -and $_.Profile -match "Public") }
if ($everywhere) {
Say "Firewall: Vibepollo is allowed on every network type."
} elseif ($Check) {
Say "Would add a firewall rule letting Vibepollo in on every network type."
} else {
New-NetFirewallRule -DisplayName "Vibepollo (Frametop)" -Program $exe -Direction Inbound -Action Allow -Profile Any | Out-Null
Say "Firewall: added a rule letting Vibepollo in on every network type (a Steam Link dongle's network is Public)."
}
$dongle = Get-NetAdapter -ErrorAction SilentlyContinue | Where-Object { $_.InterfaceDescription -match "For Valve" } | Select-Object -First 1
if ($dongle -and $dongle.Status -eq "Up") {
$ip = (Get-NetIPAddress -InterfaceIndex $dongle.ifIndex -AddressFamily IPv4 -ErrorAction SilentlyContinue | Select-Object -First 1).IPAddress
Say "Steam Link dongle: connected$(if ($ip) { " ($ip)" }). Frametop can stream over it, past your router."
} elseif ($dongle) {
Say "Steam Link dongle: plugged in, not connected to the Steam Frame. Frametop uses your network until it is."
} else {
Say "Steam Link dongle: none. Frametop streams over your network."
}
if (-not $Check) {
if (Wait-WebUi) { Say "The Web UI answers (https://localhost:47990)." } else { Say "The Web UI didn't answer yet; check that Vibepollo is running." }
}
$addresses = Get-NetIPAddress -AddressFamily IPv4 -ErrorAction SilentlyContinue | Where-Object {
$_.IPAddress -notmatch "^(127\.|169\.254\.)" -and $_.InterfaceAlias -notmatch "Tailscale|vEthernet|Loopback" -and
(-not $dongle -or $_.InterfaceIndex -ne $dongle.ifIndex) } | ForEach-Object { $_.IPAddress }
Write-Host ""
Say "Done. On the Steam Frame: open Frametop Remote Displays, Add computer, pick $([Net.Dns]::GetHostName()) ($($addresses -join ', ')), and sign in as '$user'."
}
try {
Main
} catch {
Say "Failed: $_"
if (-not $Check -and (Get-Service $Service -ErrorAction SilentlyContinue) -and (Get-Service $Service).Status -ne "Running") {
Start-Service $Service -ErrorAction SilentlyContinue
}
}
if (-not $env:FRAMETOP_NO_PAUSE) { Read-Host "Press Enter to close" | Out-Null }
+2 -3
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# Launch Frametop Input Settings from a Plasma session on the Frame host.
# The app runs in the dev container (PySide6 and Kirigami come from Fedora there).
# The app runs in Frametop's container (PySide6 and Kirigami come from Fedora there).
# podman needs the real XDG_RUNTIME_DIR and the real user bus (to reach systemd for
# the container's cgroup; the Frametop session runs on a private bus from
# dbus-run-session). The session's Wayland socket and bus go to the app itself.
@@ -10,8 +10,7 @@ case $wl in /*) ;; *) wl="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/$wl" ;; esac
session_bus=${DBUS_SESSION_BUS_ADDRESS:-}
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
"$here/../scripts/container-up.sh"
exec "$HOME/.local/bin/distrobox" enter dev -- env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
exec "$here/../scripts/in-box" env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
QT_QPA_PLATFORM="wayland;xcb" \
python3 "$here/ft_input_settings.py" "$@"
+6 -1
View File
@@ -1235,7 +1235,12 @@ class Backend(QObject):
@Slot()
def applyBluetoothFixes(self):
if not os.path.exists("/run/host/etc/steamframe/bt-fixups.sh") and not os.path.exists("/etc/steamframe/bt-fixups.sh"):
self.message.emit("Bluetooth fixes aren't installed (setup/bluetooth/install.sh)", True)
# The unit outlives the script: SteamOS updates keep /etc's units, not what they run.
if any(os.path.exists(r + "/etc/systemd/system/steamframe-bt-fixups.service") for r in ("/run/host", "")):
self.message.emit("A SteamOS update deleted the Bluetooth fixes: reinstall them with "
"setup/bluetooth/install.sh install", True)
else:
self.message.emit("Bluetooth fixes aren't installed (setup/bluetooth/install.sh)", True)
return
result = host("pkexec", "/etc/steamframe/bt-fixups.sh")
if result.returncode == 0:
+18 -5
View File
@@ -27,7 +27,7 @@ look; held, the pointer stops there and your head steers it (it stays put in you
the release clicks; held still for half a second, it's a real press that your head drags
("gazekey left|right 1|0" to the helper; by default Meta+J and Meta+K, DEFAULT_KEY_BINDINGS),
gaze_quickcal = the gaze service's one-dot check ("quickcal" to @ft_gazed), sens_up, sens_down,
layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens,
layout_reset = open the profile in use again, or put the desktop screens back in their layout (ft-layout reset), screens_toggle = hide or show the desktop screens,
keyboard_toggle = open or close Frametop's keyboard, float_toggle = float the desktop window under the
pointer (else the active one) in VR, or put it back if it floats, dock_all = put every floating
window back (both to ft-floatd, @frametop_float), spin_next and spin_prev = turn every panel in the
@@ -91,7 +91,7 @@ default: only while no pass-through keyboard is connected; a program's uinput ke
doesn't count), "button" (only the keyboard_toggle action opens it), or "never"
(keyboard_toggle does nothing either). With "vr_keyboard_persist" (the default), it stays
open when the text field loses focus, until its Close key, keyboard_toggle, or a layout reset
(ft-layout apply) closes it.
(ft-layout reset) closes it.
Volume keys, from every device that has them (the headset's own buttons included),
are handled here: wpctl steps the default output. Nothing else may see a volume key,
@@ -658,7 +658,7 @@ class Pointer:
pass # ft-screens not running
elif name == "layout_reset":
# Runs a few seconds and borrows the pointer; ft-layout refuses a second copy.
subprocess.Popen([FT_LAYOUT, "apply"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
subprocess.Popen([FT_LAYOUT, "reset"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, start_new_session=True)
log("layout reset")
elif name in ("sens_up", "sens_down"):
@@ -932,9 +932,22 @@ def main():
if not focused:
if not state["rules"].get("vr_keyboard_persist", True):
vr_keyboard("hide") # ft-screens closes it only if it opened it for a text field
elif mode == "always" or (mode == "no_keyboard" and not any(
n.candidate and n.is_keyboard and n.role == "passthrough" and not n.uinput for n in nodes.values())):
return
keyboards = sorted({n.name for n in nodes.values()
if n.candidate and n.is_keyboard and n.role == "passthrough" and not n.uinput})
if mode == "always" or (mode == "no_keyboard" and not keyboards):
vr_keyboard("show")
why = "asking ft-screens to open Frametop's keyboard"
elif mode == "no_keyboard":
why = (f"not opening Frametop's keyboard: a keyboard is connected ({', '.join(keyboards)}), "
"and the Keyboard setting opens it only without one")
else:
why = f"not opening Frametop's keyboard (Keyboard setting: {mode})"
# For bug reports (scripts/report.sh): once per decision, again after 30 s.
now = time.monotonic()
if why != state.get("text_field_said") or now - state.get("text_field_said_at", 0.0) > 30:
log(f"text field focused: {why}")
state["text_field_said"], state["text_field_said_at"] = why, now
def do_action(action, value, now, source="mouse"):
"""A mapped mouse or controller button, or key combination (pointer mode only, but
+35 -1
View File
@@ -93,9 +93,11 @@ class NoDevice:
relay.Virtual = NoDevice
relay.Volume.key = lambda self, fd, code, value, now: VOLUME.append((code, value))
relay.log = lambda *args: None # the relay's own log
LOG = [] # the relay's own log
relay.log = lambda *args: LOG.append(" ".join(map(str, args)))
bindings = {"now": None} # the rules' key_bindings; None: the relay's defaults
devices = {"roles": {}, "buttons": {}} # the rules' "devices" roles and per-device "buttons"
vr_keyboard = {"mode": None} # the rules' vr_keyboard (Frametop's keyboard); None: the default
MOUSE_ID = "usb:0003:0004:test mouse" # the fake mouse's id (Node.id)
@@ -103,6 +105,8 @@ def read_rules(path=None):
rules = {"devices": {i: {"role": r} for i, r in devices["roles"].items()},
"buttons": {i: dict(b) for i, b in devices["buttons"].items()}, "controller_buttons": {}}
rules["key_bindings"] = dict(relay.DEFAULT_KEY_BINDINGS if bindings["now"] is None else bindings["now"])
if vr_keyboard["mode"]:
rules["vr_keyboard"] = vr_keyboard["mode"]
return rules
@@ -413,6 +417,36 @@ def tests():
check("steam_menu, pause_toggle and commands work without pointer mode",
(relay.needs_pointer("steam_menu"), relay.needs_pointer("pause_toggle"), relay.needs_pointer("command:ls")),
(False, False, False))
# A text field on the desktop got focus (ft-textinput): Frametop's keyboard by the Keyboard
# setting, and a log line saying why, once per decision (scripts/report.sh reads them).
def text_field():
c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
c.sendto(b"textfield 1", f"\0{tag}_relay")
time.sleep(0.1)
return [m for m in typed() if m.startswith("vrkeyboard")]
def said():
got = [m for m in LOG if m.startswith("text field focused")]
LOG.clear()
return got
typed(), said()
check("text field, a keyboard connected (default setting): no keyboard", text_field(), [])
check("...and the log says why", said(), [("text field focused: not opening Frametop's keyboard: a keyboard is "
"connected (test keyboard), and the Keyboard setting opens it only "
"without one")])
check("the same again: not logged twice", (text_field(), said()), ([], []))
vr_keyboard["mode"] = "always"
use(None)
check("text field, setting always: it opens", text_field(), ["vrkeyboard show"])
check("...logged", said(), ["text field focused: asking ft-screens to open Frametop's keyboard"])
vr_keyboard["mode"] = "never"
use(None)
check("text field, setting never: no keyboard, logged",
(text_field(), said()), ([], ["text field focused: not opening Frametop's keyboard (Keyboard setting: never)"]))
vr_keyboard["mode"] = None
use(None)
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
shutil.rmtree(OUT, ignore_errors=True)
os._exit(1 if failures else 0)
+38 -13
View File
@@ -4,22 +4,29 @@
# eye tracker for it, and the Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
# for example after `git pull`. (Hand tracking, hands/, is deferred: it isn't offered here.)
# (It also works from a PC over SSH; see "Developing from a PC" in the README.)
# In a release (pack/install-release.sh), the programs come built from its image: this installs the
# distrobox the release brings, makes the release's container from its image, and builds nothing.
#
# Usage: ./install.sh [--yes] [--no-bluetooth]
# Usage: ./install.sh [--yes] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
# --yes don't ask; installs gaze mode, and our eye tracker if sudo can run without
# a password prompt; skips the Bluetooth fixes and the SteamVR restart
# --no-eye-tracker don't offer our own eye tracker
# --no-bluetooth don't offer the Bluetooth fixes
# --bluetooth install the Bluetooth fixes without asking (with --yes: if sudo can run
# without a password prompt)
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
. "$root/scripts/_env.sh"
assume_yes=0 bluetooth=1
assume_yes=0 bluetooth=1 eye_tracker=1
for arg in "$@"; do
case $arg in
--yes) assume_yes=1 ;;
--no-bluetooth) bluetooth=0 ;;
-h|--help) sed -n '2,12p' "$0"; exit 0 ;;
--bluetooth) bluetooth=2 ;;
--no-eye-tracker) eye_tracker=0 ;;
-h|--help) sed -n '2,17p' "$0"; exit 0 ;;
*) echo "unknown option: $arg" >&2; exit 2 ;;
esac
done
@@ -45,8 +52,12 @@ sudo_quiet() {
grep -q '^steamos_root_pwd=.' "$REPO_ROOT/.env" 2>/dev/null && return 0
on_frame 'sudo -n true' 2>/dev/null
}
# A release never builds (FRAME_RELEASE, scripts/_env.sh): its programs come in its image.
build() { [ "$FRAME_RELEASE" = 1 ] || "$@"; }
if [ "$FRAME_LOCAL" = 1 ]; then
if [ "$FRAME_RELEASE" = 1 ]; then
echo "Installing Frametop $(sed -n 's/^VERSION=//p' "$root/.frametop-release") on this Steam Frame from $FRAME_REPO"
elif [ "$FRAME_LOCAL" = 1 ]; then
echo "Installing on this Steam Frame from $FRAME_REPO"
else
echo "Installing on $FRAME_HOST over SSH (repo copy at $FRAME_REPO)"
@@ -60,6 +71,9 @@ fi
step "1/10 distrobox (container tool, installed in your home folder)"
if on_frame 'test -x ~/.local/bin/distrobox'; then
echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')"
elif [ "$FRAME_RELEASE" = 1 ]; then
# The release's image brings the distrobox it was tested with (pack/Containerfile).
on_frame 'cd pack/build/distrobox && ./install --prefix ~/.local'
else
on_frame 'set -e; mkdir -p ~/dev/src
# A tested release, so upstream changes cannot break new installs.
@@ -67,29 +81,36 @@ else
cd ~/dev/src/distrobox && ./install --prefix ~/.local'
fi
step "2/10 build container (Fedora 44 'dev', about 1-2 GB the first time)"
"$root/setup/dev-container.sh"
if [ "$FRAME_RELEASE" = 1 ]; then
step "2/10 Frametop's container (from this release's image)"
"$root/pack/release-box.sh"
else
step "2/10 build container (Fedora 44 'dev', about 1-2 GB the first time)"
"$root/setup/dev-container.sh"
fi
step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
"$root/desktops.sh" relay install
step "4/10 3D mouse: SteamVR driver"
"$root/pointer/driver/build.sh"
build "$root/pointer/driver/build.sh"
"$root/pointer/driver/install.sh" install
step "5/10 3D mouse: pointer helper service"
"$root/pointer/helper/build.sh"
build "$root/pointer/helper/build.sh"
"$root/pointer/helper/run.sh" install
step "6/10 power service (turns the displays off while the headset isn't used, even on a stand)"
"$root/power/build.sh"
build "$root/power/build.sh"
"$root/power/run.sh" install
step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
"$root/screens/build.sh"
step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, Display Settings, and Remote Displays"
build "$root/screens/build.sh"
build "$root/stream/build.sh" # ft-stream: remote displays (Frametop Remote Displays)
"$root/desktops.sh" install >/dev/null
"$root/input-settings/install.sh"
"$root/display-settings/install.sh"
"$root/remote-displays/install.sh"
"$root/remote/install.sh"
on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
@@ -109,9 +130,11 @@ fi
step "9/10 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
if [ "$gaze" = 0 ]; then
echo "skipped: gaze mode isn't installed. Install it later with: gaze/tracker/install.sh"
elif [ "$eye_tracker" = 0 ]; then
echo "skipped. Install it later with: gaze/tracker/install.sh"
elif [ "$assume_yes" = 1 ] && ! sudo_quiet; then
echo "skipped: it needs your password (sudo), and --yes doesn't ask. Install it later with: gaze/tracker/install.sh"
elif ask "Install our own eye tracker? Gaze mode then uses it instead of SteamVR's. Its frame grabber is a small system service, so it needs your password (sudo), and it downloads about 165 MB (numpy, OpenCV)." y; then
elif ask "Install our own eye tracker? Gaze mode then uses it instead of SteamVR's. Its frame grabber is a small system service, so it needs your password (sudo)$([ "$FRAME_RELEASE" = 1 ] || echo ", and it downloads about 165 MB (numpy, OpenCV)")." y; then
if "$root/gaze/tracker/install.sh" install; then
# Configs made before GAZE_TRACKER=auto say steam, which keeps SteamVR's.
tracker=$(on_frame "sed -n 's/^GAZE_TRACKER=\([a-z]*\).*/\1/p' ~/.config/frametop.conf | tail -1")
@@ -133,7 +156,9 @@ else
fi
step "10/10 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
if [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; then
if [ "$bluetooth" = 2 ] && [ "$assume_yes" = 1 ] && ! sudo_quiet; then
echo "skipped: they need your password (sudo), and --yes doesn't ask. Install later with: setup/bluetooth/install.sh install"
elif [ "$bluetooth" = 2 ] || { [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; }; then
"$root/setup/bluetooth/install.sh" install
else
echo "skipped. Install later with: setup/bluetooth/install.sh install"
+66
View File
@@ -0,0 +1,66 @@
# just — the runner for the actions that happen inside the image.
# The CI smoke job runs these with: docker run ... IMAGE just test
# Host-side image management (build, shell, update) lives in ./ft; these
# recipes only ever run where the toolchain and the venv are: in the image.
# default: list the recipes
default:
@just --list
# everything, strict (the recipes are the source of truth)
test: test-python test-c test-bash
# the Python suites as a strict gate: every failure fails the run. The image's
# python3 has to import the dnf Qt stack and the locked packages together;
# without that check the Qt tests would only skip. session/test is not here:
# test_accessibility.py needs the host's AT-SPI and PyGObject.
test-python:
#!/usr/bin/env bash
set -uo pipefail
python3 -c 'import PySide6, numpy, cv2' || { echo "python3 can't import PySide6, numpy, and cv2"; exit 1; }
status=0
for t in input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py gaze/test/*.py pack/test/*.py; do
[ -f "$t" ] || continue
if python3 "$t" >/dev/null; then echo "$t: ok"; else echo "$t: FAILED"; status=1; fi
done
pytest -q session/tests || status=1
exit $status
# the C/C++ unit tests that need no model runtime: header-only logic with
# made-up events. sides_test.cpp is NOT here: it pulls in ncnn (hands'
# deferred heavy build) and stays with `make check` in the hands build
# (hands/Makefile).
test-c:
#!/usr/bin/env bash
set -e
mkdir -p build/tests
gcc -std=c11 -Wall -Wextra -Werror -I. screens/tests/controller-click-test.c \
-lm -o build/tests/controller-click-test
build/tests/controller-click-test && echo "screens/tests/controller-click-test.c: ok"
gcc -std=c11 -Wall -Wextra -Werror screens/tests/relay-buttons-test.c \
-o build/tests/relay-buttons-test
build/tests/relay-buttons-test && echo "screens/tests/relay-buttons-test.c: ok"
# shell scripts: syntax-gate everything that ships. No git: the image has
# none, and a mounted checkout can trip "dubious ownership" — find lists
# what the repo actually ships. The count guard makes a silently short
# list fail the gate instead of passing it.
test-bash:
#!/usr/bin/env bash
set -e
mapfile -t scripts < <(find . -name '*.sh' -type f \
-not -path './.git/*' -not -path '*/build/*' | sort)
[ "${#scripts[@]}" -ge 20 ] || { echo "only ${#scripts[@]} scripts found — the list is broken"; exit 1; }
status=0
for f in "${scripts[@]}" ft; do
bash -n "$f" && echo "bash -n $f: ok" || status=1
done
exit $status
# lint the Python side (ruff from the locked dev group). Report only for now:
# the codebase has pre-existing findings; making it a gate is its own cleanup.
lint:
-ruff check .
# the whole gate: lint, then tests
check: lint test
+1 -1
View File
@@ -1,4 +1,4 @@
#!/bin/bash
# Reset Screen Layout's own program. Steam lists menu entries by program, so entries that
# shared ft-layout (with different arguments) launched as one.
exec "$(dirname "$(readlink -f "$0")")/ft-layout" apply "$@"
exec "$(dirname "$(readlink -f "$0")")/ft-layout" reset "$@"
+521 -7
View File
@@ -36,7 +36,24 @@ you face (yaw only), like a recenter. It lives in ~/.config/frametop-layout.json
visibility mode (ft-layout hide N)
"pos": [x, y, z], "face": [yaw, pitch], "roll": 0, custom layout
"rotation": "normal" | "left" | "right"}, ...], gamescope only
"panel_size": [w, h]} gamescope: last measured panel size
"panel_size": [w, h], gamescope: last measured panel size
"hosts": [{"name": "desk-pc", "address": "192.168.1.20", ft-screens: other machines' displays
"direct": ["10.35.78.20"], its Steam Link dongle on the Frame's hotspot
"route": "auto", auto (the dongle when it answers) | network |
dongle (only; ft-stream reads both)
"displays": [{"id": "desk-pc-oled", (docs/remote-displays.md), each a panel
"client": "frametop-1", streamed by ft-stream: its paired client,
"app": "display:{GUID}", what it streams (display:DEVICE, monitor:
a Vibepollo Remote Monitor, primary),
"label": "OLED", "size": [5120, 1440], "fps": 60, "bitrate": 50000,
"metres": 1.8, "curve": 0, "pin": ..., "hidden": true,
"off": true, disconnected: no stream, no panel
"pos": ..., "face": ..., "roll": 0}]}]} as a screen's
Remote displays are screens 101 and up, in the order they're listed. Without a place of their
own they go in a row above the screens. A profile keeps the ones connected when it was saved,
like its apps: their places and hidden state by id (profiles[NAME]["remote"]). Opening it, or
arranging while it's the one in use, connects each whose host answers and puts it there; the
others stay as they are.
Custom positions: x right, y up, -z forward from the head, in metres; face = the
direction you look to see the screen's front straight on, in degrees, relative to your
heading; roll = the panel turned about its front, counterclockwise as you see it.
@@ -47,6 +64,9 @@ is top left, then left to right.
Usage (on the Frame host; Frametop Display Settings calls it too):
ft-layout apply [--wait SECONDS] arrange every screen; --wait is for desktop start:
wait for the screens, skip if "auto" is off
ft-layout reset a quick reset (Meta+Shift+R, a screen's reset button, a mapped
button): open the profile in use again, as use NAME does, or
else apply
ft-layout capture save the current arrangement as the custom layout
ft-layout save NAME save it as a named layout too, and use that; with the
desktop's apps and hidden screens, as a profile
@@ -70,7 +90,19 @@ Usage (on the Frame host; Frametop Display Settings calls it too):
ft-layout hidden the screens hidden on their own
ft-layout pin all|N left|right|head pin screens to a wrist or your head as they are;
unpin all|N
ft-layout remote list the remote displays, their numbers and their streams' state
ft-layout remote monitors ADDRESS the host's displays (Vibepollo's /api/display-devices, JSON)
ft-layout remote add HOST ADDRESS APP [--size WxH] [--fps F] [--bitrate KBPS] [--label TEXT]
a display of a host (pairs its client with the host's token
first) -> its id
ft-layout remote set ID size=WxH|fps=F|bitrate=KBPS|label=TEXT|app=APP|metres=M ...
ft-layout remote host NAME route=auto|network|dongle direct=ADDRESS[,ADDRESS]|none
how a host's streams reach it; its running streams start over
ft-layout remote connect ID... start their streams again (and keep them on)
ft-layout remote disconnect ID... stop their streams and panels until connected again
ft-layout remote remove ID stops it and unpairs its client (the host stays)
"""
import concurrent.futures
import fcntl
import json
import math
@@ -100,7 +132,12 @@ VISIBILITY = {"mode": "always", "wrist_angle": 60, "gesture_hand": "left", "gest
SPATIAL = ("pos", "face", "roll", "metres", "curve", "pin") # what a named layout keeps of a screen
DEFAULTS = {"auto": True, "mode": "preset",
"preset": {"kind": "arc", "rows": 1, "distance": 2.0, "gap": 0.05, "height": 0.0},
"screens": [], "panel_size": list(DEFAULT_PANEL)}
"screens": [], "panel_size": list(DEFAULT_PANEL), "hosts": []}
REMOTE_FIRST = 101 # ft-screens' remote screens (screens/remote.h)
REMOTE_MAX = 16
REMOTE_PIXELS_PER_METRE = 2400 # a new remote display's width in VR (5120 px: 2.1 m), at least 1 m
WEB_UI_PORT = 47990 # Vibepollo's Web UI: a host answers there when it's up
STREAM = os.path.join(REPO, "stream", "build", "ft-stream")
def log(*args):
@@ -432,9 +469,20 @@ def set_hidden(which, hidden):
desktop runs."""
layout = load_layout()
n = screen_count(layout)
picked = range(n) if which == "all" else [int(which) - 1] if which.isdigit() else []
if not picked or not all(0 <= i < n for i in picked):
raise RuntimeError(f"no screen {which} (1 to {n})")
remotes = {num: d for num, _, d in remote_displays(layout)}
if which == "all":
picked, picked_remote = range(n), list(remotes)
elif which.isdigit() and int(which) in remotes:
picked, picked_remote = [], [int(which)]
else:
picked, picked_remote = ([int(which) - 1] if which.isdigit() else []), []
if not (picked or picked_remote) or not all(0 <= i < n for i in picked):
raise RuntimeError(f"no screen {which} (1 to {n}, or a remote display's number)")
for num in picked_remote:
if hidden:
remotes[num]["hidden"] = True
else:
remotes[num].pop("hidden", None)
screens = layout.setdefault("screens", [])
while len(screens) < n:
screens.append({})
@@ -446,8 +494,8 @@ def set_hidden(which, hidden):
save_layout(layout)
try:
sock = screens_socket()
for i in picked:
sock.ask(f"{'conceal' if hidden else 'reveal'} {i + 1}")
for num in [i + 1 for i in picked] + picked_remote:
sock.ask(f"{'conceal' if hidden else 'reveal'} {num}")
except RuntimeError as e:
log(f"saved; not applied now: {e}")
@@ -482,6 +530,8 @@ def apply_screens(wait=0):
if time.time() >= deadline:
raise
time.sleep(1)
start_remotes(sock, layout)
save_layout(layout) # screen numbers given to new remote displays
f = sock.ask("head").split()
eye, heading = tuple(map(float, f[1:4])), float(f[4])
send_visibility(sock, layout)
@@ -501,6 +551,7 @@ def apply_screens(wait=0):
except RuntimeError as e:
log(f"screen {i + 1}: {e}") # that controller isn't on
send_hidden(sock, layout)
place_remotes(sock, layout, count, eye, heading)
try:
sock.ask("vrkeyboard close") # the keyboard, if open, goes too: a reset starts over
except RuntimeError:
@@ -527,6 +578,7 @@ def capture_screens():
screens.append(entry)
layout["screens"] = screens + layout.get("screens", [])[len(screens):]
layout["mode"] = "custom"
capture_remotes(sock, layout, eye, heading)
save_layout(layout)
return screens
@@ -645,6 +697,430 @@ def capture_gamescope():
return screens
# ---------------------------------------------------------------- remote displays (docs/remote-displays.md)
def remote_displays(layout):
"""[(number, host, display)]: each display's screen number (its "screen", kept so it
doesn't move when another is removed; or the first free one)."""
entries = [(h, d) for h in layout.get("hosts", []) for d in h.get("displays", [])]
used, out = set(), []
for h, d in entries:
n = d.get("screen")
if isinstance(n, int) and REMOTE_FIRST <= n < REMOTE_FIRST + REMOTE_MAX and n not in used:
used.add(n)
d["_kept"] = True
for h, d in entries:
if not d.pop("_kept", False):
free = [k for k in range(REMOTE_FIRST, REMOTE_FIRST + REMOTE_MAX) if k not in used]
if not free:
continue
d["screen"] = free[0]
used.add(free[0])
out.append((d["screen"], h, d))
return out
def find_display(layout, display_id):
for n, host, d in remote_displays(layout):
if d.get("id") == display_id:
return n, host, d
raise RuntimeError(f"no remote display {display_id!r}")
def remote_size(d):
"""A remote display's size in VR (width, height) in metres."""
w, h = d.get("size", [2560, 1440])
m = float(d.get("metres", max(1.0, w / REMOTE_PIXELS_PER_METRE)))
return m, m * h / w
def plan_remotes(layout, count, displays):
"""The remote displays' poses in the head frame (as plan()): their own place, or else a
row above the screens, hinged like the arc preset."""
out = [None] * len(displays)
for k, (_, _, d) in enumerate(displays):
if "pos" in d:
out[k] = {"pos": tuple(d["pos"]), "face": tuple(d.get("face", yaw_pitch(d["pos"]))),
"roll": float(d.get("roll", 0))}
free = [k for k in range(len(displays)) if out[k] is None and not displays[k][2].get("off")]
if not free:
return out
p = layout["preset"]
dist = max(0.3, float(p.get("distance", 2.0)))
gap = max(0.0, float(p.get("gap", 0.05)))
span = lambda m, at: 2 * math.degrees(math.atan(m / 2 / at))
top = 0.0
for i, t in enumerate(plan(layout, count)):
x, y, z = t["pos"]
at = max(0.3, math.sqrt(x * x + y * y + z * z))
top = max(top, math.degrees(math.atan2(y, math.hypot(x, z))) + span(screen_size(layout, i)[1], at) / 2)
sizes = [remote_size(displays[k][2]) for k in free]
pitch = top + span(gap, dist) + max(span(h, dist) for _, h in sizes) / 2
cp, sp = math.cos(math.radians(pitch)), math.sin(math.radians(pitch))
for k, (x, z, yaw) in zip(free, _chain([w for w, _ in sizes], dist, gap)):
out[k] = {"pos": (x * cp, math.hypot(x, z) * sp, z * cp), "face": (yaw, pitch), "roll": 0.0}
return out
def remotes_running(sock):
"""ft-screens' remote screens, {number: state}; None from an ft-screens without them."""
try:
fields = sock.ask("remotes").split()[2:]
except RuntimeError:
return None
return {int(e.split(":")[0]): e.split(":")[2] for e in fields}
def start_remote(sock, n, host, d):
"""Its stream (ft-screens starts it over only if the settings changed). ft-screens'
reply: "ok restored" when its panel is back where it was before a disconnect."""
w, h = d.get("size", [2560, 1440])
label = f"{host.get('name') or host['address']}: {d.get('label') or d['id']}"
return sock.ask(f"remote {n} start {d['client']} {host['address']} {d['app']} {int(w)}x{int(h)} "
f"{int(d.get('fps', 60))} {int(d.get('bitrate', 0))} {remote_size(d)[0]:.3f} {label}")
def place_remote(sock, n, d, t, eye, heading):
center = tuple(e + v for e, v in zip(eye, turn_yaw(t["pos"], heading)))
sock.ask(f"width {n} {remote_size(d)[0]:.4f}")
sock.ask(f"curve {n} {float(d.get('curve', 0)):.3f}")
sock.ask("place %d %.4f %.4f %.4f %.3f %.3f %.3f" % (n, *center, t["face"][0] + heading, t["face"][1], t["roll"]))
pin = d.get("pin")
if pin and len(pin.get("rel", [])) == 12:
try:
sock.ask(f"pin {n} {pin['hand']} " + " ".join(f"{v:.5f}" for v in pin["rel"]))
except RuntimeError as e:
log(f"remote display {d['id']}: {e}") # that controller isn't on
sock.ask(f"{'conceal' if d.get('hidden') else 'reveal'} {n}")
def host_answers(host, timeout=1.5):
"""Whether a host's Vibepollo answers, at its address or its Steam Link dongle's (as its
route allows)."""
route = host.get("route", "auto")
addresses = ([] if route == "dongle" else [host.get("address")]) + \
([] if route == "network" else list(host.get("direct", [])))
for address in filter(None, addresses):
try:
with socket.create_connection((address, WEB_UI_PORT), timeout=timeout):
return True
except OSError:
pass
return False
def profile_remotes(layout, name):
"""Remote displays the profile has a place for go there; the others stay where they are."""
remote = layout.get("profiles", {}).get(name, {}).get("remote", {})
for _, _, d in remote_displays(layout):
if d.get("id") in remote:
for k in SPATIAL:
d.pop(k, None)
d.update(json.loads(json.dumps({k: v for k, v in remote[d["id"]].items() if k in SPATIAL})))
def open_remotes(layout, name):
"""Connect the remote displays profile `name` was saved with (in the layout: apply starts
their streams), each whose host answers now. A host that doesn't is skipped, and its
displays stay disconnected."""
remote = layout.get("profiles", {}).get(name, {}).get("remote", {})
wanted = [(h, d) for _, h, d in remote_displays(layout) if d.get("id") in remote and d.get("off")]
hosts = list({id(h): h for h, _ in wanted}.values())
if not hosts:
return
with concurrent.futures.ThreadPoolExecutor(len(hosts)) as pool:
up = dict(zip(map(id, hosts), pool.map(host_answers, hosts)))
for h, d in wanted:
if up[id(h)]:
d.pop("off", None)
log(f"remote display {d['id']}: connecting (profile {name!r})")
else:
log(f"remote display {d['id']}: {h.get('name') or h['address']} isn't answering; skipped")
def start_remotes(sock, layout, only=None):
"""Start the remote displays' streams (or just the ids in `only`; ft-screens leaves one
running with the same settings alone), and stop the streams of ones no longer listed or
disconnected ("off"). They start without a head pose too: placing them waits for one
(place_remotes)."""
displays = remote_displays(layout)
running = remotes_running(sock)
if running is None:
if displays:
log("remote displays: this ft-screens can't show them (build it again)")
return False
if only is None:
for n in set(running) - {n for n, _, d in displays if not d.get("off")}:
sock.ask(f"remote {n} stop")
for n, host, d in displays:
if d.get("off"):
continue
if only is None or d.get("id") in only:
try:
start_remote(sock, n, host, d)
except RuntimeError as e:
log(f"remote display {d.get('id')}: {e}")
return True
def place_remotes(sock, layout, count, eye, heading, only=None):
displays = remote_displays(layout)
for (n, host, d), t in zip(displays, plan_remotes(layout, count, displays)):
if d.get("off"):
continue # disconnected: no panel
if only is None or d.get("id") in only:
try:
place_remote(sock, n, d, t, eye, heading)
except RuntimeError as e:
log(f"remote display {d.get('id')}: {e}")
on = [d for _, _, d in displays if not d.get("off") and (only is None or d.get("id") in only)]
if on:
log(f"arranged {len(on)} remote display(s)")
def capture_remotes(sock, layout, eye, heading):
"""Where the remote displays are now (each one running) into their entries."""
for n, _, d in remote_displays(layout):
try:
g = parse_get(sock.ask(f"get {n}"))
except RuntimeError:
continue # not running
d.update(relative_pose(g["center"], g["x"], g["z"], eye, heading))
d["metres"] = round(g["metres"], 4)
d["curve"] = round(g["curve"], 3)
d.pop("pin", None)
if "rel" in g:
d["pin"] = {"hand": g["hand"], "rel": g["rel"]}
def in_container():
return os.path.exists("/run/.containerenv")
def run_stream(*args, timeout=60):
"""ft-stream, in the container ft-screens runs in: "dev" for a clone, the release's own for a
release (scripts/in-box). A missing "dev" made distrobox ask whether to create it, and the
question hung until the timeout."""
cmd = [STREAM, *args]
if not in_container():
cmd = [os.path.join(REPO, "scripts", "in-box"), *cmd]
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
return r.returncode, r.stdout, r.stderr
def slug(text):
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-") or "display"
def remote_anchor(sock, layout):
"""(eye, heading) to place remote displays from: the head, or with no head pose (the
headset off), where the last arrangement was made from, worked out from where screen 1
is and where the layout puts it."""
try:
f = sock.ask("head").split()
return tuple(map(float, f[1:4])), float(f[4])
except RuntimeError:
g = parse_get(sock.ask("get 1"))
if g["hand"] != "none":
raise RuntimeError("no head pose, and screen 1 is pinned")
t = plan(layout, screen_count(layout))[0]
heading = yaw_pitch(tuple(-c for c in g["z"]))[0] - t["face"][0]
return tuple(c - v for c, v in zip(g["center"], turn_yaw(t["pos"], heading))), heading
def start_remotes_now(layout, only):
"""Start and place remote displays now, if the desktop runs."""
try:
sock = screens_socket()
if not start_remotes(sock, layout, only):
return
eye, heading = remote_anchor(sock, layout)
place_remotes(sock, layout, screen_count(layout), eye, heading, only=only)
except RuntimeError as e:
log(f"not placed now: {e}")
def remote_command(args):
"""ft-layout remote list|monitors|add|set|connect|disconnect|remove (see the usage)."""
what = args[0] if args else ""
if what == "list" and len(args) == 1:
layout = load_layout()
try:
running = remotes_running(screens_socket()) or {}
except RuntimeError:
running = {}
for n, host, d in remote_displays(layout):
w, h = d.get("size", [2560, 1440])
state = "disconnected" if d.get("off") else running.get(n, "off")
state += f" route {host.get('route', 'auto')}"
print(f"{n} {d['id']} {host.get('name')} {host['address']} {d['app']} {w}x{h}@{d.get('fps', 60)} "
f"{state}{' hidden' if d.get('hidden') else ''} {d.get('label', '')}")
return
if what == "host" and len(args) >= 3:
layout = load_layout()
host = next((h for h in layout.get("hosts", []) if h.get("name") == args[1]), None)
if host is None:
raise RuntimeError(f"no host called {args[1]!r}")
for kv in args[2:]:
k, _, v = kv.partition("=")
if k == "route" and v in ("auto", "network", "dongle"):
host["route"] = v
elif k == "direct" and (v == "none" or re.fullmatch(r"[A-Za-z0-9.:-]+(,[A-Za-z0-9.:-]+)*", v)):
host["direct"] = [] if v == "none" else v.split(",")
else:
raise RuntimeError(f"remote host: {kv!r}? (route=auto|network|dongle direct=ADDRESS[,ADDRESS]|none)")
save_layout(layout)
# Its streams start over the new way, their panels where they are (ft-screens keeps
# their places across a stop in the same run).
try:
sock = screens_socket()
running = remotes_running(sock) or {}
for n, h, d in remote_displays(layout):
if h is host and n in running and not d.get("off"):
sock.ask(f"remote {n} stop")
start_remote(sock, n, h, d)
log(f"{d['id']}: starting over")
except RuntimeError as e:
log(f"saved; not applied now: {e}")
return
if what in ("connect", "disconnect") and len(args) >= 2:
layout = load_layout()
found = [find_display(layout, i) for i in args[1:]]
for _, _, d in found:
if what == "connect":
d.pop("off", None)
else:
d["off"] = True
save_layout(layout)
if what == "connect":
# Back where it was if ft-screens still knows (this run); else placed like the
# others, from where you look now.
try:
sock = screens_socket()
unplaced = {d["id"] for n, host, d in found if start_remote(sock, n, host, d) != "ok restored"}
except RuntimeError as e:
log(f"saved; not started now: {e}")
return
for _, _, d in found:
log(f"connected {d['id']}" + ("" if d["id"] in unplaced else " (where it was)"))
if unplaced:
try:
eye, heading = remote_anchor(sock, layout)
place_remotes(sock, layout, screen_count(layout), eye, heading, only=unplaced)
except RuntimeError as e:
log(f"not placed now: {e}")
return
try:
sock = screens_socket()
for n, _, d in found:
sock.ask(f"remote {n} stop")
log(f"disconnected {d['id']}")
except RuntimeError as e:
log(f"saved; not stopped now: {e}")
return
if what == "monitors" and len(args) == 2:
code, out, err = run_stream("monitors", args[1])
if code:
raise RuntimeError(err.strip() or out.strip() or "ft-stream monitors failed")
print(out.strip())
return
if what == "add" and len(args) >= 4:
host_name, address, app = args[1], args[2], args[3]
opts = dict(zip(args[4::2], args[5::2]))
unknown = set(opts) - {"--size", "--fps", "--bitrate", "--label", "--client"}
if unknown or len(args[4:]) % 2:
raise RuntimeError("remote add: options are --size WxH --fps F --bitrate KBPS --label TEXT --client NAME")
if not re.fullmatch(r"[A-Za-z0-9.:-]+", address) or not re.fullmatch(r"[A-Za-z0-9{}._:-]+", app):
raise RuntimeError("remote add: the address or app has other characters")
layout = load_layout()
displays = remote_displays(layout)
if len(displays) >= REMOTE_MAX:
raise RuntimeError(f"at most {REMOTE_MAX} remote displays")
label = opts.get("--label") or ("Virtual" if app == "monitor" else app.split(":", 1)[-1])
ids = {d["id"] for _, _, d in displays}
display_id = base = slug(f"{host_name}-{label}")
for k in range(2, 100):
if display_id not in ids:
break
display_id = f"{base}-{k}"
client = opts.get("--client") or f"frametop-{display_id}"
if not re.fullmatch(r"[A-Za-z0-9._-]+", client):
raise RuntimeError("remote add: the client name has other characters")
size = [2560, 1440]
if "--size" in opts:
size = [int(v) for v in opts["--size"].lower().split("x")]
elif app.startswith("display:"): # the monitor's own resolution
code, out, _ = run_stream("monitors", address)
try:
for m in (json.loads(out[out.index("["):]) if code == 0 else []):
if m.get("device_id") == app[8:]:
res = m["info"]["resolution"]
size = [int(res["width"]), int(res["height"])]
except (ValueError, KeyError, TypeError):
pass
code, out, err = run_stream("pair", address, "--id", client, timeout=30)
if code:
raise RuntimeError(f"pairing {client} with {address} failed: {(err or out).strip()}")
log(out.strip().splitlines()[-1] if out.strip() else f"paired {client}")
host = next((h for h in layout.setdefault("hosts", []) if h.get("name") == host_name), None)
if host is None:
host = {"name": host_name, "address": address, "displays": []}
layout["hosts"].append(host)
host["address"] = address
host["displays"].append({"id": display_id, "client": client, "app": app, "label": label, "size": size,
"fps": int(opts.get("--fps", 60)), "bitrate": int(opts.get("--bitrate", 0)),
"metres": round(max(1.0, size[0] / REMOTE_PIXELS_PER_METRE), 3)})
remote_displays(layout) # gives it a screen number
save_layout(layout)
print(display_id)
start_remotes_now(layout, {display_id})
return
if what == "set" and len(args) >= 3:
layout = load_layout()
n, host, d = find_display(layout, args[1])
for kv in args[2:]:
k, _, v = kv.partition("=")
if k == "size" and re.fullmatch(r"\d+x\d+", v.lower()):
d["size"] = [int(x) for x in v.lower().split("x")]
elif k in ("fps", "bitrate") and v.isdigit():
d[k] = int(v)
elif k == "label" and v:
d["label"] = v
elif k == "metres" and re.fullmatch(r"\d+(\.\d+)?", v) and 0.15 <= float(v) <= 12:
d["metres"] = float(v)
elif k == "app" and re.fullmatch(r"[A-Za-z0-9{}._:-]+", v):
d["app"] = v
else:
raise RuntimeError(f"remote set: {kv!r}? (size=WxH fps=F bitrate=KBPS label=TEXT app=APP metres=M)")
save_layout(layout)
if d.get("off"):
return # disconnected: the new settings are for its next connection
try:
start_remote(screens_socket(), n, host, d) # starts over with the new settings
except RuntimeError as e:
log(f"saved; not applied now: {e}")
return
if what == "remove" and len(args) == 2:
layout = load_layout()
n, host, d = find_display(layout, args[1])
try:
screens_socket().ask(f"remote {n} stop")
time.sleep(2) # its stream releases a Remote Monitor on the way out
except RuntimeError:
pass
host["displays"].remove(d) # the host stays (Remote Displays removes hosts)
for profile in layout.get("profiles", {}).values():
profile.get("remote", {}).pop(d["id"], None)
save_layout(layout)
code, out, err = run_stream("unpair", host["address"], "--id", d["client"], timeout=30)
said = (out or err).strip().splitlines()
log(f"removed {d['id']}; {said[-1] if said else f'unpair exited {code}'}")
return
raise RuntimeError("remote list | monitors ADDRESS | add HOST ADDRESS APP [options] | set ID KEY=VALUE... | "
"host NAME route=...|direct=... | connect ID... | disconnect ID... | remove ID")
def apply(wait=0):
return apply_screens(wait) if backend() == "screens" else apply_gamescope(wait)
@@ -680,6 +1156,14 @@ def save_named(layout, name):
if not screens:
raise RuntimeError("nothing to save: no arrangement captured")
layout.setdefault("layouts", {})[name] = [{k: s[k] for k in SPATIAL if k in s} for s in screens]
# The remote displays connected now, like the apps open now.
remote = {d["id"]: {k: d[k] for k in SPATIAL if k in d} for _, _, d in remote_displays(layout)
if not d.get("off") and "pos" in d}
profile = layout.setdefault("profiles", {}).setdefault(name, {})
if remote:
profile["remote"] = remote
else:
profile.pop("remote", None)
layout["mode"], layout["active"] = "custom", name
return name
@@ -704,6 +1188,7 @@ def use_named(layout, name):
elif "pos" not in screens[i]:
screens[i].update({"pos": list(preset[i]["pos"]), "face": list(preset[i]["face"]),
"roll": preset[i]["roll"]})
profile_remotes(layout, name)
layout["mode"], layout["active"] = "custom", name
@@ -758,6 +1243,10 @@ def capture_profile(layout, name):
hidden = [i + 1 for i in range(screen_count(layout)) if screen_entry(layout, i).get("hidden")]
profile = layout.setdefault("profiles", {}).setdefault(name, {})
profile["hidden"] = hidden
remote = profile.get("remote", {}) # the ones connected (save_named)
for _, _, d in remote_displays(layout):
if d["id"] in remote:
remote[d["id"]]["hidden"] = bool(d.get("hidden"))
reply = ask_float("windows")
if reply and reply.startswith("ok "):
profile["windows"] = json.loads(reply[3:])
@@ -780,6 +1269,13 @@ def use_hidden(layout, name):
screens[i]["hidden"] = True
else:
screens[i].pop("hidden", None)
remote = profile.get("remote", {})
for _, _, d in remote_displays(layout):
if "hidden" in remote.get(d.get("id"), {}):
if remote[d["id"]]["hidden"]:
d["hidden"] = True
else:
d.pop("hidden", None)
def open_apps(name, wait=0):
@@ -1086,6 +1582,7 @@ def main(argv):
return main([argv[0], "apply"] + argv[2:])
use_named(layout, name)
use_hidden(layout, name)
open_remotes(layout, name)
save_layout(layout)
log(f"starting in profile {name!r}")
wait = float(argv[argv.index("--wait") + 1]) if "--wait" in argv else 60
@@ -1102,6 +1599,12 @@ def main(argv):
else:
log(f"kwin: {last}")
open_apps(name, wait=90) # ft-floatd starts with Plasma
elif cmd == "reset":
layout = load_layout()
name = layout.get("active") if layout.get("mode") == "custom" else None
return main([argv[0], "use", name] if name in layout.get("layouts", {}) else [argv[0], "apply"])
elif cmd == "remote":
remote_command(argv[2:])
elif cmd in ("pin", "unpin") and len(argv) >= 3:
log(screens_socket().ask(" ".join(argv[1:])))
kwin_follow() # pinned screens go last
@@ -1124,9 +1627,19 @@ def main(argv):
time.sleep(1)
send_visibility(sock, load_layout())
send_hidden(sock, load_layout())
layout = load_layout()
if start_remotes(sock, layout): # streams start, arranged or not
save_layout(layout)
except RuntimeError as e:
log(f"visibility: {e}")
else:
layout = load_layout()
name = layout.get("active") if layout.get("mode") == "custom" else None
if name in layout.get("profiles", {}):
# Arranged as the profile in use: its remote displays too.
profile_remotes(layout, name)
open_remotes(layout, name)
save_layout(layout)
apply(wait)
if not wait:
kwin_follow()
@@ -1159,6 +1672,7 @@ def main(argv):
layout = load_layout()
use_named(layout, argv[2])
use_hidden(layout, argv[2])
open_remotes(layout, argv[2])
save_layout(layout)
log(f"using layout {argv[2]!r}")
try:
+109
View File
@@ -0,0 +1,109 @@
# Frametop runtime image (see pack/README.md).
#
# One image that holds everything Frametop needs: the frozen toolchain and C
# libraries, its native binaries, and its locked Python environment. A given
# image tag always behaves the same, on a PC and on the Frame.
#
# setup/dev-container.sh stays the source of truth for the package list on the
# Frame's dev container; the list below freezes the runtime-relevant subset of
# it plus the toolchain (the binaries are built in this image). Keep the two in
# sync on purpose: dev-container.sh documents *why* each package is there.
FROM registry.fedoraproject.org/fedora-toolbox:44@sha256:034cb7c472038e2d879ddc19568106a1342aa24403f0d641a0f73dda638707ad
# Toolchain (the binaries are built in here) and the libraries they link.
RUN dnf -y install --setopt=install_weak_deps=False \
gcc gcc-c++ make pkgconf-pkg-config curl unzip just git cmake ninja-build \
pipewire-devel libxkbcommon-devel libinput-devel systemd-devel dbus-devel \
libdrm-devel mesa-libgbm-devel wayland-devel vulkan-loader-devel \
vulkan-headers plasma-wayland-protocols wlroots-devel pixman-devel \
libstdc++-static glibc-static jsoncpp-devel zstd \
# remote displays (stream/): moonlight-embedded's libgamestream and
# moonlight-common-c, and the host's sound (Opus, PulseAudio)
openssl-devel libcurl-devel expat-devel libuuid-devel json-devel \
opus-devel pulseaudio-libs-devel \
# Frametop Input Settings (Kirigami, PySide6) and remote desktop (krdp):
# the KDE/Qt stack has no PyPI wheels, so it stays a dnf package
python3-pyside6 kf6-kirigami kf6-qqc2-desktop-style qt6-qtwayland \
breeze-icon-theme plasma-breeze krdp freerdp tigervnc-x11-server xrandr \
&& dnf -y remove 'glibc-langpack-*' \
&& dnf -y install glibc-langpack-en \
&& dnf clean all \
&& rm -rf /usr/share/doc/* /usr/share/man/* /usr/share/info/* \
/usr/share/wallpapers
# uv, pinned and checked. Python dependencies come from the committed uv.lock,
# not from whatever PyPI serves on build day.
RUN curl -fsSL https://github.com/astral-sh/uv/releases/download/0.12.23/uv-aarch64-unknown-linux-gnu.tar.gz \
-o /tmp/uv.tar.gz \
&& echo "6524bd338177ed50d035d39354e12545e993bbeba2ecbddf0480c5b3a81d313f /tmp/uv.tar.gz" | sha256sum -c \
&& tar -xzf /tmp/uv.tar.gz --strip-components=1 -C /usr/local/bin \
&& rm /tmp/uv.tar.gz
# OpenVR's API library for linking, from the SDK tag the build scripts pin
# (scripts/openvr.sh fetches the headers). The image has no SteamVR, so the
# binaries link this copy; their rpath names SteamVR's folder first, so on the
# Frame they load SteamVR's own libopenvr_api, as the on-device builds do.
RUN install -d /opt/frametop/bin /opt/frametop/lib \
&& curl -fsSL https://github.com/ValveSoftware/openvr/raw/v2.15.6/bin/linuxarm64/libopenvr_api.so \
-o /opt/frametop/lib/libopenvr_api.so \
&& echo "cc3671d24dd23fb61494e8a8327cf4fe7f638a60508ff7225984a03f2f5947e9 /opt/frametop/lib/libopenvr_api.so" | sha256sum -c
# Python: Fedora's own python3 (the digest-pinned base freezes it), with the
# locked packages in a venv on top. The venv sees the system site-packages,
# because PySide6 and Kirigami come from dnf for this python3: one interpreter
# imports both, as in the dev container on the Frame. Rebuilt only when the
# lockfile changes, so source edits below reuse this layer.
# uv sync --frozen --group tracker: the default dev group (pytest/ruff for CI's
# in-image tests) plus the tracker stack; the types group (mypy) stays out.
# --no-cache keeps uv's download cache out of the image.
ENV UV_PROJECT_ENVIRONMENT=/opt/frametop/venv UV_PYTHON=/usr/bin/python3 UV_PYTHON_DOWNLOADS=never
COPY pyproject.toml uv.lock .python-version /src/frametop/
RUN uv venv --system-site-packages /opt/frametop/venv \
&& cd /src/frametop && uv sync --frozen --group tracker --no-cache \
&& /opt/frametop/venv/bin/python -c 'import PySide6, numpy, cv2'
# The sources, then the binaries, built by the same build scripts as on the
# Frame (FRAME_IN_BOX=1: they run here instead of entering the dev container).
# Each program stays in its build/ folder in /src/frametop and is copied to
# /opt/frametop. Hand tracking is deferred exactly as in install.sh: it builds
# ncnn from source and isn't needed until hands/run.sh install runs.
COPY . /src/frametop
WORKDIR /src/frametop
ENV FRAME_IN_BOX=1 OPENVR_LIB=/opt/frametop/lib
RUN screens/build.sh \
&& install -t /opt/frametop/bin screens/build/ft-screens screens/build/ft-handtest
RUN pointer/helper/build.sh && pointer/driver/build.sh \
&& install -t /opt/frametop/bin pointer/helper/build/ft-pointer \
&& install -t /opt/frametop/lib pointer/driver/build/driver_ft_pointer.so
RUN gaze/build.sh \
&& install -t /opt/frametop/bin gaze/build/ft-gaze gaze/build/ft-gazepanel
RUN power/build.sh \
&& install -t /opt/frametop/bin power/build/ft-powerd
RUN stream/build.sh \
&& install -t /opt/frametop/bin stream/build/ft-stream
# Our own eye tracker: ft-eyegrab (it runs on the host, as root) and ft-eyes'
# build/venv, which here only points at the venv above.
RUN gaze/tracker/build.sh \
&& install -t /opt/frametop/bin gaze/tracker/build/ft-eyegrab
# A release installs from this image (pack/install-release.sh): it copies
# /src/frametop, built, to the Frame for its host side (services, the driver,
# the desktop's scripts), and its install.sh installs this distrobox (1.8.2.5,
# as a source install clones it, here by commit) to run the image as the
# container those programs start in (scripts/in-box).
RUN git init -q pack/build/distrobox \
&& cd pack/build/distrobox \
&& git fetch -q --depth 1 https://github.com/89luca89/distrobox.git 40c3cd724faa434aeb0a23e28776665b92de68bd \
&& git checkout -q FETCH_HEAD \
&& rm -rf .git
# Programs built against OpenVR look for SteamVR's library here first. In the
# container on the Frame, distrobox mounts the host at /run/host; elsewhere the
# link leads nowhere and they use /opt/frametop/lib's.
RUN ln -s /run/host/opt/steamvr /opt/steamvr
# The default environment: Frametop's binaries and the venv's python3 (Fedora's
# interpreter, with the locked packages and the dnf Qt stack) first.
# sleep infinity as the default command keeps the image usable with distrobox
# create, like the toolbox base.
ENV PATH=/opt/frametop/venv/bin:/opt/frametop/bin:/usr/local/bin:/usr/bin:/usr/sbin
CMD ["sleep", "infinity"]
+218
View File
@@ -0,0 +1,218 @@
# Frametop as an image
The Frame's runtime is already a container: the dev container (Fedora toolbox
in distrobox) is where everything builds and runs, and `setup/dev-container.sh`
is its recipe. The recipe is the weak point — nothing in it is pinned. Rebuild
the container tomorrow and you get a different Fedora, a different wlroots, a
different glibc. This directory turns the container from a recipe into an
**artifact**: an OCI image whose every input is frozen.
## What the image contains
`Containerfile` builds one image, `frametop`, from three frozen inputs:
1. **The base image by digest** — `fedora-toolbox:44` pinned to the exact
image, not the floating tag. This freezes the C world: glibc, wlroots 0.20,
the Qt/KDE stack.
2. **Python from `uv.lock`** — the committed lockfile pins every Python
dependency (pytest, ruff, the tracker's NumPy/OpenCV stack) to exact
versions with aarch64 wheels, installed into `/opt/frametop/venv`. The venv
sits on Fedora's own `python3`, which the base digest freezes, and sees the
system site-packages: PySide6 and Kirigami come from dnf for that
interpreter, so one `python3` imports both, as in the dev container.
3. **OpenVR from the pinned public tag** — `v2.15.6`, in
`scripts/openvr.sh`. The build scripts fetch its headers and check their
sha256. The image has no SteamVR, so it links the SDK's prebuilt
`libopenvr_api.so` (in `/opt/frametop/lib`, also checked); the binaries'
rpath names SteamVR's folder first, so on the Frame they load SteamVR's
own library, as the on-device builds do.
The native binaries (`ft-screens`, `ft-pointer`, the `ft_pointer` driver,
`ft-gaze`, `ft-gazepanel`, `ft-powerd`) are built by the components' own
`build.sh` scripts, run inside the image with `FRAME_IN_BOX=1`, so there is
one build recipe for the image and the Frame. They stay in their `build/`
folders under `/src/frametop` and are copied to `/opt/frametop`. Hand
tracking stays deferred exactly as in `install.sh` (ncnn is a heavy,
separately triggered build).
What deliberately stays outside the image for now: the **host payload** — the
SteamVR driver registration (`vrpathreg`), ft-camd's file capabilities
(setcap), systemd units, the KWin script. Those need the SteamOS host, and
they are what `install.sh` does today. Packaging them into a checksummed
payload tarball is the next slice.
## The `ft` wrapper
Everything that runs Frametop goes through the wrapper at the repo root, so
the integration points live in one place:
```
ft dev build # build the image from this repo (docker or podman)
ft update # pull the published (versioned) image, pin its digest
ft dev shell # interactive shell, repo at /src/frametop
ft ft-screens ... # run any program from /opt/frametop
ft dev test # the Python test suites inside the image
ft clean # remove frametop's leftovers only (see the store section)
```
`FT_IMAGE` overrides the image reference (repo mode defaults to the locally
built `frametop:local`). Development commands live behind `ft dev` so an
installed copy — which has no repo to build from — refuses them. Containers
run through the wrapper are named `frametop-<program>-<pid>`. Runs get no
access to the host beyond the repo mount: how Frametop's programs run on the
Frame is still open (see [design.md](design.md)).
### No :latest on a headset
A moving tag has no place on a headset: it cannot be reproduced in a bug
report and cannot be rolled back. The wrapper enforces this for what gets
**deployed** — installed mode refuses to run without a pinned reference, and
both the update source and the pinned reference are rejected if they say
`:latest`. Local development can use any tag it likes (`FT_IMAGE` is never
second-guessed in repo mode).
How pinning works:
- `install.sh` (next slice) writes a **version tag** to
`~/.config/frametop/published` — the only place updates come from.
- `ft update` pulls that reference and then writes the **digest**
(`repo@sha256:...`) to `~/.config/frametop/image`. Every later run, and
every bug report, names exactly that image.
- Rollback is editing one file: put the previous digest back into
`~/.config/frametop/image`. The old image is still in the local store
(remove it with `ft clean` when you are done with it).
Releases (below) don't use these: a release is a file, its image is pinned
by ID in `.frametop-release`, and you roll back by running the previous
release's `install.sh`.
## The shared podman store
On SteamOS, rootless podman has **one container/image store**, and Frametop
shares it with Valve's Android layer: Lepton names its containers
`lepton-<context>`, and they live in the same store as ours. Two rules
follow, and both are enforced or documented rather than hoped for:
1. **Frametop never touches anything it does not own.** Containers get stable
`frametop-<program>` names; `ft clean` deletes only containers matching
`frametop-*` and only images whose reference names frametop. It never runs
store-wide commands.
2. **Nobody should run store-wide podman commands on a Frame.**
`podman rm -a`, `podman rmi -a`, and `podman system prune` would delete
Valve's containers and images too. Use `ft clean` for Frametop's share of
the store and leave the rest of it to Steam.
## Network path for pulls
`ft update` (and install) contact the registry over the Frame's normal
Wi-Fi client interface (`wlan0`), not the 6 GHz streaming link to the PC
dongle. The image is roughly 1–1.5 GB compressed — pulls ride the home
network, so a weak Wi-Fi link is the bottleneck, not the streaming antenna.
## Storage
The image replaces the build toolchain, not adds to it: today's distrobox dev
container with its dnf history is the heavyweight; the pulled image carries
only the runtime plus the frozen build inputs (~1–1.5 GB compressed, a few GB
unpacked, in `~/.local/share/containers`). Once install.sh switches to the
image, the dev container is only needed for development and can be removed
from user machines. `ft clean` reclaims space from superseded frametop
images.
## Running on the Frame
The programs need much more of the host than a plain `podman run` gives them;
[design.md](design.md#the-runtime-on-the-frame) lists what, the device
findings so far, and the options. What's built so far is option A, a
distrobox made from the image, the same kind of container as the dev
container. The headset trial (step 2 below) decides.
Every program an installed Frametop runs in its container goes through
`scripts/in-box`: the pointer and power services, the gaze service's ft-gaze,
ft-eyes and calibration panel, the desktop's ft-screens, the remote desktop,
and the settings apps. A source install runs them in `dev`; a release names
its own container in `.frametop-release`.
## Releases
A release is one file, **Frametop.zip** (about 1.1 GB), attached to a GitHub
release: the image as an OCI archive (`frametop-image.tar`, from `podman save`),
`frametop-release.json` (version, commit, the archive's sha256, the image's
ID, and the SteamOS table), the installer `install-release.sh`, and the
install window from `framedrop/installer`. No container registry. It installs
three ways, all through the same installer:
- FrameDrop on a PC copies the zip's folder to the headset and adds "Frametop"
to the library; Play opens the install window (`framedrop/README.md`).
- Unpacked on the headset, `Frametop/frametop-install.sh` opens the same
window.
- `get.sh --release` downloads the zip from GitHub (the newest stable release,
or `--experimental`, `--version V`, `--zip FILE`) and runs
`install-release.sh` in the terminal.
`install-release.sh`:
1. Checks this SteamOS build (`BUILD_ID` in `/etc/os-release`) against the
SteamOS table, `steamos.json`: the newest one, from main on GitHub, when it
can fetch it, else the copy in the release. Tested: on. Not tested yet: a
warning, and a question (`--yes` goes on). Broken for this release: it stops
and names the release that fixes it (`--any-steamos` goes on). A broken build
counts from its `from` release up to its `fixed_in`, so a release that needs
a newer SteamOS refuses an older one, and the other way round.
2. Checks the archive's sha256 (a damaged copy stops it, exit status 3, and
`get.sh` downloads again), loads it into podman, checks the image's ID, and
tags it `localhost/frametop:VERSION`.
3. Copies the image's `/src/frametop` (the repo at that commit, built) to
`~/.local/share/frametop/releases/VERSION` with a `.frametop-release`
(version, commit, channel, image, ID, and the container's name, `frametop-`
and the ID's start), and runs that copy's `install.sh`.
4. In a release, `install.sh` builds nothing: it installs the distrobox the
image brings, makes the release's container from the image
(`release-box.sh`, which checks the ID again), and installs the services
from the release's folder. Each release has its own container, so
installing one doesn't stop the one running.
5. The release installed before stays; running its `install.sh` goes back to
it. Older ones are removed, with their containers and images.
`uninstall.sh` removes them all.
Every update is a full 1.1 GB download, since the zip holds the whole image
(user decision, 2026-10-07: one file, no registry). The `ft` wrapper's
installed mode (`ft update`, `~/.config/frametop/published` and `image`) pulls
from a registry, so releases don't use it.
## CI
`.github/workflows/release.yml` runs on Depot's arm64 runners
(`depot-ubuntu-24.04-arm-4`), only for a `v*` tag or by hand: no pushes or pull
requests, since the runners are paid and a fork's pull request would run on
them. It builds the image with podman (`ft dev build`, as on the Frame), runs
the test gate inside it (`ft dev test`: Python, C, and shell), checks what a
release installs from the image, and builds Frametop.zip with
`framedrop/build.sh --image`. A tag makes a draft GitHub release with the zip,
its FrameDrop manifest, `frametop-release.json`, and `SHA256SUMS`, as a
prerelease when the tag has a `-` (`v0.3.0-exp.1`); someone publishes it. A
manual run keeps the zip as an artifact for a week. The repo needs Depot's
GitHub app for the runner label to work.
The repo moved from DeeJanuz/frametop to the Frametop organization on
2026-10-09, because Depot's runners need an organization. GitHub redirects the
old repo URLs; the old Pages one-liner (`deejanuz.github.io/frametop/get.sh`)
is kept by DeeJanuz/deejanuz.github.io, whose `frametop/get.sh` and
`frametop/uninstall.sh` hand over to `frametop.github.io/frametop`. A release:
merge into experimental (and main for a stable one, by merge commit), push,
tag, and publish the draft.
## The longer arc
The image only pays off when it makes installing Frametop easier, so none of
this ships until the whole path works:
1. Image, wrapper, and CI (this directory).
2. A headset trial: Frametop's services run from the image (the runtime
question above), next to an install time measured against today's
on-device build. Go or no-go here.
3. A release pipeline: a tag builds the image, tests it, and attaches
Frametop.zip to a draft GitHub release (CI above). The image carries the
host side too (its `/src/frametop`), so there's no separate tarball.
4. FrameDrop, the unpacked zip, and `get.sh --release` install a release
(Releases above). The source install stays for development.
+242
View File
@@ -0,0 +1,242 @@
# Frametop packaging: the OCI image as the product
This is the design rationale for packaging Frametop as a CI-built OCI image.
[pack/README.md](README.md) is the practical reference (what the image
contains, the mount matrix, CI); this file explains why it is built this way,
what it solves, and how it is meant to be used.
## The problem today
Frametop on the Frame is three worlds that today get assembled *on the user's
headset* at install time:
- **The C world**: six native binaries (`ft-screens`, `ft-pointer`, `ft-powerd`,
`ft-gaze`, and friends) built against Qt, OpenVR, and wlroots.
- **The Python world**: the session, gaze, and settings code, with a NumPy /
OpenCV / PySide6 stack.
- **The host payload**: the SteamVR driver, ft-camd, systemd units, desktop
files, the KWin script — the parts that must live on the SteamOS host.
The path there is `get.sh` → `install.sh` → `setup/dev-container.sh`: a
distrobox environment is set up, packages are installed via dnf, binaries are
compiled, Python dependencies are fetched. That design has structural costs:
1. **Every install is a build.** The first install downloads 1–2 GB and
compiles on the device. A flaky mirror, a renamed dnf package, or a failing
build step fails installation — individually, for every user, at the worst
possible moment (first contact with the project).
2. **No two installations are alike.** Unpinned dnf and pip installs mean user
A has one OpenCV and user B another. Bug reports become "works on my
headset" stories that cannot be reproduced.
3. **SteamOS updates hit the host boundary.** The container's toolchain lives
in the home folder and survives an update — but every update replaces
SteamVR, KWin, and gamescope, and anything in the container that points at
a host path, interface, or quirk can break. `scripts/update-check.py`
tracks exactly these dependencies. Packaging does not remove this risk —
the host boundary stays the host boundary — but it shrinks it to what it
genuinely is: with the toolchain frozen in the image there is no on-device
build left to rot, and revalidating against a new SteamOS means one CI run
instead of every headset finding out individually.
4. **Using and developing are coupled.** A user needs a build ecosystem on
their headset to *run* Frametop. Conversely, a developer tests in an
environment no user shares.
## The design
> **The OCI image is the product. CI builds it, users pull it, `ft` is the
> only interface in between.**
```
CI (GitHub Actions, native arm64) Frame / PC
┌──────────────────────────────┐ ┌─────────────────────────────┐
│ pack/Containerfile │ push → │ ghcr.io/<org>/frametop │
│ · toolbox base, digest-pinned│ │ ↓ podman pull │
│ · uv sync --frozen (lockfile)│ │ ~/.local/bin/ft │
│ · OpenVR tag, stb pinned │ │ · runs programs │
│ · six binaries compiled │ │ · owns the mounts │
│ · smoke-tested in CI │ │ · ft update = pull │
└──────────────────────────────┘ └─────────────────────────────┘
```
The `ft` wrapper holds the image-reference resolution (pinning, updates,
cleanup) and the development commands. How the programs themselves run from
the image on the Frame is still open: see "The runtime on the Frame" below.
## What it solves, point by point
| Today | With the image |
| --- | --- |
| Install = build on the device | `podman pull`. Minutes, no compilers, no dnf |
| Every environment drifts | Every user runs the *exact* CI environment |
| SteamOS updates rot the on-device build | No on-device build left; what an update can still break is the host boundary — revalidated once in CI instead of per headset |
| Users need a developer ecosystem | Users need podman (SteamOS ships it) and the host payload |
| "Works on my headset" | A bug report names an image tag; the developer reproduces it in `ft dev shell` within minutes |
| Shipping fixes | `ft update`. The user never compiles anything |
And the subtler win: **decoupled failure domains.** When Frametop breaks on a
device, it is now either (a) the image — in which case it breaks identically
for everyone and reproducibly in CI — or (b) one of the small, documented host
boundaries. Today (a) and (b) are one soup.
## Why OCI — and not uv alone, not Flatpak
**Why not "just uv"?** uv (lockfile + frozen sync) solves the Python world
properly, and we use it exactly that way *inside* the image. But Frametop is
more than Python: the six C binaries, Qt, OpenVR, wlroots, the compile step
itself. uv freezes no compiler, no dnf package, no `libopenvr_api`. uv alone
still leaves the user compiling on the device with a drifting C toolchain —
the largest source of divergence untouched.
**Why not Flatpak?** Flatpak would be the more "native" answer for desktop
apps on SteamOS, and for the settings GUIs it is not absurd in the long run.
But Frametop is not a collection of desktop apps. It is a set of daemons that
live in SteamVR, on Wayland sockets, on `/dev/shm`, and on localhost sockets —
precisely what Flatpak sandboxing turns into an adventure of `--talk-*` and
`--filesystem` holes. One would spend the effort knocking holes in the sandbox
until it is no longer a sandbox, while still maintaining a separate build
system (flatpak-builder, manifests, a runtime dependency) that does the same
environment-freezing work again, in Flatpak currency. podman, by contrast,
ships with SteamOS (distrobox, which `install.sh` adds, runs on it), and OCI is
the one artifact format that CI, GHCR, and local development all speak
natively.
**Why not "distrobox, but pinned"?** That is what we have today — the dev
container. But a distrobox container is *state on the device* (dnf
transactions accumulating over months, drift), not an *artifact*. The
transition is exactly the point: a container you maintain becomes an image you
replace. `podman pull` is idempotent; a container filesystem with six months
of history is not. (A distrobox *created from* the pinned image, and created
again on every update, is a different thing: it holds no state. It is one of
the runtime options below.)
## How it is used
**Developer (PC, from a checkout):**
```
./ft dev build # build the image from the checkout
./ft dev test # the test suites inside the image (strict)
./ft dev shell # interactive shell, repo at /src/frametop
./ft ft-screens # run a program
```
Repo mode defaults to the locally built `frametop:local` and mounts the
checkout at `/src/frametop`.
**User (Frame, once the install path exists):**
```
ft update # pull the published image, pin its digest
```
No repo on the device, no build. The image reference resolves `FT_IMAGE` →
`~/.config/frametop/image` (written by the installer, so installs pin what was
installed) → the published image. Development commands live behind `ft dev`
and are refused in installed mode — a user should not reach the build world by
accident, and an installed wrapper has no checkout to build from anyway.
**Container naming.** Containers run through the wrapper are named
`frametop-<program>-<pid>`: `podman ps` names the program, two runs of one
program don't replace each other, and `ft clean` finds the leftovers of
crashed runs by the prefix.
## What the tests do
Two levels, both running *inside the built image*:
- **`just test` (strict)** — the Python suites, the header-only C tests, and
a syntax check of every shell script, inside the image. Any failing suite
fails the run, and `python3` must import the dnf Qt stack and the locked
packages together, so Qt tests can't pass by skipping. This is the real
gain over "CI runs pytest on the runner": the tests run in exactly the
environment the user receives. What is green is green *in the product*.
- **The CI smoke job** — pulls the built image and checks it from the outside:
the binaries exist, the venv is intact, programs execute and answer. This
catches broken layers, missing files, and architecture mistakes.
`just lint` (report-only while the pre-existing ruff findings are worked down)
is the on-ramp to strict linting later.
## The runtime on the Frame
Open. Today the programs that run in a container run in the `dev` distrobox,
which is privileged, shares the host's PID, network, and IPC namespaces,
mounts `/dev`, `/sys`, `/tmp`, `/run/user/<uid>`, and the home folder, and
keeps the user's groups (`run.oci.keep_original_groups`). The programs rely on
that:
| Program | Needs from the host |
| --- | --- |
| ft-screens | `/dev/dri/renderD128` (GBM), `XDG_RUNTIME_DIR` (its Wayland socket, for KWin on the host) |
| ft-powerd | `/dev/input` (use), the backlight in `/sys`, writable through the `video` group, `~/.config/frametop.conf`, `~/.cache/frametop` (the brightness to put back after a crash) |
| ft-pointer | `~/.config/frametop.conf`, `/opt/steamvr` (it runs `vrcmd`) |
| ft-gaze, ft-gazepanel | `/dev/shm` (SteamVR's `eye-server.mmap`), `/dev/dri` |
| Settings apps | the Wayland socket and session bus, `distrobox-host-exec` (they run `systemctl --user` on the host) |
| All of them | the host network namespace: they talk over abstract sockets (`@ft_screens`, `@ft_pointer`, ...), and the host's datagram queue length (`net.unix.max_dgram_qlen`, 512 from systemd; a container's own namespace starts at 10, and the input relay's burst of releases then loses its last ones, which leaves buttons held) |
OpenVR clients need more, found on the device with a containerized ft-powerd
(SteamOS 0.4.3, SteamVR 2.18.2): the path registry `~/.config/openvr`, also at
the absolute `/home/steamos/...` paths it names; SteamVR's IPC control file in
`/tmp` (with a private `/tmp`, `VR_Init` fails with `Init_Internal` 124);
`HOME` set explicitly; and no `--user`, since rootless podman maps the
container's root to the desktop user and a forced uid breaks that mapping.
A `podman run` with a hand-picked list of mounts (the wrapper's first
`FT_FRAME=1` mode) got ft-powerd connected to SteamVR, but it had no
`/dev/dri`, `/dev/input`, writable `/sys`, host groups, config files, or
`XDG_RUNTIME_DIR`, so the programs couldn't do their jobs. Two ways give them
what the dev container gives them:
1. **A distrobox created from the pinned image.** The image keeps `sleep
infinity` as its command for this. It gets every mount, group, and
namespace above with no list to maintain; the units keep `distrobox
enter` and point at `/opt/frametop/bin`. An update creates the box again
from the new digest, so it holds no state. Its first start runs
distrobox's own setup, which once made installs over SSH stop at a sudo
prompt (issue #9).
2. **Quadlet units** (podman 5.5 on SteamOS ships the generator) with the
same flags as distrobox: privileged, host PID, network, and IPC, the same
mounts, the user's groups. No distrobox setup step, and systemd tracks the
container itself rather than a `podman` client.
Either way the image adds no isolation (the dev container has none either).
What it adds is a pinned environment that was built and tested before it
reached the headset. Starting a program costs about the same: on the Frame a
`podman run` starts in about 0.23 s, `distrobox enter` in about 0.35 s.
Rootless storage lands in `~/.local/share/containers`, shared with Valve's
`lepton-*` containers (see README.md).
## What `install.sh` does in this world
1. `podman pull` the image by digest, write the reference to
`~/.config/frametop/image`
2. copy the wrapper to `~/.local/bin/ft` (already installed-mode capable)
3. set up the runtime (a distrobox from the image, or Quadlet units) and
install the units, pointing at `/opt/frametop`
4. the host payload from the same release: SteamVR driver registration
(`vrpathreg`), the KWin script, desktop files, and the optional parts that
need sudo (ft-camd's capabilities, the eye tracker's frame grabber, the
Bluetooth fixes)
`get.sh` stays the front door, and the FrameDrop package runs the same steps;
the difference is that step 1 ships the frozen image instead of building on
the device. The image and the host payload come from one tagged commit.
## Open decisions
1. **Tagging**: decided — `:latest` is refused by the wrapper (see
pack/README.md, "Never :latest"). Releases cut version tags; `ft update`
pins the digest of whatever version tag `install.sh` recorded. What
remains open is only the cadence: a tag per release vs. per CI build.
2. **Registry home**: `ghcr.io/deejanuz/frametop`, published from `main` and
`v*` tags only.
3. **Pull without auth**: depends on package visibility; install.sh can pin
the reference either way.
4. **Settings apps**: currently dnf-provided (PySide6/Kirigami) and run via
the image. Whether the GUIs migrate toward Flatpak/host packages later is left
open deliberately.
5. **Host payload distribution**: payload tarball + `get.sh` as artifact
installer is the current proposal.
6. **Runtime on the Frame**: a distrobox from the image, or Quadlet units
(see above). Decided by a headset trial, which also measures the install
time against today's on-device build.
+234
View File
@@ -0,0 +1,234 @@
#!/usr/bin/env bash
# Install the Frametop release in this folder: a GitHub release's Frametop.zip, unpacked by
# FrameDrop (into ~/devkit-game/Frametop), by hand, or by get.sh --release. Next to this script
# are frametop-image.tar (Frametop, built, as a container image) and frametop-release.json (its
# version, commit, the image file's sha256 and the image's ID, and the SteamOS table). Nothing
# builds on the headset, and nothing else is downloaded.
#
# 1. It checks this SteamOS build against the SteamOS table: Frametop's newest from GitHub
# (pack/steamos.json on main) when it can get it, else the one in the release. Tested: on.
# Not tested yet: it says so and asks (--yes goes on). Broken for this release: it stops,
# and names the release that fixes it (--any-steamos goes on anyway).
# 2. It checks the image file's sha256, and loads it into podman as localhost/frametop:VERSION.
# 3. It copies the release's files out of the image to ~/.local/share/frametop/releases/VERSION
# and runs their install.sh, which makes the release's container and builds nothing.
# 4. The release installed before stays: running its install.sh goes back to it. Older ones
# are removed, with their containers and images.
#
# Usage: install-release.sh [--yes] [--any-steamos] [--unpack-only] [--dir DIR]
# [--no-eye-tracker] [--no-bluetooth | --bluetooth]
# --unpack-only stop after step 2 and the copy: don't run install.sh
# --dir DIR where the releases go (default ~/.local/share/frametop/releases)
# The rest go to install.sh. FRAMETOP_STEAMOS_TABLE names another table to fetch (a URL), or
# "none" to use only the release's. Exit status 3: the image file is damaged.
set -euo pipefail
here=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
TABLE_URL=${FRAMETOP_STEAMOS_TABLE:-https://raw.githubusercontent.com/Frametop/frametop/main/pack/steamos.json}
# check_release RELEASE_JSON TABLE_JSON BUILD_ID: is the release usable, and is it for this
# SteamOS build? Prints seven lines: status (tested, untested, or broken), version, commit,
# channel, the image file's sha256, the image's ID, and a note for the user.
check_release() {
python3 - "$@" <<'EOF'
import json, re, sys
release_path, table_path, build = sys.argv[1:4]
def fail(msg):
print(f"the release's frametop-release.json isn't usable: {msg}", file=sys.stderr)
sys.exit(1)
try:
with open(release_path) as f:
r = json.load(f)
except (OSError, ValueError) as e:
fail(str(e))
if not isinstance(r, dict) or r.get("schema") != "frametop.release/v1":
fail("not frametop.release/v1")
image = r.get("image") if isinstance(r.get("image"), dict) else {}
fields = {
"version": (r.get("version"), r"[0-9A-Za-z][0-9A-Za-z.+_-]{0,63}"),
"commit": (r.get("commit"), r"[0-9a-f]{7,40}"),
"channel": (r.get("channel", ""), r"(stable|experimental|)"),
"image sha256": (image.get("sha256"), r"[0-9a-f]{64}"),
"image id": (image.get("id"), r"[0-9a-f]{64}"),
}
for name, (value, pattern) in fields.items():
if not isinstance(value, str) or not re.fullmatch(pattern, value):
fail(f"bad {name}: {str(value)[:80]!r}")
version = r["version"]
def base(v):
"""0.3.0-exp.1 -> (0, 3, 0): the release it leads up to."""
return tuple(int(p) for p in re.findall(r"\d+", str(v).split("-")[0].split("+")[0]))
table = r.get("steamos")
if table_path: # Frametop's newest table, when it could be fetched
try:
with open(table_path) as f:
newer = json.load(f)
if isinstance(newer, dict) and isinstance(newer.get("tested"), list):
table = newer
except (OSError, ValueError):
pass
table = table if isinstance(table, dict) else {}
def entries(kind):
listed = table.get(kind)
return [e for e in listed if isinstance(e, dict)] if isinstance(listed, list) else []
def oneline(s):
return " ".join(str(s).split())[:300]
broken = [b for b in entries("broken") if b.get("build") == build
and (not b.get("from") or base(version) >= base(b["from"]))
and (not b.get("fixed_in") or base(version) < base(b["fixed_in"]))]
tested = [t for t in entries("tested") if t.get("build") == build]
if broken:
status = "broken"
note = oneline(broken[0].get("reason", "it doesn't work on this SteamOS build"))
if broken[0].get("fixed_in"):
note += f" (fixed in Frametop {oneline(broken[0]['fixed_in'])})"
elif tested:
status, note = "tested", ""
else:
status = "untested"
others = entries("tested")[-3:]
note = ("tested on " + ", ".join(f"SteamOS {e.get('version', '?')} (build {e.get('build', '?')})"
for e in others)) if others else "not tested on any SteamOS build yet"
for line in (status, version, r["commit"], r["channel"], image["sha256"], image["id"], oneline(note)):
print(line)
EOF
}
main() {
local yes=0 any=0 unpack_only=0 base=$HOME/.local/share/frametop/releases tty=0 answer
local pass=()
while [ $# -gt 0 ]; do
case $1 in
--yes) yes=1; pass+=("$1") ;;
--any-steamos) any=1 ;;
--unpack-only) unpack_only=1 ;;
--dir) base=${2:?--dir needs a folder}; shift ;;
--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
-h|--help) sed -n '2,23p' "$0"; return 0 ;;
*) echo "unknown option: $1" >&2; return 2 ;;
esac
shift
done
if ! { grep -qx 'ID=steamos' /etc/os-release && grep -qE '^VARIANT_ID="?vr"?$' /etc/os-release; } 2>/dev/null; then
echo "Frametop installs on a Steam Frame (SteamOS, VR variant)." >&2
return 1
fi
for f in frametop-release.json frametop-image.tar; do
[ -f "$here/$f" ] || { echo "$here/$f is missing: unpack the whole Frametop.zip" >&2; return 1; }
done
{ : </dev/tty; } 2>/dev/null && tty=1
# podman's, even from a terminal in a VR desktop (its session has its own)
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
local build osver tmp status version commit channel sha id note
build=$(sed -n 's/^BUILD_ID=//p' /etc/os-release | tr -d '"')
osver=$(sed -n 's/^VERSION_ID=//p' /etc/os-release | tr -d '"')
tmp=$(mktemp)
if [ "$TABLE_URL" = none ] || ! curl -fsS --max-time 8 --proto '=https' "$TABLE_URL" -o "$tmp" 2>/dev/null; then
: >"$tmp" # offline: the release's own table
fi
{ read -r status; read -r version; read -r commit; read -r channel; read -r sha; read -r id; read -r note; } \
< <(check_release "$here/frametop-release.json" "$([ -s "$tmp" ] && echo "$tmp")" "$build") ||
{ rm -f "$tmp"; return 1; }
rm -f "$tmp"
[ -n "${id:-}" ] || return 1
case $status in
tested) echo "Frametop $version: tested on this SteamOS ($osver, build $build)" ;;
untested)
echo "Frametop $version hasn't been tested on this SteamOS yet ($osver, build $build); $note."
echo "It usually works: SteamOS updates rarely change what Frametop uses. If something's"
echo "wrong afterwards, scripts/doctor.sh in the release's folder says what changed."
if [ "$yes" = 0 ]; then
[ "$tty" = 1 ] || { echo "No terminal to ask in: add --yes to install it anyway." >&2; return 1; }
read -r -p "Install it anyway? [Y/n] " answer </dev/tty || answer=
[[ ${answer:-y} =~ ^[Yy] ]] || return 1
fi ;;
broken)
echo "Frametop $version doesn't work on this SteamOS ($osver, build $build): $note." >&2
if [ "$any" = 0 ]; then
echo "Nothing installed. --any-steamos installs it anyway." >&2
return 1
fi ;;
esac
local image=localhost/frametop:$version box=frametop-${id:0:12} dest=$base/$version
if [ -f "$dest/.frametop-release" ] && ! grep -qxF "IMAGE_ID=$id" "$dest/.frametop-release"; then
echo "$dest holds another build of Frametop $version. Move it away, then run this again." >&2
return 1
fi
printf '\n\033[1m== 0/10 loading Frametop %s (a minute or two)\033[0m\n' "$version"
if [ "$(podman image inspect -f '{{.Id}}' "$image" 2>/dev/null)" = "$id" ]; then
echo "already loaded"
else
local free
free=$(df -P -BG "$HOME" | awk 'NR == 2 { sub("G", "", $4); print $4 }')
if [ "${free:-0}" -lt 5 ]; then
echo "Only ${free:-0} GB free in your home folder; Frametop needs about 4 GB. Free some space first." >&2
return 1
fi
echo "checking the image file"
echo "$sha $here/frametop-image.tar" | sha256sum -c --status ||
{ echo "frametop-image.tar is damaged (its sha256 doesn't match): download Frametop.zip again" >&2; return 3; }
podman load -q -i "$here/frametop-image.tar" >/dev/null
podman image exists "$id" ||
{ echo "the image file didn't load as the image frametop-release.json names" >&2; return 1; }
podman tag "$id" "$image"
fi
if [ ! -f "$dest/.frametop-release" ]; then
echo "copying the release's files to $dest"
mkdir -p "$base"
rm -rf "$dest.new"
mkdir "$dest.new"
local copied=0
podman rm -f "frametop-copy-$$" >/dev/null 2>&1 || true
podman create --name "frametop-copy-$$" "$image" true >/dev/null &&
podman cp "frametop-copy-$$:/src/frametop/." "$dest.new/" || copied=1
podman rm -f "frametop-copy-$$" >/dev/null 2>&1 || true
[ "$copied" = 0 ] || { rm -rf "$dest.new"; echo "couldn't copy the release's files out of its image" >&2; return 1; }
printf 'VERSION=%s\nCOMMIT=%s\nCHANNEL=%s\nIMAGE=%s\nIMAGE_ID=%s\nBOX=%s\n' \
"$version" "$commit" "$channel" "$image" "$id" "$box" >"$dest.new/.frametop-release"
mv "$dest.new" "$dest"
fi
echo "Frametop $version (${commit:0:7}) in $dest"
if [ "$unpack_only" = 1 ]; then
echo "Install with: $dest/install.sh"
return 0
fi
if [ "$tty" = 1 ]; then
"$dest/install.sh" ${pass[@]+"${pass[@]}"} </dev/tty
else
"$dest/install.sh" ${pass[@]+"${pass[@]}"} </dev/null
fi
# This release is now "current"; the one before stays, to go back to. Older ones go, with
# their containers and images (only Frametop's: frametop-... and localhost/frametop:...).
local prev= d old_image old_box
[ -L "$base/current" ] && prev=$(readlink "$base/current")
ln -sfn "$version" "$base/current"
for d in "$base"/*/; do
d=${d%/}
[ -L "$d" ] && continue # "current" itself, a link to the release just installed
[ -f "$d/.frametop-release" ] || continue
case ${d##*/} in "$version"|"$prev") continue ;; esac
old_image=$(sed -n 's/^IMAGE=//p' "$d/.frametop-release")
old_box=$(sed -n 's/^BOX=//p' "$d/.frametop-release")
echo "removing Frametop ${d##*/}, two releases back"
if [[ $old_box =~ ^frametop-[A-Za-z0-9_.-]+$ ]]; then podman rm -f "$old_box" >/dev/null 2>&1 || true; fi
if [[ $old_image == localhost/frametop:* ]]; then podman rmi "$old_image" >/dev/null 2>&1 || true; fi
rm -rf "$d"
done
}
main "$@"
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
# Runs on the Frame host, from install.sh in a release (pack/install-release.sh): make the
# release's container from its image, as setup/dev-container.sh makes "dev" for a source
# install. Both come from .frametop-release: IMAGE (localhost/frametop:VERSION, loaded from the
# release's image file already), IMAGE_ID (checked), and BOX, the container's name, which
# scripts/in-box runs the release's programs in. Each release gets a container of its own, so
# installing one never stops the programs of the one running now; install-release.sh removes
# the containers of releases it removes.
# Usage: pack/release-box.sh
set -euo pipefail
tree=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
field() { sed -n "s/^$1=//p" "$tree/.frametop-release" | tail -1; }
image=$(field IMAGE)
id=$(field IMAGE_ID)
box=$(field BOX)
[[ $image == localhost/frametop:* ]] && [[ $id =~ ^[0-9a-f]{64}$ ]] && [[ $box =~ ^frametop-[A-Za-z0-9_.-]+$ ]] ||
{ echo "$tree/.frametop-release doesn't name a localhost/frametop image, its ID, and a frametop-... container" >&2; exit 1; }
export XDG_RUNTIME_DIR=/run/user/$(id -u)
[ "$(podman image inspect -f '{{.Id}}' "$image" 2>/dev/null)" = "$id" ] ||
{ echo "$image isn't the release's image (load it with the release's install-release.sh)" >&2; exit 1; }
if podman container exists "$box"; then
echo "the $box container is there already"
else
echo "creating the $box container"
# --no-entry: no "enter this container" entry among the apps (Launch a program lists them).
"$HOME/.local/bin/distrobox" create --yes --no-entry --name "$box" --image "$image"
fi
"$tree/scripts/container-up.sh" "$box"
echo "$box ready: $image"
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Write frametop-release.json, for a release's Frametop.zip (framedrop/build.sh --image):
the version, the commit, the channel, the image file's sha256, the image's ID (its config's
digest: podman's ID for it once loaded), and the SteamOS table, pack/steamos.json.
pack/install-release.sh reads it.
pack/release-info.py --image-file frametop-image.tar --version 0.3.0 --commit SHA \\
[--channel stable|experimental] > frametop-release.json
The image file is an OCI archive (podman save --format oci-archive). The channel defaults to
experimental for a version with a "-" (0.3.0-exp.1), else stable.
"""
import argparse
import hashlib
import json
import re
import sys
import tarfile
from pathlib import Path
HERE = Path(__file__).resolve().parent
def image_id(path):
"""The config digest of the one image in an OCI archive."""
with tarfile.open(path) as tar:
def blob(digest):
algo, _, hexd = digest.partition(":")
return json.load(tar.extractfile(f"blobs/{algo}/{hexd}"))
index = json.load(tar.extractfile("index.json"))
manifests = index.get("manifests", [])
if len(manifests) != 1:
raise SystemExit(f"{path}: {len(manifests)} images in it, not one")
manifest = blob(manifests[0]["digest"])
return manifest["config"]["digest"].partition(":")[2]
def sha256(path):
h = hashlib.sha256()
with open(path, "rb") as f:
while chunk := f.read(1 << 20):
h.update(chunk)
return h.hexdigest()
def main():
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--image-file", required=True, type=Path)
ap.add_argument("--version", required=True)
ap.add_argument("--commit", required=True)
ap.add_argument("--channel", choices=("stable", "experimental"))
ap.add_argument("--steamos", type=Path, default=HERE / "steamos.json")
a = ap.parse_args()
if not re.fullmatch(r"[0-9A-Za-z][0-9A-Za-z.+_-]{0,63}", a.version):
ap.error(f"--version is malformed: {a.version!r}")
if not re.fullmatch(r"[0-9a-f]{7,40}", a.commit):
ap.error(f"--commit is malformed: {a.commit!r}")
table = json.loads(a.steamos.read_text())
json.dump({
"schema": "frametop.release/v1",
"version": a.version,
"commit": a.commit,
"channel": a.channel or ("experimental" if "-" in a.version else "stable"),
"image": {"file": a.image_file.name, "sha256": sha256(a.image_file), "id": image_id(a.image_file)},
"steamos": {"tested": table.get("tested", []), "broken": table.get("broken", [])},
}, sys.stdout, indent=2)
print()
if __name__ == "__main__":
main()
+11
View File
@@ -0,0 +1,11 @@
## Install
On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1.1 GB), then either:
- **With [FrameDrop](https://framedropvr.com) on a PC:** drop Frametop.zip on your paired headset. In the headset, open Frametop in your library and press Play.
- **On the headset:** unpack Frametop.zip (in the desktop's Dolphin or Konsole) and run `Frametop/frametop-install.sh`.
- **In a terminal on the headset:** `curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --release`
A window asks what to install, and your SteamOS password for the two optional parts that need it (our own eye tracker and the Bluetooth fixes); the password is only used for this install. Nothing is compiled on the headset. Restart SteamVR once afterwards.
Before installing, the installer checks your SteamOS build against the builds Frametop was tested on, and stops if this release is known not to work there.
+22
View File
@@ -0,0 +1,22 @@
{
"about": "The SteamOS builds Frametop has been tested on, and the ones it's known to break on. Every release's Frametop.zip carries a copy (pack/release-info.py), and pack/install-release.sh checks this SteamOS build against the newest one, this file on main, when it can fetch it. Add a build once Frametop passed the headset tests on it (HEADSET-TESTS.md; scripts/doctor.sh --mark-good prints the versions). A broken build names the release that fixes it (fixed_in), or the first release that needs something the build lacks (from), or both: the releases in between refuse to install on it and name the fix.",
"tested": [
{
"build": "20260922.6101926",
"version": "0.3.0",
"branch": "stable",
"steamvr": "2.17.10",
"frametop": "0.2.1",
"date": "2026-10-05"
},
{
"build": "20261007.6125817",
"version": "0.4.5",
"branch": "stable",
"steamvr": "2.18.2",
"frametop": "0.2.2",
"date": "2026-10-09"
}
],
"broken": []
}
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env python3
"""Offline test of a release's SteamOS check and its frametop-release.json: install-release.sh's
check_release (the script without its last line, which would install), on files
pack/release-info.py writes from a small made-up OCI archive. No network, podman, or install.
pack/test/release-test.py
"""
import hashlib
import io
import json
import os
import subprocess
import sys
import tarfile
import tempfile
from pathlib import Path
HERE = os.path.dirname(os.path.abspath(__file__))
REPO = os.path.join(HERE, "..", "..")
TMP = tempfile.mkdtemp(prefix="ft-release-test-")
SCRIPT = Path(REPO, "pack", "install-release.sh").read_text().rsplit('\nmain "$@"', 1)[0]
OURS = "20260922.6101926" # tested
NEW = "20261007.6180005" # in no table
failures = []
def check(label, got, want):
ok = got == want
print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
if not ok:
failures.append(label)
def write(name, obj):
path = os.path.join(TMP, name)
with open(path, "w") as f:
f.write(obj if isinstance(obj, str) else json.dumps(obj))
return path
def oci_archive(path, config=b'{"architecture":"arm64"}'):
"""An OCI archive with one image: index.json -> manifest -> config."""
def digest(b):
return hashlib.sha256(b).hexdigest()
manifest = json.dumps({"schemaVersion": 2, "config": {"digest": "sha256:" + digest(config)}, "layers": []}).encode()
index = json.dumps({"schemaVersion": 2, "manifests": [{"digest": "sha256:" + digest(manifest)}]}).encode()
with tarfile.open(path, "w") as tar:
for name, data in (("index.json", index), (f"blobs/sha256/{digest(manifest)}", manifest),
(f"blobs/sha256/{digest(config)}", config)):
info = tarfile.TarInfo(name)
info.size = len(data)
tar.addfile(info, io.BytesIO(data))
return digest(config)
def info(version, steamos, *extra):
table = write("steamos.json", steamos)
r = subprocess.run([sys.executable, os.path.join(REPO, "pack", "release-info.py"), "--image-file", IMAGE,
"--version", version, "--commit", "1234567", "--steamos", table, *extra],
capture_output=True, text=True)
return r.returncode, r.stdout
def pick(release, build, live=""):
r = subprocess.run(["bash", "-c", SCRIPT + '\ncheck_release "$@"', "install-release.sh", release, live, build],
capture_output=True, text=True)
return r.returncode, r.stdout.splitlines()
IMAGE = os.path.join(TMP, "frametop-image.tar")
ID = oci_archive(IMAGE)
SHA = hashlib.sha256(Path(IMAGE).read_bytes()).hexdigest()
tested = [{"build": OURS, "version": "0.3.0", "steamvr": "2.17.10"}]
code, out = info("0.3.0", {"tested": tested, "broken": []})
check("release-info.py writes it", code, 0)
rel = json.loads(out)
check("with the image file's sha256 and the image's ID", (rel["image"]["sha256"], rel["image"]["id"]), (SHA, ID))
check("a version without a \"-\" is stable", rel["channel"], "stable")
check("an experimental one", json.loads(info("0.3.0-exp.1", {"tested": tested})[1])["channel"], "experimental")
check("release-info.py refuses a malformed version", info("../0.3", {"tested": tested})[0], 2)
two = os.path.join(TMP, "two.tar")
with tarfile.open(two, "w") as tar:
data = json.dumps({"manifests": [{"digest": "sha256:a"}, {"digest": "sha256:b"}]}).encode()
t = tarfile.TarInfo("index.json")
t.size = len(data)
tar.addfile(t, io.BytesIO(data))
r = subprocess.run([sys.executable, os.path.join(REPO, "pack", "release-info.py"), "--image-file", two,
"--version", "1.0", "--commit", "1234567"], capture_output=True, text=True)
check("and an archive with two images", r.returncode != 0, True)
stable = write("stable.json", out)
code, out = pick(stable, OURS)
check("a tested build: tested", (code, out[0], out[1]), (0, "tested", "0.3.0"))
check("and the image's sha256 and ID come through", out[4:6], [SHA, ID])
code, out = pick(stable, NEW)
check("a build in no table: untested", out[0], "untested")
check("which says where it was tested", out[6], "tested on SteamOS 0.3.0 (build 20260922.6101926)")
broken = {"tested": tested, "broken": [{"build": NEW, "reason": "the 3D mouse\nhas no laser", "fixed_in": "0.3.1"}]}
code, out = pick(write("b.json", info("0.3.0", broken)[1]), NEW)
check("a build broken for this release: broken", out[0], "broken")
check("with the reason on one line, and the fix", out[6], "the 3D mouse has no laser (fixed in Frametop 0.3.1)")
code, out = pick(write("b2.json", info("0.3.1-exp.2", broken)[1]), NEW)
check("the release with the fix isn't broken there", out[0], "untested")
needs = {"tested": tested, "broken": [{"build": OURS, "reason": "needs SteamVR 2.18", "from": "0.4.0"}]}
check("a break from a later release doesn't count for this one", pick(write("n1.json", info("0.3.0", needs)[1]), OURS)[1][0],
"tested")
check("it counts from that release on, over tested", pick(write("n2.json", info("0.4.0", needs)[1]), OURS)[1][0], "broken")
live = write("live.json", {"tested": tested + [{"build": NEW, "version": "0.5.5"}], "broken": []})
check("the newest table from GitHub counts over the release's", pick(stable, NEW, live)[1][0], "tested")
check("a table that isn't one is ignored", pick(stable, NEW, write("junk.json", "<html>"))[1][0], "untested")
live_broken = write("live-broken.json", {"tested": tested, "broken": [{"build": OURS, "reason": "x", "fixed_in": "0.3.1"}]})
check("and it can say a build broke after the release came out", pick(stable, OURS, live_broken)[1][0], "broken")
good = json.loads(Path(stable).read_text())
for label, change in (("a version with a slash", {"version": "../1.0"}),
("a commit that isn't hex", {"commit": "$(reboot)"}),
("an image ID that isn't one", {"image": {"sha256": SHA, "id": "latest"}}),
("no image sha256", {"image": {"id": ID}}),
("another schema", {"schema": "frametop.release/v0"})):
check(f"a release file with {label} fails", pick(write("bad.json", {**good, **change}), OURS), (1, []))
check("a release file that isn't JSON fails", pick(write("notjson.json", "{"), OURS), (1, []))
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
sys.exit(1 if failures else 0)
Executable
+110
View File
@@ -0,0 +1,110 @@
#!/usr/bin/env bash
# The runtime trial (HEADSET-TESTS.md, "Frametop from the image"): switch this Frame's Frametop
# to a release, installed from its Frametop.zip, and back to what was there.
# pack/trial.sh on DIR # DIR: the unpacked zip's Frametop folder (install-release.sh in it)
# pack/trial.sh desktop # restart the VR desktop from the release (ft-screens in its container)
# pack/trial.sh status # what runs where
# pack/trial.sh off # put it all back
# on saves what the release's install.sh replaces to ~/.local/state/frametop-trial/saved: the
# frametop-* user units with their drop-ins and .wants links, the launcher and menu entries, the
# SteamVR driver's folder and registry, frametop.conf, and the desktop's shortcuts file. It
# moves the drop-ins aside (they would override the release's units), runs the release's
# install-release.sh --yes --no-eye-tracker --no-bluetooth (no sudo: the installed eye grabber
# stays), and restarts SteamVR so it loads the release's driver and services.
# off removes those files and puts the saved ones back, stops the release's container, and
# restarts SteamVR. The release stays in ~/.local/share/frametop/releases for another try.
# Restarting SteamVR closes everything open in VR.
set -euo pipefail
shopt -s nullglob
self=$(readlink -f "$0")
state=$HOME/.local/state/frametop-trial
saved=$state/saved
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
# What install.sh changes (pack/README.md, Releases), relative to the home folder, as globs.
paths() {
cat <<'EOF'
.config/systemd/user/frametop-*.service
.config/systemd/user/frametop-*.service.d
.config/systemd/user/*.wants/frametop-*
.local/share/applications/deckard-nested-desktop.desktop
.local/share/applications/native-deckard-nested-desktop.desktop
.local/share/applications/ft-*.desktop
.local/share/applications/frametop-profile-*.desktop
.local/share/frametop/ft_pointer
.config/openvr/openvrpaths.vrpath
.config/frametop.conf
.config/frametop/kglobalshortcutsrc
EOF
}
restart_steamvr() {
echo "restarting SteamVR"
systemctl --user restart steamvr.service
for _ in $(seq 60); do
systemctl --user -q is-active frametop-pointer.service && break
sleep 1
done
}
case ${1:-status} in
on)
src=$(cd "${2:?usage: pack/trial.sh on DIR (the Frametop folder of the unpacked zip)}" && pwd)
[ -x "$src/install-release.sh" ] || { echo "$src/install-release.sh isn't there" >&2; exit 1; }
[ -e "$saved" ] && { echo "a trial is on already: pack/trial.sh off first" >&2; exit 1; }
mkdir -p "$saved"
cd "$HOME"
while read -r glob; do
for p in $glob; do
[ -e "$p" ] || [ -L "$p" ] || continue # a plain name stays as is when it's missing
cp -a --parents "$p" "$saved/"
done
done < <(paths)
echo "saved $(find "$saved" -mindepth 1 -not -type d | wc -l) files in $saved"
rm -rf .config/systemd/user/frametop-*.service.d
systemctl --user daemon-reload
"$src/install-release.sh" --yes --no-eye-tracker --no-bluetooth
restart_steamvr
"$self" status ;;
desktop)
release=$HOME/.local/share/frametop/releases/current
[ -e "$saved" ] && [ -x "$release/desktops.sh" ] || { echo "no trial on" >&2; exit 1; }
"$release/desktops.sh" restart ;;
off)
[ -d "$saved" ] || { echo "no trial on (nothing saved in $saved)" >&2; exit 1; }
cd "$HOME"
while read -r glob; do
for p in $glob; do rm -rf "$p"; done
done < <(paths)
cp -a "$saved/." "$HOME/"
systemctl --user daemon-reload
restart_steamvr # first: the release's programs run in its container until SteamVR stops them
pkill -x ft-screens || true # the VR desktop from the release, if it's still up
for box in $(podman ps --format '{{.Names}}' | grep -E '^frametop-[0-9a-f]{12}$' || true); do
echo "stopping $box"
podman stop -t 5 "$box" >/dev/null || echo "couldn't stop $box (podman stop $box)" >&2
done
mv "$saved" "$state/restored-$(date +%Y%m%d-%H%M%S)"
echo "back as before; restart the VR desktop from Launch a program to leave the release's"
"$self" status ;;
status)
[ -d "$saved" ] && echo "trial: ON (saved in $saved)" || echo "trial: off"
for u in frametop-input-relay frametop-pointer frametop-power frametop-gaze frametop-desktop; do
printf '%-22s %-9s %s\n' "$u" "$(systemctl --user is-active $u.service)" \
"$(systemctl --user show -P ExecStart $u.service | grep -o 'argv\[\]=[^;]*' | cut -c8- | cut -c1-110)"
done
for name in ft-pointer ft-powerd ft-gaze ft-eyes ft-screens; do
if [ "$name" = ft-eyes ]; then pids=$(pgrep -f "[/]ft-eyes " || true); else pids=$(pgrep -x "$name" || true); fi
for pid in $pids; do
id=$(grep -o 'libpod-[0-9a-f]\{12\}' "/proc/$pid/cgroup" 2>/dev/null | head -1 | cut -c8-) || id=
box=host
[ -n "$id" ] && box=$(podman ps --filter "id=$id" --format '{{.Names}}' 2>/dev/null || echo "$id")
printf '%-11s pid %-7s in %-22s %s\n' "$name" "$pid" "$box" \
"$(tr '\0' ' ' </proc/$pid/cmdline 2>/dev/null | grep -o '/home/[^ ]*' | head -1 || true)"
done
done
readlink -f "$HOME/.local/share/frametop/releases/current" 2>/dev/null | sed 's/^/release: /' || true ;;
*) sed -n '2,6p' "$0" >&2; exit 2 ;;
esac
+2 -1
View File
@@ -9,9 +9,10 @@ root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C pointer/driver 'set -e
mkdir -p build
. ../../scripts/openvr.sh
g++ -std=c++17 -O2 -fPIC -shared -fvisibility=hidden -fno-math-errno -Wall -Wno-unused-parameter \
-static-libstdc++ -static-libgcc -Wl,--exclude-libs,ALL \
-I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers \
$OPENVR_CFLAGS \
-o build/driver_ft_pointer.so driver_ft_pointer.cpp -lpthread
max=$(objdump -T build/driver_ft_pointer.so | grep -oE "GLIBC_[0-9.]+" | sort -uV | tail -1)
echo "built build/driver_ft_pointer.so, newest glibc symbol: $max"
+3 -2
View File
@@ -4,6 +4,7 @@ set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C pointer/helper 'set -e; mkdir -p build
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers -I../common \
-o build/ft-pointer ft-pointer.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
. ../../scripts/openvr.sh
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter $OPENVR_CFLAGS -I../common \
-o build/ft-pointer ft-pointer.cpp $OPENVR_LIBS -lpthread
echo "built build/ft-pointer"'
+32
View File
@@ -0,0 +1,32 @@
// The gaze calibration panel's answers (see ft-pointer.cpp's top): while ft-gazed says the panel
// is up, a press from the relay answers it instead of clicking. Here so it can be tested without
// SteamVR (pointer/test/calpanel-test.sh).
//
// Accept (take this dot): the left button's press ("btn trigger 1", whatever mouse button,
// controller button or key combination is mapped to the left action), Meta+J ("gazekey left 1"),
// and a gaze precision or gaze drag press ("precision|gazedrag <source> 1"): those are the left
// button for someone who mapped it to one, and were dropped until 2026-10-09, so the panel
// never took a dot from them. Quit: the right button ("btn b 1") and Meta+K ("gazekey right 1").
// Their releases, and the other buttons, do nothing while the panel is up.
#pragma once
#include <cstdio>
#include <cstring>
enum class CalPanelAnswer { None, Accept, Quit, Ignore };
inline CalPanelAnswer calPanelAnswer(const char *msg) {
char source[16];
int value;
if (!std::strncmp(msg, "btn trigger 1", 13) || !std::strncmp(msg, "gazekey left 1", 14))
return CalPanelAnswer::Accept;
if ((std::sscanf(msg, "precision %15s %d", source, &value) == 2 ||
std::sscanf(msg, "gazedrag %15s %d", source, &value) == 2) && value == 1)
return CalPanelAnswer::Accept;
if (!std::strncmp(msg, "btn b 1", 7) || !std::strncmp(msg, "gazekey right 1", 15))
return CalPanelAnswer::Quit;
if (!std::strncmp(msg, "btn ", 4) || !std::strncmp(msg, "gazekey ", 8) ||
!std::strncmp(msg, "precision ", 10) || !std::strncmp(msg, "gazedrag ", 9))
return CalPanelAnswer::Ignore;
return CalPanelAnswer::None;
}
+4 -6
View File
@@ -8,13 +8,11 @@ PartOf=steamvr.service
Requisite=steamvr.service
[Service]
# Runs in the dev container (built there against its libraries). The helper process
# lives in the container, so clean it up explicitly around distrobox enter.
# Start the container in a scope of its own first: started by this service (distrobox enter
# does that on demand), stopping the service would stop the container and all in it.
ExecStartPre=-@REPO@/scripts/container-up.sh
# Runs in the container it was built in (the dev container, or a release's: scripts/in-box,
# which starts it in a scope of its own, so stopping this service can't stop it). The helper
# process lives in the container, so clean it up explicitly around it.
ExecStartPre=-/usr/bin/pkill -x ft-pointer
ExecStart=%h/.local/bin/distrobox enter dev -- @REPO@/pointer/helper/build/ft-pointer
ExecStart=@REPO@/scripts/in-box @REPO@/pointer/helper/build/ft-pointer
ExecStopPost=-/usr/bin/pkill -x ft-pointer
Restart=on-failure
RestartSec=3
+12 -12
View File
@@ -227,8 +227,9 @@
// drags again from there. Without gaze mode they work from wherever the pointer is.
// The gaze calibration panel (gaze/panel/ft-gazepanel, run by the gaze service): while
// ft-gazed says it's up ("calpanel 1", renewed every second; it lapses 3 s after the last),
// the dot hides and a press answers the panel instead of clicking: a left click or gaze_left
// sends "calaccept" to @ft_gazed (take this dot now), a right click or gaze_right "calquit".
// the dot hides and a press answers the panel instead of clicking: a left click, gaze_left, or
// a gaze_precision or gaze_drag press sends "calaccept" to @ft_gazed (take this dot now), a right
// click or gaze_right "calquit" (calpanel.h).
// POINTER_ROLE (right, left, or stylus): the hand role our device takes while connected. A
// Frame controller in your hand counts as used through its touch sensors and takes its hand's
// role back, and then no click lands (see "no hand role" in the main loop): with a controller
@@ -316,6 +317,7 @@
// POINTER_ROLE (right): gaze precision and keyboard clicks, above.
#include <openvr.h>
#include "calpanel.h"
#include "vrbuttons.h"
#include "vrmath.h"
@@ -479,11 +481,11 @@ std::string ExeDir() {
return p.substr(0, p.rfind('/'));
}
// One of ft-screens' panels showing a desktop: a screen (frametop.screen.N), a floating
// window (frametop.float.N), or a floating window's popup (frametop.float.N.sub.K), not a
// control of theirs.
// One of ft-screens' panels showing a desktop: a screen (frametop.screen.N), another
// machine's display (frametop.remote.N), a floating window (frametop.float.N), or a floating
// window's popup (frametop.float.N.sub.K), not a control of theirs.
bool FramePanel(const std::string &key) {
for (const char *prefix : {"frametop.screen.", "frametop.float."}) {
for (const char *prefix : {"frametop.screen.", "frametop.remote.", "frametop.float."}) {
if (key.rfind(prefix, 0) != 0) continue;
const std::string rest = key.substr(std::strlen(prefix));
const size_t dot = rest.find('.');
@@ -1485,14 +1487,12 @@ int main() {
}
}
if (Clock::now() < calPanelUntil) {
const bool accept = !std::strncmp(buf, "btn trigger 1", 13) || !std::strncmp(buf, "gazekey left 1", 14);
const bool quit = !std::strncmp(buf, "btn b 1", 7) || !std::strncmp(buf, "gazekey right 1", 15);
if (accept || quit) {
SendTo(out, "ft_gazed", accept ? "calaccept" : "calquit");
const CalPanelAnswer answer = calPanelAnswer(buf);
if (answer == CalPanelAnswer::Accept || answer == CalPanelAnswer::Quit) {
SendTo(out, "ft_gazed", answer == CalPanelAnswer::Accept ? "calaccept" : "calquit");
continue;
}
if (!std::strncmp(buf, "btn ", 4) || !std::strncmp(buf, "gazekey ", 8) ||
!std::strncmp(buf, "precision ", 10) || !std::strncmp(buf, "gazedrag ", 9))
if (answer == CalPanelAnswer::Ignore)
continue; // their releases, and the other buttons: nothing to click now
}
{
+49
View File
@@ -0,0 +1,49 @@
// Offline test of the gaze calibration panel's answers (pointer/helper/calpanel.h): which relay
// messages take the dot, close the panel, or do nothing while it's up. Needs no SteamVR.
//
// pointer/test/calpanel-test.sh
#include "../helper/calpanel.h"
#include <cstdio>
int main() {
struct Case {
const char *msg;
CalPanelAnswer want;
} cases[] = {
{"btn trigger 1", CalPanelAnswer::Accept},
{"gazekey left 1", CalPanelAnswer::Accept},
{"precision mouse 1", CalPanelAnswer::Accept},
{"precision keyboard 1", CalPanelAnswer::Accept},
{"gazedrag mouse 1", CalPanelAnswer::Accept},
{"gazedrag keyboard 1", CalPanelAnswer::Accept},
{"btn b 1", CalPanelAnswer::Quit},
{"gazekey right 1", CalPanelAnswer::Quit},
{"btn trigger 0", CalPanelAnswer::Ignore},
{"btn b 0", CalPanelAnswer::Ignore},
{"btn a 1", CalPanelAnswer::Ignore}, // the relay's laser claim
{"btn x 1", CalPanelAnswer::Ignore},
{"btn system 1", CalPanelAnswer::Ignore},
{"gazekey left 0", CalPanelAnswer::Ignore},
{"gazekey right 0", CalPanelAnswer::Ignore},
{"precision mouse 0", CalPanelAnswer::Ignore},
{"gazedrag mouse 0", CalPanelAnswer::Ignore},
{"move 0.1000 -0.2000", CalPanelAnswer::None},
{"show", CalPanelAnswer::None},
{"recenter", CalPanelAnswer::None},
{"scroll 0 1", CalPanelAnswer::None},
{"typing", CalPanelAnswer::None},
};
const char *names[] = {"none", "accept", "quit", "ignore"};
int failed = 0;
for (const Case &c : cases) {
const CalPanelAnswer got = calPanelAnswer(c.msg);
if (got != c.want) {
std::printf("FAIL \"%s\": %s, want %s\n", c.msg, names[int(got)], names[int(c.want)]);
++failed;
}
}
std::printf("%s: %d of %zu cases\n", failed ? "FAILED" : "ok", int(sizeof cases / sizeof cases[0]) - failed,
sizeof cases / sizeof cases[0]);
return failed ? 1 : 0;
}
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# Offline test of the gaze calibration panel's answers (pointer/test/calpanel-test.cpp): builds
# it in the dev container and runs it there. Nothing reaches SteamVR, so it's safe next to it.
#
# pointer/test/calpanel-test.sh
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C pointer 'set -e; mkdir -p test/build
g++ -std=c++17 -O2 -Wall -o test/build/calpanel-test test/calpanel-test.cpp
test/build/calpanel-test'
+3 -2
View File
@@ -4,6 +4,7 @@ set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C power 'set -e; mkdir -p build
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers \
-o build/ft-powerd ft-powerd.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64
. ../scripts/openvr.sh
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter $OPENVR_CFLAGS \
-o build/ft-powerd ft-powerd.cpp $OPENVR_LIBS
echo "built build/ft-powerd"'
+3 -5
View File
@@ -7,12 +7,10 @@ After=steamvr.service
PartOf=steamvr.service
[Service]
# Runs in the dev container (built there against its libraries). Start the container in a
# scope of its own first (see scripts/container-up.sh), and clean up the process explicitly
# around distrobox enter.
ExecStartPre=-@REPO@/scripts/container-up.sh
# Runs in the container it was built in (the dev container, or a release's: scripts/in-box,
# which starts it in a scope of its own). Clean up the process explicitly around it.
ExecStartPre=-/usr/bin/pkill -x ft-powerd
ExecStart=%h/.local/bin/distrobox enter dev -- @REPO@/power/build/ft-powerd
ExecStart=@REPO@/scripts/in-box @REPO@/power/build/ft-powerd
# SIGTERM makes ft-powerd turn the displays back on before it exits.
ExecStopPost=-/usr/bin/pkill -x ft-powerd
Restart=on-failure
+31
View File
@@ -0,0 +1,31 @@
[project]
name = "frametop"
version = "0.1.0"
description = "Desktops in VR on the Steam Frame"
readme = "README.md"
requires-python = ">=3.12"
# Frametop's Python dependencies, locked in uv.lock. The Frame's runtime apps
# (input settings uses Kirigami) still come from the container's dnf packages;
# these groups cover everything uv can pin: dev tooling and the hand tracker's
# NumPy/OpenCV stack (Fedora's python3-opencv pulls in over a gigabyte).
[dependency-groups]
dev = [
"pytest>=8",
"ruff>=0.14",
"pyflakes>=3.4",
]
# type checking is dev tooling outside the image: the image installs the
# default groups (dev for CI's pytest/ruff, tracker for the hand tracker)
# but not this one
types = [
"mypy>=1.19",
]
tracker = [
"numpy>=2.2",
"opencv-python-headless>=4.11",
"huggingface_hub>=0.34",
]
[tool.uv]
package = false
+14
View File
@@ -0,0 +1,14 @@
#!/bin/bash
# Launch Frametop Remote Displays from a Plasma session on the Frame host.
# The app runs in Frametop's container (PySide6 and Kirigami come from Fedora there), as
# Display Settings does (see display-settings/ft-display-settings for the buses).
here=$(cd "$(dirname "$(readlink -f "$0")")" && pwd)
wl=${WAYLAND_DISPLAY:-wayland-0}
case $wl in /*) ;; *) wl="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/$wl" ;; esac
session_bus=${DBUS_SESSION_BUS_ADDRESS:-}
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
exec "$here/../scripts/in-box" env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
QT_QPA_PLATFORM="wayland;xcb" \
python3 "$here/ft_remote_displays.py" "$@"
@@ -0,0 +1,9 @@
[Desktop Entry]
Type=Application
Name=Frametop Remote Displays
GenericName=Other computers' monitors in VR
Comment=Connect to computers running Vibepollo and show their monitors as Frametop screens
Exec=@REPO@/remote-displays/ft-remote-displays
Icon=network-workgroup
Categories=Settings;HardwareSettings;Network;
Keywords=remote;display;monitor;moonlight;vibepollo;sunshine;stream;pc;frametop;
+525
View File
@@ -0,0 +1,525 @@
#!/usr/bin/env python3
"""Frametop Remote Displays: other computers' monitors as Frametop screens, streamed with
Moonlight's protocol from Vibepollo (docs/remote-displays.md).
A Kirigami (QML) app with a Python backend, like Frametop Display Settings. It runs in
the dev container:
- Hosts: computers running Vibepollo, found on the network (they announce _nvstream._tcp
over mDNS; avahi-browse on the host) or typed in. You sign in once with the host's Web UI
user name and password: Frametop asks it for an API token that can do only what it needs
(SCOPES), keeps that in ~/.local/share/frametop-stream/hosts/ADDRESS.token, readable only
by you, and forgets the password. The Web UI's certificate is self-signed, so the one seen
then is pinned (ADDRESS.pin, its public key's SHA-256): ft-stream and later sign-ins
refuse another. Whether each host answers on its Web UI port is checked now and then.
- Displays: a host's existing monitor, or a virtual one at any size, each its own stream
(ft-layout remote add pairs a client for it). Each one connects and disconnects on its
own, or all of a host's at once; a disconnected one keeps its place and settings.
Its stream's resolution, frame rate and bitrate, its width in VR, and whether it shows.
- Connection, per host with a Steam Link dongle on the Frame's hotspot (found with it on
the network, or with Find): auto (the dongle when it answers, else the network), network
only, or dongle only. Find looks for the host among the hotspot's clients (the same
<uniqueid> in their serverinfo as at its own address). Hosts without one use the network.
Where the displays are in VR is kept like the screens' (move them there; Display Settings'
Save as profile keeps it). Anything that touches the streams runs layout/ft-layout on the
host; the streams' state comes from ft-screens (@ft_screens: "remotes", "remote N info").
Launch with remote-displays/ft-remote-displays (host wrapper).
"""
import base64
import hashlib
import http.client
import json
import os
import re
import shutil
import socket
import ssl
import subprocess
import sys
import threading
import urllib.request
import uuid
from PySide6.QtCore import Property, QObject, QProcess, Qt, QTimer, QUrl, Signal, Slot
from PySide6.QtGui import QGuiApplication, QIcon
from PySide6.QtQml import QQmlApplicationEngine
from PySide6.QtQuickControls2 import QQuickStyle
HERE = os.path.dirname(os.path.abspath(__file__))
LAYOUT_DIR = os.path.join(HERE, "..", "layout")
sys.path.insert(0, LAYOUT_DIR)
import ft_layout # noqa: E402
FT_LAYOUT = os.path.join(LAYOUT_DIR, "ft-layout")
FT_SCREENS = "\0ft_screens"
TOKEN_DIR = os.path.expanduser("~/.local/share/frametop-stream/hosts") # each host's API token (ft-stream)
WEB_UI_PORT = 47990 # Vibepollo's Web UI, where ft-stream pairs and lists monitors
STREAM_RATES = [30, 60, 72, 90, 120]
# What Frametop's API token may do: pair its displays' clients (submit their PINs), list the
# clients and set their permissions, and list the host's monitors.
SCOPES = [{"path": "/api/pin", "methods": ["POST"]}, {"path": "/api/clients/list", "methods": ["GET"]},
{"path": "/api/clients/update", "methods": ["POST"]}, {"path": "/api/display-devices", "methods": ["GET"]}]
NAME_RE = r"[A-Za-z0-9][A-Za-z0-9 ._-]{0,39}"
RESOLUTIONS = [(1920, 1080, ""), (2560, 1440, ""), (3840, 2160, "4K"), (2560, 1080, "ultrawide"),
(3440, 1440, "ultrawide"), (5120, 1440, "super ultrawide"), (1920, 1200, "16:10"),
(2560, 1600, "16:10"), (1080, 1920, "portrait"), (1440, 2560, "portrait")]
def host_command(*cmd):
"""argv to run a command on the SteamOS host (we live in the dev container).
distrobox-host-exec reaches the host through the user's real session bus; inside the
desktop our DBUS_SESSION_BUS_ADDRESS is the nested session's private one, where it
fails (exit 127, silently)."""
if not shutil.which("distrobox-host-exec"):
return list(cmd)
bus = f"unix:path=/run/user/{os.getuid()}/bus"
return ["env", f"DBUS_SESSION_BUS_ADDRESS={bus}", "distrobox-host-exec"] + list(cmd)
def token_path(address):
return os.path.join(TOKEN_DIR, f"{address}.token")
def pin_path(address):
return os.path.join(TOKEN_DIR, f"{address}.pin")
def write_private(path, text):
os.makedirs(TOKEN_DIR, mode=0o700, exist_ok=True)
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_NOFOLLOW, 0o600)
with os.fdopen(fd, "w") as f:
f.write(text + "\n")
def spki_pin(der):
"""A certificate's public key pin as curl takes it (CURLOPT_PINNEDPUBLICKEY)."""
pem = subprocess.run(["openssl", "x509", "-inform", "der", "-pubkey", "-noout"], input=der,
capture_output=True, check=True).stdout
key = subprocess.run(["openssl", "pkey", "-pubin", "-outform", "der"], input=pem,
capture_output=True, check=True).stdout
return "sha256//" + base64.b64encode(hashlib.sha256(key).digest()).decode()
def parse_avahi(text):
"""avahi-browse -rpt _nvstream._tcp -> [{name, address, dongle}]: a host's address on the
network, and on the Frame's hotspot (wlanap: its Steam Link dongle), if it's there."""
unescape = lambda v: re.sub(r"\\(\d{3})", lambda m: chr(int(m.group(1))), v).replace("\\.", ".")
found = {}
for line in text.splitlines():
f = line.split(";")
if len(f) < 9 or f[0] != "=" or f[2] != "IPv4" or f[1].startswith("tailscale"):
continue
name = unescape(f[3])
h = found.setdefault(name, {"name": name, "address": "", "dongle": ""})
key = "dongle" if f[1] == "wlanap" else "address"
h[key] = h[key] or f[7]
out = []
for h in found.values():
if not h["address"]: # only on the hotspot
h["address"], h["dongle"] = h["dongle"], ""
out.append(h)
return sorted(out, key=lambda h: h["name"].lower())
class Backend(QObject):
changed = Signal()
busyChanged = Signal()
message = Signal(str, bool) # text, is error
monitorsReady = Signal(str, "QVariantList", str) # a host's address, its monitors, an error
dongleFound = Signal(str, str, str) # a host's name, its dongle's address ("" none), what happened
hostsFound = Signal("QVariantList") # computers on the network: [{name, address, dongle, added}]
signedIn = Signal(str, bool, str) # a host's name, whether it worked, what went wrong
_signInDone = Signal(str, str, str, str, str, str) # name, address, dongle, token, pin, error
_reached = Signal(str, bool) # a host's address, whether its Web UI answered
def __init__(self):
super().__init__()
self._sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
self._sock.bind("") # an abstract address ft-screens can reply to
self._sock.settimeout(1.0)
self._running = False
self._states = {} # remote screen number -> its stream's state ("lost can't connect")
self._online = {} # host address -> True/False (None: not checked yet)
self._queue = [] # ft-layout runs waiting their turn: (label, args)
self._proc = None
self._busy = ""
self._reached.connect(self._host_reached, Qt.QueuedConnection)
self._signInDone.connect(self._sign_in_done, Qt.QueuedConnection)
self.poll = QTimer(interval=2000, timeout=self._check)
self.poll.start()
self.ping = QTimer(interval=15000, timeout=self._ping_hosts)
self.ping.start()
self._check()
self._ping_hosts()
# --- state ---
def _ask(self, text):
"""Request/reply to ft-screens; None if it isn't running."""
try:
self._sock.sendto(text.encode(), FT_SCREENS)
return self._sock.recv(4096).decode()
except OSError:
return None
def _check(self):
running = os.path.exists(f"/run/user/{os.getuid()}/frametop/wayland-0")
states = {}
reply = self._ask("remotes") if running else None
if reply and reply.startswith("ok"):
for e in reply.split()[2:]:
n, state = int(e.split(":")[0]), e.split(":")[2]
if state in ("lost", "live"): # why ("lost can't connect"), or which way ("live via dongle")
info = self._ask(f"remote {n} info") or ""
state = info[3:].strip() if info.startswith("ok ") else state
states[n] = state
if running != self._running or states != self._states:
self._running, self._states = running, states
self.changed.emit()
def _ping_hosts(self):
for h in ft_layout.load_layout().get("hosts", []):
address = h.get("address", "")
if address:
threading.Thread(target=self._ping, args=(address,), daemon=True).start()
def _ping(self, address):
try:
with socket.create_connection((address, WEB_UI_PORT), timeout=2):
ok = True
except OSError:
ok = False
self._reached.emit(address, ok)
def _host_reached(self, address, ok):
if self._online.get(address) != ok:
self._online[address] = ok
self.changed.emit()
@Property(bool, notify=changed)
def desktopRunning(self):
return self._running
@Property(str, notify=busyChanged)
def busy(self):
return self._busy
@Property("QVariantList", constant=True)
def streamRates(self):
return STREAM_RATES
@Property("QVariantList", constant=True)
def resolutions(self):
return [{"text": f"{w} × {h}" + (f" ({t})" if t else ""), "width": w, "height": h} for w, h, t in RESOLUTIONS]
@Property("QVariantList", notify=changed)
def hosts(self):
"""Each host, whether it answers, and its displays with their streams' state."""
layout = ft_layout.load_layout()
numbers = {d["id"]: n for n, _, d in ft_layout.remote_displays(layout)}
out = []
for h in layout.get("hosts", []):
displays = []
for d in h.get("displays", []):
n = numbers.get(d.get("id"), 0)
w, hh = d.get("size", [2560, 1440])
if d.get("off"):
state = "disconnected"
elif not self._running:
state = "desktop off"
else:
state = self._states.get(n, "not running")
displays.append({"id": d["id"], "number": n, "label": d.get("label") or d["id"], "app": d["app"],
"virtual": d["app"] == "monitor", "width": int(w), "height": int(hh),
"fps": int(d.get("fps", 60)), "bitrate": int(d.get("bitrate", 0)),
"metres": ft_layout.remote_size(d)[0], "shown": not d.get("hidden"),
"connected": not d.get("off"), "state": state})
address = h.get("address", "")
out.append({"name": h.get("name", ""), "address": address, "hasToken": os.path.exists(token_path(address)),
"online": self._online.get(address), "route": h.get("route", "auto"),
"direct": ", ".join(h.get("direct", [])), "displays": displays})
return out
# --- hosts ---
@Slot()
def discoverHosts(self):
"""Vibepollo (and Sunshine) computers on the network (answer: hostsFound)."""
def work():
try:
r = subprocess.run(host_command("avahi-browse", "-rpt", "_nvstream._tcp"), capture_output=True,
text=True, timeout=15)
found = parse_avahi(r.stdout)
except (OSError, subprocess.SubprocessError):
found = []
known = {h.get("address") for h in ft_layout.load_layout().get("hosts", [])}
for h in found:
h["added"] = h["address"] in known
self.hostsFound.emit(found)
threading.Thread(target=work, daemon=True).start()
@Slot(str, str, str, str, str)
def signIn(self, name, address, dongle, user, password):
"""Asks the host for Frametop's API token with its Web UI login (answer: signedIn).
Over its dongle when it has one that answers, so the password skips the router."""
name, address, dongle = " ".join(name.split()), address.strip(), dongle.strip()
if not re.fullmatch(NAME_RE, name):
return self.signedIn.emit(name, False, "A host's name needs 1 to 40 letters, digits, spaces, dots or dashes")
if not re.fullmatch(r"[A-Za-z0-9.:-]{1,64}", address) or (dongle and not re.fullmatch(r"[0-9.]{7,15}", dongle)):
return self.signedIn.emit(name, False, "The address is a host name or an IP address")
layout = ft_layout.load_layout()
if any(h.get("name") == name and h.get("address") != address for h in layout.get("hosts", [])):
return self.signedIn.emit(name, False, f"There's already a host called {name}")
def work():
known = None
try:
with open(pin_path(address)) as f:
known = f.read().strip() or None
except OSError:
pass
error = ""
for target in ([dongle] if dongle else []) + [address]:
try:
ctx = ssl.create_default_context()
ctx.check_hostname, ctx.verify_mode = False, ssl.CERT_NONE # self-signed: pinned below
conn = http.client.HTTPSConnection(target, WEB_UI_PORT, timeout=15, context=ctx)
conn.connect()
pin = spki_pin(conn.sock.getpeercert(binary_form=True))
if known and pin != known:
conn.close()
return self._signInDone.emit(name, address, dongle, "", "", "Its certificate isn't the one "
"seen when it was added. If Vibepollo was installed again, "
"remove the host and add it again.")
basic = base64.b64encode(f"{user}:{password}".encode()).decode()
conn.request("POST", "/api/token", json.dumps({"scopes": SCOPES}),
{"Authorization": "Basic " + basic, "Content-Type": "application/json"})
r = conn.getresponse()
text = r.read().decode(errors="replace")
conn.close()
if r.status == 401:
return self._signInDone.emit(name, address, dongle, "", "", "Wrong user name or password")
token = json.loads(text).get("token") if r.status == 200 else None
if not token:
return self._signInDone.emit(name, address, dongle, "", "",
f"It didn't make a token ({r.status} {text[:120]})")
return self._signInDone.emit(name, address, dongle, token, pin, "")
except (OSError, ValueError, ssl.SSLError, http.client.HTTPException,
subprocess.SubprocessError) as e:
error = str(e) or type(e).__name__
self._signInDone.emit(name, address, dongle, "", "", f"Couldn't reach it: {error}")
threading.Thread(target=work, daemon=True).start()
def _sign_in_done(self, name, address, dongle, token, pin, error):
if error:
return self.signedIn.emit(name, False, error)
write_private(token_path(address), token)
write_private(pin_path(address), pin)
layout = ft_layout.load_layout()
host = next((h for h in layout.setdefault("hosts", []) if h.get("address") == address), None)
if host is None:
host = {"name": name, "address": address, "displays": []}
layout["hosts"].append(host)
if dongle and dongle not in host.get("direct", []):
host["direct"] = [dongle] + host.get("direct", [])
ft_layout.save_layout(layout)
self.changed.emit()
threading.Thread(target=self._ping, args=(address,), daemon=True).start()
self.signedIn.emit(host["name"], True, "")
@Slot(str)
def removeHost(self, name):
layout = ft_layout.load_layout()
host = next((h for h in layout.get("hosts", []) if h.get("name") == name), None)
if host is None:
return
if host.get("displays"):
return self.message.emit("Remove its displays first", True)
layout["hosts"].remove(host)
ft_layout.save_layout(layout)
if not any(h.get("address") == host.get("address") for h in layout["hosts"]):
for path in (token_path(host.get("address", "")), pin_path(host.get("address", ""))):
try:
os.remove(path)
except OSError:
pass
self.changed.emit()
@Slot(str, bool)
def setHostConnected(self, name, on):
"""All of a host's displays at once."""
layout = ft_layout.load_layout()
host = next((h for h in layout.get("hosts", []) if h.get("name") == name), None)
ids = [d["id"] for d in (host or {}).get("displays", []) if bool(d.get("off")) == on]
if ids:
self._run(f"{'Connecting' if on else 'Disconnecting'} {name}", "remote", "connect" if on else "disconnect", *ids)
@Slot(str, str)
def setRoute(self, name, route):
self._run(f"Connecting {name} {dict(auto='either way', network='over the network', dongle='over the dongle')[route]}",
"remote", "host", name, f"route={route}")
@Slot(str, str)
def setDirect(self, name, text):
addresses = [a for a in re.split(r"[\s,]+", text.strip()) if a]
if any(not re.fullmatch(r"[A-Za-z0-9.:-]{1,64}", a) for a in addresses):
return self.message.emit("The dongle's address is an IP address", True)
self._run(f"Setting {name}'s dongle", "remote", "host", name, "direct=" + (",".join(addresses) or "none"))
@staticmethod
def _host_id(address):
"""A host's <uniqueid> from its serverinfo (Vibepollo can take seconds to answer)."""
url = f"http://{address}:{47989}/serverinfo?uniqueid=0123456789ABCDEF&uuid={uuid.uuid4()}"
with urllib.request.urlopen(url, timeout=12) as r:
m = re.search(r"<uniqueid>([^<]+)</uniqueid>", r.read().decode(errors="replace"))
return m.group(1) if m else None
@Slot(str)
def findDongle(self, name):
"""The host among the Frame hotspot's clients (answer: dongleFound)."""
host = next((h for h in ft_layout.load_layout().get("hosts", []) if h.get("name") == name), None)
if host is None:
return
def work():
try:
want = self._host_id(host["address"])
if not want:
return self.dongleFound.emit(name, "", "it didn't say who it is at its own address")
with open("/proc/net/arp") as f: # IP, HW type, flags, MAC, mask, device
rows = [line.split() for line in f.read().splitlines()[1:]]
candidates = [r[0] for r in rows if len(r) >= 6 and r[5] == "wlanap" and r[2] != "0x0"]
for ip in candidates:
try:
if self._host_id(ip) == want:
return self.dongleFound.emit(name, ip, "")
except OSError:
continue
self.dongleFound.emit(name, "", "it isn't on the Frame's hotspot" if candidates else
"nothing is on the Frame's hotspot (is Steam Link's dongle paired?)")
except OSError as e:
self.dongleFound.emit(name, "", str(e))
threading.Thread(target=work, daemon=True).start()
# --- displays ---
@Slot(str)
def listMonitors(self, address):
"""The host's monitors, from its Web UI API (answer: monitorsReady)."""
def work():
monitors, error = [], ""
try:
code, out, err = ft_layout.run_stream("monitors", address, timeout=30)
if code:
error = (err or out).strip().splitlines()[-1] if (err or out).strip() else f"exit {code}"
else:
for m in json.loads(out[out.index("["):]):
info = m.get("info") or {}
res = info.get("resolution") or {}
virtual = (m.get("edid") or {}).get("manufacturer_id") == "SDD" or \
str(m.get("friendly_name", "")).startswith("frametop")
monitors.append({"app": "display:" + m.get("device_id", ""),
"name": m.get("friendly_name") or m.get("display_name") or "?",
"width": int(res.get("width", 0)), "height": int(res.get("height", 0)),
"primary": bool(info.get("primary")), "active": bool(info),
"virtual": virtual})
except Exception as e: # noqa: BLE001 (shown to the user, not raised in a thread)
error = str(e)
self.monitorsReady.emit(address, monitors, error)
threading.Thread(target=work, daemon=True).start()
@Slot(str, str, str, int, int, int, int)
def addDisplay(self, host, app, label, width, height, fps, bitrate):
"""Pairs a client for it with the host's token and starts its stream (ft-layout)."""
layout = ft_layout.load_layout()
h = next((x for x in layout.get("hosts", []) if x.get("name") == host), None)
if h is None:
return self.message.emit(f"No host called {host}", True)
label = " ".join(label.split()) or ("Virtual" if app == "monitor" else "Display")
self._run(f"Adding {label} from {host}", "remote", "add", host, h["address"], app, "--label", label,
"--size", f"{width}x{height}", "--fps", str(fps), "--bitrate", str(bitrate))
@Slot(str, bool)
def setConnected(self, display_id, on):
self._run("Connecting" if on else "Disconnecting", "remote", "connect" if on else "disconnect", display_id)
@Slot(str, int, int, int, int)
def setStream(self, display_id, width, height, fps, bitrate):
"""Its stream's resolution, frame rate and bitrate (it starts over)."""
self._run("Changing the stream", "remote", "set", display_id, f"size={width}x{height}", f"fps={fps}",
f"bitrate={bitrate}")
def _edit(self, display_id, fn):
layout = ft_layout.load_layout()
n, _, d = ft_layout.find_display(layout, display_id)
fn(d)
ft_layout.save_layout(layout)
self.changed.emit()
return n, d
@Slot(str, float)
def setMetres(self, display_id, m):
n, d = self._edit(display_id, lambda d: d.__setitem__("metres", round(m, 3)))
if self._running and not d.get("off"):
self._ask(f"width {n} {m:.3f}")
@Slot(str, bool)
def setShown(self, display_id, shown):
n, d = self._edit(display_id, lambda d: d.pop("hidden", None) if shown else d.__setitem__("hidden", True))
if self._running and not d.get("off"):
self._ask(f"{'reveal' if shown else 'conceal'} {n}")
@Slot(str)
def removeDisplay(self, display_id):
self._run("Removing the display", "remote", "remove", display_id)
# --- ft-layout on the host, one at a time ---
def _run(self, label, *args):
self._queue.append((label, args))
if self._proc is None:
self._next()
def _next(self):
if not self._queue:
return
label, args = self._queue.pop(0)
self._busy = label
self.busyChanged.emit()
proc = QProcess(self)
proc.setProcessChannelMode(QProcess.MergedChannels)
argv = host_command(os.path.abspath(FT_LAYOUT), *args)
proc.finished.connect(lambda code, _status: self._done(proc, label, code))
self._proc = proc
proc.start(argv[0], argv[1:])
def _done(self, proc, label, code):
out = bytes(proc.readAllStandardOutput()).decode(errors="replace").strip()
self._proc = None
self._busy = ""
self.busyChanged.emit()
self.changed.emit()
self._check()
last = out.splitlines()[-1] if out else ""
if code == 0:
self.message.emit(f"{label}: done" + (f" ({last})" if last and not last.startswith("ok") else ""), False)
else:
self.message.emit(f"{label} failed: {last or 'exit code ' + str(code)}", True)
self._next()
def main():
app = QGuiApplication(sys.argv)
app.setApplicationName("ft-remote-displays")
app.setApplicationDisplayName("Frametop Remote Displays")
app.setDesktopFileName("ft-remote-displays")
if not QIcon.themeName():
QIcon.setThemeName("breeze")
QQuickStyle.setStyle("org.kde.desktop")
engine = QQmlApplicationEngine()
backend = Backend()
engine.rootContext().setContextProperty("backend", backend)
engine.load(QUrl.fromLocalFile(os.path.join(HERE, "main.qml")))
if not engine.rootObjects():
sys.exit(1)
sys.exit(app.exec())
if __name__ == "__main__":
main()
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
# Install (or remove) Frametop Remote Displays' menu entry on the Frame.
# Usage: remote-displays/install.sh [install|uninstall]
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
. "$root/scripts/_env.sh"
"$root/scripts/sync.sh" >/dev/null
apps=.local/share/applications
case ${1:-install} in
install)
fill_template "$root/remote-displays/ft-remote-displays.desktop" | on_frame "mkdir -p ~/$apps && cat > ~/$apps/ft-remote-displays.desktop"
on_frame "chmod +x remote-displays/ft-remote-displays"
echo "installed: Frametop Remote Displays" ;;
uninstall)
on_frame "rm -f ~/$apps/ft-remote-displays.desktop; echo removed" ;;
*) echo "usage: $0 [install|uninstall]" >&2; exit 2 ;;
esac
+575
View File
@@ -0,0 +1,575 @@
// Frametop Remote Displays (Kirigami). Backend: ft_remote_displays.py ("backend").
import QtQuick
import QtQuick.Controls as Controls
import QtQuick.Layouts
import org.kde.kirigami as Kirigami
Kirigami.ApplicationWindow {
id: root
title: "Frametop Remote Displays"
width: Kirigami.Units.gridUnit * 42
height: Kirigami.Units.gridUnit * 34
pageStack.initialPage: hostsPage
Connections {
target: backend
function onMessage(text, isError) {
root.showPassiveNotification(text, isError ? "long" : "short")
}
}
// What a stream is doing, in words (ft-screens' state, or ours for a disconnected one).
function stateText(state) {
const words = { live: "streaming", connecting: "connecting", starting: "starting", queued: "waiting its turn",
lost: "lost, trying again", disconnected: "disconnected",
"desktop off": "connects when the Frametop desktop starts", "not running": "not running" }
if (state in words) return words[state]
if (state.startsWith("live via ")) return "streaming over the " + state.slice(9)
if (state.startsWith("lost ")) return "lost (" + state.slice(5) + "), trying again"
return state
}
function stateColor(state) {
if (state.startsWith("live")) return Kirigami.Theme.positiveTextColor
if (state.startsWith("lost")) return Kirigami.Theme.negativeTextColor
if (state === "connecting" || state === "starting" || state === "queued") return Kirigami.Theme.neutralTextColor
return Kirigami.Theme.disabledTextColor
}
footer: Controls.ToolBar {
visible: backend.busy !== ""
RowLayout {
anchors.fill: parent
Controls.BusyIndicator { running: backend.busy !== ""; Layout.preferredHeight: Kirigami.Units.iconSizes.medium }
Controls.Label { text: backend.busy + "…"; Layout.fillWidth: true }
}
}
// ---------------------------------------------------------------- dialogs
// A host: picked from the computers found on the network (or typed in), then a sign-in
// with its Vibepollo Web UI login, for Frametop's own API token. Also signs in again to
// one already added (openFor).
Kirigami.PromptDialog {
id: hostDialog
objectName: "hostDialog"
title: existing ? "Sign in to " + fixedName : "Add a computer"
standardButtons: Kirigami.Dialog.NoButton
property bool existing: false
property string fixedName: ""
property string fixedAddress: ""
property string fixedDongle: ""
property var found: []
property bool searching: false
property bool working: false
property string error: ""
property int picked: -1 // index in found; found.length: typed in
readonly property bool typed: picked === found.length
readonly property string pickedName: existing ? fixedName
: typed ? (otherName.text.trim() || otherAddress.text.trim()) : picked >= 0 ? found[picked].name : ""
readonly property string pickedAddress: existing ? fixedAddress
: typed ? otherAddress.text.trim() : picked >= 0 ? found[picked].address : ""
readonly property string pickedDongle: existing ? fixedDongle : !typed && picked >= 0 ? found[picked].dongle : ""
readonly property bool ok: pickedName !== "" && pickedAddress !== "" && user.text !== "" && password.text !== "" && !working
function look() {
searching = true
backend.discoverHosts()
}
function openNew() {
existing = false; found = []; picked = -1; error = ""; working = false
otherName.text = ""; otherAddress.text = ""; password.text = ""
open()
look()
}
function openFor(h) {
existing = true; fixedName = h.name; fixedAddress = h.address; fixedDongle = h.direct.split(",")[0].trim()
error = ""; working = false; password.text = ""
open()
user.forceActiveFocus()
}
function accept() {
if (!ok) return
working = true; error = ""
backend.signIn(pickedName, pickedAddress, pickedDongle, user.text, password.text)
}
Connections {
target: backend
function onHostsFound(list) {
hostDialog.searching = false
hostDialog.found = list
if (hostDialog.picked < 0 || hostDialog.picked > list.length) {
const free = list.findIndex(h => !h.added)
hostDialog.picked = free >= 0 ? free : list.length
}
}
function onSignedIn(name, ok, why) {
if (!hostDialog.working) return
hostDialog.working = false
if (!ok) {
hostDialog.error = why
return
}
password.text = ""
hostDialog.close()
root.showPassiveNotification("Signed in to " + name)
const h = backend.hosts.find(x => x.name === name)
if (h && !hostDialog.existing) displayDialog.openFor(h)
}
}
ColumnLayout {
RowLayout {
visible: !hostDialog.existing
Controls.Label { text: "Computers running Vibepollo on your network:"; Layout.fillWidth: true }
Controls.ToolButton {
icon.name: "view-refresh"
text: "Look again"
display: Controls.AbstractButton.IconOnly
enabled: !hostDialog.searching
onClicked: hostDialog.look()
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
}
}
Controls.BusyIndicator { visible: hostDialog.searching; running: visible; Layout.alignment: Qt.AlignHCenter }
Repeater {
model: hostDialog.existing ? [] : hostDialog.found
Controls.RadioButton {
required property var modelData
required property int index
enabled: !modelData.added
text: modelData.name + " (" + modelData.address + (modelData.dongle ? ", and on the dongle" : "") + ")"
+ (modelData.added ? ": added" : "")
checked: hostDialog.picked === index
onToggled: if (checked) hostDialog.picked = index
}
}
Controls.RadioButton {
visible: !hostDialog.existing && !hostDialog.searching
text: hostDialog.found.length ? "Another one, by its address" : "None found: type its address"
checked: hostDialog.typed
onToggled: if (checked) hostDialog.picked = hostDialog.found.length
}
Kirigami.FormLayout {
Layout.fillWidth: true
visible: hostDialog.typed && !hostDialog.existing
Controls.TextField { id: otherAddress; Kirigami.FormData.label: "Address:"; placeholderText: "192.168.1.20" }
Controls.TextField { id: otherName; Kirigami.FormData.label: "Name:"; placeholderText: "My PC"; maximumLength: 40 }
}
Controls.Label {
Layout.fillWidth: true
Layout.topMargin: Kirigami.Units.largeSpacing
wrapMode: Text.Wrap
text: "Sign in with its Vibepollo Web UI user name and password. Frametop keeps a token that can only pair its displays, set their permissions and list the monitors; the password isn't kept."
}
Kirigami.FormLayout {
Layout.fillWidth: true
Controls.TextField { id: user; Kirigami.FormData.label: "User name:" }
Controls.TextField {
id: password
Kirigami.FormData.label: "Password:"
echoMode: TextInput.Password
onAccepted: hostDialog.accept()
}
}
Controls.BusyIndicator { visible: hostDialog.working; running: visible; Layout.alignment: Qt.AlignHCenter }
Controls.Label {
visible: hostDialog.error !== ""
Layout.fillWidth: true
wrapMode: Text.Wrap
color: Kirigami.Theme.negativeTextColor
text: hostDialog.error
}
}
customFooterActions: [
Kirigami.Action { text: "Sign in"; icon.name: "go-next"; enabled: hostDialog.ok; onTriggered: hostDialog.accept() },
Kirigami.Action { text: "Cancel"; icon.name: "dialog-cancel"; onTriggered: { password.text = ""; hostDialog.close() } }
]
}
// A host's displays to add: its monitors as they are (all of them ticked, but the ones
// already added), and a virtual one at any size.
Kirigami.PromptDialog {
id: displayDialog
objectName: "displayDialog"
title: "Add displays from " + host
standardButtons: Kirigami.Dialog.NoButton
property string host: ""
property string address: ""
property var added: [] // what its displays stream (app), so a monitor isn't added twice
property var monitors: []
property var chosen: ({}) // app -> ticked
property bool virtualChosen: false
property string error: ""
property bool loading: false
readonly property int count: monitors.filter(m => chosen[m.app]).length + (virtualChosen ? 1 : 0)
function openFor(h) {
host = h.name; address = h.address; added = h.displays.map(d => d.app)
monitors = []; chosen = ({}); virtualChosen = false; error = ""; loading = true
open()
backend.listMonitors(address)
}
function tick(app, on) {
const c = Object.assign({}, chosen)
c[app] = on
chosen = c
}
function accept() {
if (!count) return
close()
const fps = displayRate.currentValue, kbps = Math.round(displayBitrate.value * 1000)
for (const m of monitors)
if (chosen[m.app]) backend.addDisplay(host, m.app, m.name, m.width, m.height, fps, kbps)
if (virtualChosen) {
const r = backend.resolutions[displayRes.currentIndex]
backend.addDisplay(host, "monitor", "Virtual", r.width, r.height, fps, kbps)
}
}
Connections {
target: backend
function onMonitorsReady(address, monitors, error) {
if (address !== displayDialog.address) return
displayDialog.loading = false
displayDialog.monitors = monitors.filter(m => !m.virtual && m.active)
const c = {}
for (const m of displayDialog.monitors) c[m.app] = !displayDialog.added.includes(m.app)
displayDialog.chosen = c
displayDialog.error = error
}
}
ColumnLayout {
Controls.BusyIndicator { visible: displayDialog.loading; running: visible; Layout.alignment: Qt.AlignHCenter }
Controls.Label {
visible: displayDialog.error !== ""
Layout.fillWidth: true
wrapMode: Text.Wrap
text: "Couldn't list its monitors: " + displayDialog.error
}
Repeater {
model: displayDialog.monitors
Controls.CheckBox {
required property var modelData
readonly property bool already: displayDialog.added.includes(modelData.app)
enabled: !already
text: modelData.name + " (" + modelData.width + " × " + modelData.height + (modelData.primary ? ", main" : "") + ")"
+ (already ? ": added" : "")
checked: displayDialog.chosen[modelData.app] === true
onToggled: displayDialog.tick(modelData.app, checked)
}
}
Controls.CheckBox {
visible: !displayDialog.loading
text: "A virtual display (the host makes a new monitor at the size you pick)"
checked: displayDialog.virtualChosen
onToggled: displayDialog.virtualChosen = checked
}
Kirigami.FormLayout {
Layout.fillWidth: true
visible: displayDialog.count > 0
Controls.ComboBox {
id: displayRes
Kirigami.FormData.label: "Virtual display:"
visible: displayDialog.virtualChosen
model: backend.resolutions
textRole: "text"
Component.onCompleted: currentIndex = Math.max(0, backend.resolutions.findIndex(r => r.width === 2560 && r.height === 1440))
}
Controls.ComboBox {
id: displayRate
Kirigami.FormData.label: "Frame rate:"
model: backend.streamRates.map(r => ({ text: r + " fps", value: r }))
textRole: "text"
valueRole: "value"
Component.onCompleted: currentIndex = Math.max(0, indexOfValue(60))
}
RowLayout {
Kirigami.FormData.label: "Bitrate:"
Controls.SpinBox { id: displayBitrate; from: 0; to: 150; stepSize: 5; value: 0; editable: true }
Controls.Label { text: displayBitrate.value === 0 ? "Mbit/s (0: by size and rate)" : "Mbit/s" }
}
}
}
customFooterActions: [
Kirigami.Action {
text: displayDialog.count > 1 ? "Add " + displayDialog.count : "Add"
icon.name: "list-add"
enabled: displayDialog.count > 0
onTriggered: displayDialog.accept()
},
Kirigami.Action { text: "Later"; icon.name: "dialog-cancel"; onTriggered: displayDialog.close() }
]
}
// ---------------------------------------------------------------- hosts and their displays
Component {
id: hostsPage
Kirigami.ScrollablePage {
title: "Remote displays"
actions: [
Kirigami.Action {
text: "Add computer"
icon.name: "list-add"
onTriggered: hostDialog.openNew()
}
]
header: Kirigami.InlineMessage {
position: Kirigami.InlineMessage.Position.Header
visible: true
type: backend.desktopRunning ? Kirigami.MessageType.Information : Kirigami.MessageType.Warning
text: !backend.desktopRunning
? "The Frametop desktop isn't running. The connected displays start with it."
: backend.hosts.length === 0
? "Another computer's monitors as Frametop screens, with the screens' controls, in your layouts and profiles. The computer runs Vibepollo: add it, sign in with its Web UI login, then pick its displays."
: "Each display is a panel like a screen: move, size, curve and pin it in VR, and Display Settings' Save as profile keeps where it is. A disconnected display keeps its place and settings."
}
ColumnLayout {
spacing: Kirigami.Units.largeSpacing
Repeater {
model: backend.hosts
delegate: Kirigami.AbstractCard {
id: hostCard
required property var modelData
readonly property bool anyConnected: modelData.displays.some(d => d.connected)
Layout.fillWidth: true
contentItem: ColumnLayout {
RowLayout {
Kirigami.Icon {
source: "computer"
Layout.preferredWidth: Kirigami.Units.iconSizes.medium
Layout.preferredHeight: Kirigami.Units.iconSizes.medium
}
ColumnLayout {
spacing: 0
Kirigami.Heading { level: 3; text: hostCard.modelData.name }
Controls.Label {
text: hostCard.modelData.address + " · "
+ (hostCard.modelData.online === true ? "online"
: hostCard.modelData.online === false ? "not answering" : "checking…")
+ (hostCard.modelData.hasToken ? "" : " · not signed in")
color: hostCard.modelData.online === false ? Kirigami.Theme.negativeTextColor : Kirigami.Theme.textColor
opacity: hostCard.modelData.online === false ? 1 : 0.7
}
}
Item { Layout.fillWidth: true }
Controls.Switch {
text: "Connected"
visible: hostCard.modelData.displays.length > 0
checked: hostCard.anyConnected
onToggled: backend.setHostConnected(hostCard.modelData.name, checked)
Controls.ToolTip.text: "Connect or disconnect all of its displays"
Controls.ToolTip.visible: hovered
}
Controls.Button {
visible: hostCard.modelData.hasToken
text: "Add displays"
icon.name: "list-add"
onClicked: displayDialog.openFor(hostCard.modelData)
}
Controls.Button {
visible: !hostCard.modelData.hasToken
text: "Sign in"
icon.name: "unlock"
onClicked: hostDialog.openFor(hostCard.modelData)
}
Controls.ToolButton {
visible: hostCard.modelData.hasToken
icon.name: "unlock"
display: Controls.AbstractButton.IconOnly
text: "Sign in again"
Controls.ToolTip.text: "Sign in again: a new token replaces the one kept here (revoke the old one in its Web UI, API Tokens)"
Controls.ToolTip.visible: hovered
onClicked: hostDialog.openFor(hostCard.modelData)
}
Controls.ToolButton {
icon.name: "edit-delete-remove"
display: Controls.AbstractButton.IconOnly
enabled: hostCard.modelData.displays.length === 0
text: "Remove this host"
Controls.ToolTip.text: enabled ? text : "Remove its displays first"
Controls.ToolTip.visible: hovered
onClicked: backend.removeHost(hostCard.modelData.name)
}
}
// How its streams reach it: its Steam Link dongle on the Frame's hotspot
// (no router in the way) or the home network. Without a dongle, the
// network, and Find to look for one.
RowLayout {
id: route
property string finding: ""
readonly property bool hasDongle: hostCard.modelData.direct !== ""
Controls.Label { text: route.hasDongle ? "Connection:" : "Connection: the network" }
Controls.ComboBox {
visible: route.hasDongle
model: [{ text: "Auto (dongle when it's up)", value: "auto" },
{ text: "Network only", value: "network" },
{ text: "Dongle only", value: "dongle" }]
textRole: "text"
valueRole: "value"
currentIndex: Math.max(0, indexOfValue(hostCard.modelData.route))
onActivated: if (currentValue !== hostCard.modelData.route) backend.setRoute(hostCard.modelData.name, currentValue)
Controls.ToolTip.text: "Changing it starts the host's streams over (a virtual display is made again, so its windows move)"
Controls.ToolTip.visible: hovered
}
Controls.Label { visible: route.hasDongle; text: "Dongle:" }
Controls.TextField {
id: dongle
visible: route.hasDongle
text: hostCard.modelData.direct
placeholderText: "not found yet"
Layout.preferredWidth: Kirigami.Units.gridUnit * 7
onEditingFinished: if (text !== hostCard.modelData.direct) backend.setDirect(hostCard.modelData.name, text)
}
Controls.Button {
text: route.finding === "…" ? "Looking…" : route.hasDongle ? "Find" : "Find a dongle"
icon.name: "edit-find"
flat: !route.hasDongle
enabled: route.finding !== "…"
onClicked: { route.finding = "…"; backend.findDongle(hostCard.modelData.name) }
Controls.ToolTip.text: "Look for this computer on the Frame's hotspot (Steam Link's dongle)"
Controls.ToolTip.visible: hovered
}
Controls.Label {
visible: route.finding !== "" && route.finding !== "…"
text: route.finding
opacity: 0.7
Layout.fillWidth: true
elide: Text.ElideRight
}
Connections {
target: backend
function onDongleFound(name, address, why) {
if (name !== hostCard.modelData.name) return
route.finding = address ? "found " + address : "not found: " + why
if (address && address !== hostCard.modelData.direct) backend.setDirect(name, address)
}
}
}
Controls.Label {
visible: hostCard.modelData.displays.length === 0
text: "No displays yet."
opacity: 0.7
}
Repeater {
model: hostCard.modelData.displays
delegate: ColumnLayout {
id: disp
required property var modelData
property bool expanded: false
Layout.fillWidth: true
Kirigami.Separator { Layout.fillWidth: true }
RowLayout {
Kirigami.Icon {
source: disp.modelData.virtual ? "video-display-symbolic" : "video-display"
Layout.preferredWidth: Kirigami.Units.iconSizes.smallMedium
Layout.preferredHeight: Kirigami.Units.iconSizes.smallMedium
}
ColumnLayout {
spacing: 0
Kirigami.Heading { level: 4; text: disp.modelData.label }
Controls.Label {
text: (disp.modelData.virtual ? "virtual display" : "monitor") + ", "
+ disp.modelData.width + " × " + disp.modelData.height + " at " + disp.modelData.fps
+ " fps, screen " + disp.modelData.number
opacity: 0.7
}
Controls.Label {
text: root.stateText(disp.modelData.state)
color: root.stateColor(disp.modelData.state)
}
}
Item { Layout.fillWidth: true }
Controls.Switch {
text: "Connected"
checked: disp.modelData.connected
onToggled: backend.setConnected(disp.modelData.id, checked)
}
Controls.Switch {
text: "Shown"
enabled: disp.modelData.connected
checked: disp.modelData.shown
onToggled: backend.setShown(disp.modelData.id, checked)
Controls.ToolTip.text: "Hide its panel in VR; the stream keeps going, slowly"
Controls.ToolTip.visible: hovered
}
Controls.ToolButton {
icon.name: disp.expanded ? "go-up" : "configure"
display: Controls.AbstractButton.IconOnly
text: disp.expanded ? "Hide its settings" : "Its stream and size"
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
onClicked: disp.expanded = !disp.expanded
}
Controls.ToolButton {
icon.name: "edit-delete-remove"
display: Controls.AbstractButton.IconOnly
text: "Remove this display"
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
onClicked: backend.removeDisplay(disp.modelData.id)
}
}
Kirigami.FormLayout {
Layout.fillWidth: true
visible: disp.expanded
RowLayout {
Kirigami.FormData.label: "Stream:"
Controls.SpinBox {
id: sw
from: 640; to: 7680; stepSize: 8; editable: true
value: disp.modelData.width
}
Controls.Label { text: "×" }
Controls.SpinBox {
id: sh
from: 360; to: 4320; stepSize: 8; editable: true
value: disp.modelData.height
}
Controls.ComboBox {
id: sr
model: backend.streamRates.map(r => ({ text: r + " fps", value: r }))
textRole: "text"
valueRole: "value"
Component.onCompleted: currentIndex = Math.max(0, indexOfValue(disp.modelData.fps))
}
Controls.SpinBox {
id: sb
from: 0; to: 150; stepSize: 5; editable: true
value: Math.round(disp.modelData.bitrate / 1000)
}
Controls.Label { text: sb.value === 0 ? "Mbit/s (auto)" : "Mbit/s" }
Controls.Button {
text: "Apply"
enabled: sw.value !== disp.modelData.width || sh.value !== disp.modelData.height
|| sr.currentValue !== disp.modelData.fps || sb.value * 1000 !== disp.modelData.bitrate
onClicked: backend.setStream(disp.modelData.id, sw.value, sh.value, sr.currentValue, sb.value * 1000)
}
}
Controls.Label {
visible: !disp.modelData.virtual
Layout.fillWidth: true
wrapMode: Text.Wrap
opacity: 0.7
text: "A monitor is scaled to the stream's size; its own resolution stays as it is."
}
RowLayout {
Kirigami.FormData.label: "Width in VR:"
Controls.Slider {
id: rm
from: 0.3; to: 6.0; stepSize: 0.05
value: disp.modelData.metres
Layout.preferredWidth: Kirigami.Units.gridUnit * 12
onMoved: backend.setMetres(disp.modelData.id, value)
}
Controls.Label {
text: rm.value.toFixed(2) + " m wide, "
+ (rm.value * disp.modelData.height / disp.modelData.width).toFixed(2) + " m tall"
}
}
}
}
}
}
}
}
}
}
}
}
+9 -8
View File
@@ -1,28 +1,29 @@
#!/usr/bin/env bash
# Build ft-screens in the dev container on the Frame (screens/build/ft-screens), and
# ft-handtest, which tries the hand cutouts (handcut.cpp) on a test panel of its own.
# compositor.c is the wlroots side (C; wlroots headers aren't C++), vr.cpp the OpenVR side.
# compositor.c is the wlroots side (C; wlroots headers aren't C++), vr.cpp the OpenVR side,
# remote.c the remote screens (their streams are ft-stream's, stream/build.sh).
# vr.cpp needs OpenVR's IVRIPCResourceManagerClient (ImportDmabuf), which the header
# shipped with SteamVR on the Frame predates, so the build uses the public header from
# Valve's openvr repo (pinned; the Frame's runtime supports its interface versions).
# Valve's openvr repo (pinned in scripts/openvr.sh).
# keyboard.cpp draws its key labels with stb_truetype (public domain, one header, pinned).
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C screens 'set -e; mkdir -p build/include
openvr=v2.15.6
[ -f build/include/openvr-$openvr ] || { curl -fsSL "https://raw.githubusercontent.com/ValveSoftware/openvr/$openvr/headers/openvr.h" -o build/include/openvr.h && touch build/include/openvr-$openvr; }
. ../scripts/openvr.sh
stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
[ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
gcc -std=c11 -O2 -Wall -Wno-unused-parameter -c -o build/compositor.o compositor.c \
$(pkg-config --cflags wlroots-0.20 wayland-server xkbcommon libdrm pixman-1)
cc="gcc -std=c11 -O2 -Wall -Wno-unused-parameter $(pkg-config --cflags wlroots-0.20 wayland-server xkbcommon libdrm pixman-1)"
$cc -c -o build/compositor.o compositor.c
$cc -c -o build/remote.o remote.c
cxx="g++ -std=c++17 -O2 -Wall -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags egl glesv2 gbm libdrm)"
$cxx -c -o build/vr.o vr.cpp
$cxx -c -o build/keyboard.o keyboard.cpp
$cxx -c -o build/handcut.o handcut.cpp
$cxx -c -o build/handtest.o handtest.cpp
vrlibs="$(pkg-config --libs egl glesv2 gbm) -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64"
g++ -o build/ft-screens build/compositor.o build/vr.o build/keyboard.o build/handcut.o \
vrlibs="$(pkg-config --libs egl glesv2 gbm) $OPENVR_LIBS"
g++ -o build/ft-screens build/compositor.o build/remote.o build/vr.o build/keyboard.o build/handcut.o \
$(pkg-config --libs wlroots-0.20 wayland-server xkbcommon pixman-1) $vrlibs
g++ -o build/ft-handtest build/handtest.o build/handcut.o $vrlibs
echo "built build/ft-screens build/ft-handtest"'
+70 -9
View File
@@ -12,13 +12,16 @@
// buffers are in pixels, so a panel position in pixels is divided by the screen's KWin
// scale first ("scale <screen> <s>", from ft-layout).
//
// Usage: ft-screens [--socket NAME] [--control NAME] [--no-vr] [--screen WxH@METRES]...
// Usage: ft-screens [--socket NAME] [--control NAME] [--no-vr] [--beside] [--screen WxH@METRES]...
// [--spares N] [--rates F,V,H] [-- COMMAND ARGS...]
// --socket Wayland socket name in $XDG_RUNTIME_DIR (default ft-screens-0)
// --control the control socket's abstract name (default ft_screens)
// --no-vr run without SteamVR, for tests next to the running desktop: no panels, no
// input, and nothing sent to the input relay. Commands still work, and
// "toplevels" shows what KWin opened.
// --beside with SteamVR, next to the running desktop, for remote screens (remote.c):
// nothing goes to the input relay, ft-floatd or ft-layout. Use other --socket
// and --control names, and no COMMAND.
// --screen one per screen, in KWin's order (default: 3440x1440@2.4)
// --spares KWin's outputs after the screens: spares for floating windows (ft-floatd
// turns them on and sizes them; see docs/floating-windows.md)
@@ -63,6 +66,7 @@
#include "vr.h"
#include "controller-click.h"
#include "remote.h"
#include "relay-buttons.h"
#define MAX_SCREENS 24 // screens and spare outputs
@@ -138,6 +142,7 @@ struct server {
// panel. The input relay grabs the keyboards while it's the screens (see keys_update).
bool keys_clicked; // the last click was on a screen
bool keys_desktop; // ...and the screens are showing: typing goes to the desktop
int key_remote; // the last click was on this remote screen (index): typing goes to it; -1 not
int relay_fd; // unbound, so the relay can't reply into our control socket
uint32_t relay_sent; // when the relay last heard from us (ms)
unsigned ticks;
@@ -146,6 +151,7 @@ struct server {
bool kb_auto; // opened for a focused text field (not by a button)
unsigned kb_close_at; // ticks: close it then (a text field lost focus), 0 not
bool vr; // connected to SteamVR (not --no-vr)
bool beside; // a second instance next to the desktop (--beside): remote screens only
};
static uint32_t now_ms(void) {
@@ -330,6 +336,13 @@ static void new_decoration(struct wl_listener *l, void *data) {
static void panel_key(struct server *s, uint32_t code, bool pressed);
// Typing goes to remote screen `index` (or to the desktop: -1). The one it leaves lets go of
// the keys it held.
static void set_key_remote(struct server *s, int index) {
if (s->key_remote >= 0 && s->key_remote != index) ft_remote_blur(s->key_remote);
s->key_remote = index;
}
static void handle_vr_event(const struct ft_event *e, void *data) {
struct server *s = data;
if (e->type == FT_QUIT) {
@@ -346,6 +359,16 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
s->kb_close_at = 0;
return;
}
if (ft_remote_is(e->screen)) {
// Another machine's display: its stream takes the input, and a click or a spin to it
// takes the typing there too.
if ((e->type == FT_BUTTON && e->pressed) || e->type == FT_FRONT) {
set_key_remote(s, e->screen);
s->keys_clicked = true;
}
ft_remote_event(e);
return;
}
if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
struct ft_event filtered = *e;
if (e->screen < s->n_config &&
@@ -359,6 +382,8 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
// and ft-floatd makes its window (or the top one on a screen) KWin's active window.
wlr_seat_keyboard_notify_enter(s->seat, surface, NULL, 0, NULL);
s->keys_clicked = true;
set_key_remote(s, -1);
if (s->beside) return;
char msg[32];
snprintf(msg, sizeof msg, "front %d", e->screen + 1);
struct sockaddr_un addr = {.sun_family = AF_UNIX};
@@ -390,6 +415,7 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
if (e->pressed) {
wlr_seat_keyboard_notify_enter(s->seat, surface, NULL, 0, NULL);
s->keys_clicked = true;
set_key_remote(s, -1);
}
}
break;
@@ -434,7 +460,7 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
// gamescope's focus) doesn't get the keys too. Without word from us for a few seconds,
// the relay gives the keyboards back, so a closed desktop doesn't keep them.
static void keys_update(struct server *s) {
if (!s->vr) return; // a test instance leaves the running desktop's keyboards alone
if (!s->vr || s->beside) return; // a test instance leaves the running desktop's keyboards alone
const bool desktop = s->keys_clicked && ft_vr_screens_shown();
const uint32_t t = now_ms();
if (desktop == s->keys_desktop && t - s->relay_sent < 1000) return;
@@ -508,6 +534,7 @@ static int tick(int fd, uint32_t mask, void *data) {
const ssize_t got = read(fd, &expirations, sizeof expirations); // clears it; how many doesn't matter
(void)got;
ft_vr_poll(handle_vr_event, s);
ft_remote_tick();
if (s->relay_buttons.held && !relay_can_press(s))
release_relay_buttons(s, ft_vr_paused() ? "paused" : "its screen hid");
if (++s->ticks % 9 == 0) keys_update(s);
@@ -539,7 +566,9 @@ static int child_exited(int sig, void *data) {
int status;
pid_t pid;
while ((pid = waitpid(-1, &status, WNOHANG)) > 0)
if (pid == s->child) {
if (ft_remote_child(pid, status)) {
continue;
} else if (pid == s->child) {
wlr_log(WLR_INFO, "session exited");
wl_display_terminate(s->display);
}
@@ -607,6 +636,11 @@ static void handle_key(struct server *s, uint32_t code, int value, char *reply,
return;
}
if (kind == FT_RELAY_DROP) return (void)snprintf(reply, size, "ok not a key or mouse button");
// A remote screen has the typing, except for the release of a key the desktop holds.
if (s->key_remote >= 0 && ft_remote_is(s->key_remote) && (value || !key_held(&s->keyboard, code))) {
ft_remote_key(s->key_remote, code, value != 0);
return (void)snprintf(reply, size, "ok remote");
}
if (value || !key_held(&s->keyboard, code)) {
if (!s->seat->keyboard_state.focused_surface) return (void)snprintf(reply, size, "ok no focus");
if (!s->keys_desktop) return (void)snprintf(reply, size, "ok typing goes to Steam");
@@ -652,9 +686,15 @@ static void keyboard_command(struct server *s, const char *what, char *reply, in
return (void)snprintf(reply, size, "ok closed");
}
if (open) return (void)snprintf(reply, size, "ok open");
if (!ft_vr_screens_shown()) return (void)snprintf(reply, size, "ok screens hidden");
if (!ft_vr_screens_shown()) {
wlr_log(WLR_INFO, "keyboard not opened (%s): the screens are hidden", toggle ? "button" : "text field");
return (void)snprintf(reply, size, "ok screens hidden");
}
const int screen = focused_screen(s);
if (screen < 0 || !ft_vr_keyboard_show(screen)) return (void)snprintf(reply, size, "error not shown");
if (screen < 0 || !ft_vr_keyboard_show(screen)) {
wlr_log(WLR_INFO, "keyboard not opened (%s) for screen %d", toggle ? "button" : "text field", screen + 1);
return (void)snprintf(reply, size, "error not shown");
}
wlr_log(WLR_INFO, "keyboard open for screen %d (%s)", screen + 1, toggle ? "button" : "text field");
s->kb_screen = screen;
s->kb_auto = !toggle;
@@ -664,6 +704,10 @@ static void keyboard_command(struct server *s, const char *what, char *reply, in
// A key from our keyboard, for the focused screen. Its release always goes through, so
// no key stays held.
static void panel_key(struct server *s, uint32_t code, bool pressed) {
if (s->key_remote >= 0 && ft_remote_is(s->key_remote) && (pressed || !key_held(&s->keyboard, code))) {
ft_remote_key(s->key_remote, code, pressed);
return;
}
if (pressed ? s->seat->keyboard_state.focused_surface != NULL : key_held(&s->keyboard, code))
send_key(s, code, pressed);
}
@@ -739,7 +783,7 @@ static int control_readable(int fd, uint32_t mask, void *data) {
else if (got >= 4 && (strcmp(what, "down") == 0 || strcmp(what, "up") == 0))
e.type = FT_BUTTON, e.pressed = what[0] == 'd';
else index = 0;
if (index < 1 || index > MAX_SCREENS || !s->screens[index - 1]) {
if ((index < 1 || index > MAX_SCREENS || !s->screens[index - 1]) && !ft_remote_is(index - 1)) {
snprintf(reply, sizeof reply, "error input <screen> move|down|up|leave [x y [button]]");
} else {
handle_vr_event(&e, s);
@@ -793,7 +837,15 @@ static int control_readable(int fd, uint32_t mask, void *data) {
if (strcmp(buf + 6, "-") != 0 && strncmp(buf + 6, "frametop.", 9) != 0) s->keys_clicked = false;
len = sizeof from;
continue;
} else {
} else if (strcmp(buf, "debug") == 0) {
// ft-screens' side of scripts/report.sh's debug line; vr.cpp adds SteamVR's.
char vr[1024] = "ok vr=off";
if (s->vr) ft_vr_command("debug", vr, sizeof vr);
snprintf(reply, sizeof reply, "%s kb_screen=%d kb_auto=%d typing=%s focused_screen=%d pointer_screen=%d",
vr, s->kb_screen + 1, s->kb_auto ? 1 : 0,
s->key_remote >= 0 ? "remote" : s->keys_desktop ? "desktop" : s->keys_clicked ? "desktop (hidden)" : "steam",
focused_screen(s) + 1, s->pointer_focus ? s->pointer_focus->index + 1 : 0);
} else if (!ft_remote_command(buf, reply, sizeof reply)) {
ft_vr_command(buf, reply, sizeof reply);
}
if (len > offsetof(struct sockaddr_un, sun_path))
@@ -861,6 +913,7 @@ int main(int argc, char **argv) {
s.controller_click.threshold = 32;
for (int i = 0; i < MAX_SCREENS; ++i) s.scale[i] = 1;
s.kb_screen = -1;
s.key_remote = -1;
s.rate[FT_FOCUSED] = 0, s.rate[FT_IN_VIEW] = 15, s.rate[FT_HIDDEN] = 1;
s.period_ns = 1000000000LL / 90;
s.phase_ms = 1;
@@ -876,6 +929,8 @@ int main(int argc, char **argv) {
s.spares = atoi(argv[++i]);
} else if (strcmp(argv[i], "--no-vr") == 0) {
s.vr = false;
} else if (strcmp(argv[i], "--beside") == 0) {
s.beside = true;
} else if (strcmp(argv[i], "--rates") == 0 && i + 1 < argc) {
if (sscanf(argv[++i], "%d,%d,%d", &s.rate[FT_FOCUSED], &s.rate[FT_IN_VIEW], &s.rate[FT_HIDDEN]) != 3) {
fprintf(stderr, "bad --rates %s (want FOCUSED,IN_VIEW,HIDDEN in Hz, 0 full)\n", argv[i]);
@@ -895,7 +950,7 @@ int main(int argc, char **argv) {
break;
} else {
fprintf(stderr,
"usage: %s [--socket NAME] [--control NAME] [--no-vr] [--screen WxH@METRES]... "
"usage: %s [--socket NAME] [--control NAME] [--no-vr] [--beside] [--screen WxH@METRES]... "
"[--spares N] [--rates F,V,H] [-- COMMAND ARGS...]\n",
argv[0]);
return 2;
@@ -906,10 +961,12 @@ int main(int argc, char **argv) {
setvbuf(stdout, NULL, _IOLBF, 0); // vr.cpp prints to stdout; keep it in order with the log
wlr_log_init(WLR_INFO, NULL);
if (s.vr && !ft_vr_init()) return 1;
if (s.beside) ft_vr_beside();
if (!s.vr) wlr_log(WLR_INFO, "--no-vr: running without SteamVR");
s.display = wl_display_create();
s.loop = wl_display_get_event_loop(s.display);
ft_remote_init(s.loop, s.vr);
wl_list_init(&s.buffers);
wlr_compositor_create(s.display, 6, NULL);
wlr_subcompositor_create(s.display);
@@ -981,12 +1038,16 @@ int main(int argc, char **argv) {
wl_display_run(s.display);
wlr_log(WLR_INFO, "stopping");
// SteamVR first, while every buffer the panels show still exists (KWin's and the remote
// streams'): vrcompositor leaves standby the moment we disconnect, and twice it drew a
// remote panel's texture that was already gone (SIGBUS, 2026-10-07).
ft_vr_shutdown();
ft_remote_shutdown();
if (s.child > 0) kill(s.child, SIGTERM);
wl_display_destroy_clients(s.display);
// wlroots asserts that nothing still listens to its globals when they go.
wl_list_remove(&s.new_toplevel.link);
wl_list_remove(&s.new_decoration.link);
ft_vr_shutdown();
wl_display_destroy(s.display);
return 0;
}
+212 -27
View File
@@ -88,7 +88,7 @@ bool Hands::Read() {
const Mat head = HeadAt(captureNs_);
// each hand's palm in the room, and its velocity from the last time it was seen
ids_.clear();
ids_.clear(), basePts_.clear();
std::vector<int> owners; // the hand each capsule belongs to, in file order
for (uint32_t k = 0; k < nhands; ++k) {
const fh_hand_t &h = copy.hands[k];
@@ -97,6 +97,13 @@ bool Hands::Read() {
const int idx = int(ids_.size());
ids_.push_back(id);
owners.insert(owners.end(), std::min<uint32_t>(h.ncapsules, kMaxCapsules), idx);
HandPoints hp{id, (h.flags & FH_HAND_RIGHT) != 0, {}};
bool finite = true;
for (int j = 0; j < 21; ++j) {
for (float v : pts[j]) finite = finite && std::isfinite(v) && std::fabs(v) < 10;
Apply(head, pts[j], hp.p[j]);
}
if (finite) basePts_.push_back(hp);
double palm[3] = {0, 0, 0};
bool ok = true;
for (int j : {0, 5, 9, 13, 17}) {
@@ -162,7 +169,7 @@ bool Hands::Update(const Mat &head, int64_t nowNs) {
history_.push_back({nowNs, head});
while (!history_.empty() && nowNs - history_.front().ns > kHistoryNs) history_.erase(history_.begin());
Read();
if (nowNs - publishNs_ > kStaleNs) base_.clear(), owner_.clear();
if (nowNs - publishNs_ > kStaleNs) base_.clear(), owner_.clear(), basePts_.clear();
// move each hand ahead to when this frame will be on the displays; a slow hand's
// velocity is mostly tracking noise, so it fades out below kStillSpeed
const double ahead = std::clamp((nowNs + leadNs_ - captureNs_) / 1e9, 0.0, kMaxAhead);
@@ -182,6 +189,21 @@ bool Hands::Update(const Mat &head, int64_t nowNs) {
return !world_.empty();
}
void Hands::Points(int64_t nowNs, bool predict, double leadMs, std::vector<HandPoints> &out) const {
out = basePts_;
if (!predict) return;
const double ahead = std::clamp((nowNs + leadMs * 1e6 - captureNs_) / 1e9, 0.0, kMaxAhead);
for (HandPoints &hp : out) {
const auto m = motion_.find(hp.id);
if (m == motion_.end()) continue;
const double *v = m->second.v;
const double speed = std::sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2]);
const double gain = std::clamp((speed - kStillSpeed) / kStillSpeed, 0.0, 1.0); // as Update
for (auto &p : hp.p)
for (int i = 0; i < 3; ++i) p[i] += float(v[i] * gain * ahead);
}
}
void EyePositions(const Mat &head, double out[2][3]) {
const vr::EVREye eyes[2] = {vr::Eye_Left, vr::Eye_Right};
for (int e = 0; e < 2; ++e) {
@@ -279,6 +301,9 @@ PFNEGLCREATEIMAGEKHRPROC pCreateImage;
PFNEGLDESTROYIMAGEKHRPROC pDestroyImage;
PFNGLEGLIMAGETARGETTEXTURE2DOESPROC pImageTargetTexture;
PFNGLEGLIMAGETARGETRENDERBUFFERSTORAGEOESPROC pImageTargetRenderbuffer;
PFNEGLCREATESYNCKHRPROC pCreateSync;
PFNEGLDESTROYSYNCKHRPROC pDestroySync;
PFNEGLCLIENTWAITSYNCKHRPROC pClientWaitSync;
const char *kVertex = R"(
attribute vec2 pos; // the unit square
@@ -314,6 +339,20 @@ void main() {
gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0 - smoothstep(rad - feather, rad + feather, d));
})";
// A probe dot: the colour inside, a dark ring around it so it reads on any background.
const char *kMark = R"(
precision highp float;
uniform vec2 c;
uniform float r;
uniform vec3 color;
varying vec2 px;
void main() {
float d = length(px - c);
float a = 1.0 - smoothstep(r - 0.75, r + 0.75, d);
if (a <= 0.0) discard;
gl_FragColor = vec4(d > r - 2.0 ? vec3(0.0) : color, a);
})";
unsigned Shader(GLenum type, const char *src) {
const GLuint s = glCreateShader(type);
glShaderSource(s, 1, &src, nullptr);
@@ -400,7 +439,11 @@ bool Renderer::Init(const std::vector<uint64_t> &modifiers, std::function<void(c
pImageTargetTexture = reinterpret_cast<PFNGLEGLIMAGETARGETTEXTURE2DOESPROC>(eglGetProcAddress("glEGLImageTargetTexture2DOES"));
pImageTargetRenderbuffer = reinterpret_cast<PFNGLEGLIMAGETARGETRENDERBUFFERSTORAGEOESPROC>(
eglGetProcAddress("glEGLImageTargetRenderbufferStorageOES"));
if (!gbm_ || !pGetPlatformDisplay || !pCreateImage || !pImageTargetTexture || !pImageTargetRenderbuffer) {
pCreateSync = reinterpret_cast<PFNEGLCREATESYNCKHRPROC>(eglGetProcAddress("eglCreateSyncKHR"));
pDestroySync = reinterpret_cast<PFNEGLDESTROYSYNCKHRPROC>(eglGetProcAddress("eglDestroySyncKHR"));
pClientWaitSync = reinterpret_cast<PFNEGLCLIENTWAITSYNCKHRPROC>(eglGetProcAddress("eglClientWaitSyncKHR"));
if (!gbm_ || !pGetPlatformDisplay || !pCreateImage || !pImageTargetTexture || !pImageTargetRenderbuffer ||
!pCreateSync || !pDestroySync || !pClientWaitSync) {
std::fprintf(stderr, "handcut: GBM or EGL extensions missing\n");
return false;
}
@@ -415,7 +458,8 @@ bool Renderer::Init(const std::vector<uint64_t> &modifiers, std::function<void(c
ctx_ = ctx;
copyProg_ = Program(kCopy);
cutProg_ = Program(kCut);
if (!copyProg_ || !cutProg_) return false;
markProg_ = Program(kMark);
if (!copyProg_ || !cutProg_ || !markProg_) return false;
const float quad[] = {0, 0, 1, 0, 0, 1, 1, 1};
glGenBuffers(1, &vbo_);
glBindBuffer(GL_ARRAY_BUFFER, vbo_);
@@ -451,6 +495,9 @@ void Renderer::Forget(const void *key) {
glDeleteTextures(1, &it->second.tex);
pDestroyImage(EGLDisplay(dpy_), EGLImageKHR(it->second.image));
imported_.erase(it);
for (auto &[k, r] : rings_) // a new buffer at the same address isn't this one
for (Output &o : r.out)
if (o.key == key) o.drawn = false;
}
bool Renderer::MakeOutput(Output &o, int w, int h) {
@@ -489,6 +536,7 @@ bool Renderer::MakeOutput(Output &o, int w, int h) {
void Renderer::FreeOutput(Output &o) {
if (o.bo && released_) released_(&o);
if (o.fence) pDestroySync(EGLDisplay(dpy_), EGLSyncKHR(o.fence));
if (o.fbo) glDeleteFramebuffers(1, &o.fbo);
if (o.rb) glDeleteRenderbuffers(1, &o.rb);
if (o.image) pDestroyImage(EGLDisplay(dpy_), EGLImageKHR(o.image));
@@ -505,26 +553,68 @@ void Renderer::DropPanel(int panel) {
rings_.erase(it);
}
const Output *Renderer::Composite(int panel, const void *key, const ft_dmabuf &src, const std::vector<Capsule2D> eyes[2]) {
if (!ready_) return nullptr;
const auto t0 = std::chrono::steady_clock::now();
const int w = src.width, h = src.height;
Ring &ring = rings_[panel];
if (ring.w != w || ring.h != h) {
for (Output &old : ring.out) FreeOutput(old);
ring.w = w, ring.h = h, ring.next = 0;
}
Output &o = ring.out[ring.next];
if (!o.bo && !MakeOutput(o, 2 * w, h)) return nullptr;
const GLuint tex = Texture(key, src);
if (!tex) return nullptr;
ring.next = (ring.next + 1) % 3;
namespace {
// The pixels a cutout's quad covers (see Draw), as x0 y0 x1 y1 in the eye's half.
void Bounds(const Capsule2D &c, float b[4]) {
const float feather = std::max(1.5f, 0.15f * std::min(c.ra, c.rb));
const float r = std::max(c.ra, c.rb) + feather;
b[0] = std::min(c.ax, c.bx) - r, b[1] = std::min(c.ay, c.by) - r;
b[2] = std::max(c.ax, c.bx) + r, b[3] = std::max(c.ay, c.by) + r;
}
// Within a quarter pixel: the same picture.
bool SameSpots(const std::vector<Capsule2D> a[2], const std::vector<Capsule2D> b[2]) {
for (int e = 0; e < 2; ++e) {
if (a[e].size() != b[e].size()) return false;
for (size_t i = 0; i < a[e].size(); ++i) {
const Capsule2D &p = a[e][i], &q = b[e][i];
for (float d : {p.ax - q.ax, p.ay - q.ay, p.bx - q.bx, p.by - q.by, p.ra - q.ra, p.rb - q.rb})
if (std::fabs(d) > 0.25f) return false;
}
}
return true;
}
int64_t SteadyNs() {
return std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();
}
} // namespace
bool Renderer::Passed(Output &o, int64_t timeoutNs) {
if (!o.fence) return true;
const EGLint r = pClientWaitSync(EGLDisplay(dpy_), EGLSyncKHR(o.fence), 0, EGLTimeKHR(timeoutNs));
if (r == EGL_TIMEOUT_EXPIRED_KHR) return false;
pDestroySync(EGLDisplay(dpy_), EGLSyncKHR(o.fence)); // passed, or failed: don't wait on it again
o.fence = nullptr;
return true;
}
// Draws one buffer. Partial: the buffer holds this client frame already, with o.spots cut
// out, so each eye is drawn again only inside the box around those and the new cutouts.
void Renderer::Draw(Output &o, unsigned tex, int w, int h, const std::vector<Capsule2D> eyes[2], bool partial) {
glBindFramebuffer(GL_FRAMEBUFFER, o.fbo);
glBindBuffer(GL_ARRAY_BUFFER, vbo_);
glEnableVertexAttribArray(0);
glVertexAttribPointer(0, 2, GL_FLOAT, GL_FALSE, 0, nullptr);
for (int e = 0; e < 2; ++e) {
if (partial) {
float box[4] = {1e9f, 1e9f, -1e9f, -1e9f}, b[4];
const std::vector<Capsule2D> *lists[2] = {&o.spots[e], &eyes[e]};
for (const std::vector<Capsule2D> *list : lists)
for (const Capsule2D &c : *list) {
Bounds(c, b);
box[0] = std::min(box[0], b[0]), box[1] = std::min(box[1], b[1]);
box[2] = std::max(box[2], b[2]), box[3] = std::max(box[3], b[3]);
}
// Window y is the buffer's row, the same way down as the cutouts' y (see kVertex).
const int x0 = std::clamp(int(std::floor(box[0])) - 1, 0, w), y0 = std::clamp(int(std::floor(box[1])) - 1, 0, h);
const int x1 = std::clamp(int(std::ceil(box[2])) + 1, 0, w), y1 = std::clamp(int(std::ceil(box[3])) + 1, 0, h);
if (x1 <= x0 || y1 <= y0) continue; // no cutout in this eye, then or now
glEnable(GL_SCISSOR_TEST);
glScissor(e * w + x0, y0, x1 - x0, y1 - y0);
}
glViewport(e * w, 0, w, h);
glDisable(GL_BLEND);
glUseProgram(copyProg_);
@@ -543,22 +633,117 @@ const Output *Renderer::Composite(int panel, const void *key, const ft_dmabuf &s
uB = glGetUniformLocation(cutProg_, "b"), uR = glGetUniformLocation(cutProg_, "r"),
uF = glGetUniformLocation(cutProg_, "feather");
for (const Capsule2D &c : eyes[e]) {
const float feather = std::max(1.5f, 0.15f * std::min(c.ra, c.rb));
const float r = std::max(c.ra, c.rb) + feather;
glUniform4f(uRect, std::min(c.ax, c.bx) - r, std::min(c.ay, c.by) - r, std::max(c.ax, c.bx) + r,
std::max(c.ay, c.by) + r);
float b[4];
Bounds(c, b);
glUniform4f(uRect, b[0], b[1], b[2], b[3]);
glUniform2f(uA, c.ax, c.ay);
glUniform2f(uB, c.bx, c.by);
glUniform2f(uR, c.ra, c.rb);
glUniform1f(uF, feather);
glUniform1f(uF, std::max(1.5f, 0.15f * std::min(c.ra, c.rb)));
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
}
glDisable(GL_SCISSOR_TEST);
}
glDisable(GL_BLEND);
// SteamVR reads the buffer from another process and GPU queue; make sure it's done.
glFinish();
lastMs_ = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count();
return &o;
}
const Output *Renderer::Composite(int panel, const void *key, uint64_t serial, const ft_dmabuf &src,
const std::vector<Capsule2D> eyes[2]) {
if (!ready_) return nullptr;
const int64_t t0 = SteadyNs();
const int w = src.width, h = src.height;
Ring &ring = rings_[panel];
if (ring.w != w || ring.h != h) {
for (Output &old : ring.out) FreeOutput(old);
ring.w = w, ring.h = h, ring.shown = ring.before = ring.drawing = -1;
}
// After a pause the panel showed its client buffer, so nothing of ours is on it.
if (t0 - ring.lastCall > 30'000'000) ring.shown = ring.before = -1;
ring.lastCall = t0;
auto promote = [&ring] {
ring.before = ring.shown, ring.shown = ring.drawing, ring.drawing = -1;
};
if (ring.drawing >= 0 && Passed(ring.out[ring.drawing], 0)) promote();
const int newest = ring.drawing >= 0 ? ring.drawing : ring.shown;
const Output *n = newest >= 0 ? &ring.out[newest] : nullptr;
if (n && n->key == key && n->serial == serial && SameSpots(n->spots, eyes)) {
++stats_.same;
} else if (ring.drawing >= 0) {
++stats_.busy; // drawn on a later tick, from what's current then
} else {
int i = 0;
while (i == ring.shown || i == ring.before) ++i;
Output &o = ring.out[i];
if (!o.bo && !MakeOutput(o, 2 * w, h)) return nullptr;
const GLuint tex = Texture(key, src);
if (!tex) return nullptr;
const bool partial = o.drawn && o.key == key && o.serial == serial;
Draw(o, tex, w, h, eyes, partial);
o.fence = pCreateSync(EGLDisplay(dpy_), EGL_SYNC_FENCE_KHR, nullptr);
glFlush();
if (!o.fence) glFinish(); // no fence: wait here, as before
o.key = key, o.serial = serial, o.drawn = true;
for (int e = 0; e < 2; ++e) o.spots[e] = eyes[e];
ring.drawing = i;
++stats_.draws, stats_.partial += partial;
}
// Nothing of ours to show yet: wait for this one rather than show none.
if (ring.shown < 0 && ring.drawing >= 0) {
++stats_.waits;
if (Passed(ring.out[ring.drawing], 50'000'000)) promote();
}
lastMs_ = (SteadyNs() - t0) / 1e6;
stats_.cpuMs += lastMs_, stats_.worstMs = std::max(stats_.worstMs, lastMs_);
return ring.shown >= 0 ? &ring.out[ring.shown] : nullptr;
}
const Output *Renderer::Marks(int panel, int w, int h, const std::vector<Mark> eyes[2]) {
if (!ready_) return nullptr;
Ring &ring = rings_[panel];
if (ring.w != w || ring.h != h) {
for (Output &old : ring.out) FreeOutput(old);
ring.w = w, ring.h = h, ring.shown = ring.before = ring.drawing = -1;
}
auto promote = [&ring] {
ring.before = ring.shown, ring.shown = ring.drawing, ring.drawing = -1;
};
if (ring.drawing >= 0 && Passed(ring.out[ring.drawing], 0)) promote();
if (ring.drawing < 0) {
int i = 0;
while (i == ring.shown || i == ring.before) ++i;
Output &o = ring.out[i];
if (!o.bo && !MakeOutput(o, 2 * w, h)) return nullptr;
glBindFramebuffer(GL_FRAMEBUFFER, o.fbo);
glViewport(0, 0, 2 * w, h);
glClearColor(0, 0, 0, 0);
glClear(GL_COLOR_BUFFER_BIT);
glBindBuffer(GL_ARRAY_BUFFER, vbo_);
glEnableVertexAttribArray(0);
glVertexAttribPointer(0, 2, GL_FLOAT, GL_FALSE, 0, nullptr);
glDisable(GL_BLEND);
glUseProgram(markProg_);
glUniform2f(glGetUniformLocation(markProg_, "size"), float(w), float(h));
const GLint uRect = glGetUniformLocation(markProg_, "rect"), uC = glGetUniformLocation(markProg_, "c"),
uR = glGetUniformLocation(markProg_, "r"), uColor = glGetUniformLocation(markProg_, "color");
for (int e = 0; e < 2; ++e) {
glViewport(e * w, 0, w, h);
for (const Mark &m : eyes[e]) {
glUniform4f(uRect, m.x - m.r - 1, m.y - m.r - 1, m.x + m.r + 1, m.y + m.r + 1);
glUniform2f(uC, m.x, m.y);
glUniform1f(uR, m.r);
glUniform3f(uColor, m.rgb[0], m.rgb[1], m.rgb[2]);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
}
}
o.fence = pCreateSync(EGLDisplay(dpy_), EGL_SYNC_FENCE_KHR, nullptr);
glFlush();
if (!o.fence) glFinish();
o.key = nullptr, o.drawn = false; // holds no client frame
ring.drawing = i;
}
if (ring.shown < 0 && ring.drawing >= 0 && Passed(ring.out[ring.drawing], 50'000'000)) promote();
return ring.shown >= 0 ? &ring.out[ring.shown] : nullptr;
}
} // namespace handcut
+64 -7
View File
@@ -58,6 +58,17 @@ public:
bool predicting() const { return predict_; }
double leadMs() const { return leadNs_ / 1e6; }
// Each hand's 21 landmarks in the room, for ft-handtest --probe: as the cameras saw them
// (predict false), or moved ahead along the hand's velocity to now + leadMs, as the
// capsules are. Empty while no fresh hands are known (as capsules()).
struct HandPoints {
uint32_t id;
bool right;
float p[21][3];
};
void Points(int64_t nowNs, bool predict, double leadMs, std::vector<HandPoints> &out) const;
int64_t captureNs() const { return captureNs_; }
private:
bool Read();
Mat HeadAt(int64_t ns) const;
@@ -67,10 +78,11 @@ private:
std::vector<Capsule> base_; // the capsules at capture time, in the room
std::vector<int> owner_; // each capsule's hand (index into ids_), or -1
std::vector<uint32_t> ids_; // the hands in the file
std::vector<HandPoints> basePts_; // their landmarks at capture time, in the room
std::map<uint32_t, Motion> motion_;
std::vector<Capsule> world_; // base_, moved ahead
bool predict_ = true;
int64_t leadNs_ = 25'000'000;
int64_t leadNs_ = 36'000'000; // 25 ms to the displays, plus the tick a cutout buffer waits for its fence
int fd_ = -1;
const void *map_ = nullptr;
uint64_t seq_ = 0;
@@ -91,6 +103,28 @@ struct Output {
void *bo = nullptr;
unsigned fbo = 0, rb = 0;
void *image = nullptr;
// What's drawn in it: the client buffer and its frame, and the cutouts (a later draw
// with the same frame only redraws around the old and new cutouts).
void *fence = nullptr; // the GPU is still drawing it
const void *key = nullptr;
uint64_t serial = 0;
bool drawn = false;
std::vector<Capsule2D> spots[2];
};
// A dot for ft-handtest --probe: centre and radius in pixels of the eye's half, colour.
struct Mark {
float x, y, r;
float rgb[3];
};
// Composite's counts since the last TakeStats.
struct CutStats {
int draws = 0, partial = 0; // buffers drawn, of them only around the cutouts
int same = 0; // nothing changed: the newest buffer stays
int busy = 0; // the GPU hadn't finished the last one: drawn next tick
int waits = 0; // a panel's first buffer, waited for
double cpuMs = 0, worstMs = 0;
};
class Renderer {
@@ -99,31 +133,54 @@ public:
// modifiers: what SteamVR takes for DRM_FORMAT_ABGR8888, the outputs' format.
// released: an output is about to be freed (drop its SteamVR import).
bool Init(const std::vector<uint64_t> &modifiers, std::function<void(const Output *)> released);
// Draw client buffer `src` (identified by `key`) into the next output buffer of
// panel `panel`, both eyes, cutting out `eyes`. Returns that buffer, or null.
const Output *Composite(int panel, const void *key, const ft_dmabuf &src, const std::vector<Capsule2D> eyes[2]);
// Draw client buffer `src` (identified by `key`; `serial` counts its frames) into a
// buffer of panel `panel`, both eyes, cutting out `eyes`. Returns the newest buffer the
// GPU has finished, or null.
//
// It doesn't wait for the GPU: a buffer is drawn, fenced, and returned from a later call
// once the fence has passed, so what SteamVR shows is a tick behind. Each panel has three
// buffers: the one shown, the one shown before it (SteamVR may still be reading it), and
// the one being drawn. Nothing is drawn when the frame and the cutouts are what the newest
// buffer has, and when only the cutouts moved, a buffer that holds the same client frame
// is drawn again only around them. The first call after a pause (no call for 30 ms, about
// 3 ticks: the panel showed its client buffer meanwhile) waits for its buffer, so a stale
// one never shows.
const Output *Composite(int panel, const void *key, uint64_t serial, const ft_dmabuf &src,
const std::vector<Capsule2D> eyes[2]);
// ft-handtest --probe: a transparent w x h buffer per eye (side by side) with only the
// dots in it, drawn every call, fenced as Composite's. Returns the newest finished one.
const Output *Marks(int panel, int w, int h, const std::vector<Mark> eyes[2]);
// A client buffer is going away.
void Forget(const void *key);
// A panel is gone: drop its outputs.
void DropPanel(int panel);
// How long the last Composite took, ms (it waits for the GPU).
// How long the last Composite took on the CPU, ms.
double lastMs() const { return lastMs_; }
CutStats TakeStats() { CutStats s = stats_; stats_ = {}; return s; }
private:
unsigned Texture(const void *key, const ft_dmabuf &src);
bool MakeOutput(Output &o, int w, int h);
void FreeOutput(Output &o);
bool Passed(Output &o, int64_t timeoutNs);
void Draw(Output &o, unsigned tex, int w, int h, const std::vector<Capsule2D> eyes[2], bool partial);
bool ready_ = false;
int drm_ = -1;
void *gbm_ = nullptr, *dpy_ = nullptr, *ctx_ = nullptr;
unsigned copyProg_ = 0, cutProg_ = 0, vbo_ = 0;
unsigned copyProg_ = 0, cutProg_ = 0, markProg_ = 0, vbo_ = 0;
std::vector<uint64_t> modifiers_;
std::function<void(const Output *)> released_;
struct Imported { void *image; unsigned tex; };
std::map<const void *, Imported> imported_;
struct Ring { Output out[3]; int next = 0; int w = 0, h = 0; };
struct Ring {
Output out[3];
int shown = -1, before = -1, drawing = -1; // indices into out
int w = 0, h = 0;
int64_t lastCall = 0; // steady clock ns
};
std::map<int, Ring> rings_;
double lastMs_ = 0;
CutStats stats_;
};
} // namespace handcut
+150 -10
View File
@@ -20,6 +20,7 @@
#include <cstdlib>
#include <cstring>
#include <map>
#include <string>
#include <thread>
namespace {
@@ -77,14 +78,142 @@ bool TestPattern(int drm, int w, int h, gbm_bo **out, ft_dmabuf *b) {
return true;
}
// --probe: where the cutouts would land, against the hand Room View shows. The panel is
// see-through, with dots on the wrist, middle knuckle and fingertips of each tracked hand,
// one colour per timing: magenta where the cameras saw the hand, cyan moved ahead to now,
// green moved ahead to now + lead (what the cutouts use). White dots mark the panel's
// corners. Record the headset view meanwhile and compare (frame-hands/probes/
// probe_video.py), or look: which colour sits on your fingertips, still and moving?
// Each tick goes to the log as a JSON line, after a header line with the panel and eyes.
struct Variant {
const char *name;
bool predict;
double leadMs;
float rgb[3];
};
const int kProbePoints[] = {0, 9, 4, 8, 12, 16, 20};
void Matrix(FILE *f, const handcut::Mat &m) {
std::fprintf(f, "[");
for (int r = 0; r < 3; ++r)
for (int c = 0; c < 4; ++c) std::fprintf(f, "%s%.6f", r || c ? "," : "", m.m[r][c]);
std::fprintf(f, "]");
}
void Point(FILE *f, const float p[3]) { std::fprintf(f, "[%.5f,%.5f,%.5f]", p[0], p[1], p[2]); }
int RunProbe(vr::VROverlayHandle_t ov, const handcut::Panel &panel, handcut::Renderer &renderer,
std::map<const void *, vr::SharedTextureHandle_t> &imports, double seconds, double leadMs, double dotMm,
const char *logPath) {
handcut::Hands hands;
if (leadMs < 0) leadMs = hands.leadMs();
const Variant variants[] = {{"seen", false, 0, {1, 0, 1}}, {"now", true, 0, {0, 1, 1}}, {"lead", true, leadMs, {0, 1, 0}}};
FILE *log = std::fopen(logPath, "w");
if (!log) return std::perror(logPath), 1;
std::fprintf(log, "{\"probe\":1,\"panel\":{\"pose\":");
Matrix(log, panel.pose);
std::fprintf(log, ",\"width\":%.4f,\"height\":%.4f,\"px\":[%d,%d]},\"eyes\":[", panel.width, panel.height,
panel.pxWidth, panel.pxHeight);
for (vr::EVREye e : {vr::Eye_Left, vr::Eye_Right}) {
Matrix(log, vr::VRSystem()->GetEyeToHeadTransform(e));
std::fprintf(log, e == vr::Eye_Left ? "," : "],");
}
std::fprintf(log, "\"points\":[0,9,4,8,12,16,20],\"dot_mm\":%.2f,\"variants\":[", dotMm);
for (size_t k = 0; k < 3; ++k)
std::fprintf(log, "%s{\"name\":\"%s\",\"predict\":%s,\"lead_ms\":%.1f,\"rgb\":[%.0f,%.0f,%.0f]}", k ? "," : "",
variants[k].name, variants[k].predict ? "true" : "false", variants[k].leadMs,
variants[k].rgb[0] * 255, variants[k].rgb[1] * 255, variants[k].rgb[2] * 255);
std::fprintf(log, "]}\n");
vr::VROverlay()->SetOverlayFlag(ov, vr::VROverlayFlags_IgnoreTextureAlpha, false);
vr::VROverlay()->SetOverlayFlag(ov, vr::VROverlayFlags_SideBySide_Parallel, true);
vr::SharedTextureHandle_t shown = 0;
vr::Texture_t tex = {&shown, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
std::printf("probe: magenta = as seen, cyan = now, green = now + %.0f ms; log %s\n", leadMs, logPath);
vr::TrackedDevicePose_t poses[vr::k_unMaxTrackedDeviceCount];
const int64_t start = MonoNs();
int64_t lastReport = start;
int ticks = 0, withHands = 0;
std::vector<handcut::Hands::HandPoints> pts;
while (!g_stop && (seconds <= 0 || (MonoNs() - start) / 1e9 < seconds)) {
const auto tick = std::chrono::steady_clock::now();
vr::VRSystem()->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, poses, vr::k_unMaxTrackedDeviceCount);
const auto &head = poses[vr::k_unTrackedDeviceIndex_Hmd].mDeviceToAbsoluteTracking;
const int64_t now = MonoNs();
hands.Update(head, now);
double eyes[2][3];
handcut::EyePositions(head, eyes);
std::vector<handcut::Mark> marks[2];
const float W = float(panel.pxWidth), H = float(panel.pxHeight);
for (int e = 0; e < 2; ++e)
for (float x : {14.f, W - 14}) for (float y : {14.f, H - 14}) marks[e].push_back({x, y, 6, {1, 1, 1}});
std::fprintf(log, "{\"t\":%lld,\"cap\":%lld,\"head\":", (long long)now, (long long)hands.captureNs());
Matrix(log, head);
std::fprintf(log, ",\"hands\":[");
bool any = false;
for (size_t v = 0; v < 3; ++v) {
hands.Points(now, variants[v].predict, variants[v].leadMs, pts);
for (size_t h = 0; h < pts.size(); ++h) {
std::fprintf(log, "%s{\"v\":%zu,\"id\":%u,\"right\":%d,\"p\":[", any ? "," : "", v, pts[h].id, pts[h].right);
any = true;
for (size_t j = 0; j < sizeof kProbePoints / sizeof *kProbePoints; ++j) {
const float *p = pts[h].p[kProbePoints[j]];
if (j) std::fputc(',', log);
Point(log, p);
const handcut::Capsule c{{p[0], p[1], p[2]}, {p[0], p[1], p[2]}, float(dotMm / 1000), float(dotMm / 1000)};
std::vector<handcut::Capsule2D> on[2];
if (!handcut::Project(panel, {c}, eyes, on)) continue;
for (int e = 0; e < 2; ++e)
for (const auto &d : on[e])
marks[e].push_back({d.ax, d.ay, std::max(4.f, d.ra), {variants[v].rgb[0], variants[v].rgb[1], variants[v].rgb[2]}});
}
std::fprintf(log, "]}");
}
}
std::fprintf(log, "]}\n");
withHands += any;
const handcut::Output *out = renderer.Marks(0, panel.pxWidth, panel.pxHeight, marks);
if (out) {
auto it = imports.find(out);
if (it == imports.end()) it = imports.emplace(out, Import(out->buf)).first;
if (it->second && shown != it->second) {
const bool first = !shown;
shown = it->second;
vr::VROverlay()->SetOverlayTexture(ov, &tex);
if (first) vr::VROverlay()->ShowOverlay(ov);
}
}
++ticks;
if (now - lastReport > 2'000'000'000) {
std::printf("%.0f s: %d ticks, %d with hands\n", (now - start) / 1e9, ticks, withHands);
std::fflush(stdout);
std::fflush(log);
lastReport = now, ticks = withHands = 0;
}
std::this_thread::sleep_until(tick + std::chrono::microseconds(11111));
}
std::fclose(log);
return 0;
}
} // namespace
int main(int argc, char **argv) {
double distance = 0.8, width = 1.0, seconds = 0;
for (int i = 1; i + 1 < argc; i += 2) {
if (!std::strcmp(argv[i], "--distance")) distance = std::atof(argv[i + 1]);
else if (!std::strcmp(argv[i], "--width")) width = std::atof(argv[i + 1]);
else if (!std::strcmp(argv[i], "--seconds")) seconds = std::atof(argv[i + 1]);
double distance = 0.8, width = 1.0, seconds = 0, leadMs = -1, dotMm = 3;
bool probe = false;
std::string logPath = "/tmp/handprobe-" + std::to_string(time(nullptr)) + ".jsonl";
for (int i = 1; i < argc; ++i) {
const bool more = i + 1 < argc;
if (!std::strcmp(argv[i], "--probe")) probe = true;
else if (!std::strcmp(argv[i], "--distance") && more) distance = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--width") && more) width = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--seconds") && more) seconds = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--lead") && more) leadMs = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--dot-mm") && more) dotMm = std::atof(argv[++i]);
else if (!std::strcmp(argv[i], "--log") && more) logPath = argv[++i];
else return std::fprintf(stderr, "usage: ft-handtest [--distance m] [--width m] [--seconds s] "
"[--probe [--lead ms] [--dot-mm mm] [--log FILE]]\n"), 2;
}
signal(SIGINT, Stop);
signal(SIGTERM, Stop);
@@ -137,6 +266,16 @@ int main(int argc, char **argv) {
P.m[2][3] = float(hm.m[2][3] - std::cos(yaw) * distance);
panel.width = width, panel.height = width * H / W, panel.curve = 0, panel.pxWidth = W, panel.pxHeight = H;
vr::VROverlay()->SetOverlayTransformAbsolute(ov, vr::TrackingUniverseStanding, &P);
if (probe) {
const int rc = RunProbe(ov, panel, renderer, imports, seconds, leadMs, dotMm, logPath.c_str());
vr::VROverlay()->DestroyOverlay(ov);
for (auto &[k, h] : imports)
if (h) vr::VRIPCResourceManager()->UnrefResource(h);
imports.clear();
vr::VRIPCResourceManager()->UnrefResource(plain);
vr::VR_Shutdown();
return rc;
}
vr::VROverlay()->SetOverlayFlag(ov, vr::VROverlayFlags_IgnoreTextureAlpha, true);
vr::SharedTextureHandle_t shown = plain;
vr::Texture_t tex = {&shown, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
@@ -163,7 +302,7 @@ int main(int argc, char **argv) {
handcut::EyePositions(head, eyes);
cut = handcut::Project(panel, hands.capsules(), eyes, eyes2d);
}
const handcut::Output *out = cut ? renderer.Composite(0, bo, client, eyes2d) : nullptr;
const handcut::Output *out = cut ? renderer.Composite(0, bo, 1, client, eyes2d) : nullptr;
if (out) {
auto it = imports.find(out);
if (it == imports.end()) {
@@ -177,8 +316,7 @@ int main(int argc, char **argv) {
vr::VROverlay()->SetOverlayFlag(ov, vr::VROverlayFlags_SideBySide_Parallel, true);
cutting = true;
}
shown = it->second;
vr::VROverlay()->SetOverlayTexture(ov, &tex);
if (shown != it->second) shown = it->second, vr::VROverlay()->SetOverlayTexture(ov, &tex);
ms += renderer.lastMs(), worst = std::max(worst, renderer.lastMs());
++cutFrames;
caps2d += eyes2d[0].size() + eyes2d[1].size();
@@ -192,10 +330,12 @@ int main(int argc, char **argv) {
}
++frames;
if (now - lastReport > 2'000'000'000) {
const handcut::CutStats st = renderer.TakeStats();
std::printf("%.0f s: %d ticks, %d with a cutout (%.1f capsules per eye), composite %.2f ms avg %.2f ms worst, "
"%zu hand capsules known\n",
"%zu hand capsules known; %d draws (%d partial), %d same, %d busy\n",
(now - start) / 1e9, frames, cutFrames, cutFrames ? caps2d / 2.0 / cutFrames : 0.0,
cutFrames ? ms / cutFrames : 0.0, worst, hands.capsules().size());
cutFrames ? ms / cutFrames : 0.0, worst, hands.capsules().size(), st.draws, st.partial, st.same,
st.busy);
std::fflush(stdout);
lastReport = now, frames = cutFrames = 0, ms = worst = 0, caps2d = 0;
}
+476
View File
@@ -0,0 +1,476 @@
// Remote screens: displays of other machines as panels of their own (docs/remote-displays.md).
// Each is streamed by its own ft-stream process (stream/ft-stream.cpp, GPLv3, a separate
// program so this one stays MIT), started here with one end of a SOCK_SEQPACKET socket pair
// as its fd 3. ft-stream decodes into a ring of three RGBA buffers and hands their dmabufs
// over once; then "frame I" says buffer I holds a new picture. It goes to the panel through
// ft_vr_screen_present, like a KWin buffer, and the buffer shown before it goes back
// ("release"). The panel's pointer input, the keys typed while it has the keyboard, and how
// much of it you see ("attention") go the other way; the protocol is in ft-stream.cpp.
//
// Commands (on @ft_screens, from ft-layout):
// remote <N> start <client> <host> <app> <W>x<H> <fps> <kbit/s> <metres> [label]
// runs remote screen N (FT_REMOTE_FIRST and up): ft-stream's client name (its paired
// certificate), the host's address, what to stream (display:DEVICE, monitor, primary),
// the stream's size and rate (kbit/s 0: ft-stream picks), the panel's width until the
// layout says, and the panel's name. Again with the same settings: nothing changes;
// with others, the stream starts over.
// remote <N> stop
// remotes -> "ok <count> <N>:<client>:<state>:<W>x<H> ..." (state: queued, starting,
// connecting, live, lost)
// A host's streams start one after another ("queued" until then): see host_busy.
// A stream that ends starts again, after 2 s, then longer after quick failures (up to 30 s).
// Its panel keeps the last picture meanwhile. ft-stream logs to
// $XDG_RUNTIME_DIR/frametop-remote-<N>.log.
#define _GNU_SOURCE
#include "remote.h"
#include <drm_fourcc.h>
#include <errno.h>
#include <linux/input-event-codes.h>
#include <fcntl.h>
#include <limits.h>
#include <signal.h>
#include <spawn.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>
#define WLR_USE_UNSTABLE
#include <wlr/util/log.h>
extern char **environ;
struct remote {
bool used;
int index; // the screen's index (its number - 1)
char client[64], host[64], app[192], label[96];
int width, height, fps, bitrate;
double metres;
pid_t pid; // ft-stream, 0 when not running
int fd; // our end of its socket, -1 when not connected
struct wl_event_source *source;
struct ft_dmabuf ring[3]; // its buffers; each is also the key its SteamVR import is kept under
bool have_ring;
int shown; // the ring buffer on the panel, -1 none
char state[96]; // as ft-stream last said
int attention; // what it was last told, -1 nothing yet
uint32_t restart_at; // ms: start it then (0: not waiting to)
uint32_t started_ms;
int failures; // quick ends in a row, for the back-off
};
static struct remote g_remotes[FT_REMOTE_MAX];
static struct wl_event_loop *g_loop;
static bool g_vr;
static char g_stream[PATH_MAX]; // ft-stream (found at start: a rebuild replaces our own file)
// Which remote screen each mouse button (BTN_LEFT + n) went down on, -1 none. Vibepollo takes a
// button's release only from the client that pressed it (mouse_press_owner in its input.cpp),
// so a window carried onto another of the host's displays is dropped through the stream it was
// picked up on: the release there went to the display under the laser, and the window stayed
// on the pointer (2026-10-07). The pointer's moves still go to the display it's on.
static int g_pressed_on[8] = {-1, -1, -1, -1, -1, -1, -1, -1};
static uint32_t now_ms(void) {
struct timespec t;
clock_gettime(CLOCK_MONOTONIC, &t);
return (uint32_t)(t.tv_sec * 1000 + t.tv_nsec / 1000000);
}
static struct remote *find(int index) {
for (int i = 0; i < FT_REMOTE_MAX; ++i)
if (g_remotes[i].used && g_remotes[i].index == index) return &g_remotes[i];
return NULL;
}
static void say(struct remote *r, const char *fmt, ...) __attribute__((format(printf, 2, 3)));
static void say(struct remote *r, const char *fmt, ...) {
if (r->fd < 0) return;
char msg[600];
va_list a;
va_start(a, fmt);
const int n = vsnprintf(msg, sizeof msg, fmt, a);
va_end(a);
if (n > 0 && n < (int)sizeof msg) send(r->fd, msg, (size_t)n, MSG_NOSIGNAL | MSG_DONTWAIT);
}
// Drops the ring's imports and closes its buffers (a new ring came, or the screen went).
static void forget_ring(struct remote *r) {
if (!r->have_ring) return;
for (int i = 0; i < 3; ++i) {
ft_vr_forget(&r->ring[i]);
close(r->ring[i].fd[0]);
}
r->have_ring = false;
r->shown = -1;
}
static void disconnect(struct remote *r) {
if (r->source) wl_event_source_remove(r->source);
r->source = NULL;
if (r->fd >= 0) close(r->fd);
r->fd = -1;
}
static void message(struct remote *r, const char *msg, const int *fds, int nfds) {
unsigned w, h, format, offset, stride;
unsigned long long modifier;
int i;
if (strncmp(msg, "state ", 6) == 0) {
snprintf(r->state, sizeof r->state, "%.95s", msg + 6);
wlr_log(WLR_INFO, "remote %d (%s): %s", r->index + 1, r->client, r->state);
if (strncmp(r->state, "live", 4) == 0) r->failures = 0;
} else if (sscanf(msg, "buffers %u %u %x %llx %u %u", &w, &h, &format, &modifier, &offset, &stride) == 6 &&
nfds == 3 && w > 0 && h > 0 && w <= 16384 && h <= 16384) {
forget_ring(r);
for (int k = 0; k < 3; ++k) {
r->ring[k] = (struct ft_dmabuf){.width = (int)w, .height = (int)h, .format = format,
.modifier = modifier, .n_planes = 1};
r->ring[k].offset[0] = offset, r->ring[k].stride[0] = stride, r->ring[k].fd[0] = fds[k];
}
r->have_ring = true;
wlr_log(WLR_INFO, "remote %d: %ux%u buffers, modifier 0x%llx", r->index + 1, w, h, modifier);
return; // the fds are the ring's now
} else if (sscanf(msg, "frame %d", &i) == 1 && r->have_ring && i >= 0 && i < 3) {
if (ft_vr_screen_present(r->index, &r->ring[i], &r->ring[i])) {
if (r->shown >= 0 && r->shown != i) say(r, "release %d", r->shown);
r->shown = i;
} else {
say(r, "release %d", i);
}
}
for (int k = 0; k < nfds; ++k) close(fds[k]);
}
static void schedule_restart(struct remote *r) {
const uint32_t ran = now_ms() - r->started_ms;
r->failures = ran < 20000 ? r->failures + 1 : 0;
uint32_t wait = 2000;
for (int k = 1; k < r->failures && wait < 30000; ++k) wait *= 2;
if (wait > 30000) wait = 30000;
r->restart_at = now_ms() + wait;
if (!r->restart_at) r->restart_at = 1;
wlr_log(WLR_INFO, "remote %d: starting it again in %u s", r->index + 1, wait / 1000);
}
static int readable(int fd, uint32_t mask, void *data) {
struct remote *r = data;
for (;;) {
char buf[512];
char control[CMSG_SPACE(sizeof(int) * 4)];
struct iovec iov = {buf, sizeof buf - 1};
struct msghdr m = {.msg_iov = &iov, .msg_iovlen = 1, .msg_control = control, .msg_controllen = sizeof control};
const ssize_t n = recvmsg(fd, &m, MSG_DONTWAIT | MSG_CMSG_CLOEXEC);
if (n < 0 && (errno == EAGAIN || errno == EINTR)) break;
if (n <= 0) { // ft-stream is gone: its exit (SIGCHLD) sets up the restart
wlr_log(WLR_INFO, "remote %d: stream closed", r->index + 1);
snprintf(r->state, sizeof r->state, "lost");
disconnect(r);
return 0;
}
buf[n] = 0;
int fds[4], nfds = 0;
for (struct cmsghdr *c = CMSG_FIRSTHDR(&m); c; c = CMSG_NXTHDR(&m, c)) {
if (c->cmsg_level != SOL_SOCKET || c->cmsg_type != SCM_RIGHTS) continue;
const int got = (int)((c->cmsg_len - CMSG_LEN(0)) / sizeof(int));
for (int k = 0; k < got; ++k) {
int f;
memcpy(&f, CMSG_DATA(c) + k * sizeof(int), sizeof f);
if (nfds < 4) fds[nfds++] = f;
else close(f);
}
}
message(r, buf, fds, nfds);
}
return 0;
}
// <repo>/screens/build/ft-screens -> <repo>/stream/build/ft-stream, or $FT_STREAM.
static bool stream_path(char *out, size_t size) {
const char *env = getenv("FT_STREAM");
if (env && *env) return snprintf(out, size, "%s", env) < (int)size;
char exe[PATH_MAX];
if (!realpath("/proc/self/exe", exe)) return false; // gone already: ft-screens was rebuilt
for (int up = 0; up < 3; ++up) {
char *slash = strrchr(exe, '/');
if (!slash) return false;
*slash = 0;
}
return snprintf(out, size, "%s/stream/build/ft-stream", exe) < (int)size;
}
static void log_path(const struct remote *r, char *out, size_t size) {
const char *runtime = getenv("XDG_RUNTIME_DIR");
snprintf(out, size, "%s/frametop-remote-%d.log", runtime && *runtime ? runtime : "/tmp", r->index + 1);
}
static bool start_stream(struct remote *r) {
r->restart_at = 0;
r->started_ms = now_ms();
snprintf(r->state, sizeof r->state, "lost");
char *exe = g_stream, log[PATH_MAX];
if (!*exe) return false;
log_path(r, log, sizeof log);
int sv[2];
if (socketpair(AF_UNIX, SOCK_SEQPACKET | SOCK_CLOEXEC, 0, sv) != 0) return false;
if (sv[1] == 3) { // dup2 onto itself would keep close-on-exec
const int moved = fcntl(sv[1], F_DUPFD_CLOEXEC, 10);
close(sv[1]);
sv[1] = moved;
}
char size[32], fps[16], bitrate[16];
snprintf(size, sizeof size, "%dx%d", r->width, r->height);
snprintf(fps, sizeof fps, "%d", r->fps);
snprintf(bitrate, sizeof bitrate, "%d", r->bitrate);
char *argv[] = {exe, "stream", r->host, "--id", r->client, "--fd", "3", "--app", r->app,
"--size", size, "--fps", fps, "--bitrate", bitrate, NULL};
posix_spawn_file_actions_t io;
posix_spawn_file_actions_init(&io);
posix_spawn_file_actions_addopen(&io, 0, "/dev/null", O_RDONLY, 0);
posix_spawn_file_actions_addopen(&io, 1, log, O_WRONLY | O_CREAT | O_APPEND | O_NOFOLLOW, 0600);
posix_spawn_file_actions_adddup2(&io, 1, 2);
posix_spawn_file_actions_adddup2(&io, sv[1], 3);
// Not our event loop's blocked signals (SIGTERM, SIGINT, SIGCHLD go to its signalfd): with
// SIGTERM blocked, ft-stream heard neither kill nor its death signal.
posix_spawnattr_t attr;
posix_spawnattr_init(&attr);
sigset_t none;
sigemptyset(&none);
posix_spawnattr_setsigmask(&attr, &none);
posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGMASK);
pid_t pid;
const int rc = posix_spawn(&pid, exe, &io, &attr, argv, environ);
posix_spawnattr_destroy(&attr);
posix_spawn_file_actions_destroy(&io);
close(sv[1]);
if (rc != 0) {
wlr_log(WLR_ERROR, "remote %d: can't run %s: %s", r->index + 1, exe, strerror(rc));
close(sv[0]);
return false;
}
r->pid = pid;
r->fd = sv[0];
fcntl(r->fd, F_SETFL, O_NONBLOCK);
r->source = wl_event_loop_add_fd(g_loop, r->fd, WL_EVENT_READABLE, readable, r);
r->attention = -1;
snprintf(r->state, sizeof r->state, "starting");
// What SteamVR imports: ft-stream allocates its ring with one of these.
uint64_t mods[32];
const int n = ft_vr_modifiers(DRM_FORMAT_ABGR8888, mods, 32);
char msg[600];
int at = snprintf(msg, sizeof msg, "modifiers");
for (int k = 0; k < n && at < (int)sizeof msg - 20; ++k) at += snprintf(msg + at, sizeof msg - at, " %llx", (unsigned long long)mods[k]);
send(r->fd, msg, (size_t)at, MSG_NOSIGNAL | MSG_DONTWAIT);
wlr_log(WLR_INFO, "remote %d: started ft-stream (pid %d) for %s on %s, %s at %d fps (log %s)", r->index + 1, pid,
r->app, r->host, size, r->fps, log);
return true;
}
// Asks a running ft-stream to end (it releases a Remote Monitor on the way out).
static void stop_stream(struct remote *r) {
say(r, "quit");
disconnect(r);
if (r->pid > 0) kill(r->pid, SIGTERM);
}
// Vibepollo takes one stream operation at a time ("Another stream operation is still
// running"), and a Remote Monitor appearing while another display's capture starts leaves that
// one without a picture (2026-10-07: three started at once, two never got a frame). So a host's
// streams start one after another: the next once the last is live, lost, or 20 s old.
static bool host_busy(const struct remote *r) {
for (int i = 0; i < FT_REMOTE_MAX; ++i) {
const struct remote *o = &g_remotes[i];
if (o == r || !o->used || o->pid == 0 || strcmp(o->host, r->host) != 0) continue;
const bool starting = !strncmp(o->state, "starting", 8) || !strncmp(o->state, "connecting", 10);
if (starting && now_ms() - o->started_ms < 20000) return true;
}
return false;
}
void ft_remote_init(struct wl_event_loop *loop, bool vr) {
g_loop = loop;
g_vr = vr;
if (!stream_path(g_stream, sizeof g_stream)) g_stream[0] = 0;
for (int i = 0; i < FT_REMOTE_MAX; ++i) g_remotes[i].fd = -1, g_remotes[i].shown = -1;
}
void ft_remote_shutdown(void) {
for (int i = 0; i < FT_REMOTE_MAX; ++i) {
struct remote *r = &g_remotes[i];
stop_stream(r);
if (r->used) ft_vr_screen_destroy(r->index); // the panel goes before its textures
forget_ring(r);
}
}
bool ft_remote_is(int index) { return find(index) != NULL; }
void ft_remote_event(const struct ft_event *e) {
struct remote *r = find(e->screen);
if (!r) return;
const int w = r->have_ring ? r->ring[0].width : r->width, h = r->have_ring ? r->ring[0].height : r->height;
switch (e->type) {
case FT_MOTION:
say(r, "move %.1f %.1f", e->x, e->y);
break;
case FT_BUTTON: {
if (e->x >= 0 && e->y >= 0 && e->x < w && e->y < h) say(r, "move %.1f %.1f", e->x, e->y);
struct remote *by = r;
const int b = (int)e->button - BTN_LEFT;
if (b >= 0 && b < 8) {
if (e->pressed) {
g_pressed_on[b] = r->index;
} else {
struct remote *p = find(g_pressed_on[b]);
if (p && p->fd >= 0) by = p;
g_pressed_on[b] = -1;
}
}
say(by, "button %u %d", e->button, e->pressed ? 1 : 0);
wlr_log(WLR_INFO, "remote %d: button %u %s at %.0f,%.0f%s", r->index + 1, e->button, e->pressed ? "down" : "up",
e->x, e->y, by != r ? " (through the stream it went down on)" : "");
break;
}
case FT_SCROLL:
say(r, "scroll %.3f %.3f", e->dx, e->dy);
break;
default:
break;
}
}
void ft_remote_key(int index, uint32_t code, bool pressed) {
struct remote *r = find(index);
if (r) say(r, "key %u %d", code, pressed ? 1 : 0);
}
void ft_remote_blur(int index) {
struct remote *r = find(index);
if (r) say(r, "blur");
}
void ft_remote_tick(void) {
static const char *const names[] = {"hidden", "view", "focused"}; // enum ft_attention
const uint32_t t = now_ms();
for (int i = 0; i < FT_REMOTE_MAX; ++i) {
struct remote *r = &g_remotes[i];
if (r->used && r->restart_at && (int32_t)(t - r->restart_at) >= 0 && r->pid == 0 && r->fd < 0 &&
!host_busy(r) && !start_stream(r))
schedule_restart(r);
if (!r->used || r->fd < 0) continue;
const enum ft_attention a = ft_vr_screen_attention(r->index);
if ((int)a == r->attention) continue;
r->attention = (int)a;
say(r, "attention %s", names[a]);
}
}
bool ft_remote_child(pid_t pid, int status) {
for (int i = 0; i < FT_REMOTE_MAX; ++i) {
struct remote *r = &g_remotes[i];
if (r->pid != pid) continue;
r->pid = 0;
if (WIFEXITED(status)) wlr_log(WLR_INFO, "remote %d: ft-stream exited (%d)", r->index + 1, WEXITSTATUS(status));
else wlr_log(WLR_INFO, "remote %d: ft-stream killed (signal %d)", r->index + 1, WTERMSIG(status));
disconnect(r);
if (r->used && !r->restart_at) schedule_restart(r);
return true;
}
return false;
}
// Names, addresses and app ids go to ft-stream's command line: letters, digits and a few
// marks only (a display's device id is {GUID}).
static bool plain(const char *s, const char *extra) {
if (!*s) return false;
for (; *s; ++s)
if (!((*s >= 'a' && *s <= 'z') || (*s >= 'A' && *s <= 'Z') || (*s >= '0' && *s <= '9') || strchr(extra, *s)))
return false;
return true;
}
bool ft_remote_command(const char *cmd, char *reply, int size) {
if (strcmp(cmd, "remotes") == 0) {
int n = 0;
for (int i = 0; i < FT_REMOTE_MAX; ++i) n += g_remotes[i].used;
int at = snprintf(reply, size, "ok %d", n);
for (int i = 0; i < FT_REMOTE_MAX && at < size; ++i) {
const struct remote *r = &g_remotes[i];
if (!r->used) continue;
char state[32];
sscanf(r->state, "%31s", state);
at += snprintf(reply + at, size - at, " %d:%s:%s:%dx%d", r->index + 1, r->client, state,
r->have_ring ? r->ring[0].width : r->width, r->have_ring ? r->ring[0].height : r->height);
}
return true;
}
int number;
char word[16];
if (sscanf(cmd, "remote %d %15s", &number, word) != 2) return false;
if (number < FT_REMOTE_FIRST || number >= FT_REMOTE_FIRST + FT_REMOTE_MAX) {
snprintf(reply, size, "error remote screens are %d to %d", FT_REMOTE_FIRST, FT_REMOTE_FIRST + FT_REMOTE_MAX - 1);
return true;
}
const int index = number - 1;
struct remote *r = find(index);
if (strcmp(word, "stop") == 0) {
if (!r) return snprintf(reply, size, "error no remote screen %d", number), true;
stop_stream(r);
ft_vr_screen_destroy(index); // the panel goes before its textures
forget_ring(r);
r->used = false; // keeps its pid until the exit is reaped
r->restart_at = 0;
wlr_log(WLR_INFO, "remote %d: stopped", number);
return snprintf(reply, size, "ok"), true;
}
if (strcmp(word, "info") == 0) {
// Its stream's whole state ("lost can't connect"), for Remote Displays.
if (!r) return snprintf(reply, size, "error no remote screen %d", number), true;
return snprintf(reply, size, "ok %s", r->state), true;
}
if (strcmp(word, "start") != 0) return snprintf(reply, size, "error remote <N> start|stop|info"), true;
if (!g_vr) return snprintf(reply, size, "error no SteamVR (--no-vr)"), true;
struct remote c = {0};
int label_at = 0;
bool restored = false; // its panel is back where it was before a stop (vr.cpp, Parked)
if (sscanf(cmd, "remote %*d start %63s %63s %191s %dx%d %d %d %lf %n", c.client, c.host, c.app, &c.width, &c.height,
&c.fps, &c.bitrate, &c.metres, &label_at) != 8)
return snprintf(reply, size, "error remote <N> start <client> <host> <app> <W>x<H> <fps> <kbit/s> <metres> [label]"), true;
if (!plain(c.client, "._-") || !plain(c.host, ".:-") || !plain(c.app, "{}._:-"))
return snprintf(reply, size, "error client, host or app has other characters"), true;
if (c.width < 64 || c.height < 64 || c.width > 8192 || c.height > 8192 || c.fps < 1 || c.fps > 240 ||
c.bitrate < 0 || c.bitrate > 500000 || !(c.metres >= 0.15 && c.metres <= 20))
return snprintf(reply, size, "error size, fps, bitrate or width out of range"), true;
snprintf(c.label, sizeof c.label, "%s", label_at && cmd[label_at] ? cmd + label_at : c.client);
const bool same = r && !strcmp(r->client, c.client) && !strcmp(r->host, c.host) && !strcmp(r->app, c.app) &&
r->width == c.width && r->height == c.height && r->fps == c.fps && r->bitrate == c.bitrate;
if (same) return snprintf(reply, size, "ok running"), true;
if (!r) {
for (int i = 0; i < FT_REMOTE_MAX && !r; ++i)
if (!g_remotes[i].used && g_remotes[i].pid == 0) r = &g_remotes[i];
if (!r) return snprintf(reply, size, "error too many remote screens"), true;
if (!ft_vr_remote_create(index, c.label, c.metres, &restored))
return snprintf(reply, size, "error SteamVR made no panel (too many overlays?)"), true;
*r = (struct remote){.used = true, .index = index, .fd = -1, .shown = -1, .attention = -1};
} else {
stop_stream(r); // new settings: it starts over once the old one has gone
}
memcpy(r->client, c.client, sizeof r->client);
memcpy(r->host, c.host, sizeof r->host);
memcpy(r->app, c.app, sizeof r->app);
memcpy(r->label, c.label, sizeof r->label);
r->width = c.width, r->height = c.height, r->fps = c.fps, r->bitrate = c.bitrate, r->metres = c.metres;
if (!*g_stream) return snprintf(reply, size, "error no ft-stream next to ft-screens (or $FT_STREAM)"), true;
char log[PATH_MAX];
log_path(r, log, sizeof log);
const int f = open(log, O_WRONLY | O_CREAT | O_TRUNC | O_NOFOLLOW | O_CLOEXEC, 0600); // a fresh log
if (f >= 0) close(f);
// ft_remote_tick starts it: when its host is free, and once an old stream's exit is reaped.
r->restart_at = now_ms() | 1;
snprintf(r->state, sizeof r->state, "queued");
return snprintf(reply, size, restored ? "ok restored" : "ok"), true;
}
+34
View File
@@ -0,0 +1,34 @@
// Remote screens (remote.c): displays of other machines, streamed by ft-stream
// (stream/ft-stream.cpp) and shown as panels of their own, with every screen's controls.
#pragma once
#include <stdbool.h>
#include <stdint.h>
#include <sys/types.h>
#include <wayland-server-core.h>
#include "vr.h"
// Remote screens are numbered from here (from 1, like the others), clear of KWin's screens
// and spare outputs, so their numbers don't move when the desktop's screens change.
#define FT_REMOTE_FIRST 101
#define FT_REMOTE_MAX 16
// vr: connected to SteamVR (without it, no stream starts: there'd be no panel).
void ft_remote_init(struct wl_event_loop *loop, bool vr);
void ft_remote_shutdown(void);
// Whether screen `index` (from 0) is a remote screen that's running.
bool ft_remote_is(int index);
// Pointer input on a remote screen's panel.
void ft_remote_event(const struct ft_event *e);
// A key (linux KEY_* code) for remote screen `index`.
void ft_remote_key(int index, uint32_t code, bool pressed);
// Typing went elsewhere: the keys it holds on the host come up.
void ft_remote_blur(int index);
// Every tick: each stream hears how much of its screen you see, and lost ones restart.
void ft_remote_tick(void);
// A child process exited: true if it was one of ours.
bool ft_remote_child(pid_t pid, int status);
// "remote ..." and "remotes" commands; false for anything else.
bool ft_remote_command(const char *cmd, char *reply, int size);
+3
View File
@@ -1,3 +1,6 @@
// Keep assert() live even if someone builds the tests with -DNDEBUG: this
// file must always actually test.
#undef NDEBUG
#include <assert.h>
#include "../controller-click.h"
static struct ft_event event(enum ft_event_type t, bool down, double x, double y) {
Loaded 100 of 132 files, more files were not shown because too many files have changed in this diff. Show more