Hands: new hand models reach installs without a release

The pin moves out of models.sh into hands/models.pin. When hand tracking starts,
frametop-hands.service runs `hands/models.sh auto`: it reads models.pin from
GitHub (main for stable releases, experimental for experimental ones and clones
on that branch), and if it pins other models, fetches them into
models/.student-new, checks every sha256 and swaps the folder in whole with
one rename. At most every 15 minutes and within a minute; offline, a 404 or
anything not matching keeps the models it has.

A pin from GitHub is checked line by line before use: a 40-hex commit, a
plain folder name, file names from a fixed list, 64-hex sums, and a format
number, so a model that needs a newer ft-hands can be held back from older
installs. The Hugging Face repo stays fixed in the script.
HANDS_MODELS_UPDATE=0 turns it off; `hands/models.sh update` checks now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.dockerignore

@@ -16,3 +16,6 @@
 **/.env.*
 frametop-report-*.txt
 .frame-job.d/
+# the fine-tuned hand models: the image fetches its own pinned copy (hands/models.sh)
+hands/models/student/
+hands/models/.student-*

.github/workflows/release.yml

@@ -57,6 +57,9 @@ jobs:
               test -x /opt/frametop/bin/$p
             done
             test -f /src/frametop/pointer/driver/build/driver_ft_pointer.so
+            test -x /src/frametop/hands/build/ft-camd
+            test -x /src/frametop/hands/build/ft-hands
+            /src/frametop/hands/models.sh check
             test -x /src/frametop/pack/build/distrobox/install
             /src/frametop/gaze/tracker/build/venv/bin/python -c "import numpy, cv2"
             /src/frametop/voice/build/venv/bin/python -c "import numpy, sentencepiece, sherpa_onnx"

README.md

@@ -7,6 +7,7 @@ Frametop is a desktop for the Steam Frame that runs on the headset itself, with
 - **Your other computers' monitors.** Stream a Windows PC's or a Mac's monitors, or a virtual monitor at any size, as screens of their own, with the same controls. See [Remote displays](#remote-displays).
 - **Profiles.** Save where your screens are, which ones show, and which apps are open where. Switch to a profile from Display Settings, a key, or its launcher entry, or start the desktop in one.
 - **One mouse for all of SteamVR.** A Bluetooth mouse drives a small dot anchored in the room. It works the screens, the dashboard, Steam, and overlays, and hands the laser back when you pick up a controller.
+- **Your hands (experimental).** Hand tracking from the headset's own cameras: poke a screen to click, pinch and grab screens to move, resize, and throw them, and see your hands over the screens.
 - **Look and click (experimental).** In gaze mode the pointer goes where you look. Meta+J and Meta+K, or the mouse buttons, click and fine-tune, and each correction teaches the tracker. Calibration runs in the headset.
 - **Made for long sessions.** The displays turn off when the headset isn't used, even on a stand that makes it seem worn. It stays awake on the charger, and you can reach the desktop remotely over VNC.
 
@@ -36,7 +37,7 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
 
    The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.1 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
 
-   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you five things along the way: whether to install voice dictation (yes by default; it downloads about 320 MB of speech models), gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
+   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you six things along the way: whether to install voice dictation (yes by default; it downloads about 320 MB of speech models), hand tracking (experimental, yes by default), gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. Hand tracking, the eye tracker, and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
 
 After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings. SteamOS's own single-screen desktop is still there, as Native Desktop in the same list.
 
@@ -144,6 +145,21 @@ In gaze mode the pointer goes where you look, and the mouse or the keyboard does
 
 By default the mouse only corrects: while the gaze has the pointer, moving the mouse does nothing until you hold a button. Each correction before a click teaches the tracker where it was off. A correction past the learning limit (55 degrees by default, about half of what you can see) starts a quick check instead. Calibrate, on the Gaze page, runs a full calibration in a panel in front of you: look at each dot and click. Check headset fit shows how well the eye tracker sees your eyes. [gaze/README.md](gaze/README.md) has the details.
 
+### Hand tracking (experimental)
+
+Hand tracking uses the headset's own cameras, with no controllers. The installer offers it (or run `hands/run.sh install` later; it needs `sudo` once, to let its camera broker read the cameras). Installed, it runs while the Frametop desktop does, and parks while Frametop is paused for a VR game. It uses a lot of the headset's CPU.
+
+| Do this | To get this |
+| --- | --- |
+| Poke a screen with your index finger | A click there |
+| Pinch a screen's bar, edge, or corner, and move | Moves or resizes the screen |
+| Hold a screen like a card, or in a fist | Carries it; turn your wrist to turn it. Flick it away, or let go fast to throw it |
+| Hold both hands on a screen and pull them apart | Stretches it |
+| Hold an open hand out to a far screen, then make a fist and pull it back ("grab that") | Brings it to arm's reach |
+| Turn a hand palm up toward your face, look at it, and pinch the ring that shows | Opens the Steam menu |
+
+The Hands page of Frametop Input Settings sets when it runs (with Frametop, always with SteamVR, or off) and switches each hand interaction on or off, live. `ft-handsctl status` shows how it's doing. [hands/README.md](hands/README.md) has the details.
+
 ### Leave the headset on a stand and reach it remotely
 
 To keep the Frame on and connected while you're not wearing it, for SSH, remote desktop, or anything else running on it, open the Power tab in Frametop Display Settings:
@@ -234,7 +250,7 @@ A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland com
 | `power/` | ft-powerd: turns the displays off while the headset isn't used. |
 | `voice/` | Frametop Voice: dictation and wake words, recognised on the headset; the input relay types the text. See [voice/README.md](voice/README.md). |
 | `gaze/` | Gaze mode (experimental): the gaze service, its calibration panel, our own eye tracker, and the gaze probe. See [gaze/README.md](gaze/README.md). |
-| `hands/` | Hand tracking (experimental, deferred: the installer doesn't offer it). See [hands/README.md](hands/README.md). |
+| `hands/` | Hand tracking (experimental): the camera broker and the tracker. See [hands/README.md](hands/README.md). |
 | `display-settings/`, `input-settings/` | The two settings apps (Kirigami, Python). |
 | `remote/` | Frametop Remote Access, the app that turns remote desktop over VNC on and off. |
 | `remote-displays/` | Frametop Remote Displays, the app that adds other computers' monitors as screens. |

docs/reference.md

@@ -312,15 +312,15 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke
 
 [gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool.
 
-## Hand tracking (experimental, deferred)
+## Hand tracking (experimental)
 
-Deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it. It still builds and runs, installed by hand, for working on it.
+It costs a lot of the headset's CPU. `install.sh` offers it (yes by default, `--no-hands` skips it, and it needs `sudo` for ft-camd's file capabilities), and a release brings `ft-camd` and `ft-hands` built in its image. Installed, it runs while the Frametop desktop does (`HANDS_MODE=frametop`).
 
 Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen (a round hole per hand, `HANDS_MATTE=circle`; `mesh` and `capsules` are the older shapes). ft-handview (`frametop-handview.service`) draws your hands as a 3D hand model posed by the tracking: over games and the void with Room View off, and over the hole with it on (`HANDS_VIEW_ROOMVIEW=1`, the default). `HANDS_VIEW_MODEL` picks the model: `asset` (the default), `babylon`, `mrtk3`, `godot`, `mrtk2`, `robot` (a robot hand built in code from the tracked skeleton: light shells, dark joints, cyan accents), or `capsules` for the older tubes; `HANDS_VIEW_STYLE=ghost|solid` the look. The models, in `screens/assets/hands` with their licences: the WebXR Input Profiles' generic hand (`asset`; MIT, Copyright (c) 2019 Amazon), Babylon.js's hand meshes by Patrick Ryan (`babylon`; CC BY 4.0, https://github.com/BabylonJS/Assets), MRTK3's OpenXR hand (`mrtk3`; BSD-3-Clause, Copyright (c) 2025, Mixed Reality Toolkit Contributors), Godot XR Tools' hands by DigitalN8m4r3 (`godot`; CC0 1.0) and MRTK2's rigged hand (`mrtk2`; MIT, Copyright (c) Microsoft Corporation). The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode your eyes aim and your hands click, Vision Pro style: a pinch clicks where you were looking as it began, and the hand ray stays off (`POINTER_GAZE_HAND_RAY=1` brings it back beside the gaze). Pinch, hold and move the hand a few centimetres to nudge the pointer slowly onto the target, then let go to click there; the correction teaches the gaze tracker, as the mouse's do. Pinch and hold still for half a second to press and drag with the hand. A middle pinch right-clicks the same way. Hold a pinch with one hand and pinch with the other to hold both buttons, left and right, each until its pinch opens. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`. With `POINTER_HANDS=1` and gaze mode off, a bare hand raised and pointing also shows a ray that drives the pointer like a controller's laser (a pinch clicks, a middle pinch right-clicks or scrolls, reaching out and in pushes and pulls a carried panel), a controller used recently takes over its own hand only, and during a paused VR game hand tracking is parked rather than stopped, waking when the dashboard opens. Details: [hands/README.md](../hands/README.md), "Pinches and grips in the pointer". When tracking goes wrong, `ft-handsctl flag [note]` (or a voice command through frame-voice, "flag that") saves the last 12 seconds of hand crops and what the tracker made of them for the offline tools: ft-hands keeps them in RAM (`HANDS_FLAG_BUFFER=1`, about 50 MB), the drawn hands pulse cyan when a flag is taken (`HANDS_FLAG_FLASH`, `HANDS_FLAG_TONE`), and flags go to `HANDS_FLAG_DIR`. See [hands/README.md](../hands/README.md), "Flags".
 
 - `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. The Hand Recorder's installer (`hands/rec/install.sh`) sets them the same way. `hands/run.sh uncaps` takes them back while neither the services nor the Hand Recorder is installed. `hands/run.sh uninstall` and `hands/rec/install.sh uninstall` run it after removing their own part, and `uninstall.sh` always takes them back.
 - `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring.
-- `HANDS_MODE` says when they run: `off` (the default) only after `ft-handsctl on`, until `ft-handsctl off` or SteamVR stops; `frametop` while the Frametop desktop runs; `always` whenever SteamVR runs, in games too. Frametop Input Settings' Hands page sets it and applies it at once (`hands/handsmode.py`), next to the hand cutouts' margin (`HANDS_MATTE_MARGIN`, live). The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live.
+- `HANDS_MODE` says when they run: `off` (the default until install.sh installs hand tracking, which sets `frametop`) only after `ft-handsctl on`, until `ft-handsctl off` or SteamVR stops; `frametop` while the Frametop desktop runs; `always` whenever SteamVR runs, in games too. Frametop Input Settings' Hands page sets it and applies it at once (`hands/handsmode.py`), next to the hand cutouts' margin (`HANDS_MATTE_MARGIN`, live). The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live.
 - Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (`auto`, the default: ft-hands tells from the hands when some SteamVR restart has swapped the side cameras' names, and fixes them; `0` or `1` force them, and `hands/tools/check_sides.py --ring` tells which is right), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, typing's `POINTER_PINCH_TYPING`, `POINTER_TYPING_RAISE`, `POINTER_TYPING_BELOW`, `POINTER_TYPING_POSTURE`, `POINTER_TYPING_RECENT`, the gaze pinch's `POINTER_GAZE_PINCH`, `POINTER_GAZE_HAND_RAY`, `POINTER_GAZE_PINCH_DEADZONE`, and `POINTER_GAZE_PINCH_GAIN`, the hand ray's `POINTER_HAND_RAY`, `POINTER_RAY_DEADZONE`, `POINTER_RAY_PUSH`, `POINTER_RAY_SCROLL`, `POINTER_RAY_RELEASE`, `POINTER_RAY_GRACE`, its smoothing and taps (`POINTER_RAY_SMOOTH_MINCUTOFF`, `POINTER_RAY_SMOOTH_BETA`, `POINTER_RAY_SMOOTH_HOLD`, `POINTER_RAY_STEADY`, `POINTER_RAY_TAP`, `POINTER_RAY_TAP_DEADZONE`), its body and hand models (`POINTER_RAY_TORSO_WRISTS`, `POINTER_RAY_HAND_WEIGHT`, `POINTER_RAY_HAND_PITCH`, `POINTER_RAY_HAND_YAW`, `POINTER_RAY_ANCHOR`), and ft-screens' near reach (`HANDS_NEAR_REACH`, 0.075 m), pokes (`HANDS_POKE_ANGLE`, `HANDS_POKE_TRAVEL`, `HANDS_POKE_REST_BELOW`, `HANDS_POKE_REST_NEAR`) and near grabs (the border band `HANDS_GRAB_BAND`, a pinch on the content clicking `HANDS_PINCH_CLICK`, the card pinch `HANDS_CARD_PINCH`, the fist grab `HANDS_FIST_GRAB` and its reach `HANDS_FIST_REACH`, a second hand joining a held panel `HANDS_JOIN_REACH`, two hands turning a panel on every axis `HANDS_STRETCH_TURN`, the round buttons' hit area for hands `HANDS_BUTTON_REACH` and pinches pressing them `HANDS_PINCH_BUTTONS`, `HANDS_GRAB_SMOOTH_MINCUTOFF`, `HANDS_GRAB_SMOOTH_BETA`, `HANDS_GRAB_DEADZONE`, the wrist turning a carried panel: `HANDS_GRAB_ROTATE`, `HANDS_GRAB_ROTATE_DEADZONE`, holding through tracking loss `HANDS_HOLD_LOST_MS`, ignoring tracking no hand makes `HANDS_TRACK_GATE`), throws (the flick `HANDS_FLICK`, `HANDS_FLICK_SPEED`, `HANDS_FLICK_OPENING`, a carry let go fast `HANDS_THROW`, `HANDS_THROW_SPEED`, the push `HANDS_PUSH`, `HANDS_PUSH_SPEED`, where they go, `HANDS_THROW_DISTANCE`, and at what width, `HANDS_SEND_AWAY_WIDTH`), its "grab that" gesture (`HANDS_GRAB_THAT`, `HANDS_GRAB_THAT_DISTANCE`, `HANDS_GRAB_THAT_WIDTH`, which panels: `HANDS_GRAB_THAT_FROM`, `HANDS_GRAB_THAT_MARGIN`, `HANDS_GRAB_THAT_MARGIN_DEG`, `HANDS_GRAB_THAT_FACING`), the soft white halo round a gesture's target panel (`HANDS_TARGET_GLOW`), and the soft glow round the corner, edge or button a hand is at (`HANDS_EDGE_GLOW`). The example config explains each.
 
 Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md).

framedrop/README.md

@@ -1,6 +1,6 @@
 # Install with FrameDrop (proof of concept)
 
-[FrameDrop](https://framedropvr.com) is a Windows app that sideloads onto a Steam Frame: it copies a build to the headset and adds it to the Steam library. Issue #25 asks for an "Install with FrameDrop" button. Frametop isn't an app FrameDrop can copy over as is: it installs user services, a SteamVR driver, and a container, and two optional parts need sudo. So FrameDrop installs a small installer instead. Playing "Frametop" from the library opens a window that asks what to install, and your password for the parts that need it, then installs and shows its progress. A release's Frametop.zip (about 1.1 GB, built by CI on a tag, see [pack/README.md](../pack/README.md), Releases) carries Frametop built, as an image, and installs it with `install-release.sh`: nothing compiles on the headset and nothing else downloads. The same zip works unpacked on the headset. A test zip (a few KB) clones Frametop with `get.sh` instead.
+[FrameDrop](https://framedropvr.com) is a Windows app that sideloads onto a Steam Frame: it copies a build to the headset and adds it to the Steam library. Issue #25 asks for an "Install with FrameDrop" button. Frametop isn't an app FrameDrop can copy over as is: it installs user services, a SteamVR driver, and a container, and three optional parts need sudo. So FrameDrop installs a small installer instead. Playing "Frametop" from the library opens a window that asks what to install, and your password for the parts that need it, then installs and shows its progress. A release's Frametop.zip (about 1.1 GB, built by CI on a tag, see [pack/README.md](../pack/README.md), Releases) carries Frametop built, as an image, and installs it with `install-release.sh`: nothing compiles on the headset and nothing else downloads. The same zip works unpacked on the headset. A test zip (a few KB) clones Frametop with `get.sh` instead.
 
 Nothing here is published yet: no release, no button.
 
@@ -26,7 +26,7 @@ It uses Valve's SteamOS Devkit path: pair once with the headset's devkit service
 
 1. In the container, it starts itself again on the host with `flatpak-spawn --host`.
 2. It clears Steam's preload and library paths, and opens `installer/progress.py` (GTK 4 and libadwaita).
-3. The window asks which optional parts to install: our own eye tracker (on by default) and the Bluetooth fixes. Both need sudo, so it asks for your SteamOS password and checks it with `sudo -v`. If your user has no password (SteamOS starts without one), it says how to set one and leaves both out.
+3. The window asks which optional parts to install: voice dictation (on by default), hand tracking (on by default), our own eye tracker (on by default), and the Bluetooth fixes. All but voice dictation need sudo, so it asks for your SteamOS password and checks it with `sudo -v`. If your user has no password (SteamOS starts without one), it says how to set one and leaves those three out.
 4. It runs the zip's `install-release.sh --yes` (a test zip: `get.sh --yes`) in a transient user service, `frametop-framedrop-install`. The service is used because Steam ends the title's whole process tree when it's quit, and starts it with an OOM score of 900. Opened from a VR desktop (the zip unpacked by hand), it still uses the user's real bus and runtime folder for the service.
 5. It follows the service's log, shows the steps as a progress bar, and reports the result. Closing it leaves the install running. Playing the title again reattaches.
 

framedrop/installer/progress.py

@@ -101,7 +101,7 @@ def in_unit(pid):
         return False
 
 
-def install_command(eye_tracker, bluetooth, voice=True):
+def install_command(eye_tracker, bluetooth, voice=True, hands=True):
     if RELEASE:
         args = [str(HERE / "install-release.sh"), "--yes"]
         if DRY_RUN:
@@ -112,6 +112,8 @@ def install_command(eye_tracker, bluetooth, voice=True):
             args += ["--clone-only", "--dir", str(STATE / "dry-run")]
     if not voice:
         args.append("--no-voice")
+    if not hands:
+        args.append("--no-hands")
     if not eye_tracker:
         args.append("--no-eye-tracker")
     if bluetooth:
@@ -304,10 +306,12 @@ class Window(Adw.ApplicationWindow):
     def show_choices(self):
         what = f"Frametop {release_version()}" if RELEASE else "Frametop from GitHub"
         self.detail.set_label(f"This installs {what}: the multi-screen desktop, the 3D mouse, gaze "
-                              "mode, and Frametop's settings apps. Optional: voice dictation, and two "
+                              "mode, and Frametop's settings apps. Optional: voice dictation, and three "
                               "parts that need your SteamOS password (sudo):")
         self.voice = Gtk.CheckButton(label="Voice dictation: a Right Ctrl tap or holding the left d-pad down, and "
                                            "what you say is typed (downloads about 320 MB)", active=True)
+        self.hands = Gtk.CheckButton(label="Hand tracking (experimental): poke, pinch and grab the screens with "
+                                           "your hands. It uses a lot of the headset's CPU", active=True)
         self.eye = Gtk.CheckButton(label="Our own eye tracker for gaze mode (more accurate than SteamVR's)",
                                    active=True)
         self.bt = Gtk.CheckButton(label="Bluetooth fixes (LE mice and keyboards, like the Swiftpoint Z3, "
@@ -323,27 +327,26 @@ class Window(Adw.ApplicationWindow):
         self.go.connect("clicked", self.install)
         self.pw.connect("activate", self.install)
         pw_row, keypad = keypad_row(self.pw)
-        self.choices = [self.voice, self.eye, self.bt, pw_row, keypad, pw_note, self.error, self.go]
+        self.choices = [self.voice, self.hands, self.eye, self.bt, pw_row, keypad, pw_note, self.error, self.go]
         if not password_set():
-            for c in (self.eye, self.bt):
+            for c in (self.hands, self.eye, self.bt):
                 c.set_active(False)
                 c.set_sensitive(False)
             pw_row.set_visible(False)
             keypad.set_visible(False)
-            pw_note.set_label("Your user has no password, so sudo can't run and these two can't be "
+            pw_note.set_label("Your user has no password, so sudo can't run and these three can't be "
                               "installed from here. To set one, run passwd in Konsole; then play "
                               "Frametop again, or install them later from a terminal "
-                              "(gaze/tracker/install.sh, setup/bluetooth/install.sh).")
-        for c in (self.eye, self.bt):
-            c.connect("toggled", lambda *_: pw_row.set_sensitive(self.eye.get_active() or self.bt.get_active()))
+                              "(hands/run.sh install, gaze/tracker/install.sh, setup/bluetooth/install.sh).")
+        for c in (self.hands, self.eye, self.bt):
+            c.connect("toggled", lambda *_: pw_row.set_sensitive(self.needs_password()))
         for w in self.choices:
             self.box.append(w)
 
     def install(self, *_):
         if self.checking:
             return
-        needs = self.eye.get_active() or self.bt.get_active()
-        if not needs:
+        if not self.needs_password():
             return self.begin(None)
         text = self.pw.get_text()
         if not text:
@@ -370,12 +373,16 @@ class Window(Adw.ApplicationWindow):
 
         threading.Thread(target=check, daemon=True).start()
 
+    def needs_password(self):
+        return self.hands.get_active() or self.eye.get_active() or self.bt.get_active()
+
     def say(self, text):
         self.error.set_label(text)
         self.error.set_visible(True)
 
     def begin(self, pw):
-        args = install_command(self.eye.get_active(), self.bt.get_active(), self.voice.get_active())
+        args = install_command(self.eye.get_active(), self.bt.get_active(), self.voice.get_active(),
+                               self.hands.get_active())
         if pw is not None:
             try:
                 self.askpass = Askpass(self.ask_again)

get.sh

@@ -23,17 +23,17 @@
 #   --version V     with --release: that release (tag vV), not the newest
 #   --zip FILE|URL  with --release: this Frametop.zip instead of GitHub's newest
 #   --any-steamos   with --release: install even on a SteamOS build the release breaks on
-#   --yes, --no-voice, --no-eye-tracker, --no-bluetooth, --bluetooth   passed to install.sh (--yes also
+#   --yes, --no-voice, --no-hands, --no-eye-tracker, --no-bluetooth, --bluetooth   passed to install.sh (--yes also
 #                   answers this script's questions: the version already there, or stable)
 set -euo pipefail
 
 usage() {
   cat <<'EOF'
 usage: get.sh [--stable | --experimental | --branch NAME] [--dir DIR] [--clone-only] [--yes]
-              [--no-voice] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
+              [--no-voice] [--no-hands] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
        get.sh --release [--stable | --experimental | --version V | --zip FILE|URL]
-              [--any-steamos] [--dir DIR] [--clone-only] [--yes] [--no-voice] [--no-eye-tracker]
-              [--no-bluetooth | --bluetooth]
+              [--any-steamos] [--dir DIR] [--clone-only] [--yes] [--no-voice] [--no-hands]
+              [--no-eye-tracker] [--no-bluetooth | --bluetooth]
 piped: curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- [options]
 EOF
 }
@@ -129,7 +129,7 @@ main() {
       --version) want=${2:?--version needs a version}; shift ;;
       --any-steamos) any=1 ;;
       --yes) yes=1; pass+=("$1") ;;
-      --no-bluetooth|--bluetooth|--no-eye-tracker|--no-voice) pass+=("$1") ;;
+      --no-bluetooth|--bluetooth|--no-eye-tracker|--no-voice|--no-hands) pass+=("$1") ;;
       -h|--help) usage; return 0 ;;
       *) echo "unknown option: $1" >&2; usage >&2; return 2 ;;
     esac

hands/.gitignore

@@ -1,6 +1,10 @@
 # Model sources that tools/convert_models.py downloads; the converted ncnn models are kept
 models/onnx/
 models/*.task
+# the fine-tuned models hands/models.sh fetches (their own licence, CC BY-NC 4.0: not in the repo)
+models/student/
+# hands/models.sh update's download folder, lock and last check
+models/.student-*
 # frame-job settings (where replays run, and the lab's capture folder): personal, not shipped
 .frame-job
 dumps/

hands/README.md

@@ -1,6 +1,6 @@
-# Hands (experimental, deferred)
+# Hands (experimental)
 
-Hand tracking from the headset's own cameras. It's deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it and the README doesn't list it. It still builds and runs, installed by hand (below), for working on it.
+Hand tracking from the headset's own cameras. It's experimental and costs a lot of the headset's CPU. `install.sh` offers it (yes by default; `--no-hands` skips it), a release brings it built, and installed, it runs while the Frametop desktop does (`HANDS_MODE=frametop`). `hands/run.sh install` (below) installs it later.
 
 It serves two things in Frametop:
 
@@ -14,9 +14,9 @@ Two programs, each a user service that stops when SteamVR does:
 - `ft-camd` (`camd/`, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host.
 - `ft-hands` (`track/`, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container.
 
-`hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services, and links `hands/ft-handsctl` into `~/.local/bin`. `install.sh` doesn't install it. When they run is `HANDS_MODE` in `~/.config/frametop.conf`, set on the Hands page of Frametop Input Settings or with `ft-handsctl mode`; `hands/handsmode.py` does the starting and stopping, for these two and the hand view (`frametop-handview.service`, when it's installed):
+`hands/run.sh install` builds them (a release brings them built), gives ft-camd its capabilities, installs both services, and links `hands/ft-handsctl` into `~/.local/bin`. `install.sh` runs it when you choose hand tracking, then sets `HANDS_MODE=frametop` unless it's `always`. When they run is `HANDS_MODE` in `~/.config/frametop.conf`, set on the Hands page of Frametop Input Settings or with `ft-handsctl mode`; `hands/handsmode.py` does the starting and stopping, for these two and the hand view (`frametop-handview.service`, when it's installed):
 
-- `off` (the default): nothing starts them. `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it, as before; it stops with SteamVR.
+- `off` (the default before hand tracking is installed): nothing starts them. `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it, as before; it stops with SteamVR.
 - `frametop`: they run while the Frametop desktop does. The session (`session/frametop-session.sh`, screens backend) starts them once ft-screens is up and stops them as it ends.
 - `always`: they're wanted by `steamvr.service` (`systemctl --user add-wants`), so they run whenever SteamVR does, in games and the SteamVR dashboard too, with or without the desktop. Pausing Frametop for a VR game leaves them running and unparked.
 
@@ -50,7 +50,8 @@ Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides
 - `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (`sides.json`), so recordings are labelled right. The example config said `0` until 2026-10-05; `scripts/conf-migrate.sh` (run by `install.sh` and `hands/rec/install.sh`) turns that untouched line into `auto`.
 - `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
 - `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
-- `HANDS_MODELS` (unset: `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to run fine-tuned models, such as the ones trained on the hand dataset, without passing options to every launcher. Those folders usually lack the `*-int8` files, so `--int8` won't load them.
+- `HANDS_MODELS` (unset: `hands/models/student`, the fine-tuned models, when `hands/models.sh` fetched them, else `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to try other models without passing options to every launcher. The fine-tuned folders lack the `*-int8` files, so `--int8` without `--models` uses the stock ones.
+- `HANDS_MODELS_UPDATE=1`: when hand tracking starts, fetch the fine-tuned models GitHub's branch pins if they changed, without a new release (`models.sh` below). `0` keeps the ones it came with.
 - `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
 - `HANDS_SKELETON=1`: fit a skeleton with your bone lengths and publish it (see "Skeleton" below); `0` publishes the filtered landmarks as before. As `--skeleton`.
 - `HANDS_PINCH_DIP=0`: `1` also starts a pinch on a quick dip of the thumb-index distance that never reaches 2 cm (see the pinch under "Gestures" below). As `--pinch-dip`.
@@ -298,6 +299,7 @@ Python, with NumPy and OpenCV. `setup/dev-container.sh` doesn't install them, be
 - `tools/depth_report.py DEPTH`: the depth measures above.
 - `tools/cut_sets.py REC OUT [--sets N | --at I,J,...]`: copies a few frame sets (by default 8, spread evenly) out of a recording into a small one, to look at or check elsewhere without moving gigabytes. Plain Python, so it also runs on the Frame's host.
 - `tools/convert_models.py`: how `models/ncnn` was made from the OpenCV Zoo ONNX ports of MediaPipe's models (see `models/NOTICE`).
+- `models.sh`: fetches the fine-tuned models into `models/student` (`hands/build.sh` runs it, so source installs and the release image both get them): MediaPipe's two models retrained on the Frame's infrared cameras with the [Frametop Hands](https://huggingface.co/datasets/DeeJanuz/frametop-hands) dataset, from a fixed commit of [DeeJanuz/frametop-hands-models](https://huggingface.co/DeeJanuz/frametop-hands-models), each file checked against its SHA-256. They're licensed CC BY-NC 4.0 (non-commercial), not MIT, so they aren't in this repository; `models/student/LICENSE` and `README.md` come with them. `hands/models.pin` pins them: the commit, the version folder and every file's sum (frame-hands' `model-repo/pin.py` writes it); `hands/models.sh check` checks them against it. Installs don't need a release for a new version: when hand tracking starts, `frametop-hands.service` runs `hands/models.sh auto`, which reads `hands/models.pin` from GitHub (main for a stable release, experimental for an experimental one or a clone on that branch; at most every 15 minutes, within a minute) and, if it pins other models, fetches them into `models/.student-new`, checks every sum, and swaps that folder in whole. Offline, or anything not matching, it keeps the models it has. A pin only names the files above, and its `format` says which ft-hands it needs: raise it when a model needs a newer one, and installs without it keep theirs. `HANDS_MODELS_UPDATE=0` turns it off; `hands/models.sh update` checks now.
 
 To try the hand cutouts without restarting the desktop, `screens/build/ft-handtest [--distance m] [--width m] [--seconds s] [--matte circle|mesh|capsules] [--margin deg]` (built by `screens/build.sh`, run in the dev container, with hand tracking on) shows a test panel of its own, a light grid 1 m wide and 0.8 m ahead by default, and cuts your hands out of it the way ft-screens cuts them out of the screens. Every 2 s it prints the composite's cost.
 

hands/build.sh

@@ -1,11 +1,13 @@
 #!/usr/bin/env bash
 # Build hand tracking in the dev container on the Frame, into hands/build/: ft-camd and ft-hands,
 # and with --tools also ft-handreplay and ft-ringplay. The first build fetches ncnn and builds
-# it (a few minutes); NCNN=DIR, an ncnn install already on the Frame, skips that.
+# it (a few minutes); NCNN=DIR, an ncnn install already on the Frame, skips that. It also fetches
+# the fine-tuned hand models (hands/models.sh) into hands/models/student, which ft-hands prefers.
 # A rebuilt ft-camd has lost its capabilities: hands/run.sh install sets them again.
 set -euo pipefail
 root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
 targets=all
 [ "${1:-}" = --tools ] && targets="all tools"
 "$root/scripts/sync.sh" >/dev/null
+"$root/hands/models.sh" fetch
 exec "$root/scripts/frame.sh" -C hands "make -s ${NCNN:+NCNN=$NCNN} $targets && echo built \$(ls build/ft-* | tr '\n' ' ')"

hands/frametop-hands.service

@@ -10,7 +10,10 @@ PartOf=steamvr.service
 # In the container it was built in (scripts/in-box: dev for a clone, the release's own for a
 # release; it's built against Fedora's libraries). It reads ft-camd's ring and
 # writes /run/user/UID/frametop-hands/hands and gestures. Settings: HANDS_* in ~/.config/frametop.conf.
+# Before it starts, newer fine-tuned models if its branch pins them (hands/models.sh auto: at most
+# every 15 minutes, never more than a minute, offline or failing it starts with the ones it has).
 ExecStartPre=-/usr/bin/pkill -x ft-hands
+ExecStartPre=-/usr/bin/timeout 60 @REPO@/hands/models.sh auto
 ExecStart=@REPO@/scripts/in-box @REPO@/hands/build/ft-hands --status 60
 ExecStopPost=-/usr/bin/pkill -x ft-hands
 Restart=always

hands/models.pin

@@ -0,0 +1,16 @@
+# The fine-tuned hand models ft-hands uses by default (hands/models.sh): a version folder of the
+# Hugging Face repo DeeJanuz/frametop-hands-models at a fixed commit, and every file's sha256.
+# Installs read this file from GitHub (main or experimental, by their channel) when hand tracking
+# starts, so a new pin pushed there reaches them without a release. "format" is what ft-hands
+# needs to run them: raise it when a model needs a newer ft-hands, and older installs keep theirs.
+# Made by frame-hands' model-repo/pin.py.
+format 1
+rev 41afd78fbd4527456fc4cbd840ffc9f39fdc939a
+model v13b-p1
+file NOTICE 0e6b4fa8ea37665cb699abc36fcf1ee0ffaa63a78dfee788be6613dea6607f84
+file hand.ncnn.bin d6a6f00d845fb9ee490276d84b1e978da884361c372614546ed2b1b4d2f5c449
+file hand.ncnn.param 54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd
+file palm.ncnn.bin 9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909
+file palm.ncnn.param c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e
+file LICENSE 41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0
+file README.md 29c6c5460e47412a9a37b22d6a7c8a9ed266a872bbcce93d3867f9e56cfe0ca1

hands/models.sh

@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+# Fetch the fine-tuned hand models into hands/models/student/ on the Frame (in the image build,
+# here: pack/Containerfile runs hands/build.sh, which runs this). hands/models.pin names them: a
+# fixed commit of the Hugging Face repo below, and every file's sha256. ft-hands uses them when
+# they're there (HANDS_MODELS overrides; hands/models/ncnn, the stock MediaPipe models, otherwise).
+#   palm.ncnn.*, hand.ncnn.*   MediaPipe's palm detector and hand landmark model, fine-tuned on the
+#                              Frametop hand dataset (DeeJanuz/frametop-hands)
+#   LICENSE, README.md         their licence (CC BY-NC 4.0, not Frametop's MIT) and model card
+# Usage: hands/models.sh [fetch]   fetch what's missing or doesn't match models.pin (the default)
+#        hands/models.sh check     check every file against the pin it was fetched by (exit 1 if not)
+#        hands/models.sh update    fetch the models the install's branch pins on GitHub now (main for
+#                                  a stable release, experimental for an experimental one or a
+#                                  clone on that branch), if they changed. Into a folder of its own,
+#                                  swapped in once every file matches. Offline or nothing new: no-op.
+#        hands/models.sh auto      update, as frametop-hands.service runs it when hand tracking
+#                                  starts: not with HANDS_MODELS_UPDATE=0 in ~/.config/frametop.conf,
+#                                  and at most every 15 minutes.
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+. "$root/scripts/_env.sh"
+case ${1:-fetch} in fetch | check | update | auto) ;; *) sed -n '9,18p' "$0" >&2; exit 2 ;; esac
+
+on_frame_script "${1:-fetch}" "$FRAME_REPO" <<'SCRIPT'
+set -euo pipefail
+mode=$1 tree=$2
+models=$tree/hands/models dir=$tree/hands/models/student
+hf=DeeJanuz/frametop-hands-models gh=Frametop/frametop
+names=" NOTICE hand.ncnn.bin hand.ncnn.param palm.ncnn.bin palm.ncnn.param LICENSE README.md "
+
+# read_pin FILE: check every line of a pin, then set rev, model and files ("name sha256" each).
+# Anything unexpected fails it: a pin from GitHub only names files this list allows.
+read_pin() {
+  local key a b n format=
+  rev= model= files=()
+  while read -r key a b; do
+    case $key in '' | '#'*) continue ;; esac
+    case $key in
+      format) [ "$a" = 1 ] && [ -z "$b" ] || return 1; format=$a ;;
+      rev) [[ $a =~ ^[0-9a-f]{40}$ ]] && [ -z "$b" ] || return 1; rev=$a ;;
+      model) [[ $a =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$ ]] && [ -z "$b" ] || return 1; model=$a ;;
+      file) [[ $names == *" $a "* && $b =~ ^[0-9a-f]{64}$ ]] || return 1; files+=("$a $b") ;;
+      *) return 1 ;;
+    esac
+  done <"$1"
+  [ -n "$format" ] && [ -n "$rev" ] && [ -n "$model" ] || return 1
+  for n in hand.ncnn.bin hand.ncnn.param palm.ncnn.bin palm.ncnn.param; do
+    [[ " ${files[*]} " == *" $n "* ]] || return 1
+  done
+}
+
+matches() { [ -f "$1" ] && [ "$(sha256sum <"$1" | cut -d' ' -f1)" = "$2" ]; }
+
+# fetch_into DIR: fetch the pin's files that are missing there or don't match.
+fetch_into() {
+  local d=$1 entry name sum src
+  mkdir -p "$d"
+  for entry in "${files[@]}"; do
+    read -r name sum <<<"$entry"
+    matches "$d/$name" "$sum" && continue
+    src=https://huggingface.co/$hf/resolve/$rev/$model/$name
+    case $name in LICENSE | README.md) src=https://huggingface.co/$hf/resolve/$rev/$name ;; esac
+    echo "fetching hand model $name ($model)"
+    if ! curl -fsSL --retry 3 --connect-timeout 10 --max-filesize 100000000 -o "$d/$name.part" "$src"; then
+      rm -f "$d/$name.part"; return 1
+    fi
+    if ! matches "$d/$name.part" "$sum"; then
+      echo "hand models: $name's sha256 doesn't match the pin; not kept" >&2
+      rm -f "$d/$name.part"; return 1
+    fi
+    mv "$d/$name.part" "$d/$name"
+  done
+}
+
+stamp() { echo "$hf@${rev::7} $model" >"$1/VERSION"; }
+
+case $mode in
+  fetch)
+    read_pin "$tree/hands/models.pin" || { echo "hand models: hands/models.pin isn't a valid pin" >&2; exit 1; }
+    fetch_into "$dir"
+    cp "$tree/hands/models.pin" "$dir/PIN"; stamp "$dir"
+    echo "hand models: $model in $dir"
+    ;;
+  check)
+    # Against the pin they were fetched by: an update may have replaced the release's own.
+    pin=$dir/PIN; [ -f "$pin" ] || pin=$tree/hands/models.pin
+    [ -d "$dir" ] || { echo "hand models: $dir missing" >&2; exit 1; }
+    read_pin "$pin" || { echo "hand models: $pin isn't a valid pin" >&2; exit 1; }
+    bad=0
+    for entry in "${files[@]}"; do
+      read -r name sum <<<"$entry"
+      matches "$dir/$name" "$sum" || { echo "hand models: $name missing or changed" >&2; bad=1; }
+    done
+    [ "$bad" = 0 ] || exit 1
+    echo "hand models: $model in $dir, all files match"
+    ;;
+  update | auto)
+    if [ "$mode" = auto ]; then
+      conf=$(sed -n 's/#.*//; s/^[[:space:]]*HANDS_MODELS_UPDATE[[:space:]]*=[[:space:]]*//p' ~/.config/frametop.conf 2>/dev/null | tail -1)
+      conf=${FT_HANDS_MODELS_UPDATE:-$conf}
+      [ "$(echo $conf | tr -d "\"'")" = 0 ] && { echo "hand models: not updating (HANDS_MODELS_UPDATE=0)"; exit 0; }
+      [ -n "$(find "$models/.student-checked" -mmin -15 2>/dev/null)" ] && exit 0
+    fi
+    if [ -f "$tree/.frametop-release" ]; then
+      case $(sed -n 's/^CHANNEL=//p' "$tree/.frametop-release" | tail -1) in
+        experimental) branch=experimental ;; *) branch=main ;;
+      esac
+    else
+      branch=$(git -C "$tree" symbolic-ref --short -q HEAD 2>/dev/null || true)
+    fi
+    case $branch in main | experimental) ;; *) echo "hand models: no updates for branch '$branch'"; exit 0 ;; esac
+    mkdir -p "$models"
+    exec 9>"$models/.student-lock"
+    flock -n 9 || { echo "hand models: another update is running"; exit 0; }
+    touch "$models/.student-checked"
+    tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
+    code=$(curl -sS --connect-timeout 4 --max-time 15 --max-filesize 16384 -o "$tmp" -w '%{http_code}' \
+      "https://raw.githubusercontent.com/$gh/$branch/hands/models.pin") || code=
+    case $code in
+      200) ;;
+      '') echo "hand models: couldn't reach GitHub; keeping these"; exit 0 ;;
+      *) echo "hand models: $branch pins none ($code); keeping these"; exit 0 ;;
+    esac
+    cmp -s "$tmp" "$dir/PIN" 2>/dev/null && { echo "hand models: up to date ($(cat "$dir/VERSION" 2>/dev/null))"; exit 0; }
+    read_pin "$tmp" || { echo "hand models: $branch's pin isn't valid (or needs a newer Frametop); keeping these" >&2; exit 1; }
+    # Into .student-new, seeded with the files we have (a stopped update keeps what it fetched),
+    # then swapped with student in one rename: ft-hands never sees half a set.
+    new=$models/.student-new
+    mkdir -p "$new"
+    [ -d "$dir" ] && cp -a --update=none "$dir/." "$new/"
+    rm -f "$new"/*.part
+    fetch_into "$new" || { echo "hand models: update failed; keeping these" >&2; exit 1; }
+    cp "$tmp" "$new/PIN"; stamp "$new"
+    if [ -d "$dir" ]; then mv -T --exchange "$new" "$dir"; rm -rf "$new"; else mv -T "$new" "$dir"; fi
+    echo "hand models: updated to $model from $branch's pin"
+    ;;
+esac
+SCRIPT

hands/models/NOTICE

@@ -10,3 +10,7 @@ Apache License, Version 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
 Changes made here: converted to ncnn with pnnx, with the palm detector's channel pads
 rewritten as ncnn Padding layers, and quantized to 8 bits (the *-int8.ncnn.* files) with
 ncnn's tools.
+
+hands/models.sh fetches fine-tuned versions of the same two models into student/ (not in this
+repository): their weights are licensed CC BY-NC 4.0, not Apache-2.0 or MIT; see student/LICENSE and
+student/README.md once fetched, or https://huggingface.co/DeeJanuz/frametop-hands-models.

hands/rec/install.sh

@@ -5,7 +5,7 @@
 # give ft-camd its capabilities (hands/run.sh caps: asks for the password, once per build), and add
 # "Frametop Hand Recorder" to the app menu (for Frametop's desktop: it isn't tested from SteamVR's
 # "Launch a program", which runs apps outside it; the standalone recorder is for that).
-# It doesn't turn on Frametop's live hand tracking (that's hands/run.sh install, still deferred).
+# It doesn't turn on Frametop's live hand tracking (that's hands/run.sh install, which install.sh offers).
 # Usage: hands/rec/install.sh            install or update
 #        hands/rec/install.sh uninstall  remove the menu entry, and ft-camd's capabilities unless
 #                                        hand tracking's services use them (hands/run.sh uncaps).

hands/run.sh

@@ -58,7 +58,8 @@ states="for u in $units; do echo \"\$u: \$(systemctl --user is-active \$u)\"; do
 
 case ${1:-status} in
   install)
-    "$root/hands/build.sh"
+    # A release brings them built (pack/Containerfile), and never builds (FRAME_RELEASE).
+    [ "$FRAME_RELEASE" = 1 ] || "$root/hands/build.sh"
     set_caps
     for u in $units; do
       fill_template "$root/$(unit_dir $u)/$u" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$u"

hands/track/main.cpp

@@ -250,7 +250,10 @@ int main(int argc, char **argv) {
     double seconds = 0, status = 5;
     int threads = 3, niceness = 5;
     bool int8 = false, publish = true, track = true, gestures_on = true;
-    std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
+    // The models: HANDS_MODELS or --models, else the fine-tuned ones hands/models.sh fetched
+    // (models/student, chosen below once --int8 is known: they have no 8-bit files), else the stock ones.
+    const std::string model_root = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/";
+    std::string models;
     std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME;
     // SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for
     // SteamVR's compositor, whose threads there run at real-time priority, so they always
@@ -453,6 +456,10 @@ int main(int argc, char **argv) {
         std::fflush(stdout);
         return true;
     };
+    if (models.empty())
+        models = model_root + (!int8 && access((model_root + "student/hand.ncnn.param").c_str(), R_OK) == 0 &&
+                                       access((model_root + "student/palm.ncnn.param").c_str(), R_OK) == 0
+                                   ? "student" : "ncnn");
     if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
         (publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
         (!record.empty() && !start_recording(record, err))) {

hands/track/replay.cpp

@@ -109,6 +109,7 @@
 
 #include <json/json.h>
 #include <sys/stat.h>
+#include <unistd.h>
 
 #include <algorithm>
 #include <chrono>
@@ -175,7 +176,12 @@ int main(int argc, char **argv) {
     std::string flag_out;
     FlagConfig flag_cfg;
     flag_cfg.max_bytes = size_t(400) << 20, flag_cfg.writer_cpus.clear();
-    std::string timeline, depth, poses, models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
+    // as ft-hands: the fine-tuned models (models/student) when hands/models.sh fetched them, else the stock ones
+    const std::string model_root = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/";
+    std::string timeline, depth, poses,
+        models = model_root + (access((model_root + "student/hand.ncnn.param").c_str(), R_OK) == 0 &&
+                                       access((model_root + "student/palm.ncnn.param").c_str(), R_OK) == 0
+                                   ? "student" : "ncnn");
     for (int i = 2; i < argc; ++i) {
         const std::string a = argv[i];
         const bool more = i + 1 < argc;

install.sh

@@ -1,16 +1,18 @@
 #!/usr/bin/env bash
 # Install everything on the Steam Frame: the build container, Frametop (multi-screen
 # desktop, input relay, universal 3D mouse, settings app), and optionally voice dictation,
-# gaze mode, our own eye tracker for it, and the Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
-# for example after `git pull`. (Hand tracking, hands/, is deferred: it isn't offered here.)
+# hand tracking, gaze mode, our own eye tracker for it, and the Bluetooth fixes. Run it on the
+# headset in a terminal, from this repo. It's safe to re-run, for example after `git pull`.
 # (It also works from a PC over SSH; see "Developing from a PC" in the README.)
 # In a release (pack/install-release.sh), the programs come built from its image: this installs the
 # distrobox the release brings, makes the release's container from its image, and builds nothing.
 #
-# Usage: ./install.sh [--yes] [--no-voice] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
-#   --yes           don't ask; installs voice dictation, gaze mode, and our eye tracker if sudo can
-#                   run without a password prompt; skips the Bluetooth fixes and the SteamVR restart
+# Usage: ./install.sh [--yes] [--no-voice] [--no-hands] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
+#   --yes           don't ask; installs voice dictation and gaze mode, and hand tracking and our
+#                   eye tracker if sudo can run without a password prompt; skips the Bluetooth
+#                   fixes and the SteamVR restart
 #   --no-voice      don't offer voice dictation (Frametop Voice: it downloads about 320 MB)
+#   --no-hands      don't offer hand tracking
 #   --no-eye-tracker  don't offer our own eye tracker
 #   --no-bluetooth  don't offer the Bluetooth fixes
 #   --bluetooth     install the Bluetooth fixes without asking (with --yes: if sudo can run
@@ -20,7 +22,7 @@ set -euo pipefail
 root=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
 . "$root/scripts/_env.sh"
 
-assume_yes=0 bluetooth=1 eye_tracker=1 voice=1
+assume_yes=0 bluetooth=1 eye_tracker=1 hands=1 voice=1
 for arg in "$@"; do
   case $arg in
     --yes) assume_yes=1 ;;
@@ -28,7 +30,8 @@ for arg in "$@"; do
     --bluetooth) bluetooth=2 ;;
     --no-eye-tracker) eye_tracker=0 ;;
     --no-voice) voice=0 ;;
-    -h|--help) sed -n '2,18p' "$0"; exit 0 ;;
+    --no-hands) hands=0 ;;
+    -h|--help) sed -n '2,19p' "$0"; exit 0 ;;
     *) echo "unknown option: $arg" >&2; exit 2 ;;
   esac
 done
@@ -70,7 +73,7 @@ if [ "$FRAME_LOCAL" = 0 ] || [ -n "${SSH_CONNECTION:-}" ]; then
   echo "that need SteamVR start with it if it isn't running now."
 fi
 
-step "1/11 distrobox (container tool, installed in your home folder)"
+step "1/12 distrobox (container tool, installed in your home folder)"
 if on_frame 'test -x ~/.local/bin/distrobox'; then
   echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')"
 elif [ "$FRAME_RELEASE" = 1 ]; then
@@ -84,30 +87,30 @@ cd ~/dev/src/distrobox && ./install --prefix ~/.local'
 fi
 
 if [ "$FRAME_RELEASE" = 1 ]; then
-  step "2/11 Frametop's container (from this release's image)"
+  step "2/12 Frametop's container (from this release's image)"
   "$root/pack/release-box.sh"
 else
-  step "2/11 build container (Fedora 44 'dev', about 1-2 GB the first time)"
+  step "2/12 build container (Fedora 44 'dev', about 1-2 GB the first time)"
   "$root/setup/dev-container.sh"
 fi
 
-step "3/11 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
+step "3/12 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
 "$root/desktops.sh" relay install
 
-step "4/11 3D mouse: SteamVR driver"
+step "4/12 3D mouse: SteamVR driver"
 build "$root/pointer/driver/build.sh"
 "$root/pointer/driver/install.sh" install
 
-step "5/11 3D mouse: pointer helper service, and the dashboard patch that lets it move SteamVR's windows"
+step "5/12 3D mouse: pointer helper service, and the dashboard patch that lets it move SteamVR's windows"
 build "$root/pointer/helper/build.sh"
 "$root/pointer/helper/run.sh" install
 "$root/pointer/dashpatch/run.sh" install
 
-step "6/11 power service (turns the displays off while the headset isn't used, even on a stand)"
+step "6/12 power service (turns the displays off while the headset isn't used, even on a stand)"
 build "$root/power/build.sh"
 "$root/power/run.sh" install
 
-step "7/11 multi-screen desktop (ft-screens), Frametop Input Settings, Display Settings, and Remote Displays"
+step "7/12 multi-screen desktop (ft-screens), Frametop Input Settings, Display Settings, and Remote Displays"
 build "$root/screens/build.sh"
 build "$root/stream/build.sh"  # ft-stream: remote displays (Frametop Remote Displays)
 "$root/desktops.sh" install >/dev/null
@@ -119,7 +122,7 @@ on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^PO
 echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
 "$root/scripts/conf-migrate.sh"
 
-step "8/11 voice dictation (optional: speak, and it's typed where you type; nothing leaves the headset)"
+step "8/12 voice dictation (optional: speak, and it's typed where you type; nothing leaves the headset)"
 if [ "$voice" = 0 ]; then
   echo "skipped (--no-voice). Install later with: voice/run.sh install"
 elif ask "Install voice dictation? A Right Ctrl tap or holding the left d-pad down starts it; it downloads about 320 MB of speech models." y; then
@@ -130,7 +133,34 @@ else
   echo "skipped. Install later with: voice/run.sh install"
 fi
 
-step "9/11 gaze mode (optional, experimental: the pointer goes where you look)"
+# Hand tracking needs sudo once per build of ft-camd, its camera broker (file capabilities to
+# borrow XRService's camera buffers). Installed, it runs with the Frametop desktop
+# (HANDS_MODE=frametop), unless the settings already pick always.
+step "9/12 hand tracking (optional, experimental: your hands work the screens; it uses a lot of the headset's CPU)"
+if [ "$hands" = 0 ]; then
+  echo "skipped (--no-hands). Install later with: hands/run.sh install"
+elif [ "$assume_yes" = 1 ] && ! sudo_quiet; then
+  echo "skipped: it needs your password (sudo), and --yes doesn't ask. Install it later with: hands/run.sh install"
+elif ask "Install hand tracking? Poke, pinch and grab the screens with your hands. It runs while the Frametop desktop does (change that on the Hands page of Frametop Input Settings), and its camera broker needs your password (sudo)." y; then
+  if "$root/hands/run.sh" install; then
+    mode=$(on_frame "sed -n 's/^HANDS_MODE=\([a-z]*\).*/\1/p' ~/.config/frametop.conf | tail -1") || mode=
+    if [ "$mode" = always ]; then
+      echo "hand tracking runs whenever SteamVR does (HANDS_MODE=always, kept from your settings)"
+    else
+      if on_frame "python3 $(printf %q "$FRAME_REPO/hands/handsmode.py") set frametop" >/dev/null; then
+        echo "hand tracking runs while the Frametop desktop does (HANDS_MODE=frametop)"
+      else
+        echo "hand tracking is installed, but HANDS_MODE didn't change: choose With Frametop on the Hands page"
+      fi
+    fi
+  else
+    echo "hand tracking: not installed (above). Try again later with: hands/run.sh install"
+  fi
+else
+  echo "skipped. Install later with: hands/run.sh install"
+fi
+
+step "10/12 gaze mode (optional, experimental: the pointer goes where you look)"
 gaze=0
 if ask "Install gaze mode? You turn it on and calibrate it in Frametop Input Settings, on the Gaze page." y; then
   "$root/gaze/run.sh" install
@@ -141,7 +171,7 @@ fi
 
 # Ours is gaze mode's default once it's installed (GAZE_TRACKER=auto). Without it, gaze mode
 # uses SteamVR's eye tracker.
-step "10/11 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
+step "11/12 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
 if [ "$gaze" = 0 ]; then
   echo "skipped: gaze mode isn't installed. Install it later with: gaze/tracker/install.sh"
 elif [ "$eye_tracker" = 0 ]; then
@@ -169,7 +199,7 @@ else
   echo "skipped: gaze mode uses SteamVR's eye tracker. Install ours later with: gaze/tracker/install.sh"
 fi
 
-step "11/11 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
+step "12/12 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
 if [ "$bluetooth" = 2 ] && [ "$assume_yes" = 1 ] && ! sudo_quiet; then
   echo "skipped: they need your password (sudo), and --yes doesn't ask. Install later with: setup/bluetooth/install.sh install"
 elif [ "$bluetooth" = 2 ] || { [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; }; then

justfile

@@ -29,7 +29,7 @@ test-python:
 
 # the C/C++ unit tests that need no model runtime: header-only logic with
 # made-up events (pointer/tests: the helper's hand interaction logic). sides_test.cpp is NOT here: it pulls in ncnn (hands'
-# deferred heavy build) and stays with `make check` in the hands build
+# heavy build) and stays with `make check` in the hands build
 # (hands/Makefile).
 test-c:
     #!/usr/bin/env bash

pack/Containerfile

@@ -69,8 +69,7 @@ RUN uv venv --system-site-packages /opt/frametop/venv \
 # The sources, then the binaries, built by the same build scripts as on the
 # Frame (FRAME_IN_BOX=1: they run here instead of entering the dev container).
 # Each program stays in its build/ folder in /src/frametop and is copied to
-# /opt/frametop. Hand tracking is deferred exactly as in install.sh: it builds
-# ncnn from source and isn't needed until hands/run.sh install runs.
+# /opt/frametop.
 COPY . /src/frametop
 WORKDIR /src/frametop
 ENV FRAME_IN_BOX=1 OPENVR_LIB=/opt/frametop/lib
@@ -93,6 +92,12 @@ RUN gaze/tracker/build.sh \
 # above. Its speech models aren't in the image: voice/models.sh fetches them on install.
 RUN voice/build.sh \
  && install -t /opt/frametop/bin voice/build/bin/whisper-server
+# Hand tracking: ft-camd (static: it runs on the host, with the file
+# capabilities hands/run.sh install sets) and ft-hands, which builds ncnn at the
+# tag hands/Makefile pins. Both stay in hands/build/, where their units and
+# file capabilities point, so they aren't copied to /opt/frametop.
+RUN hands/build.sh \
+ && test -x hands/build/ft-camd && test -x hands/build/ft-hands
 
 # A release installs from this image (pack/install-release.sh): it copies
 # /src/frametop, built, to the Frame for its host side (services, the driver,

pack/README.md

@@ -32,8 +32,9 @@ The native binaries (`ft-screens`, `ft-pointer`, the `ft_pointer` driver,
 `build.sh` scripts, run inside the image with `FRAME_IN_BOX=1`, so there is
 one build recipe for the image and the Frame. They stay in their `build/`
 folders under `/src/frametop` and are copied to `/opt/frametop`. Hand
-tracking stays deferred exactly as in `install.sh` (ncnn is a heavy,
-separately triggered build).
+tracking's `ft-camd` and `ft-hands` (with ncnn, built from the tag
+`hands/Makefile` pins) stay in `hands/build/`, where their units and
+`ft-camd`'s file capabilities point.
 
 What deliberately stays outside the image for now: the **host payload** — the
 SteamVR driver registration (`vrpathreg`), ft-camd's file capabilities

pack/install-release.sh

@@ -4,7 +4,8 @@
 # are frametop-image.tar (Frametop, built, as a container image) and frametop-release.json (its
 # version, commit, the image file's sha256 and the image's ID, and the SteamOS table). Nothing
 # builds on the headset, and nothing else is downloaded, but voice dictation's speech models
-# (about 320 MB, fixed URLs checked by sha256: voice/models.sh) if you install it.
+# (about 320 MB, fixed URLs checked by sha256: voice/models.sh) if you install it. Hand tracking
+# comes built too: its install only sets its camera broker's file capabilities (sudo).
 #
 # 1. It checks this SteamOS build against the SteamOS table: Frametop's newest from GitHub
 #    (pack/steamos.json on main) when it can get it, else the one in the release. Tested: on.
@@ -17,7 +18,7 @@
 #    are removed, with their containers and images.
 #
 # Usage: install-release.sh [--yes] [--any-steamos] [--unpack-only] [--dir DIR]
-#                           [--no-voice] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
+#                           [--no-voice] [--no-hands] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
 #   --unpack-only  stop after step 2 and the copy: don't run install.sh
 #   --dir DIR      where the releases go (default ~/.local/share/frametop/releases)
 # The rest go to install.sh. FRAMETOP_STEAMOS_TABLE names another table to fetch (a URL), or
@@ -110,7 +111,7 @@ main() {
       --any-steamos) any=1 ;;
       --unpack-only) unpack_only=1 ;;
       --dir) base=${2:?--dir needs a folder}; shift ;;
-      --no-voice|--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
+      --no-voice|--no-hands|--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
       -h|--help) sed -n '2,24p' "$0"; return 0 ;;
       *) echo "unknown option: $1" >&2; return 2 ;;
     esac
@@ -167,7 +168,7 @@ main() {
     return 1
   fi
 
-  printf '\n\033[1m== 0/11 loading Frametop %s (a minute or two)\033[0m\n' "$version"
+  printf '\n\033[1m== 0/12 loading Frametop %s (a minute or two)\033[0m\n' "$version"
   if [ "$(podman image inspect -f '{{.Id}}' "$image" 2>/dev/null)" = "$id" ]; then
     echo "already loaded"
   else

pack/release-notes.md

@@ -6,7 +6,7 @@ On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1
 - **On the headset:** unpack Frametop.zip (in the desktop's Dolphin or Konsole) and run `Frametop/frametop-install.sh`.
 - **In a terminal on the headset:** `curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --release`
 
-A window asks what to install, and your SteamOS password for the two optional parts that need it (our own eye tracker and the Bluetooth fixes); the password is only used for this install. Nothing is compiled on the headset. Restart SteamVR once afterwards.
+A window asks what to install, and your SteamOS password for the three optional parts that need it (hand tracking, our own eye tracker, and the Bluetooth fixes); the password is only used for this install. Nothing is compiled on the headset. Restart SteamVR once afterwards.
 
 Before installing, the installer checks your SteamOS build against the builds Frametop was tested on, and stops if this release is known not to work there.
 

session/frametop.conf.example

@@ -42,7 +42,7 @@ POINTER_HEAD_DEADZONE=0.5  # keyboard clicks at the gaze (Meta+J, Meta+K): degre
 POINTER_KEY_TAP=0.25       # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
 GAZE_TRACKER=auto          # gaze service: auto = our own eye tracker when it's installed (gaze/tracker/install.sh, which install.sh offers), else SteamVR's | own = ours | steam = SteamVR's. Each keeps its own calibration (Calibrate on Frametop Input Settings' Gaze page)
 GAZE_EYE=auto              # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
-HANDS_MODE=off             # hand tracking (hands/run.sh install), when it runs: frametop = while the Frametop desktop runs | always = whenever SteamVR runs, in games and the dashboard too | off = only by hand (ft-handsctl on). Frametop Input Settings' Hands page applies it at once
+HANDS_MODE=off             # hand tracking (hands/run.sh install; install.sh sets frametop when it installs it), when it runs: frametop = while the Frametop desktop runs | always = whenever SteamVR runs, in games and the dashboard too | off = only by hand (ft-handsctl on). Frametop Input Settings' Hands page applies it at once
 HANDS_SWAP_SIDES=auto      # hand tracking: which side camera is which. auto = ft-hands tells from the hands and fixes ft-camd's names, which some SteamVR restarts swap | 0 = keep ft-camd's | 1 = exchange them
 HANDS_CPUS=5,6,7           # hand tracking: the CPUs its model threads run on
 HANDS_CAMERAS=auto         # hand tracking: auto (by the light) | mono (the IR cameras; also keeps ft-camd off the colour ones) | color | all
@@ -51,7 +51,8 @@ HANDS_BRIGHT_ON=40         # hand tracking, auto: the colour frames' mean bright
 HANDS_BRIGHT_OFF=25        # ... and back to the mono cameras under this
 HANDS_COLOR_LEFT=color_video0  # hand tracking: which colour camera is passthrough_left (hands/tools/check_color.py tells)
 HANDS_COLOR_CROP=subtract  # hand tracking: how the colour calibration's crop applies, subtract | none (check_color.py tells)
-#HANDS_MODELS=            # hand tracking: a folder of palm.ncnn.* and hand.ncnn.* to use instead of the stock models (hands/models/ncnn)
+#HANDS_MODELS=            # hand tracking: a folder of palm.ncnn.* and hand.ncnn.* to use instead of the fine-tuned models (hands/models/student; hands/models/ncnn has the stock ones)
+HANDS_MODELS_UPDATE=1      # hand tracking: 1 = when it starts, fetch newer fine-tuned models if GitHub's main (stable) or experimental branch pins them (hands/models.pin), without a new release | 0 = keep the ones it came with
 HANDS_MATTE=circle         # hand cutouts: circle = a round hole per hand | mesh = hand-shaped, from the skeleton | capsules = ft-hands' capsules (the old look)
 HANDS_MATTE_MARGIN=1.5     # hand cutouts: degrees (seen from the eyes) the hole reaches past the hand, held still (Hand cutout margin on the Hands page, live)
 HANDS_MATTE_MOTION=25      # ... plus how far the hand moves in this many ms

setup/dev-container.sh

@@ -18,7 +18,7 @@ packages=(
   mesa-libgbm-devel wayland-devel vulkan-loader-devel vulkan-headers plasma-wayland-protocols wlroots-devel
   # ft_pointer SteamVR driver: static C++ runtime (the host has an older glibc)
   libstdc++-static
-  # hand tracking (deferred: not installed by install.sh; hands/run.sh install builds it): ft-hands
+  # hand tracking (install.sh offers it; hands/run.sh install builds it): ft-hands
   # reads the calibration with jsoncpp; ft-camd runs on the host, linked statically. Its Python
   # tools (hands/tools) need NumPy and OpenCV, which aren't here: Fedora's python3-opencv pulls
   # in over a gigabyte (hands/README.md says how to get them)