Releases: ship the Hand Recorder

The image builds ft-handpanel (hands/rec/build.sh, against scripts/openvr.sh like gaze), and a release's installer offers the recorder: ft-camd's capabilities and the menu entry, nothing to build. The FrameDrop window has a Hand Recorder box, and --no-hand-recorder leaves it out. An entry from the release before is renewed without asking, since it points into that release's folder. The recorder runs ft-hands through scripts/in-box instead of the "dev" container, so a release uses its own container, and its repair hints say to reinstall the release rather than run a build script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Report: an area per flag, and --watch records each area live
2026-10-14 23:00:44 +02:00 · 2026-10-14 14:32:07 -06:00 · 2026-10-14 14:16:16 -06:00 · 2026-10-14 13:17:00 -06:00 · 2026-10-14 13:17:00 -06:00 · 2026-10-14 09:46:01 -06:00
29 changed files with 952 additions and 179 deletions
@@ -16,3 +16,6 @@
 **/.env.*
 frametop-report-*.txt
 .frame-job.d/
+# the fine-tuned hand models: the image fetches its own pinned copy (hands/models.sh)
+hands/models/student/
+hands/models/.student-*
@@ -59,10 +59,13 @@ jobs:
            test -f /src/frametop/pointer/driver/build/driver_ft_pointer.so
            test -x /src/frametop/hands/build/ft-camd
            test -x /src/frametop/hands/build/ft-hands
+            test -x /src/frametop/hands/rec/build/ft-handpanel
+            /src/frametop/hands/models.sh check
            test -x /src/frametop/pack/build/distrobox/install
            /src/frametop/gaze/tracker/build/venv/bin/python -c "import numpy, cv2"
            /src/frametop/voice/build/venv/bin/python -c "import numpy, sentencepiece, sherpa_onnx"
-            python3 -c "import PySide6"'
+            python3 -c "import PySide6, huggingface_hub"
+            command -v zstd >/dev/null'

      # The FrameDrop manifest names the zip's URL in this repo's release (a fork's, in a fork).
      - name: Frametop.zip
@@ -197,9 +197,19 @@ cd ~/frametop && scripts/report.sh

 From a release, start in `~/.local/share/frametop/releases/current` instead of `~/frametop`.

-This writes `frametop-report-<date>.txt` with version numbers, service states, settings, Frametop's keyboard and the Steam menu, and recent logs. Bluetooth addresses and the headset's serial number are masked.
+This writes `frametop-report-<date>.txt` with version numbers, settings, service states, CPU, memory and heat, and the state and recent logs of every part: the desktop (Frametop's keyboard, the Steam menu, the screens), gaze, hand tracking (its models, cameras, and how often it sees each hand), and voice (its models and the microphone). It takes up to half a minute, because it starts the gaze service for a moment to see whether the eye tracker sends. Bluetooth addresses and the headset's serial number are masked, and what you said to Frametop Voice is left out.

-If the problem is something you can make happen, like a window that won't drag or a keyboard that doesn't open, run `scripts/report.sh --watch` instead. After the usual report it records for 60 seconds (`--watch 120` for longer) while you make it happen in the headset. It notes when the Steam menu opens and closes, which laser drags what, where typing goes, and when Frametop's keyboard opens or why it doesn't. It takes up to half a minute, because it also checks gaze mode: it starts the gaze service for a moment to see whether the eye tracker sends. If gaze or its calibration doesn't work, run it while you wear the headset. `scripts/gaze-report.py` prints only the gaze part, with what looks wrong first. Then [open an issue](https://github.com/Frametop/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
+To report on one part only, name it: `--system`, `--desktop`, `--gaze`, `--hands` or `--voice` (several are fine, for example `scripts/report.sh --hands --voice`).
+
+If the problem is something you can make happen, like a window that won't drag, a pinch that doesn't register, or dictation that types nothing, add `--watch`. After the report it records for 60 seconds (`--watch 120` for longer) while you make it happen in the headset, and keeps the logs from that time:
+
+- desktop: when the Steam menu opens and closes, which laser drags what, where typing goes, and when Frametop's keyboard opens or why it doesn't
+- gaze: which eye tracker the gaze service uses, and whether its samples come in
+- hands: each second, how many frames hand tracking made, how often it saw each hand, and how close the fingertips came
+- voice: what Frametop Voice did (the number of words, not the words) and the microphone's level each second (no sound is kept)
+- system: every 5 seconds, the CPU's load, clocks and heat, and the busiest programs
+
+If gaze or its calibration doesn't work, run the report while you wear the headset. `scripts/gaze-report.py` prints only the gaze part, with what looks wrong first. Then [open an issue](https://github.com/Frametop/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.

 ## Update

@@ -1,6 +1,6 @@
 # Install with FrameDrop (proof of concept)

-[FrameDrop](https://framedropvr.com) is a Windows app that sideloads onto a Steam Frame: it copies a build to the headset and adds it to the Steam library. Issue #25 asks for an "Install with FrameDrop" button. Frametop isn't an app FrameDrop can copy over as is: it installs user services, a SteamVR driver, and a container, and three optional parts need sudo. So FrameDrop installs a small installer instead. Playing "Frametop" from the library opens a window that asks what to install, and your password for the parts that need it, then installs and shows its progress. A release's Frametop.zip (about 1.1 GB, built by CI on a tag, see [pack/README.md](../pack/README.md), Releases) carries Frametop built, as an image, and installs it with `install-release.sh`: nothing compiles on the headset and nothing else downloads. The same zip works unpacked on the headset. A test zip (a few KB) clones Frametop with `get.sh` instead.
+[FrameDrop](https://framedropvr.com) is a Windows app that sideloads onto a Steam Frame: it copies a build to the headset and adds it to the Steam library. Issue #25 asks for an "Install with FrameDrop" button. Frametop isn't an app FrameDrop can copy over as is: it installs user services, a SteamVR driver, and a container, and four optional parts need sudo. So FrameDrop installs a small installer instead. Playing "Frametop" from the library opens a window that asks what to install, and your password for the parts that need it, then installs and shows its progress. A release's Frametop.zip (about 1.1 GB, built by CI on a tag, see [pack/README.md](../pack/README.md), Releases) carries Frametop built, as an image, and installs it with `install-release.sh`: nothing compiles on the headset and nothing else downloads. The same zip works unpacked on the headset. A test zip (a few KB) clones Frametop with `get.sh` instead.

 Nothing here is published yet: no release, no button.

@@ -26,7 +26,7 @@ It uses Valve's SteamOS Devkit path: pair once with the headset's devkit service

 1. In the container, it starts itself again on the host with `flatpak-spawn --host`.
 2. It clears Steam's preload and library paths, and opens `installer/progress.py` (GTK 4 and libadwaita).
-3. The window asks which optional parts to install: voice dictation (on by default), hand tracking (on by default), our own eye tracker (on by default), and the Bluetooth fixes. All but voice dictation need sudo, so it asks for your SteamOS password and checks it with `sudo -v`. If your user has no password (SteamOS starts without one), it says how to set one and leaves those three out.
+3. The window asks which optional parts to install: voice dictation (on by default), hand tracking (on by default), the Hand Recorder (on by default; a release's zip only, since a clone builds it with `hands/rec/install.sh`), our own eye tracker (on by default), and the Bluetooth fixes. All but voice dictation need sudo, so it asks for your SteamOS password and checks it with `sudo -v`. If your user has no password (SteamOS starts without one), it says how to set one and leaves those out.
 4. It runs the zip's `install-release.sh --yes` (a test zip: `get.sh --yes`) in a transient user service, `frametop-framedrop-install`. The service is used because Steam ends the title's whole process tree when it's quit, and starts it with an OOM score of 900. Opened from a VR desktop (the zip unpacked by hand), it still uses the user's real bus and runtime folder for the service.
 5. It follows the service's log, shows the steps as a progress bar, and reports the result. Closing it leaves the install running. Playing the title again reattaches.

@@ -13,7 +13,7 @@ The install runs in a user service of its own (UNIT): Steam ends the title's who
 tree when it's quit, and starts it with a high OOM score. Closing the window doesn't stop the
 install; playing the title again reattaches to it.

-Our eye tracker and the Bluetooth fixes need sudo. The window asks for your password on the
+Hand tracking, the Hand Recorder, our eye tracker and the Bluetooth fixes need sudo. The window asks for your password on the
 headset, checks it with sudo, and keeps it in this process's memory until the install ends.
 sudo in the install gets it through askpass (SUDO_ASKPASS), from a socket in a folder only you
 can open, and the window answers only programs in the install's service. It's never written to
@@ -101,7 +101,7 @@ def in_unit(pid):
        return False


-def install_command(eye_tracker, bluetooth, voice=True, hands=True):
+def install_command(eye_tracker, bluetooth, voice=True, hands=True, hand_recorder=True):
    if RELEASE:
        args = [str(HERE / "install-release.sh"), "--yes"]
        if DRY_RUN:
@@ -114,6 +114,8 @@ def install_command(eye_tracker, bluetooth, voice=True, hands=True):
        args.append("--no-voice")
    if not hands:
        args.append("--no-hands")
+    if RELEASE and not hand_recorder:   # a clone (get.sh) doesn't offer it: hands/rec/install.sh builds it
+        args.append("--no-hand-recorder")
    if not eye_tracker:
        args.append("--no-eye-tracker")
    if bluetooth:
@@ -306,12 +308,16 @@ class Window(Adw.ApplicationWindow):
    def show_choices(self):
        what = f"Frametop {release_version()}" if RELEASE else "Frametop from GitHub"
        self.detail.set_label(f"This installs {what}: the multi-screen desktop, the 3D mouse, gaze "
-                              "mode, and Frametop's settings apps. Optional: voice dictation, and three "
+                              "mode, and Frametop's settings apps. Optional: voice dictation, and "
                              "parts that need your SteamOS password (sudo):")
        self.voice = Gtk.CheckButton(label="Voice dictation: a Right Ctrl tap or holding the left d-pad down, and "
                                           "what you say is typed (downloads about 320 MB)", active=True)
        self.hands = Gtk.CheckButton(label="Hand tracking (experimental): poke, pinch and grab the screens with "
                                           "your hands. It uses a lot of the headset's CPU", active=True)
+        # A release brings it built; a clone builds it with hands/rec/install.sh, so isn't offered it
+        self.recorder = Gtk.CheckButton(label="Hand Recorder: record your hands for Frametop's open hand "
+                                              "dataset, which trains hand tracking (nothing is uploaded "
+                                              "until you press Upload)", active=True, visible=RELEASE)
        self.eye = Gtk.CheckButton(label="Our own eye tracker for gaze mode (more accurate than SteamVR's)",
                                   active=True)
        self.bt = Gtk.CheckButton(label="Bluetooth fixes (LE mice and keyboards, like the Swiftpoint Z3, "
@@ -327,18 +333,20 @@ class Window(Adw.ApplicationWindow):
        self.go.connect("clicked", self.install)
        self.pw.connect("activate", self.install)
        pw_row, keypad = keypad_row(self.pw)
-        self.choices = [self.voice, self.hands, self.eye, self.bt, pw_row, keypad, pw_note, self.error, self.go]
+        self.choices = [self.voice, self.hands, self.recorder, self.eye, self.bt, pw_row, keypad, pw_note,
+                        self.error, self.go]
        if not password_set():
-            for c in (self.hands, self.eye, self.bt):
+            for c in (self.hands, self.recorder, self.eye, self.bt):
                c.set_active(False)
                c.set_sensitive(False)
            pw_row.set_visible(False)
            keypad.set_visible(False)
-            pw_note.set_label("Your user has no password, so sudo can't run and these three can't be "
+            pw_note.set_label("Your user has no password, so sudo can't run and these can't be "
                              "installed from here. To set one, run passwd in Konsole; then play "
                              "Frametop again, or install them later from a terminal "
-                              "(hands/run.sh install, gaze/tracker/install.sh, setup/bluetooth/install.sh).")
-        for c in (self.hands, self.eye, self.bt):
+                              "(hands/run.sh install, hands/rec/install.sh, gaze/tracker/install.sh, "
+                              "setup/bluetooth/install.sh).")
+        for c in (self.hands, self.recorder, self.eye, self.bt):
            c.connect("toggled", lambda *_: pw_row.set_sensitive(self.needs_password()))
        for w in self.choices:
            self.box.append(w)
@@ -374,7 +382,8 @@ class Window(Adw.ApplicationWindow):
        threading.Thread(target=check, daemon=True).start()

    def needs_password(self):
-        return self.hands.get_active() or self.eye.get_active() or self.bt.get_active()
+        return (self.hands.get_active() or (RELEASE and self.recorder.get_active()) or self.eye.get_active()
+                or self.bt.get_active())

    def say(self, text):
        self.error.set_label(text)
@@ -382,7 +391,7 @@ class Window(Adw.ApplicationWindow):

    def begin(self, pw):
        args = install_command(self.eye.get_active(), self.bt.get_active(), self.voice.get_active(),
-                               self.hands.get_active())
+                               self.hands.get_active(), self.recorder.get_active())
        if pw is not None:
            try:
                self.askpass = Askpass(self.ask_again)
@@ -12,6 +12,8 @@ Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions and hel

 For now the recorder runs inside Frametop's desktop, so these steps install Frametop first. A standalone recorder that runs from the SteamVR dashboard without Frametop is planned.

+Installing a Frametop release (Frametop.zip, FrameDrop, or `get.sh --release`) instead? Its installer offers the Hand Recorder, built, so there's nothing to compile: leave Hand Recorder ticked (or answer yes) and skip step 2. To add it to a release later, run `~/.local/share/frametop/releases/current/hands/rec/install.sh`.
+
 ## Before you start

 - You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/Frametop/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
@@ -1,6 +1,10 @@
 # Model sources that tools/convert_models.py downloads; the converted ncnn models are kept
 models/onnx/
 models/*.task
+# the fine-tuned models hands/models.sh fetches (their own licence, CC BY-NC 4.0: not in the repo)
+models/student/
+# hands/models.sh update's download folder, lock and last check
+models/.student-*
 # frame-job settings (where replays run, and the lab's capture folder): personal, not shipped
 .frame-job
 dumps/
@@ -50,7 +50,8 @@ Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides
 - `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (`sides.json`), so recordings are labelled right. The example config said `0` until 2026-10-05; `scripts/conf-migrate.sh` (run by `install.sh` and `hands/rec/install.sh`) turns that untouched line into `auto`.
 - `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
 - `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
- `HANDS_MODELS` (unset: `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to run fine-tuned models, such as the ones trained on the hand dataset, without passing options to every launcher. Those folders usually lack the `*-int8` files, so `--int8` won't load them.
+- `HANDS_MODELS` (unset: `hands/models/student`, the fine-tuned models, when `hands/models.sh` fetched them, else `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to try other models without passing options to every launcher. The fine-tuned folders lack the `*-int8` files, so `--int8` without `--models` uses the stock ones.
+- `HANDS_MODELS_UPDATE=1`: when hand tracking starts, fetch the fine-tuned models GitHub's branch pins if they changed, without a new release (`models.sh` below). `0` keeps the ones it came with.
 - `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
 - `HANDS_SKELETON=1`: fit a skeleton with your bone lengths and publish it (see "Skeleton" below); `0` publishes the filtered landmarks as before. As `--skeleton`.
 - `HANDS_PINCH_DIP=0`: `1` also starts a pinch on a quick dip of the thumb-index distance that never reaches 2 cm (see the pinch under "Gestures" below). As `--pinch-dip`.
@@ -214,7 +215,7 @@ What it measured, `ft-handreplay` on three recordings and three takes of the 202

 ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gestures file. The layout, and how to read it without missing quick taps, is in `include/fh_gestures.h`.

- A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost).
+- A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.045 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost).
 - A pinch held 0.3 s or longer is a drag, and holds on harder: it ends only once the tips are 5 cm apart (for the same 2 frames), or 0.12 s after they first went past 3.5 cm unless they closed back under 3 cm meanwhile. In the 2026-10-09 headset test drags ended with the tips 3.5-4.0 cm apart, a thumb hidden for a frame or fingers loosened a little mid-drag. On the lit recording's replayed distances, pinches held 0.3 s or more (typing among them) ended 33-134 ms later than before, 200 at most, and one of 25 ran into the next pinch's begin. A grip ends a pinch (as lost) only within 0.3 s of its begin, when closing the hand passes through a pinch; after that no grip begins on that hand, so curling the other fingers around a held pinch doesn't drop it (3 of 28 pinches in the same test). `tests/pinch_test.cpp` (`make check`).
 - The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. `--pinch-triangulated` uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. In the dim recording, typing still gave 2 pinches a minute (see the next point).
 - `--pinch-palm-down MAX` holds back pinches begun with the palm facing down (MAX is the palm normal's share of the head's up axis). The default, 1, turns it off. A close held back that way has to open again before a pinch can begin. Typing curls the thumb onto the index: in the lit recording of 2026-09-30, typing on a keyboard in the lap began 23 pinches in about 2 minutes, all with the palm facing down (0.69-1.00), while the 26 deliberate ones read 0.00-0.50. But in the headset, deliberate pinches with the hand raised in front read 0.90-0.99 too, so the limit is off. Typing is caught by the pointer helper instead: the input relay tells it when you press a key, and the hands on the keys are held back (`POINTER_PINCH_TYPING` and the settings after it, below).
@@ -298,6 +299,7 @@ Python, with NumPy and OpenCV. `setup/dev-container.sh` doesn't install them, be
 - `tools/depth_report.py DEPTH`: the depth measures above.
 - `tools/cut_sets.py REC OUT [--sets N | --at I,J,...]`: copies a few frame sets (by default 8, spread evenly) out of a recording into a small one, to look at or check elsewhere without moving gigabytes. Plain Python, so it also runs on the Frame's host.
 - `tools/convert_models.py`: how `models/ncnn` was made from the OpenCV Zoo ONNX ports of MediaPipe's models (see `models/NOTICE`).
+- `models.sh`: fetches the fine-tuned models into `models/student` (`hands/build.sh` runs it, so source installs and the release image both get them): MediaPipe's two models retrained on the Frame's infrared cameras with the [Frametop Hands](https://huggingface.co/datasets/DeeJanuz/frametop-hands) dataset, from a fixed commit of [DeeJanuz/frametop-hands-models](https://huggingface.co/DeeJanuz/frametop-hands-models), each file checked against its SHA-256. They're licensed CC BY-NC 4.0 (non-commercial), not MIT, so they aren't in this repository; `models/student/LICENSE` and `README.md` come with them. `hands/models.pin` pins them: the commit, the version folder and every file's sum (frame-hands' `model-repo/pin.py` writes it); `hands/models.sh check` checks them against it. Installs don't need a release for a new version: when hand tracking starts, `frametop-hands.service` runs `hands/models.sh auto`, which reads `hands/models.pin` from GitHub (main for a stable release, experimental for an experimental one or a clone on that branch; at most every 15 minutes, within a minute) and, if it pins other models, fetches them into `models/.student-new`, checks every sum, and swaps that folder in whole. Offline, or anything not matching, it keeps the models it has. A pin only names the files above, and its `format` says which ft-hands it needs: raise it when a model needs a newer one, and installs without it keep theirs. `HANDS_MODELS_UPDATE=0` turns it off; `hands/models.sh update` checks now.

 To try the hand cutouts without restarting the desktop, `screens/build/ft-handtest [--distance m] [--width m] [--seconds s] [--matte circle|mesh|capsules] [--margin deg]` (built by `screens/build.sh`, run in the dev container, with hand tracking on) shows a test panel of its own, a light grid 1 m wide and 0.8 m ahead by default, and cuts your hands out of it the way ft-screens cuts them out of the screens. Every 2 s it prints the composite's cost.

@@ -1,11 +1,13 @@
 #!/usr/bin/env bash
 # Build hand tracking in the dev container on the Frame, into hands/build/: ft-camd and ft-hands,
 # and with --tools also ft-handreplay and ft-ringplay. The first build fetches ncnn and builds
-# it (a few minutes); NCNN=DIR, an ncnn install already on the Frame, skips that.
+# it (a few minutes); NCNN=DIR, an ncnn install already on the Frame, skips that. It also fetches
+# the fine-tuned hand models (hands/models.sh) into hands/models/student, which ft-hands prefers.
 # A rebuilt ft-camd has lost its capabilities: hands/run.sh install sets them again.
 set -euo pipefail
 root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
 targets=all
 [ "${1:-}" = --tools ] && targets="all tools"
 "$root/scripts/sync.sh" >/dev/null
+"$root/hands/models.sh" fetch
 exec "$root/scripts/frame.sh" -C hands "make -s ${NCNN:+NCNN=$NCNN} $targets && echo built \$(ls build/ft-* | tr '\n' ' ')"
@@ -10,7 +10,10 @@ PartOf=steamvr.service
 # In the container it was built in (scripts/in-box: dev for a clone, the release's own for a
 # release; it's built against Fedora's libraries). It reads ft-camd's ring and
 # writes /run/user/UID/frametop-hands/hands and gestures. Settings: HANDS_* in ~/.config/frametop.conf.
+# Before it starts, newer fine-tuned models if its branch pins them (hands/models.sh auto: at most
+# every 15 minutes, never more than a minute, offline or failing it starts with the ones it has).
 ExecStartPre=-/usr/bin/pkill -x ft-hands
+ExecStartPre=-/usr/bin/timeout 60 @REPO@/hands/models.sh auto
 ExecStart=@REPO@/scripts/in-box @REPO@/hands/build/ft-hands --status 60
 ExecStopPost=-/usr/bin/pkill -x ft-hands
 Restart=always
@@ -0,0 +1,16 @@
+# The fine-tuned hand models ft-hands uses by default (hands/models.sh): a version folder of the
+# Hugging Face repo DeeJanuz/frametop-hands-models at a fixed commit, and every file's sha256.
+# Installs read this file from GitHub (main or experimental, by their channel) when hand tracking
+# starts, so a new pin pushed there reaches them without a release. "format" is what ft-hands
+# needs to run them: raise it when a model needs a newer ft-hands, and older installs keep theirs.
+# Made by frame-hands' model-repo/pin.py.
+format 1
+rev 41afd78fbd4527456fc4cbd840ffc9f39fdc939a
+model v13b-p1
+file NOTICE 0e6b4fa8ea37665cb699abc36fcf1ee0ffaa63a78dfee788be6613dea6607f84
+file hand.ncnn.bin d6a6f00d845fb9ee490276d84b1e978da884361c372614546ed2b1b4d2f5c449
+file hand.ncnn.param 54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd
+file palm.ncnn.bin 9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909
+file palm.ncnn.param c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e
+file LICENSE 41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0
+file README.md 29c6c5460e47412a9a37b22d6a7c8a9ed266a872bbcce93d3867f9e56cfe0ca1
@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+# Fetch the fine-tuned hand models into hands/models/student/ on the Frame (in the image build,
+# here: pack/Containerfile runs hands/build.sh, which runs this). hands/models.pin names them: a
+# fixed commit of the Hugging Face repo below, and every file's sha256. ft-hands uses them when
+# they're there (HANDS_MODELS overrides; hands/models/ncnn, the stock MediaPipe models, otherwise).
+#   palm.ncnn.*, hand.ncnn.*   MediaPipe's palm detector and hand landmark model, fine-tuned on the
+#                              Frametop hand dataset (DeeJanuz/frametop-hands)
+#   LICENSE, README.md         their licence (CC BY-NC 4.0, not Frametop's MIT) and model card
+# Usage: hands/models.sh [fetch]   fetch what's missing or doesn't match models.pin (the default)
+#        hands/models.sh check     check every file against the pin it was fetched by (exit 1 if not)
+#        hands/models.sh update    fetch the models the install's branch pins on GitHub now (main for
+#                                  a stable release, experimental for an experimental one or a
+#                                  clone on that branch), if they changed. Into a folder of its own,
+#                                  swapped in once every file matches. Offline or nothing new: no-op.
+#        hands/models.sh auto      update, as frametop-hands.service runs it when hand tracking
+#                                  starts: not with HANDS_MODELS_UPDATE=0 in ~/.config/frametop.conf,
+#                                  and at most every 15 minutes.
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+. "$root/scripts/_env.sh"
+case ${1:-fetch} in fetch | check | update | auto) ;; *) sed -n '9,18p' "$0" >&2; exit 2 ;; esac
+
+on_frame_script "${1:-fetch}" "$FRAME_REPO" <<'SCRIPT'
+set -euo pipefail
+mode=$1 tree=$2
+models=$tree/hands/models dir=$tree/hands/models/student
+hf=DeeJanuz/frametop-hands-models gh=Frametop/frametop
+names=" NOTICE hand.ncnn.bin hand.ncnn.param palm.ncnn.bin palm.ncnn.param LICENSE README.md "
+
+# read_pin FILE: check every line of a pin, then set rev, model and files ("name sha256" each).
+# Anything unexpected fails it: a pin from GitHub only names files this list allows.
+read_pin() {
+  local key a b n format=
+  rev= model= files=()
+  while read -r key a b; do
+    case $key in '' | '#'*) continue ;; esac
+    case $key in
+      format) [ "$a" = 1 ] && [ -z "$b" ] || return 1; format=$a ;;
+      rev) [[ $a =~ ^[0-9a-f]{40}$ ]] && [ -z "$b" ] || return 1; rev=$a ;;
+      model) [[ $a =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$ ]] && [ -z "$b" ] || return 1; model=$a ;;
+      file) [[ $names == *" $a "* && $b =~ ^[0-9a-f]{64}$ ]] || return 1; files+=("$a $b") ;;
+      *) return 1 ;;
+    esac
+  done <"$1"
+  [ -n "$format" ] && [ -n "$rev" ] && [ -n "$model" ] || return 1
+  for n in hand.ncnn.bin hand.ncnn.param palm.ncnn.bin palm.ncnn.param; do
+    [[ " ${files[*]} " == *" $n "* ]] || return 1
+  done
+}
+
+matches() { [ -f "$1" ] && [ "$(sha256sum <"$1" | cut -d' ' -f1)" = "$2" ]; }
+
+# fetch_into DIR: fetch the pin's files that are missing there or don't match.
+fetch_into() {
+  local d=$1 entry name sum src
+  mkdir -p "$d"
+  for entry in "${files[@]}"; do
+    read -r name sum <<<"$entry"
+    matches "$d/$name" "$sum" && continue
+    src=https://huggingface.co/$hf/resolve/$rev/$model/$name
+    case $name in LICENSE | README.md) src=https://huggingface.co/$hf/resolve/$rev/$name ;; esac
+    echo "fetching hand model $name ($model)"
+    if ! curl -fsSL --retry 3 --connect-timeout 10 --max-filesize 100000000 -o "$d/$name.part" "$src"; then
+      rm -f "$d/$name.part"; return 1
+    fi
+    if ! matches "$d/$name.part" "$sum"; then
+      echo "hand models: $name's sha256 doesn't match the pin; not kept" >&2
+      rm -f "$d/$name.part"; return 1
+    fi
+    mv "$d/$name.part" "$d/$name"
+  done
+}
+
+stamp() { echo "$hf@${rev::7} $model" >"$1/VERSION"; }
+
+case $mode in
+  fetch)
+    read_pin "$tree/hands/models.pin" || { echo "hand models: hands/models.pin isn't a valid pin" >&2; exit 1; }
+    fetch_into "$dir"
+    cp "$tree/hands/models.pin" "$dir/PIN"; stamp "$dir"
+    echo "hand models: $model in $dir"
+    ;;
+  check)
+    # Against the pin they were fetched by: an update may have replaced the release's own.
+    pin=$dir/PIN; [ -f "$pin" ] || pin=$tree/hands/models.pin
+    [ -d "$dir" ] || { echo "hand models: $dir missing" >&2; exit 1; }
+    read_pin "$pin" || { echo "hand models: $pin isn't a valid pin" >&2; exit 1; }
+    bad=0
+    for entry in "${files[@]}"; do
+      read -r name sum <<<"$entry"
+      matches "$dir/$name" "$sum" || { echo "hand models: $name missing or changed" >&2; bad=1; }
+    done
+    [ "$bad" = 0 ] || exit 1
+    echo "hand models: $model in $dir, all files match"
+    ;;
+  update | auto)
+    if [ "$mode" = auto ]; then
+      conf=$(sed -n 's/#.*//; s/^[[:space:]]*HANDS_MODELS_UPDATE[[:space:]]*=[[:space:]]*//p' ~/.config/frametop.conf 2>/dev/null | tail -1)
+      conf=${FT_HANDS_MODELS_UPDATE:-$conf}
+      [ "$(echo $conf | tr -d "\"'")" = 0 ] && { echo "hand models: not updating (HANDS_MODELS_UPDATE=0)"; exit 0; }
+      [ -n "$(find "$models/.student-checked" -mmin -15 2>/dev/null)" ] && exit 0
+    fi
+    if [ -f "$tree/.frametop-release" ]; then
+      case $(sed -n 's/^CHANNEL=//p' "$tree/.frametop-release" | tail -1) in
+        experimental) branch=experimental ;; *) branch=main ;;
+      esac
+    else
+      branch=$(git -C "$tree" symbolic-ref --short -q HEAD 2>/dev/null || true)
+    fi
+    case $branch in main | experimental) ;; *) echo "hand models: no updates for branch '$branch'"; exit 0 ;; esac
+    mkdir -p "$models"
+    exec 9>"$models/.student-lock"
+    flock -n 9 || { echo "hand models: another update is running"; exit 0; }
+    touch "$models/.student-checked"
+    tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
+    code=$(curl -sS --connect-timeout 4 --max-time 15 --max-filesize 16384 -o "$tmp" -w '%{http_code}' \
+      "https://raw.githubusercontent.com/$gh/$branch/hands/models.pin") || code=
+    case $code in
+      200) ;;
+      '') echo "hand models: couldn't reach GitHub; keeping these"; exit 0 ;;
+      *) echo "hand models: $branch pins none ($code); keeping these"; exit 0 ;;
+    esac
+    cmp -s "$tmp" "$dir/PIN" 2>/dev/null && { echo "hand models: up to date ($(cat "$dir/VERSION" 2>/dev/null))"; exit 0; }
+    read_pin "$tmp" || { echo "hand models: $branch's pin isn't valid (or needs a newer Frametop); keeping these" >&2; exit 1; }
+    # Into .student-new, seeded with the files we have (a stopped update keeps what it fetched),
+    # then swapped with student in one rename: ft-hands never sees half a set.
+    new=$models/.student-new
+    mkdir -p "$new"
+    [ -d "$dir" ] && cp -a --update=none "$dir/." "$new/"
+    rm -f "$new"/*.part
+    fetch_into "$new" || { echo "hand models: update failed; keeping these" >&2; exit 1; }
+    cp "$tmp" "$new/PIN"; stamp "$new"
+    if [ -d "$dir" ]; then mv -T --exchange "$new" "$dir"; rm -rf "$new"; else mv -T "$new" "$dir"; fi
+    echo "hand models: updated to $model from $branch's pin"
+    ;;
+esac
+SCRIPT
@@ -10,3 +10,7 @@ Apache License, Version 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
 Changes made here: converted to ncnn with pnnx, with the palm detector's channel pads
 rewritten as ncnn Padding layers, and quantized to 8 bits (the *-int8.ncnn.* files) with
 ncnn's tools.
+
+hands/models.sh fetches fine-tuned versions of the same two models into student/ (not in this
+repository): their weights are licensed CC BY-NC 4.0, not Apache-2.0 or MIT; see student/LICENSE and
+student/README.md once fetched, or https://huggingface.co/DeeJanuz/frametop-hands-models.
@@ -1,18 +1,18 @@
 #!/usr/bin/env bash
 # Build the hand recorder's headset panel ft-handpanel in the dev container on the Frame
-# (hands/rec/build/; it also runs there). Like gaze/build.sh: the pinned public OpenVR header
-# (the DMA-BUF import is newer than the header shipped with SteamVR's samples), stb_truetype
+# (hands/rec/build/; it also runs there), or in the release image (pack/Containerfile). Like
+# gaze/build.sh: the pinned public OpenVR header (scripts/openvr.sh; the DMA-BUF import is newer
+# than the header shipped with SteamVR's samples), stb_truetype
 # for the text and stb_image (PNG and GIF) for the pose pictures and the demo clips, at the same stb commit.
 set -euo pipefail
 root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
 "$root/scripts/sync.sh" >/dev/null
 exec "$root/scripts/frame.sh" -C hands/rec 'set -e; mkdir -p build/include
-openvr=v2.15.6
-[ -f build/include/openvr-$openvr ] || { curl -fsSL "https://raw.githubusercontent.com/ValveSoftware/openvr/$openvr/headers/openvr.h" -o build/include/openvr.h && touch build/include/openvr-$openvr; }
+. ../../scripts/openvr.sh
 stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
 [ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
 [ -f build/include/stb_image-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_image.h" -o build/include/stb_image.h && touch build/include/stb_image-$stb; }
 g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
-  -o build/ft-handpanel panel/ft-handpanel.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 \
+  -o build/ft-handpanel panel/ft-handpanel.cpp $OPENVR_LIBS \
  $(pkg-config --libs gbm libdrm) -lpthread
 echo "built build/ft-handpanel"'
@@ -1,6 +1,6 @@
 #!/bin/bash
 # Launch the Frametop Hand Recorder from a Plasma session on the Frame host.
-# The app runs in the dev container (PySide6, Kirigami and zstd come from Fedora there).
+# The app runs in Frametop's container (PySide6, Kirigami and zstd come from Fedora there).
 # podman needs the real XDG_RUNTIME_DIR and the real user bus (to reach systemd for
 # the container's cgroup; the Frametop session runs on a private bus from
 # dbus-run-session). The session's Wayland socket and bus go to the app itself.
@@ -14,8 +14,7 @@ export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
 # From the home folder: the app's host commands (distrobox-host-exec) run in its folder on the
 # host, and a folder only the container has, such as /run/host/tmp, makes them all fail.
 cd "$HOME" || exit 1
-"$here/../../scripts/container-up.sh"
-exec "$HOME/.local/bin/distrobox" enter dev -- env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
+exec "$here/../../scripts/in-box" env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
  XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
  QT_QPA_PLATFORM="wayland;xcb" \
  python3 "$here/ft_handrec.py" "$@"
@@ -6,6 +6,8 @@
 # "Frametop Hand Recorder" to the app menu (for Frametop's desktop: it isn't tested from SteamVR's
 # "Launch a program", which runs apps outside it; the standalone recorder is for that).
 # It doesn't turn on Frametop's live hand tracking (that's hands/run.sh install, which install.sh offers).
+# A release (FRAME_RELEASE) brings the three built in its image (pack/Containerfile), so there it only
+# sets the capabilities and adds the menu entry; its install.sh offers this.
 # Usage: hands/rec/install.sh            install or update
 #        hands/rec/install.sh uninstall  remove the menu entry, and ft-camd's capabilities unless
 #                                        hand tracking's services use them (hands/run.sh uncaps).
@@ -17,13 +19,17 @@ entry='~/.local/share/applications/frametop-handrec.desktop'

 case ${1:-install} in
  install)
-    echo "== 1/4 dev container packages"
-    "$root/setup/dev-container.sh"
-    echo "== 2/4 hand tracking: ft-camd and ft-hands"
-    "$root/hands/build.sh"
-    echo "== 3/4 the headset panel: ft-handpanel"
-    "$root/hands/rec/build.sh"
-    echo "== 4/4 ft-camd's capabilities (asks for your password) and the menu entry"
+    if [ "$FRAME_RELEASE" = 1 ]; then
+      echo "== ft-camd's capabilities (asks for your password, unless it has them) and the menu entry"
+    else
+      echo "== 1/4 dev container packages"
+      "$root/setup/dev-container.sh"
+      echo "== 2/4 hand tracking: ft-camd and ft-hands"
+      "$root/hands/build.sh"
+      echo "== 3/4 the headset panel: ft-handpanel"
+      "$root/hands/rec/build.sh"
+      echo "== 4/4 ft-camd's capabilities (asks for your password) and the menu entry"
+    fi
    "$root/hands/run.sh" caps
    "$root/scripts/conf-migrate.sh"   # HANDS_SWAP_SIDES=0, the old default, becomes auto
    fill_template "$root/hands/rec/ft-handrec.desktop" |
@@ -55,6 +55,9 @@ import takes  # noqa: E402  (hands/rec/takes.py: the version, and the standalone
 FT_HANDS = os.path.join(HANDS, "build", "ft-hands")
 FT_CAMD = os.path.join(HANDS, "build", "ft-camd")
 PANEL_BIN = os.path.join(HERE, "build", "ft-handpanel")
+# Runs a program in the container this tree was built for: "dev" in a clone, the release's own in a
+# release (it starts the container first)
+IN_BOX = os.path.join(REPO, "scripts", "in-box")
 SCRIPT_PATH = os.path.join(HERE, "script.json")
 POSES_DIR = os.path.join(HERE, "poses")   # the pose pictures: poses.json and its PNGs
 DEMOS_DIR = os.path.join(HERE, "demos")   # the demo clips: <id>.gif (DESIGN.md "Demo clips")
@@ -83,6 +86,8 @@ HANDS_UNIT = "frametop-handrec-hands.service"
 # The standalone Hand Recorder (takes.standalone()): its binaries run on the host, so ft-hands
 # starts directly rather than in the dev container, and repairs mean its install command.
 STANDALONE = takes.standalone()
+# A Frametop release (pack/install-release.sh) rather than a clone
+RELEASE = os.path.exists(os.path.join(REPO, ".frametop-release"))

 RECORD_HZ = 10
 SIDES_READ_S = 0.5       # how often the tracking ft-hands' side camera decision is read
@@ -161,9 +166,13 @@ def host_command(*cmd):


 def fix_hint(frametop):
-    """How to repair a missing or broken part: Frametop's command, or the standalone install's."""
+    """How to repair a missing or broken part: Frametop's command, or the standalone install's.
+    A Frametop release never builds (its programs come in its image), so a build there means
+    installing the release again."""
    if STANDALONE:
        return STANDALONE.get("reinstall") or "run the Hand Recorder's install command again"
+    if RELEASE and frametop.endswith("build.sh"):
+        return "install this Frametop release again"
    return frametop


@@ -696,7 +705,7 @@ class Recorder:
        if ring:
            argv += ["--ring", ring]
        if not in_container() and not STANDALONE:
-            argv = [os.path.expanduser("~/.local/bin/distrobox"), "enter", "dev", "--"] + argv
+            argv = [IN_BOX] + argv
        self.proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=log_file, stderr=log_file)
        self.started_ns = mono_ns()

@@ -1408,7 +1417,7 @@ def camera_text(result):
    if camcheck.is_ring_short(result):
        return ("The headset's tracking cameras are all running, but the recorder can't read some of them (%s). "
                "Close the Hand Recorder, %s, and open it again. If that doesn't help, ask in the Frametop "
-                "Discord." % (result.get("reason", ""), fix_hint("run ~/frametop/hands/rec/install.sh again")))
+                "Discord." % (result.get("reason", ""), fix_hint("run %s again" % os.path.join(REPO, "hands", "rec", "install.sh"))))
    return ("Not all of the headset's tracking cameras are running (%s). Restart SteamVR, or restart the "
            "headset if that doesn't fix it." % result.get("reason", ""))

@@ -1846,11 +1855,8 @@ class Session:

    def _start_tracker(self):
        argv = [FT_HANDS, "--no-gestures", "--status", "0"]
-        if not STANDALONE:   # built in the dev container: it runs there
-            up = os.path.join(REPO, "scripts", "container-up.sh")
-            if os.access(up, os.X_OK):
-                subprocess.run(host_command(up), capture_output=True, timeout=120)
-            argv = [os.path.expanduser("~/.local/bin/distrobox"), "enter", "dev", "--"] + argv
+        if not STANDALONE:   # built in Frametop's container: it runs there
+            argv = [IN_BOX] + argv
        if self.ring:
            argv += ["--ring", self.ring]
        self._start_unit(HANDS_UNIT, "hand tracking for feedback", argv)
@@ -918,18 +918,23 @@ class StandaloneTest(unittest.TestCase):
        self.assertIn("--no-gestures", argv)
        run.assert_not_called()   # no container to bring up

-    def test_ft_hands_in_the_dev_container(self):
-        self.assertTrue(self.recorder_argv(None)[0].endswith("distrobox"))
+    def test_ft_hands_in_frametops_container(self):
+        # scripts/in-box: "dev" in a clone, the release's own container in a release
+        self.assertEqual(self.recorder_argv(None)[:2], [session.IN_BOX, session.FT_HANDS])
        argv, _ = self.tracker_argv(None)
-        self.assertEqual(argv[:4], [os.path.expanduser("~/.local/bin/distrobox"), "enter", "dev", "--"])
-        self.assertEqual(argv[4], session.FT_HANDS)
+        self.assertEqual(argv[:2], [session.IN_BOX, session.FT_HANDS])
+        self.assertTrue(os.access(session.IN_BOX, os.X_OK))

    def test_fix_hint(self):
        with mock.patch.object(session, "STANDALONE", self.INFO):
            self.assertEqual(session.fix_hint("hands/build.sh"), "run the install command again")
            text = session.camera_text({"status": "degraded", "reason": "2 of 4", "ring_missing": ["upper_left"]})
-        with mock.patch.object(session, "STANDALONE", None):
+        with mock.patch.object(session, "STANDALONE", None), mock.patch.object(session, "RELEASE", False):
            self.assertEqual(session.fix_hint("hands/build.sh"), "hands/build.sh")
+        with mock.patch.object(session, "STANDALONE", None), mock.patch.object(session, "RELEASE", True):
+            # a release never builds: its programs come in its image
+            self.assertEqual(session.fix_hint("hands/rec/build.sh"), "install this Frametop release again")
+            self.assertEqual(session.fix_hint("hands/run.sh caps (asks for sudo)"), "hands/run.sh caps (asks for sudo)")
        self.assertIn("run the install command again", text)
        self.assertNotIn("~/frametop", text)

@@ -75,12 +75,40 @@ struct Run {
    bool down() const { return pinch.side(1).flags & FH_PINCH_DOWN; }
 };

+// The values these tests were written for (the defaults before 2026-10-14): they test the rules,
+// TestDefaults the defaults.
 PinchParams Params() {
    PinchParams p;
    p.other_finger = 12;   // as ft-hands
+    p.end_m = 0.035, p.drag_open_m = 0.05, p.drag_open_s = 0.12, p.grip_takeover_s = 0.3;
    return p;
 }

+void TestDefaults() {
+    // since 2026-10-14 a pinch lets go less easily: a tap still reading 4 cm hasn't ended
+    PinchParams p;
+    p.other_finger = 12;
+    Run tap{Pinch(p), Grip()};
+    for (int k = 0; k < 5; ++k) tap.Step(0.06);
+    for (int k = 0; k < 3; ++k) tap.Step(0.01);
+    tap.Step(0.04), tap.Step(0.04);
+    CHECK(tap.down(), "defaults: a tap at 4 cm is still down");
+    tap.Step(0.05), tap.Step(0.05);
+    CHECK(!tap.down() && tap.lost == 0, "defaults: a tap ends on the second frame past 4.5 cm");
+    // a drag loosened to 5.5 cm ends after drag_open_s (0.2 s: six or seven frames), opened to 7 cm at once
+    Run drag{Pinch(p), Grip()};
+    for (int k = 0; k < 5; ++k) drag.Step(0.06);
+    for (int k = 0; k < 30; ++k) drag.Step(0.01);
+    int frames = 0;
+    while (drag.down() && frames < 20) drag.Step(0.055), ++frames;
+    CHECK(!drag.down() && frames >= 6 && frames <= 8, "defaults: a drag at 5.5 cm ended after %d frames", frames);
+    Run open{Pinch(p), Grip()};
+    for (int k = 0; k < 5; ++k) open.Step(0.06);
+    for (int k = 0; k < 30; ++k) open.Step(0.01);
+    open.Step(0.07), open.Step(0.07);
+    CHECK(!open.down() && open.lost == 0, "defaults: a drag opened to 7 cm ends on the second frame");
+}
+
 void TestDragSpike() {
    // a drag (held 1 s): the tips read 3.8 cm for two frames (a hidden thumb), then close again
    Run r{Pinch(Params()), Grip()};
@@ -267,6 +295,7 @@ int main() {
    TestGripTakeover();
    TestDip();
    TestFistGuard();
+    TestDefaults();
    if (failures) {
        std::printf("%d failures\n", failures);
        return 1;
@@ -250,7 +250,10 @@ int main(int argc, char **argv) {
    double seconds = 0, status = 5;
    int threads = 3, niceness = 5;
    bool int8 = false, publish = true, track = true, gestures_on = true;
-    std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
+    // The models: HANDS_MODELS or --models, else the fine-tuned ones hands/models.sh fetched
+    // (models/student, chosen below once --int8 is known: they have no 8-bit files), else the stock ones.
+    const std::string model_root = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/";
+    std::string models;
    std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME;
    // SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for
    // SteamVR's compositor, whose threads there run at real-time priority, so they always
@@ -368,7 +371,7 @@ int main(int argc, char **argv) {
                        "          [--flag-buffer 0|1] (1: keep the last seconds for `ft-handsctl flag`; HANDS_FLAG_BUFFER)\n"
                        "          [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
                        "          [--color-left color_video0|color_video3] [--color-crop subtract|none]\n"
-                        "          [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (1: off)\n"
+                        "          [--pinch-begin M] (0.020) [--pinch-end M] (0.045) [--pinch-triangulated] [--pinch-palm-down MAX] (1: off)\n"
                        "          [--pinch-dip 0|1] (0; 1: also on a sharp dip, see PinchParams::dip; HANDS_PINCH_DIP)\n"
                        "          [--pinch-fist-curl R] (1.2; 0: off: no pinch begins on a fist, see PinchParams::fist_curl;\n"
                        "          HANDS_PINCH_FIST_CURL)\n"
@@ -453,6 +456,10 @@ int main(int argc, char **argv) {
        std::fflush(stdout);
        return true;
    };
+    if (models.empty())
+        models = model_root + (!int8 && access((model_root + "student/hand.ncnn.param").c_str(), R_OK) == 0 &&
+                                       access((model_root + "student/palm.ncnn.param").c_str(), R_OK) == 0
+                                   ? "student" : "ncnn");
    if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
        (publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
        (!record.empty() && !start_recording(record, err))) {
@@ -15,7 +15,10 @@ extern "C" {

 struct PinchParams {
    double begin_m = 0.020;   // thumb and index tips closer than this: the pinch begins
-    double end_m = 0.035;     // further apart than this: it ends (the gap keeps it from flickering)
+    // further apart than this: it ends (the gap keeps it from flickering). 0.045 since 2026-10-14
+    // (0.035 before): taps and pinches let go while still held (user, 2026-10-11, gaze + hands: ends
+    // came at 4-6 cm); these four were the headset's HANDS_PINCH_* settings since, now everyone's.
+    double end_m = 0.045;
    int end_frames = 2;       // processed frames in a row past end_m before it ends, so one
                              // noisy frame doesn't drop a drag
    double grace_s = 0.25;    // a pinching hand lost this long ends its pinch (FH_PINCH_LOST)
@@ -27,11 +30,12 @@ struct PinchParams {
    // replay (pinches held 0.3 s or more, typing among them) the ends came 33-134 ms later, at most
    // 200, where any dip under end_m starting the wait over kept one held 6 s longer. A tap ends as
    // before: its release is the click.
-    double drag_s = 0.3, drag_open_m = 0.05, drag_open_s = 0.12, drag_reclose_m = 0.03;
+    double drag_s = 0.3, drag_open_m = 0.065, drag_open_s = 0.2, drag_reclose_m = 0.03;   // 0.05, 0.12 before 2026-10-14
    // A grip takes this pinch's hand (ends it, as lost) only within grip_takeover_s of its begin:
    // closing the hand into a fist passes through a pinch. Later the pinch is being held (a drag),
    // and the other fingers curling around it don't end it (3 of 28 pinches ended so, the same test).
-    double grip_takeover_s = 0.3;
+    // 0.1 since 2026-10-14 (0.3 before: grips took pinches 130 ms in).
+    double grip_takeover_s = 0.1;
    // Where the distance comes from: MediaPipe's world landmarks (the model's own 3D hand
    // pose, averaged over the hand's views, at the user's hand size), or the tracker's
    // triangulated tips. The model's pose should hold up better when the fingers hide each
@@ -109,6 +109,7 @@

 #include <json/json.h>
 #include <sys/stat.h>
+#include <unistd.h>

 #include <algorithm>
 #include <chrono>
@@ -175,7 +176,12 @@ int main(int argc, char **argv) {
    std::string flag_out;
    FlagConfig flag_cfg;
    flag_cfg.max_bytes = size_t(400) << 20, flag_cfg.writer_cpus.clear();
-    std::string timeline, depth, poses, models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
+    // as ft-hands: the fine-tuned models (models/student) when hands/models.sh fetched them, else the stock ones
+    const std::string model_root = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/";
+    std::string timeline, depth, poses,
+        models = model_root + (access((model_root + "student/hand.ncnn.param").c_str(), R_OK) == 0 &&
+                                       access((model_root + "student/palm.ncnn.param").c_str(), R_OK) == 0
+                                   ? "student" : "ncnn");
    for (int i = 2; i < argc; ++i) {
        const std::string a = argv[i];
        const bool more = i + 1 < argc;
@@ -7,12 +7,15 @@
 # In a release (pack/install-release.sh), the programs come built from its image: this installs the
 # distrobox the release brings, makes the release's container from its image, and builds nothing.
 #
-# Usage: ./install.sh [--yes] [--no-voice] [--no-hands] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
-#   --yes           don't ask; installs voice dictation and gaze mode, and hand tracking and our
-#                   eye tracker if sudo can run without a password prompt; skips the Bluetooth
-#                   fixes and the SteamVR restart
+# Usage: ./install.sh [--yes] [--no-voice] [--no-hands] [--no-hand-recorder] [--no-eye-tracker]
+#                    [--no-bluetooth | --bluetooth]
+#   --yes           don't ask; installs voice dictation and gaze mode, and hand tracking, the Hand
+#                   Recorder and our eye tracker if sudo can run without a password prompt; skips
+#                   the Bluetooth fixes and the SteamVR restart
 #   --no-voice      don't offer voice dictation (Frametop Voice: it downloads about 320 MB)
-#   --no-hands      don't offer hand tracking
+#   --no-hands      don't offer hand tracking or the Hand Recorder
+#   --no-hand-recorder  don't offer the Hand Recorder (a release's; a clone builds it with
+#                   hands/rec/install.sh)
 #   --no-eye-tracker  don't offer our own eye tracker
 #   --no-bluetooth  don't offer the Bluetooth fixes
 #   --bluetooth     install the Bluetooth fixes without asking (with --yes: if sudo can run
@@ -22,7 +25,7 @@ set -euo pipefail
 root=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
 . "$root/scripts/_env.sh"

-assume_yes=0 bluetooth=1 eye_tracker=1 hands=1 voice=1
+assume_yes=0 bluetooth=1 eye_tracker=1 hands=1 hand_recorder=1 voice=1
 for arg in "$@"; do
  case $arg in
    --yes) assume_yes=1 ;;
@@ -30,8 +33,9 @@ for arg in "$@"; do
    --bluetooth) bluetooth=2 ;;
    --no-eye-tracker) eye_tracker=0 ;;
    --no-voice) voice=0 ;;
-    --no-hands) hands=0 ;;
-    -h|--help) sed -n '2,19p' "$0"; exit 0 ;;
+    --no-hands) hands=0 hand_recorder=0 ;;
+    --no-hand-recorder) hand_recorder=0 ;;
+    -h|--help) sed -n '2,22p' "$0"; exit 0 ;;
    *) echo "unknown option: $arg" >&2; exit 2 ;;
  esac
 done
@@ -143,6 +147,7 @@ elif [ "$assume_yes" = 1 ] && ! sudo_quiet; then
  echo "skipped: it needs your password (sudo), and --yes doesn't ask. Install it later with: hands/run.sh install"
 elif ask "Install hand tracking? Poke, pinch and grab the screens with your hands. It runs while the Frametop desktop does (change that on the Hands page of Frametop Input Settings), and its camera broker needs your password (sudo)." y; then
  if "$root/hands/run.sh" install; then
+    hands_caps=1
    mode=$(on_frame "sed -n 's/^HANDS_MODE=\([a-z]*\).*/\1/p' ~/.config/frametop.conf | tail -1") || mode=
    if [ "$mode" = always ]; then
      echo "hand tracking runs whenever SteamVR does (HANDS_MODE=always, kept from your settings)"
@@ -159,6 +164,26 @@ elif ask "Install hand tracking? Poke, pinch and grab the screens with your hand
 else
  echo "skipped. Install later with: hands/run.sh install"
 fi
+# The Hand Recorder (hands/rec): a release brings it built, so it's only ft-camd's capabilities
+# (set already if hand tracking just installed) and the menu entry. An entry from the release
+# before points into that release's folder, which goes two releases on: it's renewed without asking.
+# A clone builds it with hands/rec/install.sh (a few minutes), so it isn't offered here.
+if [ "$FRAME_RELEASE" = 1 ]; then
+  recorder="the Hand Recorder (records your hands for Frametop's open hand dataset)"
+  if [ "$hand_recorder" = 0 ]; then
+    echo "$recorder: skipped. Add it later with: hands/rec/install.sh"
+  elif on_frame '[ -e ~/.local/share/applications/frametop-handrec.desktop ]'; then
+    "$root/hands/rec/install.sh" install >/dev/null && echo "$recorder: updated" ||
+      echo "$recorder: couldn't update it (above). Try again with: hands/rec/install.sh"
+  elif [ "$assume_yes" = 1 ] && [ "${hands_caps:-0}" = 0 ] && ! sudo_quiet; then
+    echo "$recorder: skipped: it needs your password (sudo), and --yes doesn't ask. Add it later with: hands/rec/install.sh"
+  elif ask "Add the Hand Recorder to the app menu? It records your hands for Frametop's open hand dataset, which trains hand tracking; nothing leaves the headset until you press Upload. Its camera broker needs your password (sudo), as hand tracking does." y; then
+    "$root/hands/rec/install.sh" install ||
+      echo "$recorder: not installed (above). Try again later with: hands/rec/install.sh"
+  else
+    echo "$recorder: skipped. Add it later with: hands/rec/install.sh"
+  fi
+fi

 step "10/12 gaze mode (optional, experimental: the pointer goes where you look)"
 gaze=0
@@ -98,6 +98,10 @@ RUN voice/build.sh \
 # file capabilities point, so they aren't copied to /opt/frametop.
 RUN hands/build.sh \
 && test -x hands/build/ft-camd && test -x hands/build/ft-hands
+# The Hand Recorder's headset panel: it stays in hands/rec/build/, where the recorder
+# (hands/rec/session.py) runs it from.
+RUN hands/rec/build.sh \
+ && test -x hands/rec/build/ft-handpanel

 # A release installs from this image (pack/install-release.sh): it copies
 # /src/frametop, built, to the Frame for its host side (services, the driver,
@@ -5,7 +5,8 @@
 # version, commit, the image file's sha256 and the image's ID, and the SteamOS table). Nothing
 # builds on the headset, and nothing else is downloaded, but voice dictation's speech models
 # (about 320 MB, fixed URLs checked by sha256: voice/models.sh) if you install it. Hand tracking
-# comes built too: its install only sets its camera broker's file capabilities (sudo).
+# and the Hand Recorder come built too: their installs only set their camera broker's file
+# capabilities (sudo), and the recorder's adds its menu entry.
 #
 # 1. It checks this SteamOS build against the SteamOS table: Frametop's newest from GitHub
 #    (pack/steamos.json on main) when it can get it, else the one in the release. Tested: on.
@@ -18,7 +19,8 @@
 #    are removed, with their containers and images.
 #
 # Usage: install-release.sh [--yes] [--any-steamos] [--unpack-only] [--dir DIR]
-#                           [--no-voice] [--no-hands] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
+#                           [--no-voice] [--no-hands] [--no-hand-recorder] [--no-eye-tracker]
+#                           [--no-bluetooth | --bluetooth]
 #   --unpack-only  stop after step 2 and the copy: don't run install.sh
 #   --dir DIR      where the releases go (default ~/.local/share/frametop/releases)
 # The rest go to install.sh. FRAMETOP_STEAMOS_TABLE names another table to fetch (a URL), or
@@ -111,8 +113,8 @@ main() {
      --any-steamos) any=1 ;;
      --unpack-only) unpack_only=1 ;;
      --dir) base=${2:?--dir needs a folder}; shift ;;
-      --no-voice|--no-hands|--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
-      -h|--help) sed -n '2,24p' "$0"; return 0 ;;
+      --no-voice|--no-hands|--no-hand-recorder|--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
+      -h|--help) sed -n '2,27p' "$0"; return 0 ;;
      *) echo "unknown option: $1" >&2; return 2 ;;
    esac
    shift
@@ -6,7 +6,7 @@ On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1
 - **On the headset:** unpack Frametop.zip (in the desktop's Dolphin or Konsole) and run `Frametop/frametop-install.sh`.
 - **In a terminal on the headset:** `curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --release`

-A window asks what to install, and your SteamOS password for the three optional parts that need it (hand tracking, our own eye tracker, and the Bluetooth fixes); the password is only used for this install. Nothing is compiled on the headset. Restart SteamVR once afterwards.
+A window asks what to install, and your SteamOS password for the optional parts that need it (hand tracking, the Hand Recorder, our own eye tracker, and the Bluetooth fixes); the password is only used for this install. Nothing is compiled on the headset. Restart SteamVR once afterwards.

 Before installing, the installer checks your SteamOS build against the builds Frametop was tested on, and stops if this release is known not to work there.

@@ -0,0 +1,328 @@
+#!/usr/bin/env python3
+"""report-watch: what scripts/report.sh --watch records while the problem happens. On the Frame:
+  python3 scripts/report-watch.py SECONDS AREA[,AREA...]
+Each line: the time, the area, and what it saw. Areas (any of them, each in a thread of its own):
+  system   every 5 s: the load, the CPU clocks, the hottest thermal zone, free memory, and the
+           processes that used the most CPU in those 5 s
+  desktop  ft-screens' debug line (the Steam menu, our keyboard, which laser drags what, where
+           typing goes) whenever it changes
+  gaze     the gaze service's status (which tracker, awake, samples coming in, the headset worn)
+           whenever it changes
+  hands    each second ft-hands publishes or stops: its frames per second, their latency, how often
+           each hand was in them and how sure, the closest thumb-index and thumb-middle tips; and
+           sides.json whenever the side cameras' decision changes
+  voice    Frametop Voice's events, with what was said left out (its word count stays), and the
+           microphone's level each second (peak and RMS, dBFS; no sound is kept)
+Read-only: it asks the services over their sockets and reads their published files.
+"""
+import json
+import math
+import mmap
+import os
+import socket
+import struct
+import subprocess
+import sys
+import threading
+import time
+from array import array
+from pathlib import Path
+
+RUN = Path(f"/run/user/{os.getuid()}")
+lock = threading.Lock()
+
+
+def say(area, text):
+    now = time.time()
+    with lock:
+        print(time.strftime("%H:%M:%S", time.localtime(now)) + f".{int(now * 100) % 100:02d}", f"[{area}]", text, flush=True)
+
+
+def ask(name, msg, timeout=1.0):
+    """A request to one of Frametop's abstract datagram sockets; the reply, or None."""
+    s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    s.bind("")
+    s.settimeout(timeout)
+    try:
+        s.sendto(msg.encode(), "\0" + name)
+        return s.recv(65536).decode()
+    except OSError:
+        return None
+    finally:
+        s.close()
+
+
+def read(path):
+    try:
+        return Path(path).read_text().strip()
+    except OSError:
+        return ""
+
+
+# ---- system
+
+
+def cpu_times():
+    """pid -> (name, utime + stime in ticks)."""
+    out = {}
+    for d in os.listdir("/proc"):
+        if not d.isdigit():
+            continue
+        try:
+            s = open(f"/proc/{d}/stat").read()
+        except OSError:
+            continue
+        name, rest = s[s.index("(") + 1:s.rindex(")")], s[s.rindex(")") + 2:].split()
+        out[int(d)] = (name, int(rest[11]) + int(rest[12]))
+    return out
+
+
+def system(end):
+    tick = os.sysconf("SC_CLK_TCK")
+    zones = sorted(Path("/sys/class/thermal").glob("thermal_zone*"))
+    cpus = sorted(Path("/sys/devices/system/cpu").glob("cpu[0-9]*/cpufreq/scaling_cur_freq"),
+                  key=lambda p: int(p.parent.parent.name[3:]))
+    before, t0 = cpu_times(), time.monotonic()
+    while time.monotonic() < end:
+        time.sleep(min(5, max(0.1, end - time.monotonic())))
+        now, t1 = cpu_times(), time.monotonic()
+        used = sorted(((v[1] - before[p][1]) / tick / (t1 - t0) * 100, v[0]) for p, v in now.items()
+                      if p in before and before[p][0] == v[0])
+        top = ", ".join(f"{name} {pct:.0f}%" for pct, name in reversed(used[-5:]) if pct >= 1)
+        before, t0 = now, t1
+        temps = []
+        for z in zones:
+            t = read(z / "temp")
+            if t.lstrip("-").isdigit():
+                temps.append((int(t) / 1000, read(z / "type")))
+        hot = max(temps) if temps else (float("nan"), "?")
+        mhz = " ".join(str(int(read(c) or 0) // 1000) for c in cpus)
+        mem = {l.split(":")[0]: int(l.split()[1]) for l in open("/proc/meminfo") if l.split(":")[0] in ("MemAvailable", "MemTotal")}
+        say("system", f"load {read('/proc/loadavg').split(' ')[0]}  CPU MHz {mhz}  hottest {hot[0]:.1f} C ({hot[1]})  "
+                      f"memory free {mem.get('MemAvailable', 0) // 1024} of {mem.get('MemTotal', 0) // 1024} MB  top: {top or '-'}")
+
+
+# ---- desktop
+
+
+def desktop(end):
+    last = None
+    while time.monotonic() < end:
+        line = ask("ft_screens", "debug") or "no answer from ft-screens"
+        if line != last:
+            say("desktop", line.strip())
+            last = line
+        time.sleep(0.2)
+
+
+# ---- gaze
+
+GAZE_KEYS = ("tracker", "source", "awake", "idle", "ft_gaze", "own_running", "eyes_process", "headset_on",
+             "eye_misses", "lessons", "calibration_samples")
+
+
+def gaze(end):
+    last = None
+    while time.monotonic() < end:
+        r = ask("ft_gazed", "status", 0.5)
+        try:
+            st = json.loads(r) if r else None
+        except ValueError:
+            st = None
+        if st is None:
+            text = "no answer from the gaze service (ft-gazed)"
+        else:
+            age = st.get("sample_age_s")
+            samples = "none" if age is None else "coming in" if age < 0.5 else f"stopped ({age:.0f} s ago)" if age < 600 else "stopped"
+            text = f"samples {samples}  " + "  ".join(f"{k} {json.dumps(st[k])}" for k in GAZE_KEYS if k in st)
+        if text != last:
+            say("gaze", text)
+            last = text
+        time.sleep(1)
+
+
+# ---- hands (include/fh_hands.h)
+
+HEAD = struct.Struct("<8sIIQQQII16x")          # magic, version, size, seq, capture_ns, publish_ns, nhands, ncapsules
+HAND = struct.Struct("<IIff252xI")             # id, flags, confidence, reserved, pts[21][3], ncapsules: 272 bytes
+V1_SIZE = HEAD.size + 2 * HAND.size + 64 * 32  # fh_hands_t
+FIT_PINCH = 912                                # fh_hand_fit_t.pinch_d (thumb-index, thumb-middle), then fit_rms
+
+
+def hands(end):
+    path = RUN / "frametop-hands/hands"
+    sides_last, quiet = None, None
+    next_report = time.monotonic() + 1
+    frames, seen, conf, lat, tips = 0, {0: 0, 1: 0}, {0: [], 1: []}, [], {0: [], 1: []}
+    last_seq = None
+    m, ino = None, None
+    while time.monotonic() < end:
+        # under its sequence lock: seq, a copy, seq again; the copy counts if both are the same even number
+        data, again = b"", None
+        try:
+            st = path.stat()   # a restarted ft-hands writes a new file
+            if m is None or (st.st_ino, st.st_size) != ino:
+                with open(path, "rb") as f:
+                    m, ino = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ), (st.st_ino, st.st_size)
+            first = struct.unpack_from("<Q", m, 16)[0]
+            data = bytes(m)
+            again = struct.unpack_from("<Q", m, 16)[0]
+        except (OSError, ValueError):
+            m = None
+        if len(data) >= V1_SIZE:
+            magic, version, _, seq, cap, pub, n, _ = HEAD.unpack_from(data)
+            if magic == b"FHHANDS1" and seq == first == again and seq % 2 == 0 and seq != last_seq and pub:
+                new, last_seq = last_seq is not None, seq
+            else:
+                new = False
+            if new:   # not the frame already there when the watch began
+                frames += 1
+                lat.append((time.monotonic_ns() - cap) / 1e6)
+                for k in range(min(n, 2)):
+                    _, flags, c, _, _ = HAND.unpack_from(data, HEAD.size + k * HAND.size)
+                    side = flags & 1
+                    seen[side] += 1
+                    conf[side].append(c)
+                    off = V1_SIZE + k * 1024
+                    if version >= 2 and len(data) >= off + 1024 and struct.unpack_from("<I", data, off + 4)[0] & 1:
+                        tips[side].append(struct.unpack_from("<fff", data, off + FIT_PINCH))
+        if time.monotonic() >= next_report:
+            next_report += 1
+            if frames:
+                lat.sort()
+                def side(s, name):
+                    if not seen[s]:
+                        return f"{name} -"
+                    t = tips[s]
+                    fit = (f" closest thumb-index {1000 * min(x[0] for x in t):.0f} mm, thumb-middle "
+                           f"{1000 * min(x[1] for x in t):.0f} mm, fit {1000 * sum(x[2] for x in t) / len(t):.1f} mm") if t else ""
+                    return f"{name} {100 * seen[s] // frames}% conf {sum(conf[s]) / len(conf[s]):.2f}{fit}"
+                say("hands", f"{frames} frames/s  age at read {lat[len(lat) // 2]:.0f} ms (median)  "
+                             f"{side(0, 'left')}  |  {side(1, 'right')}")
+                quiet = False
+            elif quiet is not True:
+                say("hands", "no new frames from ft-hands" + ("" if path.exists() else f" ({path} missing)"))
+                quiet = True
+            frames, seen, conf, lat, tips = 0, {0: 0, 1: 0}, {0: [], 1: []}, [], {0: [], 1: []}
+            sides = read(RUN / "frametop-hands/sides.json")
+            try:
+                s = json.loads(sides)
+                sides = f"side cameras {s.get('state')}, swapped {s.get('swapped')} (by {s.get('decided_by')})"
+            except ValueError:
+                sides = sides or "no sides.json"
+            if sides != sides_last:
+                say("hands", sides)
+                sides_last = sides
+        time.sleep(1 / 120)
+
+
+# ---- voice
+
+SAID = ("text", "matched", "ran")   # what was said, and the command it ran (its words)
+
+
+def voice_events(end):
+    path = RUN / "frametop-voice/control.sock"
+    while time.monotonic() < end:
+        s = socket.socket(socket.AF_UNIX)
+        try:
+            s.connect(str(path))
+        except OSError as e:
+            say("voice", f"Frametop Voice isn't running ({e.strerror})")
+            s.close()
+            while time.monotonic() < end and not path.exists():
+                time.sleep(1)
+            continue
+        s.sendall(b"events\n")
+        s.settimeout(0.5)
+        buf = b""
+        while time.monotonic() < end:
+            try:
+                chunk = s.recv(4096)
+            except socket.timeout:
+                continue
+            except OSError:
+                chunk = b""
+            if not chunk:
+                say("voice", "Frametop Voice closed its events")
+                break
+            buf += chunk
+            while b"\n" in buf:
+                line, buf = buf.split(b"\n", 1)
+                try:
+                    ev = json.loads(line)
+                except ValueError:
+                    continue
+                ev.pop("time", None)
+                for k in SAID:
+                    if isinstance(ev.get(k), str):
+                        ev[k] = f"<{len(ev[k].split())} words>"
+                    elif k in ev:
+                        ev[k] = "<left out>"
+                say("voice", json.dumps(ev))
+        s.close()
+        time.sleep(1)
+
+
+def gain():
+    """The mic's ALSA gain, which SteamVR's proxmicmute sets to 0 while nobody wears the headset."""
+    vals = []
+    for numid in (23, 24):   # VA_DEC0, VA_DEC1 on card 0
+        try:
+            out = subprocess.run(["amixer", "-c", "0", "cget", f"numid={numid}"], capture_output=True, text=True, timeout=3).stdout
+        except (OSError, subprocess.SubprocessError):
+            return "unknown"
+        vals += [l.split("=", 1)[1] for l in out.splitlines() if ": values=" in l]
+    if not vals:
+        return "unknown"
+    return " ".join(vals) + (" (muted by proxmicmute: the headset off)" if all(v == "0" for v in vals) else "")
+
+
+def mic(end):
+    rate = 16000
+    try:
+        p = subprocess.Popen(["pw-record", "--raw", "--rate", str(rate), "--channels", "1", "--format", "s16", "-"],
+                             stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+    except OSError as e:
+        return say("voice", f"mic: can't run pw-record ({e})")
+    silent = None
+    try:
+        while time.monotonic() < end:
+            raw = p.stdout.read(rate * 2)
+            if not raw:
+                return say("voice", "mic: pw-record stopped")
+            a = array("h", raw[: len(raw) // 2 * 2])
+            peak = max(abs(x) for x in a) / 32768 if a else 0
+            rms = math.sqrt(sum(x * x for x in a) / len(a)) / 32768 if a else 0
+            if peak == 0:   # digital silence: once, then how long it lasted
+                if silent is None:
+                    say("voice", f"mic: digital silence (all zeros); its gain: {gain()}")
+                    silent = time.monotonic()
+                continue
+            if silent is not None:
+                say("voice", f"mic: silent for {time.monotonic() - silent:.0f} s")
+                silent = None
+            say("voice", f"mic peak {20 * math.log10(peak):.0f} dBFS, RMS {20 * math.log10(max(rms, 1e-9)):.0f} dBFS")
+        if silent is not None:
+            say("voice", f"mic: silent all the {time.monotonic() - silent:.0f} s since")
+    finally:
+        p.kill()
+
+
+def main():
+    if len(sys.argv) != 3 or not sys.argv[1].isdigit():
+        sys.exit(__doc__)
+    end = time.monotonic() + int(sys.argv[1])
+    jobs = {"system": [system], "desktop": [desktop], "gaze": [gaze], "hands": [hands], "voice": [voice_events, mic]}
+    areas = [a for a in sys.argv[2].split(",") if a]
+    if not areas or any(a not in jobs for a in areas):
+        sys.exit(f"areas: {', '.join(jobs)}")
+    threads = [threading.Thread(target=f, args=(end,), daemon=True) for a in areas for f in jobs[a]]
+    for t in threads:
+        t.start()
+    for t in threads:
+        t.join(max(0, end - time.monotonic()) + 3)
+
+
+if __name__ == "__main__":
+    main()
@@ -1,31 +1,53 @@
 #!/usr/bin/env bash
-# Collect what a Frametop bug report needs into one text file: versions, service states,
-# settings, Frametop's keyboard and the Steam menu, recent logs, and the gaze report
-# (scripts/gaze-report.py). Bluetooth addresses and the headset's serial number are masked.
-# Usage: scripts/report.sh [--watch [SECONDS]]   (in a terminal on the Frame, or from a PC over SSH)
-#   --watch   then record SECONDS (default 60) while you make the problem happen: the
-#             desktop's state (the Steam menu, our keyboard, which laser drags what, where
-#             typing goes) whenever it changes, and the logs from that time
+# Collect what a Frametop bug report needs into one text file. In a terminal on the Frame, or from
+# a PC over SSH:
+#   scripts/report.sh [--system] [--desktop] [--gaze] [--hands] [--voice] [--watch [SECONDS]]
+# With no area, all of them. Every report has the versions and the settings (~/.config/frametop.conf).
+#   --system   services, what Frametop needs from SteamOS, CPU, memory, heat, the busiest processes,
+#              the session's log, SteamVR's errors
+#   --desktop  the screens' layout, Plasma's panels, our keyboard and the Steam menu, the input
+#              relay's, pointer helper's and power service's logs, ft-screens' log
+#   --gaze     (or --eyes) the gaze report (scripts/gaze-report.py): SteamVR's eye tracker and ours,
+#              calibrations, the gaze service; it wakes an idle gaze service for about 20 s
+#   --hands    hand tracking: its services, models, cameras, side cameras, ft-hands' counts, ft-camd
+#   --voice    Frametop Voice: its services, state, settings, speech models, and the microphone: its
+#              gain, PipeWire's capture chain, and its level over one second (no sound is kept)
+#   --watch    then record SECONDS (default 60, 5 to 600) of the chosen areas while you make the
+#              problem happen (scripts/report-watch.py), and their logs from that time
+# Left out: what was said to Frametop Voice (its word counts stay). Bluetooth addresses and the
+# headset's serial number are masked.
 set -euo pipefail
 root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
 . "$root/scripts/_env.sh"
-watch=0
-case ${1:-} in
-  "") ;;
-  --watch) watch=${2:-60}
-           [[ $watch =~ ^[0-9]+$ ]] && [ "$watch" -ge 5 ] && [ "$watch" -le 600 ] ||
-             { echo "--watch takes 5 to 600 seconds" >&2; exit 2; } ;;
-  -h|--help) sed -n '2,8p' "$0"; exit 0 ;;
-  *) echo "usage: $0 [--watch [SECONDS]]" >&2; exit 2 ;;
-esac
-out=$root/frametop-report-$(date +%Y%m%d-%H%M%S).txt
-echo "Collecting (up to half a minute: the gaze service is woken to see whether the eye tracker sends)..."
+all=system,desktop,gaze,hands,voice
+areas= watch=0
+usage() { echo "usage: $0 [--system] [--desktop] [--gaze] [--hands] [--voice] [--watch [SECONDS]]  (-h: more)" >&2; exit 2; }
+while [ $# -gt 0 ]; do
+  case $1 in
+    --system | --desktop | --gaze | --hands | --voice) areas+=${areas:+,}${1#--} ;;
+    --eyes) areas+=${areas:+,}gaze ;;
+    --watch)
+      watch=60
+      if [[ ${2:-} =~ ^[0-9]+$ ]]; then watch=$2; shift; fi
+      [ "$watch" -ge 5 ] && [ "$watch" -le 600 ] || { echo "--watch takes 5 to 600 seconds" >&2; exit 2; } ;;
+    -h | --help) sed -n '2,20p' "$0"; exit 0 ;;
+    *) usage ;;
+  esac
+  shift
+done
+areas=${areas:-$all}
+out=$root/frametop-report-$(date +%Y%m%d-%H%M%S)$([ "$areas" = "$all" ] || echo "-${areas//,/-}").txt
+echo "Collecting: ${areas//,/, }$([[ ,$areas, == *,gaze,* ]] && echo " (up to half a minute: the gaze service is woken to see whether the eye tracker sends)")..."

-on_frame_script "$FRAME_REPO" > "$out" 2>&1 <<'EOF' || true
-repo=$1
+on_frame_script "$FRAME_REPO" "$areas" > "$out" 2>&1 <<'EOF' || true
+repo=$1 areas=$2
 export XDG_RUNTIME_DIR=/run/user/$(id -u) DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus
 logs=~/.local/share/Steam/logs
+has() { [[ ,$areas, == *,$1,* ]]; }
 section() { printf '\n===== %s\n' "$*"; }
+units() { for u in "$@"; do echo "$u: $(systemctl --user is-enabled $u 2>/dev/null) / $(systemctl --user is-active $u 2>/dev/null)"; done; }
+# podman's lines for each command run in the container, and ft-hands' pinch and grip lines
+noise='container (exec|exec_died|init|start|died|attach|cleanup)'
 # ask SOCKET COMMAND: a request to one of Frametop's abstract datagram sockets; prints the reply.
 ask() {
  python3 -I - "$1" "$2" <<'PY' 2>&1
@@ -41,7 +63,7 @@ except OSError as e:
 PY
 }

-section Versions
+section "Versions (report: ${areas//,/, })"
 grep -E '^(PRETTY_NAME|VERSION_ID|BUILD_ID|VARIANT_ID)=' /etc/os-release
 # The log keeps earlier runs: the last start is this SteamVR.
 grep -a -o 'vrcompositor [0-9.]* startup' $logs/vrcompositor.txt 2>/dev/null | tail -n 1 | grep . || echo "SteamVR: not found"
@@ -57,25 +79,49 @@ fi
 ~/.local/bin/distrobox version 2>/dev/null || echo "distrobox: not installed"
 podman container inspect -f "container $box: running={{.State.Running}} image={{.ImageName}}" "$box" 2>/dev/null ||
  echo "container $box: missing"
-
-section Services
-for u in frametop-input-relay frametop-pointer frametop-power frametop-gaze frametop-voice frametop-desktop; do
-  echo "$u: $(systemctl --user is-enabled $u 2>/dev/null) / $(systemctl --user is-active $u 2>/dev/null)"
-done
-for u in frametop-eyegrab steamframe-bt-fixups; do
-  echo "$u (system): $(systemctl is-enabled $u 2>/dev/null) / $(systemctl is-active $u 2>/dev/null)"
-done
-echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)"
-echo "paused for VR games: $(cat /run/user/$(id -u)/frametop-pause.json 2>/dev/null || echo 'no (never paused since boot)')"
-# SteamVR's config folder: a terminal in the desktop has the session's own (docs/design.md).
-XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p'
-
-section "What Frametop needs from SteamOS (scripts/update-check.py)"
-python3 "$repo/scripts/update-check.py" 2>&1 || true
+echo "up $(uptime -p 2>/dev/null | sed 's/^up //'), since $(uptime -s 2>/dev/null)"

 section Settings
 grep -v '^\s*#' ~/.config/frametop.conf 2>/dev/null | sed 's/\s*#.*//' | grep . || echo "no ~/.config/frametop.conf"
-python3 - <<'PY' 2>/dev/null || echo "no ~/.config/frametop-layout.json"
+
+if has system; then
+  section Services
+  units frametop-input-relay frametop-pointer frametop-power frametop-gaze frametop-desktop frametop-voice \
+    frametop-voice-whisper frametop-camd frametop-hands frametop-handview
+  for u in frametop-eyegrab steamframe-bt-fixups; do
+    echo "$u (system): $(systemctl is-enabled $u 2>/dev/null) / $(systemctl is-active $u 2>/dev/null)"
+  done
+  echo "failed user units: $(systemctl --user --failed --no-legend --plain 2>/dev/null | awk '{print $1}' | xargs | grep . || echo none)"
+  echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)"
+  echo "paused for VR games: $(cat /run/user/$(id -u)/frametop-pause.json 2>/dev/null || echo 'no (never paused since boot)')"
+  # SteamVR's config folder: a terminal in the desktop has the session's own (docs/design.md).
+  XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p'
+
+  section "What Frametop needs from SteamOS (scripts/update-check.py)"
+  python3 "$repo/scripts/update-check.py" 2>&1 || true
+
+  section "CPU, memory, heat"
+  echo "load: $(cut -d' ' -f1-3 /proc/loadavg)"
+  free -m | sed -n 1,2p
+  df -h / /home 2>/dev/null | awk 'NR == 1 || !seen[$1]++'
+  for f in /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq; do
+    c=${f%/cpufreq/*}; printf '%s %s/%s MHz  ' "${c##*/}" $(( $(cat $f) / 1000 )) $(( $(cat ${f%/*}/cpuinfo_max_freq) / 1000 ))
+  done; echo
+  for z in /sys/class/thermal/thermal_zone*; do
+    echo "$(cat $z/temp 2>/dev/null) $(cat $z/type 2>/dev/null)"
+  done | sort -rn | head -n 5 | awk '{ printf "%.1f C %s\n", $1 / 1000, $2 }'
+  echo "the busiest processes now:"
+  top -b -n 1 -w 160 2>/dev/null | sed -n '7,17p'
+
+  section "/tmp/frametop-session.log (last 60 lines)"
+  tail -n 60 /tmp/frametop-session.log 2>/dev/null || echo "missing"
+  section "SteamVR server: Frametop and errors (last 60 lines)"
+  grep -a -i -E 'ft_pointer|frametop|\[error\]' $logs/vrserver.txt 2>/dev/null | tail -n 60
+fi
+
+if has desktop; then
+  section "Screens' layout"
+  python3 - <<'PY' 2>/dev/null || echo "no ~/.config/frametop-layout.json"
 import json, os
 d = json.load(open(os.path.expanduser("~/.config/frametop-layout.json")))
 print("mode:", d.get("mode"), " auto:", d.get("auto"), " primary:", d.get("primary"))
@@ -84,9 +130,9 @@ for i, s in enumerate(d.get("screens", []), 1):
 print("visibility:", d.get("visibility"))
 PY

-section "Plasma panels and outputs"
-python3 "$repo/session/fix-panels.py" --check 2>&1 || true
-python3 - "$repo" <<'PY' 2>&1 || true
+  section "Plasma panels and outputs"
+  python3 "$repo/session/fix-panels.py" --check 2>&1 || true
+  python3 - "$repo" <<'PY' 2>&1 || true
 import json, subprocess, sys
 sys.path.insert(0, sys.argv[1] + "/layout")
 import ft_layout
@@ -103,8 +149,8 @@ r = subprocess.run(["qdbus6", "org.kde.plasmashell", "/PlasmaShell", "org.kde.Pl
 print("live panels:", (r.stdout or r.stderr).strip())
 PY

-section "Frametop's keyboard and the Steam menu"
-python3 - "$(ask frametop_relay devices)" <<'PY' 2>&1
+  section "Frametop's keyboard and the Steam menu"
+  python3 - "$(ask frametop_relay devices)" <<'PY' 2>&1
 import json, os, sys
 try:
    rules = json.load(open(os.path.expanduser("~/.config/frametop-input.json")))
@@ -121,85 +167,195 @@ print(f"  pointer mode: {d.get('pointer_mode')}, paused: {d.get('paused')}")
 for n in d.get("nodes", []):
    print("  " + json.dumps(n))
 PY
-echo "ft-textinput (tells the relay about text fields): $(pgrep -f '[f]t-textinput' >/dev/null && echo running || echo 'not running')"
-echo "the desktop's KWin input method: $(pgrep -a kwin_wayland | grep -o -- '--inputmethod [^ ]*' | head -n 1 | grep . || echo 'none (the keyboard never opens for text fields)')"
-for p in $(pgrep -x plasmashell); do
-  env=$(tr '\0' '\n' < /proc/$p/environ 2>/dev/null) || continue
-  echo "$env" | grep -q '^XDG_CONFIG_HOME=.*/frametop$' || continue
-  for v in QT_IM_MODULE GTK_IM_MODULE; do  # xim here: Qt and GTK apps never ask for the keyboard
-    printf "the desktop's %s: %s\n" $v "$(echo "$env" | sed -n "s/^$v=//p" | grep . || echo unset)"
+  echo "ft-textinput (tells the relay about text fields): $(pgrep -f '[f]t-textinput' >/dev/null && echo running || echo 'not running')"
+  echo "the desktop's KWin input method: $(pgrep -a kwin_wayland | grep -o -- '--inputmethod [^ ]*' | head -n 1 | grep . || echo 'none (the keyboard never opens for text fields)')"
+  for p in $(pgrep -x plasmashell); do
+    env=$(tr '\0' '\n' < /proc/$p/environ 2>/dev/null) || continue
+    echo "$env" | grep -q '^XDG_CONFIG_HOME=.*/frametop$' || continue
+    for v in QT_IM_MODULE GTK_IM_MODULE; do  # xim here: Qt and GTK apps never ask for the keyboard
+      printf "the desktop's %s: %s\n" $v "$(echo "$env" | sed -n "s/^$v=//p" | grep . || echo unset)"
+    done
  done
-done
-echo "ft-screens debug: $(ask ft_screens debug)"
-echo "ft-screens state (mode, manual, wrist, gesture hand and angle, lasers, game, in games): $(ask ft_screens state)"
-echo "ft-screens pause: $(ask ft_screens 'pause state'), concealed: $(ask ft_screens concealed)"
-echo "SteamVR overlays shown now:"
-XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 timeout 10 \
-  /opt/steamvr/bin/linuxarm64/vrcmd --overlays 2>/dev/null |
-  awk '/^\x27/ { show = $0 !~ /not_visible/ } show' | sed 's/^/  /' | head -n 80
-echo "keyboard, drag and Steam menu lines from ft-screens (last 40):"
-grep -a -E 'keyboard|Steam (in front|out of the way)|: (move|resize|roll)( by| ended| not started)' \
-  /tmp/frametop-screens.log 2>/dev/null | tail -n 40 | sed 's/^/  /'
-echo "text fields, from the relay (last 20):"
-journalctl --user -u frametop-input-relay --no-pager -o short 2>/dev/null | grep 'text field' | tail -n 20 | sed 's/^/  /'
+  echo "ft-screens debug: $(ask ft_screens debug)"
+  echo "ft-screens state (mode, manual, wrist, gesture hand and angle, lasers, game, in games): $(ask ft_screens state)"
+  echo "ft-screens pause: $(ask ft_screens 'pause state'), concealed: $(ask ft_screens concealed)"
+  echo "SteamVR overlays shown now:"
+  XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 timeout 10 \
+    /opt/steamvr/bin/linuxarm64/vrcmd --overlays 2>/dev/null |
+    awk '/^\x27/ { show = $0 !~ /not_visible/ } show' | sed 's/^/  /' | head -n 80
+  echo "keyboard, drag and Steam menu lines from ft-screens (last 40):"
+  grep -a -E 'keyboard|Steam (in front|out of the way)|: (move|resize|roll)( by| ended| not started)' \
+    /tmp/frametop-screens.log 2>/dev/null | tail -n 40 | sed 's/^/  /'
+  echo "text fields, from the relay (last 20):"
+  journalctl --user -u frametop-input-relay --no-pager -o short 2>/dev/null | grep 'text field' | tail -n 20 | sed 's/^/  /'

-section "Input relay (last 60 lines)"
-journalctl --user -u frametop-input-relay -n 60 --no-pager -o short 2>/dev/null
-section "Pointer helper (last 60 lines)"
-journalctl --user -u frametop-pointer -n 60 --no-pager -o short 2>/dev/null
-section "Power service (last 30 lines)"
-journalctl --user -u frametop-power -n 30 --no-pager -o short 2>/dev/null
-# Frametop Voice's log has what was said (each dictation window's words, transcripts, wake word
-# commands): those lines are left out of the report.
-if systemctl --user is-enabled frametop-voice >/dev/null 2>&1; then
-  section "Voice dictation (last 30 lines, without what was said)"
-  journalctl --user -u frametop-voice -u frametop-voice-whisper -n 60 --no-pager -o short 2>/dev/null |
-    grep -v -E 'transcribed|voice command|wake word:|window [0-9.]+s|decode skipped|line up with' | tail -n 30
+  section "Input relay (last 60 lines)"
+  journalctl --user -u frametop-input-relay -n 60 --no-pager -o short 2>/dev/null
+  section "Pointer helper (last 60 lines)"
+  journalctl --user -u frametop-pointer -n 60 --no-pager -o short 2>/dev/null
+  section "Power service (last 30 lines)"
+  journalctl --user -u frametop-power -n 30 --no-pager -o short 2>/dev/null
+  for f in /tmp/frametop-screens.log "$XDG_RUNTIME_DIR/frametop-layout.log"; do
+    section "$f (last 60 lines)"
+    tail -n 60 "$f" 2>/dev/null || echo "missing"
+  done
 fi
+
 # The gaze service, SteamVR's eye tracker and ours, the checks and calibrations, with what
 # looks wrong first. It wakes an idle gaze service for about 20 s, to see the tracker send.
-python3 "$repo/scripts/gaze-report.py" </dev/null 2>&1 || echo "scripts/gaze-report.py failed"
-for f in /tmp/frametop-session.log /tmp/frametop-screens.log "$XDG_RUNTIME_DIR/frametop-layout.log"; do
-  section "$f (last 60 lines)"
-  tail -n 60 "$f" 2>/dev/null || echo "missing"
-done
-section "SteamVR server: Frametop and errors (last 60 lines)"
-grep -a -i -E 'ft_pointer|frametop|\[error\]' $logs/vrserver.txt 2>/dev/null | tail -n 60
+if has gaze; then
+  python3 "$repo/scripts/gaze-report.py" </dev/null 2>&1 || echo "scripts/gaze-report.py failed"
+fi
+
+# Hand tracking: whether it runs, which models ft-hands loaded and whether they're the pinned ones,
+# the cameras (hands/camcheck.py), and ft-hands' own counts: how often each hand was seen, what it
+# lost, how long a step took. Its pinch and grip lines are left out (the counts are in the status).
+if has hands; then
+  section "Hand tracking"
+  python3 "$repo/hands/handsmode.py" status 2>&1 || true
+  units frametop-camd frametop-hands frametop-handview
+  student=$repo/hands/models/student
+  echo "fine-tuned models: $(cat "$student/VERSION" 2>/dev/null || echo 'none in this install (then the stock MediaPipe ones)')"
+  [ -d "$student" ] && "$repo/hands/models.sh" check 2>&1 | tail -n 3
+  echo "last checked for newer ones: $(date -r "$repo/hands/models/.student-checked" '+%F %T' 2>/dev/null || echo never)"
+  grep -E '^\s*(FT_)?HANDS_MODELS(_UPDATE)?=' ~/.config/frametop.conf 2>/dev/null | sed 's/\s*#.*//; s/^/override (ft-hands loads these instead): /'
+  python3 "$repo/hands/camcheck.py" 2>&1 | head -n 30
+  echo "sides: $(cat "$XDG_RUNTIME_DIR/frametop-hands/sides.json" 2>/dev/null || echo 'no sides.json')"
+  for c in 5 6 7; do   # ft-hands' CPUs (HANDS_CPUS)
+    f=/sys/devices/system/cpu/cpu$c/cpufreq
+    [ -r $f/scaling_cur_freq ] && echo "cpu$c: $(( $(cat $f/scaling_cur_freq) / 1000 )) of $(( $(cat $f/cpuinfo_max_freq) / 1000 )) MHz"
+  done
+  echo "hottest thermal zone: $(cat /sys/class/thermal/thermal_zone*/temp 2>/dev/null | sort -n | tail -n 1 | awk '{printf "%.1f C", $1 / 1000}')"
+  gestures='pinch (left|right)|grip  (left|right)'
+  echo "ft-hands' starts, cameras and side decisions (last 20):"
+  journalctl --user -u frametop-hands -n 5000 --no-pager -o short 2>/dev/null | grep -a -v -E "$noise|$gestures" |
+    grep -a -E 'cameras|models|colour|side cameras|hand model|heartbeat|error|failed|Failed|Started|not ' | tail -n 20 | sed 's/^/  /'
+  echo "ft-hands' last status (every minute):"
+  journalctl --user -u frametop-hands -n 5000 --no-pager -o cat 2>/dev/null | grep -a -v -E "$noise|$gestures" |
+    awk '/^[0-9.]+s +[0-9.]+ sets\/s/ { block = $0; next } /^ +(sets with|flag buffer|pinches)/ { block = block "\n" $0 }
+         END { print block }' | sed 's/^/  /'
+  # ft-camd prints each camera's rate once a minute: its other lines (starts, cameras, errors), then
+  # only the last 3 of those.
+  section "Camera broker ft-camd (last 20 lines but its rates, then its last 3 rates)"
+  journalctl --user -u frametop-camd -n 3000 --no-pager -o short 2>/dev/null | grep -a -v -E "$noise" > /tmp/ft-report-camd.$$
+  grep -a -v -E '\]: \[[0-9]+ s\] ' /tmp/ft-report-camd.$$ | tail -n 20
+  grep -a -E '\]: \[[0-9]+ s\] ' /tmp/ft-report-camd.$$ | tail -n 3
+  rm -f /tmp/ft-report-camd.$$
+fi
+
+# Frametop Voice. Its log has what was said (each dictation window's words, transcripts, wake word
+# commands): those lines are left out. The microphone: SteamVR's proxmicmute sets the mic's gain
+# (VA_DEC0/1) to 0 while nobody wears the headset, and PipeWire's echo canceller can fail at boot,
+# which leaves everything after it silent; the level check reads one second and keeps only its level.
+if has voice; then
+  section "Voice"
+  units frametop-voice frametop-voice-whisper steamvr-proxmicmute
+  echo "state: $(timeout 5 python3 "$repo/voice/ft-voice" status 2>&1 | tr -d '\n')"
+  python3 - "$repo" <<'PY' 2>&1 || true
+import sys
+sys.path.insert(0, sys.argv[1] + "/voice")
+import voice_settings as v
+s = v.load()
+print(f"settings ({v.PATH}{'' if v.PATH.exists() else ', not there: the defaults'}): controller {s['controller']} "
+      f"{s['controller_mode']}, live typing {s['live_typing']}, sounds {s['sounds']}, wake word {s['wake_word']} "
+      f"({len(s['wake_phrases'])} phrase(s)), {len(s['commands'])} command(s), advanced: {sorted(s['advanced']) or 'none'}")
+PY
+  [ -x "$repo/voice/models.sh" ] && timeout 60 "$repo/voice/models.sh" check 2>&1 | tail -n 4
+  echo "the mic's gain (0 = muted by proxmicmute, the headset off):" \
+    "VA_DEC0 $(amixer -c 0 cget numid=23 2>/dev/null | sed -n 's/.*: values=//p')," \
+    "VA_DEC1 $(amixer -c 0 cget numid=24 2>/dev/null | sed -n 's/.*: values=//p')"
+  r=$(ask ft_gazed status)
+  echo "headset worn (the gaze service): $(echo "$r" | python3 -c 'import json, sys; print(json.load(sys.stdin).get("headset_on"))' 2>/dev/null || echo unknown)"
+  echo "default source: $(wpctl inspect @DEFAULT_AUDIO_SOURCE@ 2>/dev/null | sed -n 's/.*node.name = "\(.*\)"/\1/p' | head -n 1)," \
+    "$(wpctl get-volume @DEFAULT_AUDIO_SOURCE@ 2>/dev/null)"
+  echo "PipeWire's capture chain (state, error):"
+  pw-dump 2>/dev/null | python3 -c '
+import json, sys
+for o in json.load(sys.stdin):
+    if o.get("type") == "PipeWire:Interface:Node":
+        i = o.get("info") or {}
+        p = i.get("props") or {}
+        if p.get("media.class") in ("Audio/Source", "Stream/Input/Audio"):
+            name, state, error = p.get("node.name"), i.get("state"), i.get("error")
+            print(f"  {name}: {state}" + (f", {error}" if error else ""))
+' 2>&1 || echo "  pw-dump failed"
+  echo "who has the capture device open: $(fuser /dev/snd/pcmC*c 2>/dev/null | xargs -r -n 1 sh -c 'cat /proc/$0/comm' 2>/dev/null | sort | uniq -c | xargs | grep . || echo nobody)"
+  python3 -I - <<'PY' 2>&1
+import math, subprocess
+from array import array
+try:
+    raw = subprocess.run(["timeout", "1.2", "pw-record", "--raw", "--rate", "16000", "--channels", "1", "--format", "s16", "-"],
+                         capture_output=True).stdout
+except OSError as e:
+    raise SystemExit(f"mic level: can't run pw-record ({e})")
+a = array("h", raw[: len(raw) // 2 * 2])
+if not a:
+    raise SystemExit("mic level: no samples from the default source")
+peak = max(abs(x) for x in a) / 32768
+rms = math.sqrt(sum(x * x for x in a) / len(a)) / 32768
+print("mic level over 1 s: " + ("digital silence (all zeros)" if peak == 0 else
+      f"peak {20 * math.log10(peak):.0f} dBFS, RMS {20 * math.log10(max(rms, 1e-9)):.0f} dBFS"))
+PY
+  echo "PipeWire and WirePlumber errors (last 15):"
+  journalctl --user -u pipewire -u wireplumber -n 3000 --no-pager -o short 2>/dev/null | grep -a -i -E 'error|cannot|failed' | tail -n 15 | sed 's/^/  /'
+  section "Voice dictation (last 30 lines, without what was said)"
+  journalctl --user -u frametop-voice -u frametop-voice-whisper -n 200 --no-pager -o short 2>/dev/null | grep -a -v -E "$noise" |
+    grep -v -E 'transcribed|voice command|wake word:|window [0-9.]+s|decode skipped|line up with' | tail -n 30
+fi
 EOF

 if [ "$watch" -gt 0 ]; then
  echo
-  echo "Now make the problem happen within $watch seconds: for example, open the Steam menu, try to"
-  echo "drag a window by its bar, and click into a text field. Recording..."
-  on_frame_script "$watch" >> "$out" 2>&1 <<'EOF' || true
-seconds=$1
-printf '\n===== Watched for %s s from %s\n' "$seconds" "$(date '+%F %T')"
+  echo "Now make the problem happen within $watch seconds. For example:"
+  [[ ,$areas, == *,desktop,* ]] && echo "  desktop: open the Steam menu, drag a window by its bar, click into a text field"
+  [[ ,$areas, == *,gaze,* ]] && echo "  gaze: point with your eyes, at what goes wrong"
+  [[ ,$areas, == *,hands,* ]] && echo "  hands: pinch and tap, move your hands in and out of view, where tracking fails"
+  [[ ,$areas, == *,voice,* ]] && echo "  voice: dictate, or say your wake word (only the mic's level and word counts are kept)"
+  echo "Recording..."
+  on_frame_script "$FRAME_REPO" "$areas" "$watch" >> "$out" 2>&1 <<'EOF' || true
+repo=$1 areas=$2 seconds=$3
+has() { [[ ,$areas, == *,$1,* ]]; }
+noise='container (exec|exec_died|init|start|died|attach|cleanup)'
 since=$(date '+%F %T')
 start=$(wc -l < /tmp/frametop-screens.log 2>/dev/null || echo 0)
-python3 -I - "$seconds" <<'PY'
-import socket, sys, time
-s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
-s.bind("")
-s.settimeout(1)
-end, last = time.monotonic() + float(sys.argv[1]), None
-print("ft-screens' debug line whenever it changed:")
-while time.monotonic() < end:
-    try:
-        s.sendto(b"debug", "\0ft_screens")
-        line = s.recv(4096).decode()
-    except OSError as e:
-        line = f"no answer ({e})"
-    if line != last:
-        print(time.strftime("%H:%M:%S") + f"{time.time() % 1:.2f}"[1:], line, flush=True)
-        last = line
-    time.sleep(0.2)
-PY
-printf '\n===== ft-screens log while watching\n'
-tail -n +"$((start + 1))" /tmp/frametop-screens.log 2>/dev/null | tail -n 300
-for u in frametop-input-relay frametop-pointer; do
-  printf '\n===== %s while watching\n' "$u"
-  journalctl --user -u $u --since "$since" --no-pager -o short 2>/dev/null | tail -n 200
-done
+printf '\n===== Watched %s for %s s from %s\n' "${areas//,/, }" "$seconds" "$since"
+if [ -f "$repo/scripts/report-watch.py" ]; then
+  python3 -I "$repo/scripts/report-watch.py" "$seconds" "$areas"
+else
+  echo "scripts/report-watch.py isn't in this install (older than 0.3.0-exp.12): logs only"
+  sleep "$seconds"
+fi
+log() {  # log TITLE UNIT...: their journal while watching
+  local title=$1; shift
+  printf '\n===== %s while watching\n' "$title"
+  journalctl --user $(printf -- '-u %s ' "$@") --since "$since" --no-pager -o short 2>/dev/null | grep -a -v -E "$noise" | tail -n 300
+}
+if has system; then
+  printf '\n===== warnings and errors while watching\n'
+  journalctl --user -p warning --since "$since" --no-pager -o short 2>/dev/null | grep -a -v -E "$noise" | tail -n 100
+  journalctl -k -p warning --since "$since" --no-pager -o short 2>/dev/null | tail -n 50
+fi
+if has desktop; then
+  printf '\n===== ft-screens log while watching\n'
+  tail -n +"$((start + 1))" /tmp/frametop-screens.log 2>/dev/null | tail -n 300
+  log "input relay" frametop-input-relay
+  log "pointer helper" frametop-pointer
+fi
+if has gaze; then
+  log "gaze service" frametop-gaze
+  has desktop || { printf '\n===== the pointer helper'"'"'s gaze lines while watching\n'
+    journalctl --user -u frametop-pointer --since "$since" --no-pager -o short 2>/dev/null | grep -a -i gaze | tail -n 100; }
+fi
+if has hands; then
+  log "ft-hands (pinches and grips too)" frametop-hands
+  log "ft-camd" frametop-camd
+fi
+if has voice; then
+  printf '\n===== Frametop Voice while watching (without what was said)\n'
+  journalctl --user -u frametop-voice -u frametop-voice-whisper --since "$since" --no-pager -o short 2>/dev/null |
+    grep -a -v -E "$noise" | grep -v -E 'transcribed|voice command|wake word:|window [0-9.]+s|decode skipped|line up with' | tail -n 200
+  log "PipeWire, WirePlumber and proxmicmute" pipewire wireplumber steamvr-proxmicmute
+fi
 EOF
  echo "Recorded."
 fi
@@ -51,7 +51,8 @@ HANDS_BRIGHT_ON=40         # hand tracking, auto: the colour frames' mean bright
 HANDS_BRIGHT_OFF=25        # ... and back to the mono cameras under this
 HANDS_COLOR_LEFT=color_video0  # hand tracking: which colour camera is passthrough_left (hands/tools/check_color.py tells)
 HANDS_COLOR_CROP=subtract  # hand tracking: how the colour calibration's crop applies, subtract | none (check_color.py tells)
-#HANDS_MODELS=            # hand tracking: a folder of palm.ncnn.* and hand.ncnn.* to use instead of the stock models (hands/models/ncnn)
+#HANDS_MODELS=            # hand tracking: a folder of palm.ncnn.* and hand.ncnn.* to use instead of the fine-tuned models (hands/models/student; hands/models/ncnn has the stock ones)
+HANDS_MODELS_UPDATE=1      # hand tracking: 1 = when it starts, fetch newer fine-tuned models if GitHub's main (stable) or experimental branch pins them (hands/models.pin), without a new release | 0 = keep the ones it came with
 HANDS_MATTE=circle         # hand cutouts: circle = a round hole per hand | mesh = hand-shaped, from the skeleton | capsules = ft-hands' capsules (the old look)
 HANDS_MATTE_MARGIN=1.5     # hand cutouts: degrees (seen from the eyes) the hole reaches past the hand, held still (Hand cutout margin on the Hands page, live)
 HANDS_MATTE_MOTION=25      # ... plus how far the hand moves in this many ms
@@ -167,6 +168,6 @@ DISPLAY_MOVE_DEG=0.5 # displays off: or turning this far (degrees) within 10 s
 LAYOUT_GRAB_OFFSET=0.075   # arranging screens: where the floating window's grab bar is, in metres below the screen
 LAYOUT_SLIDE_SPEED=0.5     # arranging screens: how fast the invisible controller slides a grabbed screen (m/s)

-HANDS_PINCH_END=0.035      # ft-hands: a pinch ends once thumb and index are this far apart (m)
-HANDS_PINCH_DRAG_OPEN=0.05 # ... a pinch held 0.3 s or more (a drag) ends this far apart, or HANDS_PINCH_DRAG_OPEN_S (0.12) past HANDS_PINCH_END
-HANDS_GRIP_TAKEOVER_S=0.3  # ... a grip (fist) ends a pinch only this soon after it began
+HANDS_PINCH_END=0.045      # ft-hands: a pinch ends once thumb and index are this far apart (m)
+HANDS_PINCH_DRAG_OPEN=0.065 # ... a pinch held 0.3 s or more (a drag) ends this far apart, or HANDS_PINCH_DRAG_OPEN_S (0.2) past HANDS_PINCH_END
+HANDS_GRIP_TAKEOVER_S=0.1  # ... a grip (fist) ends a pinch only this soon after it began
Author	SHA1	Message	Date
DeeJanuzandClaude Opus 5.5	01796eb299	Releases: ship the Hand Recorder The image builds ft-handpanel (hands/rec/build.sh, against scripts/openvr.sh like gaze), and a release's installer offers the recorder: ft-camd's capabilities and the menu entry, nothing to build. The FrameDrop window has a Hand Recorder box, and --no-hand-recorder leaves it out. An entry from the release before is renewed without asking, since it points into that release's folder. The recorder runs ft-hands through scripts/in-box instead of the "dev" container, so a release uses its own container, and its repair hints say to reinstall the release rather than run a build script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 14:32:07 -06:00
DeeJanuzandClaude Opus 5.5	3e6dba8276	Report: an area per flag, and --watch records each area live scripts/report.sh takes --system, --desktop, --gaze (--eyes), --hands and --voice; with none it reports on all of them, as before. Every report keeps the versions and the settings. New in it: CPU, memory, heat and the busiest processes (--system), and Frametop Voice (--voice): its services, state, settings (counts, not phrases or commands), speech models, and the microphone: proxmicmute's gain, PipeWire's capture chain with its errors, who holds the capture device, and its level over one second (no sound is kept). --watch SECONDS now runs scripts/report-watch.py for the chosen areas, one thread each: ft-screens' debug line (desktop), the gaze service's status (gaze), ft-hands' published frames read under their sequence lock each second (rate, age, each hand's share and confidence, closest thumb-index and thumb-middle tips) and the side cameras' decision (hands), Frametop Voice's events with what was said replaced by word counts and the mic's level each second with its gain when it goes silent (voice), and load, clocks, heat, memory and the busiest processes every 5 s (system). Then each area's logs from that time. README's Reporting problems says how. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 14:16:16 -06:00
DeeJanuzandClaude Opus 5.5	098ba9e71e	Hands: pinches let go less easily by default Fresh installs ended taps and pinches at 3.5 cm, the values the headset's tester found "release way too easily" on 2026-10-11 (ends came at 4-6 cm, grips took pinches 130 ms in). The headset has run with HANDS_PINCH_END=0.045, HANDS_PINCH_DRAG_OPEN=0.065, HANDS_PINCH_DRAG_OPEN_S=0.2 and HANDS_GRIP_TAKEOVER_S=0.1 since; they are now the defaults. pinch_test keeps the old values for its rule tests and checks the new defaults on their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 13:17:00 -06:00
DeeJanuzandClaude Opus 5.5	eb31589aed	Report: hand tracking's state (models, cameras, sides, ft-hands counts) scripts/report.sh had no hand tracking at all, so a "tracking is bad" report couldn't tell a stock model from the pinned one, a failed camera start from bad light, or a starved tracker. Its new section: HANDS_MODE and the three services, the fine-tuned models' version and models.sh check, any HANDS_MODELS override, hands/camcheck.py, sides.json, CPU 5-7 clocks and the hottest thermal zone, ft-hands' starts and last status, and ft-camd's start lines. --watch adds ft-hands' log (pinches) too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 13:17:00 -06:00
DeeJanuzandClaude Opus 5.5	1b256289b9	Hands: new hand models reach installs without a release The pin moves out of models.sh into hands/models.pin. When hand tracking starts, frametop-hands.service runs `hands/models.sh auto`: it reads models.pin from GitHub (main for stable releases, experimental for experimental ones and clones on that branch), and if it pins other models, fetches them into models/.student-new, checks every sha256 and swaps the folder in whole with one rename. At most every 15 minutes and within a minute; offline, a 404 or anything not matching keeps the models it has. A pin from GitHub is checked line by line before use: a 40-hex commit, a plain folder name, file names from a fixed list, 64-hex sums, and a format number, so a model that needs a newer ft-hands can be held back from older installs. The Hugging Face repo stays fixed in the script. HANDS_MODELS_UPDATE=0 turns it off; `hands/models.sh update` checks now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 09:46:01 -06:00
DeeJanuzandClaude Opus 5.5	9eac1e29a9	Hands: ship the fine-tuned hand models (v13b-p1) instead of stock MediaPipe hands/models.sh fetches MediaPipe's palm detector and hand landmark model, retrained on the Frame's infrared cameras with the Frametop Hands dataset, from a pinned commit of the Hugging Face repo DeeJanuz/frametop-hands-models, and checks every file's sha256. hands/build.sh runs it, so source installs and the release image both get them; the release smoke test checks them. The weights are CC BY-NC 4.0, not MIT, so they stay out of this repository (gitignored, dockerignored); their LICENSE and model card come with them. ft-hands and ft-handreplay now default to hands/models/student when it's there (HANDS_MODELS and --models still override; --int8 keeps the stock models, which are the only ones with 8-bit files). Replays through ft-hands, hands tracked, stock -> v13b-p1: bright room 442 -> 960, dim 912 -> 1224, lit 782 -> 1433. On held-out contributors the landmark error drops from 5.2 to 3.8 px (median). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 08:30:04 -06:00