# Hand recorder: design (Phase 1 of the hands plan)

The hand recorder guides a person through recording their hands with the headset's cameras. It saves the recordings as files, lets them review and delete anything, then exports a package to contribute to the open hand dataset. Recordings never leave the headset unless the person uploads them.

The plan this belongs to is `~/Desktop/Projects/frame-hands/notes/hands-plan.md` (on the maintainer's Frame). In short, the dataset trains a small hand model for Frametop's hand cutouts.

## Parts

| Part | What it does |
|---|---|
| `hands/rec/panel/ft-handpanel.cpp` (C++, dev container) | The headset panel: a SteamVR overlay fixed to the head that shows the prompts. It also places the "touch the dot" target in the room and logs head and controller poses. Driven over `@ft_handpanel`. |
| `hands/rec/session.py` (Python, no Qt) | The session runner. It reads `script.json`, starts and stops the recordings, and drives the panel. It reads the live hands file for feedback and writes each take's files. It also runs from the command line (`--dry-run`) for testing. |
| `hands/rec/script.json` | The guided script: sections, prompts, timings. |
| `hands/rec/poses/` | The pose pictures, `poses.json` and its PNGs (below). |
| `hands/rec/takes.py` (Python, no Qt) | Reads sessions and takes from disk: frame sets for review, deleted ranges, export (compress, strip, manifest, checksums). |
| `hands/rec/ft_handrec.py` + `main.qml` (PySide6 + Kirigami, dev container) | The desktop window: consent, the before-you-start checklist, the session controls, review, export and upload instructions. |
| `hands/rec/ft-handrec` | The host launcher, like `input-settings/ft-input-settings`. |
| `hands/rec/build.sh` | Builds `ft-handpanel` into `hands/rec/build/`, like `gaze/build.sh`. |
| `hands/rec/CONSENT.md`, `hands/rec/UPLOAD.md` | The texts the window shows. |
| ft-hands `--record-hz N` (done) | Records at most N frame sets a second. The recorder uses 10. |
| `hands/camcheck.py` (shared, standard library) | Are all four mono cameras running? The recorder runs it before a session and when a step sees no hands (below; `hands/README.md`, "Camera check"). |

Every part runs in the dev container, as ft-hands and Input Settings do. The host Python isn't used: it lacks PySide6 and zstd. `setup/dev-container.sh` gains `zstd`.

## Processes during a session

- **ft-camd** publishes the camera ring. If it isn't running, the session starts it as the transient user unit `frametop-handrec-camd.service`, the way `hands/ft-cutouts` starts `frametop-cutouts-camd.service` (needs `hands/build/ft-camd` with capabilities: `hands/run.sh caps`). An ft-camd already running from ft-cutouts or ft-handsctl is used as it is.
- **A tracking ft-hands** gives feedback through the hands file: which hands are seen, the palm's distance, the index tip. If none is running, the session starts `ft-hands --no-gestures --status 0` (unit `frametop-handrec-hands.service`). An ft-hands already running is used as it is.
- **A recording ft-hands** runs once per recording part: `ft-hands --record-only --record DIR --record-for SECONDS --record-hz 10 --status 0 --sides auto|0|1` (below, "Side cameras"). It runs as a plain child process of the session, ended with SIGTERM when the part ends. SIGTERM ends ft-hands' loop, and `Recorder` writes out its queue when it's destroyed. In step mode (below) a part is one step's countdown and hold, so a take has one part per step (about 40 in the hand poses); in auto mode a take is one part, plus one more after each pause. `--record-for` is only a safety net.
  - Why a process per part rather than one kept alive and paused: measured in the dev container with `ft-ringplay`'s ring (2026-10-02), `ft-hands --record-only` writes its first set 16-27 ms after it starts and ends 4-6 ms after SIGTERM, so a new part costs nothing the 3 s countdown doesn't cover. Every reader already takes parts in order (`takes.py`, `validate.py` through the export's single stream, the labeller's `fhl_io.py`, numbering `sets-10.bin` after `sets-9.bin`), ft-hands needs no new control, and nothing is written while a step waits. Before the hold starts the session checks that the part has written a set (`Recorder.has_data`, up to 3 s more), so the hold is recorded from its first frame.
- **Side cameras.** ft-camd can name the two side cameras the wrong way round (hands/README.md, "Which camera is which"). The tracking ft-hands decides from the hands within about 2 s of them being in view (`HANDS_SWAP_SIDES=auto`, hands/track/sides.h) and publishes that in `/run/user/UID/frametop-hands/sides.json`. The session reads it (`sides.py`, `read_live`) and stores it in session.json `"sides"`. Each later part is recorded named right (`--sides 1` or `0`). Parts recorded before the decision use ft-camd's names (`--sides auto`; a record-only ft-hands can't tell), and readers rename them (`sides.py`). Each part's `names_swapped` goes into take.json `"parts"`. Without a tracking ft-hands nothing decides: `"swapped": null`, the names stay as recorded, and the maintainer's check (`hub_review check`, check_sides on a few sets per take) tells. `takes.py sides SESSION --set swapped|named` records a decision by hand.
- **ft-handpanel** runs as a child process with `--watch-stdin`. It shows the panel and logs poses during each take.
- **The headset button's reader** is a thread of the session (`ButtonReader`, below), not a process.

Test hooks:
- `--ring PATH` goes to both ft-hands (`ft-ringplay` publishes a recording there, so the whole flow runs without the headset).
- `--no-start` uses only what's already running.
- `--dry-run` runs no processes and only prints the panel commands, with timing sped up by `--speed X`.
- `--next-after S` presses Next by itself after S seconds of waiting (real time), so a step-mode session runs unattended. Lines on stdin steer it too: `n` or an empty line is Next, `p` pause or resume, `r` redo, `s` skip the section, `q` stop.
- `--no-headset-button` leaves the headset's button alone (`ft-handrec --no-headset-button` too). `--button-device PATH` reads it from PATH instead, an event device or a FIFO of `input_event` structs, also in a dry run: a simulated button for tests.
- `--auto` runs the timed flow; `--poses DIR` takes the pose pictures from DIR; `--plan` prints the sections, their steps and length.
- `--ignore-cameras` (`ft-handrec --ignore-cameras` too) starts even when the camera check fails. A dry run and `--ring` skip the check by themselves.

## Files

```
~/.local/share/frametop/hands/contrib/
  profile.json                  consent and contributor id (below)
  sessions/<YYYYMMDD-HHMMSS>/   (a second session started in the same second gets -2, and so on)
    session.json                the session: checklist answers, lighting, versions, mode, takes
    calibration.json            /persist/xrservice.json with identifying fields removed (below)
    device.json                 the rig's pose in the CAD frame from /persist/device_config.json (below)
    takes/<NN>-<section>/
      sets.bin                  ft-hands' recording (FHSET01, hands/track/record.h), 10 sets/s: the first part
      sets-2.bin, sets-3.bin, ...  the next parts (step mode: one per step; any mode: after a pause)
      prompts.jsonl             what the person was asked to do, when (below)
      poses.jsonl               head and controller poses from ft-handpanel (below)
      take.json                 {"section", "title", "started_ns", "ended_ns", "status": "complete"|"stopped"|"skipped",
                                 "deleted": [[from_ns, to_ns], ...], "notes": "",
                                 "parts": {"sets.bin": {"names_swapped": false}, "sets-2.bin": {...}}}
  exports/<session>/            what export writes (below)
```

All `_ns` times are CLOCK_MONOTONIC nanoseconds, the clock of `dqbuf_ns` in sets.bin and of `capture_ns` in the hands file. sets.bin's `capture_ns` is the camera clock (CLOCK_MONOTONIC_RAW).

### profile.json

```json
{"schema": 1, "contributor": "<random uuid4>", "consent": {"version": "2026-10-02", "accepted": "<ISO time>", "adult": true},
 "optional": {"handedness": "right|left|both|", "notes": ""}}
```

No name, email, or account. The contributor id is random, so several sessions from one person can be held out together in evaluation. Withdrawal also goes by that id.

### session.json

```json
{"schema": 1, "tool": "ft-handrec <git describe>", "started": "<ISO>", "contributor": "<uuid>",
 "lighting": {"chosen": "dim|room|daylight|indoor", "source": "measured|picked", "measured": "indoor|daylight|",
              "ambient_ir": 0.0, "ring": {"<cam>": {"mean": 0.0, "dark_mean": 0.0}}},
 "checklist": {"objects": ["pencil", "phone", "cup", "keyboard", "mouse", "gamepad", "small"], "own_objects": ["..."],
               "controllers": "straps|none", "sleeves": "short|long|", "rings": false, "watch": false, "notes": ""},
 "device": {"steamos": "<VERSION_ID from /etc/os-release>", "steamvr": "<version if known>", "cameras": [{"name", "width", "height"}]},
 "mode": "step|auto", "quick": false, "shuffle": {"seed": 123, "sweeps": {"pose-sweeps": [{"id", "hands", "cues"}]}},
 "takes": ["01-hand-size", "..."],
 "camera": {"status": "ok|unknown|degraded", "reason": "..."}, "stop_reason": "...",
 "sides": {"swapped": true|false|null, "decided_by": "auto|config|option|manual", "state": "decided|confirmed|...",
           "evidence": {"as_named": 0, "swapped": 10, "seconds": 1.8, "miss_mm": [-1, 4.2], "probes": 30, "found": 10},
           "decided_at": "<ISO>", "decided_ns": 0}}
```

`sides`: whether ft-camd's side camera names were backwards during the session (`swapped`), as the live tracker decided it (above, "Side cameras"); `null` while nobody knows. A part's names are right when its take.json `names_swapped` equals `swapped`. Parts without a `parts` entry were recorded with ft-camd's names. Sessions from before this have no `sides`.

`camera` is the camera check's verdict at the start (no paths or log lines: those go to `session.log`). `stop_reason` is there when a session was stopped at the no-hands screen, with the check's result.

A session started before step mode existed has no `mode`: it ran as `auto`. `quick` and `shuffle` (below, "Sweeps" and "Quick round") are missing from sessions before the sweeps.

### calibration.json

This is a copy of `/persist/xrservice.json` (`/run/host/persist/` in the container). Keep the cameras' intrinsics and extrinsics, and drop anything that identifies the unit: keys containing `serial`, `sn`, `uuid`, `mac` or `id`, or values that look like serial numbers. List what was removed in `session.json` (`"calibration_removed": [...]`), so a reviewer can check it.

### device.json

The head frame needs the rig's pose in the CAD frame, from `/persist/device_config.json`. Only two of its keys are kept, `cv.cad_from_cal` (Cam0 in the CAD frame) and `head` (the head in CAD), in the shape the labeller in frame-hands `train/label` reads, as its `cut.py` writes it:

```json
{"cv": {"cad_from_cal": {"method": "FrontAndUpperCamPositions", "plus_x": [x, y, z], "plus_z": [x, y, z], "position": [x, y, z]}},
 "head": {"plus_x": [x, y, z], "plus_z": [x, y, z], "position": [x, y, z]}}
```

The rest of that file identifies the unit (serial number, display EDID) and is never copied. The two kept keys go through `strip_calibration` as well, and anything it removes is listed in `calibration_removed` as `device.json:<path>`. Sessions recorded before device.json existed have none: they still validate, with a warning, and the labeller falls back to another unit's pose.

### prompts.jsonl

One JSON object per line:

```json
{"t": 123, "event": "take", "section": "static-poses", "take": "03-static-poses"}
{"t": 123, "event": "prompt", "id": "static-poses/fist/left/near", "text": "...", "hands": "left|right|both|none",
 "pose": "fist", "distance": "near|mid|far|", "position": "centre|left|right|up|down|", "object": "", "controller": false}
{"t": 123, "event": "target", "id": "touch/3", "head": [x, y, z], "room": [x, y, z], "state": "show|hold|done|timeout"}
{"t": 123, "event": "bar", "target": 0.0}
{"t": 123, "event": "feedback", "left": true, "right": false, "palm_m": [0.0, 0.0]}
{"t": 123, "event": "pause"}
{"t": 123, "event": "resume"}
{"t": 123, "event": "ready", "id": "static-poses/fist/left/near", "seconds": 3}
{"t": 123, "event": "wait"}
{"t": 123, "event": "redo", "id": "static-poses/fist/left/near", "from": 123, "to": 123}
{"t": 123, "event": "nohands", "id": "hand-size/flat/both", "reads": 120, "published": 118}
{"t": 123, "event": "end", "status": "complete|stopped|skipped"}
```

`feedback` is written about twice a second while recording. It's the live tracker's view, kept for later checks; it's not a label.

A prompt holds from its `prompt` until the next `prompt`, `ready`, `wait` or `end`. A step in step mode reads:

```
ready      Next pressed: recording part N starts, the 3-2-1 countdown runs (recorded, no label)
prompt     the hold: its labels start here
(bar, target, feedback, pause/resume during the hold)
wait       the hold is over: no labels from here; part N stops
```

The touch-the-dot targets after the first follow straight on: no `ready` or `wait` between them. `redo` marks a try done again (R): `from` is that step's `ready` (or its `prompt` if it had none), `to` its end. Its prompt is skipped; the sets stay. In auto mode there's no `ready` or `wait`, and the intro is a recorded prompt `<section>/intro`. `nohands` marks the first hand-size step stopped because no hand was seen (below, "Camera check"); a `redo` over the same range follows it, so the try gets no labels. A sweep step (below, "Sweeps") has a `prompt` per cue, each with its `pose`, `"cue": true` and `"step"`, the step's id; its `ready`, `wait` and `redo` carry the step's id. Readers that knew only `prompt` and `end` keep working, but they'd give the countdown to the step before: `hub_review.py` (frame-hands `train/hub`) shows it as "(countdown)", redone prompts as "(redone)" and cues as "[pose] text"; `FORMAT.md` in the dataset repo has the rules.

### poses.jsonl

ft-handpanel writes one line per sample, 250 a second, from `GetDeviceToAbsoluteTrackingPose(TrackingUniverseStanding, 0)`:

```json
{"t": 123, "hmd": {"m": [12 floats, row-major 3x4], "r": 200, "ok": true},
 "left": {"m": [...], "r": 200, "ok": true}, "right": null}
```

`r` is `ETrackingResult` (200 = Running_OK, 201 = Running_OutOfRange, and so on). `left` and `right` are the devices holding those controller roles, or null. No device serials are logged.

## The panel (`ft-handpanel`)

- **Placement.** A SteamVR overlay fixed to the head, like `gaze/panel/ft-gazepanel.cpp`: key `frametop.handpanel`, sort order 250. It sits 1.2 m ahead, centred 12 degrees above straight ahead, so the hands stay clear below it. It's 36 degrees wide, 4:3, 1024x768 pixels, dim and see-through. It's drawn on the CPU with stb_truetype (and stb_image for the pose pictures, PNG only, the same pinned stb commit) into three shared DMA-BUFs SteamVR imports once, as ft-gazepanel does, and is drawn again only when something changes.
- **Layout.** The title and step on top (a red "Rec" by the step while recording), a rule. With a pose picture or a diagram, a 14-degree column on the left holds the picture (or the flipped copy and the picture side by side) and the where-to diagram under it; the text takes the right. The text column holds the instruction, the orange note, the big countdown ("3", "2", "1", then "Hold" or "Go") and the cyan action line ("Ready? Press Space or click Next"), centred together. At the bottom: the near/far bar, the hand chips, the time-left bar and the key hints.
- **Socket.** Abstract unix datagram `@ft_handpanel` (`--socket NAME`). A sender with an address gets `ok ...` or `error ...`.
- **Options:** `--watch-stdin` (quit when stdin closes), `--socket NAME`, `--distance M`, `--no-vr`. `--no-vr` makes no SteamVR connection and prints each picture's text to stdout: for testing without a headset.

Commands (UTF-8; `|` starts a new line in text):

| Command | Effect |
|---|---|
| `show` / `hide` | The panel. A "show" makes it visible with its first picture. |
| `title <text>` | Big line at the top. |
| `step <text>` | Small line at the top right, e.g. `Section 3 of 11`. |
| `text <text>` | The instruction, large, wrapped to the panel's width, centred. |
| `note <text>` | An orange line under the instruction: a warning ("I can't see your left hand"). Empty clears it. |
| `countdown <0..1>` / `countdown off` | A thin bar along the bottom: the share of this prompt's time left. |
| `hands <left> <right>` | Two chips, "Left hand" and "Right hand", each `seen` (green), `lost` (orange) or `off` (hidden). |
| `bar <target 0..1> <current 0..1 or -1> [near label] [far label]` / `bar off` | The near/far bar for the push out and back: a horizontal track with a target marker and the hand's current position. |
| `paused on` / `paused off` | A "Paused" overlay over the picture. |
| `image <path> [mirror\|both]` / `image off` | The pose picture, a PNG (below), in the left column. `mirror` flips it (a left hand); `both` draws a flipped copy on its left. A file that can't be read: `error ...` and no picture. |
| `where <position\|-> <distance\|->` / `where off` | The where-to diagram under the picture: a 3x3 front view with the asked cell lit (`centre`, `left`, `right`, `up`, `down`, and the push sections' `chest`, `desk`, `eye`), and a side view of the head and three marks for `near` ("Close"), `mid` ("Halfway out") and `far` ("Arm out"). `-` leaves that half out. |
| `action <text>` | The cyan line under the instruction (empty clears it). |
| `big <text>` | Large cyan text under the instruction: the countdown (empty clears it). |
| `keys <text>` | The faint key hints along the bottom. |
| `rec on` / `rec off` | The red "Rec" by the step line. |
| `strip <cue> <path>\|<mode>\|<label>;...` / `strip off` | A sweep's row of pose pictures (path `-`: none; mode `-` or `mirror`), each with its label under it; the one at index `cue` (from 0, -1 for none) framed in cyan and bright, the others dim. It sits along the bottom of the space left, the where-to diagram at its right end if there is one, and the text above it; it takes the left column's place. Each picture is loaded once. A file that can't be read: `error can't read ...`, and that item shows an empty frame. |
| `target <x> <y> <z> [show\|hold <0..1>\|done]` / `target off` | The touch target: a small sphere-like dot about 2 cm across, in its own overlay (`frametop.handpanel.target`). The point is in the head frame (metres, +x right, +y up, -z forward). The first `target` with a new point places it in the room with the current HMD pose, and it stays there. Later commands with the same point change only the state. `hold` draws a filling ring, `done` turns it green. Reply: `ok <room x> <room y> <room z>`. |
| `poses start <path>` / `poses stop` | Log poses to the path (appending, `poses.jsonl` format above) from a thread at 250 Hz, until stopped. |
| `devices` | Reply: `ok hmd <r> left <r or -> right <r or ->`, the current `ETrackingResult` values (`-` for no device in that role). |
| `head` | Reply: `ok <12 floats>`, the current HMD pose (standing universe). |
| `ping` | `ok shown` or `ok hidden`. |

It quits on SteamVR's quit event, as ft-gazepanel does. `--no-vr --dump DIR` writes each picture to `DIR/panel.pam`, to check the layout without a headset.

## The pose pictures (`poses/`)

`poses/poses.json` maps the script's `pose` ids to pictures: `{"<pose>": {"file": "<name>.png", "two_hands": false, "caption": "..."}}`. Each PNG is RGBA, square (512x512), drawn as the wearer sees it: a right hand, unless `two_hands` (then it shows both). How a prompt shows it (`session.pose_view`):

| Prompt's `hands` | Picture |
|---|---|
| `right` (and `any`, `none`, empty) | as drawn |
| `left` | flipped left to right (`image ... mirror`) |
| `both`, not `two_hands` | a flipped copy on the left, the picture on the right (`image ... both`) |
| anything, `two_hands` | as drawn |

A pose with no entry, or whose file is missing, shows no picture: the text alone. The session reads `poses.json` when it starts. The window shows the same picture (QML `Image.mirror`), its caption, and the same diagram.

## The script (`script.json`)

```json
{"version": 1,
 "cue_names": {"thumbs-up": "Thumbs up", "...": "..."},
 "sections": [
   {"id": "hand-size", "title": "Hand size", "requires": [], "intro": "text shown before the first prompt: 4 s, or until Next", "go": "Hold",
    "quick": true, "prompts": [{"text": "...", "cues": ["flat", "flat-back", "spread"], "cue_s": 5, "hands": "both", "distance": "mid"}]},
   {"id": "pose-sweeps", "title": "Hand poses", "kind": "sweep", "quick": true, "cue_s": 4, "step_s": 20,
    "groups": [["open", "fist", "point", "pinch"], ["ok", "thumbs-up", "spread", "claw"], ["count-1", "...", "count-5"]],
    "sweeps": [{"hands": "both", "group": "next", "text": "..."}, {"hands": "left", "group": "any", "quick": false, "text": "..."}]},
   {"id": "objects", "title": "Things you hold", "requires": ["objects"], "for_each": "object",
    "prompts": [{"text": "Pick up the {object} and use it the way you normally would.", "seconds": 15, "hands": "both", "object": "{object}"}]},
   {"id": "touch", "title": "Touch the dot", "kind": "targets", "hold_s": 1.0, "timeout_s": 8,
    "targets": [[0.0, -0.15, -0.40], ...], "text": "Touch the dot with your index fingertip and hold still."},
   {"id": "controller-push", "title": "Depth with controllers", "requires": ["controllers"], "kind": "bar",
    "intro": "...", "heights": ["chest", "desk", "eye", "left", "right"], "reps": 5, "period_s": 6, "near_m": 0.2, "far_m": 0.6}
 ]}
```

- `requires`: `objects` (at least one object ticked), `controllers` (straps ticked). A section whose requirements aren't met is skipped and logged.
- `go`: the word the countdown ends on, "Go" unless set ("Hold" for the still poses).
- `quick`: the section is in a quick round; a step with `"quick": false` isn't (below).
- `kind`:
  - `prompts` (the default): each prompt shows for its `seconds` with a countdown. A prompt with `cues` (and `cue_s`) is a sweep step with those cues in that order, `seconds` = cues x cue_s (hand size; the mouse and switch step).
  - `sweep`: steps built from `sweeps` (below).
  - `targets`: each target shows until the live index tip is within 3 cm of it for `hold_s`, or `timeout_s` passes.
  - `bar`: the target marker sweeps near to far and back, `reps` times per height, at `period_s` per sweep. The current marker follows the live palm distance.
- Each section is one take. Prompts within it are marked in `prompts.jsonl`.
- `cue_names`: the short labels under the strip's pictures (otherwise the pose id).

### Sweeps

The second in-headset session (sessions/20261002-202734) took about 16 min and was "super long and kind of annoying": the 36 held poses alone took 7.9 min, 3.2 of it reading, and the person skipped the wrist turns and gave up on the bare push after 3 steps. The labels come from the auto-labeller (teacher model and triangulation, frame-hands `train/label`), not from the prompts, so what the recordings need is variety (shapes, distances, angles), not clean holds. So the poses are swept:

- A sweep step shows a strip of 2-5 pose pictures and asks for slow, continuous movement (near and far, all around) while the hands change shape with the lit picture. The light moves on every `cue_s` (4 s), cycling, over `step_s` (20 s): 5 cues for a group of 4 or 5. Each cue is a `prompt` event with that `pose`, `"cue": true` and `"step"` (the step's id); `distance` and `position` are "" (varied). So the timeline tags each pose roughly, and the countdown, `wait` and `redo` work per step as before. The time-left bar covers the whole step.
- `groups`: lists of poses. A step takes `"group": "next"` (the groups in turn) or `"any"` (one drawn at random, a different one for each "any" while there are groups left), or names its own `cues`. `"shuffle": false` (gestures) keeps the script's order; `"cycle": false` runs the cues once; `"fixed": true` keeps one step's order.
- The pose sweeps: 3 two-hand sweeps, one per group ([open, fist, point, pinch], [ok, thumbs-up, spread, claw], [count-1 ... count-5]); then a left-hand and a right-hand sweep (the other hand in the lap) on groups drawn from those three, which also turn the wrist as they go, so the wrist angles come for free (the old wrist-turns section is gone).
- **The shuffle, per session.** The groups' order, the "any" draws and the cues' order in each step are shuffled with a seed from the session's id (`session_seed`: the first 8 hex digits of its SHA-256), separately per section (`random.Random("<seed>/<section>")`). The sections' order isn't shuffled (the controller sections need their before and after). session.json records `"shuffle": {"seed", "sweeps": {section: [{"id", "hands", "cues"}]}}`, and `build_plan(script, checklist, seed=...)` gives the same again. `--seed N` overrides it; `--plan` without one shows the script's order.
- The strip has one picture per pose, a single hand: both hands make the same shape, and five pairs wouldn't fit. A left hand's pictures are flipped.
- Gestures are sweeps too, in order and once: tap then drag; grab, cross, overlap; near the face then a screen's distance. Hand size is one step of three held shapes (flat, backs, spread; 5 s each), still the no-hands check's first step.

### Quick round

For more lighting rounds: "Quick round (about 3 min)" on the checklist page, `session.py --quick`. It has the sections marked `quick` (hand size, the pose sweeps without the one-hand ones, touch the dot, no hands), about 2 min recorded in 6 steps. session.json gets `"quick": true`. The checklist page suggests it (and picks it) once a full session has gone to the end (`Backend.hasFullSession`: a session.json with status `done`, not quick, not a dry run).

### Step mode (the default) and auto mode

The first in-headset session (2026-10-02) went too fast: each prompt advanced after 4-8 s, before there was time to read it and find the hand shape. So by default every step waits:

1. **Ready.** The panel shows the step: section title, "step N of M", the instruction, the pose picture, the where-to diagram, and the Next hint ("Ready? Press the button on the right side of the headset", or with a mouse also Space and Next: see Controls). The hand chips show which hands are seen, with no warnings yet. It waits as long as it takes, and nothing records.
2. **Countdown.** Next starts a new recording part and a "ready" event, and the panel counts 3, 2, 1 (big), recorded so the hold is captured from its first frame. In the push sections the bar sits at near meanwhile.
3. **Hold.** The `prompt` event, the section's word ("Hold" or "Go") and the time-left bar for the prompt's seconds (or the bar's sweeps, or the targets). Then a `wait` event and the part stops.

Steps that wait: each prompt, each bar height, and the first touch-the-dot target (the others follow straight on, as each waits for the touch anyway). Before a section, one screen shows the section's intro (with its `before` text, such as putting on the controllers) and waits for Next too; the welcome screen as well. The take starts with the section's first countdown, so a section skipped at its intro leaves no take. A pause in a hold works as before (the part stops; resume starts the next one).

Auto mode ("Advance by itself" on the checklist page, `session.py --auto`) is the old timed flow: the welcome, the between and before screens, the intro (recorded) and each prompt for its seconds, one recording per take. R still works there: it restarts the step running.

Steps are 20 s sweeps, 16-18 s gestures, 10-12 s desk and object steps, the touch targets (6) and the push heights (2, 3 reps of 6 s). The core session (no objects, no controllers) records about 5 min in 15 steps; with 5 s of reading a step that's about 6 min. Everything ticked (four objects, controllers): about 7.3 min recorded in 24 steps. A quick round: about 2 min in 6 steps. The window and `session.py --plan` give these (`plan_summary`); in step mode they leave the reading time out and say so.

The sections, in this order (see the plan):
1. hand size (one step: flat, backs, spread)
2. pose sweeps (above: 3 with both hands, one with each hand)
3. gestures (3 sweeps: pinch taps and drags; grabs, crossing and overlapping; near the face, then pointing at screen distance)
4. desk work (typing or pretend typing; the mouse, with switching to the keyboard when both are there)
5. objects (one prompt per ticked object, own objects included)
6. touch the dot (6 dots spread near and far, left and right, low)
7. controller depth, straps (push out and back at chest and eye level, 3 reps each; then the wrists and fingers with the controllers on). "Out and back" is straight away from the headset and back toward it: the first in-headset session took the left-right bar for sideways. The texts say so, the bar's ends read "At your chest" and "Arm out" (`near_label`, `far_label`, sent as `bar ... At your chest|Arm out`), and the picture is a side view.
8. bridge (one controller on, the bare fingertip on its thumbstick while that hand moves from close to arm's length and back, then swap; then a controller on the desk, touched from the front and above, then from the sides: 4 steps, the controller changes during the ready screens)
9. bare repeat of section 7's pushes, controllers off
10. no hands (10 s)

Before section 7: "Put on both controllers and tighten the straps". Before section 9: "Take the controllers off and put them out of view".

### Feedback while recording

- **Hands seen:** from the hands file. A hand counts as seen if its flags match the side and the file is fresh (publish within 0.3 s). Prompts with `hands` set show the `hands` chips. If an asked-for hand is lost for more than 1.5 s, the note says "I can't see your left hand: bring it into view".
- **Controller tracking:** in sections 8 and 9, `devices` is polled once a second. A result other than 200 for more than 1 s says "The left controller lost tracking: turn your palm slightly toward you". Each such stretch goes into `prompts.jsonl` as `feedback` with `"controller": {...}`.
- **Lighting, measured:** the checklist page measures the light when it opens, starting ft-camd (`frametop-handrec-camd.service`) if nothing runs it; the window stops it again on quit if it started it. The mean of every mono camera's `mean` and `dark_mean` from the ring (`hands/tools/ring.py` layout; struct only, no numpy) is compared with the person's earlier sessions. If it matches an earlier round's within 15%, the window says so before starting.
- **Lighting label:** "Measured by the cameras" is the default. `ambient_ir`, the mono cameras' mean `dark_mean` (the room's infrared), labels the round `daylight` from 6.0 and `indoor` below (`session.classify_lighting`). Lamps and LEDs give off hardly any infrared, so a dim room and a lit one read about the same (1.8 by one lamp, 2.2 in a lamp-lit room) and the cameras can't tell them apart; the person can pick dim, room or daylight instead (`source: "picked"`). The 6.0 threshold is a guess until a daylight round is measured.

### Camera check

On 2026-10-02 a whole session showed "I can't see your hands": after the headset slept, SteamVR had failed to load the colour module's FPGA image, which left the upper cameras and the IR light off (`hands/README.md`, "Camera check"). Two checks keep that from wasting a session:

- **Before the session.** The window runs `hands/camcheck.py` when the checklist page opens ("Tracking cameras:", with the evidence under Details and Check again), and again when Start is pressed; `session.py` runs it first thing, before it makes the session's folder or starts anything (`Session._preflight`). If it finds the cameras degraded, the session doesn't start. The window says: "The headset's upper cameras and IR light are off. SteamVR couldn't start the colour camera module (it happens sometimes after the headset sleeps). Restart SteamVR, or restart the headset if that doesn't fix it." (another `degraded:` reason gets a sentence naming it), and Start stays off. `unknown` (SteamVR not running, the cameras asleep) doesn't stop it: the session's own start fails clearly then. From the command line, `session.py` prints the check and exits with status 3.
- **Restart SteamVR.** The message has a Restart SteamVR button. After a confirmation it runs `systemctl --user restart --no-block steamvr.service` on the host (through `distrobox-host-exec`), then checks the cameras every 3 s, for up to 2 minutes, until a new XRService has opened its cameras. The confirmation says what really happens: every VR app closes, and so does the Frametop desktop with all its windows, this one included, and it doesn't come back by itself (`hands/README.md`, "What a SteamVR restart does to Frametop"). So the check after the restart mostly happens when the recorder is opened again; it re-runs when the checklist page opens. `--no-block` lets the restart finish after the window is gone.
- **No hands in the first step.** The first step of the hand-size section has both hands up, about 40 cm away. During its hold, the session counts the hands file's reads (about 20 a second). If the tracker published in at least half of at least 10 reads and never saw a hand, the step stops: a `wait` and the recording stops as usual, then a `nohands` event and a `redo` over the try. The session runs the camera check and shows "I can't see your hands" with its result (the camera text above when it's the VCINT failure, else "The camera check found nothing wrong"). The state is `nohands`, waiting: Next (the headset button, Space, Try again) or R starts the same step's countdown at once; Stop (Esc) ends the session with `stop_reason` set; S skips the section. One missed step costs a retry, not the session, and every hold of that step is logged ("hands check ...: a hand in N of M reads"). Without a tracker publishing the session can't tell, logs that, and goes on. Only that one step is checked: later steps have their notes ("I can't see your left hand") as before. Auto mode does the same; its recording pauses meanwhile.

### Controls

- The window has Start, a big Next (while a step waits; its hint is the panel's), Pause/Resume, Redo step, Skip section and Stop. While it has focus: Space is Next, P pauses or resumes, R redoes, S skips the section, Esc stops. The panel's bottom line and the window list them. The window also shows the step's picture, diagram, countdown and "Hold".
- **The headset's button.** The Frame has a click button on its right side, for use without controllers: `KEY_SELECT` (353) on the evdev device `gpio-keys`. While a step waits (the welcome, a section's intro, a step's ready screen) a press is Next; during a countdown or hold (and auto mode's timed screens) it pauses; while paused it resumes. The session finds the device in `/proc/bus/input/devices` by name and its KEY bitmap (`event3` on the maintainer's Frame) and reads `input_event` structs with plain `struct` (no python-evdev), from a thread. Only key-downs count (value 1: releases and autorepeat, value 2, don't), and presses closer than 0.3 s count once.
  - It's read, never grabbed (`EVIOCGRAB`). Frametop's input relay (`input/input-relay.py`), ft-powerd, SteamVR and gamescope read the same device, and the relay remaps its volume keys (the experimental branch's `docs/hazards.md`). A grab would take the volume keys from the relay.
  - `steamos` is in the `input` group, so it needs no sudo. The dev container sees the host's `/dev/input` and `/proc/bus/input/devices`, and the group carries over (checked 2026-10-02), so it works from the window there and from `session.py` on the host. If the device is missing or can't be opened, the session logs it, keeps trying every 5 s, and the hints don't mention the button.
  - **Not known yet (needs the headset):** what SteamVR and gamescope do with the same press. They read it too, so it may also click whatever is under the head or gaze pointer in VR, or open something. Check on the first session; if it does, the fix is on their side or a different button, not a grab.
- **The Next hint follows what's there.** No mouse connected: "Ready? Press the button on the right side of the headset" (the window's Next can't be clicked, and Space needs the window focused). A mouse: "Ready? Press Space, click Next, or press the headset button". No button: "Ready? Press Space or click Next". The panel's bottom line leads with "Headset button: next, pause". A mouse is a device in `/proc/bus/input/devices` with EV_REL, REL_X and REL_Y that isn't made in software: uinput devices (Frametop's virtual mouse, frame-voice's keyboard) sit under `/devices/virtual/input` or on the virtual bus (6), and are left out; Bluetooth mice come through uhid, under `/devices/virtual/misc/uhid`, and count. It's looked at again at each step, so a mouse plugged in mid-session counts from the next step.
- **R (redo).** During a step's countdown or hold: that step starts again from its ready screen. At a step's ready screen: the step before it (in this section) goes again. Either way a `redo` event marks the range of the try being redone, so its labels are skipped; the sets stay, to delete in review if wanted.
- A pause stops the take's recording and starts it again on resume as the next part of the same take (`sets-2.bin`, and so on). takes.py reads all parts in order. A pause while a step waits only shows "Paused"; a Next pressed while paused doesn't count.

## Review and export (`takes.py`, the window)

- **Review.** Each session lists its takes: title, duration, status, a thumbnail (the first set's `slam_left`). A viewer shows one frame set (all cameras side by side, 8-bit grey) at a time with a slider. It can mark a range and delete it. Deleted ranges go into `take.json`, and export leaves them out; the files keep everything until export. A whole take or session can be deleted (files removed, after a confirmation).
- **Export.** It writes `exports/<session>/`:
  - `manifest.json`: profile fields except `optional.notes` unless kept, session.json (without its `uploads` records), takes, schema, tool version, the consent version.
  - `calibration.json` and `device.json`, when the session has them.
  - Per take: `prompts.jsonl`, `poses.jsonl`, `take.json`, and `sets.bin.zst` (sets in deleted ranges removed, then zstd -10 with 2 threads).
  - The side cameras are named right in every exported set when the session's `sides.swapped` is known: parts that need it get slam_left and slam_right (and their `_dk`) exchanged in the set headers as they're compressed. The exported take.json says `"parts": {"sets.bin": {"names_swapped": <swapped>}}`, and the manifest's take entry `"sides": {"names_swapped", "renamed_sets"}`. Unknown, the names stay as recorded.
  - `SHA256SUMS`.

  Compression runs at nice 19. Before starting, the window warns if the headset is worn, judged the way `frame-job` does: `vrcompositor` runs and a `/sys/class/backlight/*/brightness` reads over 0 (SteamVR turns the panel off 5 s after the headset comes off). CPU work while in VR causes stutter. The proximity sensor is no use here: it read 9-43 with the headset sitting unworn.
- **Upload.** The Upload page uploads an export from the window. It also keeps the manual way as a fallback: `UPLOAD.md` with the export's path and size filled in, plus the copyable command (`validate.py`, then `hf upload ... --create-pr`).
  - **`hands/rec/validate.py`** (standard library) checks an export. The window runs it before an upload, and the maintainer runs it on each submission (`validate.py DIR [--json]`, exit status 1 on errors). It checks:
    - `SHA256SUMS`: every file listed and matching.
    - An allow-list: `manifest.json`, `calibration.json`, `device.json` and `SHA256SUMS` at the top, and `prompts.jsonl`, `poses.jsonl`, `take.json` and `sets.bin.zst` in `takes/<NN>-<section>/`. Anything else is an error, and so is a symlink.
    - The manifest's schema, keys and types, and that it matches the files.
    - The consent version is present and the contributor confirmed being an adult. The contributor id is a uuid4.
    - `calibration.json` has nothing that `session.py`'s `strip_calibration` would still remove.
    - `device.json` holds only `cv.cad_from_cal` and `head`, each `plus_x`, `plus_z` and `position` as 3 numbers (plus `cad_from_cal`'s `method`). Without it: a warning.
    - Each `sets.bin.zst` decompresses to its end, so a truncated one fails, and every set's FHSET01 header is sane: camera names, sizes, record length. Set counts and raw bytes match the manifest. Pixels aren't decoded.
    - Every jsonl line parses.
    - `session.sides` is `{"swapped": true|false|null}`, and every take's `sides.names_swapped` equals it (else an error: export again). Without `sides`, or `null`: a warning (the maintainer's check tells).
    - The total size: a warning over 15 GB, an error over 40 GB.

    Warnings cover notes kept in the export, a home folder path in the manifest, and missing `poses.jsonl` files.

    It runs on Linux and on Windows (the maintainer's PC) with Python 3.12 or later. It decompresses with Python 3.14's `compression.zstd`, else the `zstandard` package, else the `zstd` program, and handles several zstd frames in a row.
  - **`hands/rec/hub.py`** does the upload. It runs as a child process of the window (Cancel ends it), or from the command line (`hub.py [--base DIR] whoami | upload SESSION [--dry-run] [--again] [--json]`). It uses `huggingface_hub` (`python3-huggingface-hub` in the dev container) with the token `hf auth login` saved, and never handles a token itself. An upload goes through these steps:
    1. Validate, and stop on errors.
    2. Stop if the same export was uploaded before (same `SHA256SUMS`), unless asked again.
    3. Stop while the texts are drafts, unless `FT_HANDREC_ALLOW_UPLOAD=1`.
    4. `whoami`: a read-only token is refused.
    5. `auth_check` on the dataset: a gated dataset whose terms aren't accepted gives "accept the dataset's terms first", with the link.
    6. `upload_folder(repo_id=HF_DATASET, repo_type="dataset", folder_path=EXPORT, path_in_repo="contributions/<contributor>/<session>", create_pr=True, commit_message=..., commit_description=...)`. The description summarizes the manifest: takes, minutes, sets, lighting, objects, controllers, the consent and tool versions, the size, and validate's warnings.
    7. Add `{"repo", "pr_url", "uploaded", "export_sha"}` to `uploads` in `session.json`. `export_sha` is the SHA256 of `SHA256SUMS`. The file keeps its modification time, so the export doesn't count as out of date.

    Errors get a plain explanation: not logged in, a token Hugging Face rejects (401), a token that can't open a pull request (403), terms not accepted, dataset not found, network errors. `--dry-run` does everything except the network calls and the record, and lists what it would upload.
  - **The page** shows the login (`whoami`, with "Check again"). If nobody is logged in, it explains how to run `distrobox enter dev -- hf auth login` in Konsole with a write token: the token goes only into that terminal. The page then has Upload and Cancel, the phase with a progress bar (a share while the export is checked, a sweep while it's sent, as `huggingface_hub` reports no progress), and the pull request's link when it's done. If this export was uploaded before, the page says so, and uploading it again asks first. A stale export can't be uploaded.
  - **While the texts are drafts**, Upload stays off unless `FT_HANDREC_ALLOW_UPLOAD=1`, so the maintainer can rehearse against a private test repo. `FT_HANDREC_DATASET` overrides `HF_DATASET`. `ft-handrec --hub-dry-run` makes Upload a dry run: no network, so it isn't held back by the drafts.
  - **Rehearsal: `hands/rec/rehearse.sh [--repo ID]`** runs it all without the headset, in the dev container, in one `frame-job --local` scope when frame-job is installed. `ft-ringplay` plays 30 s of a recording into a ring in `/run/user/UID`. A tracking ft-hands that's already running is used, or one is started on that ring. `ft-handpanel --no-vr` stands in for the panel. `session.py --no-start --next-after 0.3` records a three-section test script (a prompt, a two-cue sweep, no hands) in step mode, three parts of 6 s (countdown and hold), about 360 MB once exported. Then `takes.py` exports, `validate.py` checks, and `hub.py` uploads: a dry run by default, or for real to `--repo ID` with `FT_HANDREC_ALLOW_UPLOAD=1`. It prints a summary, deletes its temporary folders (camera images of a room) and stops everything it started, Ctrl+C included. The `--no-vr` panel logs no poses, so `poses.jsonl` is missing there (a warning).

## Licensing and consent (texts in `CONSENT.md`)

- The dataset is CC BY-NC 4.0.
- Contributors also grant the maintainer (DeeJanuz) a broad, non-exclusive license to their contribution.
- Contributors confirm they're 18 or older.
- The text explains what's recorded and that nothing uploads automatically, how review works, how to withdraw (by contributor id), and that a withdrawal is purged from the repo's history.
- The texts need a legal review before the dataset launches. Until then they carry a "draft" banner, and the Upload page says contributions aren't open yet.
- The dataset repo: `HF_DATASET` in `hub.py`, `DeeJanuz/frametop-hands` (private until launch).