Frametop--frametop/hands

Hands (experimental, deferred)

Hand tracking from the headset's own cameras. It's deferred: it costs a lot of the headset's CPU and needs more work, so install.sh doesn't offer it and the README doesn't list it. It still builds and runs, installed by hand (below), for working on it.

It serves two things in Frametop:

• Hand cutouts: where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, screens/handcut.cpp), so your hands show over the screens the way they do on a Vision Pro. The hole is round, one per hand, until the tracking is good enough for a hand-shaped one (also there, built from the skeleton; see "Hand cutouts and near input" below).
• Rendered hands: ft-handview draws your hands as a 3D hand model posed by the tracking, Quest-style, over everything (screens/handview.cpp; see "Rendered hands" below): with Room View off (a VR game, SteamVR Home, the dashboard, or the panels over the void), where ft-screens doesn't cut, and by default with Room View on too, floating over the hole.
• Near input on Frametop's panels: poke a screen with your index finger to click, pinch its bar, edges or any corner to move or resize it, hold it like a card or in a fist to carry and turn it, hold it with both hands to stretch it, flick it away or throw it (ft-screens, screens/handinput.cpp; see "Hand cutouts and near input").
• Pinches and grips: with POINTER_HANDS=1, the pointer helper takes them as clicks and drags. Look at something and pinch to click it, with the eye tracker doing the looking (gaze/), or close your hand to press and drag what the pointer is on. See "Pinches and grips in the pointer" below.

Two programs, each a user service that stops when SteamVR does:

• ft-camd (camd/, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host.
• ft-hands (track/, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container.

hands/run.sh install builds them, gives ft-camd its capabilities, installs both services, and links hands/ft-handsctl into ~/.local/bin. install.sh doesn't install it. When they run is HANDS_MODE in ~/.config/frametop.conf, set on the Hands page of Frametop Input Settings or with ft-handsctl mode; hands/handsmode.py does the starting and stopping, for these two and the hand view (frametop-handview.service, when it's installed):

• off (the default): nothing starts them. ft-handsctl on starts hand tracking and ft-handsctl off stops it, as before; it stops with SteamVR.
• frametop: they run while the Frametop desktop does. The session (session/frametop-session.sh, screens backend) starts them once ft-screens is up and stops them as it ends.
• always: they're wanted by steamvr.service (systemctl --user add-wants), so they run whenever SteamVR does, in games and the SteamVR dashboard too, with or without the desktop. Pausing Frametop for a VR game leaves them running and unparked.

Choosing a mode applies it at once: the services start or stop, with no desktop restart. Without ft-camd's capabilities nothing starts, and the Hands page says to run hands/run.sh caps (it asks for the password in the terminal; the page never does). hands/run.sh install and caps apply the mode too, but stop nothing, so what ft-handsctl on or ft-cutouts on started keeps running.

For the cutouts alone, hands/ft-cutouts on starts the same two programs with ft-hands' --no-gestures: your hands show through the screens, and no pinch or grip is detected, so nothing clicks or drags. It runs this checkout's build as transient user units, so it needs hands/build.sh and ft-camd's capabilities (hands/run.sh caps) but not hands/run.sh install. It and ft-handsctl on stop each other's services, and it stops with SteamVR too.

ft-handsctl on | off            # on the Frame: start or stop hand tracking (SteamVR must be running)
ft-handsctl mode [frametop|always|off]  # HANDS_MODE: show it, or set and apply it
ft-handsctl status              # the services, and ft-hands' last status lines
ft-handsctl log [lines]
ft-handsctl cutouts on|off|state  # ft-screens' hand cutouts, without stopping tracking
ft-handsctl gestures            # pinches, grips and middle pinches, live (tools/watch_gestures.py --distance)
ft-handsctl flag [note...]      # flag a tracking failure: the last seconds go to a folder (see "Flags")

hands/ft-cutouts on | off | status   # the cutouts only: no pinches or grips

hands/run.sh install            # build, give ft-camd its capabilities (sudo, once per build), install, apply HANDS_MODE
hands/run.sh start|stop         # start or stop the services
hands/run.sh restart            # after changing a setting
hands/run.sh status
hands/run.sh log [lines]
hands/run.sh caps               # after rebuilding ft-camd (a rebuild clears its capabilities)
hands/run.sh uncaps             # take them back, unless the Hand Recorder or the services use them
hands/run.sh uninstall          # the services, then uncaps

Settings in ~/.config/frametop.conf (FT_<name> in the environment overrides them), read when ft-camd and ft-hands start:

• HANDS_SWAP_SIDES=auto (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). 1 forces them exchanged and 0 forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (sides.json), so recordings are labelled right. The example config said 0 until 2026-10-05; scripts/conf-migrate.sh (run by install.sh and hands/rec/install.sh) turns that untouched line into auto.
• HANDS_CPUS=5,6,7: the CPUs the model threads run on (below).
• HANDS_CAMERAS (auto), HANDS_BRIGHT (all), HANDS_BRIGHT_ON (40), HANDS_BRIGHT_OFF (25): which cameras ft-hands tracks with, as --cams, --bright, --bright-on and --bright-off (see ft-hands). HANDS_CAMERAS=mono also keeps ft-camd off the colour cameras.
• HANDS_MODELS (unset: hands/models/ncnn, the stock MediaPipe models): a folder holding palm.ncnn.* and hand.ncnn.*, as --models. Use it to run fine-tuned models, such as the ones trained on the hand dataset, without passing options to every launcher. Those folders usually lack the *-int8 files, so --int8 won't load them.
• HANDS_COLOR_LEFT (color_video0), HANDS_COLOR_CROP (subtract): how the colour module's calibration maps onto its images, as --color-left and --color-crop.
• HANDS_SKELETON=1: fit a skeleton with your bone lengths and publish it (see "Skeleton" below); 0 publishes the filtered landmarks as before. As --skeleton.
• HANDS_PINCH_DIP=0: 1 also starts a pinch on a quick dip of the thumb-index distance that never reaches 2 cm (see the pinch under "Gestures" below). As --pinch-dip.
• HANDS_FLAG_BUFFER=1: keep the last seconds of hand crops in RAM for ft-handsctl flag (see "Flags"); 0 keeps nothing. As --flag-buffer. HANDS_FLAG_SECONDS (12), HANDS_FLAG_CROP (224 px), HANDS_FLAG_MB (80) size it. HANDS_FLAG_DIR (~/.local/share/frametop/hands/flags) is where flags go; HANDS_FLAG_FLASH=1 pulses the drawn hands when a flag is taken, HANDS_FLAG_TONE=1 adds a short tone, HANDS_FLAG_VIEW=1 saves a frame of your view.

The pointer helper's POINTER_HANDS and POINTER_PINCH_*/POINTER_GRIP_* settings are in "Pinches and grips in the pointer" below.

Files, all in /run/user/UID/frametop-hands/ (private to the user; not /run/user/UID/frametop/, which the desktop session deletes whenever it starts):

File	Written by	Layout	Read by
cam-ring	ft-camd	camd/fhring.h	ft-hands, tools/ring.py
hands	ft-hands	include/fh_hands.h (version 2: version 1, then a skeleton per hand)	ft-screens (screens/handcut.cpp, version 1's part), the hand recorder (rec/session.py)
gestures	ft-hands	include/fh_gestures.h	the pointer helper (pointer/helper/ft-pointer.cpp), tools/watch_gestures.py
flag	ft-handsctl flag (handsflag.py)	key=value lines: dir, note, flag_unix, said_unix	ft-hands, which removes it when it takes it

The source keeps the fh_ names and magic strings of frame-hands, the project it started as, so recordings made with it still work.

ft-camd

XRService owns the headset cameras. ft-camd borrows its DMA-BUFs read-only with pidfd_getfd, the same way FrameEyeCameraFeed does. It never touches XRService's V4L2 descriptors. discovery in camd/xrcams.c is adapted from FrameEyeCameraFeed (MIT, see camd/LICENSE.FrameEyeCameraFeed).

Polling buffers for changes can catch a frame while the camera is still writing it. Instead, ft-camd listens to the v4l2:v4l2_dqbuf tracepoint, which fires when XRService takes a buffer. It gives the buffer index, the sequence number and the capture timestamp. ft-camd learns which DMA-BUF holds each V4L2 index by watching which buffer changes at each dequeue:

• Right after XRService allocates its buffers, the mapping is allocation order.
• After XRService restarts streaming, the order is shuffled, and the mapping is learned index by index.
• The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart.
• It also re-maps an index on the fly when its buffer holds no new frame.

Privileges. Setting up needs three things. pidfd_getfd on XRService needs CAP_SYS_PTRACE, because the Frame has ptrace_scope=1. The system-wide tracepoint needs CAP_PERFMON, because perf_event_paranoid is 2. Its format files are root-only, which needs CAP_DAC_READ_SEARCH. hands/run.sh install gives the binary those capabilities with sudo setcap. File capabilities need a filesystem mounted without nosuid. The Frame's /home (ext4) has no nosuid. ft-camd drops them all once it has set up, before it reads a frame, and then runs as you. XRService runs as you too. It also runs under sudo, for trying it by hand, and then drops to the user who ran sudo. It reads nothing from the ring's readers.

The ring is mode 0600, in a folder only you can write. Frame handling:

• Only complete, bright frames are published. The cameras alternate a normal exposure with a near-black one, so each camera gets 30 of its 60 fps.
• A copy torn by the camera overwriting the buffer is dropped.
• Each copy takes about 0.1 ms, and a cache sync about 0.15 ms.

Options:

• --dark R: a frame dimmer than R times the camera's recent brightest counts as near-black. Default 0.4.
• --with-dark: also publish the near-black frames, as extra ring cameras flagged FH_CAM_DARK. They show only light sources, so they're no use for hands.
• --with-color: also publish the two Arcturus colour cameras, flagged FH_CAM_COLOR. The service leaves it off and runs the mono cameras only (see "Known issues"). To try the colour cameras, add it to ExecStart in hands/frametop-camd.service and run hands/run.sh install again; ft-hands then picks the cameras by the light. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (--color-scale 2: 986x1232). They run at --color-idle (2 fps), enough for ft-hands to tell how bright it is, until a reader asks for more in /run/user/UID/frametop-hands/color-fps (ft-hands writes 30 while it tracks or records with them), up to --color-fps (30; the cameras run at 60). HANDS_CAMERAS=mono leaves them out. Frames that carry the module's warped half-size copy are dropped. Their capture_ns is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by dqbuf_ns. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core.
• Each mono camera's latest near-black frame's mean goes in the ring (dark_mean): a short fixed exposure, so it follows the room's IR light, sunlight above all.
• The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 colour cameras.
• Colour isn't reliable yet. In the lit-room test of 2026-09-30, the colour cameras kept losing their buffer mapping while the headset was worn: 30 frames in a row looked unchanged, the camera relearned, and after 5 relearns ft-camd exited. Each relearn probed all 32 colour buffers, a whole-buffer cache sync each, which also made the mono cameras miss frames. Runs with the headset idle had none of this. So the passthrough compositor may be writing into the colour buffers while Room View shows. Since then a colour camera never takes the mono ones down: it probes at most 4 buffers a frame, and one that goes stale twice in a row is paused (10 s, doubling up to 160 s) and learned again, without ft-camd exiting. Whether a frame is new is judged on the luma rows only: the chroma after them hardly changes in a lit room.
• FT_CAMD_DEBUG=1 in the environment: at each stale colour frame, ft-camd logs to stderr the camera, the frame's time, V4L2 index and sequence number, and, for every candidate buffer, how many of its sampled words changed, in all and in the last eighth of the samples.
• --sensor S: only the mono cameras whose sensor name contains S.
• --status S: a status line every S seconds (0: never).

It exits when XRService exits, when a camera's buffers keep going stale, which means XRService has reallocated them, or when a camera stops while the others keep running (3 s), which means XRService has moved it to another device (the colour module was unplugged or plugged in). The service starts it again, and it attaches to the new buffers.

Which camera is which: video9 is slam_right (sensor og01a1b 4-0036), video13 is slam_left (4-0060), video6 is upper_left and video7 is upper_right; without the colour module the side pair is on video0 (slam_right) and video3. XRService says so in its log: Found camera 'slam_left': ... v4l_subdev=/dev/v4l-subdev30 names each sensor, and each TrackingCameraInit line gives the device and subdev it opened. ft-hands and camcheck.py name the devices that way. Until 2026-10-05 they named them by the TrackingCameraInit index instead, which is only the order XRService opens them in (index 0 was slam_right on every start logged), and ft-camd bound each buffer queue to a device by the order of XRService's file descriptors, which changes when XRService restarts its cameras. The two mistakes made the side names come out swapped on most starts and right on some. ft-camd now asks each device which buffers it holds (VIDIOC_QUERYBUF names the descriptor XRService queued at each index) and only falls back to the order if that fails. With the names swapped, every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands still checks by itself, in case (track/sides.h, HANDS_SWAP_SIDES=auto):

• Whenever a hand's landmarks are found in two cameras at once (one of them a side camera), it intersects the rays through the 21 landmarks twice: once with the calibrations as named, once with the two side cameras exchanged. The same hand seen the right way meets within a few mm, in front of both cameras and as far away as its size says. The wrong way misses by centimetres or meets behind a camera.
• With the names wrong, the tracker never gets such pairs on its own: it hands the hand over to where the wrong calibration puts it and finds nothing there. So 5 times a second while undecided, the check places a tracked hand in 3D under the other naming and runs the landmark model where that puts it in the other side camera.
• It decides after 10 votes one way and none the other, or 20 with at most a fifth the other way, over at least 1 s. That takes about 1-2 s of hands in view. If the names are backwards, it exchanges them; the tracked views move with their images. Then it checks once more, more strictly.
• The log says what it found (side cameras: SWAPPED, now exchanged after 3.2 s (votes ...)). So does /run/user/UID/frametop-hands/sides.json, which the hand recorder reads. Recordings get a DIR/sides.json (hands/rec/sides.py has the rules).
• --record-only can't tell (it tracks nothing): it records ft-camd's names unless --sides 0|1 says otherwise.

tools/check_sides.py --ring is the independent check from the scene (ORB matches meeting under each naming): exit 0 as named, 3 swapped, 2 can't tell. --pair upper checks the upper pair the same way: in every recording so far (3 XRService starts, both side namings) the upper pair was named right. The colour cameras are video3 (arcimx616 0-0010) and video0 (0-001a); which of them is passthrough_left in the module's calibration is for tools/check_color.py to settle, on a recording with texture in view.

ft-hands

hands/build/ft-hands                  # status every 5 s; Ctrl+C to stop
hands/build/ft-hands --int8           # the 8-bit models (models/ncnn/*-int8.ncnn.*)

Run it in the dev container (distrobox enter dev -- ...). It reads the factory calibration from /persist (/run/host/persist in the container).

Options:

• --threads N: model threads, pinned to the --cpus list. Default 3.
• --cpus LIST: CPUs for the model threads and the main loop. Default 5,6,7 (HANDS_CPUS). SteamOS starts user processes on CPUs 0-4, and XRService's head tracking runs on 2-3. With the headset on, a step took 8.4 ms on 5-7 against 13.2 ms on 2-4, and SteamVR's frame timing didn't change (2026-09-29, three rounds of the same replayed frames).
• --contrast MODE or PALM/HAND: how crops are equalized before the models see them: clahe[:CLIP], none, or stretch (1st-99th percentile). Default clahe:2/none. In the dim recording, CLAHE let the palm search find about 10% more hands, but it made the landmarks jitter more (published median 6.9 mm, against 6.0 mm with plain landmark crops).
• --sides auto|0|1: which side camera is which (HANDS_SWAP_SIDES, see "Which camera is which"); --swap-sides is --sides 1.
• --seconds N: stop after N seconds.
• --status S: how often to print status, in seconds.
• --models DIR: where the models are.
• --nice N: niceness. Default 5, so the VR stack wins contested CPUs.
• --no-publish: don't write the hands and gestures files.
• Parked: while /run/user/UID/frametop-hands/park exists, ft-hands processes no frames (the models stay loaded), publishes no hands, and ends any gesture down, as lost; it writes 0 to hands-want there every second, and ft-camd stops copying the mono cameras' frames while that file is fresh (3 s) and says 0. The input relay parks it while Frametop is paused for a VR game with hand interaction on (POINTER_HANDS=1), instead of stopping the services, and unparks it while the dashboard is open, so the hands work there; it tracks again from a palm search. Not while recording.
• --no-gestures: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. ft-cutouts runs it this way.
• --record DIR, --record-for S, --record-hz N: save every frame set for S seconds (default 120) to DIR/sets.bin, or at most N sets a second with --record-hz (the hand recorder uses 10). Every set is about 80 MB/s. Sending the tracker SIGUSR1 (pkill -USR1 -x ft-hands) starts a recording in ~/.local/share/frametop/hands/rec-<time> without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
• --record-only: record without tracking or publishing, so it can run beside the live tracker. Give it --record DIR, since SIGUSR1 would reach both trackers. With ft-camd --with-dark, recordings also hold each camera's newest dark frame as <name>_dk, which doubles the rate. With --with-color, each colour camera's newest frame is saved with every set, as color_video<N>, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
• --reacquire S (HANDS_REACQUIRE, default 1.5): how long after losing a hand the palm search aims at where it should be (see "Scheduling"). 0 turns it off.
• --pair-sides 0|1 (HANDS_PAIR_SIDES, default 1): two hands seen together are one left and one right (see "Scheduling").
• --hold S (default 0.6): how long a lost hand tracked for 30+ frames is held before it's forgotten (0.3 s with --reacquire 0, as before).
• --keep-presence P: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.
• --ring PATH: read frames from another ring, such as ft-ringplay's.
• --cams auto|mono|color|all (HANDS_CAMERAS, default auto): which cameras to track with. The mono IR cameras light the hands themselves and track well in dim rooms, but in bright light they expose for the room and the hands come out dark. The colour pair is the other way round. auto goes by the colour frames' mean brightness: at --bright-on (HANDS_BRIGHT_ON, 40) or over for 2 s it tracks with --bright (HANDS_BRIGHT: all, every camera, the default, or color), and under --bright-off (HANDS_BRIGHT_OFF, 25) for 2 s with the mono cameras again. A dim evening room read 9. The switch is logged (cameras: mono -> all (...)), and the status line gives the colour level, the mono cameras' ambient IR, and how many steps had colour frames. Colour frames arrive on their own schedule, so a step holds the mono set, the colour pair, or both, and views wait in their camera for its next frame. With no colour cameras in the ring (ft-camd without --with-color, as the service runs it), ft-hands tracks with the mono cameras whatever this says.
• --color-left NODE (HANDS_COLOR_LEFT, color_video0) and --color-crop subtract|none (HANDS_COLOR_CROP, subtract): how the colour module's calibration maps onto the images. Not settled yet: tools/check_color.py on a recording with a lit, textured view tells.
• --skeleton 0|1 (HANDS_SKELETON, 1): the skeleton fit (below). With 0 no skeleton is published, and version 1's points and capsules are the filtered landmarks, as before 2026-10-08.
• --pinch-dip 0|1 (HANDS_PINCH_DIP, 0): the dip rule for pinches (below).
• --grip-begin R, --grip-end R: the grip detector (below). Defaults 1.2 and 1.45.
• --gesture-log: print what the pinch and grip detectors measure, 10 times a second: each hand's thumb-to-index distance (world and triangulated), its palm-down reading and its finger curl.

Gestures (/run/user/UID/frametop-hands/gestures, include/fh_gestures.h), for the pointer helper:

• A pinch: the thumb and index tips within 2 cm, ending past 3.5 cm (see "Pinch" below).
  • With HANDS_PINCH_DIP=1, also on a sharp dip that doesn't reach 2 cm. Quick taps with the hand raised often read 25-30 mm at contact and spring back, so they never click. The dip rule begins a pinch when, within 150 ms, the distance falls by 40% of its recent highest to under 3.2 cm, with the palm moving under 0.3 m/s and the thumb at least 4 cm from the middle fingertip (a hand closing into a grab brings both in). It ends once the distance is 6 mm over its lowest, and becomes an ordinary pinch (drags and all) if the tips get within 2 cm. It is off by default. In ft-handreplay on frame-hands' guided tap capture (2026-10-09, student v4), it caught 25 of 30 quick taps on a 2 Hz click instead of 10, and 74% of the teacher's clear taps instead of 62% at the same precision (0.91 against 0.93). The cost is that fingers stopped about 1 cm apart click too: 16 pinches in the near-miss segment instead of 5. Pointing and grabs in the hand recorder's takes gave 22 pinches, against 21 without it (PinchParams::dip; frame-hands notes/taps.md, Part D).
• A grip, a closed hand: every finger's tip nearer the wrist than 1.2 times its knuckle is (from the model's 3D hand, so hand size doesn't matter), ending when they open past 1.45 on average. It begins only on a hand seen open within the last second (closing it is the gesture), with the palm at most 35 degrees below straight ahead and at least 15 cm in front of the eyes, and not with the thumb within 3 cm of the index tip (that's a pinch with the other fingers curled). A grip ends a pinch on the same hand, as lost. In the 2026-09-30 lit recording (no deliberate fists), the checks cut false grips from 14 to 6, all with the hands on the desk while looking down at it. The pointer helper ignores grips that begin more than POINTER_GRIP_BELOW (0.35 m) below the eyes, which it can tell and ft-hands can't.
• A middle pinch (gestures version 3): the thumb on the middle fingertip, with the pinch's thresholds. It begins only with the thumb at least 1 cm nearer the middle tip than the index tip, the index extended (its tip at least 1.5 times as far from the wrist as its knuckle), and the hand tracked for 5 frames; a pinch and a middle pinch on one hand exclude each other, and a pinch begins only with the thumb nearer the index tip than the middle one. The pointer helper's hand ray turns it into a right click or scrolling. Closing the hand into a grab brought the middle tip to the thumb with the index curled: before the index and frames rules, the recorder's gestures take (PR #8, about 95 s of pinches and grabs, none of them middle pinches) gave 6 false middle pinches; after, 1. The 2026-09-30 lit recording (typing and 26 deliberate pinches) gave none, and the same 48 pinches as without the middle pinch.
• tools/watch_gestures.py --distance shows them live; ft-handreplay --timeline logs them (middle pinches as pinch2) and each hand's finger curl.

The status line also says how often a hand was on each side (by where the wrist is), and why views and hands came and went: views lost (the landmark model stopped seeing the hand), handoff misses (a crop projected from the hand's 3D position found nothing), duplicates, splits (two views disagreed in 3D), and hands created, merged and forgotten.

Scheduling

• Each hand is tracked in its best two cameras, the way MediaPipe tracks: the landmark model runs on a crop placed from the previous landmarks, with no palm detection.
• A hand seen in too few cameras is projected into the others through the calibration. Where it lands well inside a camera, that camera gets a crop to try. This is how a hand raised out of the side cameras reaches the upper ones.
• The palm detector runs only while fewer than two hands are tracked, at most 5 times a second, on a few zoomed tiles per search. Tiles are picked in proportion to how likely hands are there. Each tile is turned so the expected shoulder-to-hand direction points up. There are 42 tiles and a search tries 2 (3 with no hand tracked), so even the likeliest tile comes round only every 10-11 searches: about 2 s at 5 searches a second, and 4-5 s live, where a search round takes about 50 ms and the tracker manages 2-3 a second.
• Re-acquiring a lost hand (--reacquire, on by default since 2026-10-08). A hand that stops being seen is first held, not published: 0.6 s if it was tracked for 30 frames or more (--hold), else 0.3 s. Meanwhile its crops are retried where it was heading (its last position plus its velocity). From the first step it's missing and for 1.5 s after it's forgotten, the palm search aims up to 2 tiles at where it should be, in the cameras that see that spot best. The aimed tiles grow with the time since the loss, and they run every processed set while such a hand is missing, not only 5 times a second. The rotation keeps at least one tile per search. In replays this brought a lost hand back on the next processed set (median 0.05 s, against 0.12-0.25 s), and the slowest cases from seconds down to under 1 s (see frame-hands' notes/reacquire.md).
• Left and right (--pair-sides, on by default since 2026-10-08). Each hand used to be called left or right by a running average of the landmark model's call, and both hands could get the same side for seconds. Now each hand also accumulates the model's calls as fading log-odds, and two hands seen together are one left and one right: the hand with more right-hand evidence is the right one, and which wrist is further right only tips a close call.
• Frame sets are processed at 30 Hz while a hand moves faster than 0.25 m/s (or a pinch is down or closing), at 15 Hz otherwise, and at 5 Hz while no hand is in view. A reader that needs the full rate writes it to /run/user/UID/frametop-hands/want-hz and keeps the file fresh: ft-screens writes 30 there while a hand is near one of its panels, because at 15 Hz a poke's press would come up to 67 ms later. A file older than 1 s lapses.

3D

• Two or more views: each landmark is triangulated from the camera rays, weighted by the model's presence score. The median ray distance is reported as the residual.
• Pairing views across cameras. The side cameras sit side by side, so two hands next to each other at the same height fall on the same epipolar lines, and rays to two different hands can nearly meet close to the cameras. That made phantom hands 12-15 cm in front of the eyes, which tore holes through the screens. Each step now scores every way of pairing the views in two cameras and keeps the best. A pair scores well when its rays meet, when each view's apparent size matches the triangulated distance, and when the model calls both the same hand. The size check uses a fixed prior: with the model's average hand, clean pairs measure 0.71-1.51 times the one-view distance, and mismatched pairs mostly far less.
• One view: depth comes from the model's metric world landmarks, their spread across the palm against the angle it covers in the image, scaled by the user's hand size (learned while two views are available). That distance is off by 10-30% and wanders about 10% between frames, so a hand that drops to one camera keeps its last distance and drifts toward the one-view guess by 10% a frame.
• Smoothing. The landmarks go through a One Euro filter: it smooths hard while the hand is still (tracking noise is several mm per frame) and hardly at all while it moves fast. The palm speed that sets the update rate is the filtered one; the raw speed read about 0.25 m/s from noise alone. The gestures (pinch point, palm-down reading) use these filtered landmarks. What's published is the skeleton's (below), unless --skeleton 0.
• Capsules. Forearms follow the hand's own axis, and nothing within 12 cm in front of the eyes is published.

How good the depth is, measured from recordings (2026-09-30, --depth below): the two lower cameras see the hands about 77% of the time, a lower and an upper camera 7-12%, and one camera 12-15%. Depth is the noisy direction. With the lower pair, it jitters 4-6 times as much as sideways position (published: 3-7 mm against 1-2 mm). The one-camera guess is a median 2-6 cm off. When a camera drops out, drifting 10% a frame toward that guess is worse than keeping the last distance (after 0.5 s a median 23-30 mm off, against 11-12 mm).

Skeleton

After the 3D step, ft-hands fits a skeleton with your own bone lengths to each hand (track/skeleton.h), so fingers keep their length and don't wobble. It's the base for a hand mesh, colliders and the hand ray. --skeleton 0 (HANDS_SKELETON=0) turns it off.

• Bone lengths, per side. From clean stereo frames (residual under 4 mm, the model sure of the side), each of 30 numbers is learned: the thumb's CMC and the four knuckles' places in the palm, and the 15 finger bones. A landmark whose two rays miss each other by more than 4 mm, a finger bent more than 90 degrees, or a bone far off the average hand's proportions (under 0.7 or over 1.5 times them) gives no sample. Each settles on the median of its first 45 samples (3 s of a clean hand at 15 Hz), then moves at most 0.02 mm a sample. Until then the estimate is pulled toward a prior: the other side's lengths if it has more, else MediaPipe's average hand times Hand::scale. Nothing is saved: each start learns again.
• The fit, every frame. The palm is rigid: its frame from the wrist and knuckles, then Gauss-Newton on its rotation and position. Then the thumb and each finger as a chain from its base: the thumb's CMC turns three ways (±50 degrees from a rest pose, ±35 twist), its MCP bends and spreads a little, its IP is a hinge; a finger's knuckle bends (-35 to 95 degrees), spreads (±25, the little finger ±30) and twists a little (±25), the two joints above it are hinges (-10 to 110, -20 to 90). A chain's base may sit up to 15 mm off the rigid palm, pulled back with a 4 mm prior: the measured knuckles wander far more than a palm bends (the little finger's by 3 cm across the palm). Each chain runs 4 Gauss-Newton steps (the limits as clamps) twice, from the measured bone directions and from last frame's pose, and keeps the better fit, so it doesn't hop between two poses that fit about as well.
• Weights. Stereo: each landmark by its own ray miss (2 mm at least), and 3.5 times less sure along the line of sight. One view: 3 mm across, 15 mm along it.
• Which side. The model's, but a tracked hand keeps its side while the model's running call is between 0.25 and 0.75. The mirror image is tried when the fit is worse than 15 mm rms, and taken after it fits at least twice as well 3 frames in a row; it then sticks while the hand is tracked (a finger can't bend backward, so a hand fitted as the wrong side fits badly once its fingers bend). Switching restarts the filter.
• Filtering the pose, not the points. The palm centre and the palm's rotation go through One Euro filters (2 Hz still, plus 30 Hz per m/s and 10 Hz per rad/s); each finger's angles and base through one whose cutoff follows that finger's own speed (2 Hz plus 3 Hz per rad/s), so a pinch on a still palm doesn't lag the way the One Euro on the points does. A chain that fits its landmarks worse than 15 mm this frame moves its filter less (by the square of 15 mm over its error): that's mostly a hidden landmark misplaced.
• Elbow. frame-hands' arm model (notes/forearm.md): a neck pivot fixed in the head frame, shoulders 18 cm to each side and 6 cm down, upper arm 1.5 times the forearm, the forearm 2.8 times the palm (wrist to middle knuckle), the elbow on the circle the two bones allow, turned 3 degrees from straight down, and never more than 75 degrees off the hand's own axis. ft-hands has no HMD pose, so "down" is the head frame's -y: published with FH_FIT_ELBOW_HEAD_UP. The model's other parameters (neck and shoulder offsets) are its starting values: the fitted ones are on the training PC.
• Published in the hands file's version 2 (include/fh_hands.h): after version 1's struct, which is unchanged, an fh_hand_fit_t per hand with the 21 joints (position, a rough radius, and a quaternion in OpenXR's hand-joint axes), the bone lengths, the joint angles, the elbow, the pinch point (between the index and middle knuckles) and the thumb-index contact point, the thumb-index and thumb-middle distances, the fit's rms and how settled the calibration is, and the palm's velocity and angular velocity. Those two are in the head frame and include the head's own motion; for motion in the room, difference room-frame points across publishes as screens/handcut.cpp does. Version 1's points and capsules are the skeleton's too, so ft-screens' cutouts get it without a change. The hand recorder's reader takes version 1 or 2.
• The gestures don't use it: pinches still measure the model's world landmarks, and the pinch point and palm-down reading still come from the filtered landmarks. ft-handreplay --pinch-fitted counts pinches on the fitted tips for comparison: they begin about as often as on the triangulated tips, more than on the world landmarks.

What it measured, ft-handreplay on three recordings and three takes of the 2026-10-03 hand-recorder session (frame-hands notes/skeleton-fit.md has every number):

2026-09-29/30 recordings, 30 Hz	dim (typing)	lit	bright
Bone length spread per track (median CV): measured / One Euro / skeleton	0.41 / 0.37 / 0.05	0.24 / 0.22 / 0.03	0.39 / 0.35 / 0.01
Index tip jitter, hand and fingers still, mm (median; 90%): One Euro / skeleton	4.5 / 3.9; 16.9 / 24.3	3.4 / 2.8; 11.0 / 14.7	4.2 / 3.2; 11.5 / 10.2
Thumb tip jitter, still, mm (median; 90%)	3.0 / 2.7; 12.0 / 11.8	2.4 / 2.7; 7.8 / 12.0	3.0 / 2.2; 10.1 / 8.7
Index tip jitter, palm over 0.3 m/s, median mm	14.0 / 11.2	11.0 / 8.6	17.7 / 14.7
Palm jitter, still, median mm	1.8 / 1.8	1.9 / 1.8	1.9 / 1.8
Tips trailing, moving over 0.5 m/s, median mm	6.5 / 5.7	5.1 / 4.9	5.7 / 4.9
Tips trailing, fingers moving on the palm, median mm	7.8 / 3.5	5.9 / 4.7	4.6 / 3.5
Palm normal's turn per update, palm still, fingers moving, median degrees	2.8 / 1.7	2.2 / 1.3	2.3 / 2.0
Fit rms against the measured landmarks, stereo, median mm	10.4	9.0	7.6
Calibration settled after each side's first sample, s (left, right)	20.5, 10.9	10.8, 3.5	15.6, 8.9

• Against the auto-labels (frame-hands' teacher, triangulated, smoothed and fitted: not ground truth; three takes of the PR #3 session, 10 Hz, stereo hands): mean landmark error measured 10.2, 12.1 and 7.0 mm, the fit before its filter 10.4, 12.2 and 6.5 mm (pose sweeps, gestures, desk work). The index tip: 8.1, 7.7 and 7.9 mm measured, 8.7, 8.7 and 5.5 mm fitted. The labels' ring and little knuckles sit far from MediaPipe's (their wrist to little knuckle is 110-140 mm against our 80-99), so part of every number is the two models' conventions.
• Cost: 41-43 us a hand (median; 90% 61-70 us, 99% 91-164 us) on an A720 core, the fit alone replayed on CPU 3; 320-350 us on an A520 (CPU 0). ft-hands runs on CPUs 5-7, which are A720s and the X4.
• Pinches don't change: the same counts in every replay, because they don't use the skeleton. On the fitted tips they'd begin more often (lit: 77 against 51; dim: 50 against 35), which is why they stay on the world landmarks.
• Worse: the fingertips' jitter tail with the hand still in the dim and lit recordings (index 90% 24 against 17 mm, 15 against 11 mm; thumb in lit 12 against 8 mm). That's the price of each finger's own cutoff: a misplaced tip that the skeleton can reach (a few frames of a hidden finger) gets through where the palm-speed One Euro held it. The fitted landmarks also sit further from a centred average of the measured ones (moving fast: 11.7 against 9.3 mm in lit), mostly the fit's own residual: the labels don't say the fit is the worse of the two.
• Elbow: the forearm comes out 231-247 mm, a median 21-27 degrees off the hand's own axis. With gravity's up from the takes' HMD poses instead of the head's, it moves a median 0.6-2.2 degrees (90%: 2.2-5.0): the head-up assumption costs little at a desk. No elbow labels are here to say how right it is.

Pinch

ft-hands detects a pinch per hand (track/pinch.h) and publishes it to the gestures file. The layout, and how to read it without missing quick taps, is in include/fh_gestures.h.

• A pinch begins when the thumb and index tips come within --pinch-begin (default 0.020 m). It ends when they open past --pinch-end (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost).
• A pinch held 0.3 s or longer is a drag, and holds on harder: it ends only once the tips are 5 cm apart (for the same 2 frames), or 0.12 s after they first went past 3.5 cm unless they closed back under 3 cm meanwhile. In the 2026-10-09 headset test drags ended with the tips 3.5-4.0 cm apart, a thumb hidden for a frame or fingers loosened a little mid-drag. On the lit recording's replayed distances, pinches held 0.3 s or more (typing among them) ended 33-134 ms later than before, 200 at most, and one of 25 ran into the next pinch's begin. A grip ends a pinch (as lost) only within 0.3 s of its begin, when closing the hand passes through a pinch; after that no grip begins on that hand, so curling the other fingers around a held pinch doesn't drop it (3 of 28 pinches in the same test). tests/pinch_test.cpp (make check).
• The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. --pinch-triangulated uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. In the dim recording, typing still gave 2 pinches a minute (see the next point).
• --pinch-palm-down MAX holds back pinches begun with the palm facing down (MAX is the palm normal's share of the head's up axis). The default, 1, turns it off. A close held back that way has to open again before a pinch can begin. Typing curls the thumb onto the index: in the lit recording of 2026-09-30, typing on a keyboard in the lap began 23 pinches in about 2 minutes, all with the palm facing down (0.69-1.00), while the 26 deliberate ones read 0.00-0.50. But in the headset, deliberate pinches with the hand raised in front read 0.90-0.99 too, so the limit is off. Typing is caught by the pointer helper instead: the input relay tells it when you press a key, and the hands on the keys are held back (POINTER_PINCH_TYPING and the settings after it, below).
• A hand a pinch is down on stays with that side until the pinch ends. The left/right call is a running average of the model's, and when it flipped mid-pinch, the other side took the same hand and both sides pinched at once.
• The pinch point is between the index and middle knuckles, which hold still while the fingers open and close. The tips' midpoint moved 1-2 cm as a pinch opened, which dragged every release off its press. A drag is the pinch point now, minus where it was when the pinch began, both turned into the room with the HMD pose at their capture times.
• tools/watch_gestures.py prints begins, ends and drag offsets live, and --distance prints each hand's distance.

Pinches and grips in the pointer

With POINTER_HANDS=1, the pointer helper (pointer/helper/ft-pointer.cpp) reads the gestures file every frame. It's off by default.

The hand ray (POINTER_HAND_RAY=1, the default with POINTER_HANDS): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (include/fh_ray.h: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless POINTER_GAZE_HAND_RAY=1. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, pointer/helper/handoff.h). The rules, timings and measurements are at the top of ft-pointer.cpp ("Handoff", "Hand ray") and in frame-hands' notes/hand-interaction.md. pointer/tools/rayeval runs the ray over a replay (ft-handreplay --ray).

• Pinch to click, in gaze mode. Your eyes aim and your hands click, Vision Pro style, and the hand ray stays off (POINTER_GAZE_HAND_RAY). A pinch locks the pointer where you were looking just before the fingers began to close (75 ms before ft-hands saw the pinch, since the eyes often move on as the hand closes), and a quick tap clicks there. Pinch, hold and move the hand to nudge the pointer: nothing moves until the hand has moved 4 degrees (about 3 cm at arm's length), then the pointer follows at 0.3 times the hand's movement. Letting go clicks where the pointer is, and the correction is a lesson for the gaze tracker, as with the mouse. Held still for POINTER_GAZE_HOLD (0.5 s) instead, the pinch is a real press at that point, and the hand drags at POINTER_GRIP_GAIN. A middle pinch is a right click the same way. Hold a pinch with one hand and pinch with the other: both buttons are down, left for the first hand and right for the second, each until its own pinch opens (a first pinch not yet pressed becomes a press at once, where the pointer is), as when you hold both of a mouse's buttons. The rules and timings are in pointer/helper/gazehand.h.
• Without gaze mode, a pinch is a real press, like the mouse's button: pressed when it closes, released when it opens, and while it's held the hand drags the pointer. A tap is still a click where the pointer is.
• Grip to drag. Closing the hand presses where the pointer is, the hand moves the pointer, and opening the hand releases. So a title bar moves its window, a panel's grab bar carries the panel, and text gets selected. A panel the hand ray carries by its bar (or a floating window by its title bar) stays upright and facing you wherever the ray takes it (2026-10-13: the ray can't tilt it, so the wrist's turn shouldn't either); a controller's laser still carries it rigidly.
• A pinch ended by losing the hand, or by a grip taking over, doesn't click.
• The hand's movement is taken in the room, from where the eye was when the gesture began, so turning your head doesn't move the pointer. The first gesture while the pointer is off only wakes it. Gestures are ignored in a VR game (unless the dashboard is up), with the headset off, and while the mouse's button is held.

Settings in ~/.config/frametop.conf:

• POINTER_HANDS (0): 1 turns pinches and grips on.
• POINTER_GAZE_PINCH (1): in gaze mode, 0 turns clicks at the gaze off (pinches and grips with no ray do nothing), leaving the ray when gaze is off and direct touch. Frametop Input Settings' Hands page has the three hand inputs as switches of their own (hands/handsmode.py, methods_of).
• POINTER_PINCH_GAIN (0.5): without gaze mode, a held pinch moves the pointer this many times the hand's angle, seen from the eye. Under 1 gives precision.
• POINTER_PINCH_DEADZONE (1.5): without gaze mode, how many degrees the pinching hand moves before the pointer does, so a tap's jitter and the pinch point shifting as the fingers close don't move it.
• POINTER_GAZE_HAND_RAY (0): in gaze mode, 1 keeps the hand ray beside the gaze (a raised, pointing hand's pinches are then ray pinches).
• POINTER_GAZE_PINCH_DEADZONE (4): in gaze mode, how many degrees a held pinch moves before it nudges the pointer. A tap's jitter and a hand drifting while you look stay inside it.
• POINTER_GAZE_PINCH_GAIN (0.3): in gaze mode, past the dead zone the pointer moves this many times the hand's angle. Low, so a nudge is a fine correction.
• POINTER_GRIP_GAIN (1): a grip moves the pointer this many times the hand's angle.
• POINTER_GRIP_BELOW (0.35): grips that begin more than this many metres below the eyes are ignored, because hands resting on a desk curl like a loose fist. Pinches have no such limit: deliberate ones sat 0.35-0.45 m below the eyes with the elbow resting.
• POINTER_PINCH_TYPING (2), POINTER_TYPING_RAISE (0.08 m), POINTER_TYPING_BELOW (0.2 m), POINTER_TYPING_POSTURE (0.3 s), POINTER_TYPING_RECENT (600 s): typing on a real keyboard. For POINTER_PINCH_TYPING seconds after a key, no hand pinches, grips, pokes or shows a ray, except one lifted off the keys; a hand that was on the keys stays held back through any pause until it rises POINTER_TYPING_RAISE above where it typed (or turns its palm up); both hands down on the keys together count as typing before the first key. pointer/helper/typing.h and docs/hands-typing.md have the rules and the measurements.
• POINTER_HAND_RAY (1), POINTER_RAY_DEADZONE (2.5 degrees the ray turns during a pinch before the press drags), POINTER_RAY_PUSH (1), POINTER_RAY_SCROLL (1; -1 turns scrolling round), POINTER_RAY_RELEASE (2 s after the ray goes before a pointer it woke lets go), POINTER_RAY_GRACE (0.5 s a hand keeps the laser after its ray drops; the other hand's ray takes it after that, or at once by pinching or raising): the hand ray.
• The ray's smoothing (include/fh_smooth.h, through fh_ray.h): One Euro filters run in continuous time, so the pointer's pose moves at the helper's rate (about 100 Hz) instead of stepping at the tracker's 15-30 Hz; the direction is filtered on the sphere. POINTER_RAY_SMOOTH_MINCUTOFF (0.8 Hz with the hand still), POINTER_RAY_SMOOTH_BETA (0.06 Hz more per degree a second it turns), POINTER_RAY_SMOOTH_HOLD (0.5 Hz still while a pinch is held: a drag's precision). The reach, push and pull's input, has a heavier one of its own. POINTER_RAY_STEADY (1): while a pinch is held (and 0.25 s after), the ray pivots on where the shoulder was at the press, then eases back to the body model over 0.5 s; the model's shoulder follows the head, which made looking around move what was dragged. POINTER_RAY_TAP (0.3 s) and POINTER_RAY_TAP_DEADZONE (6 degrees): a pinch's first 0.3 s takes 6 degrees of turn before it drags, so a quick tap whose hand swung stays a click; after that POINTER_RAY_DEADZONE applies. When the press starts dragging the pointer carries on from where it was, no jump. On the probe-moves replay (pointer/tools/rayeval on frame-hands/dumps/probe-moves-v4.dump, read at 90 Hz) the pose's second difference per tick went from 0.90-0.96 degrees rms (p90 1.4) to 0.24-0.26 (p90 0.24) at about the same lag (110 ms behind the capture, against 105).
• The ray's body and hand models (include/fh_ray.h; ideas and formulas from StereoKit, MIT): POINTER_RAY_TORSO_WRISTS (1): the shoulders face between the head (counted twice) and the wrists seen, so looking aside while pointing swings the ray less. POINTER_RAY_HAND_WEIGHT (0; 0..1): how far the ray turns from the shoulder's line toward the hand's own direction (POINTER_RAY_HAND_PITCH -30 and POINTER_RAY_HAND_YAW 8 degrees off the palm's axes). POINTER_RAY_ANCHOR (0): the knuckle point pinned to the palm while a pinch is held. The last two measured worse on replays (pointer/tools/rayeval; numbers in fh_ray.h), so they're off.

Flags

When tracking goes wrong in real use, flag it: ft-handsctl flag [note], or by voice with frame-voice's voice commands (its commands setting: "flag that" runs ft-handsctl flag with the words after it as the note, e.g. "flag that, fingers crossed"). The offline tools (frame-hands' train/flags) relabel what was saved with the teacher, find what went wrong and in which situations, and make guided capture sessions for the weakest ones.

ft-hands keeps a rolling buffer in RAM (track/flagbuf.h, HANDS_FLAG_BUFFER=1): per processed set, a mono crop around each view the tracker used (its landmark crop's centre, 1.6 times its size, axis aligned, at most HANDS_FLAG_CROP px a side), with where it is in the image, so labels made on it lift back to the image and triangulate; a lost view is still cropped where it was for a second, so the teacher can say whether the hand was there; and its hands, gestures and each view's landmarks. It keeps HANDS_FLAG_SECONDS and at most HANDS_FLAG_MB of crops; nothing is written until a flag. Cost, measured (2026-10-09, a replayed ring with two hands in view): about 80 crops a second at 20 sets a second, 56 MB of crops for 14 s at 224 px (ft-hands' peak RSS 101 -> 159 MB), and 0.1 ms a crop on a tracking core (A720), so about 1% of a core; ft-hands' CPU time was the same with and without it. At 30 sets a second with four views the 80 MB cap can trim the oldest seconds.

ft-handsctl flag makes HANDS_FLAG_DIR/flag-<time>/ and asks ft-hands for the buffer through run_dir/flag; ft-hands keeps going for 2 s more, then a thread of its own (off the tracking cores, lowest priority) writes manifest.json (written last), sets.jsonl, crops.jsonl and crops.bin. Once ft-hands has taken the request, the drawn hands pulse cyan for half a second (ft-handview's flash; HANDS_FLAG_FLASH=0 turns it off; HANDS_FLAG_TONE=1 also plays a short tone). It returns then, and in the background saves one frame of your view from /dev/video99 as view.jpg (HANDS_FLAG_VIEW=0 skips it), the head's poses over the window from ft-handview (poses) as hmd.jsonl, and ctl.json. ft-hands logs each flag ("flag:", "flag written:"), and the status line has the buffer's size and cost. A flag while one is being taken adds its note to it. The format is in frame-hands' notes/flag-failures.md. The crops show your hands and some of the room: keep flags with your recordings.

ft-handreplay REC --flag-at S1,S2 --flag-out DIR (or --flag-every S) makes the same flags from a recording, with the same buffer, for testing the offline tools before real flags exist; --flag-crop PX, --flag-scale F, --flag-mb MB and --flag-seconds S change the buffer.

Recordings

hands/build.sh --tools also builds the offline tools.

ft-handreplay DIR runs a recording through the tracker with the live scheduling and reports how well it kept the hands: hands per set, left and right coverage, track lengths, pinches, jitter, and the same reasons as the status line. --publish-dump FILE also writes what ft-hands would have published after each processed set (the hands and gestures files' bytes, and the set's dequeue time), for ft-screens' offline tests; a hand-recorder take's poses.jsonl and prompts.jsonl copied next to the dump as NAME.poses.jsonl and NAME.prompts.jsonl give them the head's poses and the touch take's targets (scripts/test-handinput.sh, scripts/bench-matte.sh). To match the live tracker, replay with --reacquire 1.5 --reacquire-fast --pair-sides --hold 0.6 and the models in use.

hands/build/ft-handreplay ~/.local/share/frametop/hands/rec-20260929-120000 --cost --oracle 10 --timeline /tmp/tl.txt

• --cost: instead of timing the steps, charge each round of model calls what it typically costs live (10 ms landmarks, 18 ms palms), so results repeat exactly.
• --oracle N: every N-th set, also search every tile of every camera, and report how often the tracker had the hands that full search could find.
• --slow F: live, the tracker skips sets that arrive while it's busy. Replay counts each step's time times F as busy (default 1; the headset is busier live).
• --timeline FILE: a line per processed set and hand, with pinch events and distances.
• --ray FILE: a line per processed set and hand with what the pointer helper's hand ray reads (time, side, stereo, pinch and grip state, the published points, the wrist's frame, the elbow), for pointer/tools/rayeval with the same session's ft-handtest probe log (head poses).
• --cams mono|color|all: which cameras to track with (default mono). color tracks with the Arcturus pair alone, for comparing it with the IR cameras on the same recording. It needs a recording made with ft-camd --with-color. --color-left NODE (color_video0 or color_video3) and --color-crop subtract|none say how the module's calibration maps onto the images; tools/check_color.py finds out.
• The skeleton (see "Skeleton"): the report ends with its measures for the measured, One Euro filtered and fitted landmarks (and the fit before its filter): its time per hand, also on its own pinned to a little and a big core; its rms; each side's calibration and when it settled; how much bone lengths spread per track; jitter at rest and moving for the palm and the thumb and index tips; how far the tips trail during fast motion (against a centred average of the measured tips); how steady the palm's normal is; and the elbow. --no-skeleton replays without it; --skel NAME=VALUE changes a setting (skel::Params); --labels FILE compares with frame-hands' auto-labels (labels.jsonl), --skeleton-dump FILE writes the matched hands; --hmd-poses FILE (a take's poses.jsonl) compares the elbow with gravity's up against the head's; --pinch-fitted counts pinches on the fitted tips too.
• A hand-recorder take is a recording too: its sets.bin is followed by sets-2.bin and later parts, which are read in order. It has no sides.json: whether its side cameras need exchanging is take.json's names_swapped against session.json's sides (rec/sides.py), so give --sides 0 or 1 (--sides auto checks from the hands).
• --depth FILE: a line per hand per processed set for tools/depth_report.py, which measures the depth without ground truth: how the hands were seen, the noise along the line of sight against across it, each camera's one-view distance against the triangulated one, and what a camera dropping out would do.
• The pinch, contrast and presence options are ft-hands'. --reacquire S (off unless given; --reacquire-fast searches every set as ft-hands does, --reacquire-tiles N, --reacquire-clear M), --pair-sides, --hold S, --near F and --search-budget N are the tracker's re-acquire settings, off or at their old values by default so old replays repeat.
• --cost-hand MS, --cost-palm MS: with --cost, what a round of landmark and palm calls costs. The defaults (10, 18) are the headset's idle costs. The 2026-10-08 live probe (CPUs 0,1,4 beside SteamVR, 3 threads) measured about 14 and 52, and the live tracker loses hands far more at those costs.
• --drop-every S: the re-acquire test. Every S s, while two hands are tracked, one of them is made invisible for 0.35 s (just past the 0.3 s the tracker holds a lost hand), and the report gives how long it then took to have two hands again: the time to find a hand that never left the view.
• --flag-at S1,S2, --flag-every S, --flag-out DIR: flags as ft-handsctl flag makes them live (see "Flags"), at those times into the recording.
• With --timeline, the tracker also writes why views and hands came and went (ev lines: lost, miss, split, search and its tiles, palms kept or rejected, new, forget, pair-sides), and --oracle writes what the full search found. tools/gap_report.py TIMELINE --list turns that into every gap of 0.3 s or more per side, with its cause. tools/replay_gaps.sh OUT MODELS "ARGS" REC... runs several replays at once (on the 7i through frame-job) and reports them. For pinches, --tips FILE logs each hand's thumb, index and middle tips and the pinch detector's distance measures per processed set, with its pinch events; --rate HZ fixes the processing rate (15 or 30, against the live rule) and --want-hz HZ plays a reader asking for it; tools/replay_tips.sh OUT NAME=MODELS "ARGS" REC... runs them (frame-hands' train/taps/tapeval.py scores them against auto-labels).

ft-ringplay DIR --ring PATH [--from S] [--to S] [--loop] publishes a recording into a ring file in real time, as ft-camd would, so ft-hands --ring PATH --no-publish runs the same frames run after run. It needs no privileges, and it skips the dark frames.

Tools

Python, with NumPy and OpenCV. setup/dev-container.sh doesn't install them, because Fedora's python3-opencv pulls in over a gigabyte; in the dev container, run sudo dnf install python3-numpy python3-opencv once. Off the Frame, FRAME_JOB_DEVICE_ROOT can point at a folder with copies of the headset's calibration files.

• tools/check_sides.py --ring (or a recording, a sets file, --pair upper, --calib DIR, --json): are the side cameras named right, from the scene? ft-handreplay's --sides file|0|1|auto replays with DIR/sides.json's names (the default), as recorded, exchanged, or as auto decides, and reports what the side check found and when.
• tools/check_color.py REC: how the colour module's calibration maps onto its images.
• tools/show_set.py REC: a recording's frame sets as images.
• tools/watch_gestures.py [--distance]: pinches and grips, live.
• tools/depth_report.py DEPTH: the depth measures above.
• tools/cut_sets.py REC OUT [--sets N | --at I,J,...]: copies a few frame sets (by default 8, spread evenly) out of a recording into a small one, to look at or check elsewhere without moving gigabytes. Plain Python, so it also runs on the Frame's host.
• tools/convert_models.py: how models/ncnn was made from the OpenCV Zoo ONNX ports of MediaPipe's models (see models/NOTICE).

To try the hand cutouts without restarting the desktop, screens/build/ft-handtest [--distance m] [--width m] [--seconds s] [--matte circle|mesh|capsules] [--margin deg] (built by screens/build.sh, run in the dev container, with hand tracking on) shows a test panel of its own, a light grid 1 m wide and 0.8 m ahead by default, and cuts your hands out of it the way ft-screens cuts them out of the screens. Every 2 s it prints the composite's cost.

Hand cutouts and near input

ft-screens reads the hands file (version 2's skeleton when it's there, else version 1's points) and the gestures file once per tick (screens/handcut.cpp). It cuts only with Room View on: with it off, a hole would show the void or the game, not your hand (see "Rendered hands"). cutouts state ends with room view on|off (why); cutouts roomview auto|on|off forces it.

Off by default since 2026-10-09 (HANDS_CUTOUTS=0): ft-handview draws the hands over the panels instead (see "Rendered hands"). HANDS_CUTOUTS=1, or ft-handsctl cutouts on for the session, brings the holes back.

The matte (HANDS_MATTE=circle, the default since 2026-10-09) is a round hole per hand: a disk facing the eye, centred on the middle of what the head sees of the hand (the box around its joints, each with its half-width, the wrist's 2.5 cm), wide enough to reach the hand's furthest edge from there, plus the margin. Each eye sees the disk on the screen as an ellipse (a circle on a screen square to it), drawn as the stadium around it with the capsules' soft edge (15% of its radius). It hides more of the screen than the hand needs, and with it the hand's tracking error and the few degrees Room View's picture of the hand sits off it, until the tracking is good enough for a hand-shaped hole. The forearm isn't in it. Building and projecting it costs 4 us median on the desk-work replay (the mesh 26 us), the composite the same as the mesh's or less (scripts/bench-matte.sh on probe-moves: 2 pieces per eye against 35).

HANDS_MATTE=mesh is hand-shaped: a tapered capsule per finger bone, with real fingers' half-widths (10.5 mm at the index's knuckle down to 7.5 mm at its tip, scaled by the hand's palm length); the palm as a 2.4 cm slab through the wrist and the knuckles, widened to the hand's edges; the thumb's web; and an 18 cm stub of the forearm toward the skeleton's elbow (along the hand's axis without a skeleton). The palm, web and forearm are convex pieces, each drawn as its outline grown by the margin with a soft 2 px edge. HANDS_MATTE=capsules draws ft-hands' capsules instead, as before, for comparison (cutouts matte circle|mesh|capsules switches it live).

Its margin is an angle seen from the eyes, so it covers the same share of the tracking error at any distance: HANDS_MATTE_MARGIN (1.5 degrees) with the hand still, plus HANDS_MATTE_MOTION (25 ms) of the hand's motion, at most HANDS_MATTE_MOTION_MAX (1.5 degrees) more. The cutout placement probes of 2026-10-08 (frame-hands captures/probe-2026100*/video_analysis) measured the cutout points against the hand Room View shows: 1.8-2.4 degrees median still (p90 4.1-4.7), 2.3-3.1 moving at 0.1-0.8 m/s, 4-6 above 0.8 m/s. Part of that is the video landmark model's own error, and an edge needs only the error across it (about 70% of a point's). A margin covering the p90 (4.5 degrees, 3 cm at 40 cm) would turn each finger into a 7 cm blob; 1.5 covers about the median. cutouts margin DEG [MS [MAX]] changes it live, and so does the Hand cutout margin slider on the Hands page of Frametop Input Settings (0-3 degrees in quarter steps; it sends the still margin only and writes HANDS_MATTE_MARGIN). The prediction lead stays 36 ms (HANDS_MATTE_LEAD, cutouts lead): the probes' best lead for fast hands was 6-14 ms, but 36 costs only 0.1-0.2 degrees.

The probes also saw the hand's picture off by an amount that depended on its distance, but not the same way in any two runs, so the correction is a hook, off: HANDS_MATTE_SHIFT="0.35:0.5,-0.3 0.6:-1,0" would turn the matte's hands by those yaw and pitch angles (degrees, right and up positive) at those distances from the head, in between linearly (cutouts shift ...|off live).

Cost, offline (scripts/bench-matte.sh on the probe-moves and lit replays, a 3440x1935 screen 0.6 m ahead, GPU time to finish each buffer on an otherwise idle GPU): the mesh 3.9-4.1 ms per full buffer against 3.8-3.9 for the capsules, and 3.4-3.6 against 3.0-3.4 when only the hands' boxes are redrawn (the margin is wider than the capsules' 4 mm). CPU 0.2 ms per composite either way; building and projecting the matte 14-24 us median.

Near input (screens/handinput.cpp, frame-hands notes/hand-interaction.md sections 3, 5, 16, 17, 18 and 19): a bare hand whose index tip, thumb-index contact, knuckles or palm is over a panel (or within the grab band outside it) and within HANDS_NEAR_REACH (0.075 m, about 3 inches; 0.30 on 2026-10-10 before the live test, 0.15 before that) in front of it touches it: ft-screens tells the pointer helper so ("near ... 1"), which then hides that hand's ray and leaves its pinches and grips to ft-screens. Beyond it a pinch is the ray's or the gaze's. A hand stays near until it's 2.5 cm farther out, then 200 ms on, so the ray doesn't flicker at the edge; in that margin the ray stays hidden ("near ... 2") but the hand's pinches are the helper's again. The ray shows again once its pointing pose has held 120 ms, about 0.35 s after the fingertip passes 10 cm. A hand resting low by the body (the poke's resting rule below: typing, the lap) touches only within 4 cm, so a low panel within reach doesn't take its gaze pinches. While a hand is within 0.30 m of a panel, ft-screens asks ft-hands for its full rate (want-hz), so a tap from outside the zone is seen at 30 Hz. Near a panel:

• Poke to click: the index fingertip presses when it comes at the panel's face (user, 2026-10-08): past the surface for two hands frames in a row, or for one when it came in decisively (1.7 cm in on average over the last 0.3 s, on its way in on the frame before too, at 0.1 m/s or more, and under 2.5 cm in coming nearer on each of the last 3 frames: HANDS_POKE_TRAVEL, 0 = always two frames; 2.5 cm until the second live test of 2026-10-10, which refused taps that came 1.7 and 2.0 cm). A quick tap at 30 Hz is past the surface for one or two frames, and one the tracker has a centimetre short for none: two frames in a row lost most shallow taps in the live test of 2026-10-10. It also needs: seen by two cameras, moving into the surface at 4 cm/s or more when it crossed (over the last 100 ms), the finger (knuckle to tip) within HANDS_POKE_ANGLE (50) degrees of the panel's normal, the index straight (1.5) and either no other fingertip more than 6 mm deeper than its tip or the hand pointing (the index straighter than each other finger by 0.2; until the second live test the index had to be 5 mm nearer than the others, and the tracker often reads a pointing hand's middle finger nearly as straight, its tip within -7 to +4 mm of the index tip's depth: 14 of its 39 refusals), not past the surface for more than 5 frames or 4 cm (reaching through), and not from a hand resting low by the body (palm more than HANDS_POKE_REST_BELOW, 0.30 m, under the eyes and within HANDS_POKE_REST_NEAR, 0.32 m, of the head across the floor; it stays resting until 5 cm higher or further out). So typing under a low panel doesn't poke it, and reaching out to one does. It lets go 1.5 cm out, or 1 cm back from the deepest point (so a tracker that has the finger behind the panel still lets go). The click stays where it was pressed until the fingertip slides 1.5 cm; then it drags. No new press for 150 ms after a release, nor until the tip has been 1 cm out (a noisy pull-back pressed twice); a finger 8 cm through the panel lets go and needs to come back out first. Within 5 cm the pointer follows the fingertip. A tap up to 1.5 cm past the content's edge presses at the edge (the desktop's taskbar is on the bottom edge). The curve, reset, dock and close buttons take pokes too (below: Buttons). The index crossing the plane farther off the content (a hand reaching for the bar or a corner) isn't a poke: ft-screens logs it once, as "not a poke", with where it crossed. On synthetic taps with 3-6 mm of tracking noise, 50-56 of 60 shallow taps (0.5 cm past the surface) press, against 34-36 with the two-frame rule; a finger held still 1.5-2.5 cm out presses no more often than before (handinput-test, TestPokeNoisy).
• The hand owns the pointer: while a hand holds the button, SteamVR's laser moves (the pointer helper's head or gaze pointer, a controller) go nowhere, and on a panel whose pointer a fingertip moved in the last 250 ms neither do they. Before, the helper's pointer a few cm off the fingertip pulled the remote host's pointer away between the poke's press and release: a drag, not a click.
• Why a poke didn't press: ft-screens' log has a line for each press and release ("hands: right poke pressed screen 104 at 1932,869 (-0.6 cm, in at 0.25 m/s from 3.1 cm out, 1 frame(s) past, the finger 12 deg off)"), and for each time the fingertip crossed a panel's surface without a press, the rules that held it back ("hands: right poke on screen 104 didn't press (0.4 cm deep at most): past the surface for 1 frame(s); ..."). handinput state counts them.
• Pinch to move or resize: a pinch where the thumb and index meet (not the gestures file's point, the knuckles, 5-8 cm behind them) within the border band takes the panel: 6 cm outside its edges to 2.5 cm inside (HANDS_GRAB_BAND), anywhere in the touch zone's depth (in front of it, or a little behind), or up to 3 cm beyond it when the touch zone has the hand (the third live test: "the fingers +9.0 cm from it, out of the touch zone: nothing", the pointer helper having left the pinch to ft-screens). By any corner (6 cm) it resizes, and the opposite corner stays put: a screen or remote display along its diagonal (its aspect is its resolution's), a floating window in pixels from that corner (every corner of a window since 2026-10-10, its title bar's ends too). Anywhere else on the border, the bar, a floating window's title bar, it moves. Moved, it follows the knuckles 1:1 (closer and farther too) and keeps facing you. A soft white glow shows where a pinch would take it (below: Glows at corners, edges and buttons); with HANDS_EDGE_GLOW=0 the resize handle goes to the corner a pinch is near instead, brighter the nearer, lit blue while resizing. With HANDS_CUTOUTS=1 a glow on the panel also shows what a pinch would do there: amber at the corner (resize), blue on the border (move), green for a card pinch; a dot while it holds. (The glows are drawn into the cutout buffer, so with the cutouts off, the default since the drawn hands, they don't show.)
• Resize from any corner, by hand, gaze or laser (2026-10-10, as on a Vision Pro): each panel has one resize handle, and it goes to the corner something aims at: a laser's ray (a controller's, or the pointer helper's, which the gaze and the hand ray drive) landing within a fifth of the panel's height (5-20 cm) of a corner (or a hand's pinch at one, with HANDS_EDGE_GLOW=0; hands have their own glow since the fourth live test). Since 2026-10-13 (user: "remove the radio icon thing on corners for resizing and replace them with the glow that is used for grabbing with hand tracking") the handle draws nothing: the same soft white L round the corner as a hand's pinch gets shows it, brighter as the aim comes to the corner, brightest while resizing (the handle is still there, just outside the corner, to take the laser's press); with no aim for 0.4 s it goes back to the bottom right. Press it with the laser, or look at it and pinch (a gaze pinch on a resize handle or a grab bar is a press after 0.12 s held still, POINTER_GAZE_CONTROL_HOLD, not the 0.5 s elsewhere), and drag: the corner follows, the opposite one stays put.
• Pinch on the content to click (HANDS_PINCH_CLICK, 2026-10-10): a pinch within the touch zone, not in the band, presses where the fingertip's pointer was (if it was there in the last half second, within 4 cm of the fingers), else where the fingers meet; it's held while the pinch is, and moving the pinched hand 1.5 cm drags. Not from a hand resting low by the body or seen by one camera.
• Card pinch (HANDS_CARD_PINCH, 2026-10-10: "pinch from underneath the display and manipulate it like a card"): a pinch closing on the panel's edge with the panel between the thumb and the index (each tip at least 3 mm from its plane on some frame in the last 0.3 s, the line between them within 60 degrees of its normal, the hand along the panel: at least 45 degrees off its normal, the fingers within 5 cm of its border), and the index and middle fingers together against the thumb (thumb to middle tip within 4.5 cm, index to middle tip within 3.5 cm) or all four fingertips bunched at it (each within 5 cm: the "pinched fingers" hand), holds it like a card (since the second live test; the index alone was enough before, and is now the border's move or a corner's resize; the middle finger joining within 0.3 s of the pinch makes the move a card from there): it goes rigidly with the hand, its place and its turn about all three axes, 1:1, smoothed, past 1 cm and 3 degrees; no facing you, no level snap. An edge pinch from the front or the side has both tips on one side of the plane, or the hand pointing into it: that's the border's move.
• Fist grab (HANDS_FIST_GRAB, 2026-10-10: "use the fist to expand and rotate the display, as well as change its angle"): a grip (ft-hands' closed hand) made with the palm, knuckles or curled middle finger within the touch zone over a panel (or within 6 cm of its edge) takes it; so does a pinch by a closed hand (the index 1.15 extended or less, the other three 1.05 or less on average: ft-hands calls a fist with the thumb on the index a pinch, and in the third live test two fists clicked the content instead of grabbing it). It's held while either the grip or that pinch is, until the hand opens: the fist carries it 1:1 and turning the fist turns it about all three axes, as the wrist turns a pinched one (below); open the hand to let go. Two fists (or a fist and a pinch) on one panel stretch it and turn it, as two pinches do. Since the fourth live test (2026-10-10, "I can't grab using fists"): a fist is at a panel with its palm, knuckles or curled middle finger up to HANDS_FIST_REACH (0.11 m) in front of it or 10 cm behind it (a live fist went 8.5 cm through and wasn't "at it"), and a hand curled that far (the three fingers under 1.2) is ft-screens' that far out, so the pointer helper leaves its grip; a pinch on the content by a hand that's closing (the three fingers under 1.4 and down by 0.1 in the last 150 ms: a fist's thumb meets the index first) waits up to 150 ms to become a fist (then a fist grab) before it clicks, where it began. A fist farther away is the pointer helper's grip (a gaze drag). A fist or a pinch by a hand "grab that" has (held out open, or closing into its fist, aiming at a panel it would take) waits until that gesture fires, and then does nothing, or lets the hand go, at most 0.6 s, and then it's what the hand is by then (a fist, a second anchor, or a pinch): an open hand held out to a panel then a fist pulled back and turned over is "grab that", a fist made at the panel without that is a fist grab. At a panel within arm's reach "grab that" doesn't aim, so nothing waits.
• Turn it with the wrist (2026-10-10): while one hand carries a panel (by its border or bar, or a fist), turning the wrist turns the panel the same way, 1:1, about all three axes (yaw, pitch, roll), to line displays up. The hand's turn is read from its wrist and index and little knuckles (they hold still while the fingers pinch), smoothed (One Euro, 1 Hz still), relative to facing your head across the floor from the hand, and applied on top of the panel facing you: carrying a panel round you still only keeps it facing you, and a wrist turn adds to that. Nothing turns until the hand has turned HANDS_GRAB_ROTATE_DEADZONE (5 degrees; the hand frame jitters about 1 degree a frame and drifts up to 6 over a second and a half held still), then it turns from there without a jump. Within 2.5 degrees of level it snaps level. Let go and it keeps the turn; the next grab starts from it. Corner and tab grabs only resize (letting go with one of two hands, the other's turn starts afresh). HANDS_GRAB_ROTATE=0 keeps the old facing-only carry. A panel held near the eyes by a point far from its centre (closer than twice that distance) faces you from a point between the hand and its centre: facing from the centre turned it about the hand, which moved the centre, which turned it, and 0.6 m panels held by an edge at arm's reach spun and flipped between frames on the third live test's replays.
• Two hands: both hands holding one panel (pinches or fists) stretch it (its width follows their distance, past a 1 cm dead zone) and turn it with the line between them on every axis: swing the pair round (yaw), raise one hand (roll), bring one nearer, and tip both wrists the same way about the line between them to pitch it (the mean of the two wrists' turns about that line, past HANDS_GRAB_ROTATE_DEADZONE); it snaps level within 2.5 degrees. HANDS_STRETCH_TURN=0: about the vertical only (before 2026-10-10's third live test). Let go with one and the other carries on. The second hand doesn't have to find a grab zone (since the fourth live test, where it clicked the content instead): while one hand holds a panel, the other's pinch or fist anywhere on it, up to HANDS_JOIN_REACH (0.15 m) in front of it, 12 cm behind or 10 cm round it, joins as the second anchor; it never clicks, pokes, pushes or flicks a panel the other hand holds. Replaying the user's two-fist flags (215945 and 215955) with the panel where the live log had it, and 4 cm nearer or 4-8 cm farther: before, a click and no two-hand hold; now both hands hold it 9.4 s, stretching and turning it. A carried panel (one hand or two) turns at most 6 degrees a tick about the hand's point (a fist pulled in toward the face flipped it between ticks in a replay).
• Holding through tracking loss (2026-10-10: losing the hand dropped what it held): a pinch, card or fist grab, and a poke's or pinch's press, last until a clear release: the hand seen with its pinch or grip up and its fingers open (thumb to index tip over 3.5 cm; a fist's fingers over 1.35 extended) on 2 hands frames in a row (67 ms), or up for 6 frames whatever the fingers say. A pinch that flickers up for a frame or two in fast motion, or that ft-hands ended as lost while the hand comes back still pinched, holds on. Unseen for HANDS_HOLD_LOST_MS (1.75 s), it lets go where it is. Meanwhile the panel coasts on the hand's last velocity, slowing to a stop over 150 ms; a frame whose point jumped more than 12 cm counts as unseen (up to 2 in a row). When the hand is back, the smoothing restarts moving at the gap's average velocity and the panel blends to the hand over 100 ms (longer when that's more than 0.6 m/s on average). Offline, carries at 0.6 m/s losing the hand for 3, 6 and 10 frames hold, the panel's largest step 12-17 mm a 90 Hz tick (6.7 moving). A hold also lasts while the pointer helper's word says that hand is "off" (not a controller's) for up to HANDS_HOLD_LOST_MS: the word flickered off 24 times in the third live test's 7 minutes and dropped three holds.
• Tracking no hand makes (HANDS_TRACK_GATE, 2026-10-10: "grabbing the side of the monitor, my hand moves off to the left and tracking gets lost: the grabbed monitor is jittery"): on that flag the hand, 45-60 degrees off the head's forward, read alternately at two places 10-20 cm apart and its frame flipped 100-180 degrees between frames. A holding hand's frame that moves faster than 4 m/s from the last one kept, or lands more than 0.5 x 100 m/s² x dt² + 2 cm from where its velocity had it, is dropped (three in a row at one new place: it moved there, and the panel blends over); its frame turning faster than 540 degrees a second likewise. Frames seen by one camera, or 50-80 degrees off the head's forward, are smoothed harder (the cutoffs down to 0.35 of theirs) and don't turn the panel; past 80 degrees the hand counts as unseen, so a panel carried out of view at the side holds still until it's back. The turn's smoothing follows at most 4 Hz. On the third live test's ten flags, each pinch or fist held 0.4 s or more replayed against a 0.6 m panel at the hand (handinput-test --carries): the carried panel's centre moved with a per-tick second difference of 159 mm rms (99th percentile 898) on hands-throw, 9.7 mm (29) now; its turn 33 degrees rms (187) then, 1.6 (5.2) now; the side-grab flag alone 96 mm (599) to 6.2 (23). All 23 flags: 119 mm (719) to 7.6 (26), 26 degrees (157) to 1.3 (4.8).
• Flick it away (HANDS_FLICK, 2026-10-10): with the hand at a panel (its palm within 12 cm in front of it, over it, nearly still), flick it out fast with the fingers opening: away from you (within 40 degrees) at HANDS_FLICK_SPEED (1.2 m/s) or more, 12 cm or more within 0.2 s, the fingers ending open and opening by HANDS_FLICK_OPENING (0.15 of mean extension; 0 lets an open hand pushed out flick too). The panel goes out along the line from your eyes through it, to HANDS_THROW_DISTANCE (3 m) from them, facing you, over 0.4 s easing out. A pointing hand doesn't flick (its fingers read about 1.3), and an open hand coming through a panel faster than 0.8 m/s doesn't poke it. On the recordings, with a panel 0.3-0.45 m ahead, once in 11 minutes (6 times without the opening rule); none in the 13 "grab that" flags.
• Throw it (HANDS_THROW, 2026-10-10): let go of a panel you carry with one hand (a card, a fist, a pinch on its border, bar or title; not a resize or two hands) while the hand moves away from you (within 60 degrees) at HANDS_THROW_SPEED (1.0 m/s) or more, over the 0.1 s before the fingers opened, and it flies on that way to 3 m from your eyes, facing you. Let go slower and it stays. "Grab that" brings a thrown or flicked panel back, and the gesture again sends it back out there.
• Push it away (HANDS_PUSH, 2026-10-10: "pushing against the monitor or window does not make it go away"): an open hand (the four fingers 1.6 extended or more on average, the middle, ring and little too), palm to the panel (within 50 degrees), over it, pushed straight into it (within 30 degrees of its normal) from in front of it or resting on it: at HANDS_PUSH_SPEED (0.5 m/s) or more and 5 cm or more in, or slowly 15 cm past its surface. It goes 3 m out like a flick. A pointing index pokes; a flat open hand (1.8 or more) or an open one coming in at the push's speed doesn't. Not from a hand resting low, typing, pinching or gripping, nor within 0.6 s of a hold or a press. On the recordings with a panel 0.3-0.45 m ahead: probe-moves (open hands pushed and pulled to probe the ray's reach) 2-15 pushes in 3.5 minutes, the 2026-09-29 recording none, lit 1; on 23 flags with a panel at arm's reach, one (both hands pushed out 11-19 cm through it at 0.5-0.6 m/s); "grab that"'s open hand held out to a panel at arm's reach (5-10 cm in at 0.14-0.43 m/s) doesn't push.
• Sent away, a panel keeps its size (HANDS_SEND_AWAY_WIDTH, user, 2026-10-10: "windows should not resize to be smaller when they're sent back"): thrown, flicked, pushed or sent back by "grab that", it keeps its width; one "grab that" shrank to bring it to arm's reach (screens wider than HANDS_GRAB_THAT_WIDTH) and still that wide gets the width it had back (restore, the default; keep: never). In the third live test the 2.5 m desktop came at 0.6 m and was thrown 3 m out at 0.6.
• The target's glow (HANDS_TARGET_GLOW, user, 2026-10-10: "a glow over the window that grab that is targeting"; after the fourth live test: "too powerful. It should be a white glow and far more subtle. It should only be around the outside of the window"): while an open hand is held out to a panel for "grab that", a soft white halo shows outside that panel's edges, about 1.3 degrees wide (1 cm at arm's reach), at most half opaque at the edge and fading out, nothing over the content; it fades in as the gesture arms (0.2 s) and goes as it fires or is let go. An open palm close in front of a panel, toward it, shows the same halo dimmer (a push or a flick would throw it). One overlay (frametop.guide.target) sized and curved to the panel, in its plane, so it shows with the cutouts off.
• Buttons (the fourth live test: "it's extremely difficult to interact with the button to make the curve curve and not"): at arm's reach the curve button is 8 mm across, 3.5 mm past the bar's end, and the bar (first in the list, with a 1 cm margin) took the six pokes the log has at it, 3-5 mm left of its centre. Now the nearest control wins, the round buttons (curve, reset, dock, close) take a fingertip or a pinch at least HANDS_BUTTON_REACH (0.015 m) each way from their centre, and a pinch there presses the button (HANDS_PINCH_BUTTONS; 0: the bar's grab zone takes it, as before). A hand at one lights it and shows a soft ring round it. The log names the control pressed.
• Glows at corners, edges and buttons (HANDS_EDGE_GLOW, user, after the fourth live test: "almost like a cutout of the soft glow we had for highlighting, just around the edges and corners when you're within the interaction zone"): as a hand comes to a panel (in its touch zone), the same soft white halo shows only where a pinch would take it: an L round the corner it would resize from (along both edges from it), a short segment along the edge it would move the panel by, or a ring round a button; brighter as the fingers come closer, brightest while resizing from that corner or pressing that button. One overlay per hand (frametop.guide.hand.left, .right), so it shows with the cutouts off. With it the resize handle no longer follows hands (lasers and the gaze keep it); HANDS_EDGE_GLOW=0 brings back the handle at the hand's corner and the cutout buffer's rings.
• Remote displays take all of it as screens do: pokes, pinches, the card and fist, corner resize, throws and "grab that".
• Smoothing: a grab carries the knuckles as the cameras saw them, smoothed (include/fh_smooth.h: One Euro in continuous time, read each frame ft-screens draws), not the point moved ahead along the hand's velocity as the cutouts use: that velocity is mostly noise times the lead. It moves the panel only once it's 1 cm from where the pinch began, then carries on from there (no jump), so a still hand leaves the panel still. The head the panel turns to face is smoothed too. HANDS_GRAB_SMOOTH_MINCUTOFF (1 Hz), HANDS_GRAB_SMOOTH_BETA (15 Hz per m/s), HANDS_GRAB_DEADZONE (0.01 m). On the probe-moves replay (scripts/test-handinput.sh, 90 Hz ticks), the carried point's per-tick second difference went from 13-17 mm rms (50 mm p99) to 2.3-2.6 mm (11 mm), its still wobble from 1.3-1.5 to 0.8-1.0 mm, and it trails the capture by 85 ms while moving (it was 10 ms ahead, by the prediction, and 18 mm off any delay).
• With the cutouts on, a glow in the cutout buffer shows where the fingertip is on the screen (a ring that tightens as it comes to the surface, a dot while pressed); what a pinch would do where the fingers are shows as the soft white glow (above) either way; the bar and the buttons brighten under a pinch or a fingertip.
• It takes the helper's word on which hands are bare and whether hand input is allowed ("hands"); without that word for 3 s, near input is off. The helper sends it with POINTER_HANDS=1. A hand typing holds back (the helper's "typing" word, POINTER_PINCH_TYPING above) doesn't poke, grab, move the pointer with its fingertip, or make "grab that" (below). HANDS_NEAR (1): on has the helper send it without POINTER_HANDS too (direct touch alone, no pinches for the pointer), 0 turns it off, force turns it on without the helper (testing). ft-screens command handinput on|off|force|state.

The palm menu (screens/palmmenu.cpp, frame-hands notes/hand-interaction.md sections 11 and 18), like the Vision Pro's home indicator: turn a bare hand palm up toward your face, open, and look at it, still. After 0.3 s a small ring (2.5 cm) shows 4 cm above the palm and follows it. A plain thumb-index pinch tap by that hand opens the Steam menu (steam/ft-steam menu, what the input relay's steam_menu runs), or closes the SteamVR dashboard if it's open. The ring turns blue while the pinch is held, and the release does it; a pinch that's lost, or let go after the hand turned over, does nothing. Turn the hand away for 0.15 s and the ring goes.

• Palm up means: the palm's normal within 45 degrees of the line to your head (it leaves past 60; 60 and 75 before the third live test), the palm within 45 degrees of where your head points (leaves past 55), 15-70 cm from the head, the middle, ring and little fingers open (tip-to-wrist over knuckle-to-wrist 1.5 on average, leaves under 1.3), and the palm moving under 0.25 m/s. A palm up in your lap while you look at a screen doesn't count. The hand ray needs the palm turned away, so the two never overlap.
• Not by a panel, nor busy (the third live test: the Steam menu opened as the user took a panel's bottom edge from below, palm up, fingers bunched): no ring while the hand is near a panel (in its touch zone or margin, or its fingertip, fingers or palm within 12 cm in front of one and 8 cm round it), while near input presses or holds with it or "grab that" has it, nor for a second after any of those. The pinch must be a plain thumb-index one by a still hand: the middle finger out of it (its tip 4.5 cm or more from the thumb's, the finger 1.35 extended or straighter); a card's index and middle or the bunched fingertips do nothing on the ring.
• While the ring shows, ft-screens reports that hand near ("near"), so the pointer helper leaves its pinches alone (no gaze click), and near input leaves it too. A pinch that began before the ring showed stays the helper's.
• Where: a bare hand (not one holding a controller), with hand input allowed: on the desktop, and during a VR game while its dashboard is open or Frametop's panels are turned back on. So in a game it closes the Steam menu it opened; with the dashboard closed, the game keeps your hands. HANDS_PALM_MENU (1): 0 turns it off; ft-screens command palmmenu on|off|state.

Grab that (screens/grabthat.cpp, 2026-10-10): hold an open hand out toward a screen, a remote display or a floating window, palm to it, then pull the arm back while making a fist, and the panel comes to arm's reach over a quarter of a second: 45 cm ahead of your eyes (HANDS_GRAB_THAT_DISTANCE), at the fist's height (at most 45 cm under the eyes), facing you, so the hands can poke it, carry it and turn it. A screen wider than HANDS_GRAB_THAT_WIDTH (0.6 m; 0 keeps the size) comes at that width; floating windows keep their size. It doesn't take a panel within arm's reach (below: Which panel); with HANDS_GRAB_THAT_FROM=0, as before the fourth live test, the same gesture at a panel that came that way and is still within arm's reach (its centre within 0.8 m of the eyes, wherever you moved it about there) sends it back where it was (and its width), and at a panel within arm's reach that didn't come that way, it centres it at arm's reach. One at a time: a panel that came and is still where it came goes back as another one comes, so they don't stack up (three remote displays did in the live test of 2026-10-10). The reset button sends floating windows back too (the layout reset puts the screens and remote displays back itself). With the panel that near, the open hand held out to it is in its touch zone: its fingers pointing up don't poke it, and its fist or a pinch as it closes waits on the gesture (above).

• The gesture, from the user's own example (flag 2026-10-10 15:01): the fingers open (each tip from the wrist at least 1.55 times its knuckle's, on average), the palm facing ahead (its normal's forward part at least 0.45), the wrist at least 28 cm ahead of the eyes, all for 0.2 s; then within 0.45 s of the hand last held out: the middle, ring and little fingers close with the palm still toward the panel, and on one frame a fist (under 1.0), the wrist 8 cm or more back from its farthest in the last half second, the palm turned away from the panel. Only two-camera frames count. Not again for a second.
• Which panel (since 2026-10-13, user: "The glow should be where the palm is facing"; the line below lit panels off to the side of a palm that wasn't facing them): the one the open palm faces, the line out of the palm along its normal, with the margins below seen from the palm (HANDS_GRAB_THAT_AIM=palm; eyes brings back the line from the eyes through the palm). The palm has to face what it takes: in the user's example below the palm was held half down, 41 and 51 degrees off the eyes' line, and that line took both fires where the palm's takes none. With eyes (since the fourth live test, user: "make the trigger more precise and only activate if the window is not already held and closed up"): the one the line from your eyes through the open palm meets (HANDS_GRAB_THAT_MARGIN, 0: the panel itself; 10 cm or 3 degrees and a 20-degree head cone before), the nearest; else, since the fifth live test ("it couldn't be pulled back in": the palm is held out under the line of sight, so the line runs 12-15 degrees under where the head points and passed just under a desktop thrown 3 m out), the one whose outline it passes nearest within HANDS_GRAB_THAT_MARGIN_DEG (10 degrees since 2026-10-11, seen from the eyes; a "no panel there" log line names the nearest and how far off), curved ones on their curve, farther than HANDS_GRAB_THAT_FROM (0.8 m: its centre from the eyes). Pinned panels, panels held by a laser or a hand, and panels a hand presses on don't come. A panel at arm's reach is the hands' already: push, flick or throw it to send it away (HANDS_GRAB_THAT_FROM=0 brings back the gesture sending a near one back, or centring it). The arming hold has the palm facing along that line within HANDS_GRAB_THAT_FACING (80 degrees since 2026-10-11) on average and still (under 0.5 m/s over the 0.2 s), else it starts again; the user's example holds 41 and 51 degrees off, at 0.25 and 0.46 m/s. For a second after a panel is sent away (a push, a flick, a throw) the gesture doesn't fire: the push's open hand pulled back closing has its shape.
• The closing hand is ours: from when its fingers close (the thumb clear of the index tip, so not a pinch) until 0.4 s after it fires, ft-screens reports it near, so the pointer helper doesn't take the fist as a gaze grip. While a hand is held out it asks ft-hands for its full rate (want-hz).
• Offline (scripts/test-handinput.sh and handinput-test --grabthat FLAG_DIR..., which also replays each flag through near input with a panel at arm's reach): the example flag fires twice, right hand, at -11.35 and -5.89 s (the user's -11.4 and -5.9), and twice at a panel 1.6 m out along its aim (with a panel 0.35-0.45 m ahead, where the gesture brings one, it doesn't take it since the fourth live test: the closing fist grabs it); the other 12 flags and 15 minutes of recordings (probe-moves, the 2026-09-29 and lit recordings, the PR #3 takes) none; 1 of 143 pinches and 3 of 12 grips there began on a hand it claimed. Synthetic waves, pushes, fists without the pull, a slow pull, a hand turned over before closing, pinches held out, a hand dropped to the lap and typing don't fire.
• HANDS_GRAB_THAT (1): 0 turns it off; ft-screens command grabthat on|off|state.

The "approach from front" rules, on recordings: in the hand recorder's desk take (PR #3, with head poses), typing under panels 0.30-0.40 m ahead (at eye height, or 0.45 m low and tilted up) made no press; the hand on the mouse (0.37-0.40 m out, past the resting zone) made 2 presses in 13 s under the nearest ones. In its touch-the-dot take the fingertip stopped 0-2 cm short of the dot in the air, its last bone 44-63 degrees off the line from the head: with a panel 3 cm nearer, 2 of 6 touches pressed at 50 degrees, 3 at 65. HANDS_POKE_REST_NEAR=0.42 takes the mouse hand out too, but then a low dot 25 cm ahead no longer takes a poke.

Offline, scripts/test-handinput.sh [DUMP...] runs its unit tests and plays recorded hands (ft-handreplay --publish-dump) against panels placed around them. On the probe-moves (v4), lit and dim recordings, in 1 s and longer stretches where a hand held still and pointed (184 s in all; the fingertip still moved 19-23 mm in depth within each, median), a panel 3 cm beyond the fingertip got no press, 2 cm one (in 42 s of probe-moves), 1 cm 7, 0.5 cm 18. frame-hands notes/hand-interaction.md section 10 has the rest.

Rendered hands

ft-handview (screens/handview.cpp, built by screens/build.sh) draws each tracked hand: when Room View is off, and with HANDS_VIEW_ROOMVIEW=1 (the default) when it's on too, floating over the cutout's hole (the tracking is good enough to look right on its own; 0 draws nothing with Room View on, as before).

The model (HANDS_VIEW_MODEL): asset (the default), the WebXR Input Profiles' generic-hand (MIT), a rigged low-poly hand of 1360 vertices and 2314 triangles per hand with the 25 WebXR hand joints as its skeleton; or babylon (Babylon.js's hand meshes, CC BY 4.0, 1808 triangles), mrtk3 (MRTK3's OpenXR hand, BSD-3-Clause, 2794), godot (Godot XR Tools' hands, CC0, 3052) or mrtk2 (MRTK2's rigged hand, MIT, 2664). All are in screens/assets/hands/, each with its source, version, licence and conversion in a README (the MRTK ones were FBX, converted with assimp and screens/tools/hand-glb-clean.py; MRTK3 has only a right hand, mirrored for the left). screens/tools/hand-asset.py maps each one's skeleton onto WebXR's 25 joints by name (MRTK3's names sit a joint along; MRTK2 has no metacarpals but the little finger's, so the others go a third of the way to the knuckle, unweighted; extra joints such as a root or a palm give their weights to the wrist) and turns them all into screens/handasset.h (quantized vertices, normals, four joints and weights per vertex, the triangles, the joints' bind pose), compiled in, so there's no glTF code at run time. generic-hand's fingertip joints sit about a centimetre inside its fingertips, so its fingers reach about that far past the tracked tips; Babylon's and MRTK2's end within a few millimetres of them. screens/handskin.cpp poses it: each of the model's joints gets a frame, the same way in its bind pose and on the tracked hand (version 2's fitted skeleton, else the landmarks): its bone's direction toward the next joint, and a roll from the palm (across it for the fingers' first bones, its normal for the thumb's; each bone after takes its parent's sideways axis, the axis a finger bends about). WebXR's metacarpal joints, which MediaPipe doesn't have, follow the palm (wrist, index and little knuckles). The model is scaled by the palm's size and each bone along its length to the tracked bone (0.6-1.6 times). A left hand is drawn with the model's left hand, so nothing is mirrored. The vertex shader blends up to four joints' matrices per vertex (linear blend skinning on the GPU); the CPU only works out the 25 matrices and skins a third of the vertices to place the hand's square. The model ends just past the wrist: its heel fades out over the 3 cm before the tracked wrist, and the rest past it isn't drawn. That leaves it open there: with the hand pointing away from you, you'd look into it (a hollow tube, a fist's fingertips through the palm). So it's capped: a stencil pass counts the surfaces along each view (+1 for each front face, -1 for each back face; the GPU's early depth test would drop hidden ones, so it's a pass of its own with the depth test off), and where one more back face than front says the view went in through the cut, a disc where the fade begins (an ellipse fitted to the skinned points there) covers it, solid in the middle and fading toward its edge. Godot's hand has a longer, wider wrist, so more of it is faded, and from above a hand pointing away shows a darker patch there. HANDS_VIEW_MODEL=robot draws a robot hand built in code from the tracked skeleton (screens/handrobot.cpp; no asset, so no licence): rigid pieces, each fixed to one bone, so nothing bends or stretches out of shape. Each finger bone is a light shell (a tapered box with rounded edges and ends) with a gap at each joint; a dark ball sits at each knuckle and at the thumb's base, a dark hinge (a short cylinder along the axis the bone bends about) at the other joints. The palm is a dark core between a palm plate and a back plate (a dark seam shows round its sides, a panel line across each), ending in a short closed cuff past the wrist with a dark plate in its end. Small accents glow cyan: a pad on each fingertip, a line across the back of the knuckles, a dot in the palm, a ring round the cuff. 2373 vertices and 4360 triangles, one mesh for both hands, built once as a right hand on a reference hand (handskin::DemoPose's open one) in each segment's frame: the frames are handskin's (handskin::TrackedFrames: the bone's direction and its roll from the palm, by the models' rule), a left hand's mirrored from a right one's, so the same mesh draws the left mirrored. Posed, each of its 16 segments (the palm, three thumb bones, three for each finger) is scaled by the palm's size against the reference's, and the palm's width by the knuckles' span; each vertex is fixed to its bone's start or end, so a shell's rounded ends keep their shape, its straight middle takes up the tracked bone's length, and a fingertip ends at the tracked tip. The girth follows the palm's size, not ft-hands' joint radii (rough, mostly its defaults). The vertex shader applies one segment's matrix per vertex; the CPU works out 16 matrices and poses a quarter of the vertices to place the square. Nothing fades and nothing is open, so there's no cap pass. In solid the shells are matte ceramic white and the joints dark grey with a sharper highlight; in ghost the shells are pale and see-through (30%), the joints darker (50%), the accents nearly solid, with the same fresnel rim as the models. No free robot hand asset fit (2026-10-09, screens/assets/hands/README.md). HANDS_VIEW_MODEL=capsules draws the older hand instead: the cutouts' mesh matte with no margin (screens/handmesh.cpp), a tube per finger bone, the palm's slab, the thumb's web and the forearm's stub, fading out over the 10 cm behind the wrist.

The look (HANDS_VIEW_STYLE): ghost (the default) is a light, see-through hand (about a third opaque) that glows white toward its edges, a soft fresnel rim where the surface turns away from you (no drawn outline, so a finger over the palm shows only its own soft edge), with the fingertips a little brighter; solid is a matte light grey hand, nearly opaque, lit from above (wrapped, so the underside is dimmer, not dark) with a soft cool rim light, easier to see over bright games. Both shade only the surface nearest the eye (the body pass keeps depth), so a see-through hand never shows its inside or its far side. model <name>|capsules, style ghost|solid and inroom 1|0 on @ft_handview switch them live until a restart.

Each hand is an overlay of its own (sort order 100, over the panels and the dashboard): a square placed at the hand, facing your head, side by side, each half what one eye sees of the hand projected onto the square. The compositor keeps a placed overlay still in the room as your head turns, so a hand only waits for its own motion; a head-locked layer would have to be redrawn at the display's rate to stay put. It's drawn for each new hands frame and, while the hand moves, at most every 20 ms in between (moved ahead like the cutouts), on the GPU: the body into a scratch buffer (and the model's cap), then the colour into a DMA-BUF SteamVR imports. The --demo hands (two hands 45 cm ahead, drawn 30 times a second each) cost 0.5-0.7 ms of CPU per hand per draw (0.3 of it the mesh) and about 4.5% of one core in all on 2026-10-09 with the capsules; idle it's 0.2%, and with Room View on and HANDS_VIEW_ROOMVIEW=0 it wakes 10 times a second. Offscreen (--snapshot, two waving hands at 30 draws a second each, no SteamVR, capped to one core by frame-job, 2026-10-09) the model took 0.36-0.40 ms of CPU per draw (0.08-0.09 of it posing and placing) and 2.2-2.4% of a core, the capsules 0.55-0.62 ms (0.27-0.34) and 3.2-3.5%. With the fresnel look and the cap (the same day, the user in the headset with SteamVR running): asset 0.34 ms of CPU against 0.32-0.33 before, and 0.61-0.62 ms of waiting for the GPU per draw (each frame's glFinish, shared between its two draws) against 0.57; the other models 0.27-0.41 ms of CPU and 0.61-0.68 ms of GPU wait, the capsules 0.44-0.45 and 0.53-0.54. The robot (2026-10-09, frame-job at 4 cores, the same run as the others' numbers here): 0.21-0.24 ms of CPU per draw (0.04-0.05 posing and placing) and 0.55-0.57 ms of GPU wait, against asset 0.28 and 0.66, babylon 0.26 and 0.66, mrtk3 0.25 and 0.66.

Between frames and past them (screens/handhold.h, 2026-10-13, user: "implement interpolation for the rendered hands so that they feel smoother", and "if the hand is lost it's preferable the rendered hand display freeze instead of make impossible movements, spaz out, or vanish until a few seconds have passed"). The tracking comes about 30 times a second. Each new frame blends in, joint by joint, from the pose drawn now over about the time since the frame before (15-70 ms), drawn every 15 ms meanwhile. A hand the tracking loses stays where it was last drawn, still, for HANDS_VIEW_HOLD seconds (3), then goes. A frame whose palm is further from the last one taken than a hand moves (15 cm, or 6 m/s over the time since; the lazy susan's strokes peak at 4.5) or whose palm turned over (100 degrees) in under 0.2 s isn't drawn until two frames in a row agree with each other by the same test (or 0.25 s passes): the drawn hand holds still meanwhile, then goes straight there. A hand first seen, or back after a loss, also waits for two frames that agree, about 33 ms. A hand taken off for a controller, Room View or a lost head pose goes at once. HANDS_VIEW_SMOOTH=0 draws each frame as it comes and a hand only while it's tracked, as before. The status line counts the losses held and the jumps held off.

--snapshot FILE.png draws demo poses of both hands offscreen (open palm toward you, palm away, palm up, fist, point, pinch, spread; each row the left hand's two eye halves over dark, the right's over light) with no SteamVR or headset, then the cost loop above (and how long the GPU took after it): for checking the look and the skinning after a change. FT_HANDS_VIEW_MODEL and FT_HANDS_VIEW_STYLE in its environment pick what it draws.

A hand holding a controller isn't drawn: the pointer helper sends ft-handview its word ("hands L R allowed" on @ft_handview, as to ft-screens); while the helper is silent (3 s), a hand within 15 cm of its side's controller counts as holding it. The hands draw in VR games too (hand input keeps its own rules), but only while hand tracking runs: the relay parks it while Frametop is paused for a game, unless the dashboard is open.

Room View (screens/roomview.h): SteamVR has no public "is Room View on". In a VR game, the compositor's VREvent_RoomViewShown and VREvent_RoomViewHidden tell it; with no game, the construct shows Room View when camera/enableConstructRoomView is on (the default on the Frame). The passthrough shared state (VR_CameraPassthroughState, read only) has an enabled byte; it read 1 on 2026-10-09 and isn't yet known to follow the dashboard's Room View button, so only a 0 counts (off). HANDS_ROOMVIEW=auto|on|off in ~/.config/frametop.conf forces it for both programs.

screens/build/ft-handview [--status S] [--res PX] [--demo] [--seconds S]   # in the dev container (scripts/in-box)
screens/build/ft-handview --snapshot /tmp/hands.png [--res PX] [--seconds S]   # offscreen, no SteamVR
echo -n state | socat -t 1 - ABSTRACT-SENDTO:ft_handview,bind=   # the status line; also: roomview auto|on|off, model, style, inroom
screens/tools/hand-asset.py   # screens/handasset.h from screens/assets/hands/*.glb (--check: is it up to date)
screens/tools/hand-glb-clean.py # a converted (FBX -> glTF) hand rewritten for hand-asset.py (screens/assets/hands/*/README.md)

screens/frametop-handview.service runs it as a user service (--status 60), started and stopped with SteamVR; the installer doesn't install it on its own.

Camera check

Hand tracking needs all four mono cameras and the headset's IR light. With the Arcturus colour module attached, SteamVR's XRService loads an FPGA image ("VCINT") onto the module every time it opens the cameras: when SteamVR starts and after every wake. When that load fails, XRService runs only the two side cameras, the frames come out darker and noisier, and ft-hands finds no hands at all. It happened on 2026-10-02 at 17:02, after the headset slept; the hand recorder then said "I can't see your hands" for a whole session.

hands/camcheck.py (system Python, standard library) tells whether the four cameras run: ok, degraded: upper cameras and IR light off (VCINT FPGA failed to load) (or another degraded: reason), or unknown (SteamVR not running, the cameras closed while the headset sleeps). It prints the log lines and other evidence it used; --json is for programs; the exit status is 0, 1 or 2. It reads:

• the running XRService's log (~/.local/share/Steam/logs/xrservice.txt): the last camera start (from the FPGA check to the next "Closing tracking camera interfaces"), its VCINT result, Upper cameras FPGA interleaving support: N, Created N tasks (T tracking, P passthrough) and the TrackingCameraInit lines. A wake that works prints no "Created N tasks", so an older one doesn't count;
• which /dev/video* XRService has open (/proc/PID/fd; video9 and video13 are the side pair, video6 and video7 the upper pair). Only on the host: the dev container can't read another process's open files, so there it's skipped;
• ft-camd's ring header, when it runs: how many mono cameras it publishes.

The hand recorder runs it before a session (DESIGN.md, "Camera check"). hands/tests/test_camcheck.py runs it on the 2026-10-02 log cut at several points, and on made-up logs.

The watcher (off by default)

hands/ft-camwatch follows the XRService log (a stat every 2 s, reading only what's new). On a VCINT failure it posts a notification in the Frametop desktop, on the desktop's own D-Bus, found through its plasmashell as decoration/apply.sh does. With CAMWATCH_AUTO_RESTART=1 in ~/.config/frametop.conf it also restarts SteamVR, but only:

• while the headset isn't worn (frame-job's check: vrcompositor runs and a /sys/class/backlight/*/brightness is over 0), and after it has been off for CAMWATCH_IDLE_S (60);
• with no app Steam launched (SteamLaunch AppId=N in a process's arguments; CAMWATCH_IGNORE_APPIDS lists ids that don't count) and nobody on the remote desktop (an established connection to the VNC port, VNC_PORT, 5900);
• once per failure, and not again within CAMWATCH_COOLDOWN_MIN (30) of the last automatic restart. It remembers both in ~/.local/state/frametop/camwatch.json, so its own restart doesn't reset them.

It logs every decision to the journal. ft-camwatch --once prints the state and what it would do, and does nothing; --dry-run keeps watching without acting. hands/frametop-camwatch.service is the unit (a template, @REPO@ as in the others; nothing installs or enables it yet). It isn't PartOf=steamvr.service, so it outlives the restart it asks for. CAMWATCH_NOTIFY=0 turns the notification off. hands/tests/test_camwatch.py tests its decisions with made-up inputs.

What a SteamVR restart does to Frametop

Read from the code on the experimental branch, not tried live:

• ft-screens quits when SteamVR does: on VREvent_Quit it ends its Wayland display (screens/vr.cpp, ft_vr_poll; screens/compositor.c, handle_vr_event). It never connects to SteamVR again: ft_vr_init runs once, at its start.
• KWin runs nested in ft-screens, so the Frametop desktop ends with it, every window in it too (the hand recorder's as well). Its unit, frametop-desktop, is a transient systemd-run unit with Restart=no, so the desktop doesn't come back by itself: start it again (Desktop in the library, or desktops.sh start).
• When the unit stops, systemd ends what's left in it. session/keep-apps.sh moves programs started in the desktop out of the unit first, but only desktops.sh stop runs it; here they stop too. Programs in the dev container (ft-screens, the hand recorder) are in the container's cgroup and end when their Wayland connection goes.
• The units that are PartOf=steamvr.service restart with it: frametop-camd, frametop-hands, the pointer helper, gaze and power, and the hand recorder's own transient ft-camd and ft-hands units.

Verified, and what's a guess

Verified, from the XRService logs of 2026-10-01 and 2026-10-02 and the running system:

• The failure's log lines and its effect: "Failed to load VCINT FPGA image when passthrough cameras are connected", interleaving support 0, "Created 4 tasks (2 tracking, 2 passthrough)", and only video9 and video13 opened. At 19:38-19:59 XRService held only those two of the four (plus video0 and video3), and ft-camd published two mono cameras.
• A wake's load can work and can fail. Both wakes in the logs started from an FPGA that answered nothing ("ERROR/UNKNOWN"): the one at 2026-10-01 16:39 loaded VCINT, the one at 2026-10-02 17:02 failed ("FPGA config_done signal did not assert").
• After a reboot the FPGA reads PASSTHRU and SteamVR's start loads VCINT (2026-10-01 21:53, 2026-10-02 13:39).
• A SteamVR restart within a boot found VCINT still loaded and loaded nothing (2026-10-01 15:27): XRService checks the FPGA when it starts and loads only when it must.

Guesses, not tested:

• Whether a SteamVR restart fixes it. After a failed load the FPGA doesn't answer, so a new XRService would run the same load a wake runs, which has worked once and failed once. It's never been tried after a failure. If it doesn't help, only a reboot is known to work (the FPGA comes up as PASSTHRU, and the load at SteamVR's start has worked both times).
• That the IR light is off because of the FPGA: the frames are darker and the illuminator ring isn't seen, and the FPGA loader lists a room_led_en pin, but nothing shows the light's state directly.
• That a sleep and wake (taking the headset off long enough) would retry the load too: it should, since every wake loads VCINT, but no failure has been followed by a wake yet.
• How the Frametop desktop behaves on a SteamVR restart (above): read from the code only.
• That Steam-launched apps carry SteamLaunch AppId=N: from Steam on other Linux systems; no VR game has run on the Frame to confirm it.

Build

hands/build.sh builds in the dev container on the Frame, into hands/build/, with hands/Makefile. The first build fetches ncnn at a pinned tag (NCNN_TAG in the Makefile) and builds it into hands/build/ncnn, which takes a few minutes; NCNN=DIR points at an ncnn install already built instead. ft-camd is linked statically, because it runs on the host, which has an older glibc than the container.

Known issues

• The side cameras could come out swapped (fixed 2026-10-05, see "Which camera is which"). If they still do, ft-hands corrects it from the hands (HANDS_SWAP_SIDES=auto, the default): until it has seen about 1-2 s of hands in both namings' reach, the cutouts may sit beside the hands. ft-camd's log says bound by VIDIOC_QUERYBUF for each camera when its buffers were matched exactly.
• The colour cameras can't be used while the headset is worn. The colour module then writes only a half-size image into the top-left quarter of its buffers, and ft-camd drops those frames. So the service runs the mono cameras only, and tracking in bright light, where the mono cameras see dark hands, doesn't get the colour pair's help.
• The colour calibration mapping isn't settled. Which colour camera is passthrough_left (HANDS_COLOR_LEFT) and how the module's crop applies (HANDS_COLOR_CROP) still need tools/check_color.py on a recording with a lit, textured view.
• Depth when one camera loses the hand. A hand seen in one camera drifts 10% per update toward the one-camera depth guess (kMonoDepthGain, 0.1, in track/tracker.cpp). In the 2026-09-30 replays that was worse than keeping the last distance (see "3D" above). A smaller gain, such as 0.02, is the next thing to try.
• Pinches aren't reliable enough for everyday use yet. That's why hand tracking is off by default (HANDS_MODE=off: only ft-handsctl on starts it), and POINTER_HANDS is 0 by default.
• SteamVR can leave the upper cameras and the IR light off after a wake, and then no hands are found. See "Camera check" above: camcheck.py tells, the hand recorder won't start a session, and the fix is a SteamVR restart or a reboot.
• The skeleton (2026-10-08) is tested on recordings only, not in the headset: the cutouts now follow it (version 1's points and capsules), so check them there; HANDS_SKELETON=0 brings back the filtered landmarks.
• Floating windows don't get hand cutouts or glows. Their panels show crops of the client buffer, which the cutouts' side-by-side buffer doesn't match (screens/vr.cpp, UpdateCutouts). Near input works on them.
• The matte and near input are tested offline only (2026-10-08): HEADSET-TESTS.md has the first headset tests.