Consent 2026-10-03, after a non-lawyer review: who runs this and how to reach
them, the dataset is public (Hugging Face, possibly abroad), the Hugging Face
username shows next to the contributor id, purposes (no identification), safety,
the maintainer grant passes to whoever maintains Frametop next, withdrawal
before and after merge, rights such as the GDPR's, and what a new version means.
Residents of Illinois, Texas and Washington can't take part for now (biometric
privacy laws): a third checkbox, profile consent.region_ok, checked by
validate.py from this consent version on. The DRAFT banners are gone, so uploads
no longer need FT_HANDREC_ALLOW_UPLOAD.
hands/rec/install.sh installs the recorder on a Frame with Frametop: container
packages, hand tracking and panel builds, ft-camd's capabilities, menu entry.
test_qml_backend also checks each call's argument count against the slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the checklist says no controllers, exported poses.jsonl has left and right
null and feedback lines carry no controller state: controllers left switched on
still get tracked (one wandered 2 m in a real session) and would read as the
hands' ground truth. Poses and live-tracker feedback inside deleted ranges are
left out too, as the images there are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sets.bin's capture_ns is CLOCK_MONOTONIC_RAW; poses.jsonl and prompts.jsonl are
CLOCK_MONOTONIC. On 2026-10-03 the two were 0.80 s apart during a session and
1.11 s apart five hours later, so images can't be paired with poses by
capture_ns. take.json now samples RAW minus MONOTONIC as each recording part
starts and stops (as ft-hands' raw_minus_mono_ns), so readers can put each
exposure on the poses' clock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Upload page is now three numbered steps: choose the export, log in to
Hugging Face, upload. Log in runs hub.py login, huggingface_hub's browser
login (OAuth device code, as hf auth login does): the link opens in the
browser and the page shows the code to enter, with Copy code and Cancel.
hub.py saves the token; the window never sees one, and nobody pastes one.
The terminal upload and the login command are gone from the page, and
UPLOAD.md is now a short 'About uploading' under the steps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frametop's Konsole sets XDG_RUNTIME_DIR=/run/user/UID/frametop, where podman finds
no container state, so 'distrobox enter dev -- hf auth login' failed with a crun
error. The command the Upload page shows now sets the real runtime folder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
af2ea7c put _stay_awake between exportSession and its @Slot, so the
window's Export button called a method QML couldn't see. A new test checks
every backend call in main.qml against Backend's slots and properties.
Export and Upload now say Exporting…/Uploading… with a spinner the moment
they're pressed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Export and upload are done in the headset now: the export page only notes
that VR may stutter a little. Upload opens the pull request first (a
draft) and shows its link, telling the person to plug in the headset and
leave it until it says Uploaded; the files then go to refs/pr/N, and the
pull request is marked open at the end. A retry of the same export goes on
in the same pull request. While an export or upload runs, a host unit holds
a logind sleep inhibitor so the Frame stays awake with the headset off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dot is the current tracker's distance guess, often wrong; the ring is
where the hands should be.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host-spawn starts a host command in the caller's folder. Started from /tmp,
the app's folder in the container is /run/host/tmp, which the host doesn't
have, so starting ft-camd (and every other host command) exited 127.
host_command now runs from home (env -C), and the launcher cds there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The checklist page measures the light when it opens, starting ft-camd if
nothing runs it (and stopping it on quit), instead of saying the cameras
aren't running. The round's lighting defaults to what the cameras measure:
daylight or indoor, from the mono cameras' ambient infrared. Lamps give off
little infrared, so dim and normal rooms read alike; picking dim, room or
daylight still overrides it. session.json gets source, measured and
ambient_ir.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vrserver sends a controller's state only when something on it changes, and one lying still
or asleep may not even send its first one. The check subscribed to one controller and failed
after 3 s of silence. It now subscribes to all, and silence after a good handshake is a skip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gaze service ran ft-gaze and our own eye tracker all the time: with gaze mode off, ft-eyes
still took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. Now ft-gaze
and our tracker run only while gaze mode is on and someone wears the headset, while a check or
the calibration is open or asked for, or under a "wake" lease, which the Gaze page of Frametop
Input Settings renews while it's open. 30 s after the last use they stop, and the frame grabber
idles with our tracker.
- The pointer helper answers "gaze ? headset" with worn|away (SteamVR's activity level for the
headset); an older helper answers it as before, and the service then goes by gaze mode alone.
- A quick check, calibration, or fit check asked for while idle wakes the tracker and opens once
it sends; the automatic calibration waits quietly while it starts.
- Status has "awake" and "idle" (why), and the Gaze page shows it.
- gaze/test/idle-test.py runs the service with a fake helper and ft-gaze, offline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-camd tells the side cameras apart by XRService's buffer allocation order,
which some XRService starts reverse; both of 2026-10-02's starts did, so the
cutouts missed the hands. HANDS_SWAP_SIDES=auto (the default) has ft-hands
vote from hands seen in both side cameras: the landmark rays meet in front
of both cameras only under the right naming. While undecided it probes the
exchanged naming with the landmark model. It decides in about 2 s of hands
(right on all 7 recordings replayed), swaps the views in place, and publishes
sides.json. 0 and 1 still force it, with a warning when the hands disagree.
Recordings carry each part's naming and the session's decision; review,
export, validate and ft-handreplay put the names right, and takes.py sides
records a decision by hand.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frametop kept using the headset during games: with gaze mode off, our eye tracker still took
about 60% of a core, remote desktop about 2 cores while on, and KWin kept drawing hidden
screens because ft-screens sent their frame callbacks at 90 Hz. Pausing gives that back, and
resuming brings back only what pausing stopped. It's also a way to keep the gaze service and
our eye tracker off during games, which PR #13 asked for.
Paused (input/game_pause.py, run by the input relay):
- frametop-gaze stops (ft-eyegrab then idles by itself), and hand tracking and remote desktop
stop if they run
- the desktop hides and slows down: ft-screens "pause on" hides every panel and sends KWin a
frame callback once a second; or, with pause_desktop "close", the desktop closes and starts
again on resume
- the relay lets go of the 3D mouse, typing goes to Steam, and mapped buttons and key
combinations do only pause_toggle, steam_menu and commands
Toggled by both thumbsticks clicked together twice (configurable), read passively from
vrserver's web socket (input/vrws.py) so it works in games and takes nothing from them; by the
new pause_toggle action; by input/ft-pause; and, with pause_auto (default on), by a VR game
starting and ending, which the pointer helper now reports ("vrgame 1|0"). Frametop Input
Settings has a Games page for it. update-check.py checks the web socket, and doesn't count a
paused gaze service as failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the relay leaves a node it can't read yet for the next scan (12f2e84, PR #12), it
checks os.access first, and the test's fake /dev/input paths don't exist, so the relay never
opened them and every key check failed. The fake os now says they're readable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second real session took 16 minutes, half of it 36 still poses. Labels
come from the auto-labeller, so what matters is variety, not clean holds.
A sweep step shows a strip of pose pictures and lights one every 4 s while
the hands move slowly near and far; each cue is a prompt event with
"cue": true. The core session is now 15 steps, about 5 minutes recorded.
The pose groups, the one-hand sweeps' groups and the cue order are shuffled
per session, seeded from its id and saved in session.json. A quick round
(--quick, or the checklist's choice) is about 2 minutes for extra lighting.
Touch the dot has 6 dots, the push sections two heights.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the headset wakes, XRService sometimes fails to load the colour module's
VCINT FPGA image; then only the two side cameras run, without the IR light,
and the tracker finds no hands. hands/camcheck.py reads XRService's log, the
video nodes it holds and ft-camd's ring, and says ok, degraded or unknown.
The recorder won't start while degraded (--ignore-cameras overrides it), offers
a confirmed SteamVR restart, and stops the first hand-size step when the
tracker sees no hand at all. ft-camwatch (unit file only, not enabled) follows
the log, notifies, and with CAMWATCH_AUTO_RESTART=1 restarts SteamVR when the
headset isn't worn and nothing else uses VR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An X11 window in a Flatpak can give KWin an app id with no desktop file:
RustDesk's says com.carriez.flutter_hbb (its GTK application id), and the
Flatpak's desktop file is com.rustdesk.RustDesk. A profile recorded that
id, so it couldn't relaunch the app (PR #16 keeps that from killing
ft-floatd's socket). And the window's pid is the sandbox's own, so a
launched window matched neither by process nor by app id, and didn't
float.
desktop_name() finds the desktop file whose StartupWMClass names the
window's class (or app id) when the app id has none. Capture records
that name, a profile claims open windows by it, and a launch's window
matches by it.
Profiles saved before this keep the old id; save them again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gio.DesktopAppInfo.new() returns NULL for a desktop file that doesn't
exist, and PyGObject raises TypeError ("constructor returned NULL")
rather than returning None, so launch()'s `if info is None` never ran.
The exception escaped the control socket's GLib callback, GLib dropped
the watch, and ft-floatd stopped answering everything: the float key,
dock, Launch as Standalone, and profiles, until the desktop restarted.
Found on the Frame (2026-10-02): a profile saved with RustDesk's
Flatpak open records its window's app id, com.carriez.flutter_hbb,
which has no desktop file (the Flatpak's is com.rustdesk.RustDesk).
`ft-layout use` on that profile asked ft-floatd to launch it, and
ft-floatd went silent. With this, that launch replies "error no app
com.carriez.flutter_hbb" and the profile's other apps open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The headset's right-side click button (KEY_SELECT on gpio-keys, read without
a grab) now works the session: Next while a step waits, pause during a hold,
resume while paused. With no mouse connected the hints lead with it.
The push sections say plainly to push straight out from the headset and pull
back, with a side-view picture of the head, the headset and the arrow, and
the bar's ends read "At your chest" and "Arm out". The bar labels are sent
as one field, so labels with spaces no longer split.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first real session moved on every 5 s with text only, too fast to follow.
Each step now waits for Next (Space or the window's button), counts down 3-2-1
while recording, then holds. P pauses, R redoes a step, S skips a section;
"Advance by itself" (--auto) keeps the old timed flow. Nothing records while
a step waits: each step is its own recording part.
The panel and the window show a picture of each pose (hands/rec/poses,
generated by make_poses.py from a parametric hand, MIT) and a diagram of where
to hold the hands and how far out. prompts.jsonl gains ready, wait and redo
events; session.json gains mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/rec/validate.py (standard library; Linux and Windows, Python 3.12+)
checks an export before upload and when it's received: SHA256SUMS, an
allow-list of files, the manifest's schema and keys, the consent version,
a uuid4 contributor, no identifying fields in calibration.json or
device.json, every sets.bin.zst decompressed to its end as a stream with
each FHSET01 header checked against the manifest, jsonl lines, the total
size. It decompresses with compression.zstd, zstandard or the zstd
program. validate.py DIR [--json].
hands/rec/hub.py uploads an export with huggingface_hub, as a pull
request to contributions/<contributor>/<session>: validate first, refuse
a repeat of the same export, check the login (whoami) and access
(auth_check), upload_folder(create_pr=True) with the manifest summary as
the description, then record the PR under "uploads" in session.json.
Errors are explained (terms not accepted, not found, 401/403, network).
--dry-run makes no network calls. FT_HANDREC_DATASET overrides
HF_DATASET (DeeJanuz/frametop-hands); while the texts are drafts a real
upload needs FT_HANDREC_ALLOW_UPLOAD=1.
The Upload page shows the login with "Check again" and how to run
hf auth login in a terminal (the token never enters the window), then
Upload with a phase, progress and Cancel (hub.py as a child process), the
PR link, and a warning for an export uploaded before. The manual
command stays as the fallback. ft-handrec --hub-dry-run.
session.py also saves device.json: cv.cad_from_cal and head from
/persist/device_config.json, the labeller's shape, nothing identifying;
export copies it. Session ids with a -N suffix are accepted everywhere.
hands/rec/rehearse.sh runs it all without the headset: ft-ringplay plays
30 s of a capture into a ring, session.py records a short test script
with ft-handpanel --no-vr and a tracker, then export, validate and a
dry-run upload (--repo ID uploads for real). It runs in one frame-job
scope, deletes its data and stops its processes, also on Ctrl+C.
hands/rec/tests/test_validate.py covers good and broken exports and hub.py
without the network. The dev container gains python3-huggingface-hub.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/rec/ft_handrec.py + main.qml (Kirigami, dev container; host launcher
hands/rec/ft-handrec): consent (CONSENT.md, asked again when its version
changes; profile.json with a random contributor id), the before-you-start
checklist with the lighting and free-space checks, the session controls
(Space pauses, Esc stops), review with a frame-set viewer that deletes
ranges, takes and sessions, export with progress and cancel (warns while
the headset is worn), and the upload page (UPLOAD.md, the
huggingface-cli command; HF_DATASET is a placeholder). --dry-run runs
sessions without processes, for testing.
hands/rec/takes.py (standard library): indexes sets.bin and sets-N.bin
without reading pixels, reads one set's cameras, keeps deleted ranges in
take.json, and exports: deleted sets left out, zstd -10 -T2 at nice 19,
manifest.json and SHA256SUMS, nothing left behind on cancel.
CONSENT.md and UPLOAD.md are drafts pending a legal review; the window
says contributions aren't open yet. The dev container gains zstd.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/rec/session.py runs a recording session from script.json: it starts
ft-camd and a tracking ft-hands as transient units only if they aren't
running, records each section as one take (ft-hands --record-only at
10 sets/s, a new sets-N.bin after each pause), drives ft-handpanel, and
writes session.json, calibration.json (identifying fields removed),
prompts.jsonl and take.json. Feedback comes from the live hands file and,
in the controller sections, from the panel's device poll. It also runs
from the command line (--dry-run, --speed, --ring, --no-start).
hands/rec/script.json: 11 sections, about 9 minutes without the object
and controller sections.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A head-locked SteamVR overlay for the hand recorder (hands/rec/DESIGN.md):
1.2 m ahead, 12 degrees up, 36 degrees wide, drawn with stb_truetype
into three shared DMA-BUFs as ft-gazepanel does. It shows the title,
step, wrapped instruction, note, countdown, hand chips, near/far bar
and a "Paused" cover, driven over @ft_handpanel.
It also places the touch target, a 2 cm dot in its own overlay fixed
in the room where the head was at the first command for that point,
and logs head and controller poses to poses.jsonl at 250 Hz from a
thread of its own. Both threads take one lock around OpenVR calls.
--no-vr prints each picture's state to stdout (and --dump writes the
pictures), for testing without a headset. hands/rec/build.sh builds it
in the dev container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands --record-hz N records at most N frame sets a second, for the hand recorder (10).
DESIGN.md lays out the recorder: the headset panel, the session runner and its script,
the files, review and export, consent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From curiousjtuber's PR #13: with the gaze service running, SteamVR
restarted its eye tracker every 10 to 13 s of Beat Saber, as if the
headset came off, and each restart took input focus from the game.
The PR stopped every read in a game. Only the action path reaches
SteamVR (UpdateActionState on the gaze set at overlay-global priority,
then GetEyeTrackingDataRelativeToNow); the mmap and our tracker are
read-only files. So only the action is skipped while a scene app runs,
and gaze keeps moving the pointer over the dashboard in a game. The
action source is only used with --source action.
Co-Authored-By: CuriousJ <curious.j.tuber@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From curiousjtuber's PRs: the relay leaves a new input node for the next
scan until udev gives it to the input group, rather than marking it seen
after a failed open; and ft-screens' catcher takes a screen's overlays
from one copy of All() instead of begin() and end() of two temporaries,
which crashed libc++ builds on the first click.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
While a button pressed on a screen is held, UpdateCatcher checks every tick
whether the laser is still on one of the screen's overlays. It built that list
from s.All().begin() and s.All().end(), but All() returns a std::array by
value: iterators into two different temporaries, which is undefined behaviour.
A clang build of ft-screens got a garbage length, threw std::length_error, and
aborted on the first click, taking KWin and the desktop with it.
A new /dev/input node is root:root 0600 until udev applies GROUP=input. The
scan probed each new node once and marked it seen even when the open failed, so
a node caught in that gap was never opened. Behind a KVM, a switch brings back a
hub of devices at once: on the Frame, four nodes failed with EACCES in one switch,
the keyboard was never grabbed, and its keys went to gamescope instead of the
desktop screens. A node that isn't readable yet now waits for the next scan.
From jlneal's PR: a trigger press on a desktop screen stays put until the
laser moves more than 8 logical pixels, so controller jitter doesn't turn
a click into a drag. On top of it, only a hand controller's press starts
the filter: the 3D mouse's laser reaches the screens the same way, and the
PR as sent turned the mouse's short drags into clicks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 3D mouse drives SteamVR's laser through the ft_pointer virtual
controller, so its events reach the screens the same way a controller's
do. The filter held every press, which turned the mouse's short drags
(selecting a character or two, nudging a slider) into clicks. Mark
button events from hand controllers and start the filter only on those.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
container-up.sh starts the dev container in a scope of its own, so
distrobox enter finds it running and skips its wait for distrobox-init.
On a fresh install, init was still setting up passwordless sudo when
dev-container.sh ran sudo dnf install, and sudo asked for a password
with no terminal to read it from. container-up.sh now waits for
container_setup_done itself, and the container's sudo calls use -n,
so a password prompt fails at once with a clear message.
Fixes#9
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A user on a fresh install got "Calibration failed: only 0 of 21 dots" with
no reason. The installer never installed our tracker, so gaze used
SteamVR's, and the only way SteamVR's tracker rejects a dot is losing an
eye for most of the look. The panel just showed a red ring.
- install.sh: step 9/10 installs our tracker (gaze/tracker/install.sh)
after gaze mode, yes by default; it needs sudo, so --yes runs it only
when sudo won't prompt. If it fails, gaze keeps SteamVR's tracker.
Configs that still say GAZE_TRACKER=steam (the old template) are asked
whether to switch.
- GAZE_TRACKER=auto, the new default: ours when it's installed (the
frame grabber, its unit, and ft-eyes' Python), else SteamVR's.
ft-gazed rechecks every second, so installing it switches over. Input
Settings lists Own tracker first as recommended, and says how to
install it when it's missing (checking the host's /etc through
/run/host from the dev container).
- The calibration panel has a note line, orange over the instructions:
why a dot wasn't taken (an eye lost, a blink, the eyes disagreeing for
SteamVR's tracker, from steady_samples' new drop counts; ft-eyes' reply
for ours), what a click is still waiting for after 1.5 s, and a failed
calibration's most common reason, which the Gaze page shows too.
steady_samples keeps the same samples as before (checked on 2037
windows of recordings); a lost eye is named before a blink, since its
openness reads 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Turning gaze mode on without a calibration opened Calibrate only on the
off-to-on change, and only if it could open right then. With the headset
off, the eye tracker silent, or the panel not built, or with gaze mode
already on when the gaze service started, nothing opened and nothing said
why: the pointer just stayed a mouse.
- The gaze service now checks every second: gaze mode on, no calibration
for the tracker in use, eyes seen -> the full calibration opens. One
that closes unfinished opens again only after the headset comes off and
on, gaze mode off and on, or Calibrate, so it doesn't loop. A start that
fails retries every 10 s.
- Its status says why gaze mode can't work yet (checks.problem): not
calibrated and opening, open, closed unfinished, or can't open and why.
- Input Settings shows that under the Gaze pointer switch, along with the
gaze service not installed or not running and our tracker missing its
frame grabber.
- ft-gazectl on notes a missing calibration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/ft-cutouts on|off|status starts ft-camd and ft-hands as transient
user units with ft-hands' new --no-gestures: hands are published for
ft-screens' cutouts, but no pinch or grip is detected, so nothing clicks
or drags and a closing hand doesn't raise the tracking rate. It needs a
build and ft-camd's capabilities, not hands/run.sh install. Its units
conflict with ft-handsctl's, so each stops the other, and they stop with
SteamVR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Meta+key combination (Meta+J for gaze_left, say) hides Meta's release
from the desktop, and the relay skipped share_key for it too. frame-voice
saw Meta go down on @frametop_keys and never come up, so it held all
dictated text back, waiting for that release. Keys of grabbed keyboards
are now shared as pressed, before key_binding() decides what the desktop
gets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
