Pressed when the pinch closes, released when it opens, its hand dragging
the pointer in between (at the pinch gain, past the dead zone), like the
mouse's button. That's what the gaze probe's Click practice needs: it
freezes its own gaze dot at the press and drags it by the pointer's
movement until the release. With gaze mode on, the pinch still holds the
press back and clicks on the release.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pointer helper reads ft-hands' gestures. A pinch holds the pointer
where the gaze put it and clicks on the release; held, the hand nudges
the pointer at half its angle past a 1.5 degree dead zone, which also
teaches the gaze tracker. A grip presses where the pointer is, drags with
the hand, and releases when the hand opens, unless it began more than
30 cm below the eyes (hands on a desk). Hand movement is taken in the
room with the head pose at capture time. Hand use keeps the pointer from
the relay's idle release, as gaze mode does. POINTER_HANDS and friends in
frametop.conf.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands tracks with the mono IR cameras in dim light and with every
camera (or the colour pair, HANDS_BRIGHT) in bright light, going by the
colour frames' mean brightness with hysteresis and a 2 s hold
(HANDS_CAMERAS=auto, the default; mono, color and all fix it). Colour
frames are placed on the mono cameras' clock by their dequeue time, and a
view in a camera a step lacks waits for that camera's next frame.
A grip (a closed hand) is a second gesture next to the pinch, in version 2
of the gestures file: every fingertip curled toward the wrist, beginning
only on a hand seen open within a second and held up in front. On the
2026-09-30 lit recording that leaves 6 false grips of 14, all with the
hands on the desk; pinch counts are unchanged. ft-handreplay logs grips
and finger curl, watch_gestures.py shows them, and tools/cut_sets.py
copies a few sets out of a recording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A colour camera probes at most 4 buffers a frame (each probe syncs a
~9 MB buffer's cache, which made the mono cameras miss frames), and one
that goes stale twice in a row is paused (10 s, doubling to 160 s) and
learned again instead of ft-camd exiting. The colour cameras run at 2 fps
until a reader asks for more in frametop-hands/color-fps, which saves
most of their decoding while only their brightness is needed. Each mono
camera's latest near-black frame's mean goes in the ring (dark_mean), a
measure of the room's IR light. The service now starts with --with-color;
HANDS_CAMERAS=mono leaves the colour cameras out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hand tracking (ft-camd, ft-hands, and their tools) joins the desktop. The
hands file and ring move to /run/user/UID/frametop-hands/, which ft-screens'
hand cutouts now read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The runtime files move to /run/user/UID/frametop-hands/: the desktop
session deletes /run/user/UID/frametop at every start.
- The cutout copy shader runs at highp: mediump (16-bit on Adreno)
stepped 1.7 texels across a 3440-pixel screen.
- One hand no longer pinches both sides after its left/right call flips
mid-pinch, and --pinch-palm-down (0.6) holds back pinches with the palm
facing down (typing on a lap keyboard).
- ft-camd judges a colour frame fresh by its luma rows only, and logs
per-buffer changes at stale colour frames with FT_CAMD_DEBUG=1.
- hands/run.sh caps skips the setcap when ft-camd already has them.
- The replay tool dumps poses (--poses), and its pinch events carry the
hand id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Floating windows join the keyboard and the hand cutouts. Floating panels
don't get cutouts yet: their panel and popups show crops of the client
buffer (texture bounds), which the side-by-side cutout buffer doesn't
match. KWin gets both the spare outputs and our input method, and the
pointer helper's frametop. prefix already covers the float panels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes#7: SteamVR's keyboard never came up for the desktop's apps, and
opening it for our panels doesn't work well on the Frame (it's Steam's own
panel, mounted in the dashboard's scene, it follows the laser between
panels, and it takes the controllers over to SteamVR's laser).
- KWin starts input/ft-textinput as the desktop's input method. It tells
the input relay when a text field gains or loses focus, and the relay
asks ft-screens to open or close the keyboard. The session drops the
QT_IM_MODULE=xim and GTK_IM_MODULE=xim that the gamescope session sets,
or Qt and GTK apps never report text fields.
- The keyboard is ft-screens' own panel (screens/keyboard.cpp): a US laptop
layout, typed with a controller's laser or the 3D mouse. It opens 0.7 m
in front of you, below your eyes and facing you. It has a grab bar to
move it, a Close key, latching Shift, Ctrl and Alt, and repeat on a held
key. It's drawn into shared DMA-BUFs, so it doesn't flicker. Its keys
reach the focused screen as key presses, so every app takes them.
- It steps aside while the Steam menu or Steam's own keyboard is up and
comes back after. A layout reset closes it, and it doesn't open without
a head pose.
- Frametop Input Settings has a Keyboard page: open it for every text
field, only while no keyboard is connected (the default), only from a
mapped button (the new Open/close keyboard action, for mice and
controllers), or never. A switch keeps it open until you close it.
- The pointer helper treats every frametop.* overlay as a real panel. The
keyboard's shared texture reports 0x0 like SteamVR's scene-graph
controls, and the helper had given it their wide catch radius.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- KWin's nested backend makes an output the size it's configured to times its
scale, so after Meta+scroll every size ft-floatd sent was multiplied again,
and an odd result disconnected KWin (buffer not divisible by its scale).
ft-floatd now asks for sizes in the output's scaled terms (kwin_size), and
asks again after each scale change.
- SteamVR reports mouse positions on a panel with texture bounds in the whole
texture, not the crop, so clicks on a floating window landed up to ~200 px
off. The mouse scale is now the buffer's size, as on a screen.
- A floated window starts 30 cm in front of its screen (was 5 cm), so it's
easy to point at apart from the screen behind it.
- No 1 s wait before a spare turns on (a disabled output never commits), and
the login splash on the spares isn't taken for floating windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frame-eyes, a separate project until now, becomes gaze/tracker:
- ft-eyegrab (was fe-bufprobe) copies the eye-camera frames, read-only, out of
SteamVR's eyetracking process. It runs as the system service
frametop-eyegrab.service, which gaze/tracker/install.sh installs to
/etc/frametop with sudo. It keeps only CAP_SYS_PTRACE, CAP_DAC_READ_SEARCH, and
CAP_CHOWN, and copies frames only while /dev/shm/frametop-eyes-want is fresh,
holding none of the tracker's buffers otherwise.
- ft-eyes (was fe-trackd) runs under ft-gazed in the dev container, with
build/venv's pinned numpy and OpenCV: while Eye tracker is Own tracker, or on
the probe's lease ("eyes SECONDS"). No sudo password or fe-live script at
run time any more.
- lab/ holds the research tools (ft-eyes-score, -e2e, -record, -replay,
-session) and findings.md. Recordings live outside the repo, in
~/.local/share/frametop/eyes/captures; .gitignore catches stray frame dumps.
Its socket is now @ft_eyes, its output /dev/shm/frametop-eyes-gaze, and its state
~/.local/state/frametop/gaze/eyes. On practice1 -> practice2 the whole live path
(ft-eyes-e2e) gives 1.30 deg median and 3.18 for the worst tenth, as before the
move (1.30, 3.21).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-screens' hand cutouts read /run/user/UID/frametop/hands through
hands/include/fh_hands.h instead of their own copy of its offsets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Programs: ft-camd (the camera broker), ft-hands (the tracker), and
ft-handreplay and ft-ringplay for recordings, built by hands/build.sh
into hands/build/ with one Makefile. The first build fetches ncnn at
frame-hands' pinned tag and builds it with the same options.
- ft-camd gets its privileges from file capabilities (CAP_SYS_PTRACE,
CAP_PERFMON, CAP_DAC_READ_SEARCH) that hands/run.sh install sets with
sudo, and drops them once set up. It still works under sudo. It runs
on the host, linked statically, as frametop-camd.service. ft-hands
runs in the dev container as frametop-hands.service. Both start and
stop with SteamVR.
- Files move to /run/user/UID/frametop/ (cam-ring, hands, gestures),
not $XDG_RUNTIME_DIR, which a terminal in the Frametop desktop has
its own of. SIGUSR1 recordings go to ~/.local/share/frametop/hands.
- The calibration is read through /run/host in the container.
- Settings: HANDS_SWAP_SIDES and HANDS_CPUS in frametop.conf.
- install.sh offers hand tracking as an optional last step.
- The container gets jsoncpp-devel, glibc-static, and NumPy and OpenCV
for the Python tools.
- tools/ring.py reads the ring, and models/NOTICE credits the
Apache-2.0 models.
Checked: ft-handreplay gives identical summaries and byte-identical
depth dumps to frame-hands' fh-replay on both 2026-09-29 recordings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frametop Input Settings' Gaze page gets two settings, saved in frametop.conf and
read again by ft-gazed when the file changes: Eye tracker (GAZE_TRACKER: SteamVR's
or our own, frame-eyes' fe-trackd) and Eye bias (GAZE_EYE: auto, left, right).
ft-gazed now combines the eyes, each calibrated on its own: SteamVR's set 2 eyes
with the probe's Left eye and Right eye calibrations, or our tracker's eyes as
they come. Without per-eye calibrations, or with --source, it keeps the older
one-source path. A pointer nudge finds its look from the raw gaze the helper
echoes back; with our tracker it goes to fe-trackd as a click.
The bias leans instead of choosing (gazecal.EyeWeights): on 306 live clicks the
eyes' errors partly cancelled, both together 0.65 deg off against 0.96 and 1.11
for either alone. Left or Right counts that eye twice; auto weights each eye by
its RMS miss at its last 20 nudges, since the calibration's fit picked the wrong
eye on SteamVR's test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trackd/ becomes track/, and the calibration helper the Python tools
import moves from the prototype's folder into tools/. The model
development tools (nettest and the scripts that compare it with the
Python models or cut int8 calibration crops) stay in frame-hands with
the prototype they need.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The history of frame-hands (~/Desktop/Projects/frame-hands on the
Frame), filtered to what moves: the camera broker (camd/), the tracker
and its offline tools (trackd/), the shared file layouts (include/), the
ncnn models, the analysis tools, and the calibration and model helpers
they import from the Python prototype. The reverse-engineering notes,
probes, camprobe, and the rest of the prototype stay in frame-hands.
Unchanged here: the renames to ft- names and Frametop paths follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frame-hands (the headset cameras' hand tracking, a separate project so
far) becomes a native component under hands/. docs/hands-migration.md
has what it is, where each part goes, names, build, how ft-camd gets
its privileges (file capabilities), interfaces, open items, and steps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- fh-camd --with-color publishes the Arcturus colour pair (luma, half
size, 30 fps) to the ring, flagged FH_CAM_COLOR. fh-tracker records
them, and fh-replay --cams mono|color|all tracks with them, using the
module's EEPROM calibration (load_color_calibration).
tools/check_color.py checks which node is left and how the crop maps.
- Pinch detection per hand (trackd/pinch.h), published to
$XDG_RUNTIME_DIR/frame-hands/gestures (include/fh_gestures.h). It has
begin and end counters, times, the pinch point, and the begin point
for drags. tools/watch_gestures.py shows it live, and fh-replay
reports it.
- fh-replay --depth and tools/depth_report.py measure the depth without
ground truth: noise along the line of sight against across it, the
one-camera guess, and a simulated camera loss.
- fh-tracker --swap-sides, --ring, --keep-presence, --record-only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wlroots asserts that nothing still listens to its xdg-shell and decoration
globals when the display goes, so every desktop stop ended with ft-screens
aborting and leaving a core dump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Full screen fills the window's own panel (the margin drops to zero).
Meta+scroll over a floating window changes its scale at the same size in
pixels. Spare outputs are sized to a multiple of KWin's buffer scale, and
screens to even sizes: an odd buffer at a fractional scale is a protocol
error that disconnected KWin. The 3D mouse's drag lock now crosses onto
other Frametop panels unless the pressed one is being carried.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-screens makes a panel for each spare output (frametop.float.N), hidden
until ft-floatd floats a window on it. The panel shows only the window's
rectangle of the buffer at the density of the screen it came from, its
popups and dialogs get small panels over it, pressing its title bar
carries it while KWin's pointer stays put, the corner tab resizes the
window in pixels, and two more buttons close it and put it back on the
desktop. The session adds FLOAT_SLOTS spare outputs to KWin and starts
ft-floatd; ft-layout arranges only the screens' outputs, and the pointer
helper treats the new panels like screens. Not yet tried in the headset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-floatd loads the script into the desktop's KWin, which reports windows
over D-Bus and takes commands through a long poll. Floating a window (the
window menu's Float in VR, Meta+Shift+F, or ft-float) turns on a spare
output sized to the window plus a margin, moves the window onto it, and
tells ft-screens the panel's crop, density, and place; docking puts it
back and turns the spare off. Tested on the headless test desktop; the
panel side in ft-screens comes next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KWin's nested backend ignores the position in wl_pointer.enter, and
wlroots drops a motion to the position it entered at, so KWin kept its old
pointer until the next move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-screens --no-vr runs without SteamVR and leaves the input relay alone;
--control names its control socket; "toplevels" lists KWin's windows with
their titles and sizes; "input" feeds pointer events as if from a panel.
screens/test/headless.sh starts it with a bare nested KWin next to the
running desktop, and loads KWin scripts, runs apps, and takes screenshots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the Own tracker, the probe polls its status each second. After the headset was off
(or the tracker restarted), it shows one dot at the centre until you look at it and press
(S skips): that click resets both eyes' shifts, so the first real clicks aren't 10-17
degrees off. The screen-to-direction geometry the calibration used is shared with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
While a button is held on a screen, the laser leaving it no longer takes
KWin's pointer, and an invisible catcher overlay sits on the laser whenever
it's off every panel, so the release reaches KWin at the pointer's last
spot. The pointer helper also reports the mouse's left release as a
backstop. Not yet tested in the headset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
probes/core_ab.py with the headset on (3 rounds, the same replayed frames in
every block): on 5-7 a step took 8.4 ms against 13.2 ms on 2-4, where XRService's
head tracking also runs, and latency fell from 14.1 to 9.6 ms. The compositor's
late frames and CPU/GPU time per frame didn't change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The calibration dots for the Own tracker go out to the Calibration ring angle each way on
an oval, not to the window's corners, which were too far to look at while facing the
centre. The Own tracker learns from drags up to 25 degrees (after taking the headset off
and on, its first clicks were 11-17 degrees off, and the 6-degree limit blocked them). The
probe starts on the Own tracker when it's running, and the calibration header names the
tracker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fh-ringplay plays a recording into a frame ring in real time, and fh-tracker
--ring reads it (cameras without a device node are mapped by name). With the
same frames in every run, probes/core_ab.py compares the tracker on CPUs 2-4,
on 5-7, and not running, measuring the tracker's step time and latency, the
compositor's late frames and CPU/GPU time, XRService's timing warnings, CPU
temperature and clocks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fh-camd (camd/) borrows XRService's camera buffers and publishes the tracking
cameras' frames to a shared ring. fh-tracker (trackd/) finds hands in them with
MediaPipe's palm and landmark models on ncnn, triangulates them in 3D, and
publishes them for ft-screens. fh-replay replays recordings offline. tracker/ is
the earlier Python version; tools/ and probes/ hold the checks and experiments.
As of this commit: crop contrast defaults to CLAHE for the palm search and plain
crops for the landmarks, --swap-sides works around fh-camd naming the side
cameras backwards after some XRService restarts (tools/check_sides.py detects
it), and --record-only, --with-dark, --cpus and --keep-presence support the
bright-light and CPU-placement tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its fit goes wrong past its dots, and the ring (limited by the window's height) never
reached the sides, where frame-eyes' worst practice clicks were. With the Own tracker
the calibration now uses the centre, corners, and side middles of the practice area.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-gaze reads frame-eyes' /dev/shm/frame-eyes-gaze and reports it as the source "own",
with each eye's gaze, where each lands on the screen, and its slip. The probe gets a
SteamVR / Own tracker toggle. With Own tracker on, it hides SteamVR's gaze and draws a
red dot per eye, its calibration fits fe-trackd's own calibration, and practice clicks
teach fe-trackd instead of the probe's correction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tracked hands arrive 30-60 ms after the cameras saw them and reach the
displays later still, so holes trailed moving hands. Track each hand's palm
velocity in the room and move its capsules ahead to about when the frame is
on the displays, every tick, so the holes also move smoothly between tracker
updates. Slow hands aren't moved (their velocity is noise). The control
socket gets cutouts predict on|off and cutouts lead <ms>.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A head-locked performance overlay kept catching the 3D mouse. It has no
input method, so SteamVR's laser passes through it, but the helper hit
tests every visible overlay with ComputeOverlayIntersection, and the dot
stuck to it whenever it crossed that corner of the view.
POINTER_IGNORE in frametop.conf now lists overlay keys the helper leaves
out of the collision, comma-separated shell patterns, so "vendor.app*"
covers a whole app, including panels it opens later. The laser starts
just before the cursor point, so an ignored panel nearer to you doesn't
catch it either.
Frametop Input Settings has a new Ignored panels page. It asks the
helper for SteamVR's overlays ("overlays", answered from the list's
thread once vrcmd has run again, even while the pointer is off), groups
them by app, and has a checkbox per panel and one for the whole app.
Frametop's own screens aren't offered, since ignoring one would leave
nothing to click the app on with the mouse. Entries for apps that
aren't open are listed so they can be removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capsule end near the eyes' plane projects far across a screen with a huge
radius, so one bad hand estimate there tore a hole through the screens for a
moment. Clip capsules 12 cm in front of the eye, as the tracker now does too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Where frame-hands tracks a hand between an eye and a screen, that eye sees the
room through the screen. handcut.cpp draws the screen's buffer side by side
(one half per eye) with the hands cut out, only while a hand is in front of it.
The cutouts command turns it on or off. ft-handtest tries it on a test panel.
Not yet tested in the headset.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
