- Far stretch: both palms held out at the same far panel, then two fists
within 0.6 s, scales it round its centre (a floating window by pixels)
until a hand opens. Aims at the panel both hands' lines meet, or the
line through the palms' midpoint. Grab that holds its fire meanwhile, and
both hands are claimed from the pointer helper so a fist passing through
a pinch doesn't click or drag under a raised hand ray. Near misses log why.
- Reset button: a hand's tap shows "Reset the layout? Tap" under it for
5 s; a tap on it (or the button again) resets. A laser click resets at once.
- Lazy susan: with the other hand still or unseen, a looser shape, 18 cm
and 0.5 m/s count; any stroke rides out 2 frames off its shape; misses
log open/tilt/lead against their limits.
- Pointer helper: the two-hand gaze drag no longer logs "with the right
button" (it is the left alone).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Throws: a carry's let-go takes the fastest outward stretch in the last
150 ms before the fingers opened (a hand slows as it lets go), at 0.45 m/s
within 70 deg (was 0.55 within 60, the last 100 ms only). Every let-go
moving 0.3 m/s or more is logged.
Sent away (thrown, flicked, pushed), a panel goes along that line out as
far and as big as it was before a hand or "grab that" took it, not back
to its old spot (layouts are for that). Put down past 1.2 m, that's its
distance from then on; never taken, it flies to 3 m as before. The front
slot's screen leaves the slot the same way.
The bar's buttons sit a button's width apart, and the reset button
resets the layout on a tap (HANDS_RESET_HOLD_MS=0; 500 before).
Grab that's glow: ft_pointer's laser sits on the panel with the cursor
all the time (the 3D mouse, the gaze), so "no glow under a laser" kept
the glow off that panel. The pointer helper now tells ft-screens which
hand rays are up ("ray L R", on change and every second); our laser
counts as one only then. Controllers' lasers count as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User 2026-10-11, the second gaze test:
- "hand tracking stopped" ends were an unsigned age: ft-hands publishing just after the
loop's tnow made it negative, wrapped to 584 years, and ended held pinches, nudges,
drags and scrolls. Signed now.
- The two-hand drag mirrors the keys: hold one pinch (aim, nudge), pinch with the other:
the left button goes down where the dot is (as Meta+K with Meta+J held); the drag
lasts while either pinch holds, and the hand still pinching drags. A second pinch
during a drag joins it.
- The fast-hand filter refused dozens of still pinches (frame-to-frame palm speed is
noisy): smoothed, 1.0 m/s.
- A three-finger scroll goes on when ft-hands swaps which finger it calls pinched.
- Lazy susan: counted strokes came at 25-30 cm in 230-470 ms, at both limits: 22 cm,
quick somewhere along it (0.7 m/s peak) within 0.9 s; looser hand shape. The flags
replayed: 2 strokes in 45 flags of other use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user's 2026-10-11 test of gaze + hands:
- Two-hand drag as they meant it: hold one hand's pinch (nudge it into place), tap
with the other: index presses the left button where the pointer is, middle the
right; the first hand drags (POINTER_GAZE_TWO_HAND). The tap-first right-drag is
off by default (it could catch a tap followed by a nudge).
- 9 held pinches ended as "hand tracking stopped (1.5 s)" while ft-hands still held
them: 4 s with nothing pressed, 2.5 s with a press, and the line now says the file's
seq and ages to find the cause.
- A gaze pinch held back ends without a click when the hand closes into a fist
(ft-hands' grip may not take it past 0.1 s now); a pinch begun while the palm
moves faster than 0.6 m/s (a stroke) doesn't click (POINTER_PINCH_MAX_SPEED).
- Three-finger scroll: 3.5 cm and 0.4 s to join.
- Gaze mode hides the dot and collapses its laser for 1.5 s after each key
(POINTER_TYPING_HIDE): it sat big and bright on the text being typed.
- The front slot's keep counts clicks, scrolls, drags and keys, not the gaze pointer
passing over it (every brought screen was kept 3 s in).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User 2026-10-11, gaze + hands becoming the main input:
- Right-drag: an index tap on one hand, then an index pinch on the other within 1 s,
holds the right button at the gaze lock; that hand drags until it ends
(POINTER_GAZE_RIGHT_DRAG). It replaces the chord (POINTER_GAZE_CHORD=0).
- Three-finger scroll: the thumb on both the index and middle tips (3 cm) scrolls where
you looked as the hand moves, like the ray's middle-pinch stick (POINTER_GAZE_SCROLL).
- The fist guard (a hand closing into a fist: no click) now covers gaze and chord
pinches too, from each hand's own frame (not the ray's).
- A hand's laser shows only its dot: the beam starts just short of it, as the eye's
does, held still through a carry (POINTER_RAY_BEAM=0; SteamVR's beam width is one
global setting it reads only at start).
- The front slot gives a screen up to manual controls: grab that, a push, flick or
throw, or a carry by hand or laser releases it (it moves like any other and stays
where you put it); the next stroke puts it home and brings the next.
- The lazy susan counts only palm-led strokes (the return stroke does nothing; the
other hand palms the other way; HANDS_SWEEP_BACK=1 for both).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lazy susan (the third live test): a brought screen comes 1 m ahead, as big as its
resolution needs for text to read: its logical pixels (KWin's scale undone, now passed
to vr.cpp) at 26 px/deg, at most 100x55 deg keeping its shape, at least 40 deg wide,
curved round you past 60 deg (grabthat::ReadableSize; SCREEN_SPIN_DISTANCE, _PPD,
_MAX_DEG, _MAX_TALL_DEG). Its curve comes back on the way home. The strokes cycled
104, 1, 104 with three screens up: a screen still flying home read as straight ahead;
each one is now taken where it lives (home, or where it's going).
Accidental clicks: most were pinches with no laser up pressing where the pointer was
last left ("no ray up: a press where the pointer is", a dozen in two minutes). With
gaze off they now only wake the pointer (POINTER_PINCH_NO_RAY=1: the old air mouse).
The hand's laser shows only with the thumb and index within 6 cm (gone past 10;
POINTER_RAY_READY), and a fist drops it: fh_ray's "ready" and "open" tests, the fist
read as the index and the other three both curled under 1.1 (FistCurl; measured on 47k
flagged hands: the other three alone read as a fist in a fifth of the pinches). A ray
pinch made while the hand closes into a fist doesn't click. Input Settings switches for
all three (POINTER_ ones reload the helper).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the 2026-10-11 09:43 live test:
- Two fists anywhere near one panel (palms 30 cm in front to 15 behind, 30 cm
round it) resize it by their shape alone (HANDS_FIST_PAIR): ft-hands' grip
only begins on a hand seen open just before, at the panel.
- Two fingers (index and middle straight, together, level; ring and little
curled) touching the content: a swipe scrolls (a notch per 1.2 cm, the
content following), a lift unmoved clicks, held still they act as a poke
(HANDS_TWO_FINGER_SCROLL).
- Stuck presses: a press isn't held through a tracking loss (only grabs are);
a poke lets go as the index curls; a pinch-click as the tips are seen 4.5 cm
apart; either held still 1.5 s (HANDS_PRESS_STILL_MS).
- The hand ray hides within 22 cm in front of a panel and for 0.8 s after
(HANDS_RAY_HIDE_REACH); the pointer helper ignores a ray-less pinch there
without the gaze (it woke the laser).
- The push/flick hint glow is off (HANDS_PUSH_GLOW): it covered near panels
the palm faced while clicking.
- Grab that gives panels that look small up to 10 deg more aim margin
(HANDS_GRAB_THAT_SMALL).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the 2026-10-11 09:25 live test:
- A fist only resizes (HANDS_FIST_RESIZE_ONLY, default on): one fist holds the
panel still as an anchor; two hands with a fist among them scale it about
its centre, no move or turn. Pinches still carry and turn.
- Throws need 0.55 m/s (was 1.0): the user's releases were 0.62-0.96 m/s.
- Grab that brings a panel back at the width it had when it was last sent
away from arm's reach (g_nearWidth), not grab that's maxWidth.
- A natural reach arms grab that: palmArm 0.2 (was 0.45), aimFacing 80 deg
(was 55), aim margin 10 deg (was 6). Flag replays: 11 fires (was 8).
- Curve: a curved screen's radius follows its distance from the head after
every move, and it bends at most SCREEN_CURVE_MAX_DEG (100) round: one
curved up close and thrown 3 m out at full width had wrapped into a
cylinder.
- The pointer helper releases its laser while a hand touches a panel
(ft-screens' near claim): the laser from the head to the fingertip's
cursor got in the way and held the button, refusing pokes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A side "off" for no hand tracking logged as "off: its controller is in
use" (State() != Hand covered both), which hid the 2026-10-10 flicker
behind controller handoff. Now each change of the feed's liveness logs one
line ("hand tracking: nothing new for 1.5 s: sides off" / "publishing"),
and a ray dropped for it says "off: no hand tracking".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The loop reads its clock at the top and ft-hands' files later in the pass.
A frame written in between has publish_ns after tnow, and the unsigned
"now - publish_ns" in HandsInRoom::Live wrapped round: the feed read as
stale for that one pass, both sides went "off", and the "hands" word to
ft-screens flickered to "off off" and back. 2026-10-10 20:30-20:37: 24
flickers in 7 minutes, each with ft-hands publishing at full rate (pinches
logged at the same moments), each ending ft-screens' holds and the ray.
feedage.h: AgeNs/Fresh, age 0 for a frame from after "now". Live and the
gestures check that ends holds ("hand tracking stopped") use them.
pointer/tests/feedage-test.cpp replays the loop against 30, 15 and 1 Hz
publishing: the old check calls 28-2408 passes stale, the new one none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User, 2026-10-10: Vision Pro-style resize from all four corners. Only the
bottom right worked by laser (the .resize handle), and a resize kept the
screen centred.
- One resize handle per panel still (SteamVR's 128 overlays are tight:
a third remote display once didn't fit), but it moves to the corner
being aimed at (vr.cpp UpdateCorner): a laser's ray (a controller's,
or the pointer helper's device, which the gaze and the hand ray
drive) landing within a fifth of the screen's height (5-20 cm) of a
corner, or a hand's pinch at one (handinput's corners()). Its texture
turns for that corner, it brightens the closer the aim (from the
controls' idle 0.55 to 1), and it's lit blue while resizing. With no
aim it lingers 0.4 s, then goes back to the bottom right.
- Any corner resizes and the opposite one stays put: screens and
remote displays keep their aspect (the corner follows the diagonal,
ResizeFromCorner, on the cylinder for curved ones: SurfacePose),
floating windows take the pixels from that corner (ft-floatd's new
size is placed keeping the opposite corner, keepX/keepY; every corner
of a window resizes now, the title bar's ends included).
- Near hands: a pinch at any corner resizes the same way (the carry
now gives the pose as well as the width).
- Gaze pinches (gazehand.h): with the pointer on a screen's resize
handle or grab bar, the pinch is a press after 0.12 s still
(POINTER_GAZE_CONTROL_HOLD), not 0.5: nothing to click there, and the
hand drags the resize. The chrome overlays are announced to the
helper as they're made, so its ray and the gaze find a new remote
display's handle at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The near report to the pointer helper ("near <ns> <L> <R>") was one flag
for two things: the hand's ray hides, and its pinches and grips are
ft-screens'. With the 0.075 m zone, its 2.5 cm leave margin and 200 ms
linger (there so the ray doesn't flicker at the edge) would still have
swallowed pinches made just outside the zone, the live test's complaint.
Now ft-screens says 1 when the hand is within the zone itself (or
pressing, grabbing, or waiting on "grab that"): its gestures are ours; 2
in the margin: the ray stays hidden, the gestures are the helper's (ray or
gaze); 0 otherwise. The helper takes 1 as the claim and 1 or 2 as near for
the ray; an older helper takes 2 as 1, as before. ft-screens' own pinch
and fist begins need the zone itself too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts:
- ft-pointer.cpp, the gaze pinch comment: typing's "the rules for all
gestures hold (..., typing, ...)" kept, gazechord's chord and
POINTER_GAZE_PINCH paragraphs added after it, and a line that a hand
typing holds back makes no chord.
- ft-pointer.cpp, the hands settings: gazechord's gazePinchOn, nearOn,
handsTrack kept; typingHold dropped for typing's typing::Params.
- ft-pointer.cpp, mayPoint: both conditions (handsOn, and not a side
typing holds back).
- ft-pointer.cpp, beginGazePinch: gazechord's busy line (a chord isn't
busy), without the old sinceKey/typingHold lines. The chord's pinch
goes through typing.h's Why in the gesture loop before startGesture,
like any pinch, so a hand typing holds back can't make one.
- docs/reference.md settings line: typing's and reach's keys, plus
POINTER_GAZE_PINCH.
- hands/README.md near input: typing's hold-back sentence and
gazechord's HANDS_NEAR=on.
- session/frametop.conf.example: reach's HANDS_GRAB_ROTATE* and
HANDS_GRAB_THAT* lines, then gazechord's HANDS_NEAR line.
Reconciled: gazehand's Situation loses sinceKey and typingHold, and
Refused its "ignored (typing)": the helper had stopped setting them on
hands-typing (typing.h holds hands back first), so that guard could
never fire. gazehand-test's key checks now drive typing.h: keys hold a
hand back (2 s now, was 1), POINTER_PINCH_TYPING=0 is no guard, and a
chord's pinch by a hand on the keys is held back while one lifted off
them goes ahead. The chord's own pinch counts as a gesture held for
typing's "busy" (free if out of the keyboard zone when keys come), as
the left's hold does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper sends the chord's right button (btn b) apart from the left's
press, so both reach the panel; the chord ends with its pinch, its hand
lost, the pointer going, a controller on its side, or the calibration
panel. POINTER_GAZE_PINCH (1) turns clicks at the gaze off on their own.
HANDS_NEAR=on runs the handoff for ft-screens' direct touch without
POINTER_HANDS (HANDS_NEAR=1 keeps its meaning), and the relay feeds it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An index pinch by the other hand while a gaze pinch is held presses the
right button until that pinch ends; a first pinch still held back becomes
a real left press at once, where the pointer is. ChordButton times the
right press (30 ms after a left the chord made, a tap clicks, a lost hand
releases). Tests for the chord sequences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The posture's and a raise's start used -1 for none, so a replay of a flag
(times before it negative) never saw the posture. Found replaying the
2026-10-10 flags through it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper feeds typing.h the relay's keys and each hands frame. A hand it
holds back makes no pinch, middle pinch or grip (ray, gaze or plain), and has
no ray: one that drove the pointer drops it (to the mouse or the gaze, not
the controllers) and can't claim it again until lifted off the keys. A hand
pointing or holding a gesture out of the keyboard zone isn't held back by
the other hand's keys. ft-screens gets "typing <L> <R>" on a change and every
second. The old 1 s guards (beginHold, gazehand's) give way to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A key holds every hand back for POINTER_PINCH_TYPING (now 2 s); a hand in the
keyboard zone while typing stays held back until it rises POINTER_TYPING_RAISE
above where it typed (or turns its palm); both hands in the keyboard posture
count as typing within POINTER_TYPING_RECENT of a key. Tests with made-up keys
and hands placed as in the 2026-10-10 flags and the desk take.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A guided capture (probes/tap_session.py) has its head poses in poses.jsonl
("hmd": {"m": [...]}) and the raw-to-mono clock pairs in prompts.jsonl:
--prompts maps them, so the tap capture's drags and holds can be replayed.
From a dump it now also reads each pinch's distance and ends, and reports
a ray pinch's release kick, the raw ray's per-frame jumps, the pinch point
a near grab carries (measured and pinned to the palm) with its jitter,
drift and release kick, and where the hand's own direction sits against
the shoulder's ray. --set takes the new fh_ray settings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ideas from StereoKit (MIT, 07b9f0d), credited where a formula comes from it:
- The torso faces the head's forward counted twice plus the direction to
each wrist seen (POINTER_RAY_TORSO_WRISTS, on), so looking aside while
pointing swings the shoulders, and the ray, less. probe-moves replay: the
shoulder's share of a still hand's ray motion 20 -> 18%, p90 2.0-3.7 ->
1.7-3.6 deg/s; the tap capture: nothing worse.
- The ray can turn toward the hand's own direction, from the palm's axes
(wrist, middle, ring and little knuckles) tilted -30/8 deg
(POINTER_RAY_HAND_WEIGHT, 0). Off: every weight measured worse (at 0.3:
judder 0.07-0.14 -> 0.11-0.33 deg rms per 90 Hz tick, raw jumps over
3 deg 159 -> 330). The gate still tests the shoulder's ray.
- The knuckle point can be pinned to the palm's axes during a pinch, easing
back on the steady shoulder's schedule (POINTER_RAY_ANCHOR, 0). Off: our
pinch point is the knuckles already, and pinned it shook as much or more
and kicked no less on release, fitted or not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
While the panel was up, the helper took only "btn trigger 1" and "gazekey left 1" as
"take this dot". A left button mapped to gaze precision or gaze drag sends "precision|gazedrag
<source> 1" instead, which the panel dropped, so the click did nothing. The panel's answers
move to pointer/helper/calpanel.h, with an offline test (pointer/test/calpanel-test.sh).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ray's One Euro filters (new hands/include/fh_smooth.h) run in continuous time and the
helper reads the ray every loop, so the pose sent to ft_pointer moves at about 100 Hz
instead of jumping at the tracker's 15-30 Hz (the dashboard's dragger made those jumps
jerky). The direction is filtered on the sphere. On the probe-moves replay the pose's
second difference per 90 Hz tick went from 0.90-0.96 deg rms to 0.24-0.26 at the same lag.
While a ray pinch is held the ray pivots on the shoulder as it was at the press, so looking
around doesn't move what's dragged; the reach for push and pull is filtered. A pinch's
first 0.3 s takes 6 deg of turn before it drags. One failed read of the gestures file no
longer ends a drag, and the log says why a drag ended or the ray went off.
Settings: POINTER_RAY_SMOOTH_MINCUTOFF, POINTER_RAY_SMOOTH_BETA, POINTER_RAY_SMOOTH_HOLD,
POINTER_RAY_STEADY, POINTER_RAY_TAP, POINTER_RAY_TAP_DEADZONE. rayeval reads publish dumps
and reports the smoothness at the display's rate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With gaze mode and POINTER_HANDS on, the gaze is the pointer and the hand ray
stays off (POINTER_GAZE_HAND_RAY=1 brings it back). A pinch clicks where you
looked 75 ms before ft-hands saw it; held and moved past 4 deg of hand
movement (POINTER_GAZE_PINCH_DEADZONE) it nudges the pointer at 0.3x
(POINTER_GAZE_PINCH_GAIN), and the correction is a gaze lesson. Held still for
POINTER_GAZE_HOLD it's a press the hand drags. A middle pinch right-clicks at
the gaze. One log line per pinch. Rules in pointer/helper/gazehand.h, tested
in pointer/tests/gazehand-test.cpp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Quest-style: a see-through light hand from the tracked skeleton with a bright
rim, fading out behind the wrist, over everything. Each hand is a placed,
side-by-side overlay facing the head, so it stays put as the head turns and
redraws only as the hand moves. Hidden while a hand holds a controller (the
pointer helper's word, now also on @ft_handview). frametop-handview.service
runs it; nothing installs it yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live test 2026-10-09: taps took several tries and the laser jumped between
hands. The owner's ray dropping for a moment (closing the fingers turns the
hand) handed the laser to the other raised hand at once, for good; a pinch
by the other hand pressed where the old owner's ray pointed; a pinch whose
ray had just dropped became a press at a stale spot, or only woke the
pointer.
- The owner keeps the laser through POINTER_RAY_GRACE (0.5 s) of its ray
being down; the other hand takes it by pinching, by raising its ray while
the owner's is down, or after the grace (pointer/helper/rayowner.h).
- A pinch on a ray that dropped within the grace, or is on its way up,
brings it back (fh_ray.h ShowForPinch) and is a ray pinch.
- Aim lock: the press goes along the ray from just before the fingers
began to close, after a pose along it has gone out; a tap that ends
before its press clicks there too.
- Each ray follows one tracked hand through a brief flip of ft-hands'
left/right call.
- Log: owner changes with the reason, the owner's ray down and back, and
one line per pinch saying what it clicked or why not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A gesture that waited for ft-screens' near report is a ray gesture only if
its ray is still up when it starts; a ray pinch's press waits until a
pointer the ray just woke is bound (350 ms). keys-test's pause stub knows
the relay's new calls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With hand interaction on (POINTER_HANDS=1), the hands have to be there
when the dashboard opens during a game (design section 5.5):
- ft-hands: while /run/user/UID/frametop-hands/park exists it processes
no frames (models stay loaded), publishes no hands, ends gestures as
lost, and writes 0 to hands-want every second.
- ft-camd: stops copying the mono frames while hands-want is fresh and 0
(a dead ft-hands' stale file doesn't starve other ring readers).
- The input relay (game_pause.py) leaves the hand services running when
it pauses for a game and parks them instead; it unparks while the
dashboard is open (the helper's new "vrdash 1|0") and on resume.
- hands/README.md: gestures v3, parking, and the hand ray.
- pause-test.py: parking, and "paused" kept off the live helper.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With POINTER_HANDS=1 and POINTER_HAND_RAY=1 (design sections 2, 5):
- Per side, each hands frame goes through fh_ray's gate; the first ray up
drives the pointer (a pinch by the other hand's ray takes it). It wakes
the pointer and claims the laser unless a controller pressed within 2 s.
The mouse takes the pointer back; the ray returns after 1 s of no mouse.
- The pose is the ray's origin and direction (a visible beam); our dot only
where it meets a panel, no catcher in free space. A controller taking
the laser hides our dot until a ray pinch claims it back.
- A ray pinch is the left button, 30 ms after a claim it had to make;
the direction holds until POINTER_RAY_DEADZONE (2.5 deg), then drags.
A middle pinch (fh_gestures v3) is a right click, or scrolling when
held and moved. Gaze pinches and grips are unchanged with the ray down.
- Push and pull while carrying (POINTER_RAY_PUSH): the origin moves along
the ray, distance d0 (h/h0)^3.
- ft-screens' "near <capture_ns> <L> <R>": a gesture waits for the report
on its frame (60 ms at most) and is left to ft-screens if near.
- The ray lets go of a pointer it woke POINTER_RAY_RELEASE after it went;
at once when its side's controller is in use or in a game.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/include/fh_ray.h, shared by the pointer helper and ft-screens (design
sections 2.4-2.5): a body model (torso yaw = head yaw low-passed, at most
30 deg behind; shoulders from it), the ray from the shoulder through the
knuckles (20 ms prediction), the pointing gate (show tests for 120 ms,
hide after 250 ms of a failed keep test; near and "off" hide at once;
held through a ray pinch), and One Euro filters on yaw, pitch and origin.
- pointer/tests/handray-test.cpp: body, gate timing and tests, filter
(just test-c).
- ft-handreplay --ray FILE: per set and hand, what the helper reads.
- pointer/tools/rayeval: the ray over a replay, with the head poses from the
ft-handtest probe's log of the same session (clock matched by the
published capture times): how often it shows, why not, jitter, the ray's
shift into a pinch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The driver's default role is the treadmill (4), hinted from Activate;
the helper's POINTER_ROLE defaults to treadmill. Needs the driver
reinstalled and a SteamVR restart to take effect.
- The "no hand role" release skips a device whose hint is the treadmill.
- Hand gestures: none begins on a side whose controller is in use, or by a
hand at a controller (by position); a held one ends, as lost, when its
side turns controller.
- The moving-controller release no longer applies while a bare hand's ray
drives the pointer (that ray goes only when its own side turns controller).
- update-check.py: the virtual controller's path, and the touch components.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per side, a controller used recently wins, else the bare hand
(pointer/helper/handoff.h, design section 1): touch and presses from the
input relay, motion, charging and tracking from SteamVR, and the tracked
hands from ft-hands' hands file (handsin.h, which also takes the gestures
reader and the head pose history out of ft-pointer.cpp).
- The relay's web socket reader (game_pause.py TouchTracker) sends
"ctl <side> touch 0|1", "ctl <side> press", "ctl <side> gone" and
"ctl ws 0|1" with POINTER_HANDS=1, and the pause state as "paused 1|0".
- The helper logs transitions and sends ft-screens "hands <L> <R> <allowed>"
(hand|controller|off, 0|1) on a change and every second. Nothing acts
on it yet.
- fh_gestures v3 adds the middle pinch slots (written as zero for now).
- Tests: pointer/tests/handoff-test.cpp (just test-c), input/test/touch-test.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The driver takes "role treadmill" and hints its role from Activate, so a
treadmill gets /user/treadmill (from gaze-first 87a6393); the compositor
binding has the /user/treadmill entries. The default role stays the right
hand until the helper side is ready. docs/design.md says why (Treadmill role).
Probes for frame-hands/notes/hand-interaction.md section 7:
- pointer/probe/lasertest (from gaze-first): who has the laser, claim time
- pointer/probe/ctlnoise: controller pose noise, activity level, charging
- vrprobe prints each device's activity level and charging
- input/vrws.py --touch prints touch and value components
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every place an installed Frametop started a program with `distrobox enter dev --` now goes
through scripts/in-box: the pointer and power units, ft-gazed (ft-gaze, ft-eyes), the gaze
check's panel, the desktop's ft-screens, the remote desktop's krdp and VNC, and the three
settings apps. in-box starts the container in a scope of its own (container-up.sh, which now
takes the container's name) and becomes `distrobox enter BOX --`, so callers behave as before.
BOX is "dev" for a source install. A release names its own container in .frametop-release,
so the same tree runs from an image without editing nine call sites. FRAMETOP_BOX overrides
both. Development tools (frame.sh, the headless test, the gaze probe and lab) and hand
tracking, which install.sh doesn't install, keep using the dev container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
screens/build.sh conflicted: experimental added remote.c to ft-screens, and this branch
links OpenVR through scripts/openvr.sh. Both kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A remote display is a screen of ft-screens whose picture comes from another computer,
streamed by Vibepollo with Moonlight's protocol. ft-screens (screens/remote.c) runs one
ft-stream per remote screen (101 and up) over a socket pair: ft-stream (stream/, GPLv3:
moonlight-common-c and moonlight-embedded's libgamestream, pinned) decodes on the iris
decoder into a ring of three RGBA buffers that ft-screens shows as a panel with a
screen's controls, layout, profiles, curve, pins and hand cutouts. Mouse, keys and scroll
go back to the host; a button's release goes through the stream it was pressed on, as
Vibepollo wants, and a press carried onto another remote panel moves the pointer there.
A panel out of sight gets 10 fps; the host's sound plays from one stream per host.
ft-stream signs in with an API token and the host's pinned public key
(stream/host.cpp), and chooses its path: a Steam Link dongle on the Frame's hotspot when
it answers, or the network (the layout's "route": auto, network or dongle). ft-layout
gets "remote" commands (list, add, connect, disconnect, host route and direct), starts
the connected displays with the desktop and keeps disconnected ones parked. ft-gaze and
ft-pointer know the remote panels. ft-screens now shuts its VR side down before its
streams and waits for SteamVR before exiting (a vrcompositor crash at quit, in standby).
docs/remote-displays.md has the design, the spikes and the test results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/openvr.sh is the one place for the OpenVR SDK the programs build
against: the v2.15.6 headers, fetched into build/include and checked by
sha256, and the API library. ft-pointer, the ft_pointer driver, and
ft-powerd now build against those headers like ft-screens and ft-gaze,
instead of the 2.1.0 copy in SteamVR's samples, which has no public tag
an image could pin. The library stays SteamVR's own; OPENVR_LIB links
another copy (the image has no SteamVR), and the rpath still names
SteamVR's folder first.
FRAME_IN_BOX=1 tells the scripts they already run in the build container,
so pack/Containerfile can run them instead of keeping a second copy of
every compile line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In gaze mode a mouse press is held back while you aim, and its release
clicks. A release of the right button sets pressRight with the click, so
the click goes out as a right one. When the pointer goes off in the same
loop (the relay's stand_down sends the release, then "hide", as Frametop
pauses), the click is dropped, but pressRight stayed set. The next press
after resume, from the mouse, Meta+J or a pinch, then went out as a
right click.
Dropping the click now forgets pressRight with it. A "hide" read a loop
after the release still comes too late to stop the click; that needs the
helper's socket to be full as the pause starts, and is noted in the
relay's stand_down.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any "btn" or "scroll" from the relay woke a pointer that was off. When
Frametop pauses for a game, the relay's stand_down sends the releases of
the clicks held into the pause, and with its pointer already off (the
idle timeout or pointer_toggle during a press) it used to send them with
no "hide" after. The helper woke on them, connected the virtual
controller and took the right hand role during the game.
Now a button release or a scroll back to 0 0 doesn't wake it. The
release is still handled as before (it ends its press, or goes to the
driver), so the driver doesn't reconnect later with the button down.
Outside a pause this changes one case: a mouse button let go after a
moving controller released the pointer mid-press. That release used to
reconnect the controller, recenter it and release there; now it updates
the disconnected controller, and ft-screens still gets its "up".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The desktop has its own XDG_CONFIG_HOME (~/.config/frametop), and SteamVR's tools and OpenVR
read their path registry from there. So from a terminal in the desktop:
- pointer/driver/install.sh (install.sh step 4, every reinstall or update from Konsole) wrote
a new registry, ~/.config/frametop/openvr/openvrpaths.vrpath, with only our driver and
"runtime": null, and never registered the driver with SteamVR. That stray file then hid
SteamVR from every OpenVR program started in the desktop.
- scripts/update-check.py (and so doctor.sh and report.sh) reported OpenVR "can't connect to
SteamVR as a background app" (VRInitError_Init_PathRegistryNotFound, or
InstallationNotFound with the stray file) and "vrcmd --overlays lists no overlays". A user's
report of 2026-10-05 showed both, with SteamVR and Frametop's services fine.
- ft-layout's vrcmd calls (the gamescope backend) found no overlays.
They now run with XDG_CONFIG_HOME=~/.config: the driver installer (install, uninstall, probe,
aimhere), update-check.py (for all its checks: nothing there reads the desktop's own config),
report.sh's vrpathreg show, and ft-layout's vrcmd. report.sh doesn't set it for the whole
report, since session/fix-panels.py --check reads the desktop's Plasma config through it.
The driver installer also removes the stray registry (only vrpathreg's, with no runtime), and
update-check.py warns about one. And vrpathreg runs `xdg-open vrmonitor://driverinstalled` to
tell SteamVR's desktop monitor, which the Frame doesn't have: in the desktop that showed "could
not read file vrmonitor://driverinstalled" on every install. A stand-in xdg-open on its PATH
takes it, so install.sh no longer filters the step's output.
Tested in a fake HOME with a copy of SteamVR's registry, a stray one, and the desktop's
XDG_CONFIG_HOME: the new installer removes the stray file, registers the driver in the copy,
and never reaches xdg-open; the old one wrote the stray file, left the copy alone, and ran
xdg-open. update-check.py from that environment: OpenVR and vrcmd ok (the old one fails both);
its stray-registry warning fires on a seeded file. ft-layout's vrcmd: 119 overlays (was 0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper now reads SteamVR's list of panels every 20 s instead of every second, so a panel
made in between (a floating window's menu, frametop.float.N.sub.K, or the Frametop keyboard
the first time it opens) couldn't be clicked with the mouse until the next read. ft-screens
now sends "overlay <key>" to @ft_pointer_helper right after it makes one, and the helper adds
it to its list at once (only frametop.* keys). An older helper ignores it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The panel-edge test read visible[edgeKey], and when the last panel the cursor touched was
gone from the overlay list, that added it back as hidden. The map then had more entries than
there are handles, which made the 50 ms visibility poll run every frame, and since the last
commit it also counted as a visibility change each time, so unchanged frames were never
reused. The edge test now looks the key up without adding it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay sent the helper one "move" per SYN_REPORT, so a 1000 Hz mouse sent 1000 datagrams
a second to a helper whose loop runs every 8 ms, and each one went through a dozen sscanf
and strncmp tests in the helper before reaching the move handler. In a 200 ms test at
1000 Hz, 149 reports now make 45 moves with the same total.
flush() on a report now sends only once 4 ms have passed since the last move; tick() sends
the rest when due, and the select timeout shrinks to match. Buttons and the gaze
keys still flush first, unconditionally, so a click lands where the pointer was. In the
helper, "move" is now tested first in the command dispatch, and its handling is one lambda.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Handle() held the state lock through a chain of up to a dozen sscanf calls per command, and
RunFrame, which vrserver calls every frame, takes the same lock, so a burst of commands
(about 116 poses a second, plus moves and buttons) could hold up vrserver's frame. Commands
are now parsed into locals first, and the lock is held only to store the result.
RunFrame also called UpdateBooleanComponent six times and UpdateScalarComponent twice every
frame, and TrackedDevicePoseUpdated every frame even while disconnected. Components now go to
SteamVR only when they change (all of them on the first frame). The pose still goes out every
frame while the device is connected, as a tracked device's should; the disconnected pose goes
out once. The helper now sends a pose only when it changes, so the comment says the driver
keeps the last one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every frame (about 116 a second) the helper tested the cursor ray against every visible
overlay twice with ComputeOverlayIntersection, set the dot's alpha, width, transform and
visibility (five calls into SteamVR), and sent the driver a pose datagram, even with the
mouse and the head still.
Now a frame reuses the last collision result when the mouse, the anchor (1 mm) and the
eye (5 mm) haven't moved and no overlay showed, hid, or changed handle. The passes still run
at least every 100 ms, since overlays move on their own (a floating window's controls follow
it), and always while dragging. The dots' setters go to SteamVR only when their value changes:
the placement when the dot moved 0.2 mm or the eye 5 mm, which turns or resizes it by well
under 1%, and the width on a 0.5% change. The plain pose goes to the driver only when the
laser's origin moved 0.2 mm or its direction 0.04 deg (0.1 mm where it lands, 15 cm on), and
at least every 100 ms; the driver keeps the last pose and reports it every frame. A tilt's
pose, a placement, or waking sends the next one regardless.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The main loop slept a fixed 8 ms, about 116 wakeups a second, whether the pointer was awake
or not, and every second it looked up every overlay's handle and read a string property from
all 64 device slots to find its own device.
With the pointer off and hand gestures off, the loop now waits in poll() on its command
socket for up to 250 ms, or 20 ms while mapped Frame controller buttons are being read
(SteamVR input has no event to wait for). A mouse command ends the wait at once. The
headset's activity level, the game check, and the "vrgame" and "gazeawake" repeats keep
going at that pace. The 50 ms visibility poll and the 1 s handle lookups run only while the
pointer is awake, and waking forces both. The device index is looked for only while it's
unknown, and again after SteamVR activates or deactivates a device. The HMD pose history is
kept only with hand gestures on, its one user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
