User 2026-10-11, gaze + hands becoming the main input:
- Right-drag: an index tap on one hand, then an index pinch on the other within 1 s,
holds the right button at the gaze lock; that hand drags until it ends
(POINTER_GAZE_RIGHT_DRAG). It replaces the chord (POINTER_GAZE_CHORD=0).
- Three-finger scroll: the thumb on both the index and middle tips (3 cm) scrolls where
you looked as the hand moves, like the ray's middle-pinch stick (POINTER_GAZE_SCROLL).
- The fist guard (a hand closing into a fist: no click) now covers gaze and chord
pinches too, from each hand's own frame (not the ray's).
- A hand's laser shows only its dot: the beam starts just short of it, as the eye's
does, held still through a carry (POINTER_RAY_BEAM=0; SteamVR's beam width is one
global setting it reads only at start).
- The front slot gives a screen up to manual controls: grab that, a push, flick or
throw, or a carry by hand or laser releases it (it moves like any other and stays
where you put it); the next stroke puts it home and brings the next.
- The lazy susan counts only palm-led strokes (the return stroke does nothing; the
other hand palms the other way; HANDS_SWEEP_BACK=1 for both).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lazy susan (the third live test): a brought screen comes 1 m ahead, as big as its
resolution needs for text to read: its logical pixels (KWin's scale undone, now passed
to vr.cpp) at 26 px/deg, at most 100x55 deg keeping its shape, at least 40 deg wide,
curved round you past 60 deg (grabthat::ReadableSize; SCREEN_SPIN_DISTANCE, _PPD,
_MAX_DEG, _MAX_TALL_DEG). Its curve comes back on the way home. The strokes cycled
104, 1, 104 with three screens up: a screen still flying home read as straight ahead;
each one is now taken where it lives (home, or where it's going).
Accidental clicks: most were pinches with no laser up pressing where the pointer was
last left ("no ray up: a press where the pointer is", a dozen in two minutes). With
gaze off they now only wake the pointer (POINTER_PINCH_NO_RAY=1: the old air mouse).
The hand's laser shows only with the thumb and index within 6 cm (gone past 10;
POINTER_RAY_READY), and a fist drops it: fh_ray's "ready" and "open" tests, the fist
read as the index and the other three both curled under 1.1 (FistCurl; measured on 47k
flagged hands: the other three alone read as a fist in a fifth of the pinches). A ray
pinch made while the hand closes into a fist doesn't click. Input Settings switches for
all three (POINTER_ ones reload the helper).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The loop reads its clock at the top and ft-hands' files later in the pass.
A frame written in between has publish_ns after tnow, and the unsigned
"now - publish_ns" in HandsInRoom::Live wrapped round: the feed read as
stale for that one pass, both sides went "off", and the "hands" word to
ft-screens flickered to "off off" and back. 2026-10-10 20:30-20:37: 24
flickers in 7 minutes, each with ft-hands publishing at full rate (pinches
logged at the same moments), each ending ft-screens' holds and the ray.
feedage.h: AgeNs/Fresh, age 0 for a frame from after "now". Live and the
gestures check that ends holds ("hand tracking stopped") use them.
pointer/tests/feedage-test.cpp replays the loop against 30, 15 and 1 Hz
publishing: the old check calls 28-2408 passes stale, the new one none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User, 2026-10-10: Vision Pro-style resize from all four corners. Only the
bottom right worked by laser (the .resize handle), and a resize kept the
screen centred.
- One resize handle per panel still (SteamVR's 128 overlays are tight:
a third remote display once didn't fit), but it moves to the corner
being aimed at (vr.cpp UpdateCorner): a laser's ray (a controller's,
or the pointer helper's device, which the gaze and the hand ray
drive) landing within a fifth of the screen's height (5-20 cm) of a
corner, or a hand's pinch at one (handinput's corners()). Its texture
turns for that corner, it brightens the closer the aim (from the
controls' idle 0.55 to 1), and it's lit blue while resizing. With no
aim it lingers 0.4 s, then goes back to the bottom right.
- Any corner resizes and the opposite one stays put: screens and
remote displays keep their aspect (the corner follows the diagonal,
ResizeFromCorner, on the cylinder for curved ones: SurfacePose),
floating windows take the pixels from that corner (ft-floatd's new
size is placed keeping the opposite corner, keepX/keepY; every corner
of a window resizes now, the title bar's ends included).
- Near hands: a pinch at any corner resizes the same way (the carry
now gives the pose as well as the width).
- Gaze pinches (gazehand.h): with the pointer on a screen's resize
handle or grab bar, the pinch is a press after 0.12 s still
(POINTER_GAZE_CONTROL_HOLD), not 0.5: nothing to click there, and the
hand drags the resize. The chrome overlays are announced to the
helper as they're made, so its ray and the gaze find a new remote
display's handle at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts:
- ft-pointer.cpp, the gaze pinch comment: typing's "the rules for all
gestures hold (..., typing, ...)" kept, gazechord's chord and
POINTER_GAZE_PINCH paragraphs added after it, and a line that a hand
typing holds back makes no chord.
- ft-pointer.cpp, the hands settings: gazechord's gazePinchOn, nearOn,
handsTrack kept; typingHold dropped for typing's typing::Params.
- ft-pointer.cpp, mayPoint: both conditions (handsOn, and not a side
typing holds back).
- ft-pointer.cpp, beginGazePinch: gazechord's busy line (a chord isn't
busy), without the old sinceKey/typingHold lines. The chord's pinch
goes through typing.h's Why in the gesture loop before startGesture,
like any pinch, so a hand typing holds back can't make one.
- docs/reference.md settings line: typing's and reach's keys, plus
POINTER_GAZE_PINCH.
- hands/README.md near input: typing's hold-back sentence and
gazechord's HANDS_NEAR=on.
- session/frametop.conf.example: reach's HANDS_GRAB_ROTATE* and
HANDS_GRAB_THAT* lines, then gazechord's HANDS_NEAR line.
Reconciled: gazehand's Situation loses sinceKey and typingHold, and
Refused its "ignored (typing)": the helper had stopped setting them on
hands-typing (typing.h holds hands back first), so that guard could
never fire. gazehand-test's key checks now drive typing.h: keys hold a
hand back (2 s now, was 1), POINTER_PINCH_TYPING=0 is no guard, and a
chord's pinch by a hand on the keys is held back while one lifted off
them goes ahead. The chord's own pinch counts as a gesture held for
typing's "busy" (free if out of the keyboard zone when keys come), as
the left's hold does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An index pinch by the other hand while a gaze pinch is held presses the
right button until that pinch ends; a first pinch still held back becomes
a real left press at once, where the pointer is. ChordButton times the
right press (30 ms after a left the chord made, a tap clicks, a lost hand
releases). Tests for the chord sequences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The posture's and a raise's start used -1 for none, so a replay of a flag
(times before it negative) never saw the posture. Found replaying the
2026-10-10 flags through it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper feeds typing.h the relay's keys and each hands frame. A hand it
holds back makes no pinch, middle pinch or grip (ray, gaze or plain), and has
no ray: one that drove the pointer drops it (to the mouse or the gaze, not
the controllers) and can't claim it again until lifted off the keys. A hand
pointing or holding a gesture out of the keyboard zone isn't held back by
the other hand's keys. ft-screens gets "typing <L> <R>" on a change and every
second. The old 1 s guards (beginHold, gazehand's) give way to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A key holds every hand back for POINTER_PINCH_TYPING (now 2 s); a hand in the
keyboard zone while typing stays held back until it rises POINTER_TYPING_RAISE
above where it typed (or turns its palm); both hands in the keyboard posture
count as typing within POINTER_TYPING_RECENT of a key. Tests with made-up keys
and hands placed as in the 2026-10-10 flags and the desk take.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ideas from StereoKit (MIT, 07b9f0d), credited where a formula comes from it:
- The torso faces the head's forward counted twice plus the direction to
each wrist seen (POINTER_RAY_TORSO_WRISTS, on), so looking aside while
pointing swings the shoulders, and the ray, less. probe-moves replay: the
shoulder's share of a still hand's ray motion 20 -> 18%, p90 2.0-3.7 ->
1.7-3.6 deg/s; the tap capture: nothing worse.
- The ray can turn toward the hand's own direction, from the palm's axes
(wrist, middle, ring and little knuckles) tilted -30/8 deg
(POINTER_RAY_HAND_WEIGHT, 0). Off: every weight measured worse (at 0.3:
judder 0.07-0.14 -> 0.11-0.33 deg rms per 90 Hz tick, raw jumps over
3 deg 159 -> 330). The gate still tests the shoulder's ray.
- The knuckle point can be pinned to the palm's axes during a pinch, easing
back on the steady shoulder's schedule (POINTER_RAY_ANCHOR, 0). Off: our
pinch point is the knuckles already, and pinned it shook as much or more
and kicked no less on release, fitted or not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ray's One Euro filters (new hands/include/fh_smooth.h) run in continuous time and the
helper reads the ray every loop, so the pose sent to ft_pointer moves at about 100 Hz
instead of jumping at the tracker's 15-30 Hz (the dashboard's dragger made those jumps
jerky). The direction is filtered on the sphere. On the probe-moves replay the pose's
second difference per 90 Hz tick went from 0.90-0.96 deg rms to 0.24-0.26 at the same lag.
While a ray pinch is held the ray pivots on the shoulder as it was at the press, so looking
around doesn't move what's dragged; the reach for push and pull is filtered. A pinch's
first 0.3 s takes 6 deg of turn before it drags. One failed read of the gestures file no
longer ends a drag, and the log says why a drag ended or the ray went off.
Settings: POINTER_RAY_SMOOTH_MINCUTOFF, POINTER_RAY_SMOOTH_BETA, POINTER_RAY_SMOOTH_HOLD,
POINTER_RAY_STEADY, POINTER_RAY_TAP, POINTER_RAY_TAP_DEADZONE. rayeval reads publish dumps
and reports the smoothness at the display's rate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With gaze mode and POINTER_HANDS on, the gaze is the pointer and the hand ray
stays off (POINTER_GAZE_HAND_RAY=1 brings it back). A pinch clicks where you
looked 75 ms before ft-hands saw it; held and moved past 4 deg of hand
movement (POINTER_GAZE_PINCH_DEADZONE) it nudges the pointer at 0.3x
(POINTER_GAZE_PINCH_GAIN), and the correction is a gaze lesson. Held still for
POINTER_GAZE_HOLD it's a press the hand drags. A middle pinch right-clicks at
the gaze. One log line per pinch. Rules in pointer/helper/gazehand.h, tested
in pointer/tests/gazehand-test.cpp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live test 2026-10-09: taps took several tries and the laser jumped between
hands. The owner's ray dropping for a moment (closing the fingers turns the
hand) handed the laser to the other raised hand at once, for good; a pinch
by the other hand pressed where the old owner's ray pointed; a pinch whose
ray had just dropped became a press at a stale spot, or only woke the
pointer.
- The owner keeps the laser through POINTER_RAY_GRACE (0.5 s) of its ray
being down; the other hand takes it by pinching, by raising its ray while
the owner's is down, or after the grace (pointer/helper/rayowner.h).
- A pinch on a ray that dropped within the grace, or is on its way up,
brings it back (fh_ray.h ShowForPinch) and is a ray pinch.
- Aim lock: the press goes along the ray from just before the fingers
began to close, after a pose along it has gone out; a tap that ends
before its press clicks there too.
- Each ray follows one tracked hand through a brief flip of ft-hands'
left/right call.
- Log: owner changes with the reason, the owner's ray down and back, and
one line per pinch saying what it clicked or why not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/include/fh_ray.h, shared by the pointer helper and ft-screens (design
sections 2.4-2.5): a body model (torso yaw = head yaw low-passed, at most
30 deg behind; shoulders from it), the ray from the shoulder through the
knuckles (20 ms prediction), the pointing gate (show tests for 120 ms,
hide after 250 ms of a failed keep test; near and "off" hide at once;
held through a ray pinch), and One Euro filters on yaw, pitch and origin.
- pointer/tests/handray-test.cpp: body, gate timing and tests, filter
(just test-c).
- ft-handreplay --ray FILE: per set and hand, what the helper reads.
- pointer/tools/rayeval: the ray over a replay, with the head poses from the
ft-handtest probe's log of the same session (clock matched by the
published capture times): how often it shows, why not, jitter, the ray's
shift into a pinch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per side, a controller used recently wins, else the bare hand
(pointer/helper/handoff.h, design section 1): touch and presses from the
input relay, motion, charging and tracking from SteamVR, and the tracked
hands from ft-hands' hands file (handsin.h, which also takes the gestures
reader and the head pose history out of ft-pointer.cpp).
- The relay's web socket reader (game_pause.py TouchTracker) sends
"ctl <side> touch 0|1", "ctl <side> press", "ctl <side> gone" and
"ctl ws 0|1" with POINTER_HANDS=1, and the pause state as "paused 1|0".
- The helper logs transitions and sends ft-screens "hands <L> <R> <allowed>"
(hand|controller|off, 0|1) on a change and every second. Nothing acts
on it yet.
- fh_gestures v3 adds the middle pinch slots (written as zero for now).
- Tests: pointer/tests/handoff-test.cpp (just test-c), input/test/touch-test.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
