diff --git a/gaze/README.md b/gaze/README.md index 1289695..45854d3 100644 --- a/gaze/README.md +++ b/gaze/README.md @@ -28,7 +28,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste Looks down past the screens (under 20 degrees down, on no Frametop screen: a glance at the keyboard) aren't sent, so the pointer stays where it was instead of following you down, and eyes lost there aren't counted. It drops blinks (both eyes closing or lost), smooths with a fixation lock, and sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset. - The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse button or key combination mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on, until a controller is picked up. The dot shows all the time (`POINTER_GAZE_DOT=moving`: only while the mouse moves it, while a press is held, and as a pulse when you click). Gaze mode works with the mouse and the keyboard, not the controllers ([docs/gaze-controllers.md](../docs/gaze-controllers.md) explains why). -- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; let go of either to drop. +- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, but up to 30 degrees whatever `POINTER_GAZE_NUDGE_MAX` says: a keyboard correction is always meant). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; let go of either to drop. - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the probe's one-dot check does the same). The helper only sends nudges up to `POINTER_GAZE_NUDGE_MAX` (8 degrees), and right after putting the headset back on our tracker can be further off than that. If so, raise it for a moment, or do a quick check. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them. - **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always. diff --git a/pointer/helper/ft-pointer.cpp b/pointer/helper/ft-pointer.cpp index 3f13266..37d96c2 100644 --- a/pointer/helper/ft-pointer.cpp +++ b/pointer/helper/ft-pointer.cpp @@ -181,7 +181,9 @@ // your view, so turning your head carries it onto what you meant (past // POINTER_HEAD_DEADZONE, 0.5 deg, so a still head doesn't wobble it). The release clicks there // (left, or right for gaze_right), and a correction is a lesson for the gaze tracker, as with -// the mouse. Held still for POINTER_GAZE_HOLD instead, it's a real press, and the head drags. +// the mouse, but up to kKeyNudgeMax (30 deg) rather than POINTER_GAZE_NUDGE_MAX: a keyboard +// correction is always meant, and our tracker can be 12 deg off after the headset goes on (live, +// 2026-10-01: every correction was dropped). Held still for POINTER_GAZE_HOLD instead, it's a real press, and the head drags. // A quick tap (let go within POINTER_KEY_TAP, 0.25 s) clicks where the dot was at the press, // whatever the head did meanwhile, and tells the gaze tracker it was right there (a lesson // with no correction). Pressing gaze_right while gaze_left aims (Meta+K with Meta+J held) @@ -905,6 +907,9 @@ int main() { bool pressRight = false; std::string heldButton = "trigger"; bool confirmLesson = false; // a keyboard click's quick tap: a lesson with no correction (see the top) + // A keyboard click's correction: always meant, so learned up to kKeyNudgeMax (see the top). + bool keyLesson = false; + constexpr double kKeyNudgeMax = 30; // The gaze calibration panel is up until then ("calpanel 1"; see the top); calOpened: it just // came up, so a press in progress ends without a click. Clock::time_point calPanelUntil{}; @@ -914,8 +919,11 @@ int main() { // its own screens, but only we know when one lands on another panel. SendTo(out, "ft_screens", "click " + (lastHit.empty() ? std::string("-") : lastHit)); // A click after nudging the gaze-placed pointer: the nudge is a lesson (see the top). + const double nudgeMax = keyLesson ? kKeyNudgeMax : gazeNudgeMax; + if (debug && gazeOn && nudging && !gazeOwns && nudgeMoved > nudgeMax) + std::printf("gaze nudge %.2f deg: over %.0f, not learned\n", nudgeMoved, nudgeMax); if (gazeOn && nudging && !gazeOwns && havePoint && Clock::now() - nudgeAt < std::chrono::seconds(10) && - (confirmLesson || (nudgeMoved >= 0.2 && nudgeMoved <= gazeNudgeMax))) { + (confirmLesson || (nudgeMoved >= 0.2 && nudgeMoved <= nudgeMax))) { const Vec3 d = RotateInverse(nudgeHead, Normalize(lastPoint - Position(nudgeHead))); char msg[160]; std::snprintf(msg, sizeof msg, "lesson %.3f %.3f %.3f %.3f", nudgeRawHy, nudgeRawHp, @@ -924,7 +932,7 @@ int main() { if (debug) std::printf("gaze %s (nudged %.2f deg)\n", msg, nudgeMoved); if (debug) std::fflush(stdout); } - nudging = confirmLesson = false; + nudging = confirmLesson = keyLesson = false; leftHeld = true; dragDistance = lastDistance; pressKey.clear(); @@ -1523,6 +1531,7 @@ int main() { } else { clickPress = true; // the click is on the release, where the pointer is now pressRight = hold.right; + keyLesson = hold.src == Src::Head; gazeBack = nudgeMoved < 0.2; } } @@ -1656,6 +1665,7 @@ int main() { headAngles(hold.refYaw, hold.refPitch); gazeBack = gazeOn; pressRight = false; + keyLesson = true; pressLeft(); if (debug) std::printf("hold key left: pressed (right key)\n"); if (debug) std::fflush(stdout); @@ -2077,7 +2087,7 @@ int main() { // A held-back press (see the top): held still long enough, it's a real press (a drag); // released, it's a click where the pointer is now (this frame's pose has gone out). - if (!active) aimHeld = clickPress = aimHand = false; // released meanwhile: nothing to click + if (!active) aimHeld = clickPress = aimHand = confirmLesson = keyLesson = false; // released meanwhile: nothing to click if (aimHeld && !aimHand && nudgeMoved < 0.2 && tnow - aimSince >= std::chrono::duration(gazeHold)) { aimHeld = false; gazeBack = true;