From 580004bb00de6bd2990d1d59abf850faaf8478e9 Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Thu, 1 Oct 2026 10:21:21 -0600 Subject: [PATCH] Gaze checks: the headset going on is eyes coming back Steam's eyetracking.txt writes "HMD on" every minute or so with nobody in the headset, and SteamVR said it was worn for 12 hours straight, so the quick check opened about 40 times an hour at an empty headset. It now opens when SteamVR's tracker sees eyes for 3 s after none for 3 s, and closes when they're gone 2 s. The log reader also drops repeated "HMD on" lines and sub-second offs, so lessons stop counting as from an older wear every minute. Co-Authored-By: Claude Opus 5.5 --- gaze/README.md | 2 +- gaze/gazecal.py | 12 ++++++++-- gaze/gazecheck.py | 59 +++++++++++++++++++++++++++-------------------- 3 files changed, 45 insertions(+), 28 deletions(-) diff --git a/gaze/README.md b/gaze/README.md index 91ee7a9..1289695 100644 --- a/gaze/README.md +++ b/gaze/README.md @@ -30,7 +30,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste - The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse button or key combination mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on, until a controller is picked up. The dot shows all the time (`POINTER_GAZE_DOT=moving`: only while the mouse moves it, while a press is held, and as a pulse when you click). Gaze mode works with the mouse and the keyboard, not the controllers ([docs/gaze-controllers.md](../docs/gaze-controllers.md) explains why). - **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; let go of either to drop. - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the probe's one-dot check does the same). The helper only sends nudges up to `POINTER_GAZE_NUDGE_MAX` (8 degrees), and right after putting the headset back on our tracker can be further off than that. If so, raise it for a moment, or do a quick check. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them. -- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens 3 s after you put the headset on, and when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. +- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always. Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was. diff --git a/gaze/gazecal.py b/gaze/gazecal.py index f823c5b..1777b92 100644 --- a/gaze/gazecal.py +++ b/gaze/gazecal.py @@ -544,6 +544,9 @@ class SteamEyeLog: each time the headset goes on ("HMD on"): the eye model starts over then too.""" PATH = Path.home() / ".local" / "share" / "Steam" / "logs" / "eyetracking.txt" + # It writes "HMD on" again every minute or so while on, and flickers off for 0.01-0.3 s: + # only an on after an off of BLIP or longer counts. + BLIP = 1.5 def __init__(self): self.pos = 0 @@ -595,9 +598,14 @@ class SteamEyeLog: self.starts.append(t) restarted = not first elif "HMD on" in line: - self.wears.append(t) + off = bool(self.offs) and (not self.wears or self.offs[-1] > self.wears[-1]) + if off and self.wears and t - self.offs[-1] < self.BLIP: + self.offs.pop() # the sensor flickering: it never came off + elif off or not self.wears: + self.wears.append(t) elif "HMD off" in line: - self.offs.append(t) + if not self.offs or (self.wears and self.wears[-1] > self.offs[-1]): + self.offs.append(t) elif "Accept usercal" in line: self.accepts.append(t) elif "Reject usercal" in line: diff --git a/gaze/gazecheck.py b/gaze/gazecheck.py index 13b7583..f72b961 100644 --- a/gaze/gazecheck.py +++ b/gaze/gazecheck.py @@ -2,12 +2,14 @@ (gaze/panel/ft-gazepanel; ft-gazed runs it). Every kind is made of dots shown at head-relative directions: look at each one. - quick one dot in the middle of your view. It opens DON_DELAY after the headset goes on - (SteamVR's eye tracking log says when), when our own tracker asks for a click (its - "reseat": the headset may sit differently on your face now), at most once every - QUICK_COOLDOWN, and on "quickcal" (Frametop Input Settings, or a mouse button or key - combination mapped to Gaze quick check). Ignored, it closes after QUICK_TIMEOUT and - changes nothing. + quick one dot in the middle of your view. It opens when the headset goes on: eyes seen + for DON_DELAY after none for AWAY_MIN (SteamVR's tracker's variance for an eye under + EYE_LOST). SteamVR's "HMD on" can't say: it repeats every minute or so, and it can + stay on for hours with nobody in the headset. It also opens when our own tracker asks + for a click (its "reseat": the headset may sit differently on your face now), at most + once every QUICK_COOLDOWN, and on "quickcal" (Frametop Input Settings, or a mouse + button or key combination mapped to Gaze quick check). Ignored, it closes after + QUICK_TIMEOUT and changes nothing. five the middle and four around it, when the first FIVE_COUNT lessons after a quick check were all over FIVE_LIMIT degrees off: the quick check didn't fix it. full the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil @@ -46,7 +48,7 @@ import sys import time from pathlib import Path -from gazecal import DEFAULT_MODEL, STATE, steady_samples +from gazecal import DEFAULT_MODEL, EYE_LOST, STATE, steady_samples REPO = Path(__file__).resolve().parents[1] PANEL_PROG = REPO / "gaze" / "build" / "ft-gazepanel" @@ -66,7 +68,9 @@ ACCEPT_SPREAD = 2.5 # degrees: a capture asked for (calaccept) takes this muc DOT_TIMEOUT = 8.0 # seconds a dot of five or full waits; then it's skipped QUICK_TIMEOUT = 6.0 QUICK_COOLDOWN = 120.0 -DON_DELAY = 3.0 +DON_DELAY = 3.0 # seconds of eyes after AWAY_MIN without: the headset went on +AWAY_MIN = 3.0 +EYES_GONE = 2.0 # seconds without eyes that close a check: the headset came off FIVE_LIMIT = 2.0 FIVE_COUNT = 3 DONE_PAUSE = 0.35 # seconds the filled dot shows before the next @@ -151,8 +155,9 @@ class Checks: self.gaze_heard = 0.0 self.want_full_until = 0.0 # gaze mode came on before the tracker said whether it's calibrated self.last_quick = 0.0 - self.worn_seen = svc.steam.worn() - self.don_at = None + self.seen_at = 0.0 # eyes last seen (SteamVR's tracker's variance for them, "unc") + self.away = True # no eyes for AWAY_MIN: their coming back is the headset going on + self.back_since = None self.reseat_seen = False self.after_quick = None self.panel_proc = None @@ -254,9 +259,12 @@ class Checks: return True return svc.models[svc.source].samples > 0 + def eyes_seen(self, within=1.0): + return time.monotonic() - self.seen_at < within + def can_run(self): - """The headset is on and the tracker is sending.""" - return self.svc.steam.wearing() is not False and time.monotonic() - self.svc.last_sample < 2 + """Someone's in the headset and the tracker is sending.""" + return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2 def on_gaze_on(self): cal = self.calibrated() @@ -323,6 +331,11 @@ class Checks: c["run"], c["accept"], c["done_at"] = [], False, None def on_sample(self, s): + unc = (s["src"].get("mmap1") or {}).get("unc") + if unc and min(unc) <= EYE_LOST: + self.seen_at = time.monotonic() + if self.away and self.back_since is None: + self.back_since = self.seen_at c = self.check if not c or c["done_at"]: return @@ -575,7 +588,7 @@ class Checks: else: self.advance() return - if self.svc.steam.wearing() is False: + if not self.eyes_seen(EYES_GONE): self.close("the headset came off") return if c["done_at"]: @@ -602,17 +615,13 @@ class Checks: self.want_full_until = 0.0 if cal is False and self.gaze_on: self.start("full", "gaze mode came on without a calibration") - worn = svc.steam.worn() - if worn and worn != self.worn_seen: - self.worn_seen = worn - self.don_at = worn + DON_DELAY - # Both wait for the tracker to send (it takes a moment after the headset goes on). - if self.don_at and time.time() >= self.don_at: - if self.can_run(): - self.don_at = None - self.auto_quick("the headset went on") - elif time.time() > self.don_at + 20: - self.don_at = None + if not self.eyes_seen(AWAY_MIN): + self.away, self.back_since = True, None + elif self.back_since is not None and not self.eyes_seen(): + self.back_since = None # gone again before DON_DELAY + elif self.back_since is not None and now - self.back_since >= DON_DELAY: + self.away, self.back_since = False, None + self.auto_quick("the headset went on") if svc.kind == "own" and now - svc.own_at < 5: reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values()) if not reseat: @@ -623,7 +632,7 @@ class Checks: def status(self): c = self.check - st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), + st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(), "panel": self.panel_proc is not None, "last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None} if c: