diff --git a/docs/reference.md b/docs/reference.md index f581f35..242032e 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -206,7 +206,7 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke - Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held. - The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual. - A correction before a click teaches the gaze service the tracker's error there. A correction bigger than `POINTER_GAZE_NUDGE_MAX` (55 degrees) isn't learned; it opens a quick check instead. -- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode coming on without a calibration opens Calibrate by itself. +- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it. [gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool. diff --git a/gaze/README.md b/gaze/README.md index f42783d..8622bad 100644 --- a/gaze/README.md +++ b/gaze/README.md @@ -33,7 +33,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste - **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time. - **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it. - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them. -- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. +- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always. Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was. diff --git a/gaze/ft-gazectl b/gaze/ft-gazectl index 8a43637..8b4d83a 100755 --- a/gaze/ft-gazectl +++ b/gaze/ft-gazectl @@ -36,8 +36,17 @@ def main(): if r is None: sys.exit("the pointer helper isn't running") print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)") - if ask("ft_gazed", "status", 0.3) is None: + st = ask("ft_gazed", "status", 0.3) + if st is None: print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow") + elif r == "ok on": + try: + calibrated = (json.loads(st).get("checks") or {}).get("calibrated") + except ValueError: + calibrated = None + if calibrated is False: + print("note: no calibration for this eye tracker yet; it opens in the headset " + "(if it can't, ft-gazectl status says why: checks.problem)") elif cmd in ("status", "forget", "reload"): r = ask("ft_gazed", cmd) if r is None: diff --git a/gaze/gazecheck.py b/gaze/gazecheck.py index c37f221..724d58f 100644 --- a/gaze/gazecheck.py +++ b/gaze/gazecheck.py @@ -17,9 +17,12 @@ directions: look at each one. size, and the tracker's error with it, changes with brightness), each the middle and a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots) RING degrees out, half that in the middle round, turned 20 degrees a round. It opens - when gaze mode comes on without a calibration for the tracker in use, and on - "calibrate". Frametop's screens hide while it runs. Quitting it while there's still - no calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it. + whenever gaze mode is on without a calibration for the tracker in use and someone's + in the headset, and on "calibrate". One that closes unfinished (ignored, too few + dots) opens again only once the headset comes off and on, or gaze mode off and on. + Frametop's screens hide while it runs. Quitting it while there's still no + calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it. + Why gaze mode can't work yet goes in the status ("problem"), for Input Settings. fit the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was @@ -90,6 +93,7 @@ ROUND_BG = (0.03, 0.33, 0.8) ROUND_NAMES = ("dark", "medium", "bright") RING_SCALE = (1.0, 0.5, 1.0) PANEL_RETRY = 10.0 +FULL_RETRY = 10.0 # seconds before an automatic calibration that failed to start tries again FIT_TIMEOUT = 300.0 # seconds the fit check stays up FIT_EVERY = 0.5 # seconds between its cards' updates (each is a new picture for the panel) FIT_HINT_WIDTH = 95 # characters a hint line holds in the panel @@ -167,7 +171,9 @@ class Checks: self.check = None self.gaze_on = None self.gaze_heard = 0.0 - self.want_full_until = 0.0 # gaze mode came on before the tracker said whether it's calibrated + self.full_armed = True # gaze mode on without a calibration opens the full one (need_full) + self.full_blocked = None # why it can't open now + self.full_retry_at = 0.0 self.last_quick = 0.0 self.sample_at = 0.0 # the tracker last sent anything self.seen_at = 0.0 # eyes last seen (SteamVR's tracker's variance for them, "unc") @@ -282,11 +288,47 @@ class Checks: return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2 def on_gaze_on(self): + self.full_armed = True + self.need_full("gaze mode came on without a calibration") + + def need_full(self, reason): + """Gaze mode is on without a calibration: open the full one, once per arming (see the top), + or note why it can't open.""" cal = self.calibrated() - if cal is False: - self.start("full", "gaze mode came on without a calibration") - elif cal is None: - self.want_full_until = time.monotonic() + 20 + if cal: + self.full_armed = True # missing one later (the other tracker picked) is news again + if not self.gaze_on or self.check or not self.full_armed or cal is not False: + self.full_blocked = None + return + now = time.monotonic() + if not self.can_run(): + why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2 + else "no eyes seen (is the headset on?)") + elif now < self.full_retry_at: + return + else: + reply = self.start("full", reason) + why = None if reply == "ok" else reply.removeprefix("error ") + if why: + self.full_retry_at = now + FULL_RETRY + if why and why != self.full_blocked: + log(f"the calibration can't open: {why}") + self.full_blocked = why + if not why: + self.full_armed = False + + def problem(self): + """Why gaze mode, on, can't follow your eyes yet, or None. Our tracker not having said + yet is None: Input Settings has its own line for our tracker.""" + if not self.gaze_on or self.calibrated() is not False: + return None + if self.check and self.check["kind"] == "full": + return "Not calibrated yet: the calibration is open in the headset" + if self.full_blocked: + return f"Not calibrated, and the calibration can't open: {self.full_blocked}" + if not self.full_armed: + return "Not calibrated: the calibration closed unfinished. Use Calibrate" + return "Not calibrated: the calibration opens in the headset" def auto_quick(self, reason): now = time.monotonic() @@ -703,18 +745,13 @@ class Checks: self.gaze_on = None # the helper isn't answering if self.check: self.to_helper("calpanel 1") - if self.want_full_until: - cal = self.calibrated() - if cal is not None or now > self.want_full_until: - self.want_full_until = 0.0 - if cal is False and self.gaze_on: - self.start("full", "gaze mode came on without a calibration") if not self.eyes_seen(AWAY_MIN): self.away, self.back_since = True, None elif self.back_since is not None and not self.eyes_seen(): self.back_since = None # gone again before DON_DELAY elif self.back_since is not None and now - self.back_since >= DON_DELAY: self.away, self.back_since = False, None + self.full_armed = True # a calibration that closed unfinished opens again self.auto_quick("the headset went on") if svc.kind == "own" and now - svc.own_at < 5: reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values()) @@ -723,10 +760,12 @@ class Checks: elif not self.reseat_seen and self.can_run(): self.reseat_seen = True self.auto_quick("our tracker asked for a click") + self.need_full("gaze mode is on without a calibration") def status(self): c = self.check st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(), + "problem": self.problem(), "panel": self.panel_proc is not None, "last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None} if c: diff --git a/input-settings/main.qml b/input-settings/main.qml index 37994d4..0c7cc78 100644 --- a/input-settings/main.qml +++ b/input-settings/main.qml @@ -902,6 +902,23 @@ Kirigami.ApplicationWindow { opacity: 0.7 font: Kirigami.Theme.smallFont } + Controls.Label { + // Why the gaze pointer, on, can't follow your eyes yet, so it isn't left looking + // like a plain mouse. The calibration part is the gaze service's (gaze/gazecheck.py). + readonly property var checks: gpage.status.checks || {} + readonly property string why: !backend.gazeServiceInstalled + ? "The gaze service isn't installed: run gaze/run.sh install in the Frametop folder, in a terminal" + : !backend.gazeServiceRunning ? "The gaze service isn't running (it starts with SteamVR)" + : backend.gazeTracker === "own" && !gpage.status.eyegrab + ? "Our eye tracker needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)" + : checks.problem || "" + visible: backend.gazeMode > 0 && why !== "" + text: why + Layout.maximumWidth: Kirigami.Units.gridUnit * 30 + wrapMode: Text.WordWrap + color: checks.check ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor + font: Kirigami.Theme.smallFont + } ColumnLayout { Kirigami.FormData.label: "Mouse left button:" Repeater {