From d39fbb9146491beb84f60cc2755b79dd4c33c22f Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Mon, 5 Oct 2026 10:51:07 -0600 Subject: [PATCH 1/3] Gaze: our eye tracker can take its first calibration A fresh install that chose our tracker (the installer's default since 12ad93d) could never calibrate it, so gaze mode never worked. ft-eyes maps pupils to a gaze only once it has a calibration, so before its first one ft-gaze's "own" source is empty. ft-gazed then never counted a sample, Checks.can_run() stayed false, and the calibration refused to open: "Not calibrated, and the calibration can't open: the eye tracker isn't sending", or "the headset is off or the tracker isn't sending" from Calibrate. This Frame never hit it, because its calibration came from the gaze probe. Reported 2026-10-05 on SteamOS stable. - With our tracker in use, running, and uncalibrated ("blind"), SteamVR's tracker seeing an eye is enough to open the full calibration (ft-eyes answering is what makes calibrated() False rather than None). - Each dot stands in for the gaze, so a click takes the CHECK_WINDOW up to it. ft-eyes already checks, in calib-point, that each pupil was seen in enough frames and held still then, and its reason for a refused dot shows in the panel's note as before. - A quick or five-dot check is refused until then ("use Calibrate"): ours can't take a click without a calibration. - With no eyes seen, the reason given is that, not "the eye tracker isn't sending". gaze/test/first-calibration-test.py runs the service against a fake ft-gaze, ft-eyes and pointer helper, in a temp HOME: the calibration opens by itself when gaze mode comes on, each click sends ft-eyes the 0.6 s before it, a refused dot's reason reaches the panel, and calib-fit leaves the tracker calibrated with gaze mode still on. It passes here and fails on the previous code. gaze/test/idle-test.py still passes. Not yet tested in the headset. Co-Authored-By: Claude Opus 5.5 --- gaze/README.md | 2 +- gaze/gazecheck.py | 34 +++- gaze/test/first-calibration-test.py | 230 ++++++++++++++++++++++++++++ 3 files changed, 261 insertions(+), 5 deletions(-) create mode 100755 gaze/test/first-calibration-test.py diff --git a/gaze/README.md b/gaze/README.md index 2ee6937..de9d4e9 100644 --- a/gaze/README.md +++ b/gaze/README.md @@ -33,7 +33,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste - **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time. - **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it. - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them. -- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. +- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always. - **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there. diff --git a/gaze/gazecheck.py b/gaze/gazecheck.py index a912f2e..932b2a9 100644 --- a/gaze/gazecheck.py +++ b/gaze/gazecheck.py @@ -48,6 +48,13 @@ click with nothing taken after ACCEPT_WAIT, and a failed calibration names its m reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1", renewed every second; the helper shows it again by itself when that stops). +Our tracker's first calibration: before it has one, ft-eyes publishes no gaze (it maps pupils +to a gaze only with a calibration), so there's no gaze to hold still. Its calibration runs +anyway ("blind"): someone in the headset (SteamVR's tracker sees an eye) and ft-eyes answering +are enough to start it, each dot stands in for the gaze, and a click takes the CHECK_WINDOW up +to it. ft-eyes then checks that each pupil was seen and held still in that window (calib-point) +and says why not. Without this, a fresh install could never calibrate our tracker. + What a capture teaches: our tracker quick and five: a click ("click T YAW PITCH", like a pointer lesson); full: calib-start, a calib-point for each dot, calib-fit (its calibration) @@ -336,9 +343,17 @@ class Checks: return time.monotonic() - self.seen_at < within def can_run(self): - """Someone's in the headset and the tracker is sending.""" + """Someone's in the headset and the tracker is sending. Our tracker sends no gaze before + its first calibration: then ft-eyes answering (calibrated() is False only once it has) + is enough, since the calibration is what it needs (see "first calibration" at the top).""" + if self.blind(): + return self.eyes_seen() return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2 + def blind(self): + """Our tracker is in use, running, and not calibrated yet: it has no gaze to send.""" + return self.svc.kind == "own" and self.calibrated() is False + def on_gaze_on(self): self.full_armed = True self.need_full("gaze mode came on without a calibration") @@ -356,7 +371,7 @@ class Checks: if not self.can_run() and self.svc.waking(): return # the tracker is still starting (the service idled) if not self.can_run(): - why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2 + why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2 and not self.blind() else "no eyes seen (is the headset on?)") elif now < self.full_retry_at: return @@ -408,6 +423,9 @@ class Checks: if not self.can_run(): return "error the headset is off or the tracker isn't sending" own = svc.kind == "own" + blind = self.blind() + if blind and kind != "full": + return "error our tracker isn't calibrated yet: use Calibrate" if kind == "full" and own: reply = ask(EYES, "calib-start", 3.0) if not reply.startswith("ok"): @@ -416,8 +434,10 @@ class Checks: now = time.monotonic() self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0, "started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0, - "skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""} - log(f"{kind} check: {reason}") + "skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": "", + "blind": blind} + log(f"{kind} check: {reason}" + (" (our tracker's first: no gaze yet, so each click takes the look " + "up to it)" if blind else "")) if kind == "full": st = ask(SCREENS, "state", 0.5).split() if len(st) >= 3 and st[0] == "ok": @@ -519,6 +539,12 @@ class Checks: return if c["own"]: src = s["src"].get("own") or {} + if c["blind"]: + # Our tracker's first calibration: no gaze yet, so the dot stands in for it (the + # gaze can't seem to move) and a click takes the look up to it. ft-eyes checks + # the pupils held still (see the top). + yaw, pitch, _ = c["dots"][c["i"]] + src = {"hy": yaw, "hp": pitch} else: src = s["src"].get("mmap1") or {} if "hy" not in src: diff --git a/gaze/test/first-calibration-test.py b/gaze/test/first-calibration-test.py new file mode 100755 index 0000000..cb0d4f9 --- /dev/null +++ b/gaze/test/first-calibration-test.py @@ -0,0 +1,230 @@ +#!/usr/bin/env python3 +"""Offline test of our own eye tracker's first calibration (gaze/gazecheck.py, "blind"): before +it has a calibration, ft-eyes publishes no gaze, and the calibration must still open and take +its dots. Until 2026-10-05 it couldn't: a fresh install that chose our tracker never calibrated. + +Runs ft-gazed's Service with its sockets renamed and HOME in a temp folder (state and settings +go there), a fake pointer helper (gaze mode on, headset worn), a fake ft-gaze (SteamVR sees both +eyes; "own" is {"ok":0}, as with an uncalibrated ft-eyes), a fake ft-eyes control socket, and no +panel (a stand-in process). Nothing reaches the live gaze service, the pointer helper, ft-eyes, +or SteamVR, so it's safe next to them. + + gaze/test/first-calibration-test.py +""" +import os +import tempfile + +HOME = tempfile.mkdtemp(prefix="ft-gaze-first-cal-test-") +os.environ["HOME"] = HOME # before gazecal: its STATE, and frametop.conf, follow HOME + +import importlib.machinery # noqa: E402 +import importlib.util # noqa: E402 +import json # noqa: E402 +import selectors # noqa: E402 +import shutil # noqa: E402 +import socket # noqa: E402 +import subprocess # noqa: E402 +import sys # noqa: E402 +import threading # noqa: E402 +import time # noqa: E402 + +HERE = os.path.dirname(os.path.abspath(__file__)) +GAZE = os.path.join(HERE, "..") +sys.path.insert(0, GAZE) +loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(GAZE, "ft-gazed")) +gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader)) +loader.exec_module(gazed) +import gazecheck # noqa: E402 (the module ft-gazed imported) + +tag = f"ft_gaze_first_cal_test_{os.getpid()}" +gazed.ME = f"\0{tag}_gazed" +gazed.POINTER = gazecheck.POINTER = f"\0{tag}_helper" +gazecheck.SCREENS = f"\0{tag}_screens" +gazecheck.PANEL = f"\0{tag}_panel" +gazed.EYES_SOCKET = gazecheck.EYES = f"\0{tag}_eyes" +gazed.read_settings = lambda: ("own", "auto", "auto", 55.0) +DOTS = 3 +real_dots = gazecheck.check_dots +gazecheck.check_dots = lambda kind, own: real_dots(kind, own)[:DOTS] # a short calibration +logs = [] +gazed.log = gazecheck.log = lambda msg: logs.append(msg) + +# The fake ft-gaze: 90 samples a second, SteamVR sees both eyes, no gaze from ours. +FAKE = os.path.join(HOME, "ft-gaze") +with open(FAKE, "w") as f: + f.write('''import json, os, select, sys, time +while True: + if select.select([sys.stdin], [], [], 1 / 90)[0]: + if not os.read(0, 4096): + break + print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001], + "open": [0.8, 0.8]}, "own": {"ok": 0}}}), flush=True) +''') +SLEEPER = [sys.executable, "-c", "import sys; sys.stdin.read()"] # quits when its stdin closes + + +def start_helper(self): + """ft-gaze, straight from here instead of the dev container.""" + self.proc = subprocess.Popen([sys.executable, FAKE], stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + self.proc_sources = self.wanted_sources() + os.set_blocking(self.proc.stdout.fileno(), False) + os.set_blocking(self.proc.stderr.fileno(), False) + self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout") + self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr") + self.buf = b"" + + +def start_eyes(self): + """ft-eyes' process: a stand-in. Its control socket is the fake below.""" + self.eyes_proc = subprocess.Popen(SLEEPER, stdin=subprocess.PIPE, stderr=subprocess.PIPE) + os.set_blocking(self.eyes_proc.stderr.fileno(), False) + self.sel.register(self.eyes_proc.stderr, selectors.EVENT_READ, "eyes") + + +def start_panel(self): + self.panel_proc = subprocess.Popen(SLEEPER, stdin=subprocess.PIPE, stderr=subprocess.PIPE) + os.set_blocking(self.panel_proc.stderr.fileno(), False) + self.sel.register(self.panel_proc.stderr, selectors.EVENT_READ, "panel") + + +gazed.Service.start_helper = start_helper +gazed.Service.start_eyes = start_eyes +gazecheck.Checks.start_panel = start_panel + +# The fake ft-eyes: uncalibrated until calib-fit. "fail" answers the next calib-point with that. +eyes_state = {"cal": None, "points": [], "fail": None} +eyes = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) +eyes.bind(gazed.EYES_SOCKET) +eyes.settimeout(0.2) + + +def eyes_answer(): + while True: + try: + data, addr = eyes.recvfrom(512) + except socket.timeout: + continue + except OSError: + return + w = data.decode().split() + if w[0] == "status": + reply = json.dumps({"calibration": eyes_state["cal"], "calibrating": False, "dots": 0, + "eyes": {"right": {"reseat": False}, "left": {"reseat": False}}}) + elif w[0] == "calib-start": + eyes_state["points"] = [] + reply = "ok" + elif w[0] == "calib-point": + eyes_state["points"].append(tuple(map(float, w[1:5]))) + reply, eyes_state["fail"] = eyes_state["fail"] or "ok 50 50 1.00 1.00", None + elif w[0] == "calib-fit": + eyes_state["cal"] = {"made": "test", "dots": len(eyes_state["points"])} + reply = f"ok {len(eyes_state['points'])} dots" + else: + reply = f"fail unknown command {w[0]}" + if addr: + eyes.sendto(reply.encode(), addr) + + +threading.Thread(target=eyes_answer, daemon=True).start() + +helper_state = {"reply": "ok off worn", "heard": []} +helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) +helper.bind(gazed.POINTER) +helper.settimeout(0.2) + + +def helper_answer(): + while True: + try: + data, addr = helper.recvfrom(512) + except socket.timeout: + continue + except OSError: + return + if data.startswith(b"gaze ?") and addr: + helper.sendto(helper_state["reply"].encode(), addr) + else: + helper_state["heard"].append(data.decode()) + + +threading.Thread(target=helper_answer, daemon=True).start() +svc = gazed.Service(None, False, gazed.POINTER) +threading.Thread(target=svc.run, daemon=True).start() +ctl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) +ctl.bind("") +ctl.settimeout(3) + + +def ask(cmd): + ctl.sendto(cmd.encode(), gazed.ME) + return ctl.recv(65536).decode() + + +failures = [] + + +def check(label, got, want): + ok = got == want + print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True) + if not ok: + failures.append(label) + + +def wait(cond, seconds): + end = time.monotonic() + seconds + while time.monotonic() < end: + if cond(): + return True + time.sleep(0.05) + return cond() + + +def check_state(): + return svc.checks.check or {} + + +def take_dot(i): + """Wait for dot i to settle, then click: is it taken?""" + wait(lambda: check_state().get("i") == i and not check_state().get("done_at"), 3) + time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1) + before = check_state().get("captured", 0) + ask("calaccept") + return wait(lambda: check_state().get("captured", 0) > before, 2) + + +check("gaze mode off, Gaze page open (wake): ours runs, uncalibrated", ask("wake 60"), "ok") +check("the service knows ours has no calibration", wait(lambda: svc.checks.calibrated() is False, 6), True) +check("a quick check is refused while ours has no calibration", svc.checks.start("quick", "test"), + "error our tracker isn't calibrated yet: use Calibrate") +helper_state["reply"] = "ok on worn" +check("gaze mode on: the calibration opens by itself", wait(lambda: check_state().get("kind") == "full", 6), True) +check("it runs blind (no gaze from ours yet)", check_state().get("blind"), True) +check("nothing says it can't open", any("can't open" in m for m in logs), False) + +check("dot 1: a click takes it", take_dot(0), True) +t0, t1, yaw, pitch = eyes_state["points"][0] +dot = gazecheck.check_dots("full", True)[0] +check("its window is the look up to the click (about CHECK_WINDOW)", + abs((t1 - t0) - gazecheck.CHECK_WINDOW) < 0.15, True) +check("ft-eyes got that dot's direction", (round(yaw, 3), round(pitch, 3)), (round(dot[0], 3), round(dot[1], 3))) + +eyes_state["fail"] = "fail the left eye was seen in only 3 frames" +wait(lambda: check_state().get("i") == 1 and not check_state().get("done_at"), 3) +time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1) +ask("calaccept") +check("ft-eyes refusing a dot: its reason reaches the panel's note", + wait(lambda: "left eye in only 3 frames" in check_state().get("note", ""), 2), True) +check("dot 2, second try: taken", take_dot(1), True) +check("dot 3: taken", take_dot(2), True) + +check("all dots: ours fits its calibration (calib-fit)", + wait(lambda: eyes_state["cal"] is not None and not svc.checks.check, 4), True) +check("the service sees it calibrated", wait(lambda: svc.checks.calibrated() is True, 4), True) +check("and gaze mode stays on", "gaze off" in helper_state["heard"], False) + +print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True) +svc.running = False +time.sleep(0.7) +shutil.rmtree(HOME, ignore_errors=True) +os._exit(1 if failures else 0) From 307e2b73085adf5b89ed3ec4b65d3e3f29085da6 Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Mon, 5 Oct 2026 10:53:59 -0600 Subject: [PATCH 2/3] SteamVR's tools find SteamVR from a terminal in the Frametop desktop The desktop has its own XDG_CONFIG_HOME (~/.config/frametop), and SteamVR's tools and OpenVR read their path registry from there. So from a terminal in the desktop: - pointer/driver/install.sh (install.sh step 4, every reinstall or update from Konsole) wrote a new registry, ~/.config/frametop/openvr/openvrpaths.vrpath, with only our driver and "runtime": null, and never registered the driver with SteamVR. That stray file then hid SteamVR from every OpenVR program started in the desktop. - scripts/update-check.py (and so doctor.sh and report.sh) reported OpenVR "can't connect to SteamVR as a background app" (VRInitError_Init_PathRegistryNotFound, or InstallationNotFound with the stray file) and "vrcmd --overlays lists no overlays". A user's report of 2026-10-05 showed both, with SteamVR and Frametop's services fine. - ft-layout's vrcmd calls (the gamescope backend) found no overlays. They now run with XDG_CONFIG_HOME=~/.config: the driver installer (install, uninstall, probe, aimhere), update-check.py (for all its checks: nothing there reads the desktop's own config), report.sh's vrpathreg show, and ft-layout's vrcmd. report.sh doesn't set it for the whole report, since session/fix-panels.py --check reads the desktop's Plasma config through it. The driver installer also removes the stray registry (only vrpathreg's, with no runtime), and update-check.py warns about one. And vrpathreg runs `xdg-open vrmonitor://driverinstalled` to tell SteamVR's desktop monitor, which the Frame doesn't have: in the desktop that showed "could not read file vrmonitor://driverinstalled" on every install. A stand-in xdg-open on its PATH takes it, so install.sh no longer filters the step's output. Tested in a fake HOME with a copy of SteamVR's registry, a stray one, and the desktop's XDG_CONFIG_HOME: the new installer removes the stray file, registers the driver in the copy, and never reaches xdg-open; the old one wrote the stray file, left the copy alone, and ran xdg-open. update-check.py from that environment: OpenVR and vrcmd ok (the old one fails both); its stray-registry warning fires on a seeded file. ft-layout's vrcmd: 119 overlays (was 0). Co-Authored-By: Claude Opus 5.5 --- docs/design.md | 2 +- install.sh | 2 +- layout/ft_layout.py | 3 ++- pointer/driver/install.sh | 26 +++++++++++++++++++++----- scripts/report.sh | 3 ++- scripts/update-check.py | 13 +++++++++++++ 6 files changed, 40 insertions(+), 9 deletions(-) diff --git a/docs/design.md b/docs/design.md index c57533e..b61f77b 100644 --- a/docs/design.md +++ b/docs/design.md @@ -169,7 +169,7 @@ The Frame controllers can be mapped like mouse buttons, but they aren't input de ## The desktop session -The session is modeled on SteamOS's `steamos-nested-desktop` and runs beside it. It has its own runtime directory, config (`~/.config/frametop`), and state, so it never disturbs the stock desktop's layout or panels. It runs on a private D-Bus from `dbus-run-session`, which has two consequences. KDE only launches apps in systemd scopes when systemd is on the session bus, so everything started in the desktop lands in its systemd unit, and stopping the unit would kill all of it; `session/keep-apps.sh` moves those programs out first. And tools that need the real user bus, like podman and `distrobox-host-exec`, have to be pointed at it explicitly. +The session is modeled on SteamOS's `steamos-nested-desktop` and runs beside it. It has its own runtime directory, config (`~/.config/frametop`), and state, so it never disturbs the stock desktop's layout or panels. It runs on a private D-Bus from `dbus-run-session`, which has two consequences. KDE only launches apps in systemd scopes when systemd is on the session bus, so everything started in the desktop lands in its systemd unit, and stopping the unit would kill all of it; `session/keep-apps.sh` moves those programs out first. And tools that need the real user bus, like podman and `distrobox-host-exec`, have to be pointed at it explicitly. Its own config folder also hides SteamVR's path registry (`~/.config/openvr/openvrpaths.vrpath`) from everything started in it: OpenVR programs there fail with `VRInitError_Init_PathRegistryNotFound`, and `vrpathreg adddriver` writes a new registry under `~/.config/frametop/openvr` that has no SteamVR in it and that SteamVR never reads. So Frametop's scripts run SteamVR's tools with `XDG_CONFIG_HOME=~/.config`. The VR launcher starts the session from the Steam client, and the client's environment came along: `LD_LIBRARY_PATH` pointing at Steam's own runtime, whose `libavcodec` has no H.264 decoder, so VLC in the desktop couldn't play most videos, plus the client's overlay and launch settings. The session script drops the client's variables before it starts anything. SteamOS's global Mesa settings (`/usr/share/deckard/mesavars.sh`) stay, and the gamescope session's Vulkan layer (`ENABLE_GAMESCOPE_WSI`) is only kept for the gamescope backend. diff --git a/install.sh b/install.sh index eddab5b..9c0775e 100755 --- a/install.sh +++ b/install.sh @@ -75,7 +75,7 @@ step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, b step "4/10 3D mouse: SteamVR driver" "$root/pointer/driver/build.sh" -"$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open +"$root/pointer/driver/install.sh" install step "5/10 3D mouse: pointer helper service" "$root/pointer/helper/build.sh" diff --git a/layout/ft_layout.py b/layout/ft_layout.py index 774817e..60284f3 100755 --- a/layout/ft_layout.py +++ b/layout/ft_layout.py @@ -534,7 +534,8 @@ def capture_screens(): # ---------------------------------------------------------------- gamescope (dashboard panels) def vrcmd(*args, timeout=10): - env = dict(os.environ, LD_LIBRARY_PATH=os.path.dirname(VRCMD)) + # SteamVR's config folder: in the desktop, XDG_CONFIG_HOME is its own (docs/design.md). + env = dict(os.environ, LD_LIBRARY_PATH=os.path.dirname(VRCMD), XDG_CONFIG_HOME=os.path.expanduser("~/.config")) try: return subprocess.run([VRCMD, *args], capture_output=True, text=True, timeout=timeout, env=env).stdout except (OSError, subprocess.TimeoutExpired): diff --git a/pointer/driver/install.sh b/pointer/driver/install.sh index b781fd8..bbf4bc7 100755 --- a/pointer/driver/install.sh +++ b/pointer/driver/install.sh @@ -14,6 +14,10 @@ frame="$root/scripts/frame.sh" src=$FRAME_REPO/pointer/driver dest='$HOME/.local/share/frametop/ft_pointer' # expanded on the Frame, in the commands below reg='/opt/steamvr/bin/linuxarm64/vrpathreg' +# SteamVR's tools with SteamVR's config folder and libraries. A terminal in the Frametop desktop +# has XDG_CONFIG_HOME=~/.config/frametop (session/frametop-session.sh), where vrpathreg finds no +# registry, so it made one there that SteamVR never reads. +steamvr='env XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64' # expanded on the Frame case ${1:-install} in install) @@ -21,6 +25,11 @@ case ${1:-install} in # Replacing the files of a driver SteamVR has loaded leaves its input bindings in a bad # state (the 3D mouse no longer gets the laser) until SteamVR restarts, so an unchanged # driver is left alone. + # A registry an earlier install made in the desktop's config folder (see steamvr above) has + # no SteamVR in it, and hid SteamVR from OpenVR programs started in the desktop + # (VRInitError_Init_InstallationNotFound): it goes. + # vrpathreg runs 'xdg-open vrmonitor://driverinstalled' for SteamVR's desktop monitor, which + # the Frame doesn't have, so the desktop said "could not read file": a stand-in takes it. "$frame" --host "set -e; test -f $src/build/driver_ft_pointer.so rm -rf $dest.new; mkdir -p $dest.new/bin/linuxarm64 cp -r $src/ft_pointer/. $dest.new/ @@ -31,15 +40,22 @@ else rm -rf $dest; mv $dest.new $dest echo 'installed; restart SteamVR to load it' fi -LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg adddriver $dest -LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg show | grep -A3 -i 'external'" ;; +stray=\$HOME/.config/frametop/openvr/openvrpaths.vrpath +if grep -qs '\"jsonid\" : \"vrpathreg\"' \$stray && grep -qsE '\"runtime\"[[:space:]]*:[[:space:]]*null' \$stray; then + rm -f \$stray; rmdir \$HOME/.config/frametop/openvr 2>/dev/null || true + echo 'removed a SteamVR path registry an earlier install left in ~/.config/frametop/openvr' +fi +noopen=\$(mktemp -d); trap 'rm -rf \$noopen' EXIT +printf '#!/bin/sh\nexit 0\n' > \$noopen/xdg-open; chmod +x \$noopen/xdg-open +PATH=\$noopen:\$PATH $steamvr $reg adddriver $dest +$steamvr $reg show | grep -A3 -i 'external'" ;; uninstall) - "$frame" --host "LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 $reg removedriver $dest; rm -rf $dest; echo 'removed; restart SteamVR to unload it'" ;; + "$frame" --host "$steamvr $reg removedriver $dest; rm -rf $dest; echo 'removed; restart SteamVR to unload it'" ;; send) "$frame" --host "python3 -c 'import socket,sys; s=socket.socket(socket.AF_UNIX,socket.SOCK_DGRAM); s.sendto(sys.argv[1].encode(), \"\\0ft_pointer\")' $(printf %q "${2:?command}")" ;; - probe) "$frame" -C pointer/probe 'LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 ./build/vrprobe' ;; + probe) "$frame" -C pointer/probe "$steamvr ./build/vrprobe" ;; aimhere) - read -r yaw pitch < <("$frame" -C pointer/probe 'LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 ./build/vrprobe' | sed -n 's/^head yaw = \([-0-9.]*\) pitch = \([-0-9.]*\)$/\1 \2/p') + read -r yaw pitch < <("$frame" -C pointer/probe "$steamvr ./build/vrprobe" | sed -n 's/^head yaw = \([-0-9.]*\) pitch = \([-0-9.]*\)$/\1 \2/p') [ -n "${yaw:-}" ] || { echo "head pose not valid (headset off?)" >&2; exit 1; } "$0" send "aim $yaw $pitch" && echo "aimed at yaw $yaw pitch $pitch" ;; log) "$frame" --host "grep -iE 'ft_pointer' ~/.local/share/Steam/logs/vrserver.txt | tail -n ${2:-20}" ;; diff --git a/scripts/report.sh b/scripts/report.sh index ea031e1..f59fb7c 100755 --- a/scripts/report.sh +++ b/scripts/report.sh @@ -26,7 +26,8 @@ for u in frametop-input-relay frametop-pointer frametop-power; do done echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)" echo "paused for VR games: $(cat /run/user/$(id -u)/frametop-pause.json 2>/dev/null || echo 'no (never paused since boot)')" -LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p' +# SteamVR's config folder: a terminal in the desktop has the session's own (docs/design.md). +XDG_CONFIG_HOME=$HOME/.config LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p' section "What Frametop needs from SteamOS (scripts/update-check.py)" python3 "$repo/scripts/update-check.py" 2>&1 || true diff --git a/scripts/update-check.py b/scripts/update-check.py index 0b320b8..3e540e1 100755 --- a/scripts/update-check.py +++ b/scripts/update-check.py @@ -35,6 +35,9 @@ KNOWN_GOOD = os.path.join(HOME, ".local/state/frametop/known-good.json") STEAMVR_BIN = "/opt/steamvr/bin/linuxarm64" LAUNCHER = os.path.join(HOME, ".local/share/applications/deckard-nested-desktop.desktop") VRPATHS = os.path.join(HOME, ".config/openvr/openvrpaths.vrpath") +# The Frametop desktop has its own XDG_CONFIG_HOME (session/frametop-session.sh). An install from a +# terminal there, before pointer/driver/install.sh set SteamVR's, left vrpathreg's registry here. +STRAY_VRPATHS = os.path.join(HOME, ".config/frametop/openvr/openvrpaths.vrpath") BACKLIGHT = "/sys/class/backlight/ae94000.dsi.0/brightness" # as in power/ft-powerd.cpp EYE_MMAP = "/dev/shm/eye-server.mmap" VRSERVER = "http://127.0.0.1:27062" # its web socket: input/vrws.py @@ -227,6 +230,14 @@ def check_host(): else: report("FAIL", "pointer driver", "not registered with SteamVR; run pointer/driver/install.sh install, " "then restart SteamVR") + try: + with open(STRAY_VRPATHS) as f: + stray = json.load(f).get("runtime") is None + except (OSError, ValueError, AttributeError): + stray = False + if stray: + report("warn", "OpenVR path registry", f"{STRAY_VRPATHS} has no SteamVR in it, and hides SteamVR from " + "OpenVR programs started in the Frametop desktop; pointer/driver/install.sh install removes it") session = launcher_session() if session is None: @@ -474,6 +485,8 @@ def main(): # A terminal in the desktop has its session's runtime dir and bus; systemctl needs the real ones. os.environ["XDG_RUNTIME_DIR"] = f"/run/user/{UID}" os.environ["DBUS_SESSION_BUS_ADDRESS"] = f"unix:path=/run/user/{UID}/bus" + # And its own config folder, where SteamVR's path registry isn't: OpenVR and vrcmd need ours. + os.environ["XDG_CONFIG_HOME"] = os.path.join(HOME, ".config") versions = current_versions() check_versions(versions) From 1037d4e1253d92af7f5ea7f0f42cf0cfa510961a Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Mon, 5 Oct 2026 10:54:57 -0600 Subject: [PATCH 3/3] Report: the gaze service, our eye tracker, and their logs A gaze problem couldn't be told from a report: it had no gaze section. A user's report of 2026-10-05 showed only that the gaze service ran and our tracker was picked. The report now has a Gaze section: whether the gaze service and our tracker's frame grabber are enabled and running, when SteamVR's correction and our tracker's calibration were made (or "none"), and the service's status (ft-gazectl status, on one line: the tracker in use, whether it's awake and why not, checks.problem). Then the last 10 checks and calibration dots (checks.jsonl: each dot's reply and whether it was taken), the gaze service's last 60 lines without podman's exec lines, and the frame grabber's last 20. Nothing in them is an eye image. Tested on this Frame: each section fills in, and the status is one line. Co-Authored-By: Claude Opus 5.5 --- scripts/report.sh | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/scripts/report.sh b/scripts/report.sh index f59fb7c..ce5c59b 100755 --- a/scripts/report.sh +++ b/scripts/report.sh @@ -68,6 +68,28 @@ section "Pointer helper (last 60 lines)" journalctl --user -u frametop-pointer -n 60 --no-pager -o short 2>/dev/null section "Power service (last 30 lines)" journalctl --user -u frametop-power -n 30 --no-pager -o short 2>/dev/null +section "Gaze" +echo "frametop-gaze: $(systemctl --user is-enabled frametop-gaze 2>/dev/null) / $(systemctl --user is-active frametop-gaze 2>/dev/null)" +echo "frametop-eyegrab (our eye tracker's frame grabber): $(systemctl is-enabled frametop-eyegrab 2>/dev/null) / $(systemctl is-active frametop-eyegrab 2>/dev/null)" +for f in calibration.json eyes/calibration.json; do # SteamVR's correction, our tracker's calibration + p=~/.local/state/frametop/gaze/$f + echo "$f: $([ -f "$p" ] && date -r "$p" '+%F %T' || echo none)" +done +python3 - "$repo" <<'PY' 2>&1 || true +import json, subprocess, sys +out = subprocess.run([sys.executable, sys.argv[1] + "/gaze/ft-gazectl", "status"], capture_output=True, text=True, timeout=10) +text = (out.stdout or out.stderr).strip() +try: + print("status:", json.dumps(json.loads(text), separators=(",", ":"))) +except ValueError: + print("status:", text or "no answer from the gaze service") +PY +section "Gaze checks and calibration dots (last 10)" +tail -n 10 ~/.local/state/frametop/gaze/checks.jsonl 2>/dev/null | cut -c1-400 || true +section "Gaze service (last 60 lines, podman's left out)" +journalctl --user -u frametop-gaze -n 300 --no-pager -o short 2>/dev/null | grep -v ' podman\[' | tail -n 60 +section "Our eye tracker's frame grabber (last 20 lines)" +journalctl -u frametop-eyegrab -n 20 --no-pager -o short 2>/dev/null for f in /tmp/frametop-session.log /tmp/frametop-screens.log /tmp/frametop-layout.log; do section "$f (last 60 lines)" tail -n 60 "$f" 2>/dev/null || echo "missing"