steam-frame/DeeJanuz--frametop
1
0 Fork 0
mirror of https://github.com/DeeJanuz/frametop.git synced 2026-10-06 06:00:09 +02:00
Code Issues Packages Projects Releases 4 Wiki Activity

Bring our own eye tracker into the gaze folder, run by the gaze service

Browse Source 
frame-eyes, a separate project until now, becomes gaze/tracker:
- ft-eyegrab (was fe-bufprobe) copies the eye-camera frames, read-only, out of
  SteamVR's eyetracking process. It runs as the system service
  frametop-eyegrab.service, which gaze/tracker/install.sh installs to
  /etc/frametop with sudo. It keeps only CAP_SYS_PTRACE, CAP_DAC_READ_SEARCH, and
  CAP_CHOWN, and copies frames only while /dev/shm/frametop-eyes-want is fresh,
  holding none of the tracker's buffers otherwise.
- ft-eyes (was fe-trackd) runs under ft-gazed in the dev container, with
  build/venv's pinned numpy and OpenCV: while Eye tracker is Own tracker, or on
  the probe's lease ("eyes SECONDS"). No sudo password or fe-live script at
  run time any more.
- lab/ holds the research tools (ft-eyes-score, -e2e, -record, -replay,
  -session) and findings.md. Recordings live outside the repo, in
  ~/.local/share/frametop/eyes/captures; .gitignore catches stray frame dumps.

Its socket is now @ft_eyes, its output /dev/shm/frametop-eyes-gaze, and its state
~/.local/state/frametop/gaze/eyes. On practice1 -> practice2 the whole live path
(ft-eyes-e2e) gives 1.30 deg median and 3.18 for the worst tenth, as before the
move (1.30, 3.21).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed 2026-09-30 10:00:07 -06:00
1 parent 5565a25444
commit ed75614f62
27 files changed
+3021 -45
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

gaze/ft-gazed
+104 -17
View File
@@ -3,8 +3,8 @@
Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read
again when the file changes:
GAZE_TRACKER=steam|own SteamVR's eye tracker (default), or our own (frame-eyes' fe-trackd,
ft-gaze's source "own", when it's running)
GAZE_TRACKER=steam|own SteamVR's eye tracker (default), or our own (gaze/tracker/ft-eyes,
ft-gaze's source "own"; this service runs it, see below)
GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far
off it was at your recent nudges; left or right counts that eye twice
as much as the other. Either eye alone carries the gaze when the
@@ -46,7 +46,7 @@ there). The gap is the tracker's error there. The raw gaze is the one sent, so i
when that look was (the history of what was sent), and so what each eye read then:
- SteamVR: each eye learns its own error, unless the gaze was more than LESSON_MAX degrees
past the correction (then it wasn't a nudge onto what you looked at);
- our tracker: the look goes to it as a click ("click T YAW PITCH" on @frame_eyes), as the
- our tracker: the look goes to it as a click ("click T YAW PITCH" on @ft_eyes), as the
probe's clicks do, and it learns how far the headset has moved on your face. That's
what it gets wrong, and after the headset was off, the first click resets it;
- either way, how far off each eye was (before the lesson taught it anything) goes to the
@@ -56,6 +56,11 @@ SteamVR's eye tracking log is followed for the headset going on (its eye model s
and the error moves): lessons from before count less then, so the first few after it
relearn the offset.
Our own tracker (gaze/tracker/ft-eyes) runs here too, in the dev container, while
GAZE_TRACKER=own or the gaze probe asks for it ("eyes SECONDS", a lease the probe renews). It
reads the eye-camera frames the root service frametop-eyegrab copies (gaze/tracker/install.sh),
which copies them only while ft-eyes runs.
Nothing here writes to SteamVR, its eye tracker, or its files: ft-gaze reads the eye
tracker's shared memory read-only.
@@ -64,6 +69,8 @@ Control socket: abstract unix datagram "@ft_gazed":
status reply: one JSON object
forget drop what the lessons taught (the calibration stays)
reload read calibration.json and the settings again
eyes <seconds> keep our own tracker running that much longer (at most 120),
whatever GAZE_TRACKER says: the probe's lease. Reply: "ok"
Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever
the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever
@@ -94,13 +101,16 @@ REPO = Path(__file__).resolve().parents[1]
HELPER = REPO / "gaze" / "build" / "ft-gaze"
ME = "\0ft_gazed"
POINTER = "\0ft_pointer_helper"
FRAME_EYES = "\0frame_eyes" # our own tracker's control socket (frame-eyes fe-trackd)
EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes" # our own tracker
EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python" # numpy, OpenCV (build.sh)
EYES_SOCKET = "\0ft_eyes" # its control socket
EYES_CAMS = Path("/dev/shm/frametop-eyes-cams") # the frames it reads (frametop-eyegrab.service)
CONF = Path.home() / ".config" / "frametop.conf"
CALIBRATION = STATE / "calibration.json"
LESSONS = STATE / "pointer-lessons.json"
LESSON_LOG = STATE / "pointer-lessons.jsonl"
SOURCES = ("action", "mmap1", "mmap2", "left", "right") # the ones with a calibration here
EYES = ("left", "right") # ft-gaze's order, and the sources for each eye alone
SIDES = ("left", "right") # ft-gaze's order, and the sources for each eye alone
TRACKERS = ("steam", "own")
BIASES = ("auto", "left", "right")
LESSON_MAX = 8.0 # degrees past the correction
@@ -108,6 +118,8 @@ OWN_LESSON_MAX = 25.0 # our tracker: after the headset was off, its first click
HISTORY = 12.0 # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later)
LOOK = 0.3 # seconds of samples before that moment make the look (the probe's fixation)
RETRY = 3.0 # seconds before starting ft-gaze again
EYES_RETRY = 10.0 # seconds before starting ft-eyes again after it stopped on its own
EYES_LEASE_MAX = 120.0
SETTLE = 0.3 # seconds after an eye is found again before the fallback learns from it
KEYBOARD_PITCH = -20.0 # degrees: gaze under this, on no screen, is a look at the keyboard
@@ -184,6 +196,9 @@ class Service:
self.history = deque()
self.own = {} # our tracker's last status reply
self.own_at = 0.0
self.eyes_proc = None # ft-eyes, while it runs
self.eyes_until = 0.0 # the probe's lease (monotonic time)
self.eyes_restart_at = 0.0
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
self.sock.bind(ME)
@@ -207,7 +222,7 @@ class Service:
return "source"
if self.tracker == "own":
return "own"
return "eyes" if all(self.models[e].samples for e in EYES) else "source"
return "eyes" if all(self.models[e].samples for e in SIDES) else "source"
# --- Settings, calibration and lessons ---
@@ -233,7 +248,7 @@ class Service:
self.models[name].from_json(d[name])
mode = d.get("_meta", {}).get("model")
self.mode = mode if mode in MODELS else DEFAULT_MODEL
log(f"calibration: {self.mode}, " + ", ".join(f"{n} {self.models[n].samples}" for n in (self.source,) + EYES)
log(f"calibration: {self.mode}, " + ", ".join(f"{n} {self.models[n].samples}" for n in (self.source,) + SIDES)
+ " samples")
def load_lessons(self):
@@ -249,7 +264,7 @@ class Service:
for t, misses in (d.get("misses") or {}).items():
if t in self.weights:
self.weights[t] = EyeWeights(self.bias, misses)
log(", ".join(f"{n} {len(self.lives[n].samples)}" for n in (self.source,) + EYES) + " lessons")
log(", ".join(f"{n} {len(self.lives[n].samples)}" for n in (self.source,) + SIDES) + " lessons")
def save_lessons(self):
tmp = LESSONS.with_suffix(".tmp")
@@ -312,13 +327,13 @@ class Service:
# Its eyes come calibrated: how far off each was is its miss. The click goes to it.
miss = [math.hypot(thy - e[0], thp - e[1]) if e else None for e in eyes]
try:
self.eyes_sock.sendto(f"click {t:.6f} {thy:.4f} {thp:.4f}".encode(), FRAME_EYES)
self.eyes_sock.sendto(f"click {t:.6f} {thy:.4f} {thp:.4f}".encode(), EYES_SOCKET)
except OSError as e:
rec["refused"] = f"our tracker isn't running ({e})"
return self.taken(rec)
else:
miss = []
for name, e in zip(EYES, eyes):
for name, e in zip(SIDES, eyes):
if not e:
miss.append(None)
continue
@@ -384,6 +399,61 @@ class Service:
pass
self.proc = None
def eyes_wanted(self):
return (self.tracker == "own" and not self.override) or time.monotonic() < self.eyes_until
def start_eyes(self):
"""Our own tracker, in the dev container, with build/venv's numpy and OpenCV. Like
ft-gaze, it quits when its stdin closes."""
if not EYES_PYTHON.exists():
log(f"ft-eyes isn't built: run {REPO}/gaze/tracker/build.sh")
self.eyes_restart_at = time.monotonic() + 30
return
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
distrobox = Path.home() / ".local" / "bin" / "distrobox"
self.eyes_proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(EYES_PYTHON), str(EYES_PROG), "-v",
"--watch-stdin"], env=env, stdin=subprocess.PIPE,
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, start_new_session=True)
os.set_blocking(self.eyes_proc.stderr.fileno(), False)
self.sel.register(self.eyes_proc.stderr, selectors.EVENT_READ, "eyes")
log("ft-eyes started" + ("" if EYES_CAMS.exists() else
f": no {EYES_CAMS} yet (the frame grabber: gaze/tracker/install.sh)"))
def stop_eyes(self):
if not self.eyes_proc:
return
try:
self.sel.unregister(self.eyes_proc.stderr)
except (KeyError, ValueError):
pass
if self.eyes_proc.stdin and not self.eyes_proc.stdin.closed:
self.eyes_proc.stdin.close()
try:
self.eyes_proc.wait(timeout=3)
except subprocess.TimeoutExpired:
try:
os.killpg(self.eyes_proc.pid, signal.SIGTERM)
except ProcessLookupError:
pass
self.eyes_proc = None
self.own = {}
def read_eyes(self):
try:
data = os.read(self.eyes_proc.stderr.fileno(), 65536)
except BlockingIOError:
return
if not data:
log(f"ft-eyes stopped (exit {self.eyes_proc.poll()}); again in {EYES_RETRY:.0f} s if still wanted")
self.stop_eyes()
self.eyes_restart_at = time.monotonic() + EYES_RETRY
return
for line in data.decode("utf-8", "replace").splitlines():
if line.strip() and (self.verbose or "fps" not in line):
log(line)
def read_stdout(self):
try:
data = os.read(self.proc.stdout.fileno(), 65536)
@@ -456,7 +526,7 @@ class Service:
eyes = [tuple(e) if e else None for e in (src.get("eyes") or [None, None])]
hp, hit = src["hp"], src.get("hit")
else:
per = [s["src"].get(name) or {} for name in EYES]
per = [s["src"].get(name) or {} for name in SIDES]
eyes = [(p["hy"], p["hp"]) if "hy" in p else None for p in per]
if not any(eyes):
return
@@ -481,7 +551,7 @@ class Service:
corrected = seen
else:
corrected = []
for name, e in zip(EYES, seen):
for name, e in zip(SIDES, seen):
c = self.correction(name, *e) if e else None
corrected.append((e[0] + c[0], e[1] + c[1]) if e else None)
gy, gp = self.weights["own" if own else "steam"].combine(corrected)
@@ -581,6 +651,13 @@ class Service:
self.refit()
self.save_lessons()
reply = "ok"
elif words[:1] == ["eyes"] and len(words) == 2:
try:
secs = min(max(float(words[1]), 0.0), EYES_LEASE_MAX)
self.eyes_until = max(self.eyes_until, time.monotonic() + secs)
reply = "ok"
except ValueError:
reply = "error bad seconds"
elif words[:1] == ["reload"]:
self.load_settings()
self.load_calibration()
@@ -623,14 +700,15 @@ class Service:
st.update(eye_weights=[round(v, 3) for v in w.weights()], eye_misses=[len(m) for m in w.misses],
eye_rms=[None if r is None else round(r, 2) for r in w.rms()])
if kind == "eyes":
st.update(calibration_samples=min(self.models[e].samples for e in EYES),
lessons=max(len(self.lives[e].samples) for e in EYES))
st.update(calibration_samples=min(self.models[e].samples for e in SIDES),
lessons=max(len(self.lives[e].samples) for e in SIDES))
if kind == "own":
own = self.own if time.monotonic() - self.own_at < 5 else {}
cal = own.get("calibration") or {}
st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
lessons=max(len(m) for m in w.misses), own_running=bool(own),
own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists())
st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
if self.last_sample else None, "headset_on": self.steam.wearing(),
"headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None,
@@ -648,11 +726,17 @@ class Service:
self.refit()
if mtime(CONF) != self.conf_mtime:
self.load_settings()
if self.tracker == "own":
want = self.eyes_wanted()
if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:
self.start_eyes()
elif not want and self.eyes_proc:
log("ft-eyes no longer wanted: stopping it")
self.stop_eyes()
if self.eyes_proc:
try:
self.eyes_sock.sendto(b"status", FRAME_EYES)
self.eyes_sock.sendto(b"status", EYES_SOCKET)
except OSError:
pass # not running: status() says so once the last answer is old
pass # not up yet: status() says so once the last answer is old
if self.dirty:
self.save_lessons()
@@ -668,6 +752,8 @@ class Service:
self.on_control()
elif key.data == "own":
self.on_own()
elif key.data == "eyes" and self.eyes_proc:
self.read_eyes()
elif key.data == "stdout" and self.proc:
self.read_stdout()
elif key.data == "stderr" and self.proc:
@@ -679,6 +765,7 @@ class Service:
log(json.dumps(self.status()))
next_verbose = now + 5
self.stop_helper()
self.stop_eyes()
if self.dirty:
self.save_lessons()