steam-frame/DeeJanuz--frametop
1
0 Fork 0
mirror of https://github.com/DeeJanuz/frametop.git synced 2026-10-06 06:00:09 +02:00
Code Issues Packages Projects Releases 4 Wiki Activity

Ask for one look at a centre dot after the headset was off

Browse Source 
With the Own tracker, the probe polls its status each second. After the headset was off
(or the tracker restarted), it shows one dot at the centre until you look at it and press
(S skips): that click resets both eyes' shifts, so the first real clicks aren't 10-17
degrees off. The screen-to-direction geometry the calibration used is shared with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed 2026-09-29 23:08:07 -06:00
1 parent c37bee27c9
commit 76f3cdd2dc
2 files changed
+80 -12
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

gaze/README.md
+1 -1
View File
@@ -32,7 +32,7 @@ Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction,
| SteamVR action | An `eyetracking` action bound to `/user/head/eyetracking` (`actions/`), read with `IVRInput::GetEyeTrackingDataRelativeToNow`. This is the supported way. |
| mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space: set 1 is filtered, and its two eyes always share one pitch; set 2 is each eye's own reading. After each set come the tracker's variances for each eye, and at the end each eye's raw measurement and its variance (the tracker's confidence in that frame), which ft-gaze passes on for the fit check. |
| Left eye, right eye | Each eye alone, from set 2: calibrate and test them to see what one eye is worth against both. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. |
| Own tracker | Our own tracker, from `~/Desktop/Projects/frame-eyes` (experimental). `tools/fe-live` runs it: a root helper copies the eye-camera frames out of SteamVR's buffers (read-only) and `fe-trackd` finds the pupils and glints and publishes the gaze in `/dev/shm/frame-eyes-gaze`. It keeps its own calibration, not SteamVR's: the probe's calibration with the tracker toggle on Own tracker fits it (its dots go out to the Calibration ring angle each way, on an oval, since the fit goes wrong past its dots), and practice clicks teach it how far the headset has moved on your face since. With Own tracker on, the probe hides SteamVR's gaze and draws a red dot where each eye alone puts it. ft-gaze reports it as `own` while it's running, and as `{"ok":0}` otherwise. |
| Own tracker | Our own tracker, from `~/Desktop/Projects/frame-eyes` (experimental). `tools/fe-live` runs it: a root helper copies the eye-camera frames out of SteamVR's buffers (read-only) and `fe-trackd` finds the pupils and glints and publishes the gaze in `/dev/shm/frame-eyes-gaze`. It keeps its own calibration, not SteamVR's: the probe's calibration with the tracker toggle on Own tracker fits it (its dots go out to the Calibration ring angle each way, on an oval, since the fit goes wrong past its dots), and practice clicks teach it how far the headset has moved on your face since. After the headset was off, the probe first asks for one look at a centre dot, which resets that. With Own tracker on, the probe hides SteamVR's gaze and draws a red dot where each eye alone puts it. ft-gaze reports it as `own` while it's running, and as `{"ok":0}` otherwise. |
The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`. When the tracker loses an eye, the same log says `CEyePoseUKF L: Large dt` (or `R`) as it starts that eye over.
gaze/probe/ft-gazeprobe
+79 -11
View File
@@ -458,6 +458,8 @@ class Probe(Adw.ApplicationWindow):
self.models = {s: Correction() for s in SOURCES}
self.own = OwnTracker()
self.reseat_check = False # showing the one-dot check after the headset was off
self.reseat_skipped = False # S skipped this one
self.live = {s: LiveCorrection() for s in SOURCES}
self.steam = SteamEyeLog()
self.steam.poll()
@@ -988,6 +990,10 @@ class Probe(Adw.ApplicationWindow):
if keyval in (Gdk.KEY_s, Gdk.KEY_S) and self.calib:
self.calib_skip()
return True
if keyval in (Gdk.KEY_s, Gdk.KEY_S) and self.reseat_check:
self.reseat_check, self.reseat_skipped = False, True
self.area.queue_draw()
return True
if keyval == Gdk.KEY_Escape:
if self.calib or self.test:
self.calib = self.test = None
@@ -1029,9 +1035,60 @@ class Probe(Adw.ApplicationWindow):
if self.steam.poll():
self.on_status("SteamVR's eye tracker just started again: its own calibration started over, "
"so the raw gaze may have moved")
self.poll_own()
self.update_stats()
return True
def poll_own(self):
"""With the Own tracker: after the headset was off (or the tracker restarted), its
next click starts each eye's shift over, and until then the first clicks can be
10-17 degrees off (frame-eyes, session 5). So ask for one look at a centre dot first,
as Varjo's headsets do each time they're put on."""
if self.source != "own" or self.calib or self.capture:
return
reply = self.own.ask("status")
if reply.startswith("fail"):
return
st = json.loads(reply)
pending = bool(st.get("calibration")) and any(e.get("reseat") for e in st["eyes"].values())
if not pending:
self.reseat_skipped = False
check = pending and not self.reseat_skipped
if check != self.reseat_check:
self.reseat_check = check
self.area.queue_draw()
def reseat_press(self):
"""The press on the check dot: a click there teaches both eyes' shifts at once."""
if not self.recent:
return
t_end = self.recent[-1]["t"]
w, h = self.canvas_size()
d = self.screen_direction([smp for smp in self.recent if smp["t"] >= t_end - 0.6], (w / 2, h / 2))
if d is None:
self.on_status("no gaze samples on this screen: look at the dot and press again")
return
reply = self.own.ask(f"click {t_end:.6f} {d[0]:.4f} {d[1]:.4f}")
if reply.startswith("ok"):
self.reseat_check = False
self.on_status("Own tracker: checked after the headset was off")
else:
self.on_status(reply[5:] if reply.startswith("fail") else reply)
self.area.queue_draw()
def draw_reseat(self, cr, w, h):
cx, cy = w / 2, h / 2
cr.set_source_rgb(1, 1, 1)
cr.arc(cx, cy, 10, 0, 2 * math.pi)
cr.fill()
cr.set_source_rgb(0.1, 0.1, 0.1)
cr.arc(cx, cy, 3, 0, 2 * math.pi)
cr.fill()
ink = (0.9, 0.9, 0.9)
self.text(cr, 60, 70, "Own tracker: the headset was off, so it may sit differently now", ink, 26)
self.text(cr, 60, 108, self.status if self.status.startswith("no gaze") else
"Look at the dot and press (Enter, Space, or click). S skips.", ink, 20)
def on_sample(self, s):
self.sample = s
self.recent.append(s)
@@ -1220,6 +1277,9 @@ class Probe(Adw.ApplicationWindow):
if self.calib:
self.calib_press()
return
if self.reseat_check:
self.reseat_press()
return
if self.mode == "test":
self.test_press()
return
@@ -1961,9 +2021,19 @@ class Probe(Adw.ApplicationWindow):
def own_calib_point(self, samples, target):
"""Send one calibration dot to the Own tracker: the time you looked at it and its
direction. The direction comes from the screen geometry around it, through SteamVR's
nearby gaze and its pixels-per-degree there (only the geometry is used, not where
SteamVR thinks you looked). None if accepted, else why not."""
direction. None if accepted, else why not."""
d = self.screen_direction(samples, target)
if d is None or len(samples) < 2:
return "no gaze samples on this screen: look at the dot and press again"
reply = self.own.ask(f"calib-point {samples[0]['t']:.6f} {samples[-1]['t']:.6f} {d[0]:.4f} {d[1]:.4f}")
if reply.startswith("ok"):
return None
return reply[5:] + ": look at the dot and press again"
def screen_direction(self, samples, target):
"""The head-relative direction (yaw, pitch) of a point on the canvas, from the screen
geometry around it: SteamVR's nearby gaze and its pixels-per-degree there (only the
geometry is used, not where SteamVR thinks you looked). None without samples."""
pts = []
ox, oy = self.origin or (0, 0)
for smp in samples:
@@ -1973,18 +2043,13 @@ class Probe(Adw.ApplicationWindow):
if hit and hit["s"] == self.screen:
pts.append((hit["x"] - ox, hit["y"] - oy, hit["j"], src["hy"], src["hp"]))
break
if len(pts) < 5 or len(samples) < 2:
return "no gaze samples on this screen: look at the dot and press again"
if len(pts) < 5:
return None
x = statistics.median(q[0] for q in pts)
y = statistics.median(q[1] for q in pts)
j = [statistics.fmean(q[2][k] for q in pts) for k in range(4)]
oy_, op_ = deg_from_px(j, target[0] - x, target[1] - y)
yaw = statistics.median(q[3] for q in pts) + oy_
pitch = statistics.median(q[4] for q in pts) + op_
reply = self.own.ask(f"calib-point {samples[0]['t']:.6f} {samples[-1]['t']:.6f} {yaw:.4f} {pitch:.4f}")
if reply.startswith("ok"):
return None
return reply[5:] + ": look at the dot and press again"
return statistics.median(q[3] for q in pts) + oy_, statistics.median(q[4] for q in pts) + op_
def other_dot(self, c, g):
"""Was the gaze on another dot of this round rather than the current one?
@@ -2488,6 +2553,9 @@ class Probe(Adw.ApplicationWindow):
if self.calib:
self.draw_calib(cr, w, h)
return
if self.reseat_check:
self.draw_reseat(cr, w, h)
return
m = self.monitors.get(self.screen)
if m and self.is_fullscreen():
g = m.get_geometry()