steam-frame/DeeJanuz--frametop
1
0 Fork 0
mirror of https://github.com/DeeJanuz/frametop.git synced 2026-10-04 22:00:05 +02:00
Code Issues Packages Projects Releases 3 Wiki Activity

Bring our own eye tracker into the gaze folder, run by the gaze service

Browse Source 
frame-eyes, a separate project until now, becomes gaze/tracker:
- ft-eyegrab (was fe-bufprobe) copies the eye-camera frames, read-only, out of
  SteamVR's eyetracking process. It runs as the system service
  frametop-eyegrab.service, which gaze/tracker/install.sh installs to
  /etc/frametop with sudo. It keeps only CAP_SYS_PTRACE, CAP_DAC_READ_SEARCH, and
  CAP_CHOWN, and copies frames only while /dev/shm/frametop-eyes-want is fresh,
  holding none of the tracker's buffers otherwise.
- ft-eyes (was fe-trackd) runs under ft-gazed in the dev container, with
  build/venv's pinned numpy and OpenCV: while Eye tracker is Own tracker, or on
  the probe's lease ("eyes SECONDS"). No sudo password or fe-live script at
  run time any more.
- lab/ holds the research tools (ft-eyes-score, -e2e, -record, -replay,
  -session) and findings.md. Recordings live outside the repo, in
  ~/.local/share/frametop/eyes/captures; .gitignore catches stray frame dumps.

Its socket is now @ft_eyes, its output /dev/shm/frametop-eyes-gaze, and its state
~/.local/state/frametop/gaze/eyes. On practice1 -> practice2 the whole live path
(ft-eyes-e2e) gives 1.30 deg median and 3.18 for the worst tenth, as before the
move (1.30, 3.21).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed 2026-09-30 10:00:07 -06:00
1 parent 5565a25444
commit ed75614f62
27 files changed
+3021 -45
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitignore
+6
View File
@@ -5,3 +5,9 @@ target/
captures/
__pycache__/
frametop-report-*.txt
# Eye-camera recordings (biometric) never go in the repo: they live in
# ~/.local/share/frametop/eyes/captures. These catch strays (frame dumps, a lab venv).
*.raw
*.pgm
.venv/
.frame-job.d/
gaze/README.md
+19 -2
View File
@@ -4,6 +4,7 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (t
- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on.
- `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log.
- `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground. It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
```
@@ -12,6 +13,7 @@ gaze/probe/install.sh # build, and add Frametop Gaze Probe to the app me
gaze/probe/ft-gazeprobe --screen 1
gaze/run.sh install # the gaze service, with SteamVR
gaze/ft-gazectl on # the pointer follows your gaze (off: the mouse alone)
gaze/tracker/install.sh # our own eye tracker's frame grabber (asks for sudo)
```
## Gaze pointer
@@ -19,7 +21,7 @@ gaze/ft-gazectl on # the pointer follows your gaze (off: the mouse al
Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:
- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes:
- **Eye tracker:** SteamVR's (the default), or our own (Own tracker, from `~/Desktop/Projects/frame-eyes`, while its `tools/fe-live` runs). Ours keeps its own calibration: calibrate it in the probe with the tracker toggle on Own tracker. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
- **Eye tracker:** SteamVR's (the default), or our own (Own tracker: see "Our own eye tracker" below). The gaze service runs ours while this is on. It keeps its own calibration: calibrate it in the probe with the tracker toggle on Own tracker. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
- **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
@@ -38,10 +40,25 @@ Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction,
| SteamVR action | An `eyetracking` action bound to `/user/head/eyetracking` (`actions/`), read with `IVRInput::GetEyeTrackingDataRelativeToNow`. This is the supported way. |
| mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space: set 1 is filtered, and its two eyes always share one pitch; set 2 is each eye's own reading. After each set come the tracker's variances for each eye, and at the end each eye's raw measurement and its variance (the tracker's confidence in that frame), which ft-gaze passes on for the fit check. |
| Left eye, right eye | Each eye alone, from set 2: calibrate and test them to see what one eye is worth against both. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. |
| Own tracker | Our own tracker, from `~/Desktop/Projects/frame-eyes` (experimental). `tools/fe-live` runs it: a root helper copies the eye-camera frames out of SteamVR's buffers (read-only) and `fe-trackd` finds the pupils and glints and publishes the gaze in `/dev/shm/frame-eyes-gaze`. It keeps its own calibration, not SteamVR's: the probe's calibration with the tracker toggle on Own tracker fits it (its dots go out to the Calibration ring angle each way, on an oval, since the fit goes wrong past its dots), and practice clicks teach it how far the headset has moved on your face since. After the headset was off, the probe first asks for one look at a centre dot, which resets that. With Own tracker on, the probe hides SteamVR's gaze and draws a red dot where each eye alone puts it. The gaze pointer can use it too (Eye tracker: Own tracker, on the Gaze page of Frametop Input Settings). ft-gaze reports it as `own` while it's running, and as `{"ok":0}` otherwise. |
| Own tracker | Our own tracker (`tracker/`, experimental; see "Our own eye tracker"). It keeps its own calibration, not SteamVR's: the probe's calibration with the tracker toggle on Own tracker fits it (its dots go out to the Calibration ring angle each way, on an oval, since the fit goes wrong past its dots), and practice clicks teach it how far the headset has moved on your face since. After the headset was off, the probe first asks for one look at a centre dot, which resets that. With Own tracker on, the probe hides SteamVR's gaze and draws a red dot where each eye alone puts it, and asks the gaze service to keep the tracker running. The gaze pointer can use it too (Eye tracker: Own tracker, on the Gaze page of Frametop Input Settings). ft-gaze reports it as `own` while it's running, and as `{"ok":0}` otherwise. |
The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`. When the tracker loses an eye, the same log says `CEyePoseUKF L: Large dt` (or `R`) as it starts that eye over.
## Our own eye tracker
`gaze/tracker/` is an eye tracker of our own, because SteamVR's is about 1.5 degrees off after the best correction the gaze service can learn, and what's left is mostly look-to-look noise that no correction on top of its output can remove. Ours processes the eye cameras itself: 0.59 degrees (median) in its best live session against 0.83 for SteamVR's with the probe's correction, and after the headset was taken off and put back without recalibrating, 0.58 once your first clicks had taught it where the headset sat (`tracker/findings.md` has the measurements).
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too).
- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`.
- `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones run on a PC through `frame-job` (`gaze/tracker/.frame-job`). `lab/py` runs them with that Python (in the dev container on the Frame; frame-job's setup makes the same venv on the PC).
Ground rules, for anyone changing it:
- **Clean room.** Nothing of Valve's goes in: we don't decompile, disassemble, or patch the `eyetracking` binary or its network weights, and we don't copy their code or weights. Its public output (eye-server.mmap, read-only) is fair game as a baseline and as labels, and so are published papers and openly licensed pupil detectors (check each one's license: PuRe, PuReST, ElSe, and ExCuSe are non-commercial only).
- **Root only reads.** ft-eyegrab never writes to, stops, or signals the `eyetracking` process, vrserver, or vrcompositor, never opens `/dev/adsp`, `/dev/cdsp`, or `/dev/spidev0.1`, and never writes to `/dev/shm/eye-server.mmap` (it also carries calibration clicks into SteamVR's tracker), `/opt`, or `/persist`.
- **Eye images are biometric data.** Recordings live outside the repo, in `~/.local/share/frametop/eyes/captures` (0700), and `.gitignore` catches stray frame dumps. They go nowhere but the machine that runs your offline jobs.
- **Mind the headset's budget.** Finding a pupil takes about 0.4 ms a frame while ft-eyes follows it, and 1.4-2.1 ms when it searches the whole frame. Replays, scoring, and training go to a PC.
## Headset fit
The probe's Headset fit mode (Check headset fit on the Gaze page, or `ft-gazeprobe --mode fit`) shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it.
gaze/ft-gaze.cpp
+7 -7
View File
@@ -21,8 +21,8 @@
// (set 1's eyes always share one pitch, and while it's lost an eye it keeps that
// eye's yaw where it was: set 2 is each eye's own reading). From the head's origin,
// not the eye's.
// "own":SRC our own tracker (frame-eyes fe-trackd), from
// /dev/shm/frame-eyes-gaze; adds "age" (ms since its frame), "eyes":[[hy,hp],[hy,hp]]
// "own":SRC our own tracker (gaze/tracker/ft-eyes), from
// /dev/shm/frametop-eyes-gaze; adds "age" (ms since its frame), "eyes":[[hy,hp],[hy,hp]]
// (left, right; null for an eye it doesn't see), "ehit":[HIT,HIT] where each of those
// lands, and "slip":[[x,y],[x,y]] (left, right: each eye's shift in its camera image
// since the calibration, pixels; null until a click has measured it). {"ok":0}
@@ -163,8 +163,8 @@ bool ReadSample(const EyeFile &f, EyeSample &s) {
return false;
}
// --- Our own tracker: /dev/shm/frame-eyes-gaze, written by frame-eyes' fe-trackd ---
// Layout (fe-trackd's docstring): u32 seq (odd while written), u32 version, f64 t, f32 yaw,
// --- Our own tracker: /dev/shm/frametop-eyes-gaze, written by gaze/tracker/ft-eyes ---
// Layout (ft-eyes' docstring): u32 seq (odd while written), u32 version, f64 t, f32 yaw,
// pitch, u32 flags (bit 0 right eye, 1 left, 2 right slip known, 3 left), u32 n, then f32
// right yaw, pitch, left yaw, pitch; slip right x, y, left x, y; pupils (unused here).
struct OwnSample {
@@ -176,7 +176,7 @@ struct OwnSample {
class OwnFile {
public:
// Reopened when it appears or is replaced, since fe-trackd may start after us.
// Reopened when it appears or is replaced, since ft-eyes may start after us.
bool Read(OwnSample &o) {
const double now = NowRaw();
if (!p_ || now - checked_ > 2.0) Reopen(now);
@@ -206,10 +206,10 @@ private:
void Reopen(double now) {
checked_ = now;
struct stat st {};
if (stat("/dev/shm/frame-eyes-gaze", &st) != 0) return Close();
if (stat("/dev/shm/frametop-eyes-gaze", &st) != 0) return Close();
if (p_ && st.st_ino == ino_) return;
Close();
const int fd = open("/dev/shm/frame-eyes-gaze", O_RDONLY | O_CLOEXEC);
const int fd = open("/dev/shm/frametop-eyes-gaze", O_RDONLY | O_CLOEXEC);
if (fd < 0) return;
if (fstat(fd, &st) == 0 && size_t(st.st_size) >= kSize) {
void *m = mmap(nullptr, kSize, PROT_READ, MAP_SHARED, fd, 0);
gaze/ft-gazed
+104 -17
View File
@@ -3,8 +3,8 @@
Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read
again when the file changes:
GAZE_TRACKER=steam|own SteamVR's eye tracker (default), or our own (frame-eyes' fe-trackd,
ft-gaze's source "own", when it's running)
GAZE_TRACKER=steam|own SteamVR's eye tracker (default), or our own (gaze/tracker/ft-eyes,
ft-gaze's source "own"; this service runs it, see below)
GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far
off it was at your recent nudges; left or right counts that eye twice
as much as the other. Either eye alone carries the gaze when the
@@ -46,7 +46,7 @@ there). The gap is the tracker's error there. The raw gaze is the one sent, so i
when that look was (the history of what was sent), and so what each eye read then:
- SteamVR: each eye learns its own error, unless the gaze was more than LESSON_MAX degrees
past the correction (then it wasn't a nudge onto what you looked at);
- our tracker: the look goes to it as a click ("click T YAW PITCH" on @frame_eyes), as the
- our tracker: the look goes to it as a click ("click T YAW PITCH" on @ft_eyes), as the
probe's clicks do, and it learns how far the headset has moved on your face. That's
what it gets wrong, and after the headset was off, the first click resets it;
- either way, how far off each eye was (before the lesson taught it anything) goes to the
@@ -56,6 +56,11 @@ SteamVR's eye tracking log is followed for the headset going on (its eye model s
and the error moves): lessons from before count less then, so the first few after it
relearn the offset.
Our own tracker (gaze/tracker/ft-eyes) runs here too, in the dev container, while
GAZE_TRACKER=own or the gaze probe asks for it ("eyes SECONDS", a lease the probe renews). It
reads the eye-camera frames the root service frametop-eyegrab copies (gaze/tracker/install.sh),
which copies them only while ft-eyes runs.
Nothing here writes to SteamVR, its eye tracker, or its files: ft-gaze reads the eye
tracker's shared memory read-only.
@@ -64,6 +69,8 @@ Control socket: abstract unix datagram "@ft_gazed":
status reply: one JSON object
forget drop what the lessons taught (the calibration stays)
reload read calibration.json and the settings again
eyes <seconds> keep our own tracker running that much longer (at most 120),
whatever GAZE_TRACKER says: the probe's lease. Reply: "ok"
Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever
the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever
@@ -94,13 +101,16 @@ REPO = Path(__file__).resolve().parents[1]
HELPER = REPO / "gaze" / "build" / "ft-gaze"
ME = "\0ft_gazed"
POINTER = "\0ft_pointer_helper"
FRAME_EYES = "\0frame_eyes" # our own tracker's control socket (frame-eyes fe-trackd)
EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes" # our own tracker
EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python" # numpy, OpenCV (build.sh)
EYES_SOCKET = "\0ft_eyes" # its control socket
EYES_CAMS = Path("/dev/shm/frametop-eyes-cams") # the frames it reads (frametop-eyegrab.service)
CONF = Path.home() / ".config" / "frametop.conf"
CALIBRATION = STATE / "calibration.json"
LESSONS = STATE / "pointer-lessons.json"
LESSON_LOG = STATE / "pointer-lessons.jsonl"
SOURCES = ("action", "mmap1", "mmap2", "left", "right") # the ones with a calibration here
EYES = ("left", "right") # ft-gaze's order, and the sources for each eye alone
SIDES = ("left", "right") # ft-gaze's order, and the sources for each eye alone
TRACKERS = ("steam", "own")
BIASES = ("auto", "left", "right")
LESSON_MAX = 8.0 # degrees past the correction
@@ -108,6 +118,8 @@ OWN_LESSON_MAX = 25.0 # our tracker: after the headset was off, its first click
HISTORY = 12.0 # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later)
LOOK = 0.3 # seconds of samples before that moment make the look (the probe's fixation)
RETRY = 3.0 # seconds before starting ft-gaze again
EYES_RETRY = 10.0 # seconds before starting ft-eyes again after it stopped on its own
EYES_LEASE_MAX = 120.0
SETTLE = 0.3 # seconds after an eye is found again before the fallback learns from it
KEYBOARD_PITCH = -20.0 # degrees: gaze under this, on no screen, is a look at the keyboard
@@ -184,6 +196,9 @@ class Service:
self.history = deque()
self.own = {} # our tracker's last status reply
self.own_at = 0.0
self.eyes_proc = None # ft-eyes, while it runs
self.eyes_until = 0.0 # the probe's lease (monotonic time)
self.eyes_restart_at = 0.0
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
self.sock.bind(ME)
@@ -207,7 +222,7 @@ class Service:
return "source"
if self.tracker == "own":
return "own"
return "eyes" if all(self.models[e].samples for e in EYES) else "source"
return "eyes" if all(self.models[e].samples for e in SIDES) else "source"
# --- Settings, calibration and lessons ---
@@ -233,7 +248,7 @@ class Service:
self.models[name].from_json(d[name])
mode = d.get("_meta", {}).get("model")
self.mode = mode if mode in MODELS else DEFAULT_MODEL
log(f"calibration: {self.mode}, " + ", ".join(f"{n} {self.models[n].samples}" for n in (self.source,) + EYES)
log(f"calibration: {self.mode}, " + ", ".join(f"{n} {self.models[n].samples}" for n in (self.source,) + SIDES)
+ " samples")
def load_lessons(self):
@@ -249,7 +264,7 @@ class Service:
for t, misses in (d.get("misses") or {}).items():
if t in self.weights:
self.weights[t] = EyeWeights(self.bias, misses)
log(", ".join(f"{n} {len(self.lives[n].samples)}" for n in (self.source,) + EYES) + " lessons")
log(", ".join(f"{n} {len(self.lives[n].samples)}" for n in (self.source,) + SIDES) + " lessons")
def save_lessons(self):
tmp = LESSONS.with_suffix(".tmp")
@@ -312,13 +327,13 @@ class Service:
# Its eyes come calibrated: how far off each was is its miss. The click goes to it.
miss = [math.hypot(thy - e[0], thp - e[1]) if e else None for e in eyes]
try:
self.eyes_sock.sendto(f"click {t:.6f} {thy:.4f} {thp:.4f}".encode(), FRAME_EYES)
self.eyes_sock.sendto(f"click {t:.6f} {thy:.4f} {thp:.4f}".encode(), EYES_SOCKET)
except OSError as e:
rec["refused"] = f"our tracker isn't running ({e})"
return self.taken(rec)
else:
miss = []
for name, e in zip(EYES, eyes):
for name, e in zip(SIDES, eyes):
if not e:
miss.append(None)
continue
@@ -384,6 +399,61 @@ class Service:
pass
self.proc = None
def eyes_wanted(self):
return (self.tracker == "own" and not self.override) or time.monotonic() < self.eyes_until
def start_eyes(self):
"""Our own tracker, in the dev container, with build/venv's numpy and OpenCV. Like
ft-gaze, it quits when its stdin closes."""
if not EYES_PYTHON.exists():
log(f"ft-eyes isn't built: run {REPO}/gaze/tracker/build.sh")
self.eyes_restart_at = time.monotonic() + 30
return
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
distrobox = Path.home() / ".local" / "bin" / "distrobox"
self.eyes_proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(EYES_PYTHON), str(EYES_PROG), "-v",
"--watch-stdin"], env=env, stdin=subprocess.PIPE,
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, start_new_session=True)
os.set_blocking(self.eyes_proc.stderr.fileno(), False)
self.sel.register(self.eyes_proc.stderr, selectors.EVENT_READ, "eyes")
log("ft-eyes started" + ("" if EYES_CAMS.exists() else
f": no {EYES_CAMS} yet (the frame grabber: gaze/tracker/install.sh)"))
def stop_eyes(self):
if not self.eyes_proc:
return
try:
self.sel.unregister(self.eyes_proc.stderr)
except (KeyError, ValueError):
pass
if self.eyes_proc.stdin and not self.eyes_proc.stdin.closed:
self.eyes_proc.stdin.close()
try:
self.eyes_proc.wait(timeout=3)
except subprocess.TimeoutExpired:
try:
os.killpg(self.eyes_proc.pid, signal.SIGTERM)
except ProcessLookupError:
pass
self.eyes_proc = None
self.own = {}
def read_eyes(self):
try:
data = os.read(self.eyes_proc.stderr.fileno(), 65536)
except BlockingIOError:
return
if not data:
log(f"ft-eyes stopped (exit {self.eyes_proc.poll()}); again in {EYES_RETRY:.0f} s if still wanted")
self.stop_eyes()
self.eyes_restart_at = time.monotonic() + EYES_RETRY
return
for line in data.decode("utf-8", "replace").splitlines():
if line.strip() and (self.verbose or "fps" not in line):
log(line)
def read_stdout(self):
try:
data = os.read(self.proc.stdout.fileno(), 65536)
@@ -456,7 +526,7 @@ class Service:
eyes = [tuple(e) if e else None for e in (src.get("eyes") or [None, None])]
hp, hit = src["hp"], src.get("hit")
else:
per = [s["src"].get(name) or {} for name in EYES]
per = [s["src"].get(name) or {} for name in SIDES]
eyes = [(p["hy"], p["hp"]) if "hy" in p else None for p in per]
if not any(eyes):
return
@@ -481,7 +551,7 @@ class Service:
corrected = seen
else:
corrected = []
for name, e in zip(EYES, seen):
for name, e in zip(SIDES, seen):
c = self.correction(name, *e) if e else None
corrected.append((e[0] + c[0], e[1] + c[1]) if e else None)
gy, gp = self.weights["own" if own else "steam"].combine(corrected)
@@ -581,6 +651,13 @@ class Service:
self.refit()
self.save_lessons()
reply = "ok"
elif words[:1] == ["eyes"] and len(words) == 2:
try:
secs = min(max(float(words[1]), 0.0), EYES_LEASE_MAX)
self.eyes_until = max(self.eyes_until, time.monotonic() + secs)
reply = "ok"
except ValueError:
reply = "error bad seconds"
elif words[:1] == ["reload"]:
self.load_settings()
self.load_calibration()
@@ -623,14 +700,15 @@ class Service:
st.update(eye_weights=[round(v, 3) for v in w.weights()], eye_misses=[len(m) for m in w.misses],
eye_rms=[None if r is None else round(r, 2) for r in w.rms()])
if kind == "eyes":
st.update(calibration_samples=min(self.models[e].samples for e in EYES),
lessons=max(len(self.lives[e].samples) for e in EYES))
st.update(calibration_samples=min(self.models[e].samples for e in SIDES),
lessons=max(len(self.lives[e].samples) for e in SIDES))
if kind == "own":
own = self.own if time.monotonic() - self.own_at < 5 else {}
cal = own.get("calibration") or {}
st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
lessons=max(len(m) for m in w.misses), own_running=bool(own),
own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists())
st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
if self.last_sample else None, "headset_on": self.steam.wearing(),
"headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None,
@@ -648,11 +726,17 @@ class Service:
self.refit()
if mtime(CONF) != self.conf_mtime:
self.load_settings()
if self.tracker == "own":
want = self.eyes_wanted()
if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:
self.start_eyes()
elif not want and self.eyes_proc:
log("ft-eyes no longer wanted: stopping it")
self.stop_eyes()
if self.eyes_proc:
try:
self.eyes_sock.sendto(b"status", FRAME_EYES)
self.eyes_sock.sendto(b"status", EYES_SOCKET)
except OSError:
pass # not running: status() says so once the last answer is old
pass # not up yet: status() says so once the last answer is old
if self.dirty:
self.save_lessons()
@@ -668,6 +752,8 @@ class Service:
self.on_control()
elif key.data == "own":
self.on_own()
elif key.data == "eyes" and self.eyes_proc:
self.read_eyes()
elif key.data == "stdout" and self.proc:
self.read_stdout()
elif key.data == "stderr" and self.proc:
@@ -679,6 +765,7 @@ class Service:
log(json.dumps(self.status()))
next_verbose = now + 5
self.stop_helper()
self.stop_eyes()
if self.dirty:
self.save_lessons()
gaze/gazecal.py
+1 -1
View File
@@ -479,7 +479,7 @@ class EyeWeights:
"""How much each eye (0 left, 1 right) counts in the gaze, for ft-gazed's eye bias.
Two eyes beat either one: their errors partly cancel. On 306 live clicks with our own
tracker (frame-eyes, 2026-09-29) the eyes' sideways errors were correlated -0.37, and
tracker (gaze/tracker, 2026-09-29) the eyes' sideways errors were correlated -0.37, and
the mean of both was 0.65 degrees off (median), the left eye alone 0.96, the right 1.11.
So a bias leans instead of choosing: "left" or "right" counts that eye LEAN times the
other (on those clicks, 2:1 toward the better eye cost about 0.03 degrees, toward the
gaze/probe/ft-gazeprobe
+33 -12
View File
@@ -80,7 +80,7 @@ HELPER = REPO / "gaze" / "build" / "ft-gaze"
STATE = Path.home() / ".local" / "state" / "frametop" / "gaze"
# left and right: each eye alone (set 2's own reading of that eye), calibrated and tested
# like the rest, to see what one eye is worth against both. own: our own tracker
# (~/Desktop/Projects/frame-eyes, fe-trackd), when it's running; it has its own calibration.
# (gaze/tracker/ft-eyes, which the gaze service runs); it has its own calibration.
SOURCES = ["action", "mmap1", "mmap2", "left", "right", "own"]
SOURCE_NAMES = {"action": "SteamVR action", "mmap1": "mmap set 1", "mmap2": "mmap set 2", "left": "Left eye",
"right": "Right eye", "own": "Own tracker"}
@@ -90,9 +90,11 @@ TRIGGER_KEYS = {Gdk.KEY_Return, Gdk.KEY_KP_Enter, Gdk.KEY_space}
class OwnTracker:
"""Talks to our own tracker (frame-eyes fe-trackd) over its control socket (@frame_eyes).
"""Talks to our own tracker (gaze/tracker/ft-eyes) over its control socket (@ft_eyes).
It keeps its own calibration: the probe sends it calibration dots and clicks, each with
when you looked and where (head-relative degrees), and it learns from its own frames."""
when you looked and where (head-relative degrees), and it learns from its own frames.
The gaze service (ft-gazed) runs it: `lease` asks it to keep it running a while longer,
whatever the Eye tracker setting says."""
def __init__(self):
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
@@ -109,13 +111,22 @@ class OwnTracker:
pass
self.sock.settimeout(1.0)
try:
self.sock.sendto(command.encode(), "\0frame_eyes")
self.sock.sendto(command.encode(), "\0ft_eyes")
return self.sock.recv(4096).decode()
except (ConnectionRefusedError, FileNotFoundError):
return "fail the Own tracker isn't running (frame-eyes: tools/fe-live)"
return "fail the Own tracker isn't running yet (the gaze service starts it)"
except (socket.timeout, OSError) as e:
return f"fail no answer from the Own tracker ({e})"
def lease(self, seconds=30):
"""Ask the gaze service to keep the Own tracker running for `seconds` more. False if
the gaze service isn't running."""
try:
self.sock.sendto(f"eyes {seconds}".encode(), "\0ft_gazed")
return True
except OSError:
return False
# The math, filters, correction models, and SteamVR log reader are shared with ft-gazed.
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
@@ -915,9 +926,14 @@ class Probe(Adw.ApplicationWindow):
if own != (self.source == "own"):
self.w_source.set_selected(SOURCES.index("own") if own else SOURCES.index("mmap2"))
if own:
if not self.own.lease():
self.on_status("The gaze service isn't running, and it runs the Own tracker "
"(frametop-gaze.service, gaze/run.sh install)")
return
self.lease_at = time.monotonic()
reply = self.own.ask("status")
if reply.startswith("fail"):
self.on_status(reply[5:])
self.on_status("Starting the Own tracker (a few seconds)…")
else:
st = json.loads(reply)
self.on_status("Own tracker: " + ("calibrated " + st["calibration"].get("made", "")
@@ -1042,9 +1058,14 @@ class Probe(Adw.ApplicationWindow):
def poll_own(self):
"""With the Own tracker: after the headset was off (or the tracker restarted), its
next click starts each eye's shift over, and until then the first clicks can be
10-17 degrees off (frame-eyes, session 5). So ask for one look at a centre dot first,
as Varjo's headsets do each time they're put on."""
if self.source != "own" or self.calib or self.capture:
10-17 degrees off (gaze/tracker/findings.md, session 5). So ask for one look at a centre dot first,
as Varjo's headsets do each time they're put on. Also keeps the lease on the tracker."""
if self.source != "own":
return
if time.monotonic() - getattr(self, "lease_at", 0.0) > 10:
self.lease_at = time.monotonic()
self.own.lease()
if self.calib or self.capture:
return
reply = self.own.ask("status")
if reply.startswith("fail"):
@@ -1535,7 +1556,7 @@ class Probe(Adw.ApplicationWindow):
SNAP_LEARN_MAX = 6.0 # degrees: a lesson bigger than this (after the correction) is a wrong element
# The Own tracker takes bigger ones: after the headset is taken off and put back on, its
# first clicks were 11-17 degrees off, and a 6-degree limit kept them from teaching it
# (frame-eyes, session 5). It keeps the median of an eye's last 5 clicks, so one click
# (gaze/tracker/findings.md, session 5). It keeps the median of an eye's last 5 clicks, so one click
# where you changed your mind does little harm.
OWN_LEARN_MAX = 25.0
@@ -1897,7 +1918,7 @@ class Probe(Adw.ApplicationWindow):
ring` degrees each way (as much as fits across and up the window), turned 20 degrees
per round, and half the size in the middle round. Its fit is quadratic and goes
wrong past its dots, and the ring (limited by the window's height) left the sides
out: in frame-eyes' practice2 the worst clicks were all past the ring."""
out: in practice2 (gaze/tracker/findings.md) the worst clicks were all past the ring."""
w, h = self.canvas_size()
cx, cy = w / 2, h / 2
r = self.raw.get(self.source)
@@ -2836,7 +2857,7 @@ def main():
ap.add_argument("--screen", type=int, default=0, help="Frametop screen to open on (default: where it opens)")
ap.add_argument("--mode", choices=Probe.MODE_KEYS, help="start in this mode (fit: Headset fit)")
ap.add_argument("--source", choices=SOURCES,
help="gaze source to start with (default: own if frame-eyes' tracker is running, else mmap2)")
help="gaze source to start with (default: own if our tracker is running, else mmap2)")
args, rest = ap.parse_known_args()
app = Adw.Application(application_id="dev.frametop.GazeProbe", flags=Gio.ApplicationFlags.NON_UNIQUE)
windows = []
gaze/tracker/.frame-job
+14
View File
@@ -0,0 +1,14 @@
# frame-job settings (see `frame-job --help`): offline lab jobs (ft-eyes-score, ft-eyes-e2e)
# run on the 7i. Eye recordings may go there and nowhere else, and never into the repo: they
# live outside it, in ~/.local/share/frametop/eyes/captures, and on the 7i in
# ~/frame-compute/frametop-eyes/data/captures.
NAME=frametop-eyes
RUN_ON=7i
DATA="$HOME/.local/share/frametop/eyes/captures"
RESULTS=""
EXCLUDE=""
# The lab's Python on the 7i: build/venv from requirements.txt, as build.sh makes it on the Frame.
SETUP='cmp -s requirements.txt build/venv/requirements.done || { rm -rf build/venv && python3 -m venv build/venv && build/venv/bin/pip install -q --disable-pip-version-check -r requirements.txt && cp requirements.txt build/venv/requirements.done; }'
VENV=
# Live tools: they read the Frame's eye cameras.
LOCAL_ONLY="ft-eyes ft-eyes-record ft-eyes-session"
gaze/tracker/build.sh
Executable
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# Build our own eye tracker on the Frame, in the dev container:
# build/ft-eyegrab the frame grabber. It runs on the host, as root
# (frametop-eyegrab.service, gaze/tracker/install.sh), so this checks it
# only needs glibc symbols the SteamOS host has (2.39; the container has 2.43).
# build/venv Python with numpy and OpenCV (requirements.txt) for ft-eyes and lab/,
# remade when requirements.txt changes.
# Usage: gaze/tracker/build.sh
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C gaze/tracker 'set -e; mkdir -p build
gcc -std=gnu11 -O2 -Wall -Wextra -pthread -o build/ft-eyegrab ft-eyegrab.c
max=$(objdump -T build/ft-eyegrab | grep -oE "GLIBC_[0-9.]+" | sort -uV | tail -1)
echo "built build/ft-eyegrab, newest glibc symbol: $max"
[ "$(printf "%s\n" "$max" GLIBC_2.39 | sort -V | tail -1)" = GLIBC_2.39 ] || { echo "needs newer glibc than the host has" >&2; exit 1; }
if ! cmp -s requirements.txt build/venv/requirements.done; then
rm -rf build/venv
python3 -m venv build/venv
build/venv/bin/pip install -q --disable-pip-version-check -r requirements.txt
cp requirements.txt build/venv/requirements.done
fi
echo "build/venv: $(build/venv/bin/python -c "import numpy, cv2; print(\"numpy\", numpy.__version__, \"opencv\", cv2.__version__)")"'
gaze/tracker/eyes_model.py
+243
View File
@@ -0,0 +1,243 @@
"""The gaze calibration: from pupil and glint positions to head-relative gaze angles.
Shared by ft-eyes (live) and the lab tools (fitting and scoring on recordings). Gaze angles are
ft-gaze's: degrees relative to the head, yaw positive to the left, pitch positive up.
Per eye (0 = right, camera 0; 1 = left, camera 1), three quadratic fits:
pupil pupil centre -> gaze. The one used for output, after the slip correction.
glint pupil minus the glint pair's midpoint -> gaze. Slip moves both alike, so this
holds up when the headset shifts, but it's noisier, and the pair is often gone.
where gaze -> pupil centre: where the pupil sits for a gaze, with no slip.
Slip: wherever the pair is seen, `glint` gives the gaze, `where` says where the pupil should
be, and the difference is how far the eye has moved in the image. Slip changes slowly, so
the median over the last SLIP_WINDOW seconds shifts every frame, glints or not.
That glint estimate is only good to 1-4 px (1-3 degrees), though. Clicks are better: each one
says where the pupil should have been for a known gaze (`where`), so pupil minus that is the
shift. `Shift` keeps the median of the last few, and uses the glints only to notice a sudden
jump (the headset nudged or put back on), until clicks catch up. On practice2 with
practice1's calibration: 1.2 degrees median, against 2.7 with the glints alone (findings.md).
"""
import json
import time
from collections import deque
import numpy as np
SLIP_WINDOW = 30.0
SLIP_MIN = 10 # pair sightings needed before trusting a slip estimate
MIN_CLICKS = 12
SPREAD_MIN = 0.3 # degrees: floor for an eye's fit spread, so one eye can't take all the weight
SHIFT_KEEP = 5 # clicks in the shift estimate
SHIFT_JUMP = 3.0 # a glint slip change this big (px) since the last click is a nudge
JUMP_HOLD = 1.0 # s: ...if it holds this long (a bad glint pair gives a jump that snaps back)
JUMP_MAX = 40.0 # px: bigger is a bad glint pair, not the headset (a re-seat moved 20-30)
JUMP_WINDOW = 10.0 # seconds of glint sightings for noticing a jump
class Quad:
"""Ridged quadratic least squares from 2-D inputs, normalised on the training set."""
def __init__(self, X=None, Y=None, ridge=1e-3):
if X is None:
return
X, Y = np.asarray(X, float), np.asarray(Y, float)
self.m, self.s = X.mean(0), X.std(0) + 1e-9
A = self.terms(X)
R = ridge * np.eye(A.shape[1])
R[0, 0] = 0
self.w = np.linalg.solve(A.T @ A + R, A.T @ Y)
def terms(self, X):
P = (np.atleast_2d(np.asarray(X, float)) - self.m) / self.s
return np.c_[np.ones(len(P)), P, P ** 2, P[:, 0] * P[:, 1]]
def __call__(self, X):
return self.terms(X) @ self.w
def one(self, x, y):
"""Faster for a single point (the live path)."""
px, py = (x - self.m[0]) / self.s[0], (y - self.m[1]) / self.s[1]
t = np.array([1.0, px, py, px * px, py * py, px * py])
return t @ self.w
def to_json(self):
return {"m": self.m.tolist(), "s": self.s.tolist(), "w": self.w.tolist()}
@classmethod
def from_json(cls, d):
q = cls()
q.m, q.s, q.w = (np.array(d[k]) for k in ("m", "s", "w"))
return q
def pair_mid(pair):
return ((pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2)
class Calibration:
"""The three fits per eye. Build from clicks (ft-eyes-score's features) or load from JSON.
`spread` is each eye's RMS miss (degrees) on the clicks its pupil fit was made from.
`combine` weights the eyes by its inverse square: on practice2 (one headset position,
leave-one-out) that gave 0.75 median against 0.84 for the plain average, since one eye
is usually much better than the other (left 0.73, right 1.32 there)."""
def __init__(self, fits=None, info=None, spread=None):
self.fits = fits or {} # (name, eye) -> Quad
self.info = info or {}
self.spread = spread or {} # eye -> degrees
@classmethod
def fit(cls, clicks, info=None):
truth = lambda cs: np.array([k["truth"] for k in cs]) # noqa: E731
fits, spread = {}, {}
for c in (0, 1):
cs = [k for k in clicks if k["eye"][c] is not None]
if len(cs) < MIN_CLICKS:
continue
fits["pupil", c] = Quad([k["eye"][c]["pupil"] for k in cs], truth(cs))
miss = fits["pupil", c]([k["eye"][c]["pupil"] for k in cs]) - truth(cs)
spread[c] = float(np.sqrt(np.mean(np.sum(miss ** 2, axis=1))))
fits["where", c] = Quad(truth(cs), [k["eye"][c]["pupil"] for k in cs])
gs = [k for k in cs if k["eye"][c]["mid"] is not None]
if len(gs) >= MIN_CLICKS:
fits["glint", c] = Quad([k["eye"][c]["pupil"] - k["eye"][c]["mid"] for k in gs], truth(gs))
return cls(fits, info, spread)
def has(self, name, eye):
return (name, eye) in self.fits
def slip(self, eye, rows):
"""Median slip in pixels from pair sightings `rows` (t, px, py, mx, my), or None."""
if not self.has("glint", eye) or len(rows) < SLIP_MIN:
return None
rows = np.asarray(rows, float)
gaze = self.fits["glint", eye](rows[:, 1:3] - rows[:, 3:5])
return np.median(rows[:, 1:3] - self.fits["where", eye](gaze), axis=0)
def click_shift(self, eye, pupil, truth):
"""The shift a click measures: the pupil, less where it sits for that gaze."""
return np.asarray(pupil, float) - self.fits["where", eye].one(*truth)
def gaze(self, eye, x, y, slip=None):
"""Gaze (yaw, pitch) for a pupil centre, less a slip if there is one."""
if slip is not None:
x, y = x - slip[0], y - slip[1]
return self.fits["pupil", eye].one(x, y)
def weight(self, eye):
s = self.spread.get(eye)
return 1.0 if s is None else 1.0 / max(s, SPREAD_MIN) ** 2
def combine(self, gazes):
"""The weighted mean of {eye: (yaw, pitch)}, or None if empty."""
if not gazes:
return None
w = {c: self.weight(c) for c in gazes}
return sum(w[c] * np.asarray(g, float) for c, g in gazes.items()) / sum(w.values())
def save(self, path):
d = {"version": 1, "info": self.info, "spread": {str(e): s for e, s in self.spread.items()},
"fits": [{"name": n, "eye": e, **q.to_json()} for (n, e), q in self.fits.items()]}
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps(d, indent=1))
tmp.replace(path)
@classmethod
def load(cls, path):
d = json.loads(path.read_text())
return cls({(f["name"], f["eye"]): Quad.from_json(f) for f in d["fits"]}, d.get("info"),
{int(e): s for e, s in d.get("spread", {}).items()})
class Shift:
"""Where one eye sits in the image now, relative to the calibration (pixels).
`base` is the median shift the last SHIFT_KEEP clicks measured. `ref` is the glint slip
estimate at the last click; if the glint estimate has since moved more than SHIFT_JUMP
(and less than JUMP_MAX) and stayed there for JUMP_HOLD seconds, the headset moved, and
the change is added until the next click. That click then starts the history over,
since the older ones describe the old position. Live, bad glint pairs made the estimate
leap by up to 68 px for under a second (practice2), hence the hold. `reseat` does the
same for the next click without the glints: the frames stopped (the headset was off),
so the headset may be anywhere now."""
def __init__(self, base=(0.0, 0.0), ref=None):
self.meas = []
self.base = np.asarray(base, float)
self.ref = None if ref is None else np.asarray(ref, float)
self.jump = np.zeros(2)
self.held = None # (change, since when) while a jump waits out JUMP_HOLD
self.reseated = False
def glint(self, g, t):
"""The latest glint slip estimate (or None), at time t (s)."""
if g is None:
return
if self.ref is None:
self.ref = np.asarray(g, float)
d = np.asarray(g, float) - self.ref
size = np.hypot(*d)
if size > JUMP_MAX:
return
if size <= SHIFT_JUMP:
self.jump, self.held = np.zeros(2), None
return
if self.held is None or np.hypot(*(d - self.held[0])) > SHIFT_JUMP:
self.held = (d, t)
elif t - self.held[1] >= JUMP_HOLD:
self.jump = d
def reseat(self):
self.reseated = True
def click(self, d, g=None):
"""A click measured the shift d; g is the glint estimate then."""
if np.any(self.jump) or self.reseated:
self.meas = []
self.reseated = False
self.meas = (self.meas + [np.asarray(d, float)])[-SHIFT_KEEP:]
self.base = np.median(self.meas, axis=0)
self.jump, self.held = np.zeros(2), None
if g is not None:
self.ref = np.asarray(g, float)
@property
def value(self):
return self.base + self.jump
def to_json(self):
return {"base": self.base.tolist(), "ref": None if self.ref is None else self.ref.tolist(),
"meas": [m.tolist() for m in self.meas]}
@classmethod
def from_json(cls, d):
s = cls(d.get("base", (0, 0)), d.get("ref"))
s.meas = [np.asarray(m, float) for m in d.get("meas", [])]
return s
class SlipTracker:
"""Live slip estimate for one eye: pair sightings over the last SLIP_WINDOW seconds,
re-estimated at most every `every` seconds."""
def __init__(self, cal, eye, every=0.5, window=SLIP_WINDOW):
self.cal, self.eye, self.every, self.window = cal, eye, every, window
self.rows = deque()
self.value, self.at = None, 0.0
def add(self, t, pupil, mid):
self.rows.append((t, pupil[0], pupil[1], mid[0], mid[1]))
while self.rows and self.rows[0][0] < t - self.window:
self.rows.popleft()
def get(self, now=None):
now = time.monotonic() if now is None else now
if now - self.at >= self.every:
self.at = now
s = self.cal.slip(self.eye, list(self.rows))
if s is not None:
self.value = s
return self.value
gaze/tracker/eyes_pupil.py
+139
View File
@@ -0,0 +1,139 @@
"""Classic pupil and glint finder for one 512x400 eye-camera frame.
The pupil is a dark blob enclosed by brighter iris and skin. The lens rim and the unlit
background are just as dark, but they touch the image edge, so any dark region that reaches
the edge is dropped. Closing the glints' holes can join the pupil to that background (the
left camera's, when you look more than about 20 degrees left), so when nothing is found
the search runs again with a smaller closing.
"""
import cv2
import numpy as np
DARK = 30 # pupil pixels are below this (the face around it is 40-180)
MIN_AREA = 150 # pupil area range in pixels
MAX_AREA = 20000
MIN_FILL = 0.75 # blob area / fitted-ellipse area
MAX_ASPECT = 3.0 # long / short axis; the steep camera sees a squashed pupil
GLINT = 200 # glints are near-saturated spots on or by the pupil
CLOSE = 7 # px: closes the glints' holes in the pupil (the right eye's need this much)
CLOSE_TIGHT = 3 # px: the retry, keeps a pupil near the dark background apart from it
NEAR = 70 # the windowed search: this many pixels, or 3 pupil radii, around a hint
def find_pupil(frame, near=None):
"""Return dict(x, y, a, b, angle, area, fill, glints) or None.
`near` (a previous result) searches a window around it first (0.4 ms, not 1.4-2.1);
if the pupil isn't wholly inside the window it falls back to the whole frame."""
if near is not None:
r = int(max(NEAR, 3 * near['a']))
x0, y0 = max(int(near['x']) - r, 0), max(int(near['y']) - r, 0)
p = _find_either(frame[y0:int(near['y']) + r, x0:int(near['x']) + r], x0, y0)
if p is not None:
p['glints'] = find_glints(frame, p)
return p
p = _find_either(frame, 0, 0)
if p is not None:
p['glints'] = find_glints(frame, p)
return p
def _find_either(img, ox, oy):
p = _find(img, ox, oy)
return p if p is not None else _find(img, ox, oy, CLOSE_TIGHT)
def _find(img, ox, oy, close=CLOSE):
"""The pupil in `img` (a window at ox, oy of the frame), with frame coordinates. A dark
region touching the window's edge doesn't count: it's background, rim, or cut off."""
f = cv2.GaussianBlur(img, (5, 5), 0)
dark = (f < DARK).astype(np.uint8)
# Glints punch bright holes in the pupil; close them so the blob stays whole.
dark = cv2.morphologyEx(dark, cv2.MORPH_CLOSE, np.ones((close, close), np.uint8))
dark = cv2.morphologyEx(dark, cv2.MORPH_OPEN, np.ones((3, 3), np.uint8))
n, lab, stats, _ = cv2.connectedComponentsWithStats(dark, connectivity=8)
h, w = img.shape
best = None
for i in range(1, n):
x, y, bw, bh, area = stats[i]
if area < MIN_AREA or area > MAX_AREA:
continue
if x <= 1 or y <= 1 or x + bw >= w - 1 or y + bh >= h - 1:
continue
blob = lab[y:y + bh, x:x + bw] == i
cs, _ = cv2.findContours(blob.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
c = max(cs, key=len)
if len(c) < 5:
continue
(ex, ey), (d1, d2), ang = cv2.fitEllipse(c)
a, b = max(d1, d2) / 2, min(d1, d2) / 2
if b < 3 or a / b > MAX_ASPECT:
continue
fill = area / (np.pi * a * b)
if fill < MIN_FILL or fill > 1.25:
continue
# Prefer the darkest, fullest blob.
score = fill - img[y:y + bh, x:x + bw][blob].mean() / 255
if best is None or score > best[0]:
# fitEllipse's angle is the direction of its first axis (d1); the long axis is
# that one or the one at right angles.
major = np.radians(ang if d1 >= d2 else ang + 90)
best = (score, dict(x=ex + x + ox, y=ey + y + oy, a=a, b=b, angle=ang, major=major,
area=int(area), fill=fill, box=(x + ox, y + oy, bw, bh)))
return best[1] if best else None
GLINT_RING = 140 # a glint sits on dark iris or pupil: its surroundings are below this
GLINT_PAIR = (5, 45) # the two LEDs' reflections: this far apart, in pixels, mostly vertical
def find_glints(frame, p):
"""Small bright spots on dark iris or pupil within 2.5 pupil radii, as (x, y) list.
Bright skin has noise speckle above GLINT too, so a spot counts only if the ring
around it is dark."""
r = int(p['a'] * 2.5) + 6
x0, y0 = max(int(p['x']) - r, 0), max(int(p['y']) - r, 0)
roi = frame[y0:int(p['y']) + r, x0:int(p['x']) + r]
n, lab, stats, cents = cv2.connectedComponentsWithStats((roi >= GLINT).astype(np.uint8))
out = []
for i in range(1, n):
x, y, w, h, area = stats[i]
if not 2 <= area <= 80:
continue
ya, yb, xa, xb = max(y - 4, 0), y + h + 4, max(x - 4, 0), x + w + 4
ring = roi[ya:yb, xa:xb][lab[ya:yb, xa:xb] != i]
if ring.size and np.median(ring) < GLINT_RING:
out.append((cents[i][0] + x0, cents[i][1] + y0))
return out
def glint_pair(p):
"""The two LED reflections as ((x, y) upper, (x, y) lower), or None. Picks the
vertical-ish pair nearest the pupil centre."""
g = p['glints']
best = None
for i in range(len(g)):
for j in range(i + 1, len(g)):
dx, dy = g[j][0] - g[i][0], g[j][1] - g[i][1]
d = np.hypot(dx, dy)
if not GLINT_PAIR[0] <= d <= GLINT_PAIR[1] or abs(dy) < 2 * abs(dx):
continue
mx, my = (g[i][0] + g[j][0]) / 2, (g[i][1] + g[j][1]) / 2
cost = np.hypot(mx - p['x'], my - p['y'])
if best is None or cost < best[0]:
best = (cost, (g[i], g[j]) if dy > 0 else (g[j], g[i]))
return best[1] if best else None
def draw(frame, p, scale=1.0):
img = cv2.cvtColor(cv2.convertScaleAbs(frame, alpha=2.0), cv2.COLOR_GRAY2BGR)
if p:
cv2.ellipse(img, ((p['x'], p['y']), (2 * p['a'], 2 * p['b']), np.degrees(p['major'])),
(0, 255, 0), 1)
for gx, gy in p['glints']:
cv2.circle(img, (int(gx), int(gy)), 3, (0, 0, 255), 1)
if scale != 1.0:
img = cv2.resize(img, None, fx=scale, fy=scale)
return img
gaze/tracker/findings.md
+465
View File
@@ -0,0 +1,465 @@
# Findings so far
Our own eye tracker's research notes, newest sections last. They were written while it was a
separate project (frame-eyes), so they use its names: `fe-trackd` is now `ft-eyes`,
`fe-bufprobe` is `ft-eyegrab`, `fe_model`/`fe_pupil` are `eyes_model`/`eyes_pupil`, the
tools are in `lab/` (`fe-score` is `ft-eyes-score`, `fe-replaytest` is `ft-eyes-e2e`,
`fe-record` and `fe-replay` are `ft-eyes-record` and `ft-eyes-replay`), `fe-live` is the
gaze service running ft-eyes, and `captures/NAME` is `~/.local/share/frametop/eyes/captures/NAME`.
These were measured on the Frame on 2026-09-28 (SteamVR eyetracking 2.17.10), and all by
reading only.
## SteamVR's tracker process
- `eyetracking -b CDSP -w .../et_dsp_20250610_03136.weights` runs as the user (steamos),
started by SteamVR. The user is in the `cdsp` and `spidev` groups. `ptrace_scope` is 1,
so reading another process's fds or memory needs root (pidfd_getfd).
- Log: `~/.local/share/Steam/logs/eyetracking.txt`. Component names: `CStereoAdspCams`
(the eye cameras come in through the audio DSP; "Set framerate 72/90"),
`CGazeEstimatorCdsp` / `CDSPGazenet` (the neural net on the compute DSP), and
`CEyePoseUKF L/R` (a filter per eye; "Large dt" means it had no measurement for over
0.4 s and starts that eye over).
- "Failed to grab cdsp input buffer" came up 5,924 times in 6 hours (about 0.3 % of
frames at 90 Hz).
- "Accept usercal" is its passive calibration from quick mouse clicks.
- The eye cameras aren't V4L2 devices, and there's no fastrpc node.
- Open fds that matter:
- `/dev/spidev0.1`: modalias `spi:hid-over-spi`, role unknown.
- Six udmabufs, all `exp_name: udmabuf`: three of 16 MiB (16777216 B) and three of
32 MiB (33554432 B), fds 50, 51, 53, 54, 159 and 169 at the time.
- `/dev/shm/eye-server.mmap`: its output.
- `/dev/input/event0-7`.
- The frames most likely arrive in the udmabufs, which it shares with the DSPs.
## The eye-camera frames (found 2026-09-28, `tools/fe-bufprobe --scan`)
- **The buffers.** The six udmabuf fds are really two buffers, three fds each: a 16 MiB one
(inode 1) and a 32 MiB one (inode 2).
- **The frames.** In the 16 MiB buffer, eight slots sit 0x40000 apart from 0x230000, four
per camera: slots 0-3 are camera 0 and slots 4-7 camera 1. Each slot starts with a small
block (slot 0's holds a table of floats such as 0.00125, 4.655, 90.0, 1.0, possibly
exposure, gain, and frame rate; the others were zero), then a 512x400 8-bit grayscale
frame, row stride 512. The frame starts at slot base + 0x40c0 + 0x40 per slot index,
plus one more 0x40 for camera 1's slots. Found by the dark lens-rim column lining up;
`slot_start()` in fe-bufprobe.
- **What they show.** Infrared images, one camera per eye, dim (mean about 25-40), with a
dark band on one side (the lens rim). One camera saw its eye at a steep angle (squashed
pupil near the image edge, big reflections on the white); the other nearly head-on (a
round pupil with two small glints in it). Which camera is which eye isn't known yet.
- **Timing.**
- About 90 frames a second per camera, the two within about 0.6 ms of each other.
- A frame lands over several milliseconds, in bursts, so a copy taken when the slot
"stops changing" can be half old. The reliable rule: a slot is complete when its camera
starts writing another slot.
- Camera 0 fills its slots in turn (3, 0, 1, 2); camera 1 in a repeating order of eight
(7, 5, 4, 6, 5, 7, 6, 4), never the same slot twice in a row.
- `--rec` stamps each frame when its slot first changed, polling every 0.3 ms, so times
are only good to a few ms.
- The cameras run at 90 fps ("Set framerate 90" in the log). The first recorder copied
a frame as soon as its camera started the next one and got 94 a second in fit1 and 106
in a streaming test. The extras were half-written frames: a frame's last writes can land
after the next frame starts, and a slow poll saw both at once. The recorder now copies a
frame when its camera starts the frame after next (no slot is rewritten sooner than three
frames), ignores late writes to the frame just finished, and saves from a separate
thread. fit1 may hold a few percent of torn frames.
- Eye tracking stops when the headset is off ("HMD off, stopping eye tracking"), so
recordings are empty then.
- **Which camera is which eye** (capture fit1, 2026-09-29, closing one eye at a time):
camera 0 (slots 0-3) is the **right** eye, seen at a steep angle; camera 1 (slots 4-7) is
the **left** eye, seen nearly head-on. Both images have the lens rim dark on the left and
the lit face on the right.
- **Why the left eye is lost looking down:** at the keyboard, camera 1 sees only the upper
lid and lashes. Camera 0 still catches part of the right eye. It's the camera angle, not
the net.
- **The 32 MiB buffer.** Two 48 KiB regions (0x1522000, 0x1532000) that change every frame,
mean bytes about 148, 99 % nonzero. Probably the net's input per eye (crops, maybe not 8-bit
pixels). Not decoded.
- **`tools/fe-session NAME SECONDS`.** Records frames and ft-gaze's samples together, on the
same clock (CLOCK_MONOTONIC_RAW). Each frame's nearest SteamVR sample is a median 3.8 ms
away.
## Our first pupil finder (`tools/fe_pupil.py`, 2026-09-29)
- Threshold dark (< 30), close glint holes, drop dark regions that touch the image edge
(lens rim, background), keep the fullest, darkest ellipse-shaped blob. Glints: spots
>= 200 within 1.5 pupil radii. 3.1 ms a frame on the CPU, unoptimised.
- On fit1 (20 s: open, each eye closed, keyboard, up), pupil found vs SteamVR seeing the
eye:
| Gaze pitch | Right, SteamVR | Right, ours | Left, SteamVR | Left, ours |
| --- | --- | --- | --- | --- |
| below -20 (keyboard) | 79 % | 35 % | 22 % | 24 % |
| -20 to 15 (screens, includes closed-eye time) | 91 % | 89 % | 74 % | 72 % |
| above 15 | 100 % | 100 % | 98 % | 99 % |
It misses the right eye looking down, where the lower lid cuts the pupil. False finds
on closed eyes: 0.4 % right, 2.8 % left.
- A quadratic fit from pupil centre to SteamVR's per-eye gaze, held out by time block:
median 4.8 degrees right, 3.0 left. That isn't an accuracy figure yet. fit1 has few
distinct gaze points, it uses no glints, and SteamVR's per-eye gaze is itself off by
several degrees. It needs a recording against known targets.
## eye-server.mmap (its output)
The file is 324,122 bytes. Only bytes 0x0-0x1f3 are used; the rest is zero. It's packed and
unaligned, so read it with memcpy. Offsets are also in `~/frametop/gaze/ft-gaze.cpp`.
| Offset | What |
| --- | --- |
| 0x38 | u32 sample counter |
| 0x157 | f64 sample time, CLOCK_MONOTONIC_RAW |
| 0x15f, 0x16b | set 1: left and right eye direction (3 f32, head space, -Z forward). Filtered; both eyes always share one pitch; a lost eye keeps its yaw |
| 0x177 | set 1: 6 f32 variances (left 3, right 3; the middle one of each is shared). About 0.0005-0.002 when the eye is seen, 0.015-0.03 when it's lost |
| 0x18f | set 1 fixation point (3 f32; its length is the vergence distance) |
| 0x19b, 0x1a7 | set 2: each eye's own direction |
| 0x1b3 | set 2: 6 f32 variances |
| 0x1cb | 2 f32, 0..1: openness (0 in a blink) |
| 0x1d3 | 8 f32: left measurement x, y; right x, y (camera-relative, freezes while that eye isn't seen); then variance of left x, y, right x, y (about 2e-5 on a clear view, rising as lids or lashes get in the way) |
| 0x0-0x157 | header, plus records that look like the calibration-click channel into the tracker. Never write |
## Accuracy of SteamVR's gaze (this user, this headset)
- **Tonight's practice (71 clicks, 45 minutes):**
- raw error: median 5.1 degrees (3.0 in the 21:23 test; it varies by session);
- corrected at the press: median 1.5, with 1 in 10 past 3.3;
- best smooth correction fitted to the same clicks, each predicted from the rest: 1.7-1.9;
- weighting recent clicks more (half-lives from 20 minutes down to 1) didn't help, so
there's no slow drift to follow.
- **Look-to-look:** two looks within 3 degrees of each other, under 5 minutes apart,
differ by a median 1.15 degrees (3.0 when 5 or more minutes apart). Jitter within one
look is 0.25-0.3.
- **One eye alone (set 2), against both eyes' gaze:** median 0.8 degrees over a steady
look. Per-eye raw errors are large and opposite in yaw: at one spot, left (+8.2, +8.9)
and right (-5.0, +5.2) degrees, both (+1.6, +7.1).
- **Losses:** the left eye was lost 57-64 % of the time looking 30-50 degrees down (at
the keyboard, through the gap by the nose), and the right eye never. At screen height
both were seen over 98 % of the time. Openness looking down: left 0.45, right 0.65.
Harmless for the pointer: ft-gazed ignores looks down past the screens.
## First accuracy test against known targets (practice1, 2026-09-29)
5 minutes, 98 gaze-probe practice clicks, gaze yaw -26..25 and pitch -14..20 degrees. Truth
is SteamVR's raw gaze at the press plus the angle to the release point. `tools/fe-score.py`
fits a quadratic per method and scores each click leave-one-out. Pupil = median centre over
the frames 250-20 ms before the press; no glints yet.
| Method (85 clicks with both pupils found) | Median | 90 % |
| --- | --- | --- |
| SteamVR raw | 6.51 | 11.04 |
| SteamVR + quadratic fit | 1.52 | 3.10 |
| Ours, right pupil only | 0.74 | 1.48 |
| Ours, left pupil only | 0.62 | 1.48 |
| Ours, both pupils averaged | 0.61 | 1.13 |
SteamVR with the probe's live correction: 1.44 median over all 98. The pupil was found
before 88/98 clicks (right) and 90/98 (left). Caveats: one session with the headset
never moved (pupil-only mapping breaks when the headset slips; glints should fix that),
the truth includes the user's own drag precision, and it's offline only.
## Glints and slip (2026-09-29, practice1)
- Two IR LED reflections, a vertical pair 12-19 px apart, sit on the cornea near the pupil.
There's no alternating illumination: the pair is in every frame the geometry allows.
Before a click: right eye 45/88, left 54/90 (the steep right camera loses the pair on
the white when the eye looks across). Bright skin has noise speckle above 200, so a glint
only counts if the ring around it is dark (`find_glints`, `glint_pair` in fe_pupil.py).
- Pupil minus pair midpoint as the feature: 0.84 median, noisier than the pupil alone
(0.61), because the pair's position is noisy.
- Slip method (`tools/fe-score.py`): where the pair is seen, the glint fit gives the gaze,
a fit of gaze to pupil position says where the pupil should be, and the difference is
the slip. The median over the last 30 s shifts every frame, glints or not. In-session:
0.61 median, same as the pupil alone. The estimate stayed within 1-3 px all session.
- Simulated slip (fit on the first 49 clicks, test on the rest shifted 10 px): pupil alone
0.62 -> 2.7-3.1; glint 0.83 unchanged; slip 0.67 unchanged. That checks the math only,
for a pure image shift. A real re-seat also tilts and changes the distance.
- Next test: a second session after taking the headset off and on, scored with
`fe-score.py captures/practice1 captures/practice2`.
## Live tracker (2026-09-29)
- `fe-bufprobe --share` (root) copies each finished frame into `/dev/shm/frame-eyes-cams`
(0600, the user's); `fe-trackd` (user) finds pupils and glints and writes
`/dev/shm/frame-eyes-gaze`; ft-gaze reads that as the source `own`. `tools/fe-live` runs
both. The user side never touches SteamVR's buffers.
- The windowed pupil search gives the same results as the full frame (0.000 px apart on
2000 frames per eye), at 0.4 ms instead of 1.4-2.1; with glints, about 1 ms a frame, 90 fps
per eye.
- Replaying practice1's first minute through fe-trackd: 0.64 median at the 14 clicks
(0.61 offline with the same calibration; in-sample, so a pipeline check, not accuracy).
Sample-to-sample jitter 0.08 degrees; SteamVR's is 0.25-0.3.
- The calibration covers yaw -26..25 and pitch -14..20 degrees (practice1's clicks). Beyond
that the quadratic extrapolates.
## Test 2: a second session after re-seating (practice2, 2026-09-29 15:18)
124 practice clicks over about 4 minutes, headset nudged at about 100 s. The probe stayed on
SteamVR's mmap2 as its source, but recorded our live gaze at every press. Scored with
`fe-score.py captures/practice1 captures/practice2` (median degrees):
| Method | Fit on practice1 | Fit within practice2 (leave-one-out) |
| --- | --- | --- |
| SteamVR raw | 2.86 | 2.86 |
| SteamVR + the probe's live correction | 1.37 | 1.37 |
| SteamVR + quadratic fit | 3.84 | 1.17 |
| Ours, pupil only | 14.31 | 2.86 |
| Ours, glint | 3.31 | 1.54 |
| Ours, slip | 2.68 (live: 2.87) | 1.36 |
- The re-seat moved the eyes 20-30 px in the images: pupil-only goes 14 degrees off. The
slip correction takes that to 2.7, but no further. The rest is partly one offset (yaw
-1.6, pitch +1.2; removing it leaves 1.43), and an offset from the previous 5 clicks
gives 1.31, the same as SteamVR's live correction (1.37).
- Within one headset position (before the nudge, 45 clicks; after it, 68): pupil only
1.01 and 0.99, slip 1.18 and 1.86, SteamVR with the same fit 0.91 and 1.19. So today our
tracker is level with SteamVR within a position, not ahead of it as in practice1 (0.61
against 1.69). One session was not enough to claim a lead.
- The slip estimate adds noise: it's worse than no correction within a position, and it
wandered (the right eye's jumped 18 px near the end, after the clicks). It comes from the
glint fit, which is itself only 1.5-3 degrees good, and the left eye's pair was seen
before only 21 of 124 clicks.
- Live: fe-trackd ran 81-90 fps per eye at 2-3.6 ms a frame alongside VR (1 ms in replay);
ft-gaze's `own` came through on every line, about 37 ms old.
- What would help: a geometric eye model (the eyeball centre from how the pupil ellipse
changes shape, as Swirski's method and Pupil Labs' pye3d do) instead of 2-D regression,
so that headset movement is modelled and not fitted around. practice1 and practice2
together (re-seat plus a nudge) are the benchmark for it.
## The geometric model, and a shift taught by clicks (2026-09-29, practice1 -> practice2)
All offline: calibrate on practice1, score practice2's clicks.
- **Eyeball centre from the pupil ellipses (Swirski-style, weak perspective): worse.** The
centre it finds is steady within a session (a few px per 50 s) and moves between the
sessions about as the slip does, with a rotation radius of about 60 px (10-12 mm). But
it's off from the true shift by up to 8 px (6-7 degrees) on the right eye. Correcting
with it gave 4.3-5.8 median, against 2.7 for the glint slip. The cornea's refraction and
where you happened to look in the window likely bias it.
- **The calibration itself carries over.** The best possible pixel shift per eye, fitted
on practice2's own clicks with one shift per headset position (before and after the
nudge), gives 1.18 held out (shift+scale 1.11, affine 1.07). So practice1's fit is fine
if we know the shift; the problem was only estimating it. One shift for the whole
session gets only 2.7-2.8, because the nudge moved the eyes again.
- **How well each estimate finds that shift (px, right x/y, before the nudge):** true
(-0.7,-28.6), glints (+2.0,-25.6), eyeball centre (-6.9,-22.2). The glints are off by
1-4 px, which is 1-3 degrees; the eyeball centre is worse.
- **Clicks estimate it best.** Each click says where the pupil should have been for a
known gaze, so pupil minus that is the shift. Scored in time order with only earlier
clicks:
| Shift from | Median | 90% |
| --- | --- | --- |
| Glints only, last 10 or 30 s | 2.68-2.69 | 3.78-4.13 |
| Last 3 clicks | 1.29 | 2.80 |
| **Last 5 clicks** | **1.18** | 3.09 |
| Last 5 clicks + glint slip since | 1.30-1.34 | 3.34-3.55 |
| **Last 5 clicks, glints only to catch a jump over 3 px** | **1.22** | **2.33** |
| SteamVR + the probe's live correction (same clicks) | 1.37 | 2.71 |
Adding the glint slip to the clicks' shift adds its noise. Using it only to notice a
nudge (then the shift follows the glints until clicks catch up) keeps the median and
cuts the tail after a nudge. That's `fe_model.Shift`, and `fe-score`'s `clicks` method
reproduces it (1.22 median, 2.33 90%).
- So on this pair of sessions, ours with click correction is slightly ahead of SteamVR
with the probe's click correction: 1.22 against 1.37 median, 2.33 against 2.71 for the
worst tenth. One pair of sessions, so not yet a lead (see Test 2).
- fe-trackd now works this way: the probe's calibration with the source "Own tracker"
sends each dot to fe-trackd (`calib-point`), which fits from its own pupil history
(`calib-fit`, which replaces the old calibration and clears the shifts), and each
practice release sends a `click` that teaches the shift. See fe-trackd's docstring.
- **The live path reproduces it** (`tools/fe-replaytest captures/practice1 captures/practice2`
on the 7i: a scratch fe-trackd, calibrated through `calib-point` from practice1's clicks,
then fed practice2's clicks at the recorded pace and scored on what it published in the
300 ms before each press). 84 of 98 dots accepted (14 had an eye in under 15 frames),
fit 0.59 median. practice2: median 1.21 and 1.23 over two runs (offline 1.22), but 90%
3.02 and 2.86 (offline 2.33). The tail: the first click (19.7, nothing learned yet after
the re-seat), and two clicks at 192 s and 213 s (7.6 and 10.3) with a settled 5-click
shift and no jump. (Offline has the same two, 6.8 and 9.3: see the next section.) The
glint jump restarted the right eye's shift 9 times and the left's 4.
## Wide gaze, the left pupil, and false glint jumps (2026-09-29, practice2)
- **The bad clicks were all past the calibration, or right eye only.** practice1's clicks
reach yaw 25 and pitch 20; practice2's reach 30 and 25. Offline (median / 90%):
| Clicks | Ours | SteamVR + probe |
| --- | --- | --- |
| Inside practice1's range (104) | 1.09 / 1.97 | 1.37 / 2.71 |
| Outside it (18-20) | 1.80 / 5.01 | 1.19 / 2.67 |
| Both eyes (99) | 1.09 / 1.98 | 1.40 / 2.83 |
| Right eye only (23) | 1.66 / 4.05 | 0.99 / 2.47 |
A fit that goes linear past its data, more ridge, or a linear fit didn't help outside.
- **The left eye was lost at every click past about 19 degrees left.** The pupil is still
mid-image (x 293-310 of 512), but the left camera's image is dark from x 0 to about 360,
and the 7 px closing (which heals glint holes) joined the pupil to that background, which
touches the edge, so it was dropped. `fe_pupil` now retries with a 3 px closing when
nothing is found: left eye found in 97% of the frames before practice2's clicks (was
79%; 181 of 217 frames at 20+ degrees, was 16), right eye unchanged, centres moved at
most 0.16 px. The right eye still needs the 7 px (3 px alone: 96% against 98%).
- **Those pupils are accurate.** Within one headset position (practice2 after the nudge,
68 clicks, leave-one-out, so calibrated out there too): left eye alone 0.70 / 1.39 below
19 degrees left and 0.87 / 1.14 beyond; right eye 1.27 / 2.13 and 2.09 / 6.42; both
averaged 0.85 / 1.30 and 1.18 / 5.14. Calibrated where you look, the left eye is our best.
- **But practice1's calibration has 4 clicks per eye beyond 19 degrees**, so the newly seen
left eye extrapolates there (3.64 median alone, right 1.78), and practice1 -> practice2
got a worse tail: 1.20 / 3.20 offline (1.22 / 2.33 when the left eye was simply lost
there). Weighting the eyes, or leaving out an eye or a click past the calibrated range,
didn't recover it. The fix is coverage: the probe's calibration for the Own tracker now
puts its dots on an oval out to the `Calibration ring` angle each way (the ring was
limited by the window's height and never reached the sides). A first try put them at
the practice area's corners, which in a large window were too far to look at while
facing the centre.
- **The glint jumps.** Offline (checked once per click) there were 2 per eye, 3 of 4 real
(the next click found the shift the glints claimed). Live, bad glint pairs made the right
eye's estimate leap by up to 68 px for under a second, 20 times between clicks, and the
shift restarted 8-9 times. `Shift` now takes a jump only once it has held for 1 s
(JUMP_HOLD) and ignores ones over 40 px (JUMP_MAX). Live replay: shift restarts 1 (right)
and 0 (left), biggest leap between clicks 4.9 px, output steps over 10 degrees 86 -> 32.
Offline with the hold: 1.19 / 3.35.
- Live replay with both changes: 94 of 98 calibration dots taken (83-84 before), practice2
1.28 / 3.16. The tail stays until a calibration covers the practice area.
- `fe-score` now reads each capture's own `practice.jsonl` (cut from the probe's log by
`fe-score.py --clicks`, or the first scoring on the Frame), so the 7i can rebuild features.
## Session 3 (2026-09-29 22:04-22:10, live, SteamVR driving)
The calibration and 84 practice clicks went to SteamVR (the probe's tracker toggle was
left on SteamVR), so fe-trackd got no dots or clicks and kept practice1's calibration. The
probe still logged our gaze at 79 presses. SteamVR + the probe's correction: 1.19 median,
2.61 90% (raw 2.18 / 4.30). Ours with practice1's calibration and no clicks: 12.5 median;
with a stand-in for the click shift (the median offset of the previous 5 clicks, in gaze
angles rather than per eye in pixels): 1.38 / 2.38. No frames were recorded.
## Session 4: the Own tracker driving (2026-09-29 22:12-22:22, live)
Fresh calibration from the probe with the Own tracker: 27 dots on an oval out to 20
degrees (30 of 31 attempts accepted; one had no right eye). Then 136 practice clicks, all
with both eyes, each teaching the shift. The probe logged SteamVR at 114 of the presses,
and its correction learned from the same drags, so the comparison is fair:
| At the same 114 presses, to where you let go | Median | 90% |
| --- | --- | --- |
| **Ours** | **0.59** | **1.50** |
| SteamVR + the probe's correction | 0.83 | 2.02 |
| SteamVR raw | 3.49 | 5.21 |
Ours was closer on 76 of 114. Ours at the press (what the dot showed, all 136): 0.67
median, 1.37 90%, and steady from the first 10 clicks (0.71) on; SteamVR's correction
took about 30 clicks to get under 1 degree. No click changed an eye's shift by more than
6 px. The user: "MUCH improved". No frames were recorded, and the headset wasn't nudged or
re-seated, so this is within one position; the cross-session question is still open.
## Session 5: off and on again, no recalibration (2026-09-29 22:26-22:31, live)
Session 4's calibration and shifts, headset taken off and put back on, then 132 practice
clicks with the Own tracker driving. The re-seat moved the eyes about 15 px (right) and
33 px (left) in the images.
- Before the first taught click, the glints had moved the right eye's shift to within
about 4 px and the left's about two thirds of the way. Clicks 1-4 were still 11-17
degrees off, and the probe refused to teach them (its 6-degree limit on a lesson), so
the first taught click was the 5th (5.4 degrees). It restarted each eye's history as
designed; clicks 6, 7, 8: 2.8, 1.3, 0.4.
- After that (clicks 6 on, 127, to where you let go): ours 0.58 median, 1.42 90%, the
same as within one position (session 4: 0.59); SteamVR + the probe's correction 2.94 /
5.73 (SteamVR raw drifted from 3.5 to 4.7 through the session). Ours closer on 117 of 132.
- Changes: the probe lets the Own tracker learn from drags up to 25 degrees
(OWN_LEARN_MAX), and starts on the Own tracker when fe-trackd answers; its calibration
header names the tracker. fe-trackd restarts an eye's shift history at the next click
after frames stop for 3 s (the headset off) and after its own restart, glints or not.
Replay regression (fe-replaytest practice1 practice2): 1.28 / 3.16, as before.
## Weighting the eyes (2026-09-29, practice2 after the nudge)
One headset position, 64 clicks with both eyes, leave-one-out: plain average 0.84 / 1.59;
left eye alone 0.73 / 1.36; right alone 1.32 / 3.05; weighted by each eye's inverse
residual variance on its own calibration 0.75 / 1.26. Not in fe-trackd yet.
## Next steps (2026-09-29, after a literature search; sources in the session report)
Ranked by expected gain for the effort, checked against our own numbers:
1. Weight the eyes by each one's calibration residuals (above: 0.84 -> 0.75 median). S.
2. A one-dot re-seat check when frames come back after a gap (Varjo recalibrates with one
dot at every put-on): one look and press teaches both shifts before the first real
click, instead of 11-17 degree first clicks. S.
3. Our tracker as a source for the Frametop pointer (ft-gazed): session 5 beat SteamVR
across a re-seat. M. Done 2026-09-30 (below).
4. Record frames during live tests (fe-session), so each can be replayed. S (disk: about
2 GB a minute).
5. A less biased glint slip estimate: ours is off by 1-4 px even over hundreds of frames,
so it's bias, not noise; try taking out the part of the glint midpoint that follows the
pupil (regressed on calibration data) before using it. S-M, offline first.
6. Smooth-pursuit calibration (a moving dot): dense labels out to the edge in about 20 s,
for wider coverage. M.
7. Sub-pixel edge ellipse refit with RANSAC for the steep right eye (our weaker eye,
1.32 against 0.73). S-M.
8. Later, if needed: learned pupil segmentation (EllSeg, RITnet: MIT) on the GPU through
ncnn, a 3-D cornea model from the two glints, or a per-user network trained on the
residuals across re-seats. L. Not recommended: the eyeball-centre model (tried, and our
steep camera and +-20 degree range are outside its published conditions). PuRe,
PuReST, ElSe, and ExCuSe are licensed for non-commercial use only.
## Quick wins from the next steps (2026-09-29, late)
- Eye weighting (1): `Calibration.spread` is each eye's RMS miss on its own calibration
dots, and `combine` weights by its inverse square (floor 0.3 degrees). fe-trackd, fe-score,
and so fe-replaytest use it; older calibrations get it from their saved dots. The
22:16 calibration: right 1.48, left 1.08, so the left eye counts about twice as much.
practice1 -> practice2 offline is unchanged (1.20 / 3.35: that tail is extrapolation).
- Re-seat check (2): fe-trackd's status says when the next click will start an eye's shift
over; the probe then shows one centre dot, and a press on it sends that click. Checked
on the 7i (a pending re-seat at start on both eyes, cleared by one click); the probe's
screen for it wasn't seen (the web view didn't connect).
- Recording (4): `tools/fe-record` copies every shared frame (9 s of replay: 1620 frames,
none dropped, all identical to the source); `fe-live --record NAME` runs it alongside.
## The Frametop pointer, and the eyes on live clicks (2026-09-30)
ft-gazed (`~/frametop/gaze`) can now use our tracker: `GAZE_TRACKER=own`, the Eye tracker
setting on the Gaze page of Frametop Input Settings. A mouse nudge before a click reaches
fe-trackd as a click. The nudge's raw gaze is one ft-gazed sent, so ft-gazed finds when
that look was, and fe-trackd keeps 12 s of pupils instead of 5, because the helper sends a
nudge up to 10 s after the look.
`GAZE_EYE` (auto, left, right) weights the eyes there, from each eye's own gaze. Replayed on
the 306 live clicks of sessions 4 and 5 (`clicks.jsonl`: each eye's pupil and shift just
before the click, so each is a fresh test):
- Each eye alone: left 0.96 median (mean 1.17), right 1.11 (1.26). The eyes' RMS misses
were about equal (1.43, 1.46), unlike practice2's leave-one-out (0.73, 1.32).
- Both eyes: 0.65 (0.77) evenly. By the calibration's spread (the 22:16 one: left counts
about twice): 0.63 (0.81). By each eye's RMS miss at its last 5, 10, or 20 clicks: 0.66
(0.79-0.81).
- By share of the right eye: 0.3 gives 0.68, 0.5 gives 0.65, 0.7 gives 0.81.
- The eyes' yaw errors are correlated -0.37: they partly cancel, which is why two eyes
beat either one by a third.
So a bias leans instead of choosing: Left or Right counts that eye twice. Auto starts
even and weights by each eye's RMS miss at its last 20 nudges, once each has 5. On
SteamVR's side the calibration's own fit picked the wrong eye (its dots: left 1.78, right
1.88; new spots: left 2.50, right 1.63), so auto learns from nudges, not the fit.
fe-trackd's own `combine` still uses the spread, for the probe.
## Valve's tracker
It can't be the starting point, legally or practically:
- **No source.** `/opt/steamvr/tools/eyetracking/bin/linuxarm64/eyetracking` is a
stripped aarch64 binary. The paths left in it (`/data/src/eyetracking/eyetracklib/...`)
are Valve's build machine's.
- **The net is just numbers.** `et_dsp_20250610_03136.weights` is 393,600 bytes of raw
floats (about 98,000 parameters), with no header or architecture. The layer layout
lives in the binary and in the program it loads onto the compute DSP (`CDSPGazenet`).
Rebuilding it would mean reverse engineering both.
- **License.** SteamVR is Valve's proprietary software, used under the Steam Subscriber
Agreement. That agreement doesn't allow reverse engineering, decompiling, modifying, or
redistributing it, except where the law allows. `third_party_legal_notices.txt`
covers only the open libraries it uses (Ceres, protobuf, ...), not the tracker. Putting
their code or weights in a GitHub repo would be redistribution. (Not legal advice.)
- **Not much to gain.** Their net is small and tuned to their cameras. Improving it would
need the same thing our own tracker needs: your eye images with known gaze, for
training.
What we can use: its public output (the mmap, read-only), as a baseline and as labels.
Anything published and openly licensed is also fair game: papers and open-source pupil
detectors (check each one's license before using its code).
gaze/tracker/frametop-eyegrab.service
+36
View File
@@ -0,0 +1,36 @@
# Template: gaze/tracker/install.sh fills in @UID@ and @GID@ (the Frametop user) and installs
# it to /etc/systemd/system.
[Unit]
Description=Frametop eye-camera frames for our own eye tracker (read-only copies from SteamVR's eyetracking)
Documentation=file://@REPO@/gaze/README.md
[Service]
# Idle (no frames copied, none of the tracker's buffers held) until ft-eyes or
# ft-eyes-record touches the want file; see ft-eyegrab.c.
ExecStart=/etc/frametop/ft-eyegrab --share /dev/shm/frametop-eyes-cams --owner @UID@:@GID@ --want /dev/shm/frametop-eyes-want
Restart=on-failure
RestartSec=5
Nice=5
# Root only for what reading another process's buffers needs: CAP_SYS_PTRACE (pidfd_getfd),
# CAP_DAC_READ_SEARCH (its /proc/PID/fd), and CAP_CHOWN (the shared file goes to the user).
CapabilityBoundingSet=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_CHOWN
AmbientCapabilities=
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/dev/shm
PrivateNetwork=yes
RestrictAddressFamilies=AF_UNIX
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
RestrictNamespaces=yes
RestrictRealtime=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
[Install]
WantedBy=multi-user.target
gaze/tracker/ft-eyegrab.c
+605
View File
@@ -0,0 +1,605 @@
/*
* ft-eyegrab: the eye-camera frames for our own eye tracker (ft-eyes), copied read-only out
* of the DMA-BUFs SteamVR's eyetracking process holds. Runs as root (pidfd_getfd needs
* CAP_SYS_PTRACE; ptrace_scope is 1): the system service frametop-eyegrab.service runs
* --share, installed by gaze/tracker/install.sh. The other modes are for finding the frames
* again after a SteamVR update (run them with sudo).
*
* ft-eyegrab --share PATH [--owner UID:GID] [--want FILE]
* keep the latest frames of both cameras in PATH (shared memory,
* 0600, owned by UID:GID, or the sudo user) for ft-eyes; follows
* the tracker through SteamVR restarts. With --want, only while
* FILE (a regular file owned by that user) was touched in the
* last WANT_FRESH seconds: ft-eyes and ft-eyes-record touch it
* every second, so nothing is copied, and none of the tracker's
* buffers are held, while nobody reads the frames
* ft-eyegrab list the buffers
* ft-eyegrab --scan [N] N snapshots (default 40) about 11 ms apart: which 4 KiB pages
* change, merged into regions, with byte statistics for each
* ft-eyegrab --dump I OFF LEN FILE
* copy LEN bytes at OFF of buffer I (from the list) to FILE
* ft-eyegrab --seq I OFF LEN FRAMES DIR
* FRAMES copies of that region, one each time it changes, to
* DIR/NNNN.raw, with DIR/times.txt (CLOCK_MONOTONIC_RAW)
* ft-eyegrab --rec SECONDS DIR
* every new eye-camera frame for SECONDS: DIR/frames.raw (512x400
* 8-bit frames back to back) and DIR/index.txt, one line per frame:
* "<n> <slot> <camera 0|1> <CLOCK_MONOTONIC_RAW time seen>"
* (lab/ft-eyes-record does the same from the shared frames,
* without root)
*
* The eye frames (found with --scan): in the 16 MiB buffer, eight slots 0x40000 apart from
* 0x230000, four per camera (slots 0-3, 4-7). Each slot starts with a small block, then a
* 512x400 8-bit image at 0x40c0 + 0x40 per slot, and one more 0x40 for the second camera's.
*
* Only reads the tracker's buffers. They are borrowed with pidfd_getfd and mapped PROT_READ; nothing is
* written, and the process isn't stopped or signalled. Reads can tear while the DSP writes.
*/
#define _GNU_SOURCE
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <signal.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <time.h>
#include <unistd.h>
#define MAXBUF 32
#define PAGE 4096
typedef struct {
int xfd, fd;
size_t size;
unsigned long ino;
const uint8_t *p;
} buf_t;
static buf_t bufs[MAXBUF];
static int nbufs;
static double now(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
return ts.tv_sec + ts.tv_nsec * 1e-9;
}
static int find_tracker(void) {
DIR *d = opendir("/proc");
struct dirent *e;
int pid = -1;
while (d && (e = readdir(d))) {
char path[300], cmd[512];
if (e->d_name[0] < '0' || e->d_name[0] > '9') continue;
snprintf(path, sizeof path, "/proc/%s/cmdline", e->d_name);
FILE *f = fopen(path, "r");
if (!f) continue;
size_t n = fread(cmd, 1, sizeof cmd - 1, f);
fclose(f);
cmd[n] = 0;
if (strstr(cmd, "tools/eyetracking/bin/") && strstr(cmd, "/eyetracking")) {
pid = atoi(e->d_name);
break;
}
}
if (d) closedir(d);
return pid;
}
static int open_bufs(int pid) {
int pidfd = syscall(SYS_pidfd_open, pid, 0);
if (pidfd < 0) {
perror("pidfd_open");
return -1;
}
char dir[64];
snprintf(dir, sizeof dir, "/proc/%d/fd", pid);
DIR *d = opendir(dir);
struct dirent *e;
while (d && (e = readdir(d)) && nbufs < MAXBUF) {
char link[320], target[256];
if (e->d_name[0] == '.') continue;
snprintf(link, sizeof link, "%s/%s", dir, e->d_name);
ssize_t n = readlink(link, target, sizeof target - 1);
if (n <= 0) continue;
target[n] = 0;
if (strncmp(target, "/dmabuf:", 8) != 0) continue;
int xfd = atoi(e->d_name);
int fd = syscall(SYS_pidfd_getfd, pidfd, xfd, 0);
if (fd < 0) {
fprintf(stderr, "pidfd_getfd %d: %s\n", xfd, strerror(errno));
continue;
}
struct stat st;
fstat(fd, &st);
off_t size = lseek(fd, 0, SEEK_END);
void *p = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
if (p == MAP_FAILED) {
fprintf(stderr, "mmap fd %d (%lld bytes): %s\n", xfd, (long long)size, strerror(errno));
close(fd);
continue;
}
bufs[nbufs++] = (buf_t){xfd, fd, (size_t)size, (unsigned long)st.st_ino, p};
}
if (d) closedir(d);
close(pidfd);
return nbufs;
}
static uint64_t page_hash(const uint8_t *p) {
const uint64_t *q = (const uint64_t *)p;
uint64_t h = 1469598103934665603ull;
for (int i = 0; i < PAGE / 8; i += 4) h = (h ^ q[i]) * 1099511628211ull; // every 4th word
return h;
}
static void stats(const uint8_t *p, size_t n, double *mean, int *lo, int *hi, double *nonzero) {
uint64_t sum = 0, nz = 0;
int a = 255, b = 0;
for (size_t i = 0; i < n; i++) {
sum += p[i];
nz += p[i] != 0;
if (p[i] < a) a = p[i];
if (p[i] > b) b = p[i];
}
*mean = n ? (double)sum / n : 0;
*lo = a, *hi = b, *nonzero = n ? (double)nz / n : 0;
}
static void scan(int snaps) {
for (int b = 0; b < nbufs; b++) {
size_t pages = bufs[b].size / PAGE;
uint64_t *prev = calloc(pages, 8), *cur = calloc(pages, 8);
int *changes = calloc(pages, sizeof(int));
for (size_t i = 0; i < pages; i++) prev[i] = page_hash(bufs[b].p + i * PAGE);
double t0 = now();
for (int s = 1; s < snaps; s++) {
usleep(11000);
for (size_t i = 0; i < pages; i++) {
cur[i] = page_hash(bufs[b].p + i * PAGE);
if (cur[i] != prev[i]) changes[i]++;
prev[i] = cur[i];
}
}
double dt = now() - t0;
printf("buffer %d (fd %d, %zu bytes, ino %lu): %d snapshots over %.2f s\n", b, bufs[b].xfd, bufs[b].size,
bufs[b].ino, snaps, dt);
// Regions: runs of pages that changed at least once (gaps of up to 2 pages merged).
size_t i = 0;
int regions = 0;
while (i < pages) {
if (!changes[i]) {
i++;
continue;
}
size_t start = i, end = i, gap = 0;
int most = 0;
long total = 0;
for (; i < pages; i++) {
if (changes[i]) {
end = i, gap = 0;
total += changes[i];
if (changes[i] > most) most = changes[i];
} else if (++gap > 2) {
break;
}
}
size_t off = start * PAGE, len = (end - start + 1) * PAGE;
double mean, nz;
int lo, hi;
stats(bufs[b].p + off, len, &mean, &lo, &hi, &nz);
printf(" region 0x%08zx +0x%zx (%zu KiB): changed in up to %d of %d intervals (avg %.1f); "
"bytes mean %.1f min %d max %d nonzero %.0f%%\n",
off, len, len / 1024, most, snaps - 1, (double)total / (end - start + 1), mean, lo, hi, nz * 100);
regions++;
}
if (!regions) {
double mean, nz;
int lo, hi;
stats(bufs[b].p, bufs[b].size, &mean, &lo, &hi, &nz);
printf(" no change; bytes mean %.1f min %d max %d nonzero %.1f%%\n", mean, lo, hi, nz * 100);
}
free(prev), free(cur), free(changes);
}
}
#define EYE_W 512
#define EYE_H 400
#define EYE_SLOTS 8
static size_t slot_start(int k) {
return 0x230000 + (size_t)k * 0x40000 + 0x40c0 + (size_t)k * 0x40 + (k >= 4 ? 0x40 : 0);
}
// A cheap fingerprint of a frame: 256 words spread over it (a new frame changes nearly all).
static uint64_t frame_sig(const uint8_t *p) {
uint64_t h = 1469598103934665603ull, w;
for (int i = 0; i < 256; i++) {
memcpy(&w, p + (size_t)i * (EYE_W * EYE_H / 256), 8);
h = (h ^ w) * 1099511628211ull;
}
return h;
}
static volatile sig_atomic_t stop_rec;
static void on_stop(int sig) { (void)sig; stop_rec = 1; }
// The poller copies finished frames into a ring; a writer thread saves them, so a slow disk
// write never delays the polling.
#define RING 128
static struct {
uint8_t *frames;
int slot[RING];
double time[RING];
size_t head, tail, dropped; // head: next to fill (poller); tail: next to save (writer)
int done;
pthread_mutex_t mu;
pthread_cond_t cv;
FILE *f, *ix;
} ring = {.mu = PTHREAD_MUTEX_INITIALIZER, .cv = PTHREAD_COND_INITIALIZER};
static void *ring_writer(void *arg) {
(void)arg;
size_t fsize = EYE_W * EYE_H, n = 0;
pthread_mutex_lock(&ring.mu);
for (;;) {
while (ring.tail == ring.head && !ring.done) pthread_cond_wait(&ring.cv, &ring.mu);
if (ring.tail == ring.head) break;
size_t i = ring.tail % RING;
pthread_mutex_unlock(&ring.mu);
fwrite(ring.frames + i * fsize, 1, fsize, ring.f);
fprintf(ring.ix, "%zu %d %d %.6f\n", n++, ring.slot[i], ring.slot[i] >= 4, ring.time[i]);
pthread_mutex_lock(&ring.mu);
ring.tail++;
}
pthread_mutex_unlock(&ring.mu);
return NULL;
}
static void ring_put(const uint8_t *frame, int slot, double t) {
size_t fsize = EYE_W * EYE_H;
pthread_mutex_lock(&ring.mu);
int full = ring.head - ring.tail >= RING;
pthread_mutex_unlock(&ring.mu);
if (full) {
ring.dropped++;
return;
}
size_t i = ring.head % RING;
memcpy(ring.frames + i * fsize, frame, fsize);
ring.slot[i] = slot, ring.time[i] = t;
pthread_mutex_lock(&ring.mu);
ring.head++;
pthread_cond_signal(&ring.cv);
pthread_mutex_unlock(&ring.mu);
}
static int eye_buffer(void) {
for (int i = 0; i < nbufs; i++)
if (bufs[i].size == 16777216) return i;
return -1;
}
// Calls done(frame, slot, time) for every complete eye-camera frame until `seconds` pass
// (forever if negative), a stop signal comes, the tracker process goes away, or keep()
// (checked about every 0.25 s, when given) says to stop.
//
// A frame lands over several milliseconds, in bursts, and its last bursts can come after
// the camera has started its next frame. A slot isn't rewritten until at least three frames
// later (camera 0 cycles 3,0,1,2; camera 1 7,5,4,6,5,7,6,4), so a frame is passed on when
// its camera starts the frame after next. Changes to the slot just finished are late bursts,
// not a new frame. A frame's time is when its slot first changed.
static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8_t *, int, double),
int (*keep)(void)) {
uint64_t sig[EYE_SLOTS];
double first[EYE_SLOTS];
int cur[2] = {-1, -1}, prev[2] = {-1, -1};
for (int k = 0; k < EYE_SLOTS; k++) sig[k] = frame_sig(bufs[b].p + slot_start(k)), first[k] = 0;
double start = now(), checked = start, kept = start;
char proc[64];
snprintf(proc, sizeof proc, "/proc/%d", pid);
while ((seconds < 0 || now() - start < seconds) && !stop_rec) {
double t = now();
if (t - checked > 1.0) { // the tracker restarted: its buffers are stale
struct stat st;
if (stat(proc, &st) != 0) return;
checked = t;
}
if (keep && t - kept > 0.25) {
if (!keep()) return;
kept = t;
}
for (int k = 0; k < EYE_SLOTS; k++) {
uint64_t s = frame_sig(bufs[b].p + slot_start(k));
if (s == sig[k]) continue;
sig[k] = s;
int cam = k >= 4;
if (k == cur[cam] || k == prev[cam]) continue; // landing, or a late burst
if (prev[cam] >= 0) done(bufs[b].p + slot_start(prev[cam]), prev[cam], first[prev[cam]]);
prev[cam] = cur[cam];
cur[cam] = k;
first[k] = t;
}
usleep(300);
}
}
static void rec_frame(const uint8_t *frame, int slot, double t) { ring_put(frame, slot, t); }
static int rec(double seconds, const char *dir, int pid) {
int b = eye_buffer();
if (b < 0) {
fprintf(stderr, "no 16 MiB buffer\n");
return 1;
}
// Frames stream to disk (about 37 MB/s), so a long recording doesn't fill memory.
// Ctrl-C or SIGTERM ends it early and keeps what was recorded.
char path[512];
snprintf(path, sizeof path, "%s/frames.raw", dir);
ring.f = fopen(path, "wb");
snprintf(path, sizeof path, "%s/index.txt", dir);
ring.ix = fopen(path, "w");
ring.frames = malloc((size_t)RING * EYE_W * EYE_H);
if (!ring.f || !ring.ix || !ring.frames) {
perror(dir);
return 1;
}
setvbuf(ring.f, NULL, _IOFBF, 4 << 20);
signal(SIGINT, on_stop);
signal(SIGTERM, on_stop);
pthread_t writer;
pthread_create(&writer, NULL, ring_writer, NULL);
double start = now();
poll_frames(b, seconds, pid, rec_frame, NULL);
pthread_mutex_lock(&ring.mu);
ring.done = 1;
pthread_cond_signal(&ring.cv);
pthread_mutex_unlock(&ring.mu);
pthread_join(writer, NULL);
fclose(ring.f), fclose(ring.ix);
printf("%zu frames in %.1f s to %s", ring.head, now() - start, dir);
if (ring.dropped) printf(" (%zu dropped: disk too slow)", ring.dropped);
printf("\n");
free(ring.frames);
return 0;
}
// --- --share: the latest frames in shared memory for the live tracker ---
//
// The file (SHARE_PATH, mode 0600, owned by the --owner user) is a header, then SHARE_SLOTS
// entries per camera. Each entry is a 64-byte head and one 512x400 frame. Frame n of camera
// c goes in entry c * SHARE_SLOTS + n % SHARE_SLOTS. The head's `seq` is odd while it's
// written (read it before and after copying, and retry if it changed or was odd), and
// count[c] is how many frames camera c has published. tracker_pid is 0 while no frames
// come (nobody wants them, or SteamVR's tracker isn't running).
//
// This keeps the tracker's own buffers behind root: the user side only ever sees copies.
#define SHARE_SLOTS 8
#define SHARE_MAGIC 0x31434546u // "FEC1"
#define WANT_FRESH 3.0 // seconds a touch of the --want file lasts
typedef struct {
uint32_t magic, version, width, height, slots, entry_size;
volatile uint64_t count[2];
uint32_t tracker_pid, pad0;
uint8_t pad[16];
} share_head_t;
typedef struct {
volatile uint64_t seq;
double t;
uint64_t n;
uint32_t cam, slot;
uint8_t pad[32];
} share_entry_t;
_Static_assert(sizeof(share_head_t) == 64, "share header");
_Static_assert(sizeof(share_entry_t) == 64, "share entry");
static uint8_t *share;
static const char *want_path;
static uid_t owner_uid = (uid_t)-1;
static gid_t owner_gid = (gid_t)-1;
static void share_frame(const uint8_t *frame, int slot, double t) {
share_head_t *h = (share_head_t *)share;
int cam = slot >= 4;
uint64_t n = h->count[cam];
size_t esize = sizeof(share_entry_t) + EYE_W * EYE_H;
share_entry_t *e = (share_entry_t *)(share + sizeof *h + (cam * SHARE_SLOTS + n % SHARE_SLOTS) * esize);
e->seq++;
__atomic_thread_fence(__ATOMIC_RELEASE);
memcpy((uint8_t *)(e + 1), frame, EYE_W * EYE_H);
e->t = t, e->n = n, e->cam = cam, e->slot = slot;
__atomic_thread_fence(__ATOMIC_RELEASE);
e->seq++;
__atomic_thread_fence(__ATOMIC_RELEASE);
h->count[cam] = n + 1;
}
// Someone reads the frames: the want file was touched lately. It must be a regular file
// (lstat: a link isn't followed) owned by the frames' owner, so no one else can turn this on.
static int wanted(void) {
if (!want_path) return 1;
struct stat st;
if (lstat(want_path, &st) != 0 || !S_ISREG(st.st_mode)) return 0;
if (owner_uid != (uid_t)-1 && st.st_uid != owner_uid) return 0;
struct timespec ts;
clock_gettime(CLOCK_REALTIME, &ts);
double age = (ts.tv_sec - st.st_mtim.tv_sec) + (ts.tv_nsec - st.st_mtim.tv_nsec) * 1e-9;
return age < WANT_FRESH;
}
static void close_bufs(void) {
for (int i = 0; i < nbufs; i++) munmap((void *)bufs[i].p, bufs[i].size), close(bufs[i].fd);
nbufs = 0;
}
static int share_loop(const char *path) {
size_t esize = sizeof(share_entry_t) + EYE_W * EYE_H;
size_t size = sizeof(share_head_t) + 2 * SHARE_SLOTS * esize;
unlink(path);
int fd = open(path, O_RDWR | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC, 0600);
if (fd < 0 || ftruncate(fd, size) != 0) {
perror(path);
return 1;
}
if (owner_uid != (uid_t)-1 && fchown(fd, owner_uid, owner_gid) != 0) perror("fchown");
share = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
close(fd);
if (share == MAP_FAILED) {
perror("mmap");
return 1;
}
share_head_t *h = (share_head_t *)share;
*h = (share_head_t){.magic = SHARE_MAGIC, .version = 1, .width = EYE_W, .height = EYE_H,
.slots = SHARE_SLOTS, .entry_size = (uint32_t)esize};
signal(SIGINT, on_stop);
signal(SIGTERM, on_stop);
fprintf(stderr, "ft-eyegrab: sharing frames in %s%s%s\n", path, want_path ? " while wanted by " : "",
want_path ? want_path : "");
int pid = -1, idle = -1, missing = 0;
while (!stop_rec) {
if (!wanted()) {
// Nobody reads the frames: copy nothing, and let go of the tracker's buffers.
if (idle != 1) fprintf(stderr, "ft-eyegrab: idle (nobody wants frames)\n"), idle = 1;
close_bufs();
pid = -1;
h->tracker_pid = 0;
usleep(250000);
continue;
}
if (nbufs == 0 && ((pid = find_tracker()) < 0 || open_bufs(pid) <= 0 || eye_buffer() < 0)) {
// SteamVR's tracker isn't running (yet, or again).
if (!missing) fprintf(stderr, "ft-eyegrab: waiting for SteamVR's eyetracking\n"), missing = 1;
close_bufs();
h->tracker_pid = 0;
sleep(2);
continue;
}
if (idle != 0 || missing) fprintf(stderr, "ft-eyegrab: copying frames from eyetracking %d\n", pid);
idle = 0, missing = 0;
h->tracker_pid = pid;
poll_frames(eye_buffer(), -1, pid, share_frame, wanted);
struct stat st;
char proc[64];
snprintf(proc, sizeof proc, "/proc/%d", pid);
if (!stop_rec && stat(proc, &st) != 0) {
fprintf(stderr, "ft-eyegrab: eyetracking %d went away; waiting for it\n", pid);
close_bufs();
h->tracker_pid = 0;
}
}
close_bufs();
unlink(path);
return 0;
}
static int dump(int b, size_t off, size_t len, const char *file) {
if (b < 0 || b >= nbufs || off + len > bufs[b].size) {
fprintf(stderr, "out of range\n");
return 1;
}
FILE *f = fopen(file, "wb");
if (!f) {
perror(file);
return 1;
}
fwrite(bufs[b].p + off, 1, len, f);
fclose(f);
printf("wrote %zu bytes to %s\n", len, file);
return 0;
}
static int seq(int b, size_t off, size_t len, int frames, const char *dir) {
if (b < 0 || b >= nbufs || off + len > bufs[b].size) {
fprintf(stderr, "out of range\n");
return 1;
}
char path[512];
snprintf(path, sizeof path, "%s/times.txt", dir);
FILE *times = fopen(path, "w");
if (!times) {
perror(path);
return 1;
}
uint8_t *copy = malloc(len);
uint64_t last = 0;
int got = 0;
double start = now();
while (got < frames && now() - start < 30) {
uint64_t h = 0;
for (size_t i = 0; i + PAGE <= len; i += PAGE * 8) h ^= page_hash(bufs[b].p + off + i) + i;
if (h != last) {
last = h;
double t = now();
memcpy(copy, bufs[b].p + off, len);
snprintf(path, sizeof path, "%s/%04d.raw", dir, got);
FILE *f = fopen(path, "wb");
if (f) fwrite(copy, 1, len, f), fclose(f);
fprintf(times, "%d %.6f\n", got, t);
got++;
}
usleep(1000);
}
fclose(times);
free(copy);
printf("%d frames in %s\n", got, dir);
return 0;
}
static void usage(void) {
fprintf(stderr, "usage: ft-eyegrab [--share PATH [--owner UID:GID] [--want FILE] | --scan [N] | --dump I OFF LEN FILE |\n"
" --seq I OFF LEN FRAMES DIR | --rec SECONDS DIR]\n");
}
int main(int argc, char **argv) {
if (argc >= 3 && strcmp(argv[1], "--share") == 0) {
const char *uid = getenv("SUDO_UID"), *gid = getenv("SUDO_GID");
if (uid && gid) owner_uid = (uid_t)atoi(uid), owner_gid = (gid_t)atoi(gid);
for (int i = 3; i < argc; i++) {
unsigned u, g;
if (strcmp(argv[i], "--owner") == 0 && i + 1 < argc && sscanf(argv[i + 1], "%u:%u", &u, &g) == 2) {
owner_uid = u, owner_gid = g, i++;
} else if (strcmp(argv[i], "--want") == 0 && i + 1 < argc) {
want_path = argv[++i];
} else {
usage();
return 2;
}
}
return share_loop(argv[2]);
}
int pid = find_tracker();
if (pid < 0) {
fprintf(stderr, "SteamVR's eyetracking process isn't running\n");
return 1;
}
if (open_bufs(pid) <= 0) {
fprintf(stderr, "no buffers (run as root)\n");
return 1;
}
if (argc >= 2 && strcmp(argv[1], "--scan") == 0) {
scan(argc >= 3 ? atoi(argv[2]) : 40);
} else if (argc == 6 && strcmp(argv[1], "--dump") == 0) {
return dump(atoi(argv[2]), strtoul(argv[3], NULL, 0), strtoul(argv[4], NULL, 0), argv[5]);
} else if (argc == 4 && strcmp(argv[1], "--rec") == 0) {
return rec(atof(argv[2]), argv[3], pid);
} else if (argc == 7 && strcmp(argv[1], "--seq") == 0) {
return seq(atoi(argv[2]), strtoul(argv[3], NULL, 0), strtoul(argv[4], NULL, 0), atoi(argv[5]), argv[6]);
} else if (argc == 1) {
printf("eyetracking pid %d\n", pid);
for (int b = 0; b < nbufs; b++)
printf("buffer %d: fd %d, %zu bytes, ino %lu\n", b, bufs[b].xfd, bufs[b].size, bufs[b].ino);
} else {
usage();
return 2;
}
return 0;
}
gaze/tracker/ft-eyes
Executable
+453
View File
@@ -0,0 +1,453 @@
#!/usr/bin/env python3
"""ft-eyes: our own eye tracker, live.
Reads the eye-camera frames that ft-eyegrab (the root service frametop-eyegrab, installed by
gaze/tracker/install.sh) keeps in /dev/shm/frametop-eyes-cams, finds each eye's pupil and
glint pair (eyes_pupil.py), turns them into a gaze with the saved calibration and a running
slip estimate (eyes_model.py), and publishes the result in /dev/shm/frametop-eyes-gaze,
where ft-gaze reads it as the source "own". It touches /dev/shm/frametop-eyes-want every
second, and ft-eyegrab copies frames only while someone does.
The gaze service (gaze/ft-gazed) runs it in the dev container, with --watch-stdin (it quits
when its stdin closes), while Eye tracker is Own tracker or the gaze probe uses it. The
probe calibrates and teaches it; the gaze pointer's nudges reach it as clicks, from ft-gazed.
By hand: distrobox enter dev -- python3 gaze/tracker/ft-eyes -v
Calibration: ~/.local/state/frametop/gaze/eyes/calibration.json, made by the gaze probe's
calibration with the tracker toggle on Own tracker (or lab/ft-eyes-score --save CAPTURE).
Each eye's shift since the calibration (eyes_model.Shift, taught by clicks) is kept in
state.json next to it. After a restart, or frames stopping for GAP seconds (the headset
off), the next click starts that eye's shift over, since the headset may sit differently now.
Control socket: abstract datagram "@ft_eyes"; each command gets one reply line.
status JSON: calibration, and per eye its shift, clicks, whether a
glint jump is applied, and "reseat" (the next click starts
the shift over: the probe asks for a one-dot check then)
calib-start a new calibration: collect dots from now on
calib-point T0 T1 YAW PITCH a dot you looked at from T0 to T1 (CLOCK_MONOTONIC_RAW) in
that direction (head-relative degrees): "ok N0 N1 SD0 SD1"
(frames and spread in px per eye, right first) or "fail WHY"
calib-fit fit the dots, save, and start the shifts and clicks over
click T YAW PITCH a click: you were looking there just before T; teaches the
shift: "ok DX0 DY0 DX1 DY1" (the shift each eye measured)
Environment, for replays (lab/ft-eyes-e2e): FT_EYES_CAMS, FT_EYES_GAZE, FT_EYES_STATE (the
state folder), FT_EYES_SOCKET (the socket's name). With FT_EYES_CAMS set, the want file isn't
touched.
/dev/shm/frametop-eyes-gaze, 128 bytes, little-endian (mirrored in ft-gaze.cpp):
0 u32 seq odd while it's being written: read it before and after, retry if it moved
4 u32 version 1
8 f64 t the newest frame's time, CLOCK_MONOTONIC_RAW seconds
16 f32 yaw, pitch the gaze, head-relative degrees (yaw +left, pitch +up), eyes averaged
24 u32 flags bit 0: right eye in it, 1: left eye in it, 2: right shift from clicks, 3: left
28 u32 n samples published
32 f32 x4 right yaw, pitch, left yaw, pitch (NaN when that eye isn't seen)
48 f32 x4 each eye's shift since the calibration, pixels: right x, y, left x, y
64 f32 x4 pupil centre, pixels: right x, y, left x, y
"""
import json
import math
import mmap
import os
import socket
import struct
import sys
import threading
import time
from collections import deque
from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
import eyes_model # noqa: E402
import eyes_pupil # noqa: E402
CAMS = os.environ.get("FT_EYES_CAMS", "/dev/shm/frametop-eyes-cams") # overrides for replays
OUT = os.environ.get("FT_EYES_GAZE", "/dev/shm/frametop-eyes-gaze")
WANT = None if "FT_EYES_CAMS" in os.environ else Path("/dev/shm/frametop-eyes-want")
OUT_SIZE = 128
CALIBRATION = Path(os.environ.get("FT_EYES_STATE", Path.home() / ".local/state/frametop/gaze/eyes")) / "calibration.json"
W, H = 512, 400
FRESH = 0.03 # an eye's reading counts toward the output for this long (s)
LOST_EVERY = 3 # while an eye is lost, search the whole frame only every 3rd frame
EYES = ("right", "left")
STATE = CALIBRATION.parent / "state.json"
CLICKS = CALIBRATION.parent / "clicks.jsonl"
SOCKET = "\0" + os.environ.get("FT_EYES_SOCKET", "ft_eyes")
HISTORY = 12.0 # seconds of pupil positions kept per eye, for dots and clicks (the pointer's
# clicks come from ft-gazed up to 10 s after the look)
GAP = 3.0 # s without frames: the headset was off, and may sit differently now
CLICK_BEFORE = 0.3 # a click's frames: the 300 ms before it (like the probe's fixation)
CALIB_MIN = 15 # frames an eye needs in a calibration dot's window
CALIB_SPREAD = 4.0 # px: more than this and the eye moved during the dot
class Cams:
"""The shared frames. Header and entry layout: ft-eyegrab.c, share_head_t/share_entry_t."""
def __init__(self):
fd = os.open(CAMS, os.O_RDONLY)
try:
self.ino = os.fstat(fd).st_ino
self.mm = mmap.mmap(fd, 0, prot=mmap.PROT_READ)
finally:
os.close(fd)
magic, version, w, h, self.slots, self.esize = struct.unpack_from("<6I", self.mm, 0)
if magic != 0x31434546 or version != 1 or (w, h) != (W, H):
raise RuntimeError(f"{CAMS}: unexpected header")
def replaced(self):
"""ft-eyegrab restarted: it makes a new file, and this one is stale."""
try:
return os.stat(CAMS).st_ino != self.ino
except OSError:
return True
def count(self, cam):
return struct.unpack_from("<Q", self.mm, 24 + 8 * cam)[0]
def tracker(self):
return struct.unpack_from("<I", self.mm, 40)[0]
def frame(self, cam, n):
"""Frame n of a camera as (time, array), or None if it was overwritten meanwhile."""
off = 64 + (cam * self.slots + n % self.slots) * self.esize
for _ in range(3):
seq = struct.unpack_from("<Q", self.mm, off)[0]
if seq & 1:
continue
img = np.frombuffer(self.mm, np.uint8, W * H, off + 64).reshape(H, W).copy()
t, got = struct.unpack_from("<dQ", self.mm, off + 8)
if struct.unpack_from("<Q", self.mm, off)[0] == seq and got == n:
return t, img
return None
class Out:
def __init__(self):
fd = os.open(OUT, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600)
try:
os.ftruncate(fd, OUT_SIZE)
self.mm = mmap.mmap(fd, OUT_SIZE)
finally:
os.close(fd)
self.seq = (struct.unpack_from("<I", self.mm, 0)[0] + 1) & ~1 # even: at rest
self.n = 0
def write(self, t, gaze, flags, eyes, slips, pupils):
struct.pack_into("<I", self.mm, 0, self.seq + 1) # odd while writing
self.n += 1
struct.pack_into("<IdffII12f", self.mm, 4, 1, t, gaze[0], gaze[1], flags, self.n,
*eyes, *slips, *pupils)
self.seq = (self.seq + 2) & 0xFFFFFFFE
struct.pack_into("<I", self.mm, 0, self.seq)
class Eye:
def __init__(self, eye):
self.eye = eye
self.cal = None
self.slip = None
self.shift = eyes_model.Shift()
self.history = deque() # (t, x, y, glint mid x, y)
self.last = None # the previous pupil (window hint)
self.gaze = None # (t, yaw, pitch, x, y)
self.lost = 0
self.frames = self.found = 0
self.work = 0.0
self.last_t = None # the previous frame's time
def use(self, cal, shift=None):
self.cal = cal if cal is not None and cal.has("pupil", self.eye) else None
self.slip = eyes_model.SlipTracker(cal, self.eye, window=eyes_model.JUMP_WINDOW) if self.cal else None
self.shift = shift or eyes_model.Shift()
self.gaze = None
def feed(self, t, img):
self.frames += 1
if self.last_t is not None and t - self.last_t > GAP:
self.shift.reseat()
self.last_t = t
if self.last is None:
self.lost += 1
if self.lost % LOST_EVERY:
return
t0 = time.perf_counter()
p = eyes_pupil.find_pupil(img, self.last)
self.last = p
if p is not None:
self.found += 1
self.lost = 0
pair = eyes_pupil.glint_pair(p)
mid = eyes_model.pair_mid(pair) if pair else (math.nan, math.nan)
self.history.append((t, p["x"], p["y"], mid[0], mid[1]))
while self.history and self.history[0][0] < t - HISTORY:
self.history.popleft()
if self.cal:
if pair:
self.slip.add(t, (p["x"], p["y"]), mid)
self.shift.glint(self.slip.get(), t)
g = self.cal.gaze(self.eye, p["x"], p["y"], self.shift.value)
self.gaze = (t, float(g[0]), float(g[1]), p["x"], p["y"])
self.work += time.perf_counter() - t0
def window(self, t0, t1):
"""Median pupil and glint midpoint over [t0, t1], the frame count, and the spread."""
rows = np.array([r for r in self.history if t0 <= r[0] <= t1]).reshape(-1, 5)
if len(rows) == 0:
return None
pupil = np.median(rows[:, 1:3], axis=0)
spread = float(np.median(np.hypot(*(rows[:, 1:3] - pupil).T)))
mids = rows[~np.isnan(rows[:, 3]), 3:5]
mid = np.median(mids, axis=0) if len(mids) >= 3 else None
return dict(pupil=pupil, mid=mid, n=len(rows), spread=spread)
class Tracker:
def __init__(self):
self.eyes = [Eye(0), Eye(1)]
self.cal = None
self.dots = [] # calibration dots so far, in ft-eyes-score's click form
self.calibrating = False
if CALIBRATION.exists():
self.cal = eyes_model.Calibration.load(CALIBRATION)
if not self.cal.spread:
self.cal.spread = spread_from_dots(self.cal)
shifts = {}
try:
d = json.loads(STATE.read_text())
if self.cal and d.get("calibration") == self.cal.info.get("made"):
shifts = {int(k): eyes_model.Shift.from_json(v) for k, v in d.get("shift", {}).items()}
except (OSError, ValueError):
pass
for e in self.eyes:
e.use(self.cal, shifts.get(e.eye))
# Kept from the last run, but the headset may have been off since: the first
# click starts the history over (the saved shift is used until then).
e.shift.reseat()
def save_state(self):
d = {"calibration": self.cal.info.get("made") if self.cal else None,
"shift": {e.eye: e.shift.to_json() for e in self.eyes}}
tmp = STATE.with_suffix(".tmp")
tmp.write_text(json.dumps(d))
tmp.replace(STATE)
def command(self, line):
w = line.split()
if not w:
return "fail empty"
if w[0] == "status":
return json.dumps(self.status())
if w[0] == "calib-start":
self.dots, self.calibrating = [], True
return "ok"
if w[0] == "calib-point" and len(w) == 5:
if not self.calibrating:
return "fail no calibration started"
t0, t1, yaw, pitch = map(float, w[1:])
got = {e.eye: e.window(t0, t1) for e in self.eyes}
for c, g in got.items():
if g is None or g["n"] < CALIB_MIN:
return f"fail the {EYES[c]} eye was seen in only {0 if g is None else g['n']} frames"
if g["spread"] > CALIB_SPREAD:
return f"fail the {EYES[c]} eye moved ({g['spread']:.1f} px)"
self.dots.append(dict(truth=(yaw, pitch), eye={c: dict(pupil=g["pupil"], mid=g["mid"]) for c, g in got.items()}))
return "ok {} {} {:.2f} {:.2f}".format(got[0]["n"], got[1]["n"], got[0]["spread"], got[1]["spread"])
if w[0] == "calib-fit":
if len(self.dots) < eyes_model.MIN_CLICKS:
return f"fail only {len(self.dots)} dots (need {eyes_model.MIN_CLICKS})"
cal = eyes_model.Calibration.fit(self.dots, {"made": time.strftime("%Y-%m-%d %H:%M:%S"),
"dots": len(self.dots), "from": "probe calibration"})
if not all(cal.has(n, c) for n in ("pupil", "where") for c in (0, 1)):
return "fail not enough dots with both eyes"
errs = [float(np.hypot(*(np.mean([cal.gaze(c, *k["eye"][c]["pupil"]) for c in (0, 1)], axis=0)
- k["truth"]))) for k in self.dots]
if CALIBRATION.exists():
CALIBRATION.replace(CALIBRATION.with_name(time.strftime("calibration-%Y%m%d-%H%M%S.json")))
cal.save(CALIBRATION)
self.cal, self.calibrating = cal, False
for e in self.eyes:
e.use(cal)
self.save_state()
with open(CALIBRATION.with_name("calibration-dots.jsonl"), "a") as f:
for k in self.dots:
f.write(json.dumps({"made": cal.info["made"], "truth": k["truth"],
"eye": {c: {"pupil": v["pupil"].tolist(),
"mid": None if v["mid"] is None else v["mid"].tolist()}
for c, v in k["eye"].items()}}) + "\n")
return (f"ok {len(self.dots)} dots, fit median {np.median(errs):.2f} deg, eyes "
+ ", ".join(f"{EYES[c]} {cal.spread[c]:.2f}" for c in sorted(cal.spread)))
if w[0] == "click" and len(w) == 4:
if not self.cal:
return "fail not calibrated"
t, yaw, pitch = map(float, w[1:])
out, rec = [], {"time": time.time(), "t": t, "truth": [yaw, pitch], "eyes": {}}
for e in self.eyes:
g = e.window(t - CLICK_BEFORE, t)
if g is None or g["n"] < 5 or not e.cal:
out += ["nan", "nan"]
continue
d = self.cal.click_shift(e.eye, g["pupil"], (yaw, pitch))
before = e.shift.value.tolist()
e.shift.click(d, e.slip.value if e.slip else None)
rec["eyes"][e.eye] = {"pupil": g["pupil"].tolist(), "measured": d.tolist(), "before": before,
"after": e.shift.value.tolist()}
out += [f"{d[0]:.2f}", f"{d[1]:.2f}"]
self.save_state()
with open(CLICKS, "a") as f:
f.write(json.dumps(rec) + "\n")
return "ok " + " ".join(out)
return f"fail unknown command {w[0]}"
def status(self):
return {"calibration": self.cal.info if self.cal else None, "calibrating": self.calibrating,
"dots": len(self.dots),
"eyes": {EYES[e.eye]: {"shift": e.shift.value.tolist(), "clicks": len(e.shift.meas),
"jump": bool(np.any(e.shift.jump)),
"reseat": e.shift.reseated} for e in self.eyes}}
def spread_from_dots(cal):
"""The eyes' fit spreads for a calibration saved without them, from its dots in
calibration-dots.jsonl ({} if they aren't there: the eyes are then weighted alike)."""
try:
lines = CALIBRATION.with_name("calibration-dots.jsonl").read_text().splitlines()
except OSError:
return {}
dots = []
for line in lines:
d = json.loads(line)
if d.get("made") == cal.info.get("made"):
dots.append(dict(truth=d["truth"], eye={
int(c): dict(pupil=np.array(v["pupil"]), mid=None if v["mid"] is None else np.array(v["mid"]))
for c, v in d["eye"].items()}))
return eyes_model.Calibration.fit(dots).spread if dots else {}
class Want:
"""Touches the want file every second, so ft-eyegrab keeps copying frames."""
def __init__(self):
self.at = 0.0
def __call__(self):
if WANT is None or time.monotonic() - self.at < 1.0:
return
self.at = time.monotonic()
try:
fd = os.open(WANT, os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600)
os.utime(fd)
os.close(fd)
except OSError as e:
print(f"ft-eyes: {WANT}: {e}", file=sys.stderr, flush=True)
def wait_for_cams(sock, tracker, want):
while True:
want()
try:
return Cams()
except (OSError, ValueError, RuntimeError):
serve(sock, tracker)
time.sleep(0.2)
def serve(sock, tracker):
while True:
try:
data, addr = sock.recvfrom(512)
except BlockingIOError:
return
try:
reply = tracker.command(data.decode(errors="replace").strip())
except Exception as ex: # a bad command must not take the tracker down
reply = f"fail {type(ex).__name__}: {ex}"
if addr:
try:
sock.sendto(reply.encode(), addr)
except OSError:
pass
def main():
verbose = "-v" in sys.argv
if "--watch-stdin" in sys.argv:
# Run by ft-gazed through distrobox, which doesn't pass a stop on: quit when our
# stdin (its pipe) closes.
def watch():
while sys.stdin.buffer.read(4096):
pass
os._exit(0)
threading.Thread(target=watch, daemon=True).start()
want = Want()
CALIBRATION.parent.mkdir(parents=True, exist_ok=True)
tracker = Tracker()
eyes = tracker.eyes
out = Out()
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind(SOCKET)
sock.setblocking(False)
cal = tracker.cal
print("ft-eyes: " + (f"calibration from {cal.info.get('made')} ({cal.info.get('from', cal.info.get('capture'))})"
if cal else "not calibrated: run the probe's calibration with the Own tracker")
+ f"; waiting for {CAMS}", file=sys.stderr, flush=True)
cams = wait_for_cams(sock, tracker, want)
print("ft-eyes: frames found, tracking", file=sys.stderr, flush=True)
seen = [cams.count(0), cams.count(1)]
report = time.monotonic()
while True:
serve(sock, tracker)
want()
new = False
for c in (0, 1):
n = cams.count(c)
if n == seen[c]:
continue
seen[c] = n
got = cams.frame(c, n - 1) # only the newest: never fall behind
if got:
eyes[c].feed(*got)
new = True
if new:
latest = max((e.gaze[0] for e in eyes if e.gaze), default=None)
use = [e for e in eyes if e.gaze and latest - e.gaze[0] < FRESH]
if use:
yaw, pitch = tracker.cal.combine({e.eye: e.gaze[1:3] for e in use})
flags, per, shifts, pupils = 0, [], [], []
for i, e in enumerate(eyes):
fresh = e in use
flags |= (1 << i) if fresh else 0
flags |= (4 << i) if e.shift.meas else 0
per += [e.gaze[1], e.gaze[2]] if fresh else [math.nan, math.nan]
shifts += [float(v) for v in e.shift.value]
pupils += [e.gaze[3], e.gaze[4]] if fresh else [math.nan, math.nan]
out.write(latest, (yaw, pitch), flags, per, shifts, pupils)
else:
time.sleep(0.001)
now = time.monotonic()
if now - report >= 5:
if verbose:
parts = []
for e in eyes:
s = e.shift.value
parts.append(f"{EYES[e.eye]} {e.frames / 5:.0f} fps, found {e.found / max(e.frames, 1):.0%}, "
f"{e.work / max(e.frames, 1) * 1000:.2f} ms/frame, shift ({s[0]:+.1f},{s[1]:+.1f})"
f" from {len(e.shift.meas)} clicks" + (", jump" if np.any(e.shift.jump) else ""))
e.frames = e.found = 0
e.work = 0.0
print("ft-eyes: " + "; ".join(parts), file=sys.stderr, flush=True)
report = now
if cams.replaced():
print("ft-eyes: frames went away; waiting", file=sys.stderr, flush=True)
cams = wait_for_cams(sock, tracker, want)
seen = [cams.count(0), cams.count(1)]
if __name__ == "__main__":
try:
main()
except KeyboardInterrupt:
pass
gaze/tracker/install.sh
Executable
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
# Install (or remove) the frame grabber our own eye tracker needs: ft-eyegrab, as the system
# service frametop-eyegrab.service. It copies the eye-camera frames, read-only, out of
# SteamVR's eyetracking process into /dev/shm/frametop-eyes-cams for ft-eyes, and only while
# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when Eye tracker
# is Own tracker or the gaze probe uses it.
# Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit. On the
# Frame, sudo asks for the password in the terminal, or runs SUDO_ASKPASS when that's set.
# From a PC (or with no terminal), the password comes from steamos_root_pwd in the repo's .env
# and is sent to sudo -S on stdin, never on a command line.
# Usage: gaze/tracker/install.sh [install|uninstall|status|log [lines]]
set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
. "$root/scripts/_env.sh"
src=$FRAME_REPO/gaze/tracker
unit=frametop-eyegrab.service
sudo_run() {
if [ "$FRAME_LOCAL" = 1 ] && [ -n "${SUDO_ASKPASS:-}" ]; then
sudo -A bash -c "$1" # SUDO_ASKPASS supplies the password
return
fi
if [ "$FRAME_LOCAL" = 1 ] && [ -t 0 ]; then
sudo bash -c "$1" # asks for the password here
return
fi
local pw
pw=$(sed -n 's/^steamos_root_pwd=//p' "$root/.env" 2>/dev/null)
pw=${pw#[\"\']}; pw=${pw%[\"\']} # .env values may be quoted
[ -n "$pw" ] || { echo "no terminal for sudo, and steamos_root_pwd is missing from $root/.env" >&2; exit 1; }
printf '%s\n' "$pw" | on_frame "sudo -S -p '' bash -c $(printf %q "$1")"
}
case ${1:-install} in
install)
"$root/gaze/tracker/build.sh"
ids=$(on_frame 'echo "$(id -u):$(id -g)"')
fill_template "$root/gaze/tracker/$unit" | sed "s|@UID@|${ids%:*}|g; s|@GID@|${ids#*:}|g" |
on_frame "cat > /tmp/$unit"
sudo_run "set -e
install -D -m 0755 -o root -g root $src/build/ft-eyegrab /etc/frametop/ft-eyegrab
install -D -m 0644 -o root -g root /tmp/$unit /etc/systemd/system/$unit
rm -f /tmp/$unit
systemctl daemon-reload
systemctl enable $unit
systemctl restart $unit
sleep 1
echo \"$unit: \$(systemctl is-active $unit)\""
;;
uninstall)
sudo_run "systemctl disable --now $unit 2>/dev/null
rm -f /etc/systemd/system/$unit /etc/frametop/ft-eyegrab
rmdir /etc/frametop 2>/dev/null; systemctl daemon-reload; echo removed" ;;
status) on_frame "systemctl is-active $unit; ls -l /dev/shm/frametop-eyes-cams 2>/dev/null" || true ;;
log) on_frame "journalctl -u $unit --no-pager -o cat -n ${2:-20}" ;;
*) echo "usage: $0 [install|uninstall|status|log [lines]]" >&2; exit 2 ;;
esac
gaze/tracker/lab/eyes_lab.py
+31
View File
@@ -0,0 +1,31 @@
"""What the lab tools share: where recordings are kept, and the tracker's modules.
Recordings of the eye cameras are biometric data. They're kept outside the repo, in
~/.local/share/frametop/eyes/captures (FT_EYES_CAPTURES overrides), one folder each, 0700,
and never leave the Frame except for the 7i's copies frame-job makes for offline jobs.
"""
import os
import sys
from pathlib import Path
TRACKER = Path(__file__).resolve().parents[1] # gaze/tracker: ft-eyes, eyes_model, eyes_pupil
LAB = Path(__file__).resolve().parent
CAPTURES = Path(os.environ.get("FT_EYES_CAPTURES", Path.home() / ".local/share/frametop/eyes/captures"))
sys.path.insert(0, str(TRACKER))
def capture(arg):
"""A recording's folder: a path as given, or a bare name in CAPTURES."""
p = Path(arg).expanduser()
return p if p.exists() or os.sep in arg else CAPTURES / arg
def new_capture(name):
"""A new, private recording folder in CAPTURES; exits if it's already there."""
out = CAPTURES / name
if out.exists():
sys.exit(f"{out} exists")
out.mkdir(parents=True)
for d in (CAPTURES, out):
d.chmod(0o700)
return out
gaze/tracker/lab/eyes_track.py
+41
View File
@@ -0,0 +1,41 @@
"""Per-frame pupil ellipses through a recording, cached in the capture (numbers only).
ellipses(cap) -> {camera: array of rows (t, x, y, a, b, major, fill, glint mid x, y)}, every
EVERY-th frame of each camera; the glint midpoint is NaN when the pair isn't seen.
"""
import numpy as np
import eyes_lab # noqa: F401 (puts gaze/tracker on the path)
import eyes_pupil
EVERY = 2
VERSION = 2
def load_index(cap):
return np.array([[float(v) for v in l.split()]
for l in (cap / "index.txt").read_text().splitlines() if len(l.split()) == 4])
def ellipses(cap):
cache = cap / f"ellipses-v{VERSION}.npz"
if cache.exists():
z = np.load(cache)
return {0: z["cam0"], 1: z["cam1"]}
idx = load_index(cap)
frames = np.memmap(cap / "frames.raw", dtype=np.uint8, mode="r").reshape(-1, 400, 512)
idx = idx[:len(frames)]
out = {}
for c in (0, 1):
rows, prev = [], None
for i in np.where(idx[:, 2] == c)[0][::EVERY]:
p = eyes_pupil.find_pupil(frames[i], prev)
prev = p
if p is None:
continue
pair = eyes_pupil.glint_pair(p)
mx, my = ((pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2) if pair else (np.nan, np.nan)
rows.append((idx[i, 3], p["x"], p["y"], p["a"], p["b"], p["major"], p["fill"], mx, my))
out[c] = np.array(rows).reshape(-1, 9)
np.savez(cache, cam0=out[0], cam1=out[1])
return out
gaze/tracker/lab/ft-eyes-e2e
Executable
+183
View File
@@ -0,0 +1,183 @@
#!/usr/bin/env python3
"""ft-eyes-e2e: the live path end to end on two recordings, without the headset.
Starts a scratch ft-eyes (its own shared memory, socket, and state folder, so the real
calibration is untouched), then:
1. plays CALIB into it and sends each of its practice clicks as a calibration dot (the
300 ms before the press), as the probe's calibration would, and fits;
2. plays TEST into it and, at each of its practice clicks, scores what ft-eyes was
publishing in the 300 ms before the press, then sends the click, as the probe does.
So every TEST click is scored with only earlier data, like ft-eyes-score's `clicks` method.
Usage: frame-job -- lab/py lab/ft-eyes-e2e CALIB TEST [--for S] [--dump FILE]
CALIB and TEST are recordings (a bare name is in eyes_lab.CAPTURES; give full paths for
frame-job to copy them to the 7i). Runs at the recorded pace (about the two recordings'
length). --dump saves everything ft-eyes published during TEST (OUT_FIELDS per row) and each
click's score, as a pickle, for looking into the bad clicks.
"""
import json
import mmap
import os
import pickle
import re
import shutil
import socket
import struct
import subprocess
import sys
import tempfile
import time
from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
from eyes_lab import LAB, TRACKER, capture # noqa: E402
BEFORE = 0.3 # the probe's fixation window before a press (s)
# ft-eyes' output after its seq and version (ft-eyes' docstring): one row per sample.
OUT_FORMAT = "<dffII12f"
OUT_FIELDS = ("t", "yaw", "pitch", "flags", "n", "r_yaw", "r_pitch", "l_yaw", "l_pitch",
"r_shift_x", "r_shift_y", "l_shift_x", "l_shift_y", "r_pupil_x", "r_pupil_y", "l_pupil_x", "l_pupil_y")
class Run:
def __init__(self):
tag = f"ft-eyes-e2e-{os.getpid()}"
self.state = Path(tempfile.mkdtemp(prefix=tag + "-"))
self.env = dict(os.environ, FT_EYES_CAMS=f"/dev/shm/{tag}-cams", FT_EYES_GAZE=f"/dev/shm/{tag}-gaze",
FT_EYES_STATE=str(self.state), FT_EYES_SOCKET=tag)
self.log = self.state / "ft-eyes.log"
self.trackd = subprocess.Popen([sys.executable, str(TRACKER / "ft-eyes"), "-v"], env=self.env,
stdout=subprocess.DEVNULL, stderr=open(self.log, "w"))
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
self.sock.bind("\0" + tag + "-client")
self.sock.settimeout(2)
self.to = "\0" + tag
self.gaze = None
def cmd(self, line):
for _ in range(50): # ft-eyes may not have bound its socket yet
try:
self.sock.sendto(line.encode(), self.to)
return self.sock.recv(4096).decode()
except ConnectionRefusedError:
time.sleep(0.2)
raise RuntimeError("ft-eyes never answered")
def latest_frame_t(self):
"""The newest replayed frame's time, or None before the replay starts."""
try:
with open(self.env["FT_EYES_CAMS"], "rb") as f:
mm = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
except (OSError, ValueError):
return None
slots, esize = struct.unpack_from("<2I", mm, 16)
best = None
for cam in (0, 1):
n = struct.unpack_from("<Q", mm, 24 + 8 * cam)[0]
if n:
t = struct.unpack_from("<d", mm, 64 + (cam * slots + (n - 1) % slots) * esize + 8)[0]
best = t if best is None else max(best, t)
return best
def published(self):
"""ft-eyes' newest output (OUT_FIELDS), or None."""
if self.gaze is None:
try:
with open(self.env["FT_EYES_GAZE"], "rb") as f:
self.gaze = mmap.mmap(f.fileno(), 128, prot=mmap.PROT_READ)
except (OSError, ValueError):
return None
for _ in range(3):
seq = struct.unpack_from("<I", self.gaze, 0)[0]
v = struct.unpack_from(OUT_FORMAT, self.gaze, 8)
if not seq & 1 and struct.unpack_from("<I", self.gaze, 0)[0] == seq:
return v
return None
def stage(self, cap, secs, handle):
"""Play `cap` and call handle(click, samples) as each click's press time goes by.
Returns everything ft-eyes published meanwhile."""
clicks = pickle.load(open(cap / "features.pkl", "rb"))["clicks"]
replay = subprocess.Popen([sys.executable, str(LAB / "ft-eyes-replay"), str(cap), self.env["FT_EYES_CAMS"],
"--for", str(secs)], stdout=subprocess.DEVNULL)
todo, samples = list(clicks), []
while replay.poll() is None:
t = self.latest_frame_t()
v = self.published()
if v and v[0] > 0 and (not samples or v[0] != samples[-1][0]):
samples.append(v)
while t and todo and t > todo[0]["t"] + 0.05:
handle(todo.pop(0), samples)
time.sleep(0.003)
return samples
def close(self):
self.trackd.terminate()
self.trackd.wait()
for p in (self.env["FT_EYES_CAMS"], self.env["FT_EYES_GAZE"]):
if os.path.exists(p):
os.unlink(p)
shutil.rmtree(self.state, ignore_errors=True)
def main(argv):
args = [a for i, a in enumerate(argv) if not a.startswith("--") and (i == 0 or argv[i - 1] not in ("--for", "--dump"))]
if len(args) != 2:
sys.exit(__doc__)
calib, test = map(capture, args)
secs = argv[argv.index("--for") + 1] if "--for" in argv else "1e9"
dump = Path(argv[argv.index("--dump") + 1]) if "--dump" in argv else None
run = Run()
try:
print("calib-start:", run.cmd("calib-start"), flush=True)
dots = []
run.stage(calib, secs, lambda k, _s: dots.append(
run.cmd(f"calib-point {k['t'] - BEFORE} {k['t']} {k['truth'][0]} {k['truth'][1]}")))
fails = [d for d in dots if not d.startswith("ok")]
print(f"{calib.name}: {len(dots) - len(fails)} of {len(dots)} clicks taken as dots", flush=True)
whys = [re.sub(r"[\d.]+", "N", f) for f in fails]
for why in sorted(set(whys)):
print(f" {whys.count(why)} x {why}")
print("calib-fit:", run.cmd("calib-fit"), flush=True)
# ft-eyes-replay removes its file at the end; ft-eyes notices within 5 s and waits for the next.
time.sleep(6)
scored = []
def click(k, samples):
s = np.array([v for v in samples if k["t"] - BEFORE <= v[0] <= k["t"]]).reshape(-1, len(OUT_FIELDS))
err = float(np.hypot(*(np.median(s[:, 1:3], axis=0) - k["truth"]))) if len(s) >= 5 else None
reply = run.cmd(f"click {k['t']} {k['truth'][0]} {k['truth'][1]}")
st = json.loads(run.cmd("status"))["eyes"]
scored.append((k["t"], err, reply, [(v["clicks"], v["jump"]) for v in st.values()]))
test_samples = run.stage(test, secs, click)
if dump:
with open(dump, "wb") as f:
pickle.dump({"fields": OUT_FIELDS, "samples": np.array(test_samples, float),
"clicks": [dict(t=x[0], err=x[1], reply=x[2], eyes=x[3]) for x in scored]}, f)
print("dumped to", dump)
e = np.array([x[1] for x in scored if x[1] is not None])
print(f"{test.name}: {len(e)} of {len(scored)} clicks scored live", flush=True)
if len(e):
print(f" median {np.median(e):.2f} deg, 90% {np.percentile(e, 90):.2f}, "
f"after the first 5: median {np.median(e[5:]):.2f}")
print(" clicks ft-eyes refused:", sum(not x[2].startswith("ok") for x in scored))
t0 = scored[0][0] if scored else 0
print(" clicks that started an eye's shift over after a jump: right {}, left {}".format(
*(sum(x[3][c][0] == 1 for x in scored[1:]) for c in (0, 1))))
print(" by time (s): " + ", ".join(
f"{lo}-{lo + 30}: {np.median(b):.2f}" for lo in range(0, 300, 30)
if len(b := [x[1] for x in scored if x[1] is not None and lo <= x[0] - t0 < lo + 30])))
print(" worst: " + ", ".join(
f"{x[0] - t0:.0f}s {x[1]:.1f} (clicks/jump R {x[3][0][0]}/{x[3][0][1]:d} L {x[3][1][0]}/{x[3][1][1]:d})"
for x in sorted((x for x in scored if x[1] is not None), key=lambda x: -x[1])[:10]))
print("status:", run.cmd("status"))
print("ft-eyes' last report:", run.log.read_text().strip().splitlines()[-1:])
finally:
run.close()
if __name__ == "__main__":
main(sys.argv[1:])
gaze/tracker/lab/ft-eyes-record
Executable
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env python3
"""ft-eyes-record: record a live session from the shared frames, so it can be replayed and
scored later (ft-eyes-score, ft-eyes-e2e), without root and alongside ft-eyes.
Reads /dev/shm/frametop-eyes-cams (ft-eyegrab, frametop-eyegrab.service) and writes every
frame of both cameras to a new recording NAME in eyes_lab.CAPTURES: frames.raw, index.txt
("<n> <slot> <camera> <time>", as ft-eyegrab --rec), and clocks.txt (wall clock and
CLOCK_MONOTONIC_RAW, read together). It touches /dev/shm/frametop-eyes-want every second, so
ft-eyegrab copies frames even without ft-eyes. About 2 GB a minute. Stops after SECONDS
(default 900), on Ctrl-C or SIGTERM, or when the disk gets below MIN_FREE_GB. The probe's
clicks are added later, on the Frame: `lab/py lab/ft-eyes-score --clicks NAME`.
Usage: lab/ft-eyes-record NAME [SECONDS] (host Python is enough: no numpy)
"""
import mmap
import os
import shutil
import signal
import struct
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from eyes_lab import new_capture # noqa: E402
CAMS = os.environ.get("FT_EYES_CAMS", "/dev/shm/frametop-eyes-cams")
WANT = "/dev/shm/frametop-eyes-want"
W, H = 512, 400
MIN_FREE_GB = 20
class Share:
"""The shared frames. Layout: ft-eyegrab.c, share_head_t/share_entry_t."""
def __init__(self):
fd = os.open(CAMS, os.O_RDONLY)
try:
self.ino = os.fstat(fd).st_ino
self.mm = mmap.mmap(fd, 0, prot=mmap.PROT_READ)
finally:
os.close(fd)
magic, version, w, h, self.slots, self.esize = struct.unpack_from("<6I", self.mm, 0)
if magic != 0x31434546 or version != 1 or (w, h) != (W, H):
raise RuntimeError(f"{CAMS}: unexpected header")
def replaced(self):
try:
return os.stat(CAMS).st_ino != self.ino
except OSError:
return True
def count(self, cam):
return struct.unpack_from("<Q", self.mm, 24 + 8 * cam)[0]
def frame(self, cam, n):
"""(time, slot, bytes) of frame n of a camera, or None if it was overwritten."""
off = 64 + (cam * self.slots + n % self.slots) * self.esize
for _ in range(3):
seq = struct.unpack_from("<Q", self.mm, off)[0]
if seq & 1:
continue
data = self.mm[off + 64:off + 64 + W * H]
t, got, _cam, slot = struct.unpack_from("<dQII", self.mm, off + 8)
if struct.unpack_from("<Q", self.mm, off)[0] == seq and got == n:
return t, slot, data
return None
def touch_want():
"""Tell ft-eyegrab someone wants frames (it idles otherwise)."""
if "FT_EYES_CAMS" in os.environ:
return
try:
fd = os.open(WANT, os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600)
os.utime(fd)
os.close(fd)
except OSError:
pass
def open_share(deadline):
while time.monotonic() < deadline:
touch_want()
try:
return Share()
except (OSError, ValueError, RuntimeError):
time.sleep(0.5)
sys.exit(f"ft-eyes-record: no {CAMS} (is frametop-eyegrab.service running? gaze/tracker/install.sh)")
def main(argv):
if not argv or argv[0].startswith("-"):
sys.exit(__doc__)
secs = float(argv[1]) if len(argv) > 1 else 900.0
out = new_capture(argv[0])
stop = []
for sig in (signal.SIGINT, signal.SIGTERM):
signal.signal(sig, lambda *_: stop.append(1))
share = open_share(time.monotonic() + 10)
(out / "clocks.txt").write_text(f"{time.time()} {time.clock_gettime(time.CLOCK_MONOTONIC_RAW)}\n")
seen = [share.count(0), share.count(1)]
written = dropped = 0
end = time.monotonic() + secs
check = touched = time.monotonic()
with open(out / "frames.raw", "wb") as frames, open(out / "index.txt", "w") as index:
while not stop and time.monotonic() < end:
new = []
for c in (0, 1):
n = share.count(c)
if n - seen[c] > share.slots: # fell behind: those frames are gone
dropped += n - seen[c] - share.slots
seen[c] = n - share.slots
for i in range(seen[c], n):
f = share.frame(c, i)
if f is None:
dropped += 1
else:
new.append((f[0], f[1], c, f[2]))
seen[c] = n
for t, slot, c, data in sorted(new, key=lambda f: f[0]):
frames.write(data)
index.write(f"{written} {slot} {c} {t:.6f}\n")
written += 1
if not new:
time.sleep(0.002)
if time.monotonic() - touched > 1:
touched = time.monotonic()
touch_want()
if time.monotonic() - check > 5:
check = time.monotonic()
if shutil.disk_usage(out).free < MIN_FREE_GB * 1e9:
print(f"ft-eyes-record: under {MIN_FREE_GB} GB free, stopping", file=sys.stderr)
break
if share.replaced():
print("ft-eyes-record: the frame share was restarted; following it", file=sys.stderr)
share = open_share(time.monotonic() + 10)
seen = [share.count(0), share.count(1)]
print(f"ft-eyes-record: {written} frames ({dropped} dropped) in {out}", file=sys.stderr)
if __name__ == "__main__":
main(sys.argv[1:])
gaze/tracker/lab/ft-eyes-replay
Executable
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""ft-eyes-replay: play a recording into the shared-frame layout, as ft-eyegrab --share
would, at the recorded pace, to test ft-eyes without the headset.
Usage: lab/py lab/ft-eyes-replay NAME [PATH] [--from S] [--for S]
NAME is a recording (a bare name is in eyes_lab.CAPTURES). PATH defaults to
/dev/shm/frametop-eyes-cams-replay; run ft-eyes with FT_EYES_CAMS=PATH (and FT_EYES_GAZE=...
so it doesn't overwrite the live output).
"""
import mmap
import os
import struct
import sys
import time
from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
from eyes_lab import capture # noqa: E402
W, H, SLOTS = 512, 400, 8
ESIZE = 64 + W * H
def main(argv):
cap = capture(argv[0])
path = argv[1] if len(argv) > 1 and not argv[1].startswith("--") else "/dev/shm/frametop-eyes-cams-replay"
start = float(argv[argv.index("--from") + 1]) if "--from" in argv else 0.0
length = float(argv[argv.index("--for") + 1]) if "--for" in argv else 1e9
idx = np.array([[float(v) for v in l.split()] for l in (cap / "index.txt").read_text().splitlines()
if len(l.split()) == 4])
frames = np.memmap(cap / "frames.raw", dtype=np.uint8, mode="r").reshape(-1, H, W)
size = 64 + 2 * SLOTS * ESIZE
if os.path.exists(path):
os.unlink(path)
fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_EXCL, 0o600)
os.ftruncate(fd, size)
mm = mmap.mmap(fd, size)
os.close(fd)
struct.pack_into("<6I2QI", mm, 0, 0x31434546, 1, W, H, SLOTS, ESIZE, 0, 0, os.getpid())
count = [0, 0]
t0 = idx[0, 3] + start
wall0 = time.monotonic()
try:
for i in range(len(frames)):
t = idx[i, 3]
if t < t0:
continue
if t - t0 > length:
break
delay = (t - t0) - (time.monotonic() - wall0)
if delay > 0:
time.sleep(delay)
cam = int(idx[i, 2])
n = count[cam]
off = 64 + (cam * SLOTS + n % SLOTS) * ESIZE
seq = struct.unpack_from("<Q", mm, off)[0]
struct.pack_into("<Q", mm, off, seq + 1)
mm[off + 64:off + 64 + W * H] = frames[i].tobytes()
struct.pack_into("<dQII", mm, off + 8, t, n, cam, int(idx[i, 1]))
struct.pack_into("<Q", mm, off, seq + 2)
count[cam] = n + 1
struct.pack_into("<Q", mm, 24 + 8 * cam, n + 1)
finally:
os.unlink(path)
print(f"replayed {sum(count)} frames")
if __name__ == "__main__":
main(sys.argv[1:])
gaze/tracker/lab/ft-eyes-score
Executable
+291
View File
@@ -0,0 +1,291 @@
#!/usr/bin/env python3
"""ft-eyes-score: score our pupil tracker and SteamVR against the gaze probe's practice clicks.
Usage (from gaze/tracker; A and B are recordings: a bare name is in eyes_lab.CAPTURES):
frame-job -- lab/py lab/ft-eyes-score A fit and test on A, leave-one-out
frame-job -- lab/py lab/ft-eyes-score A B fit on A, test on B
lab/py lab/ft-eyes-score --save A fit on A, save it for ft-eyes (on the Frame)
lab/py lab/ft-eyes-score --clicks A... on the Frame: copy each capture's practice
clicks from the probe's log into it (the first scoring does it too)
For frame-job to copy a recording to the 7i, give its full path, e.g.
~/.local/share/frametop/eyes/captures/A.
Each practice click gives a known gaze direction: SteamVR's raw gaze at the press plus the
angle from it to where you released (you were looking there). For each click we take the
frames from just before the press and find each eye's pupil and glint pair.
Methods, each a quadratic fit per eye, both eyes averaged when both are seen:
pupil the pupil centre alone. Breaks when the headset slips on the face.
glint pupil minus the glint pair's midpoint. Slip moves both alike, so this holds up,
but the right eye's pair is often off the cornea.
clicks the pupil centre minus the shift the earlier clicks measured, with the glints
only noticing a sudden jump (eyes_model.Shift). The one ft-eyes uses.
slip the pupil centre minus a slip estimate. Wherever the pair is seen, the glint
method gives the gaze, the fit says where the pupil should be for that gaze, and
the difference is the slip. Slip changes slowly, so the median over the last
30 seconds applies to every frame, with or without glints (see eyes_model.py).
SteamVR gets the same quadratic fit on its raw gaze.
"""
import json
import pickle
import sys
import time
from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
from eyes_lab import capture # noqa: E402
import eyes_model # noqa: E402
import eyes_pupil # noqa: E402
PRACTICE = Path.home() / ".local/state/frametop/gaze/practice.jsonl"
BEFORE = (0.25, 0.02) # frames from 250 ms to 20 ms before the press
EYES = {0: "right", 1: "left"}
MIN_FRAMES = 5
TRACK_EVERY = 9 # the slip track uses every 9th frame per camera (10 a second)
VERSION = 4 # bump when the features change, to rebuild the caches
# --- Features -------------------------------------------------------------------------
def load_index(cap):
# Skip a half-written last line (the recorder may still be running).
return np.array([[float(v) for v in l.split()]
for l in (cap / "index.txt").read_text().splitlines() if len(l.split()) == 4])
def eye_features(frames):
"""Median pupil centre and glint-pair midpoint over some frames of one eye."""
ps = [p for p in map(eyes_pupil.find_pupil, frames) if p]
if len(ps) < MIN_FRAMES:
return None
pupil = np.median([(p["x"], p["y"]) for p in ps], axis=0)
mids = []
for p in ps:
pair = eyes_pupil.glint_pair(p)
if pair:
mids.append(((pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2))
mid = np.median(mids, axis=0) if len(mids) >= 3 else None
return dict(pupil=pupil, mid=mid)
def practice_log(cap, t0, t1, wall, mono):
"""The probe's practice records during the capture. The first time (on the Frame), cut
from the probe's log into the capture's practice.jsonl, so the capture carries its own
clicks (the 7i has no probe log)."""
own = cap / "practice.jsonl"
if not own.exists():
if not PRACTICE.exists():
sys.exit(f"{own} is missing: run `lab/py lab/ft-eyes-score --clicks {cap.name}` once on the Frame")
keep = [line for line in open(PRACTICE)
if t0 - 5 < json.loads(line)["time"] - wall + mono < t1 + 60]
own.write_text("".join(keep))
return [json.loads(line) for line in open(own)]
def features(cap):
"""Per-click and per-session features, cached in the capture (numbers only)."""
cache = cap / "features.pkl"
if cache.exists():
f = pickle.loads(cache.read_bytes())
if f.get("version") == VERSION:
return f
wall, mono = map(float, (cap / "clocks.txt").read_text().split())
idx = load_index(cap)
frames = np.memmap(cap / "frames.raw", dtype=np.uint8, mode="r").reshape(-1, 400, 512)
idx = idx[:len(frames)]
t0, t1 = idx[0, 3], idx[-1, 3]
clicks = []
for r in practice_log(cap, t0, t1, wall, mono):
src = r.get("sources", {})
s = src.get("mmap1")
if r.get("mode") != "practice" or not s or "off" not in s:
continue
press = r["time"] - r["held_s"] - wall + mono
if not t0 + BEFORE[0] < press < t1:
continue
k = np.where((idx[:, 3] > press - BEFORE[0]) & (idx[:, 3] < press - BEFORE[1]))[0]
eye = {c: eye_features([frames[i] for i in k if idx[i, 2] == c]) for c in (0, 1)}
# The truth: a source's gaze at the press plus the angle from it to the release
# point. The angle is converted with a local linear fit of the screen, so the
# closer the source, the better: ours when the live tracker was running.
own = src.get("own") if src.get("own", {}).get("off") else None
base = own or s
truth = (base["hy"] + base["off"][0], base["hp"] + base["off"][1])
clicks.append(dict(t=press, truth=truth, truth_from="own" if own else "mmap1",
steam=(s["hy"], s["hp"]),
steam_err=float(np.hypot(s["hy"] - truth[0], s["hp"] - truth[1])),
live_err=float(np.hypot(*own["off"])) if own else None,
press_err=r.get("press_err_deg"), press_source=r.get("source"), eye=eye))
# The slip track: pupil and pair midpoint on a sample of frames through the session.
track = {}
for c in (0, 1):
rows = []
for i in np.where(idx[:, 2] == c)[0][::TRACK_EVERY]:
p = eyes_pupil.find_pupil(frames[i])
pair = p and eyes_pupil.glint_pair(p)
if pair:
rows.append((idx[i, 3], p["x"], p["y"],
(pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2))
track[c] = np.array(rows).reshape(-1, 5)
f = dict(version=VERSION, clicks=clicks, track=track, span=(t0, t1))
cache.write_bytes(pickle.dumps(f))
return f
# --- Fitting --------------------------------------------------------------------------
class Model:
"""A calibration fitted on some clicks, plus SteamVR's quadratic fit on the same."""
def __init__(self, clicks):
self.cal = eyes_model.Calibration.fit(clicks)
self.steam = eyes_model.Quad([k["steam"] for k in clicks], [k["truth"] for k in clicks])
def slip(self, c, track, t):
"""Median slip (pixels) over the track in the SLIP_WINDOW seconds before t."""
tr = track[c]
if len(tr) == 0:
return None
return self.cal.slip(c, tr[(tr[:, 0] < t) & (tr[:, 0] > t - eyes_model.SLIP_WINDOW)])
def predict(self, method, k, track):
"""Gaze for one click by a method, combining the eyes it has; None if neither."""
cal, out = self.cal, {}
for c in (0, 1):
e = k["eye"][c]
if e is None or not cal.has("pupil", c):
continue
if method == "pupil":
out[c] = cal.gaze(c, *e["pupil"])
elif method == "glint" and cal.has("glint", c) and e["mid"] is not None:
out[c] = cal.fits["glint", c].one(*(e["pupil"] - e["mid"]))
elif method == "slip":
s = self.slip(c, track, k["t"])
if s is not None:
out[c] = cal.gaze(c, *e["pupil"], slip=s)
return cal.combine(out)
# --- Scoring --------------------------------------------------------------------------
METHODS = ("pupil", "glint", "slip")
def report(name, err, total):
err = np.asarray([e for e in err if e is not None])
if len(err) == 0:
print(f" {name:36s} no clicks")
return
print(f" {name:36s} {len(err):3d}/{total} median {np.median(err):5.2f} "
f"mean {err.mean():5.2f} 90% {np.percentile(err, 90):5.2f} deg")
def score(train, test, track, same):
"""Errors per click for each method; leave-one-out when train and test are the same.
"clicks" goes through the test clicks in order, as live: each is predicted with the
shift the earlier ones measured (eyes_model.Shift), then teaches it."""
errs = {m: [] for m in METHODS + ("clicks", "steam")}
model = None if same else Model(train)
shifts = {c: eyes_model.Shift() for c in (0, 1)}
for i, k in enumerate(test):
mdl = Model(train[:i] + train[i + 1:]) if same else model
for m in METHODS:
g = mdl.predict(m, k, track)
errs[m].append(None if g is None else float(np.hypot(*(g - k["truth"]))))
errs["steam"].append(float(np.hypot(*(mdl.steam(k["steam"])[0] - k["truth"]))))
out = {}
for c in (0, 1):
e = k["eye"][c]
if e is None or not mdl.cal.has("pupil", c):
continue
tr = track[c]
# The glint estimate as live would have had it over the last second, for the hold.
for back in (eyes_model.JUMP_HOLD, eyes_model.JUMP_HOLD / 2, 0.0):
te = k["t"] - back
g = mdl.cal.slip(c, tr[(tr[:, 0] < te) & (tr[:, 0] > te - eyes_model.JUMP_WINDOW)]) if len(tr) else None
shifts[c].glint(g, te)
out[c] = mdl.cal.gaze(c, *e["pupil"], slip=shifts[c].value)
shifts[c].click(mdl.cal.click_shift(c, e["pupil"], k["truth"]), g)
errs["clicks"].append(float(np.hypot(*(mdl.cal.combine(out) - k["truth"]))) if out else None)
return errs
def summary(f, label):
cl = f["clicks"]
T = np.array([k["truth"] for k in cl])
print(f"{label}: {len(cl)} clicks over {f['span'][1] - f['span'][0]:.0f} s, gaze yaw "
f"{T[:, 0].min():.0f}..{T[:, 0].max():.0f}, pitch {T[:, 1].min():.0f}..{T[:, 1].max():.0f}")
for c in (0, 1):
n = sum(k["eye"][c] is not None for k in cl)
g = sum(k["eye"][c] is not None and k["eye"][c]["mid"] is not None for k in cl)
print(f" {EYES[c]} eye: pupil before {n} clicks, glint pair before {g}; "
f"slip track {len(f['track'][c])} frames with the pair")
CALIBRATION = Path.home() / ".local/state/frametop/gaze/eyes/calibration.json"
def main(args):
if args and args[0] == "--clicks":
for cap in map(capture, args[1:]):
wall, mono = map(float, (cap / "clocks.txt").read_text().split())
lines = (cap / "index.txt").read_text().splitlines()
ts = [float(l.split()[3]) for l in (lines[0], lines[-1])]
print(f"{cap}: {len(practice_log(cap, *ts, wall, mono))} practice records")
return
if args and args[0] == "--save":
cap = capture(args[1])
f = features(cap)
cal = eyes_model.Calibration.fit(f["clicks"], {"capture": cap.name, "clicks": len(f["clicks"]),
"made": time.strftime("%Y-%m-%d %H:%M")})
cal.save(CALIBRATION)
print(f"saved {CALIBRATION}: {sorted(f'{n} {EYES[e]}' for n, e in cal.fits)}")
return
ca = capture(args[0])
a = features(ca)
summary(a, ca.name)
if len(args) > 1:
cb = capture(args[1])
b = features(cb)
summary(b, cb.name)
print(f"\nFit on {ca.name}, tested on {cb.name}:")
test, track, errs = b["clicks"], b["track"], score(a["clicks"], b["clicks"], b["track"], False)
else:
print("\nLeave-one-out within the session:")
test, track, errs = a["clicks"], a["track"], score(a["clicks"], a["clicks"], a["track"], True)
n = len(test)
report("SteamVR raw", [k["steam_err"] for k in test], n)
steam_press = [k["press_err"] for k in test if k["press_source"] != "own"]
report("SteamVR + probe's live correction", steam_press, len(steam_press))
live = [k["live_err"] for k in test if k["live_err"] is not None]
if live:
report("ours live (ft-eyes, as the probe saw it)", live, n)
own_press = [k["press_err"] for k in test if k["press_source"] == "own"]
report("ours live + probe's live correction", own_press, len(own_press))
report("SteamVR + quadratic fit", errs["steam"], n)
for m in METHODS:
report(f"ours, {m}", errs[m], n)
report("ours, clicks (shift from earlier clicks)", errs["clicks"], n)
# Like for like: the clicks every method scored.
common = [i for i in range(n) if all(errs[m][i] is not None for m in METHODS)]
print(f"\nSame {len(common)} clicks for every method:")
report("SteamVR + quadratic fit", [errs["steam"][i] for i in common], len(common))
for m in METHODS:
report(f"ours, {m}", [errs[m][i] for i in common], len(common))
if len(args) == 1:
for c in (0, 1):
tr = track[c]
if len(tr) < 20:
continue
mdl = Model(a["clicks"])
ss = [mdl.slip(c, track, t) for t in np.linspace(tr[0, 0] + eyes_model.SLIP_WINDOW, tr[-1, 0], 6)]
print(f" {EYES[c]} eye slip estimate through the session (px): "
+ " ".join(f"({s[0]:+.1f},{s[1]:+.1f})" for s in ss if s is not None))
if __name__ == "__main__":
main(sys.argv[1:] or ["practice1"])
gaze/tracker/lab/ft-eyes-session
Executable
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
# ft-eyes-session: record the eye cameras and SteamVR's gaze together, for SECONDS (default 10),
# as a new recording NAME in ~/.local/share/frametop/eyes/captures (FT_EYES_CAPTURES).
# Usage: gaze/tracker/lab/ft-eyes-session NAME [SECONDS]
#
# frames.raw, index.txt, clocks.txt every eye-camera frame (ft-eyes-record, from the shared
# frames of frametop-eyegrab.service; no root)
# gaze.jsonl ft-gaze's samples (gaze/build/ft-gaze)
# Both are timed on CLOCK_MONOTONIC_RAW ("t" in gaze.jsonl, the last column of index.txt).
set -euo pipefail
lab=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
repo=$(cd "$lab/../../.." && pwd)
name=${1:?usage: ft-eyes-session NAME [SECONDS]}
secs=${2:-10}
out=${FT_EYES_CAPTURES:-$HOME/.local/share/frametop/eyes/captures}/$name
[ -e "$out" ] && { echo "$out exists" >&2; exit 1; }
# ft-gaze runs in the dev container; start it first, it takes a moment to connect. It quits
# when its stdin closes (--watch-stdin): killing distrobox doesn't reach it in the container.
# The recorder makes the folder; ft-gaze's output waits for it.
tmp=$(mktemp -d)
sleep $((secs + 3)) | "$HOME/.local/bin/distrobox" enter dev -- "$repo/gaze/build/ft-gaze" \
--watch-stdin > "$tmp/gaze.jsonl" 2> "$tmp/gaze.log" &
gaze=$!
sleep 2
python3 "$lab/ft-eyes-record" "$name" "$secs"
wait $gaze || true
mv "$tmp/gaze.jsonl" "$tmp/gaze.log" "$out/"
rmdir "$tmp"
echo "$(wc -l < "$out/index.txt") frames, $(wc -l < "$out/gaze.jsonl") gaze samples in $out"
gaze/tracker/lab/py
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
# Python with numpy and OpenCV for the lab tools: gaze/tracker/build/venv (gaze/tracker/build.sh
# makes it in the dev container on the Frame; frame-job's SETUP makes it on the PC). On the
# Frame's host it runs in the dev container, where it was made.
# Usage: lab/py lab/TOOL [args] (from gaze/tracker)
here=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
py=$here/build/venv/bin/python
if grep -qx 'ID=steamos' /etc/os-release 2>/dev/null; then
exec "$HOME/.local/bin/distrobox" enter dev -- "$py" "$@"
fi
[ -x "$py" ] || { echo "no $py: run gaze/tracker/build.sh (or a frame-job job, whose setup makes it)" >&2; exit 1; }
exec "$py" "$@"
gaze/tracker/requirements.txt
+5
View File
@@ -0,0 +1,5 @@
# ft-eyes and the lab tools (gaze/tracker/build.sh puts them in build/venv, in the dev
# container; frame-job's SETUP does the same on the PC). Fedora's python3-opencv would pull in
# VTK, GDAL, and over a gigabyte of map data; these wheels are about 165 MB.
numpy==2.5.3
opencv-python-headless==5.0.0.93
input-settings/ft_input_settings.py
+1 -1
View File
@@ -85,7 +85,7 @@ GAZE_SETTINGS = [
("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"),
]
# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias.
GAZE_TRACKERS = {"steam": "SteamVR's eye tracker", "own": "our own eye tracker (frame-eyes)"}
GAZE_TRACKERS = {"steam": "SteamVR's eye tracker", "own": "our own eye tracker"}
GAZE_EYES = {"auto": "auto", "left": "left eye", "right": "right eye"}
# Pointer settings: key, label, default, min, max, step, unit.
POINTER_SETTINGS = [
input-settings/main.qml
+6 -4
View File
@@ -779,16 +779,18 @@ Kirigami.ApplicationWindow {
onToggled: if (checked) backend.setGazeTracker("steam")
}
Controls.RadioButton {
text: "Own tracker (frame-eyes)"
text: "Own tracker"
checked: backend.gazeTracker === "own"
onToggled: if (checked) backend.setGazeTracker("own")
}
}
Controls.Label {
// Ours runs apart from Frametop (frame-eyes: tools/fe-live) and keeps its own calibration.
// The gaze service runs ours (gaze/tracker/ft-eyes); it needs the frame grabber
// (gaze/tracker/install.sh, root) and keeps its own calibration.
visible: backend.gazeTracker === "own" && backend.gazeServiceRunning
&& (!gpage.status.own_running || gpage.status.own_reseat || !gpage.status.calibration_samples)
text: !gpage.status.own_running ? "Not running: start it in frame-eyes (tools/fe-live), then calibrate"
text: !gpage.status.eyegrab ? "Needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)"
: !gpage.status.own_running ? "Starting…"
: !gpage.status.calibration_samples ? "Not calibrated: use Calibrate… with Own tracker"
: "The headset was off: your first nudge and click resets where it sits"
color: gpage.status.own_running ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor
@@ -921,7 +923,7 @@ Kirigami.ApplicationWindow {
+ "to drag something instead. Look away to hand back: how far from the pointer you look before the "
+ "gaze takes it back from the mouse. Largest nudge to learn: bigger mouse moves before a click "
+ "are treated as using the mouse, not correcting the gaze. Eye tracker: SteamVR's, or our own "
+ "(frame-eyes), which keeps its own calibration (Calibrate… with Own tracker). Eye bias: the gaze "
+ "(gaze/tracker), which keeps its own calibration (Calibrate… with Own tracker). Eye bias: the gaze "
+ "combines both eyes, since their errors partly cancel; Left or Right counts that eye twice as "
+ "much, and Auto weights each by how far off it was at your recent nudges."
}
session/frametop.conf.example
+1 -1
View File
@@ -30,7 +30,7 @@ POINTER_GAZE_RETAKE=5 # gaze mode: how far (degrees) you look away from the
POINTER_GAZE_NUDGE_MAX=8 # gaze mode: a mouse nudge up to this far (degrees) before a click is learned as the eye tracker's error
POINTER_GAZE_HOLD=0.5 # gaze mode: a press held this long (s) without moving becomes a real press (to drag); moved or released sooner, it clicks where you let go
POINTER_GAZE_SHOW=1 # gaze mode: the dot shows this long (s) after the mouse moves it; also while a press is held, and a pulse per click
GAZE_TRACKER=steam # gaze service: steam = SteamVR's eye tracker | own = our own (frame-eyes' fe-trackd, calibrated in the gaze probe with Own tracker)
GAZE_TRACKER=steam # gaze service: steam = SteamVR's eye tracker | own = our own (gaze/tracker: its frame grabber needs gaze/tracker/install.sh; calibrated in the gaze probe with Own tracker)
GAZE_EYE=auto # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
META_DASHBOARD=0 # 1 = a Meta tap on a pass-through keyboard toggles the SteamVR dashboard (pointer mode only)
SHARE_KEYS=0 # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen)