DeeJanuz--frametop/gaze/README.md

Gaze (experimental)

The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (the pointer goes where you look, and the mouse does the last bit), and the tools to calibrate and measure it.

• ft-gaze (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on.
• gazecal.py has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log.
• probe/ft-gazeprobe (GTK 4, host Python) is a fullscreen playground. It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.

gaze/build.sh                 # build ft-gaze
gaze/probe/install.sh         # build, and add Frametop Gaze Probe to the app menu
gaze/probe/ft-gazeprobe --screen 1
gaze/run.sh install           # the gaze service, with SteamVR
gaze/ft-gazectl on            # the pointer follows your gaze (off: the mouse alone)

Gaze pointer

Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:

• ft-gazed (host Python, a user service: gaze/run.sh install) runs ft-gaze and corrects its gaze. It drops blinks and dropouts, smooths with a fixation lock, and applies the calibration from the probe (calibration.json, reloaded when the probe changes it) plus what the pointer has learned since (pointer-lessons.json). It sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
• The pointer helper's gaze mode (off by default: gaze/ft-gazectl on, POINTER_GAZE=1 in ~/.config/frametop.conf, or a button mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. Clicks and drags work as usual.
• Learning from nudges: if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. ft-gazectl status shows the lessons, and ft-gazectl forget drops them.
• Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.

Lessons are logged to pointer-lessons.jsonl: the raw gaze, the true direction, the correction at the time, and how far off it was.

Gaze sources

Source	Where it comes from
SteamVR action	An eyetracking action bound to /user/head/eyetracking (actions/), read with IVRInput::GetEyeTrackingDataRelativeToNow. This is the supported way.
mmap set 1, set 2	/dev/shm/eye-server.mmap, which SteamVR's eyetracking process writes for the HMD driver (driver_cv.so). It has two sets of per-eye directions in head space. The layout is undocumented (offsets are in ft-gaze.cpp) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written.

The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See Accept usercal in ~/.local/share/Steam/logs/eyetracking.txt.

Probe

The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The keys do the same (Tab, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place.

• Run calibration (start here): the initial calibration, modeled on Apple Vision Pro's eye setup. Face the centre and keep your head still. Look at one dot and press the trigger, then at each of six dots in a circle. That happens in three rounds, and the screen goes dark, then medium, then bright, because pupil size changes with brightness and the tracker's error with it. Each round turns the ring 20 degrees, and the middle round's ring is half the size, so the 21 dots cover the middle, halfway out, and the edge of your view. The ring's size is Calibration ring (degrees, 20 by default, less if the window is too small). Error grows toward the edge, and the calibration can only correct as far out as it has seen dots. The current dot is a bright pulsing dot with a point in the middle; finished dots fade to specks, so your eyes don't go back to them. Samples from blinks and from moments when the tracker lost an eye are dropped: openness under half of what it was during that look (not a fixed level, because your lids come down when you look down, and you squint in the bright round), or the angle between the eyes jumping more than 1.5 degrees from its median (that angle depends on how far away you're looking, so only a jump counts). Each dot is measured with medians, so one bad sample can't fail it. A look that lands where the gaze was for another dot of the round is refused as a look at the wrong dot. Mouse clicks don't count during a calibration run or a test: use Enter or Space. If a dot still fails, the message says why and the next try listens longer. After two failures, S (or the menu) skips the dot. Every attempt is logged to calibration-attempts.jsonl. At the end it fits every source's calibration from all the dots, replacing what it had learned (quadratic if the model was none). Esc cancels. The run is saved as calibration-*.json. With "Test after calibration" on (the default), the accuracy test starts right after, on new spots.
• Free look: the gaze dot. The trigger calibrates wherever you're looking (see below).
• Accuracy test: look at each target and press Enter or Space. "Test spots" picks where the targets go. Calibrated area (the default) puts 15 new spots inside the calibration ring (the centre, 7 halfway out, 7 near the ring), turned so none sits on a calibration dot. It checks the calibration where it was made, with your head facing the centre. The window grids reach past that area. On a wide screen that's far more than your eyes turn without your head, so they show how the calibration holds up beyond where it was made. For each source the test records the error before and after correction (degrees and pixels), the share of targets within 1 degree, jitter, and the corrected error by region of your view (centre, up, down-left, and so on), worst first. On screen, each target gets a faint line to the raw gaze and a solid line to where the corrected dot was, green under 1 degree, yellow under 2, red above. Tests since the last calibration are listed as a trend.
• Refine calibration: refits from the latest calibration run's dots plus every calibrated-area test since. It tries offset, affine, quadratic, and quadratic+grid, scoring each on points it wasn't fitted on (leave-one-out), and uses the best. Then test again: each test adds its targets, so test, refine, test is the loop. The scores are in the panel and in refinements.jsonl.
• Snap practice: a field of desktop-like elements (toolbar icons, list rows, buttons, tiles, small links), some close together, inside the calibrated area. The gaze snaps to the nearest element and highlights it, so the pointer lands on a whole element instead of a spot. Look at the orange one and tap Enter (or click) to click it. If the wrong one is highlighted, hold the press instead: the highlight locks and stops following your gaze. Glance toward the right one (look off to that side and back), and each glance steps the highlight to the next element that way. Or move the mouse, and the highlight follows it from where it was. Let go on the right one. A glance works however far off the tracker is, because only the eye movement counts, and the tracker gets that right: its error barely changes over a couple of degrees. Whichever element you let go on is taken as the one you were looking at when you pressed, and the gap from the gaze at the press is learned as the tracker's error there (not if it's over 6 degrees after the correction, which means a wrong element). That's what it would learn in real use, where nothing knows which element you meant. The probe does know (the orange one), so each click is also scored: right at the press, right in the end, and whether the snap would have been right with the calibration alone. Backspace takes back the last click's lesson. Clicks go to snaps.jsonl.
• Click practice: the white dot is a gaze pointer, the way it would be in real use. Look at the target and press (click, or Enter), and keep looking at it. The dot stops following your gaze. If it isn't on the target, keep holding and move the mouse: the dot moves with it. Let go on the target. You were looking at where you let go when you pressed, so the drag is the tracker's error there, and the click corrections learn it (not if it's over 6 degrees after the correction). A click without a drag teaches nothing: it only says the dot was close enough. The target is only for scoring: would a plain gaze click have hit at the press, and did the drag end on it. The drag is drawn for a moment. Presses go to practice.jsonl. The Frametop pointer (the mouse's own white dot) stays where the mouse puts it. The probe only reads its movement. An earlier version steered the Frametop pointer onto the gaze with the pointer helper's move commands. It lost the user's pointer: the helper's pointer goes idle, or a controller takes the laser, and the moves piled up. Taking over the real pointer belongs in the pointer helper itself, which knows its own state and can aim straight at the gaze.

The default trigger is freeze and look, the on-demand calibration. The press freezes the dot where the tracker says you're looking. Then look at the frozen dot: it's a target right where you're looking, and it stays put. After settle ms it averages the unsmoothed gaze for capture ms, or until you let go if you hold longer. The gap between the frozen dot and that average is the tracker's error at that spot, and the calibration learns it. The live dot is hidden while frozen so it can't pull your eye (the "Live dot while frozen" option shows it anyway). Freeze and look drops blink and dropout samples and uses medians, like the calibration run. A capture is thrown out if the gaze spread more than max spread (1 degree) or the error is over max error (12 degrees; the real error reaches 8-9 degrees looking well up or down). Each capture is logged to captures.jsonl.

The older triggers, nudge with head and nudge with eyes, are still there. Hold, then move the frozen dot onto what you meant with your head or eyes (nudge gain scales the movement), and release. When nudging with your eyes, don't look at the dot: it follows your gaze, error included, so it runs away.

Smoothing defaults to fixation lock. It holds the dot on the running mean of the current fixation and jumps when your gaze leaves the fixation radius. One Euro follows more smoothly, and its beta is per degree a second. Sitting still, raw gaze jitters by about 0.25-0.3 degrees, and mmap set 2 was the quietest source, so it's the default.

Click corrections ("Learn from clicks", on by default) are learned on the fly from snap and practice clicks, on top of the calibration. Right-click, Clear click corrections forgets them and keeps the calibration. The first click shifts the whole correction. More clicks bend it (the same quadratic terms, held near zero except the offset), and what's left near each click is added within about 3 degrees of it: on your data, errors less than 3 degrees apart are alike, and ones further apart aren't. Recent clicks count more, so it follows SteamVR's gaze as that moves. A big element only weakly says where on it you looked, so a wide list row barely counts sideways. Replayed on logged points, a calibration from an earlier session was 4.95 degrees off; one click brought that to 2.3, five to 1.6, and twenty to 1.2. They're saved with the calibration and start over with a new calibration run.

SteamVR's eye tracker also calibrates itself, from clicks (Accept usercal in ~/.local/share/Steam/logs/eyetracking.txt). It takes a quick mouse-button down and up as "you were looking there", if the gaze was held within 5 degrees of the click. In the logs, the accepted clicks were all under 0.14 s, one of 0.38 s was "too slow", and a click that moved between down and up was refused. It keeps that inside the running eyetracking process and saves nothing, so when SteamVR starts again, its calibration starts over and the raw gaze moves: your 13:39 and 21:23 sessions had a restart between them, and the error's shape changed, not just its offset. The panel shows when the eye tracker started, whether that was after your calibration, and how many clicks it has learned from since. Snap and practice clicks are what keep up with it: a drag is too slow for SteamVR to take, and a quick click on the snapped element teaches both calibrations the same spot.

Pick the Correction model in the panel or the right-click menu. Choosing one fits it right away from the latest calibration's dots and the calibrated-area tests since (points.jsonl), and the choice is saved with the calibration. The models (per source) are none, offset, affine (offset plus a straight-line change across your view), quadratic (the default), and affine+grid or quadratic+grid (plus a 10-degree grid for what's left). Quadratic is the second-order polynomial video eye trackers usually calibrate with. The Frame's error grows as your eyes turn away from the centre: it overstates vertical movement, more the further up or down you look, and looking up adds a sideways error. A straight line only follows part of that. A polynomial runs away outside the spots it was fitted on, so the model only follows it to 3 degrees past the range of view it has seen. They're keyed by where you're looking relative to your head, and saved in ~/.local/state/frametop/gaze/calibration.json. Freeze captures go to captures.jsonl, nudges to practice.jsonl, and test results to test-*.json in the same folder. Every calibration dot and test target is also added to points.jsonl: where in your view it was, the raw error, the error with the calibration of the time, and the spread. That's the data for refining, and for finding where the calibration is off.