mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-06 03:00:06 +02:00
Use the other eye when the tracker loses one, and add a headset fit check
SteamVR's combined gaze keeps going on one eye, but it holds the lost eye's yaw, so the gaze moves half as far sideways as the eyes do. ft-gaze now reads each eye's tracking uncertainty and raw measurement from eye-server.mmap. When the tracker loses an eye, ft-gazed takes the gaze from the other one, plus the offset that eye usually shows against both, learned while both are seen. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze. Glances down at the keyboard, past every screen, aren't sent. The pointer stays put, and eyes lost there don't count as lost. The gaze probe gets a Headset fit mode. It shows per-eye tracking, openness and confidence, maps where each eye gets lost, gives hints, and has a guided check. The settings app opens it from the Gaze page and shows how often each eye is lost. The probe can also test each eye alone, and its side panel now collapses to a title bar so the dot isn't hidden behind it. Snapping to UI elements is deferred; the mouse drag is the correction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed
1 parent
e60cb2f28a
commit
f01c84f9a3
10 files changed
+715
-68
No files matched your search
+3
-1
@@ -83,7 +83,7 @@ A few overlays need special handling:
|
||||
|
||||
Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
|
||||
|
||||
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse or controller button mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz, and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze; the mouse is gaze mode's only pointer device for now. The dot shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. The gaze moving the pointer doesn't show it: you know where you're looking. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured.
|
||||
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse or controller button mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze; the mouse is gaze mode's only pointer device for now. The dot shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. The gaze moving the pointer doesn't show it: you know where you're looking. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured.
|
||||
|
||||
Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
|
||||
|
||||
@@ -121,6 +121,8 @@ Steam, not systemd, suspends the Frame: after `system_idle_suspend_ac_sec` (an h
|
||||
|
||||
Flatpak apps need `XDG_DATA_DIRS` to include Flatpak's exports, or Plasma opens Discover instead of launching them, so the session sources `/etc/profile.d/flatpak.sh`.
|
||||
|
||||
The private runtime directory also moves the session's document portal to `$XDG_RUNTIME_DIR/frametop/doc`, and that broke saving and uploading in Flatpak apps. The file picker (xdg-desktop-portal 1.18.4 on SteamOS) gives a sandboxed app the host path of the file it picked, `/run/user/1000/frametop/doc/ID/NAME`. Inside the sandbox the portal is at `/run/flatpak/doc`, and `/run/user/1000` is a private per-app folder (`.flatpak/APP/xdg-run` in the runtime directory). So Brave created the missing folder there, "finished" the download into it, and the file vanished when the session cleaned up. The session script now links that path to `/run/flatpak/doc` in each installed app's folder before Plasma starts. Upstream xdg-desktop-portal fixed this after 1.22.1 (commit `69ba5e1`) by handing Flatpak apps `/run/flatpak/doc` paths, after which the links go unused.
|
||||
|
||||
A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it.
|
||||
|
||||
Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`.
|
||||
|
||||
+1
-1
@@ -110,7 +110,7 @@ A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs
|
||||
- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, faster or slower, pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
|
||||
- Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through. Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
|
||||
- Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
|
||||
- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), the gaze mode sliders, the gaze service's state (headset, samples per second, whether only one eye is tracked, the calibration, the nudges learned), and Calibrate (opens the gaze probe), Reload calibration, and Forget nudges.
|
||||
- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Calibrate (opens the gaze probe), Check headset fit (opens the probe's Headset fit mode), Reload calibration, and Forget nudges.
|
||||
- Bluetooth lists paired devices and has Apply Bluetooth fixes, which runs `/etc/steamframe/bt-fixups.sh` through `pkexec`. Pair new devices in Steam.
|
||||
|
||||
Device rules are saved in `~/.config/frametop-input.json`. `input-settings/install.sh` installs the menu entry. Its launcher hands podman the real `XDG_RUNTIME_DIR` and user bus and gives the app the session's Wayland socket, because the desktop session runs on a private D-Bus and podman fails on it.
|
||||
|
||||
+11
-4
@@ -18,7 +18,7 @@ gaze/ft-gazectl on # the pointer follows your gaze (off: the mouse al
|
||||
|
||||
Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:
|
||||
|
||||
- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. It uses SteamVR's combined gaze (mmap set 1), which keeps going when the tracker loses one eye; set 2's combined direction is off by half of whatever the lost eye reads, and the tracker does lose an eye for minutes at a time. It drops blinks (both eyes closing), smooths with a fixation lock, and applies the calibration from the probe (`calibration.json`, reloaded when the probe changes it) plus what the pointer has learned since (`pointer-lessons.json`). It sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
|
||||
- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. It uses SteamVR's combined gaze (mmap set 1). When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery. Looks down past the screens (under 20 degrees down, on no Frametop screen: a glance at the keyboard) aren't sent, so the pointer stays where it was instead of following you down, and eyes lost there aren't counted. It drops blinks (both eyes closing or lost), smooths with a fixation lock, and applies the calibration from the probe (`calibration.json`, reloaded when the probe changes it) plus what the pointer has learned since (`pointer-lessons.json`). It sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
|
||||
- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse or controller button mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on. The dot only shows while the mouse moves it, while a press is held, and as a pulse when you click.
|
||||
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
|
||||
- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
|
||||
@@ -30,13 +30,20 @@ Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction,
|
||||
| Source | Where it comes from |
|
||||
| --- | --- |
|
||||
| SteamVR action | An `eyetracking` action bound to `/user/head/eyetracking` (`actions/`), read with `IVRInput::GetEyeTrackingDataRelativeToNow`. This is the supported way. |
|
||||
| mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. |
|
||||
| mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space: set 1 is filtered, and its two eyes always share one pitch; set 2 is each eye's own reading. After each set come the tracker's variances for each eye, and at the end each eye's raw measurement and its variance (the tracker's confidence in that frame), which ft-gaze passes on for the fit check. |
|
||||
| Left eye, right eye | Each eye alone, from set 2: calibrate and test them to see what one eye is worth against both. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. |
|
||||
|
||||
The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`.
|
||||
The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`. When the tracker loses an eye, the same log says `CEyePoseUKF L: Large dt` (or `R`) as it starts that eye over.
|
||||
|
||||
## Headset fit
|
||||
|
||||
The probe's Headset fit mode (Check headset fit on the Gaze page, or `ft-gazeprobe --mode fit`) shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it.
|
||||
|
||||
Losing an eye is usually about where you look, not the tracker. On this Frame the left eye was lost 57 to 64 % of the time looking 30 to 50 degrees down (at the keyboard) and the right eye never; at screen height both were seen over 98 % of the time. Looking down, the lids come down over the eyes. That's harmless, since the gaze service ignores looks down past the screens: they show on the maps, but not in the counts or as a problem.
|
||||
|
||||
## Probe
|
||||
|
||||
The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The keys do the same (Tab, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place.
|
||||
The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The arrow in the panel's title bar collapses it to just that bar, so the dot and targets behind it stay visible; the collapsed bar stays through tests. The keys do the same (Tab, C, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place.
|
||||
|
||||
- **Run calibration (start here):** the initial calibration, modeled on Apple Vision Pro's eye setup. Face the centre and keep your head still. Look at one dot and press the trigger, then at each of six dots in a circle. That happens in three rounds, and the screen goes dark, then medium, then bright, because pupil size changes with brightness and the tracker's error with it. Each round turns the ring 20 degrees, and the middle round's ring is half the size, so the 21 dots cover the middle, halfway out, and the edge of your view. The ring's size is `Calibration ring` (degrees, 20 by default, less if the window is too small). Error grows toward the edge, and the calibration can only correct as far out as it has seen dots. The current dot is a bright pulsing dot with a point in the middle; finished dots fade to specks, so your eyes don't go back to them. Samples from blinks and from moments when the tracker lost an eye are dropped: openness under half of what it was during that look (not a fixed level, because your lids come down when you look down, and you squint in the bright round), or the angle between the eyes jumping more than 1.5 degrees from its median (that angle depends on how far away you're looking, so only a jump counts). Each dot is measured with medians, so one bad sample can't fail it. A look that lands where the gaze was for another dot of the round is refused as a look at the wrong dot. Mouse clicks don't count during a calibration run or a test: use Enter or Space. If a dot still fails, the message says why and the next try listens longer. After two failures, S (or the menu) skips the dot. Every attempt is logged to `calibration-attempts.jsonl`. At the end it fits every source's calibration from all the dots, replacing what it had learned (quadratic if the model was none). Esc cancels. The run is saved as `calibration-*.json`. With "Test after calibration" on (the default), the accuracy test starts right after, on new spots.
|
||||
- **Free look:** the gaze dot. The trigger calibrates wherever you're looking (see below).
|
||||
|
||||
@@ -0,0 +1,367 @@
|
||||
"""fitcheck: how well the eye tracker sees each eye, for fitting the headset (ft-gazeprobe's
|
||||
Headset fit mode).
|
||||
|
||||
From each ft-gaze sample it takes, per eye, whether the tracker has that eye (its variance
|
||||
for the eye's direction, "unc", under EYE_LOST; see gazecal), how open the eye is, and the
|
||||
tracker's own confidence in its latest measurement of it ("eye" "q": the measurement's
|
||||
variance, about 2e-5 on a clear view). It keeps that per direction you look in (10 degree
|
||||
cells, and a few named regions), so a map shows where each eye gets lost, and turns it into
|
||||
hints.
|
||||
|
||||
On the Frame this was written for, the left eye was lost 57-63 % of the time looking 30-50
|
||||
degrees down (at the keyboard) and the right never; at screen height both were seen over
|
||||
98 % of the time. Looking down, the lids come down over the eyes, and a glance at the
|
||||
keyboard isn't where the gaze pointer matters: ft-gazed ignores looks down past the
|
||||
screens. So those are on the maps, but not in the cards' counts or the hints' warnings.
|
||||
|
||||
Directions are head-relative degrees (yaw +left, pitch +up), the combined gaze's.
|
||||
"""
|
||||
|
||||
import math
|
||||
import statistics
|
||||
from collections import deque
|
||||
|
||||
from gazecal import EYE_FOUND, EYE_LOST
|
||||
|
||||
EYES = ("Left eye", "Right eye")
|
||||
CELL = 10.0
|
||||
YAW = (-40, 40)
|
||||
PITCH = (-50, 30)
|
||||
CLOSED = 0.12 # openness under this: closed (a blink, or squeezed shut)
|
||||
MIN_REGION = 60 # samples in a region before it's judged (two thirds of a second)
|
||||
|
||||
# Named regions, for the hints: (key, words, test on yaw and pitch).
|
||||
REGIONS = [
|
||||
("down", "down (at a keyboard or desk)", lambda y, p: p < -20),
|
||||
("up", "up", lambda y, p: p > 15),
|
||||
("left", "to the left", lambda y, p: y > 20 and -20 <= p <= 15),
|
||||
("right", "to the right", lambda y, p: y < -20 and -20 <= p <= 15),
|
||||
("centre", "straight ahead (screen height)", lambda y, p: abs(y) <= 20 and -20 <= p <= 15),
|
||||
]
|
||||
|
||||
# The guided check: dots on the screen (fractions of its size; the corners stay clear of the
|
||||
# probe's title bar and toolbar), then prompts to look past it. Seconds each.
|
||||
GUIDE = [
|
||||
("dot", (0.5, 0.5), 2.0), ("dot", (0.12, 0.2), 2.0), ("dot", (0.88, 0.2), 2.0),
|
||||
("dot", (0.88, 0.92), 2.0), ("dot", (0.12, 0.92), 2.0), ("dot", (0.5, 0.92), 2.0),
|
||||
("look", "Look down at your keyboard", 4.0), ("look", "Look up, above the screen", 3.0),
|
||||
("look", "Look far to the left", 3.0), ("look", "Look far to the right", 3.0),
|
||||
("dot", (0.5, 0.5), 2.0),
|
||||
]
|
||||
|
||||
|
||||
class FitCheck:
|
||||
def __init__(self):
|
||||
self.reset()
|
||||
|
||||
def reset(self):
|
||||
self.lost = [False, False]
|
||||
self.lost_since = [None, None]
|
||||
self.losses = [0, 0] # times each eye was lost
|
||||
self.durations = [[], []] # how long each loss lasted (s)
|
||||
self.cells = [{}, {}] # per eye: (i, j) -> [samples, lost]
|
||||
self.regions = [{k: [0, 0] for k, _, _ in REGIONS} for _ in EYES]
|
||||
self.recent = [deque(maxlen=900), deque(maxlen=900)] # (t, lost) for the last 10 s
|
||||
self.q = [deque(maxlen=180), deque(maxlen=180)] # recent fresh measurement variances
|
||||
self.open = [0.0, 0.0]
|
||||
self.unc = [0.0, 0.0]
|
||||
self.gaze = None
|
||||
self.samples = 0
|
||||
self.guide = None # {"start": t, "step": i, "results": [...]}
|
||||
self.have_eye_data = False
|
||||
|
||||
# --- Samples ---
|
||||
|
||||
def feed(self, s, now):
|
||||
m1 = s["src"].get("mmap1") or {}
|
||||
unc, opens = m1.get("unc"), m1.get("open")
|
||||
if "hy" not in m1 or not unc or not opens:
|
||||
return
|
||||
self.have_eye_data = True
|
||||
self.samples += 1
|
||||
eye = s.get("eye") or {}
|
||||
hy, hp = m1["hy"], m1["hp"]
|
||||
self.gaze = (hy, hp)
|
||||
self.unc = list(unc)
|
||||
for k in (0, 1):
|
||||
self.open[k] += 0.2 * (opens[k] - self.open[k])
|
||||
was = self.lost[k]
|
||||
self.lost[k] = unc[k] > (EYE_FOUND if was else EYE_LOST)
|
||||
if self.lost[k] and not was:
|
||||
if not looking_down(hy, hp):
|
||||
self.losses[k] += 1
|
||||
self.lost_since[k] = now
|
||||
elif was and not self.lost[k] and self.lost_since[k] is not None:
|
||||
if not looking_down(hy, hp):
|
||||
self.durations[k].append(now - self.lost_since[k])
|
||||
self.lost_since[k] = None
|
||||
q = eye.get("q")
|
||||
if q and (eye.get("new") or [1, 1])[k]:
|
||||
self.q[k].append(q[k])
|
||||
closed = [opens[k] < CLOSED for k in (0, 1)]
|
||||
if all(closed) or all(self.lost):
|
||||
return # a blink: says nothing about the fit
|
||||
key = (math.floor(hy / CELL), math.floor(hp / CELL))
|
||||
for k in (0, 1):
|
||||
c = self.cells[k].setdefault(key, [0, 0])
|
||||
c[0] += 1
|
||||
c[1] += self.lost[k]
|
||||
for rk, _, test in REGIONS:
|
||||
if test(hy, hp):
|
||||
r = self.regions[k][rk]
|
||||
r[0] += 1
|
||||
r[1] += self.lost[k]
|
||||
if not looking_down(hy, hp):
|
||||
self.recent[k].append((now, self.lost[k]))
|
||||
g = self.guide
|
||||
if g and g["step"] < len(GUIDE):
|
||||
res = g["results"][g["step"]]
|
||||
res[0] += 1
|
||||
res[1] += self.lost[0]
|
||||
res[2] += self.lost[1]
|
||||
|
||||
# --- The guided check ---
|
||||
|
||||
def toggle_guide(self, now):
|
||||
if self.guide and self.guide["step"] < len(GUIDE):
|
||||
self.guide = None
|
||||
else:
|
||||
self.guide = {"start": now, "step": 0, "step_start": now, "results": [[0, 0, 0] for _ in GUIDE]}
|
||||
|
||||
def guide_step(self, now):
|
||||
"""The current step (kind, what, seconds left), or None when there's no check running."""
|
||||
g = self.guide
|
||||
if not g or g["step"] >= len(GUIDE):
|
||||
return None
|
||||
kind, what, secs = GUIDE[g["step"]]
|
||||
if now - g["step_start"] >= secs:
|
||||
g["step"] += 1
|
||||
g["step_start"] = now
|
||||
return self.guide_step(now)
|
||||
return kind, what, secs - (now - g["step_start"])
|
||||
|
||||
# --- Summaries ---
|
||||
|
||||
def status(self, k):
|
||||
if not self.samples:
|
||||
return "no data", (0.6, 0.6, 0.6)
|
||||
if self.lost[k]:
|
||||
return "LOST", (1.0, 0.35, 0.3)
|
||||
if self.open[k] < CLOSED:
|
||||
return "closed", (0.8, 0.8, 0.8)
|
||||
return "tracking", (0.35, 1.0, 0.5)
|
||||
|
||||
def tracked_share(self, k, now, window=10.0):
|
||||
pts = [lost for t, lost in self.recent[k] if now - t <= window]
|
||||
return (1 - sum(pts) / len(pts)) if pts else None
|
||||
|
||||
def signal(self, k):
|
||||
"""The tracker's recent confidence in this eye, 0..1 (from its measurement variance:
|
||||
2e-5 or less is 1, 1e-3 or more is 0), or None."""
|
||||
if len(self.q[k]) < 10:
|
||||
return None
|
||||
q = statistics.median(self.q[k])
|
||||
return min(1.0, max(0.0, (math.log10(1e-3) - math.log10(max(q, 1e-9))) / (math.log10(1e-3) - math.log10(2e-5))))
|
||||
|
||||
def region_share(self, k, key):
|
||||
n, lost = self.regions[k][key]
|
||||
return (lost / n) if n >= MIN_REGION else None
|
||||
|
||||
def hints(self):
|
||||
if not self.have_eye_data:
|
||||
return ["No per-eye data from ft-gaze (it needs SteamVR's eye-server.mmap, and a current build)."]
|
||||
if self.samples < 3 * MIN_REGION:
|
||||
return ["Look around slowly (the screen's corners, then down at your keyboard, up, left and right) "
|
||||
"or press Enter for a guided check."]
|
||||
out = []
|
||||
bad = {}
|
||||
for k in (0, 1):
|
||||
for key, words, _ in REGIONS:
|
||||
share = self.region_share(k, key)
|
||||
if share is not None and share >= 0.15:
|
||||
bad.setdefault(key, {})[k] = share
|
||||
for key, words, _ in REGIONS:
|
||||
if key not in bad:
|
||||
continue
|
||||
eyes = bad[key]
|
||||
if len(eyes) == 2:
|
||||
if key == "down":
|
||||
out.append("Both eyes get lost looking down at the keyboard. That's fine: the gaze service "
|
||||
"ignores looks down past the screens.")
|
||||
elif key == "centre":
|
||||
out.append(f"Both eyes get lost looking {words} ({eyes[0]:.0%} and {eyes[1]:.0%} of the time): "
|
||||
"check the lenses are clean and the headset is on as usual; if it stays like this, "
|
||||
"the tracker isn't getting a clear view of either eye.")
|
||||
else:
|
||||
out.append(f"Both eyes get lost looking {words}: that's past what the tracker covers for your "
|
||||
"face, not one eye's fit.")
|
||||
continue
|
||||
k = next(iter(eyes))
|
||||
other = self.region_share(1 - k, key)
|
||||
vs = f", the {EYES[1 - k].lower()} {other:.0%}" if other is not None else ""
|
||||
line = f"{EYES[k]}: lost {eyes[k]:.0%} of the time looking {words}{vs}."
|
||||
if key == "down":
|
||||
line += (" That's fine: glancing at the keyboard, the lids come down over the eyes, and the gaze "
|
||||
"service ignores looks down past the screens, so the pointer stays put.")
|
||||
elif key == "centre":
|
||||
line += (" Even at screen height: clean that lens, and check its distance from your eye and the "
|
||||
"IPD. Lashes that touch the lens get in the camera's way too.")
|
||||
else:
|
||||
line += (" At the edge of your view: try the IPD setting, and centring the headset between your "
|
||||
"eyes.")
|
||||
out.append(line)
|
||||
s0, s1 = self.signal(0), self.signal(1)
|
||||
if s0 is not None and s1 is not None and abs(s0 - s1) > 0.25:
|
||||
k = 0 if s0 < s1 else 1
|
||||
out.append(f"The tracker is less sure of your {EYES[k].lower()} even when it has it "
|
||||
f"(signal {min(s0, s1):.0%} against {max(s0, s1):.0%}).")
|
||||
if not out:
|
||||
out.append("Both eyes are tracked everywhere you've looked so far.")
|
||||
return out
|
||||
|
||||
# --- Drawing (cairo) ---
|
||||
|
||||
def draw(self, cr, w, h, text, now):
|
||||
# Right of the probe's collapsed title bar, under its toolbar (top right).
|
||||
left = 300
|
||||
top = 190
|
||||
text(cr, left, top - 60, "Headset fit", (1, 1, 1), 30)
|
||||
text(cr, left, top - 28, "Adjust the headset and watch each eye. Enter: guided check. R: start over.",
|
||||
(0.8, 0.8, 0.8), 18)
|
||||
card_w = min(560, (w - left - 80) / 2)
|
||||
mh = max(0, min(card_w * 0.8, h - top - 280 - 200))
|
||||
for k in (0, 1):
|
||||
x = left + k * (card_w + 40)
|
||||
self.draw_card(cr, x, top, card_w, text, now, k)
|
||||
self.draw_map(cr, x, top + 280, card_w, mh, text, k)
|
||||
y = top + 280 + (mh + 60 if mh >= 80 else 0)
|
||||
for line in self.hints()[:4]:
|
||||
for part in wrap(line, max(40, int((w - left - 40) / 10))):
|
||||
if y > h - 30:
|
||||
break
|
||||
text(cr, left, y, part, (1, 0.95, 0.75), 18)
|
||||
y += 26
|
||||
y += 8
|
||||
step = self.guide_step(now)
|
||||
g = self.guide
|
||||
if step:
|
||||
kind, what, left_s = step
|
||||
if kind == "dot":
|
||||
fx, fy = what
|
||||
x, y = fx * w, fy * h
|
||||
cr.set_source_rgba(1, 0.85, 0.2, 0.95)
|
||||
cr.arc(x, y, 14 + 4 * math.sin(now * 6), 0, 2 * math.pi)
|
||||
cr.fill()
|
||||
else:
|
||||
text(cr, w / 2 - 260, h / 2, f"{what} ({left_s:.0f})", (1, 0.85, 0.2), 34)
|
||||
elif g and g["step"] >= len(GUIDE):
|
||||
self.draw_guide_results(cr, w, h, text)
|
||||
|
||||
def draw_card(self, cr, x, y, cw, text, now, k):
|
||||
cr.set_source_rgba(1, 1, 1, 0.06)
|
||||
cr.rectangle(x, y, cw, 230)
|
||||
cr.fill()
|
||||
word, col = self.status(k)
|
||||
text(cr, x + 16, y + 38, EYES[k], (1, 1, 1), 26)
|
||||
cr.select_font_face("sans")
|
||||
cr.set_font_size(26)
|
||||
text(cr, x + cw - 16 - cr.text_extents(word).x_advance, y + 38, word, col, 26)
|
||||
rows = [("Open", self.open[k]), ("Signal", self.signal(k)), ("Seen, last 10 s", self.tracked_share(k, now))]
|
||||
yy = y + 70
|
||||
for label, v in rows:
|
||||
text(cr, x + 16, yy + 16, label, (0.85, 0.85, 0.85), 17)
|
||||
bx, bw = x + 170, cw - 250
|
||||
cr.set_source_rgba(1, 1, 1, 0.12)
|
||||
cr.rectangle(bx, yy, bw, 20)
|
||||
cr.fill()
|
||||
if v is not None:
|
||||
v = min(1.0, max(0.0, v))
|
||||
cr.set_source_rgba(*bar_colour(v), 0.9)
|
||||
cr.rectangle(bx, yy, bw * v, 20)
|
||||
cr.fill()
|
||||
text(cr, bx + bw + 10, yy + 16, f"{v:.0%}", (0.9, 0.9, 0.9), 17)
|
||||
yy += 36
|
||||
d = self.durations[k]
|
||||
longest = max(d) if d else 0
|
||||
n = self.losses[k]
|
||||
text(cr, x + 16, yy + 22, f"Lost {n} time{'' if n == 1 else 's'}" + (f", longest {longest:.1f} s" if longest >= 0.05 else ""),
|
||||
(0.85, 0.85, 0.85), 17)
|
||||
|
||||
def draw_map(self, cr, x, y, mw, mh, text, k):
|
||||
"""Where you looked (yaw across, pitch up), each cell coloured by how often this eye
|
||||
was lost there: green never, red always, dark: not looked there yet."""
|
||||
if mh < 80:
|
||||
return
|
||||
cols = int((YAW[1] - YAW[0]) / CELL)
|
||||
rows = int((PITCH[1] - PITCH[0]) / CELL)
|
||||
cw, ch = mw / cols, mh / rows
|
||||
text(cr, x, y - 8, f"Where the {EYES[k].lower()} gets lost", (0.85, 0.85, 0.85), 17)
|
||||
for i in range(cols):
|
||||
yaw_i = math.floor(YAW[1] / CELL) - 1 - i # left of the map is your left (+yaw)
|
||||
for j in range(rows):
|
||||
pitch_j = math.floor(PITCH[1] / CELL) - 1 - j
|
||||
c = self.cells[k].get((yaw_i, pitch_j))
|
||||
cx, cy = x + i * cw, y + j * ch
|
||||
if c and c[0] >= 10:
|
||||
share = c[1] / c[0]
|
||||
cr.set_source_rgba(*bar_colour(1 - share), 0.75)
|
||||
else:
|
||||
cr.set_source_rgba(1, 1, 1, 0.05)
|
||||
cr.rectangle(cx + 1, cy + 1, cw - 2, ch - 2)
|
||||
cr.fill()
|
||||
# Straight ahead, and the gaze now.
|
||||
def at(yaw, pitch):
|
||||
return x + (YAW[1] - yaw) / (YAW[1] - YAW[0]) * mw, y + (PITCH[1] - pitch) / (PITCH[1] - PITCH[0]) * mh
|
||||
cr.set_source_rgba(1, 1, 1, 0.35)
|
||||
cr.set_line_width(1)
|
||||
ox, oy = at(0, 0)
|
||||
cr.move_to(ox - 10, oy)
|
||||
cr.line_to(ox + 10, oy)
|
||||
cr.move_to(ox, oy - 10)
|
||||
cr.line_to(ox, oy + 10)
|
||||
cr.stroke()
|
||||
text(cr, x, y + mh + 20, "+ ahead, bottom rows: keyboard", (0.6, 0.6, 0.6), 14)
|
||||
if self.gaze:
|
||||
gx, gy = at(max(YAW[0], min(YAW[1], self.gaze[0])), max(PITCH[0], min(PITCH[1], self.gaze[1])))
|
||||
cr.set_source_rgba(1, 1, 1, 0.95)
|
||||
cr.arc(gx, gy, 5, 0, 2 * math.pi)
|
||||
cr.fill()
|
||||
|
||||
def draw_guide_results(self, cr, w, h, text):
|
||||
res = self.guide["results"]
|
||||
lines = []
|
||||
for (kind, what, _), (n, l0, l1) in zip(GUIDE, res):
|
||||
if not n:
|
||||
continue
|
||||
name = what if kind == "look" else "dot at {:.0%}, {:.0%}".format(*what)
|
||||
lines.append(f"{name}: left lost {l0 / n:.0%}, right {l1 / n:.0%}")
|
||||
y = h / 2 - 20 * len(lines)
|
||||
text(cr, w / 2 - 300, y - 40, "Guided check", (1, 0.85, 0.2), 26)
|
||||
for line in lines:
|
||||
text(cr, w / 2 - 300, y, line, (1, 1, 1), 19)
|
||||
y += 30
|
||||
|
||||
|
||||
def looking_down(yaw, pitch):
|
||||
"""A look down at the keyboard: the "down" region, which ft-gazed doesn't send on."""
|
||||
return pitch < -20
|
||||
|
||||
|
||||
def bar_colour(v):
|
||||
"""Red (0) through amber to green (1)."""
|
||||
if v < 0.5:
|
||||
return 1.0, 0.3 + 0.9 * v, 0.3
|
||||
return 1.0 - 1.3 * (v - 0.5), 0.75 + 0.25 * (v - 0.5) * 2, 0.35
|
||||
|
||||
|
||||
def wrap(s, width):
|
||||
words, lines, cur = s.split(), [], ""
|
||||
for wd in words:
|
||||
if cur and len(cur) + 1 + len(wd) > width:
|
||||
lines.append(cur)
|
||||
cur = wd
|
||||
else:
|
||||
cur = f"{cur} {wd}".strip()
|
||||
if cur:
|
||||
lines.append(cur)
|
||||
return lines
|
||||
+45
-9
@@ -8,13 +8,24 @@
|
||||
//
|
||||
// {"t":<sample time, CLOCK_MONOTONIC_RAW s>,"age":<ms old when read>,"n":<sample counter>,
|
||||
// "head":{"yaw":..,"pitch":..,"hit":HIT}, head forward ray (for head nudging)
|
||||
// "src":{"action":SRC,"mmap1":SRC,"mmap2":SRC}}
|
||||
// "src":{"action":SRC,"mmap1":SRC,"mmap2":SRC,"left":SRC,"right":SRC},"eye":EYE}
|
||||
// SRC = {"hy":..,"hp":..,"hit":HIT} or {"ok":0} hy/hp: gaze direction relative to the
|
||||
// head, degrees (yaw +left, pitch +up)
|
||||
// mmap1 adds "open":[l,r] (probably eye openness, 0 in a blink) and "dist" (vergence
|
||||
// distance, m); both mmap sets add "lr", the angle between the eyes (deg), which
|
||||
// jumps when the tracker loses an eye, and "eyes":[[hy,hp],[hy,hp]], each eye's own
|
||||
// direction (left, right), for calibrating the eyes separately.
|
||||
// direction (left, right), for calibrating the eyes separately, and "unc":[l,r],
|
||||
// the tracker's uncertainty about each eye's direction (its filter's variance):
|
||||
// about 0.0005-0.002 while it sees the eye, 0.015-0.03 once it's lost it.
|
||||
// "left":SRC,"right":SRC each eye's own direction from set 2
|
||||
// (set 1's eyes always share one pitch, and while it's lost an eye it keeps that
|
||||
// eye's yaw where it was: set 2 is each eye's own reading). From the head's origin,
|
||||
// not the eye's.
|
||||
// EYE = {"q":[l,r],"m":[[x,y],[x,y]],"new":[l,r]} the tracker's latest measurement of
|
||||
// each eye before filtering: "m" (camera-relative, undocumented units), "q" its
|
||||
// variance (about 2e-5 on a clear view of the eye, rising as the lid or lashes get
|
||||
// in the way), "new" whether it changed since the last sample (it freezes while the
|
||||
// tracker can't see that eye, and in blinks). "eye" is null without the mmap.
|
||||
// HIT = {"s":<screen>,"x":..,"y":..,"j":[dx/dhy,dy/dhy,dx/dhp,dy/dhp],"dpp":<deg per px>}
|
||||
// or null. x, y are pixels on that screen; j is pixels per degree of head-relative
|
||||
// yaw and pitch there, so a correction in degrees can be turned into pixels and back.
|
||||
@@ -78,7 +89,13 @@ constexpr size_t kLeft1 = 0x15f, kRight1 = 0x16b; // set 1: unit vectors, head
|
||||
constexpr size_t kFix1 = 0x18f; // set 1 fixation point: length is the vergence distance (m)
|
||||
constexpr size_t kLeft2 = 0x19b, kRight2 = 0x1a7; // set 2
|
||||
constexpr size_t kOpen = 0x1cb; // two floats, 0..1: probably eye openness or confidence
|
||||
constexpr size_t kNeed = 0x1d3;
|
||||
// After each set's two directions, six floats: the left eye's variance (three), the
|
||||
// right's (three; the middle one of each is shared). They jump when an eye is lost.
|
||||
constexpr size_t kVar1 = 0x177, kVar2 = 0x1b3;
|
||||
// The measurements the filter is fed: left x, y, right x, y, then the variance of each (left
|
||||
// x, y, right x, y). An eye's pair stops changing while the tracker can't see it.
|
||||
constexpr size_t kMeas = 0x1d3;
|
||||
constexpr size_t kNeed = 0x1f3;
|
||||
|
||||
struct EyeFile {
|
||||
const uint8_t *p = nullptr;
|
||||
@@ -115,6 +132,7 @@ struct EyeSample {
|
||||
double t = 0;
|
||||
Vec3 left1, right1, fix1, left2, right2;
|
||||
float open[2] = {0, 0};
|
||||
float var1[6] = {}, var2[6] = {}, meas[8] = {};
|
||||
};
|
||||
|
||||
// A consistent copy: the writer has no seqlock we can use, so read until the counter and
|
||||
@@ -127,6 +145,9 @@ bool ReadSample(const EyeFile &f, EyeSample &s) {
|
||||
s.left1 = f.V(kLeft1), s.right1 = f.V(kRight1), s.fix1 = f.V(kFix1);
|
||||
s.left2 = f.V(kLeft2), s.right2 = f.V(kRight2);
|
||||
std::memcpy(s.open, f.p + kOpen, sizeof s.open);
|
||||
std::memcpy(s.var1, f.p + kVar1, sizeof s.var1);
|
||||
std::memcpy(s.var2, f.p + kVar2, sizeof s.var2);
|
||||
std::memcpy(s.meas, f.p + kMeas, sizeof s.meas);
|
||||
std::atomic_thread_fence(std::memory_order_acquire);
|
||||
if (f.Get<uint32_t>(kCounter) == n0 && f.Get<double>(kTime) == t0) {
|
||||
s.n = n0, s.t = t0;
|
||||
@@ -377,6 +398,7 @@ int main(int argc, char **argv) {
|
||||
screens.Start();
|
||||
PoseHistory history;
|
||||
uint32_t lastN = 0;
|
||||
float lastMeas[8] = {};
|
||||
double lastEmit = 0;
|
||||
int actionErrors = 0;
|
||||
vr::EVRInputError lastActionError = vr::VRInputError_None;
|
||||
@@ -423,7 +445,7 @@ int main(int argc, char **argv) {
|
||||
lastActionError = ae;
|
||||
}
|
||||
|
||||
std::string m1 = "{\"ok\":0}", m2 = m1;
|
||||
std::string m1 = "{\"ok\":0}", m2 = m1, left = m1, right = m1, eye = "null";
|
||||
if (haveMmap) {
|
||||
// lr: the angle between the two eyes' directions. It's a fraction of a degree
|
||||
// normally; when the tracker loses one eye (or during a blink) it jumps.
|
||||
@@ -438,12 +460,26 @@ int main(int argc, char **argv) {
|
||||
std::snprintf(b, sizeof b, "\"eyes\":[[%.4f,%.4f],[%.4f,%.4f]],", ly, lp, ry, rp);
|
||||
return std::string(b);
|
||||
};
|
||||
char extra[128];
|
||||
auto unc = [](const float *v) {
|
||||
char b[64];
|
||||
std::snprintf(b, sizeof b, "\"unc\":[%.5f,%.5f],", std::max(v[0], v[2]), std::max(v[3], v[5]));
|
||||
return std::string(b);
|
||||
};
|
||||
char extra[256];
|
||||
std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1),
|
||||
s.open[0], s.open[1], lr(s.left1, s.right1));
|
||||
m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1));
|
||||
m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1) + unc(s.var1));
|
||||
std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2));
|
||||
m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2));
|
||||
m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2) + unc(s.var2));
|
||||
left = SrcJson(list, headThen, s.left2);
|
||||
right = SrcJson(list, headThen, s.right2);
|
||||
const float *m = s.meas;
|
||||
const bool newL = m[0] != lastMeas[0] || m[1] != lastMeas[1];
|
||||
const bool newR = m[2] != lastMeas[2] || m[3] != lastMeas[3];
|
||||
std::memcpy(lastMeas, m, sizeof lastMeas);
|
||||
std::snprintf(extra, sizeof extra, "{\"q\":[%.3g,%.3g],\"m\":[[%.4f,%.4f],[%.4f,%.4f]],\"new\":[%d,%d]}",
|
||||
(m[4] + m[5]) / 2, (m[6] + m[7]) / 2, m[0], m[1], m[2], m[3], int(newL), int(newR));
|
||||
eye = extra;
|
||||
}
|
||||
|
||||
double yaw, pitch;
|
||||
@@ -451,9 +487,9 @@ int main(int argc, char **argv) {
|
||||
yaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
|
||||
pitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
|
||||
std::printf("{\"t\":%.5f,\"age\":%.1f,\"n\":%u,\"head\":{\"yaw\":%.4f,\"pitch\":%.4f,\"hit\":%s},"
|
||||
"\"src\":{\"action\":%s,\"mmap1\":%s,\"mmap2\":%s}}\n",
|
||||
"\"src\":{\"action\":%s,\"mmap1\":%s,\"mmap2\":%s,\"left\":%s,\"right\":%s},\"eye\":%s}\n",
|
||||
s.t, (now - s.t) * 1000, s.n, yaw, pitch, HitJson(list, headNow, 0, 0).c_str(), action.c_str(),
|
||||
m1.c_str(), m2.c_str());
|
||||
m1.c_str(), m2.c_str(), left.c_str(), right.c_str(), eye.c_str());
|
||||
if (std::fflush(stdout) != 0) break; // the reader went away
|
||||
}
|
||||
|
||||
|
||||
+67
-19
@@ -3,13 +3,20 @@
|
||||
|
||||
Runs ft-gaze (in the dev container), and for every eye tracker sample (90 Hz):
|
||||
|
||||
1. drops blinks: both eyes' openness under half its running median (each eye its own).
|
||||
With the default source, mmap set 1, that's all: set 1 is SteamVR's combined gaze,
|
||||
which keeps going when the tracker loses one eye (its two directions stay together).
|
||||
Set 2's combined direction is the mean of the eyes' own, so with one eye lost it's
|
||||
off by half of whatever that eye reads (9 to 14 degrees apart were seen): with set 2,
|
||||
samples with an eye under its floor, or the angle between the eyes jumping more than
|
||||
1.5 degrees from its median, are dropped too;
|
||||
1. drops blinks: both eyes' openness under half its running median (each eye its own),
|
||||
or both lost (the tracker's variance for them, ft-gaze's "unc", over EYE_LOST). With
|
||||
one eye lost or closed, the gaze comes from the other (EyeFallback: that eye's own
|
||||
reading from set 2, plus what it usually reads against the combined gaze, learned
|
||||
while both are seen). SteamVR's combined gaze (set 1) keeps going on one eye too, but
|
||||
holds the lost eye's yaw, so it moves half as far sideways as the eyes do. Before the
|
||||
fallback has learned an eye, set 1 is used as it is; set 2's combined direction is
|
||||
the mean of the eyes' own (off by half of whatever the lost eye reads), so with set 2
|
||||
that sample is dropped, as is one where the angle between the eyes jumps more than
|
||||
1.5 degrees from its median;
|
||||
Looks down past the screens (pitch under KEYBOARD_PITCH, on no Frametop screen: at the
|
||||
keyboard, through the gap by the nose) aren't sent, so the pointer stays where it was
|
||||
instead of following you down; the tracker often loses an eye there (the lids come
|
||||
down), and that isn't counted as a lost eye either;
|
||||
2. smooths it with a fixation lock (the running mean of the current fixation, 1 degree);
|
||||
3. corrects it: the calibration from ft-gazeprobe (calibration.json, reloaded when the
|
||||
probe changes it) plus what the pointer's corrections have taught since (LiveCorrection,
|
||||
@@ -56,7 +63,8 @@ from collections import deque
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from gazecal import DEFAULT_MODEL, MODELS, STATE, Correction, Fixation, LiveCorrection, SteamEyeLog # noqa: E402
|
||||
from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correction, EyeFallback, # noqa: E402
|
||||
Fixation, LiveCorrection, SteamEyeLog)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
HELPER = REPO / "gaze" / "build" / "ft-gaze"
|
||||
@@ -67,6 +75,8 @@ LESSONS = STATE / "pointer-lessons.json"
|
||||
LESSON_LOG = STATE / "pointer-lessons.jsonl"
|
||||
LESSON_MAX = 8.0 # degrees past the correction
|
||||
RETRY = 3.0 # seconds before starting ft-gaze again
|
||||
SETTLE = 0.3 # seconds after an eye is found again before the fallback learns from it
|
||||
KEYBOARD_PITCH = -20.0 # degrees: gaze under this, on no screen, is a look at the keyboard
|
||||
|
||||
|
||||
class PointerLessons(LiveCorrection):
|
||||
@@ -103,7 +113,11 @@ class Service:
|
||||
self.fix = Fixation(radius=1.0)
|
||||
self.opens = (deque(maxlen=90), deque(maxlen=90)) # left, right
|
||||
self.vergence = deque(maxlen=90)
|
||||
self.counts = {"samples": 0, "sent": 0, "blinks": 0, "one_eye": 0, "dropped": 0, "lessons_taken": 0, "refused": 0}
|
||||
self.fallback = EyeFallback()
|
||||
self.lost = [False, False]
|
||||
self.bad_at = [0.0, 0.0] # sample time an eye was last lost or closed
|
||||
self.counts = {"samples": 0, "sent": 0, "blinks": 0, "one_eye": 0, "one_eye_used": 0, "lost_left": 0,
|
||||
"lost_right": 0, "looking_down": 0, "dropped": 0, "lessons_taken": 0, "refused": 0}
|
||||
self.last_sample = 0.0
|
||||
self.last = None
|
||||
|
||||
@@ -261,21 +275,54 @@ class Service:
|
||||
floor = max(0.12, 0.5 * statistics.median(good)) if len(good) >= 30 else 0.12
|
||||
low[k] = o[k] < floor
|
||||
hist.append(o[k])
|
||||
if all(low):
|
||||
unc = m1.get("unc")
|
||||
down = src["hp"] < KEYBOARD_PITCH and not src.get("hit")
|
||||
if unc and len(unc) == 2:
|
||||
for k in (0, 1):
|
||||
self.lost[k] = unc[k] > (EYE_FOUND if self.lost[k] else EYE_LOST)
|
||||
if not down:
|
||||
self.counts["lost_left"] += self.lost[0]
|
||||
self.counts["lost_right"] += self.lost[1]
|
||||
if down:
|
||||
self.counts["looking_down"] += 1
|
||||
for k in (0, 1):
|
||||
self.bad_at[k] = s["t"] # the fallback doesn't learn from these either
|
||||
return
|
||||
bad = [low[k] or self.lost[k] for k in (0, 1)]
|
||||
if all(bad):
|
||||
self.counts["blinks"] += 1
|
||||
return
|
||||
if any(low):
|
||||
hy, hp = src["hy"], src["hp"]
|
||||
eyes = (s["src"].get("mmap2") or {}).get("eyes")
|
||||
for k in (0, 1):
|
||||
if bad[k]:
|
||||
self.bad_at[k] = s["t"]
|
||||
if any(bad):
|
||||
self.counts["one_eye"] += 1
|
||||
if self.source == "mmap2":
|
||||
jump = (lr is not None and len(self.vergence) >= 30
|
||||
and abs(lr - statistics.median(self.vergence)) > 1.5)
|
||||
if lr is not None and not any(low):
|
||||
self.vergence.append(lr)
|
||||
if any(low) or jump:
|
||||
seen = 1 if bad[0] else 0
|
||||
est = self.fallback.get(seen, eyes[seen][0], eyes[seen][1]) if eyes else None
|
||||
if est:
|
||||
hy, hp = est
|
||||
self.counts["one_eye_used"] += 1
|
||||
elif self.source == "mmap2":
|
||||
self.counts["dropped"] += 1
|
||||
return
|
||||
else:
|
||||
if self.source == "mmap2":
|
||||
jump = (lr is not None and len(self.vergence) >= 30
|
||||
and abs(lr - statistics.median(self.vergence)) > 1.5)
|
||||
if lr is not None:
|
||||
self.vergence.append(lr)
|
||||
if jump:
|
||||
self.counts["dropped"] += 1
|
||||
return
|
||||
# Learn only once both have been seen for a moment: the tracker's filter starts
|
||||
# an eye over when it finds it again.
|
||||
if eyes and s["t"] - max(self.bad_at) > SETTLE:
|
||||
for k in (0, 1):
|
||||
self.fallback.update(k, eyes[k][0], eyes[k][1], hy, hp)
|
||||
# The fixation lock works in degrees here (1 degree per "pixel").
|
||||
fy, fp = self.fix(src["hy"], src["hp"], s["t"], 1.0)
|
||||
fy, fp = self.fix(hy, hp, s["t"], 1.0)
|
||||
cy, cp = self.correction(fy, fp)
|
||||
self.last = (fy + cy, fp + cp, fy, fp)
|
||||
try:
|
||||
@@ -327,7 +374,8 @@ class Service:
|
||||
"lessons": len(self.live.samples), "lesson_offset": [round(ly, 3), round(lp, 3)],
|
||||
"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
|
||||
if self.last_sample else None, "headset_on": self.steam.wearing(), "headset_on_since": self.steam.worn(),
|
||||
"last": [round(v, 2) for v in self.last] if self.last else None, **self.counts}
|
||||
"last": [round(v, 2) for v in self.last] if self.last else None,
|
||||
"eyes_lost": self.lost, "fallback_ready": [self.fallback.ready(0), self.fallback.ready(1)], **self.counts}
|
||||
|
||||
def periodic(self):
|
||||
if self.steam.poll() or self.steam.worn() != self.live.wear_time:
|
||||
|
||||
+89
-3
@@ -2,8 +2,8 @@
|
||||
|
||||
The correction models (Correction: the calibration fitted from calibration dots;
|
||||
LiveCorrection: what clicks teach on the fly, on top of it), the smoothing filters, the
|
||||
blink and dropout filter for one look at a spot, and SteamEyeLog, which follows SteamVR's
|
||||
eye tracking log. Angles are head-relative degrees (yaw +left, pitch +up), as ft-gaze
|
||||
blink and dropout filter for one look at a spot, EyeFallback (the gaze from one eye while
|
||||
the tracker has lost the other), and SteamEyeLog, which follows SteamVR's eye tracking log. Angles are head-relative degrees (yaw +left, pitch +up), as ft-gaze
|
||||
reports them.
|
||||
"""
|
||||
|
||||
@@ -391,6 +391,90 @@ class LiveCorrection:
|
||||
return self.cy[0], self.cp[0]
|
||||
|
||||
|
||||
# The tracker's variance for an eye's direction (ft-gaze's "unc"): 0.0005-0.002 while it
|
||||
# sees the eye, 0.015-0.03 once it's lost it, falling back through 0.008-0.002 in the 0.1 s
|
||||
# after it finds it again.
|
||||
EYE_LOST = 0.004
|
||||
EYE_FOUND = 0.0025
|
||||
|
||||
|
||||
class EyeFallback:
|
||||
"""The gaze from one eye, while the tracker has lost the other.
|
||||
|
||||
SteamVR's combined gaze (mmap set 1) keeps going with one eye lost, but badly: it holds
|
||||
the lost eye's yaw where it was and gives it the other eye's pitch, so the gaze moves
|
||||
half as far sideways as the eyes do (seen: the right eye swung 5 degrees, the combined
|
||||
gaze 2.5). Set 2's eyes are each eye's own reading. While both are seen, this learns what
|
||||
each eye reads against the combined gaze (an offset: half the angle between the eyes,
|
||||
plus how differently the tracker reads each), in 10 degree cells of where that eye
|
||||
looks, blended over the four nearest; while one is lost, the other eye plus its offset
|
||||
stands in for the combined gaze. So the rest (fixation lock, calibration, lessons)
|
||||
carries on as if nothing happened.
|
||||
|
||||
On a recording, one eye alone came out 1.1 degrees (median) from both eyes' gaze, 0.8
|
||||
over a tenth of a second of a steady look, and a little more jittery (0.31-0.37 degrees
|
||||
against 0.28). Carrying on the offset from just before a loss did no better: what's
|
||||
left is fast noise, not something particular to that look.
|
||||
|
||||
`update` and `get` take head-relative degrees (yaw, pitch)."""
|
||||
|
||||
CELL = 10.0
|
||||
GLOBAL_RATE = 0.01 # per sample: about a second at 90 Hz
|
||||
CELL_RATE = 0.02 # the least a cell learns per sample, once it has CELL_FULL
|
||||
CELL_FULL = 30 # samples before a cell counts fully
|
||||
READY = 45 # samples of both eyes before an eye can stand in
|
||||
|
||||
def __init__(self):
|
||||
self.glob = [None, None] # per eye: [oy, op]
|
||||
self.seen = [0, 0]
|
||||
self.cells = [{}, {}] # per eye: (i, j) -> [oy, op, n]
|
||||
|
||||
def ready(self, eye):
|
||||
return self.seen[eye] >= self.READY
|
||||
|
||||
def update(self, eye, ey, ep, cy, cp):
|
||||
oy, op = cy - ey, cp - ep
|
||||
g = self.glob[eye]
|
||||
if g is None:
|
||||
self.glob[eye] = [oy, op]
|
||||
else:
|
||||
g[0] += self.GLOBAL_RATE * (oy - g[0])
|
||||
g[1] += self.GLOBAL_RATE * (op - g[1])
|
||||
self.seen[eye] += 1
|
||||
key = (math.floor(ey / self.CELL), math.floor(ep / self.CELL))
|
||||
c = self.cells[eye].setdefault(key, [oy, op, 0])
|
||||
c[2] += 1
|
||||
a = max(1.0 / c[2], self.CELL_RATE)
|
||||
c[0] += a * (oy - c[0])
|
||||
c[1] += a * (op - c[1])
|
||||
|
||||
def offset(self, eye, ey, ep):
|
||||
g = self.glob[eye]
|
||||
if g is None:
|
||||
return None
|
||||
# Bilinear over the four cells whose centres surround the point.
|
||||
fy, fp = ey / self.CELL - 0.5, ep / self.CELL - 0.5
|
||||
i0, j0 = math.floor(fy), math.floor(fp)
|
||||
ty, tp = fy - i0, fp - j0
|
||||
sy = sp = used = 0.0
|
||||
for di, wi in ((0, 1 - ty), (1, ty)):
|
||||
for dj, wj in ((0, 1 - tp), (1, tp)):
|
||||
c = self.cells[eye].get((i0 + di, j0 + dj))
|
||||
if c:
|
||||
w = wi * wj * min(1.0, c[2] / self.CELL_FULL)
|
||||
sy += w * c[0]
|
||||
sp += w * c[1]
|
||||
used += w
|
||||
return sy + (1 - used) * g[0], sp + (1 - used) * g[1]
|
||||
|
||||
def get(self, eye, ey, ep):
|
||||
"""The combined gaze from this eye's reading, or None before it has learned enough."""
|
||||
if not self.ready(eye):
|
||||
return None
|
||||
oy, op = self.offset(eye, ey, ep)
|
||||
return ey + oy, ep + op
|
||||
|
||||
|
||||
class SteamEyeLog:
|
||||
"""Follows SteamVR's eye tracking log (read only) for what moves the raw gaze under a
|
||||
calibration.
|
||||
@@ -494,7 +578,8 @@ def cross_validate(points, mode):
|
||||
|
||||
def steady_samples(samples, vergence_jump=1.5):
|
||||
"""The samples of one look at one spot where the tracker had both eyes: none in a blink
|
||||
(openness under half its median over the samples), and none where the angle between the eyes' directions (`lr`, the
|
||||
(openness under half its median over the samples), none where it had lost an eye (its
|
||||
variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the
|
||||
vergence) is more than `vergence_jump` degrees from its median over the samples. The
|
||||
vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
|
||||
fraction of one far off), so only a jump away from what it was during this look means
|
||||
@@ -513,6 +598,7 @@ def steady_samples(samples, vergence_jump=1.5):
|
||||
|
||||
def vergence(smp):
|
||||
return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
|
||||
opened = [smp for smp in opened if max((smp["src"].get("mmap1") or {}).get("unc") or [0]) <= EYE_LOST]
|
||||
have = [v for v in map(vergence, opened) if v is not None]
|
||||
if len(have) < 5:
|
||||
return opened
|
||||
|
||||
+96
-25
@@ -2,8 +2,8 @@
|
||||
"""ft-gazeprobe: a playground for eye tracking as pointer input on the Frametop desktop.
|
||||
|
||||
Opens fullscreen on one Frametop screen and shows where the headset's eye tracker says
|
||||
you're looking, from the three sources ft-gaze reads (SteamVR's eye tracking action and
|
||||
the two gaze sets in eye-server.mmap). Three modes:
|
||||
you're looking, from the sources ft-gaze reads (SteamVR's eye tracking action, the two
|
||||
gaze sets in eye-server.mmap, and each eye alone). The modes:
|
||||
|
||||
Free look the gaze dot; the trigger calibrates wherever you are looking.
|
||||
Accuracy test look at each target and tap the trigger; measures every source's error
|
||||
@@ -15,6 +15,11 @@ the two gaze sets in eye-server.mmap). Three modes:
|
||||
it; if it's the wrong one, hold, then glance toward the right one or
|
||||
move the mouse, and let go on it. Each click teaches the click
|
||||
corrections (LiveCorrection).
|
||||
Headset fit how well the tracker sees each eye (fitcheck.py): live per eye, whether
|
||||
it's tracked, how open it is, and the tracker's confidence, and maps of
|
||||
where you looked and where each eye got lost, with hints. Enter runs a
|
||||
guided check (dots around the screen, then down, up, left and right),
|
||||
R starts over. Adjust the headset while you watch it.
|
||||
|
||||
Run calibration: the initial calibration, after Apple Vision Pro's eye setup. One dot,
|
||||
then six in a circle, in three rounds that go from a dark to a bright screen (pupil size
|
||||
@@ -22,8 +27,8 @@ changes with brightness, and the tracker's error with it). Look at each highligh
|
||||
and press the trigger. At the end, each source's calibration is fitted from all 21 dots;
|
||||
freeze and look refines it on demand after that.
|
||||
|
||||
Trigger: Enter, Space, or a mouse button. Tab shows and hides the panel, F11 toggles
|
||||
fullscreen, Esc quits.
|
||||
Trigger: Enter, Space, or a mouse button. Tab shows and hides the panel, C collapses it to
|
||||
its title bar (or its arrow button does), F11 toggles fullscreen, Esc quits.
|
||||
|
||||
Freeze and look (the default trigger): the press freezes the dot where the tracker says
|
||||
you're looking. Look at the frozen dot. After a moment to settle, the probe averages where
|
||||
@@ -72,9 +77,13 @@ from gi.repository import Adw, Gdk, Gio, GLib, Gtk # noqa: E402
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
HELPER = REPO / "gaze" / "build" / "ft-gaze"
|
||||
STATE = Path.home() / ".local" / "state" / "frametop" / "gaze"
|
||||
SOURCES = ["action", "mmap1", "mmap2"]
|
||||
SOURCE_NAMES = {"action": "SteamVR action", "mmap1": "mmap set 1", "mmap2": "mmap set 2"}
|
||||
SOURCE_COLORS = {"action": (0.2, 0.8, 1.0), "mmap1": (1.0, 0.6, 0.1), "mmap2": (0.9, 0.3, 0.9)}
|
||||
# left and right: each eye alone (set 2's own reading of that eye), calibrated and tested
|
||||
# like the rest, to see what one eye is worth against both.
|
||||
SOURCES = ["action", "mmap1", "mmap2", "left", "right"]
|
||||
SOURCE_NAMES = {"action": "SteamVR action", "mmap1": "mmap set 1", "mmap2": "mmap set 2", "left": "Left eye",
|
||||
"right": "Right eye"}
|
||||
SOURCE_COLORS = {"action": (0.2, 0.8, 1.0), "mmap1": (1.0, 0.6, 0.1), "mmap2": (0.9, 0.3, 0.9),
|
||||
"left": (0.4, 1.0, 0.6), "right": (1.0, 1.0, 0.4)}
|
||||
TRIGGER_KEYS = {Gdk.KEY_Return, Gdk.KEY_KP_Enter, Gdk.KEY_space}
|
||||
|
||||
|
||||
@@ -82,6 +91,7 @@ TRIGGER_KEYS = {Gdk.KEY_Return, Gdk.KEY_KP_Enter, Gdk.KEY_space}
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from gazecal import (DEFAULT_MODEL, MODELS, Correction, Fixation, LiveCorrection, OneEuro, # noqa: E402
|
||||
SteamEyeLog, cross_validate, deg_from_px, px_from_deg, steady_samples)
|
||||
from fitcheck import FitCheck # noqa: E402
|
||||
|
||||
|
||||
# --- Gaze from ft-gaze ----------------------------------------------------------------
|
||||
@@ -414,6 +424,7 @@ class Probe(Adw.ApplicationWindow):
|
||||
self.pointer = None
|
||||
self.practice = None # click practice: the press being held (see practice_press)
|
||||
self.recent = deque(maxlen=60) # the last samples, for where you looked at a press
|
||||
self.fitcheck = FitCheck() # Headset fit: how well the tracker sees each eye
|
||||
|
||||
self.models = {s: Correction() for s in SOURCES}
|
||||
self.live = {s: LiveCorrection() for s in SOURCES}
|
||||
@@ -471,8 +482,8 @@ class Probe(Adw.ApplicationWindow):
|
||||
|
||||
# --- The context menu (right-click, the Menu key, or Shift+F10) ---
|
||||
|
||||
MODE_KEYS = ["free", "test", "practice", "snap"]
|
||||
MODE_NAMES = ["Free look", "Accuracy test", "Click practice", "Snap practice"]
|
||||
MODE_KEYS = ["free", "test", "practice", "snap", "fit"]
|
||||
MODE_NAMES = ["Free look", "Accuracy test", "Click practice", "Snap practice", "Headset fit"]
|
||||
|
||||
def build_menu(self):
|
||||
def action(name, fn, state=None):
|
||||
@@ -501,6 +512,8 @@ class Probe(Adw.ApplicationWindow):
|
||||
self.mode_action.connect("activate", lambda a, v: self.set_mode(v.get_string()))
|
||||
self.add_action(self.mode_action)
|
||||
self.panel_action = action("panel", self.on_panel_toggle, GLib.Variant("b", True))
|
||||
self.collapse_action = action("collapse", lambda a, v: self.set_collapsed(v.get_boolean()),
|
||||
GLib.Variant("b", False))
|
||||
self.full_action = action("fullscreen", lambda a, v: self.toggle_fullscreen(), GLib.Variant("b", True))
|
||||
action("quit", lambda *_: self.close())
|
||||
|
||||
@@ -527,6 +540,7 @@ class Probe(Adw.ApplicationWindow):
|
||||
menu.append_submenu("Correction model", models)
|
||||
view = Gio.Menu()
|
||||
view.append("Panel", "win.panel")
|
||||
view.append("Collapse panel", "win.collapse")
|
||||
view.append("Fullscreen", "win.fullscreen")
|
||||
view.append("Quit", "win.quit")
|
||||
menu.append_section(None, view)
|
||||
@@ -655,16 +669,29 @@ class Probe(Adw.ApplicationWindow):
|
||||
return s
|
||||
|
||||
def build_panel(self):
|
||||
box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=8, halign=Gtk.Align.START,
|
||||
valign=Gtk.Align.START, margin_start=40, margin_top=40)
|
||||
box.add_css_class("probe-panel")
|
||||
box.set_size_request(460, -1)
|
||||
panel = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, halign=Gtk.Align.START,
|
||||
valign=Gtk.Align.START, margin_start=40, margin_top=40)
|
||||
panel.add_css_class("probe-panel")
|
||||
|
||||
title = Gtk.Label(label="Gaze probe", xalign=0)
|
||||
# The title bar stays when the panel is collapsed, so the gaze dot isn't lost behind it.
|
||||
head = Gtk.Box(spacing=12)
|
||||
title = Gtk.Label(label="Gaze probe", xalign=0, hexpand=True)
|
||||
title.add_css_class("title-2")
|
||||
box.append(title)
|
||||
hint = Gtk.Label(label="Trigger: Enter, Space, or click. Right-click for the menu. Tab hides this panel, Esc quits. "
|
||||
"Click the window once so the keys reach it.", xalign=0, wrap=True)
|
||||
head.append(title)
|
||||
self.w_collapse = Gtk.Button(icon_name="pan-up-symbolic", tooltip_text="Collapse the panel (C)")
|
||||
self.w_collapse.connect("clicked", lambda *_: self.set_collapsed(not self.collapsed))
|
||||
head.append(self.w_collapse)
|
||||
panel.append(head)
|
||||
|
||||
box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=8, margin_top=8)
|
||||
box.set_size_request(460, -1)
|
||||
self.panel_body = box
|
||||
panel.append(box)
|
||||
# Wrapped labels ask for their whole text on one line, which made the panel as wide
|
||||
# as the screen; max_width_chars keeps it to about the grid's width.
|
||||
hint = Gtk.Label(label="Trigger: Enter, Space, or click. Right-click for the menu. Tab hides this panel, "
|
||||
"C collapses it, Esc quits. Click the window once so the keys reach it.",
|
||||
xalign=0, wrap=True, max_width_chars=45)
|
||||
hint.add_css_class("dim-label")
|
||||
box.append(hint)
|
||||
|
||||
@@ -740,10 +767,31 @@ class Probe(Adw.ApplicationWindow):
|
||||
buttons.append(b)
|
||||
box.append(buttons)
|
||||
|
||||
self.w_stats = Gtk.Label(xalign=0, yalign=0, wrap=True, selectable=False)
|
||||
self.w_stats = Gtk.Label(xalign=0, yalign=0, wrap=True, selectable=False, max_width_chars=50)
|
||||
self.w_stats.add_css_class("probe-stats")
|
||||
box.append(self.w_stats)
|
||||
return box
|
||||
return panel
|
||||
|
||||
@property
|
||||
def collapsed(self):
|
||||
return not self.panel_body.get_visible()
|
||||
|
||||
def set_collapsed(self, collapsed):
|
||||
self.panel_body.set_visible(not collapsed)
|
||||
self.w_collapse.set_icon_name("pan-down-symbolic" if collapsed else "pan-up-symbolic")
|
||||
self.w_collapse.set_tooltip_text("Expand the panel (C)" if collapsed else "Collapse the panel (C)")
|
||||
self.collapse_action.set_state(GLib.Variant("b", collapsed))
|
||||
|
||||
def under_panel(self, x0, y0, x1, y1, margin=20):
|
||||
"""Whether a box on the canvas overlaps the panel, collapsed or not."""
|
||||
if not self.panel.get_visible():
|
||||
return False
|
||||
# Collapsed: the title bar. Before the first layout: about the full panel.
|
||||
px, py, pw, ph = (40, 40, 240, 70) if self.collapsed else (40, 40, 520, 960)
|
||||
ok, r = self.panel.compute_bounds(self.area)
|
||||
if ok and r.get_width() > 0:
|
||||
px, py, pw, ph = r.get_x(), r.get_y(), r.get_width(), r.get_height()
|
||||
return x0 < px + pw + margin and px - margin < x1 and y0 < py + ph + margin and py - margin < y1
|
||||
|
||||
def place(self):
|
||||
"""Find the Frametop screens and go fullscreen on the chosen one."""
|
||||
@@ -843,6 +891,9 @@ class Probe(Adw.ApplicationWindow):
|
||||
self.snap = None
|
||||
elif not self.snap:
|
||||
self.snap_layout()
|
||||
if self.mode == "fit" and getattr(self, "last_mode", None) != "fit" and hasattr(self, "collapse_action"):
|
||||
self.set_collapsed(True) # the fit check needs the room
|
||||
self.last_mode = self.mode
|
||||
if hasattr(self, "mode_action"):
|
||||
self.mode_action.set_state(GLib.Variant("s", self.mode))
|
||||
self.update_stats()
|
||||
@@ -860,9 +911,17 @@ class Probe(Adw.ApplicationWindow):
|
||||
if keyval == Gdk.KEY_Tab:
|
||||
self.panel.set_visible(not self.panel.get_visible())
|
||||
return True
|
||||
if keyval in (Gdk.KEY_c, Gdk.KEY_C):
|
||||
self.panel.set_visible(True)
|
||||
self.set_collapsed(not self.collapsed)
|
||||
return True
|
||||
if keyval == Gdk.KEY_F11:
|
||||
self.toggle_fullscreen()
|
||||
return True
|
||||
if keyval in (Gdk.KEY_r, Gdk.KEY_R) and self.mode == "fit":
|
||||
self.fitcheck.reset()
|
||||
self.area.queue_draw()
|
||||
return True
|
||||
if keyval == Gdk.KEY_BackSpace and self.mode == "snap":
|
||||
self.snap_undo()
|
||||
return True
|
||||
@@ -916,6 +975,7 @@ class Probe(Adw.ApplicationWindow):
|
||||
def on_sample(self, s):
|
||||
self.sample = s
|
||||
self.recent.append(s)
|
||||
self.fitcheck.feed(s, time.monotonic())
|
||||
self.rate_count += 1
|
||||
self.last_arrival = time.monotonic()
|
||||
t = s["t"]
|
||||
@@ -1103,6 +1163,10 @@ class Probe(Adw.ApplicationWindow):
|
||||
if self.mode == "practice":
|
||||
self.practice_press()
|
||||
return
|
||||
if self.mode == "fit":
|
||||
self.fitcheck.toggle_guide(time.monotonic())
|
||||
self.area.queue_draw()
|
||||
return
|
||||
if self.action == "freeze":
|
||||
if not self.capture:
|
||||
self.start_capture()
|
||||
@@ -1131,7 +1195,7 @@ class Probe(Adw.ApplicationWindow):
|
||||
return self.cursor
|
||||
|
||||
def on_release(self):
|
||||
if self.mode == "test":
|
||||
if self.mode in ("test", "fit"):
|
||||
return
|
||||
if self.mode == "snap":
|
||||
self.snap_release()
|
||||
@@ -1286,8 +1350,8 @@ class Probe(Adw.ApplicationWindow):
|
||||
last = self.targets[0] if self.targets else None
|
||||
for _ in range(50):
|
||||
x, y = random.uniform(margin, w - margin), random.uniform(margin, h - margin)
|
||||
if self.panel.get_visible() and x < 560 and y < 1000:
|
||||
continue # not under the panel
|
||||
if self.under_panel(x - r, y - r, x + r, y + r):
|
||||
continue
|
||||
if not last or math.hypot(x - last[0], y - last[1]) > min(w, h) * 0.25:
|
||||
break
|
||||
self.targets = [(x, y, r)]
|
||||
@@ -1368,8 +1432,8 @@ class Probe(Adw.ApplicationWindow):
|
||||
corners = [(ox, oy), (ox + gw, oy), (ox, oy + gh), (ox + gw, oy + gh)]
|
||||
if any(math.hypot(x - cx, y - cy) > radius * 1.05 for x, y in corners):
|
||||
continue
|
||||
if self.panel.get_visible() and ox < 560 and oy < 1000:
|
||||
continue # not under the panel
|
||||
if self.under_panel(ox, oy, ox + gw, oy + gh):
|
||||
continue
|
||||
if any(ox < b[2] + margin and b[0] < ox + gw + margin and oy < b[3] + margin and b[1] < oy + gh + margin
|
||||
for b in boxes):
|
||||
continue
|
||||
@@ -2281,9 +2345,13 @@ class Probe(Adw.ApplicationWindow):
|
||||
if self.w_cells.get_active() and r and r[5]:
|
||||
self.degree_grid(cr, w, h, 1 / r[5])
|
||||
|
||||
if self.mode == "fit":
|
||||
self.fitcheck.draw(cr, w, h, self.text, time.monotonic())
|
||||
if self.fitcheck.guide_step(time.monotonic()):
|
||||
GLib.idle_add(self.area.queue_draw)
|
||||
if self.test:
|
||||
self.draw_test(cr)
|
||||
if self.results and not self.test and not self.snap:
|
||||
if self.results and not self.test and not self.snap and self.mode != "fit":
|
||||
self.draw_results(cr)
|
||||
if self.snap:
|
||||
self.draw_snap(cr)
|
||||
@@ -2534,12 +2602,15 @@ def region_errors(targets, source, key="cerr_deg"):
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Eye tracking playground for the Frametop desktop")
|
||||
ap.add_argument("--screen", type=int, default=0, help="Frametop screen to open on (default: where it opens)")
|
||||
ap.add_argument("--mode", choices=Probe.MODE_KEYS, help="start in this mode (fit: Headset fit)")
|
||||
args, rest = ap.parse_known_args()
|
||||
app = Adw.Application(application_id="dev.frametop.GazeProbe", flags=Gio.ApplicationFlags.NON_UNIQUE)
|
||||
windows = []
|
||||
|
||||
def activate(a):
|
||||
win = Probe(a, args.screen)
|
||||
if args.mode:
|
||||
win.set_mode(args.mode)
|
||||
windows.append(win)
|
||||
win.present()
|
||||
|
||||
|
||||
@@ -541,9 +541,12 @@ class Backend(QObject):
|
||||
n = status["samples"] - prev[1]["samples"]
|
||||
status["rate"] = n / (now - prev[0])
|
||||
status["one_eye_share"] = (status["one_eye"] - prev[1]["one_eye"]) / n if n else 0.0
|
||||
for key in ("lost_left", "lost_right"):
|
||||
if key in status and key in prev[1]:
|
||||
status[key + "_share"] = (status[key] - prev[1][key]) / n if n else 0.0
|
||||
elif self._gaze:
|
||||
status.setdefault("rate", self._gaze.get("rate"))
|
||||
status.setdefault("one_eye_share", self._gaze.get("one_eye_share"))
|
||||
for key in ("rate", "one_eye_share", "lost_left_share", "lost_right_share"):
|
||||
status.setdefault(key, self._gaze.get(key))
|
||||
if not prev or now - prev[0] > 0.5:
|
||||
self._gaze_prev = (now, status)
|
||||
self._gaze = status
|
||||
@@ -604,13 +607,21 @@ class Backend(QObject):
|
||||
@Slot()
|
||||
def openGazeProbe(self):
|
||||
"""Calibrate in ft-gazeprobe (a GTK app on the host, fullscreen on a Frametop screen)."""
|
||||
self._open_probe([], "Opening the gaze probe: calibrate there, then close it")
|
||||
|
||||
@Slot()
|
||||
def openHeadsetFit(self):
|
||||
"""The probe's Headset fit mode: how well the tracker sees each eye, as you adjust."""
|
||||
self._open_probe(["--mode", "fit"], "Opening the headset fit check in the gaze probe")
|
||||
|
||||
def _open_probe(self, args, done):
|
||||
runner = ["distrobox-host-exec"] if shutil.which("distrobox-host-exec") else []
|
||||
env = [f"{k}={os.environ[k]}" for k in ("WAYLAND_DISPLAY", "DISPLAY", "XAUTHORITY", "DBUS_SESSION_BUS_ADDRESS")
|
||||
if os.environ.get(k)]
|
||||
try:
|
||||
subprocess.Popen(runner + ["env"] + env + [GAZE_PROBE], stdin=subprocess.DEVNULL,
|
||||
subprocess.Popen(runner + ["env"] + env + [GAZE_PROBE] + args, stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True)
|
||||
self.message.emit("Opening the gaze probe: calibrate there, then close it", False)
|
||||
self.message.emit(done, False)
|
||||
except OSError as e:
|
||||
self.message.emit(f"Couldn't open the gaze probe: {e}", True)
|
||||
|
||||
|
||||
+21
-2
@@ -557,6 +557,12 @@ Kirigami.ApplicationWindow {
|
||||
tooltip: "Open the gaze probe to calibrate (fullscreen on a Frametop screen)"
|
||||
onTriggered: backend.openGazeProbe()
|
||||
},
|
||||
Kirigami.Action {
|
||||
text: "Check headset fit…"
|
||||
icon.name: "view-visible"
|
||||
tooltip: "How well the eye tracker sees each eye, and where it loses one, while you adjust the headset"
|
||||
onTriggered: backend.openHeadsetFit()
|
||||
},
|
||||
Kirigami.Action {
|
||||
text: "Reload calibration"
|
||||
icon.name: "view-refresh"
|
||||
@@ -645,11 +651,24 @@ Kirigami.ApplicationWindow {
|
||||
: Math.round(gpage.status.rate) + " samples/s"
|
||||
+ (gpage.status.one_eye_share > 0.5 ? " · only one eye tracked" : "")
|
||||
color: gpage.status.one_eye_share > 0.5 ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.textColor
|
||||
Controls.ToolTip.text: "Only one eye tracked: reseat the headset or check the lenses. It still works, "
|
||||
+ "probably less precisely."
|
||||
Controls.ToolTip.text: "Only one eye tracked: the gaze comes from the other eye, a little less "
|
||||
+ "precisely. Check headset fit… shows where the tracker loses it."
|
||||
Controls.ToolTip.visible: gpage.status.one_eye_share > 0.5 && ghover.hovered
|
||||
HoverHandler { id: ghover }
|
||||
}
|
||||
Controls.Label {
|
||||
// Share of the last second's samples where the tracker had lost each eye.
|
||||
property real lostL: gpage.status.lost_left_share || 0
|
||||
property real lostR: gpage.status.lost_right_share || 0
|
||||
visible: backend.gazeServiceRunning && gpage.status.lost_left !== undefined
|
||||
Kirigami.FormData.label: "Eyes:"
|
||||
text: lostL < 0.05 && lostR < 0.05 ? "both tracked"
|
||||
: (lostL >= 0.05 ? "left eye lost " + Math.round(lostL * 100) + "%" : "")
|
||||
+ (lostL >= 0.05 && lostR >= 0.05 ? " · " : "")
|
||||
+ (lostR >= 0.05 ? "right eye lost " + Math.round(lostR * 100) + "%" : "")
|
||||
+ ((lostL >= 0.05) !== (lostR >= 0.05) ? " (the other eye stands in)" : "")
|
||||
color: lostL >= 0.05 || lostR >= 0.05 ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.textColor
|
||||
}
|
||||
Controls.Label {
|
||||
visible: backend.gazeServiceRunning
|
||||
Kirigami.FormData.label: "Calibration:"
|
||||
|
||||
Reference in new issue
Block a user