mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-04 22:00:05 +02:00
Merge gaze-calibration into experimental
The gaze checks and calibration in a headset panel (quick, five, full, headset fit; shared-buffer drawing, click-to-capture), keyboard and mouse gaze clicks (Meta+J/K, the mouse's buttons alike, mouse moves only while a button is held, double right/Meta+K tilts a drag), one 55 degree learning limit with a quick check past it, eye presence for "the headset went on", the gaze probe as a development tool, and the relay releasing keys the desktop has down that no keyboard holds. Conflicts with the profiles: the relay keeps known_action/needs_pointer and the gaze defaults (Meta+J/K and the float key); Input Settings keeps the profile actions everywhere and the gaze actions in key combinations only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed
commit
20e8d6254a
15 files changed
+1894
-109
No files matched your search
+1
-1
@@ -91,7 +91,7 @@ A few overlays need special handling:
|
||||
|
||||
Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
|
||||
|
||||
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured.
|
||||
Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry, and each dot takes the gaze when it has held still rather than at a press: what the tracker says doesn't have to be close for the capture to work. See `gaze/README.md` for the service, the calibration, and what was measured.
|
||||
|
||||
Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
|
||||
|
||||
|
||||
+2
-2
@@ -104,7 +104,7 @@ pointer/helper/run.sh status | log | restart
|
||||
pointer/driver/install.sh probe # devices, hand roles, who owns the dashboard pointer
|
||||
```
|
||||
|
||||
The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, and `POINTER_GAZE_MOUSE`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
|
||||
The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
|
||||
|
||||
## Frametop Input Settings
|
||||
|
||||
@@ -116,7 +116,7 @@ A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs
|
||||
- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected.
|
||||
- Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
|
||||
- Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
|
||||
- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Calibrate (opens the gaze probe), Check headset fit (opens the probe's Headset fit mode), Reload calibration, and Forget nudges.
|
||||
- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
|
||||
- Bluetooth lists paired devices and has Apply Bluetooth fixes, which runs `/etc/steamframe/bt-fixups.sh` through `pkexec`. Pair new devices in Steam.
|
||||
|
||||
Device rules are saved in `~/.config/frametop-input.json`. `input-settings/install.sh` installs the menu entry. Its launcher hands podman the real `XDG_RUNTIME_DIR` and user bus and gives the app the session's Wayland socket, because the desktop session runs on a private D-Bus and podman fails on it.
|
||||
|
||||
+9
-4
@@ -5,7 +5,7 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (t
|
||||
- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on.
|
||||
- `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log.
|
||||
- `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
|
||||
- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground. It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
|
||||
- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
|
||||
|
||||
```
|
||||
gaze/build.sh # build ft-gaze
|
||||
@@ -27,8 +27,11 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
|
||||
With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
|
||||
|
||||
Looks down past the screens (under 20 degrees down, on no Frametop screen: a glance at the keyboard) aren't sent, so the pointer stays where it was instead of following you down, and eyes lost there aren't counted. It drops blinks (both eyes closing or lost), smooths with a fixation lock, and sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
|
||||
- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse button or key combination mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on, until a controller is picked up. The dot shows all the time (`POINTER_GAZE_DOT=moving`: only while the mouse moves it, while a press is held, and as a pulse when you click). Gaze mode works with the mouse only; [docs/gaze-controllers.md](../docs/gaze-controllers.md) explains why the controllers can't click at the gaze.
|
||||
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the probe's one-dot check does the same). The helper only sends nudges up to `POINTER_GAZE_NUDGE_MAX` (8 degrees), and right after putting the headset back on our tracker can be further off than that. If so, raise it for a moment, or do the probe's check. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
|
||||
- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse button or key combination mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on, until a controller is picked up. The dot shows all the time (`POINTER_GAZE_DOT=moving`: only while the mouse moves it, while a press is held, and as a pulse when you click). Gaze mode works with the mouse and the keyboard, not the controllers ([docs/gaze-controllers.md](../docs/gaze-controllers.md) explains why).
|
||||
- **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` lets the mouse take the pointer any time, as before.
|
||||
- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, but up to 30 degrees whatever `POINTER_GAZE_NUDGE_MAX` says: a keyboard correction is always meant). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
|
||||
- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`, 55 degrees: half of the 109 the headset shows across) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the probe's one-dot check does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
|
||||
- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
|
||||
- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
|
||||
|
||||
Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.
|
||||
@@ -61,12 +64,14 @@ Ground rules, for anyone changing it:
|
||||
|
||||
## Headset fit
|
||||
|
||||
The probe's Headset fit mode (Check headset fit on the Gaze page, or `ft-gazeprobe --mode fit`) shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it.
|
||||
Check headset fit on the Gaze page opens it in the headset panel: a card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and the hints, live while you adjust the headset. A left click or Meta+J runs the guided check (dots, then looks down, up, left and right), and a right click or Meta+K closes it. The probe's Headset fit mode (`ft-gazeprobe --mode fit`, for development) has the same check with maps: it shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it.
|
||||
|
||||
Losing an eye is usually about where you look, not the tracker. On this Frame the left eye was lost 57 to 64 % of the time looking 30 to 50 degrees down (at the keyboard) and the right eye never; at screen height both were seen over 98 % of the time. Looking down, the lids come down over the eyes. That's harmless, since the gaze service ignores looks down past the screens: they show on the maps, but not in the counts or as a problem.
|
||||
|
||||
## Probe
|
||||
|
||||
The probe is a development tool (in the Gaze page's overflow menu): calibration experiments, accuracy tests, and practice modes. Users calibrate and check in the headset panel instead.
|
||||
|
||||
The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The arrow in the panel's title bar collapses it to just that bar, so the dot and targets behind it stay visible; the collapsed bar stays through tests. The keys do the same (Tab, C, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place.
|
||||
|
||||
- **Run calibration (start here):** the initial calibration, modeled on Apple Vision Pro's eye setup. Face the centre and keep your head still. Look at one dot and press the trigger, then at each of six dots in a circle. That happens in three rounds, and the screen goes dark, then medium, then bright, because pupil size changes with brightness and the tracker's error with it. Each round turns the ring 20 degrees, and the middle round's ring is half the size, so the 21 dots cover the middle, halfway out, and the edge of your view. The ring's size is `Calibration ring` (degrees, 20 by default, less if the window is too small). Error grows toward the edge, and the calibration can only correct as far out as it has seen dots. The current dot is a bright pulsing dot with a point in the middle; finished dots fade to specks, so your eyes don't go back to them. Samples from blinks and from moments when the tracker lost an eye are dropped: openness under half of what it was during that look (not a fixed level, because your lids come down when you look down, and you squint in the bright round), or the angle between the eyes jumping more than 1.5 degrees from its median (that angle depends on how far away you're looking, so only a jump counts). Each dot is measured with medians, so one bad sample can't fail it. A look that lands where the gaze was for another dot of the round is refused as a look at the wrong dot. Mouse clicks don't count during a calibration run or a test: use Enter or Space. If a dot still fails, the message says why and the next try listens longer. After two failures, S (or the menu) skips the dot. Every attempt is logged to `calibration-attempts.jsonl`. At the end it fits every source's calibration from all the dots, replacing what it had learned (quadratic if the model was none). Esc cancels. The run is saved as `calibration-*.json`. With "Test after calibration" on (the default), the accuracy test starts right after, on new spots.
|
||||
|
||||
+9
-2
@@ -1,5 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build ft-gaze in the dev container on the Frame (gaze/build/ft-gaze; it also runs there).
|
||||
# Build ft-gaze and the calibration panel ft-gazepanel in the dev container on the Frame
|
||||
# (gaze/build/; they also run there). The panel draws its text with stb_truetype (public
|
||||
# domain, one header, pinned as in screens/build.sh).
|
||||
# The eye tracking API (IVRInput::GetEyeTrackingDataRelativeToNow) is newer than the header
|
||||
# shipped with SteamVR's samples, so this uses the pinned public header ft-screens fetches.
|
||||
set -euo pipefail
|
||||
@@ -10,4 +12,9 @@ openvr=v2.15.6
|
||||
[ -f build/include/openvr-$openvr ] || { curl -fsSL "https://raw.githubusercontent.com/ValveSoftware/openvr/$openvr/headers/openvr.h" -o build/include/openvr.h && touch build/include/openvr-$openvr; }
|
||||
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include -I../pointer/common \
|
||||
-o build/ft-gaze ft-gaze.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
|
||||
echo "built build/ft-gaze"'
|
||||
stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
|
||||
[ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
|
||||
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
|
||||
-o build/ft-gazepanel panel/ft-gazepanel.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 \
|
||||
$(pkg-config --libs gbm libdrm) -lpthread
|
||||
echo "built build/ft-gaze build/ft-gazepanel"'
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
"""fitcheck: how well the eye tracker sees each eye, for fitting the headset (ft-gazeprobe's
|
||||
Headset fit mode).
|
||||
"""fitcheck: how well the eye tracker sees each eye, for fitting the headset (the headset
|
||||
panel's fit check, gazecheck.py, and ft-gazeprobe's Headset fit mode).
|
||||
|
||||
From each ft-gaze sample it takes, per eye, whether the tracker has that eye (its variance
|
||||
for the eye's direction, "unc", under EYE_LOST; see gazecal), how open the eye is, and the
|
||||
|
||||
+51
-19
@@ -44,9 +44,11 @@ click, it sends "lesson <raw yaw> <raw pitch> <true yaw> <true pitch>": where th
|
||||
was when the mouse took over, and where the pointer was when you clicked (you were looking
|
||||
there). The gap is the tracker's error there. The raw gaze is the one sent, so it also says
|
||||
when that look was (the history of what was sent), and so what each eye read then:
|
||||
- SteamVR: each eye learns its own error, unless the gaze was more than LESSON_MAX degrees
|
||||
- SteamVR: each eye learns its own error, unless the gaze was more than
|
||||
POINTER_GAZE_NUDGE_MAX degrees (frametop.conf, 55 by default: the helper's limit too)
|
||||
past the correction (then it wasn't a nudge onto what you looked at);
|
||||
- our tracker: the look goes to it as a click ("click T YAW PITCH" on @ft_eyes), as the
|
||||
- our tracker: unless it was more than POINTER_GAZE_NUDGE_MAX off, the look goes to it as
|
||||
a click ("click T YAW PITCH" on @ft_eyes), as the
|
||||
probe's clicks do, and it learns how far the headset has moved on your face. That's
|
||||
what it gets wrong, and after the headset was off, the first click resets it;
|
||||
- either way, how far off each eye was (before the lesson taught it anything) goes to the
|
||||
@@ -61,16 +63,27 @@ GAZE_TRACKER=own or the gaze probe asks for it ("eyes SECONDS", a lease the prob
|
||||
reads the eye-camera frames the root service frametop-eyegrab copies (gaze/tracker/install.sh),
|
||||
which copies them only while ft-eyes runs.
|
||||
|
||||
Checks and calibration (gaze/gazecheck.py): a one-dot quick check when the headset goes on or
|
||||
our tracker asks for a click, and the full calibration when gaze mode comes on without one,
|
||||
both in a panel fixed to the headset (gaze/panel/ft-gazepanel, which this service runs too).
|
||||
|
||||
Nothing here writes to SteamVR, its eye tracker, or its files: ft-gaze reads the eye
|
||||
tracker's shared memory read-only.
|
||||
|
||||
Control socket: abstract unix datagram "@ft_gazed":
|
||||
lesson <rhy> <rhp> <thy> <thp> from the pointer helper (see above)
|
||||
recheck <deg> from the pointer helper: a click's correction was past
|
||||
POINTER_GAZE_NUDGE_MAX, so the quick check (gazecheck.py)
|
||||
status reply: one JSON object
|
||||
forget drop what the lessons taught (the calibration stays)
|
||||
reload read calibration.json and the settings again
|
||||
eyes <seconds> keep our own tracker running that much longer (at most 120),
|
||||
whatever GAZE_TRACKER says: the probe's lease. Reply: "ok"
|
||||
quickcal the one-dot check now (the calibration if there's none)
|
||||
calibrate the full calibration in the panel
|
||||
fitcheck the headset fit check in the panel
|
||||
calaccept | calquit from the pointer helper while the panel is up: take this dot
|
||||
now (a left click, Meta+J) | close it (a right click, Meta+K)
|
||||
|
||||
Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever
|
||||
the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever
|
||||
@@ -96,6 +109,7 @@ from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correction, EyeFallback, # noqa: E402
|
||||
EyeWeights, Fixation, LiveCorrection, SteamEyeLog)
|
||||
from gazecheck import Checks # noqa: E402
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
HELPER = REPO / "gaze" / "build" / "ft-gaze"
|
||||
@@ -113,8 +127,7 @@ SOURCES = ("action", "mmap1", "mmap2", "left", "right") # the ones with a calib
|
||||
SIDES = ("left", "right") # ft-gaze's order, and the sources for each eye alone
|
||||
TRACKERS = ("steam", "own")
|
||||
BIASES = ("auto", "left", "right")
|
||||
LESSON_MAX = 8.0 # degrees past the correction
|
||||
OWN_LESSON_MAX = 25.0 # our tracker: after the headset was off, its first clicks can be 10-17 off
|
||||
NUDGE_MAX = 55.0 # degrees: POINTER_GAZE_NUDGE_MAX's default, the largest lesson taken
|
||||
HISTORY = 12.0 # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later)
|
||||
LOOK = 0.3 # seconds of samples before that moment make the look (the probe's fixation)
|
||||
RETRY = 3.0 # seconds before starting ft-gaze again
|
||||
@@ -141,7 +154,7 @@ def log(msg):
|
||||
|
||||
|
||||
def read_settings():
|
||||
"""(tracker, eye bias) from frametop.conf, defaults for anything missing or unknown."""
|
||||
"""(tracker, eye bias, nudge max) from frametop.conf, defaults for anything missing or unknown."""
|
||||
conf = {}
|
||||
try:
|
||||
for line in CONF.read_text().splitlines():
|
||||
@@ -153,7 +166,11 @@ def read_settings():
|
||||
pass
|
||||
tracker = conf.get("GAZE_TRACKER", "steam")
|
||||
bias = conf.get("GAZE_EYE", "auto")
|
||||
return tracker if tracker in TRACKERS else "steam", bias if bias in BIASES else "auto"
|
||||
try:
|
||||
nudge = min(max(float(conf.get("POINTER_GAZE_NUDGE_MAX", NUDGE_MAX)), 1.0), 110.0) # the helper's range
|
||||
except ValueError:
|
||||
nudge = NUDGE_MAX
|
||||
return tracker if tracker in TRACKERS else "steam", bias if bias in BIASES else "auto", nudge
|
||||
|
||||
|
||||
def mtime(path):
|
||||
@@ -168,7 +185,7 @@ class Service:
|
||||
self.override, self.verbose, self.to = source, verbose, to
|
||||
self.source = source or "mmap1" # the older path's source
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
self.tracker, self.bias = read_settings()
|
||||
self.tracker, self.bias, self.nudge_max = read_settings()
|
||||
self.conf_mtime = mtime(CONF)
|
||||
self.models = {name: Correction() for name in SOURCES}
|
||||
self.mode = DEFAULT_MODEL
|
||||
@@ -209,6 +226,7 @@ class Service:
|
||||
self.sel = selectors.DefaultSelector()
|
||||
self.sel.register(self.sock, selectors.EVENT_READ, "control")
|
||||
self.sel.register(self.eyes_sock, selectors.EVENT_READ, "own")
|
||||
self.checks = Checks(self, self.sel)
|
||||
self.proc = None
|
||||
self.buf = b""
|
||||
self.restart_at = 0.0
|
||||
@@ -228,7 +246,7 @@ class Service:
|
||||
|
||||
def load_settings(self):
|
||||
self.conf_mtime = mtime(CONF)
|
||||
tracker, bias = read_settings()
|
||||
tracker, bias, self.nudge_max = read_settings()
|
||||
if (tracker, bias) != (self.tracker, self.bias):
|
||||
log(f"tracker {tracker}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
|
||||
self.tracker, self.bias = tracker, bias
|
||||
@@ -273,6 +291,13 @@ class Service:
|
||||
tmp.replace(LESSONS)
|
||||
self.dirty = False
|
||||
|
||||
def forget_lessons(self):
|
||||
"""Drop what the lessons taught (the calibration stays)."""
|
||||
self.lives = {name: PointerLessons() for name in SOURCES}
|
||||
self.weights = {t: EyeWeights(self.bias) for t in TRACKERS}
|
||||
self.refit()
|
||||
self.save_lessons()
|
||||
|
||||
def refit(self):
|
||||
for name, live in self.lives.items():
|
||||
live.wear_time = self.steam.worn()
|
||||
@@ -304,8 +329,8 @@ class Service:
|
||||
cy, cp = self.correction(self.source, rhy, rhp)
|
||||
left = math.hypot(dy - cy, dp - cp)
|
||||
rec.update(source=self.source, model=self.mode, correction=[cy, cp], lesson_deg=left)
|
||||
if left > LESSON_MAX:
|
||||
rec["refused"] = f"more than {LESSON_MAX} deg past the correction"
|
||||
if left > self.nudge_max:
|
||||
rec["refused"] = f"more than {self.nudge_max:g} deg past the correction"
|
||||
else:
|
||||
self.lives[self.source].add({"time": rec["time"], "hy": rhy, "hp": rhp, "dy": dy, "dp": dp,
|
||||
"wy": 1.0, "wp": 1.0, "how": "pointer"}, self.models[self.source], self.mode)
|
||||
@@ -316,11 +341,10 @@ class Service:
|
||||
weights = self.weights[self.tracker if kind == "own" else "steam"]
|
||||
rec.update(tracker=self.tracker if kind == "own" else "steam", bias=self.bias, lesson_deg=left, look_t=t,
|
||||
eyes=eyes, weights=[round(w, 3) for w in weights.weights()])
|
||||
limit = OWN_LESSON_MAX if kind == "own" else LESSON_MAX
|
||||
if t is None:
|
||||
rec["refused"] = "that gaze isn't in the last few seconds sent"
|
||||
elif left > limit:
|
||||
rec["refused"] = f"more than {limit} deg off"
|
||||
elif left > self.nudge_max:
|
||||
rec["refused"] = f"more than {self.nudge_max:g} deg off"
|
||||
if "refused" in rec:
|
||||
return self.taken(rec)
|
||||
if kind == "own":
|
||||
@@ -505,6 +529,7 @@ class Service:
|
||||
return low
|
||||
|
||||
def on_sample(self, s):
|
||||
self.checks.on_sample(s)
|
||||
kind = self.kind
|
||||
if kind != self.last_kind:
|
||||
log({"own": "our own tracker", "eyes": "SteamVR's eyes, each calibrated",
|
||||
@@ -636,6 +661,7 @@ class Service:
|
||||
if words[:1] == ["lesson"] and len(words) == 5:
|
||||
try:
|
||||
rec = self.lesson(*map(float, words[1:]))
|
||||
self.checks.after_lesson(rec)
|
||||
reply = "refused" if "refused" in rec else f"ok {rec['lesson_deg']:.2f}"
|
||||
log(f"lesson {rec['lesson_deg']:.2f} deg at {rec['raw'][0]:+.1f},{rec['raw'][1]:+.1f}"
|
||||
+ (f", eyes off {', '.join('-' if m is None else f'{m:.2f}' for m in rec['miss'])}"
|
||||
@@ -646,11 +672,10 @@ class Service:
|
||||
elif words[:1] == ["status"]:
|
||||
reply = json.dumps(self.status())
|
||||
elif words[:1] == ["forget"]:
|
||||
self.lives = {name: PointerLessons() for name in SOURCES}
|
||||
self.weights = {t: EyeWeights(self.bias) for t in TRACKERS}
|
||||
self.refit()
|
||||
self.save_lessons()
|
||||
self.forget_lessons()
|
||||
reply = "ok"
|
||||
elif words[:1] and words[0] in ("quickcal", "calibrate", "fitcheck", "calaccept", "calquit", "recheck"):
|
||||
reply = self.checks.command(words)
|
||||
elif words[:1] == ["eyes"] and len(words) == 2:
|
||||
try:
|
||||
secs = min(max(float(words[1]), 0.0), EYES_LEASE_MAX)
|
||||
@@ -713,7 +738,7 @@ class Service:
|
||||
if self.last_sample else None, "headset_on": self.steam.wearing(),
|
||||
"headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None,
|
||||
"eyes_lost": self.lost, "fallback_ready": [self.fallback.ready(0), self.fallback.ready(1)],
|
||||
**self.counts})
|
||||
"checks": self.checks.status(), **self.counts})
|
||||
return st
|
||||
|
||||
def periodic(self):
|
||||
@@ -737,6 +762,7 @@ class Service:
|
||||
self.eyes_sock.sendto(b"status", EYES_SOCKET)
|
||||
except OSError:
|
||||
pass # not up yet: status() says so once the last answer is old
|
||||
self.checks.periodic()
|
||||
if self.dirty:
|
||||
self.save_lessons()
|
||||
|
||||
@@ -747,9 +773,13 @@ class Service:
|
||||
now = time.monotonic()
|
||||
if not self.proc and now >= self.restart_at:
|
||||
self.start_helper()
|
||||
for key, _ in self.sel.select(timeout=0.5):
|
||||
for key, _ in self.sel.select(timeout=0.05 if self.checks.active else 0.5):
|
||||
if key.data == "control":
|
||||
self.on_control()
|
||||
elif key.data == "checks":
|
||||
self.checks.on_readable()
|
||||
elif key.data == "panel" and self.checks.panel_proc:
|
||||
self.checks.read_panel()
|
||||
elif key.data == "own":
|
||||
self.on_own()
|
||||
elif key.data == "eyes" and self.eyes_proc:
|
||||
@@ -758,12 +788,14 @@ class Service:
|
||||
self.read_stdout()
|
||||
elif key.data == "stderr" and self.proc:
|
||||
self.read_stderr()
|
||||
self.checks.tick()
|
||||
if now >= next_periodic:
|
||||
self.periodic()
|
||||
next_periodic = now + 1.0
|
||||
if self.verbose and now >= next_verbose:
|
||||
log(json.dumps(self.status()))
|
||||
next_verbose = now + 5
|
||||
self.checks.stop()
|
||||
self.stop_helper()
|
||||
self.stop_eyes()
|
||||
if self.dirty:
|
||||
|
||||
+10
-2
@@ -544,6 +544,9 @@ class SteamEyeLog:
|
||||
each time the headset goes on ("HMD on"): the eye model starts over then too."""
|
||||
|
||||
PATH = Path.home() / ".local" / "share" / "Steam" / "logs" / "eyetracking.txt"
|
||||
# It writes "HMD on" again every minute or so while on, and flickers off for 0.01-0.3 s:
|
||||
# only an on after an off of BLIP or longer counts.
|
||||
BLIP = 1.5
|
||||
|
||||
def __init__(self):
|
||||
self.pos = 0
|
||||
@@ -595,9 +598,14 @@ class SteamEyeLog:
|
||||
self.starts.append(t)
|
||||
restarted = not first
|
||||
elif "HMD on" in line:
|
||||
self.wears.append(t)
|
||||
off = bool(self.offs) and (not self.wears or self.offs[-1] > self.wears[-1])
|
||||
if off and self.wears and t - self.offs[-1] < self.BLIP:
|
||||
self.offs.pop() # the sensor flickering: it never came off
|
||||
elif off or not self.wears:
|
||||
self.wears.append(t)
|
||||
elif "HMD off" in line:
|
||||
self.offs.append(t)
|
||||
if not self.offs or (self.wears and self.wears[-1] > self.offs[-1]):
|
||||
self.offs.append(t)
|
||||
elif "Accept usercal" in line:
|
||||
self.accepts.append(t)
|
||||
elif "Reject usercal" in line:
|
||||
|
||||
@@ -0,0 +1,739 @@
|
||||
"""gazecheck: the gaze service's checks and calibration, in the panel fixed to the headset
|
||||
(gaze/panel/ft-gazepanel; ft-gazed runs it). Every kind is made of dots shown at head-relative
|
||||
directions: look at each one.
|
||||
|
||||
quick one dot in the middle of your view. It opens when the headset goes on: eyes seen
|
||||
for DON_DELAY after none for AWAY_MIN (SteamVR's tracker's variance for an eye under
|
||||
EYE_LOST). SteamVR's "HMD on" can't say: it repeats every minute or so, and it can
|
||||
stay on for hours with nobody in the headset. It also opens when our own tracker asks
|
||||
for a click (its "reseat": the headset may sit differently on your face now), at most
|
||||
once every QUICK_COOLDOWN, and on "quickcal" (Frametop Input Settings, or a mouse
|
||||
button or key combination mapped to Gaze quick check), and when a click's correction
|
||||
was past POINTER_GAZE_NUDGE_MAX (55 degrees; the helper's "recheck"): the tracker is
|
||||
far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing.
|
||||
five the middle and four around it, when the first FIVE_COUNT lessons after a quick check
|
||||
were all over FIVE_LIMIT degrees off: the quick check didn't fix it.
|
||||
full the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil
|
||||
size, and the tracker's error with it, changes with brightness), each the middle and
|
||||
a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots)
|
||||
RING degrees out, half that in the middle round, turned 20 degrees a round. It opens
|
||||
when gaze mode comes on without a calibration for the tracker in use, and on
|
||||
"calibrate". Frametop's screens hide while it runs. Quitting it while there's still
|
||||
no calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
|
||||
|
||||
fit the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a
|
||||
card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was
|
||||
seen) and hints, from the gaze probe's Headset fit (gaze/fitcheck.py), while you
|
||||
adjust the headset. A left click or Meta+J runs its guided check (dots, then looks
|
||||
down, up, left and right); a right click or Meta+K closes it, as does FIT_TIMEOUT.
|
||||
|
||||
The quick check's dot captures itself: from CHECK_SETTLE after it shows (the eyes getting
|
||||
there), once the gaze has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and
|
||||
capture time). It's the gaze holding still that counts, not where the tracker puts it, so it
|
||||
works however far off the tracker is; a left click or Meta+J (the pointer helper's
|
||||
"calaccept") takes it now. The full calibration's and five's dots wait for that click: you
|
||||
click when you're looking at the dot (the user asked for that: a steady gaze isn't always on
|
||||
the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as
|
||||
it takes, up to CLICK_IDLE. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
|
||||
renewed every second; the helper shows it again by itself when that stops).
|
||||
|
||||
What a capture teaches:
|
||||
our tracker quick and five: a click ("click T YAW PITCH", like a pointer lesson); full:
|
||||
calib-start, a calib-point for each dot, calib-fit (its calibration)
|
||||
SteamVR's quick and five: a lesson for each eye, like the pointer's; full: each source's
|
||||
calibration fitted from the dots (gazecal.Correction, the probe's way), saved
|
||||
to calibration.json, and the lessons start over on top of it
|
||||
Either way, how far off each eye was goes to the eye bias, and each capture to the lesson log.
|
||||
"""
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import selectors
|
||||
import signal
|
||||
import socket
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from fitcheck import MIN_REGION, FitCheck, wrap
|
||||
from gazecal import DEFAULT_MODEL, EYE_LOST, STATE, steady_samples
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
PANEL_PROG = REPO / "gaze" / "build" / "ft-gazepanel"
|
||||
PANEL = "\0ft_gazepanel"
|
||||
POINTER = "\0ft_pointer_helper"
|
||||
SCREENS = "\0ft_screens"
|
||||
EYES = "\0ft_eyes"
|
||||
CONF = Path.home() / ".config" / "frametop.conf"
|
||||
CALIBRATION = STATE / "calibration.json"
|
||||
POINTS = STATE / "points.jsonl" # the probe's record of calibration dots, for its refine
|
||||
CHECK_LOG = STATE / "checks.jsonl"
|
||||
|
||||
CHECK_SETTLE = 0.45
|
||||
CHECK_WINDOW = 0.6
|
||||
CHECK_SPREAD = 1.0 # degrees
|
||||
ACCEPT_SPREAD = 2.5 # degrees: a capture asked for (calaccept) takes this much
|
||||
CLICK_IDLE = 120.0 # seconds a dot of five or full waits for its click; then the check closes
|
||||
QUICK_TIMEOUT = 6.0
|
||||
QUICK_COOLDOWN = 120.0
|
||||
DON_DELAY = 3.0 # seconds of eyes after AWAY_MIN without: the headset went on
|
||||
AWAY_MIN = 3.0
|
||||
EYES_GONE = 2.0 # seconds without eyes that close a check: the headset came off
|
||||
FIVE_LIMIT = 2.0
|
||||
FIVE_COUNT = 3
|
||||
DONE_PAUSE = 0.35 # seconds the filled dot shows before the next
|
||||
RING = 20.0
|
||||
ROUND_BG = (0.03, 0.33, 0.8)
|
||||
ROUND_NAMES = ("dark", "medium", "bright")
|
||||
RING_SCALE = (1.0, 0.5, 1.0)
|
||||
PANEL_RETRY = 10.0
|
||||
FIT_TIMEOUT = 300.0 # seconds the fit check stays up
|
||||
FIT_EVERY = 0.5 # seconds between its cards' updates (each is a new picture for the panel)
|
||||
FIT_HINT_WIDTH = 95 # characters a hint line holds in the panel
|
||||
|
||||
|
||||
def log(msg):
|
||||
print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def check_dots(kind, own):
|
||||
"""(yaw, pitch, round): head-relative degrees, yaw +left, pitch +up."""
|
||||
if kind == "quick":
|
||||
return [(0.0, 0.0, 0)]
|
||||
if kind == "five":
|
||||
return [(0.0, 0.0, 0), (12.0, 0.0, 0), (-12.0, 0.0, 0), (0.0, 9.0, 0), (0.0, -9.0, 0)]
|
||||
out = []
|
||||
n = 8 if own else 6
|
||||
for rnd in range(3):
|
||||
out.append((0.0, 0.0, rnd))
|
||||
r = RING * RING_SCALE[rnd]
|
||||
for i in range(n):
|
||||
a = math.radians(-90 + rnd * 20 + i * 360 / n) # as the probe: x right, y down
|
||||
x, y = r * math.cos(a), r * math.sin(a) * (0.9 if own else 1.0)
|
||||
out.append((-x, -y, rnd))
|
||||
return out
|
||||
|
||||
|
||||
def spread(points):
|
||||
"""The median point and the spread around it (1.4826 x the median distance: a standard
|
||||
deviation that one stray sample can't move far)."""
|
||||
mx = statistics.median(p[0] for p in points)
|
||||
my = statistics.median(p[1] for p in points)
|
||||
return 1.4826 * statistics.median(math.hypot(p[0] - mx, p[1] - my) for p in points), (mx, my)
|
||||
|
||||
|
||||
def write_gaze(on, path=CONF):
|
||||
"""POINTER_GAZE=1|0 in the config, every other line kept."""
|
||||
try:
|
||||
lines = path.read_text().splitlines()
|
||||
except OSError:
|
||||
lines = []
|
||||
value = f"POINTER_GAZE={1 if on else 0}"
|
||||
for i, line in enumerate(lines):
|
||||
if line.split("#", 1)[0].split("=", 1)[0].strip() == "POINTER_GAZE":
|
||||
lines[i] = value
|
||||
break
|
||||
else:
|
||||
lines.append(value)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_text("\n".join(lines) + "\n")
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def ask(addr, command, timeout=1.0):
|
||||
"""A command to a local datagram socket, and its reply ("" without one)."""
|
||||
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC)
|
||||
try:
|
||||
s.bind("")
|
||||
s.settimeout(timeout)
|
||||
s.sendto(command.encode(), addr)
|
||||
return s.recv(4096).decode("utf-8", "replace")
|
||||
except OSError:
|
||||
return ""
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
class Checks:
|
||||
def __init__(self, svc, sel):
|
||||
self.svc = svc
|
||||
self.sel = sel
|
||||
self.out = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
|
||||
self.out.bind("") # the panel's and the helper's replies come back here
|
||||
sel.register(self.out, selectors.EVENT_READ, "checks")
|
||||
self.check = None
|
||||
self.gaze_on = None
|
||||
self.gaze_heard = 0.0
|
||||
self.want_full_until = 0.0 # gaze mode came on before the tracker said whether it's calibrated
|
||||
self.last_quick = 0.0
|
||||
self.sample_at = 0.0 # the tracker last sent anything
|
||||
self.seen_at = 0.0 # eyes last seen (SteamVR's tracker's variance for them, "unc")
|
||||
self.away = True # no eyes for AWAY_MIN: their coming back is the headset going on
|
||||
self.back_since = None
|
||||
self.reseat_seen = False
|
||||
self.after_quick = None
|
||||
self.panel_proc = None
|
||||
self.panel_restart_at = 0.0
|
||||
self.screens_shown = None
|
||||
self.last_progress = 0.0
|
||||
|
||||
@property
|
||||
def active(self):
|
||||
return self.check is not None
|
||||
|
||||
# --- The panel process ---
|
||||
|
||||
def start_panel(self):
|
||||
if not PANEL_PROG.exists():
|
||||
log(f"ft-gazepanel isn't built: run {REPO}/gaze/build.sh")
|
||||
self.panel_restart_at = time.monotonic() + 60
|
||||
return
|
||||
env = dict(os.environ)
|
||||
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
|
||||
distrobox = Path.home() / ".local" / "bin" / "distrobox"
|
||||
self.panel_proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(PANEL_PROG), "--watch-stdin"],
|
||||
env=env, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE, start_new_session=True)
|
||||
os.set_blocking(self.panel_proc.stderr.fileno(), False)
|
||||
self.sel.register(self.panel_proc.stderr, selectors.EVENT_READ, "panel")
|
||||
log("ft-gazepanel started")
|
||||
|
||||
def read_panel(self):
|
||||
try:
|
||||
data = os.read(self.panel_proc.stderr.fileno(), 65536)
|
||||
except BlockingIOError:
|
||||
return
|
||||
if not data:
|
||||
log(f"ft-gazepanel stopped (exit {self.panel_proc.poll()}); again in {PANEL_RETRY:.0f} s")
|
||||
self.stop_panel()
|
||||
self.panel_restart_at = time.monotonic() + PANEL_RETRY
|
||||
if self.check:
|
||||
self.close("the panel stopped")
|
||||
return
|
||||
for line in data.decode("utf-8", "replace").splitlines():
|
||||
if line.strip():
|
||||
log(line)
|
||||
|
||||
def stop_panel(self):
|
||||
if not self.panel_proc:
|
||||
return
|
||||
try:
|
||||
self.sel.unregister(self.panel_proc.stderr)
|
||||
except (KeyError, ValueError):
|
||||
pass
|
||||
if self.panel_proc.stdin and not self.panel_proc.stdin.closed:
|
||||
self.panel_proc.stdin.close()
|
||||
try:
|
||||
self.panel_proc.wait(timeout=2)
|
||||
except subprocess.TimeoutExpired:
|
||||
try:
|
||||
os.killpg(self.panel_proc.pid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
self.panel_proc = None
|
||||
|
||||
def to_panel(self, command):
|
||||
try:
|
||||
self.out.sendto(command.encode(), PANEL)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def to_helper(self, command):
|
||||
try:
|
||||
self.out.sendto(command.encode(), POINTER)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def on_readable(self):
|
||||
"""Replies on our socket: the helper's "ok on|off" to "gaze ?"; the panel's are dropped."""
|
||||
while True:
|
||||
try:
|
||||
data = self.out.recv(4096).decode("utf-8", "replace")
|
||||
except (BlockingIOError, OSError):
|
||||
return
|
||||
if data in ("ok on", "ok off"):
|
||||
on = data == "ok on"
|
||||
was, self.gaze_on, self.gaze_heard = self.gaze_on, on, time.monotonic()
|
||||
if on and was is False:
|
||||
self.on_gaze_on()
|
||||
|
||||
# --- State ---
|
||||
|
||||
def calibrated(self):
|
||||
"""Does the tracker in use have a calibration? None: our tracker hasn't said yet."""
|
||||
svc = self.svc
|
||||
kind = svc.kind
|
||||
if kind == "own":
|
||||
if time.monotonic() - svc.own_at > 5 or not svc.own:
|
||||
return None
|
||||
return bool((svc.own.get("calibration") or {}).get("dots"))
|
||||
if kind == "eyes":
|
||||
return True
|
||||
return svc.models[svc.source].samples > 0
|
||||
|
||||
def eyes_seen(self, within=1.0):
|
||||
return time.monotonic() - self.seen_at < within
|
||||
|
||||
def can_run(self):
|
||||
"""Someone's in the headset and the tracker is sending."""
|
||||
return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2
|
||||
|
||||
def on_gaze_on(self):
|
||||
cal = self.calibrated()
|
||||
if cal is False:
|
||||
self.start("full", "gaze mode came on without a calibration")
|
||||
elif cal is None:
|
||||
self.want_full_until = time.monotonic() + 20
|
||||
|
||||
def auto_quick(self, reason):
|
||||
now = time.monotonic()
|
||||
if self.check or not self.gaze_on or not self.can_run() or self.calibrated() is not True:
|
||||
return
|
||||
if now - self.last_quick < QUICK_COOLDOWN:
|
||||
log(f"quick check skipped ({reason}): one ran {now - self.last_quick:.0f} s ago")
|
||||
return
|
||||
self.start("quick", reason)
|
||||
|
||||
# --- Running a check ---
|
||||
|
||||
def start(self, kind, reason):
|
||||
svc = self.svc
|
||||
if self.check:
|
||||
return "error a check is running"
|
||||
if not self.panel_proc:
|
||||
return "error the panel isn't running (gaze/build.sh builds it)"
|
||||
if kind == "fit":
|
||||
return self.start_fit(reason)
|
||||
if not self.can_run():
|
||||
return "error the headset is off or the tracker isn't sending"
|
||||
own = svc.kind == "own"
|
||||
if kind == "full" and own:
|
||||
reply = ask(EYES, "calib-start", 3.0)
|
||||
if not reply.startswith("ok"):
|
||||
log(f"calibration not started: our tracker says {reply or 'nothing'}")
|
||||
return f"error our tracker: {reply or 'no reply'}"
|
||||
now = time.monotonic()
|
||||
self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
|
||||
"started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
|
||||
"skipped": 0, "captured": 0, "points": {}}
|
||||
log(f"{kind} check: {reason}")
|
||||
if kind == "full":
|
||||
st = ask(SCREENS, "state", 0.5).split()
|
||||
if len(st) >= 3 and st[0] == "ok":
|
||||
self.screens_shown = (st[2] == "0") if st[1] == "always" else (st[2] == "1")
|
||||
ask(SCREENS, "hide", 0.5)
|
||||
self.to_helper("calpanel 1")
|
||||
self.to_panel(f"show {'full' if kind == 'full' else 'quick'}")
|
||||
self.show_dot()
|
||||
if kind == "quick":
|
||||
self.last_quick = now
|
||||
return "ok"
|
||||
|
||||
def start_fit(self, reason):
|
||||
# Eyes lost are what it's there to show, so it needs only the tracker sending.
|
||||
if time.monotonic() - self.sample_at > 2:
|
||||
return "error the headset is off or the eye tracker isn't sending"
|
||||
now = time.monotonic()
|
||||
self.check = {"kind": "fit", "reason": reason, "own": False, "dots": [], "i": 0, "started": now, "shown": now,
|
||||
"run": [], "accept": False, "done_at": None, "tries": 0, "skipped": 0, "captured": 0,
|
||||
"points": {}, "fit": FitCheck(), "drawn": {}, "drawn_at": 0.0, "step": None}
|
||||
log(f"fit check: {reason}")
|
||||
self.to_helper("calpanel 1")
|
||||
self.to_panel("show fit")
|
||||
self.to_panel("title Headset fit: adjust the headset while you watch")
|
||||
self.to_panel("text Left click or Meta+J: guided check · Right click or Meta+K: done")
|
||||
return "ok"
|
||||
|
||||
def fit_tick(self, now):
|
||||
c = self.check
|
||||
fit = c["fit"]
|
||||
# The guided check: its dots at head-relative directions in the panel (the probe's
|
||||
# screen fractions, spread over the middle of the panel), and its looks as the title.
|
||||
step = fit.guide_step(now)
|
||||
key = None if step is None else (step[0], step[1])
|
||||
if key != c["step"]:
|
||||
c["step"] = key
|
||||
if step is None:
|
||||
self.to_panel("dot 0 0 off")
|
||||
self.to_panel("title Headset fit: adjust the headset while you watch")
|
||||
elif step[0] == "dot":
|
||||
fx, fy = step[1]
|
||||
self.to_panel(f"dot {(0.5 - fx) * 32:.2f} {(0.5 - fy) * 24:.2f} look")
|
||||
self.to_panel("title Look at the dot")
|
||||
else:
|
||||
self.to_panel("dot 0 0 off")
|
||||
self.to_panel(f"title {step[1]}")
|
||||
if now - c["drawn_at"] < FIT_EVERY:
|
||||
return
|
||||
c["drawn_at"] = now
|
||||
drawn = c["drawn"]
|
||||
for k in (0, 1):
|
||||
word, (r, g, b) = fit.status(k)
|
||||
sig, seen = fit.signal(k), fit.tracked_share(k, now)
|
||||
cmd = (f"eye {k} {r:.2f} {g:.2f} {b:.2f} {-1 if sig is None else round(sig, 1):g} "
|
||||
f"{-1 if seen is None else round(seen * 20) / 20:g} {word.capitalize()}")
|
||||
if drawn.get(k) != cmd:
|
||||
drawn[k] = cmd
|
||||
self.to_panel(cmd)
|
||||
if fit.have_eye_data and fit.samples < 3 * MIN_REGION:
|
||||
hints = ["Look around slowly: up, down, left and right. Or left click (Meta+J) for a guided check."]
|
||||
else:
|
||||
hints = fit.hints()
|
||||
lines = [line for h in hints for line in wrap(h, FIT_HINT_WIDTH)][:8]
|
||||
cmd = "hints " + "|".join(lines)
|
||||
if drawn.get("hints") != cmd:
|
||||
drawn["hints"] = cmd
|
||||
self.to_panel(cmd)
|
||||
|
||||
def show_dot(self):
|
||||
c = self.check
|
||||
yaw, pitch, rnd = c["dots"][c["i"]]
|
||||
if c["kind"] == "full":
|
||||
self.to_panel(f"bg {ROUND_BG[rnd]}")
|
||||
self.to_panel(f"title Gaze calibration: {ROUND_NAMES[rnd]} round, {rnd + 1} of 3")
|
||||
self.to_panel("text Look at the dot and click (left click or Meta+J). Right click or Meta+K: stop")
|
||||
elif c["kind"] == "five":
|
||||
self.to_panel(f"text Look at the dot and click ({c['i'] + 1} of {len(c['dots'])})")
|
||||
else:
|
||||
self.to_panel("text Look at the dot")
|
||||
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look")
|
||||
self.last_progress = None
|
||||
c["shown"] = time.monotonic()
|
||||
c["run"], c["accept"], c["done_at"] = [], False, None
|
||||
|
||||
def on_sample(self, s):
|
||||
self.sample_at = time.monotonic()
|
||||
unc = (s["src"].get("mmap1") or {}).get("unc")
|
||||
if unc and min(unc) <= EYE_LOST:
|
||||
self.seen_at = time.monotonic()
|
||||
if self.away and self.back_since is None:
|
||||
self.back_since = self.seen_at
|
||||
c = self.check
|
||||
if c and c["kind"] == "fit":
|
||||
c["fit"].feed(s, self.sample_at)
|
||||
return
|
||||
if not c or c["done_at"]:
|
||||
return
|
||||
if c["own"]:
|
||||
src = s["src"].get("own") or {}
|
||||
else:
|
||||
src = s["src"].get("mmap1") or {}
|
||||
if "hy" not in src:
|
||||
per = [s["src"].get(n) or {} for n in ("left", "right")]
|
||||
per = [p for p in per if "hy" in p]
|
||||
src = {"hy": statistics.fmean(p["hy"] for p in per), "hp": statistics.fmean(p["hp"] for p in per)} if per else {}
|
||||
if "hy" not in src:
|
||||
return
|
||||
now = time.monotonic()
|
||||
if now < c["shown"] + CHECK_SETTLE:
|
||||
return
|
||||
g = (src["hy"], src["hp"])
|
||||
run = c["run"]
|
||||
if run:
|
||||
recent = run[-30:]
|
||||
my, mp = statistics.median(p[2] for p in recent), statistics.median(p[3] for p in recent)
|
||||
if math.hypot(g[0] - my, g[1] - mp) > 2.5 * CHECK_SPREAD:
|
||||
run.clear() # the eyes moved on (a saccade, a blink): start over
|
||||
run.append((now, s, g[0], g[1]))
|
||||
window = [p for p in run if p[0] >= now - CHECK_WINDOW]
|
||||
held = now - run[0][0]
|
||||
# The quick check's ring fills in quarters: each step is a new picture for the panel, so
|
||||
# few of them keep it solid. The others wait for the click (see the top): no ring.
|
||||
progress = math.floor(min(1.0, held / CHECK_WINDOW) * 4) / 4
|
||||
if c["kind"] == "quick" and progress != self.last_progress:
|
||||
yaw, pitch, _ = c["dots"][c["i"]]
|
||||
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} capture {progress:.2f}")
|
||||
self.last_progress = progress
|
||||
if c["accept"] and held >= 0.3 and len(window) >= 10:
|
||||
sd, _ = spread([(p[2], p[3]) for p in window])
|
||||
if sd <= ACCEPT_SPREAD:
|
||||
self.capture(window)
|
||||
return
|
||||
if c["kind"] == "quick" and held >= CHECK_WINDOW and len(window) >= 20:
|
||||
sd, _ = spread([(p[2], p[3]) for p in window])
|
||||
if sd <= CHECK_SPREAD:
|
||||
self.capture(window)
|
||||
else:
|
||||
del run[:len(run) // 2] # not steady enough yet: keep trying with the newer half
|
||||
|
||||
def capture(self, window):
|
||||
svc = self.svc
|
||||
c = self.check
|
||||
yaw, pitch, rnd = c["dots"][c["i"]]
|
||||
samples = [p[1] for p in window]
|
||||
rec = {"time": time.time(), "check": c["kind"], "dot": c["i"], "round": rnd, "true": [yaw, pitch],
|
||||
"samples": len(samples), "own": c["own"]}
|
||||
ok = True
|
||||
if c["own"]:
|
||||
eyes = []
|
||||
for k in (0, 1):
|
||||
seen = [smp["src"]["own"]["eyes"][k] for smp in samples
|
||||
if (smp["src"].get("own") or {}).get("eyes") and smp["src"]["own"]["eyes"][k]]
|
||||
eyes.append((statistics.median(e[0] for e in seen), statistics.median(e[1] for e in seen)) if seen else None)
|
||||
miss = [math.hypot(yaw - e[0], pitch - e[1]) if e else None for e in eyes]
|
||||
rec.update(eyes=eyes, miss=miss)
|
||||
t0, t1 = samples[0]["t"], samples[-1]["t"]
|
||||
if c["kind"] == "full":
|
||||
reply = ask(EYES, f"calib-point {t0:.6f} {t1:.6f} {yaw:.4f} {pitch:.4f}", 3.0)
|
||||
rec["reply"] = reply
|
||||
ok = reply.startswith("ok")
|
||||
else:
|
||||
try:
|
||||
svc.eyes_sock.sendto(f"click {t1:.6f} {yaw:.4f} {pitch:.4f}".encode(), EYES)
|
||||
except OSError as e:
|
||||
rec["reply"], ok = f"our tracker isn't running ({e})", False
|
||||
if ok:
|
||||
svc.weights["own"].add(miss)
|
||||
else:
|
||||
steady = steady_samples(samples)
|
||||
reads = {}
|
||||
for name in ("action", "mmap1", "mmap2", "left", "right"):
|
||||
pts = [(smp["src"][name]["hy"], smp["src"][name]["hp"]) for smp in steady
|
||||
if "hy" in (smp["src"].get(name) or {})]
|
||||
if len(pts) >= 15:
|
||||
reads[name] = (statistics.median(p[0] for p in pts), statistics.median(p[1] for p in pts))
|
||||
rec["reads"] = reads
|
||||
main = ("left", "right") if svc.kind == "eyes" else (svc.source,)
|
||||
if not all(n in reads for n in main):
|
||||
ok = False
|
||||
rec["reply"] = f"only {len(steady)} of {len(samples)} samples had both eyes"
|
||||
elif c["kind"] == "full":
|
||||
for name, (hy, hp) in reads.items():
|
||||
c["points"].setdefault(name, []).append((hy, hp, yaw - hy, pitch - hp))
|
||||
try:
|
||||
with open(POINTS, "a") as f:
|
||||
for name, (hy, hp) in reads.items():
|
||||
f.write(json.dumps({"time": time.time(), "source": name, "hy": hy, "hp": hp,
|
||||
"off": [yaw - hy, pitch - hp], "layout": None, "how": "panel"}) + "\n")
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
miss = []
|
||||
for name in main:
|
||||
hy, hp = reads[name]
|
||||
cy, cp = svc.correction(name, hy, hp)
|
||||
miss.append(math.hypot(yaw - hy - cy, pitch - hp - cp))
|
||||
svc.lives[name].add({"time": time.time(), "hy": hy, "hp": hp, "dy": yaw - hy, "dp": pitch - hp,
|
||||
"wy": 1.0, "wp": 1.0, "how": "check"}, svc.models[name], svc.mode)
|
||||
rec["miss"] = miss
|
||||
if svc.kind == "eyes":
|
||||
svc.weights["steam"].add(miss)
|
||||
svc.dirty = True
|
||||
rec["accepted"] = ok
|
||||
self.log_check(rec)
|
||||
if not ok:
|
||||
c["tries"] += 1
|
||||
log(f"{c['kind']} check dot {c['i'] + 1}: not taken ({rec.get('reply', '')})")
|
||||
if c["tries"] >= 2 or c["kind"] != "full":
|
||||
self.skip()
|
||||
else:
|
||||
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
|
||||
c["run"], c["accept"] = [], False
|
||||
c["shown"] = time.monotonic() # settle again, then retry
|
||||
return
|
||||
c["captured"] += 1
|
||||
c["tries"] = 0
|
||||
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} done")
|
||||
c["done_at"] = time.monotonic() + DONE_PAUSE
|
||||
|
||||
def skip(self):
|
||||
c = self.check
|
||||
yaw, pitch, _ = c["dots"][c["i"]]
|
||||
c["skipped"] += 1
|
||||
c["tries"] = 0
|
||||
self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
|
||||
c["done_at"] = time.monotonic() + DONE_PAUSE
|
||||
|
||||
def advance(self):
|
||||
c = self.check
|
||||
c["i"] += 1
|
||||
if c["i"] < len(c["dots"]):
|
||||
self.show_dot()
|
||||
return
|
||||
self.finish()
|
||||
|
||||
def finish(self):
|
||||
svc = self.svc
|
||||
c = self.check
|
||||
kind = c["kind"]
|
||||
if kind in ("quick", "five"):
|
||||
if c["captured"]:
|
||||
log(f"{kind} check done ({c['captured']} of {len(c['dots'])} dots)")
|
||||
self.after_quick = [] if kind == "quick" else None
|
||||
self.close()
|
||||
return
|
||||
n = len(c["dots"])
|
||||
if c["captured"] < n * 2 / 3:
|
||||
log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays")
|
||||
self.to_panel(f"text Calibration failed: only {c['captured']} of {n} dots. Try again from Input Settings")
|
||||
self.to_panel("dot 0 0 off")
|
||||
c["done_at"] = time.monotonic() + 3.0
|
||||
c["closing"] = True
|
||||
return
|
||||
if c["own"]:
|
||||
reply = ask(EYES, "calib-fit", 10.0)
|
||||
log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
|
||||
else:
|
||||
mode = svc.mode if svc.mode != "none" else DEFAULT_MODEL
|
||||
for name, pts in c["points"].items():
|
||||
if name in svc.models and pts:
|
||||
svc.models[name].fit(pts, mode)
|
||||
d = {name: m.to_json() for name, m in svc.models.items()}
|
||||
d["_meta"] = {"calibrated_at": time.time(), "model": mode, "how": "panel"}
|
||||
d["_live"] = {}
|
||||
tmp = CALIBRATION.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(d, indent=1))
|
||||
tmp.replace(CALIBRATION)
|
||||
svc.forget_lessons() # a new calibration: the pointer's lessons start over on top of it
|
||||
svc.load_calibration()
|
||||
svc.refit()
|
||||
log(f"calibration ({c['captured']} of {n} dots): {mode}, saved")
|
||||
self.close()
|
||||
|
||||
def close(self, why=None):
|
||||
if not self.check:
|
||||
return
|
||||
if why:
|
||||
log(f"{self.check['kind']} check closed: {why}")
|
||||
self.to_panel("hide")
|
||||
self.to_helper("calpanel 0")
|
||||
if self.check["kind"] == "full" and self.screens_shown:
|
||||
ask(SCREENS, "show", 0.5)
|
||||
self.screens_shown = None
|
||||
self.check = None
|
||||
|
||||
def quit(self):
|
||||
c = self.check
|
||||
if not c:
|
||||
return
|
||||
self.close("quit")
|
||||
if c["kind"] == "full" and self.calibrated() is False:
|
||||
# No calibration still: gaze mode can't work, so it goes off until it's turned on again.
|
||||
try:
|
||||
write_gaze(False)
|
||||
except OSError as e:
|
||||
log(f"can't write {CONF}: {e}")
|
||||
self.to_helper("gaze off")
|
||||
self.gaze_on = False
|
||||
log("gaze mode off: no calibration")
|
||||
|
||||
def log_check(self, rec):
|
||||
try:
|
||||
with open(CHECK_LOG, "a") as f:
|
||||
f.write(json.dumps(rec) + "\n")
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# --- From the service ---
|
||||
|
||||
def command(self, words):
|
||||
"""quickcal, calibrate, calaccept, calquit -> a reply."""
|
||||
cmd = words[0]
|
||||
if cmd == "quickcal":
|
||||
return self.start("full" if self.calibrated() is False else "quick", "asked for")
|
||||
if cmd == "calibrate":
|
||||
return self.start("full", "asked for")
|
||||
if cmd == "fitcheck":
|
||||
return self.start("fit", "asked for")
|
||||
if cmd == "calaccept":
|
||||
if self.check and self.check["kind"] == "fit":
|
||||
self.check["fit"].toggle_guide(time.monotonic())
|
||||
elif self.check:
|
||||
self.check["accept"] = True
|
||||
return "ok"
|
||||
if cmd == "calquit":
|
||||
self.quit()
|
||||
return "ok"
|
||||
if cmd == "recheck" and len(words) == 2:
|
||||
self.auto_quick(f"a click was corrected {words[1]} deg")
|
||||
return "ok"
|
||||
return "error unknown command"
|
||||
|
||||
def after_lesson(self, rec):
|
||||
if self.after_quick is None or "refused" in rec:
|
||||
return
|
||||
self.after_quick.append(rec.get("lesson_deg", 0.0))
|
||||
if len(self.after_quick) < FIVE_COUNT:
|
||||
return
|
||||
off = self.after_quick
|
||||
self.after_quick = None
|
||||
if all(d > FIVE_LIMIT for d in off):
|
||||
log(f"the {FIVE_COUNT} lessons after the quick check were {', '.join(f'{d:.1f}' for d in off)} deg off")
|
||||
if self.gaze_on and self.can_run() and not self.check:
|
||||
self.start("five", "the quick check didn't fix it")
|
||||
|
||||
def tick(self):
|
||||
c = self.check
|
||||
if not c:
|
||||
return
|
||||
now = time.monotonic()
|
||||
if c["kind"] == "fit":
|
||||
# Not closed when the eyes go: adjusting the headset loses them.
|
||||
if now - c["started"] > FIT_TIMEOUT:
|
||||
self.close("timed out")
|
||||
else:
|
||||
self.fit_tick(now)
|
||||
return
|
||||
if c["done_at"] and now >= c["done_at"]:
|
||||
if c.get("closing"):
|
||||
self.close()
|
||||
else:
|
||||
self.advance()
|
||||
return
|
||||
if not self.eyes_seen(EYES_GONE):
|
||||
self.close("the headset came off")
|
||||
return
|
||||
if c["done_at"]:
|
||||
return
|
||||
if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
|
||||
self.close("ignored")
|
||||
elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
|
||||
self.close(f"no click on dot {c['i'] + 1} for {CLICK_IDLE:.0f} s")
|
||||
|
||||
def periodic(self):
|
||||
svc = self.svc
|
||||
now = time.monotonic()
|
||||
if not self.panel_proc and now >= self.panel_restart_at:
|
||||
self.start_panel()
|
||||
self.to_helper("gaze ?")
|
||||
if now - self.gaze_heard > 5:
|
||||
self.gaze_on = None # the helper isn't answering
|
||||
if self.check:
|
||||
self.to_helper("calpanel 1")
|
||||
if self.want_full_until:
|
||||
cal = self.calibrated()
|
||||
if cal is not None or now > self.want_full_until:
|
||||
self.want_full_until = 0.0
|
||||
if cal is False and self.gaze_on:
|
||||
self.start("full", "gaze mode came on without a calibration")
|
||||
if not self.eyes_seen(AWAY_MIN):
|
||||
self.away, self.back_since = True, None
|
||||
elif self.back_since is not None and not self.eyes_seen():
|
||||
self.back_since = None # gone again before DON_DELAY
|
||||
elif self.back_since is not None and now - self.back_since >= DON_DELAY:
|
||||
self.away, self.back_since = False, None
|
||||
self.auto_quick("the headset went on")
|
||||
if svc.kind == "own" and now - svc.own_at < 5:
|
||||
reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values())
|
||||
if not reseat:
|
||||
self.reseat_seen = False
|
||||
elif not self.reseat_seen and self.can_run():
|
||||
self.reseat_seen = True
|
||||
self.auto_quick("our tracker asked for a click")
|
||||
|
||||
def status(self):
|
||||
c = self.check
|
||||
st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
|
||||
"panel": self.panel_proc is not None,
|
||||
"last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None}
|
||||
if c:
|
||||
st["check"] = {"kind": c["kind"], "dot": c["i"] + 1, "dots": len(c["dots"]), "captured": c["captured"],
|
||||
"skipped": c["skipped"], "reason": c["reason"]}
|
||||
return st
|
||||
|
||||
def stop(self):
|
||||
self.close("the gaze service is stopping")
|
||||
self.stop_panel()
|
||||
@@ -0,0 +1,526 @@
|
||||
// ft-gazepanel: the gaze calibration panel (docs/design.md, gaze/README.md). A SteamVR overlay
|
||||
// fixed to the headset, so wherever you turn your head it stays in the same place in your
|
||||
// view: a dot drawn at a head-relative direction is exactly that direction from the headset,
|
||||
// which is what the gaze service needs to know where you were looking. It's drawn on the CPU
|
||||
// and takes no input: the gaze service (gaze/ft-gazed) drives it, and the pointer helper
|
||||
// passes it your presses (calaccept, calquit).
|
||||
//
|
||||
// The panel sits POINTER-like at --distance (1.5 m, about where Frametop's screens are, so
|
||||
// the eyes converge as they do in use). "quick" is a small square, QUICK_DEG across, for the
|
||||
// one-dot check; "full" is FULL_DEG across (4:3), with a solid background whose brightness the
|
||||
// service sets per round (pupil size changes with it, and the tracker's error with it); "fit"
|
||||
// is FIT_DEG across (4:3), see-through like quick, for the headset fit check: a card per eye
|
||||
// (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and hints.
|
||||
//
|
||||
// Control socket: abstract unix datagram "@ft_gazepanel" (--socket NAME); a sender with an
|
||||
// address gets "ok" or "error ...":
|
||||
// show quick|full|fit the panel, empty, in front of you
|
||||
// hide
|
||||
// bg <0..1> the background's brightness (full)
|
||||
// dot <yaw> <pitch> <state> [<progress 0..1>]
|
||||
// the dot, head-relative degrees (yaw +left, pitch +up); state:
|
||||
// look, capture (a ring filling to progress), done,
|
||||
// fail, off
|
||||
// title <text> / text <text> a line at the top / at the bottom (empty to clear)
|
||||
// eye <0|1> <r> <g> <b> <signal 0..1|-1> <seen 0..1|-1> <word>
|
||||
// fit: an eye's card (0 left): its state in that colour, the
|
||||
// tracker's signal, the share of the last 10 s it was seen
|
||||
// hints <line>|<line>|... fit: lines under the cards (empty to clear)
|
||||
// ping
|
||||
//
|
||||
// Each picture goes into the next of three shared buffers (linear DMA-BUFs SteamVR imported
|
||||
// once, the size of the biggest panel; the texture bounds show the part in use), and the panel
|
||||
// switches to it, as screens/keyboard.cpp does. SetOverlayRaw, which uploads a new texture each
|
||||
// time, flickered on every change of the full calibration's 1024x768 picture, and in a live
|
||||
// test the headset kept showing an old picture after the panel had drawn new ones (2026-10-01).
|
||||
// It's only the fallback. A "show" makes the panel visible once its first picture is in.
|
||||
//
|
||||
// Options: --watch-stdin (quit when stdin closes: the service runs it), --socket NAME,
|
||||
// --distance METRES. Runs in the dev container (gaze/build.sh builds it into gaze/build).
|
||||
#include <openvr.h>
|
||||
|
||||
#define STB_TRUETYPE_IMPLEMENTATION
|
||||
#include "stb_truetype.h"
|
||||
|
||||
#include <drm_fourcc.h>
|
||||
#include <fcntl.h>
|
||||
#include <gbm.h>
|
||||
#include <sys/socket.h>
|
||||
#include <sys/un.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
#include <csignal>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
namespace {
|
||||
|
||||
using Clock = std::chrono::steady_clock;
|
||||
constexpr double kQuickDeg = 16; // QUICK_DEG: the one-dot check's square
|
||||
constexpr double kFullDeg = 64; // FULL_DEG: the full calibration's width (4:3)
|
||||
constexpr double kFitDeg = 40; // FIT_DEG: the headset fit check's width (4:3)
|
||||
constexpr int kQuickPx = 320, kFullW = 1024, kFullH = 768, kFitW = 800, kFitH = 600;
|
||||
std::atomic<bool> g_stop{false};
|
||||
|
||||
// ---------------------------------------------------------------- text (as screens/keyboard.cpp)
|
||||
|
||||
stbtt_fontinfo g_font;
|
||||
std::vector<unsigned char> g_fontData;
|
||||
bool g_fontOk = false;
|
||||
constexpr int kMaxW = kFullW, kMaxH = kFullH; // the buffers' size: the biggest panel
|
||||
struct Glyph {
|
||||
std::vector<unsigned char> bitmap;
|
||||
int w = 0, h = 0, xoff = 0, yoff = 0, advance = 0;
|
||||
};
|
||||
std::map<std::pair<uint32_t, int>, Glyph> g_glyphs;
|
||||
|
||||
void LoadFont() {
|
||||
std::string path;
|
||||
if (FILE *p = popen("fc-match -f '%{file}' 'Noto Sans' 2>/dev/null", "r")) {
|
||||
char buf[512];
|
||||
if (std::fgets(buf, sizeof buf, p)) path = buf;
|
||||
pclose(p);
|
||||
}
|
||||
if (path.empty()) path = "/usr/share/fonts/google-noto-vf/NotoSans[wght].ttf";
|
||||
if (FILE *f = std::fopen(path.c_str(), "rb")) {
|
||||
std::fseek(f, 0, SEEK_END);
|
||||
g_fontData.resize(size_t(std::ftell(f)));
|
||||
std::fseek(f, 0, SEEK_SET);
|
||||
g_fontOk = std::fread(g_fontData.data(), 1, g_fontData.size(), f) == g_fontData.size() &&
|
||||
stbtt_InitFont(&g_font, g_fontData.data(), stbtt_GetFontOffsetForIndex(g_fontData.data(), 0));
|
||||
std::fclose(f);
|
||||
}
|
||||
if (!g_fontOk) std::fprintf(stderr, "ft-gazepanel: no font (%s); no text\n", path.c_str());
|
||||
}
|
||||
|
||||
const Glyph &GetGlyph(uint32_t cp, int size) {
|
||||
auto [it, fresh] = g_glyphs.try_emplace({cp, size});
|
||||
Glyph &g = it->second;
|
||||
if (fresh) {
|
||||
const float scale = stbtt_ScaleForPixelHeight(&g_font, float(size));
|
||||
unsigned char *b = stbtt_GetCodepointBitmap(&g_font, 0, scale, int(cp), &g.w, &g.h, &g.xoff, &g.yoff);
|
||||
if (b) g.bitmap.assign(b, b + size_t(g.w) * g.h), stbtt_FreeBitmap(b, nullptr);
|
||||
int adv, lsb;
|
||||
stbtt_GetCodepointHMetrics(&g_font, int(cp), &adv, &lsb);
|
||||
g.advance = int(std::lround(adv * scale));
|
||||
}
|
||||
return g;
|
||||
}
|
||||
|
||||
std::vector<uint32_t> Codepoints(const std::string &s) {
|
||||
std::vector<uint32_t> out;
|
||||
for (const unsigned char *p = (const unsigned char *)s.c_str(); *p;) {
|
||||
uint32_t c = *p++;
|
||||
int more = c >= 0xF0 ? 3 : c >= 0xE0 ? 2 : c >= 0xC0 ? 1 : 0;
|
||||
if (more) c &= 0x3Fu >> more;
|
||||
for (; more && (*p & 0xC0) == 0x80; --more) c = c << 6 | (*p++ & 0x3F);
|
||||
out.push_back(c);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- the picture
|
||||
|
||||
struct EyeCard {
|
||||
std::string word = "no data";
|
||||
double r = 0.6, g = 0.6, b = 0.6, signal = -1, seen = -1;
|
||||
};
|
||||
|
||||
struct Panel {
|
||||
bool full = false, fit = false;
|
||||
EyeCard eyes[2];
|
||||
std::vector<std::string> hints;
|
||||
int w = kQuickPx, h = kQuickPx;
|
||||
double wDeg = kQuickDeg; // across
|
||||
std::vector<uint8_t> px;
|
||||
double bg = 0.05;
|
||||
std::string title, text;
|
||||
bool dotOn = false;
|
||||
double dotYaw = 0, dotPitch = 0, progress = 0;
|
||||
std::string state = "off";
|
||||
};
|
||||
|
||||
void Blend(Panel &p, int x, int y, double r, double g, double b, double a) {
|
||||
if (x < 0 || y < 0 || x >= p.w || y >= p.h || a <= 0) return;
|
||||
uint8_t *q = &p.px[(size_t(y) * p.w + x) * 4];
|
||||
a = std::min(a, 1.0);
|
||||
q[0] = uint8_t(std::lround(q[0] + (r * 255 - q[0]) * a));
|
||||
q[1] = uint8_t(std::lround(q[1] + (g * 255 - q[1]) * a));
|
||||
q[2] = uint8_t(std::lround(q[2] + (b * 255 - q[2]) * a));
|
||||
q[3] = uint8_t(std::lround(q[3] + (255 - q[3]) * a));
|
||||
}
|
||||
|
||||
// A filled disc, or (inner > 0) a ring from inner to outer radius, anti-aliased; with sweep < 1,
|
||||
// only that share of the ring, clockwise from the top.
|
||||
void Disc(Panel &p, double cx, double cy, double outer, double inner, double r, double g, double b, double a,
|
||||
double sweep = 1) {
|
||||
const int x0 = int(cx - outer - 1), x1 = int(cx + outer + 1), y0 = int(cy - outer - 1), y1 = int(cy + outer + 1);
|
||||
for (int y = y0; y <= y1; ++y)
|
||||
for (int x = x0; x <= x1; ++x) {
|
||||
const double dx = x + 0.5 - cx, dy = y + 0.5 - cy, d = std::hypot(dx, dy);
|
||||
double cover = std::clamp(outer - d + 0.5, 0.0, 1.0);
|
||||
if (inner > 0) cover = std::min(cover, std::clamp(d - inner + 0.5, 0.0, 1.0));
|
||||
if (sweep < 1) {
|
||||
double ang = std::atan2(dx, -dy) / (2 * M_PI); // 0 at the top, clockwise
|
||||
if (ang < 0) ang += 1;
|
||||
if (ang > sweep) continue;
|
||||
}
|
||||
Blend(p, x, y, r, g, b, a * cover);
|
||||
}
|
||||
}
|
||||
|
||||
void Rect(Panel &p, int x0, int y0, int x1, int y1, double r, double g, double b, double a) {
|
||||
for (int y = std::max(y0, 0); y < std::min(y1, p.h); ++y)
|
||||
for (int x = std::max(x0, 0); x < std::min(x1, p.w); ++x) Blend(p, x, y, r, g, b, a);
|
||||
}
|
||||
|
||||
// Text centred on (x, cy), or starting at x (left).
|
||||
void Text(Panel &p, const std::string &s, int size, int x, int cy, double r, double g, double b, bool left = false) {
|
||||
if (!g_fontOk || s.empty()) return;
|
||||
const auto cps = Codepoints(s);
|
||||
int width = 0;
|
||||
for (uint32_t cp : cps) width += GetGlyph(cp, size).advance;
|
||||
int ascent, descent, gap;
|
||||
stbtt_GetFontVMetrics(&g_font, &ascent, &descent, &gap);
|
||||
const float scale = stbtt_ScaleForPixelHeight(&g_font, float(size));
|
||||
if (!left) x -= width / 2;
|
||||
const int baseline = cy + int(std::lround((ascent + descent) * scale / 2));
|
||||
for (uint32_t cp : cps) {
|
||||
const Glyph &gl = GetGlyph(cp, size);
|
||||
for (int gy = 0; gy < gl.h; ++gy)
|
||||
for (int gx = 0; gx < gl.w; ++gx)
|
||||
Blend(p, x + gl.xoff + gx, baseline + gl.yoff + gy, r, g, b, gl.bitmap[size_t(gy) * gl.w + gx] / 255.0);
|
||||
x += gl.advance;
|
||||
}
|
||||
}
|
||||
|
||||
void Text(Panel &p, const std::string &s, int size, int x, int cy, double lum, bool left = false) {
|
||||
Text(p, s, size, x, cy, lum, lum, lum, left);
|
||||
}
|
||||
|
||||
// The headset fit check (see the top): a card per eye, then the hints.
|
||||
void DrawFit(Panel &p, double pxPerDeg, int textSize, double faint) {
|
||||
const int margin = int(p.w * 0.06), cw = int(p.w * 0.41), ch = int(p.h * 0.32), top = int(p.h * 0.12);
|
||||
const int pad = int(pxPerDeg * 0.9), big = int(pxPerDeg * 1.6), small = int(pxPerDeg * 0.8);
|
||||
for (int k = 0; k < 2; ++k) {
|
||||
const EyeCard &e = p.eyes[k];
|
||||
const int x0 = k == 0 ? margin : p.w - margin - cw;
|
||||
Rect(p, x0, top, x0 + cw, top + ch, 1, 1, 1, 0.07);
|
||||
Text(p, k ? "Right eye" : "Left eye", textSize, x0 + pad, top + pad + textSize / 2, faint, true);
|
||||
Text(p, e.word, big, x0 + pad, top + int(ch * 0.40), e.r, e.g, e.b, true);
|
||||
// The tracker's signal: a bar, red to green.
|
||||
const int by = top + int(ch * 0.62), bh = std::max(4, int(pxPerDeg * 0.35)), bw = cw - 2 * pad;
|
||||
Text(p, "Signal", small, x0 + pad, by - small, faint, true);
|
||||
Rect(p, x0 + pad, by, x0 + pad + bw, by + bh, 1, 1, 1, 0.15);
|
||||
if (e.signal >= 0) {
|
||||
const double v = std::clamp(e.signal, 0.0, 1.0);
|
||||
const double r = v < 0.5 ? 1.0 : 1.0 - 1.3 * (v - 0.5), g = v < 0.5 ? 0.3 + 0.9 * v : 0.75 + 0.5 * (v - 0.5);
|
||||
Rect(p, x0 + pad, by, x0 + pad + int(bw * v), by + bh, r, g, 0.35, 0.95);
|
||||
}
|
||||
char seen[64] = "Seen: not yet";
|
||||
if (e.seen >= 0) std::snprintf(seen, sizeof seen, "Seen %d%% of the last 10 s", int(std::lround(e.seen * 100)));
|
||||
Text(p, seen, small, x0 + pad, top + ch - pad, faint, true);
|
||||
}
|
||||
int y = top + ch + pad * 2;
|
||||
for (const std::string &line : p.hints) {
|
||||
Text(p, line, textSize, margin, y, faint, true);
|
||||
y += int(textSize * 1.4);
|
||||
}
|
||||
}
|
||||
|
||||
// Head-relative direction -> panel pixel: the panel is a plane `d` in front of the headset.
|
||||
void ToPixel(const Panel &p, double yaw, double pitch, double &x, double &y) {
|
||||
const double yr = yaw * M_PI / 180, pr = pitch * M_PI / 180;
|
||||
const double half = std::tan(p.wDeg * M_PI / 360); // half the width, per unit of distance
|
||||
const double X = -std::tan(yr), Y = std::tan(pr) / std::cos(yr);
|
||||
x = (X / (2 * half) + 0.5) * p.w;
|
||||
y = (0.5 - Y / (2 * half) * p.w / p.h) * p.h;
|
||||
}
|
||||
|
||||
void Draw(Panel &p) {
|
||||
p.px.assign(size_t(p.w) * p.h * 4, 0);
|
||||
const double pxPerDeg = p.w / p.wDeg;
|
||||
if (p.full) {
|
||||
for (size_t i = 0; i < p.px.size(); i += 4)
|
||||
p.px[i] = p.px[i + 1] = p.px[i + 2] = uint8_t(std::lround(p.bg * 255)), p.px[i + 3] = 255;
|
||||
} else {
|
||||
// The quick check and the fit check: a dim rounded panel, see-through, so it's clear of what's behind.
|
||||
const double r = std::min(p.w, p.h) * 0.12;
|
||||
for (int y = 0; y < p.h; ++y)
|
||||
for (int x = 0; x < p.w; ++x) {
|
||||
const double dx = std::max({r - x - 0.5, x + 0.5 - (p.w - r), 0.0});
|
||||
const double dy = std::max({r - y - 0.5, y + 0.5 - (p.h - r), 0.0});
|
||||
const double cover = std::clamp(r - std::hypot(dx, dy) + 0.5, 0.0, 1.0);
|
||||
Blend(p, x, y, 0.06, 0.06, 0.07, 0.82 * cover);
|
||||
}
|
||||
}
|
||||
const bool light = p.full && p.bg > 0.5; // a dark dot on the bright round
|
||||
const double ink = light ? 0.0 : 1.0, faint = light ? 0.2 : 0.75;
|
||||
const int titleSize = int(pxPerDeg * (p.full ? 1.5 : 1.1)), textSize = int(pxPerDeg * (p.full ? 1.2 : 0.9));
|
||||
Text(p, p.title, titleSize, p.w / 2, int(titleSize * 1.2), faint);
|
||||
Text(p, p.text, textSize, p.w / 2, p.h - int(textSize * 1.3), faint);
|
||||
if (p.fit) DrawFit(p, pxPerDeg, textSize, faint);
|
||||
if (!p.dotOn || p.state == "off") return;
|
||||
double x, y;
|
||||
ToPixel(p, p.dotYaw, p.dotPitch, x, y);
|
||||
const double core = 0.22 * pxPerDeg;
|
||||
if (p.state == "look") {
|
||||
// Still, so the panel looks solid and is drawn again only when something changes.
|
||||
const double rr = 0.75 * pxPerDeg;
|
||||
Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, ink, ink, ink, 0.6);
|
||||
Disc(p, x, y, core, 0, ink, ink, ink, 1);
|
||||
} else if (p.state == "capture") {
|
||||
const double rr = 0.75 * pxPerDeg;
|
||||
Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, ink, ink, ink, 0.25);
|
||||
Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, 0.3, 0.85, 1.0, 1, std::clamp(p.progress, 0.0, 1.0));
|
||||
Disc(p, x, y, core, 0, ink, ink, ink, 1);
|
||||
} else if (p.state == "done") {
|
||||
Disc(p, x, y, 0.75 * pxPerDeg, 0, 0.25, 0.85, 0.4, 0.9);
|
||||
Disc(p, x, y, core, 0, 1, 1, 1, 1);
|
||||
} else if (p.state == "fail") {
|
||||
Disc(p, x, y, 0.75 * pxPerDeg, 0.6 * pxPerDeg, 0.95, 0.35, 0.3, 0.9);
|
||||
Disc(p, x, y, core, 0, ink, ink, ink, 1);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ---------------------------------------------------------------- the buffers (see the top)
|
||||
|
||||
struct Buffer {
|
||||
gbm_bo *bo = nullptr;
|
||||
int fd = -1;
|
||||
vr::SharedTextureHandle_t handle = 0;
|
||||
};
|
||||
|
||||
struct Buffers {
|
||||
int drm = -1;
|
||||
gbm_device *gbm = nullptr;
|
||||
Buffer b[3];
|
||||
int next = 0;
|
||||
bool ok = false;
|
||||
|
||||
bool Make() {
|
||||
drm = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC);
|
||||
if (drm >= 0) gbm = gbm_create_device(drm);
|
||||
for (Buffer &x : b) {
|
||||
// ABGR8888 is R, G, B, A in memory, like Panel::px.
|
||||
if (gbm) x.bo = gbm_bo_create(gbm, kMaxW, kMaxH, GBM_FORMAT_ABGR8888, GBM_BO_USE_RENDERING | GBM_BO_USE_LINEAR);
|
||||
if (!x.bo || (x.fd = gbm_bo_get_fd(x.bo)) < 0) break;
|
||||
vr::DmabufAttributes_t a{};
|
||||
a.unWidth = kMaxW, a.unHeight = kMaxH;
|
||||
a.unDepth = a.unMipLevels = a.unArrayLayers = a.unSampleCount = 1;
|
||||
a.unFormat = DRM_FORMAT_ABGR8888;
|
||||
a.ulModifier = DRM_FORMAT_MOD_LINEAR;
|
||||
a.unPlaneCount = 1;
|
||||
a.plane[0].unOffset = gbm_bo_get_offset(x.bo, 0);
|
||||
a.plane[0].unStride = gbm_bo_get_stride(x.bo);
|
||||
a.plane[0].nFd = x.fd;
|
||||
if (!vr::VRIPCResourceManager()->ImportDmabuf(vr::VRApplication_Overlay, &a, &x.handle)) x.handle = 0;
|
||||
if (!x.handle) break;
|
||||
}
|
||||
ok = b[2].handle != 0;
|
||||
if (!ok) {
|
||||
std::fprintf(stderr, "ft-gazepanel: no shared buffers; falling back to SetOverlayRaw (it flickers)\n");
|
||||
Drop();
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
|
||||
void Drop() {
|
||||
for (Buffer &x : b) {
|
||||
if (x.handle) vr::VRIPCResourceManager()->UnrefResource(x.handle);
|
||||
if (x.fd >= 0) close(x.fd);
|
||||
if (x.bo) gbm_bo_destroy(x.bo);
|
||||
x = Buffer{};
|
||||
}
|
||||
if (gbm) gbm_device_destroy(gbm);
|
||||
if (drm >= 0) close(drm);
|
||||
gbm = nullptr, drm = -1, ok = false;
|
||||
}
|
||||
|
||||
// p's picture to the overlay: into the next buffer, premultiplied (the overlay's flag says
|
||||
// so), then the overlay switches to it.
|
||||
void Present(vr::IVROverlay *ov, vr::VROverlayHandle_t h, const Panel &p) {
|
||||
vr::EVROverlayError e;
|
||||
if (!ok) {
|
||||
e = ov->SetOverlayRaw(h, const_cast<uint8_t *>(p.px.data()), uint32_t(p.w), uint32_t(p.h), 4);
|
||||
} else {
|
||||
Buffer &x = b[next];
|
||||
next = (next + 1) % 3;
|
||||
uint32_t stride = 0;
|
||||
void *mapping = nullptr;
|
||||
auto *dst = static_cast<uint8_t *>(gbm_bo_map(x.bo, 0, 0, p.w, p.h, GBM_BO_TRANSFER_WRITE, &stride, &mapping));
|
||||
if (!dst) {
|
||||
std::fprintf(stderr, "ft-gazepanel: can't map a buffer\n");
|
||||
return;
|
||||
}
|
||||
for (int y = 0; y < p.h; ++y) {
|
||||
const uint8_t *src = &p.px[size_t(y) * p.w * 4];
|
||||
uint8_t *row = dst + size_t(y) * stride;
|
||||
for (int i = 0; i < p.w * 4; i += 4) {
|
||||
const unsigned a = src[i + 3];
|
||||
row[i] = uint8_t(src[i] * a / 255), row[i + 1] = uint8_t(src[i + 1] * a / 255);
|
||||
row[i + 2] = uint8_t(src[i + 2] * a / 255), row[i + 3] = uint8_t(a);
|
||||
}
|
||||
}
|
||||
gbm_bo_unmap(x.bo, mapping);
|
||||
const vr::VRTextureBounds_t bounds{0, 0, float(p.w) / kMaxW, float(p.h) / kMaxH};
|
||||
ov->SetOverlayTextureBounds(h, &bounds);
|
||||
vr::Texture_t tex = {&x.handle, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
|
||||
e = ov->SetOverlayTexture(h, &tex);
|
||||
}
|
||||
if (e != vr::VROverlayError_None)
|
||||
std::fprintf(stderr, "ft-gazepanel: the picture didn't go to SteamVR: %s\n", ov->GetOverlayErrorNameFromEnum(e));
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
bool watchStdin = false;
|
||||
std::string sockName = "ft_gazepanel";
|
||||
double distance = 1.5;
|
||||
for (int i = 1; i < argc; ++i) {
|
||||
if (!std::strcmp(argv[i], "--watch-stdin")) watchStdin = true;
|
||||
else if (!std::strcmp(argv[i], "--socket") && i + 1 < argc) sockName = argv[++i];
|
||||
else if (!std::strcmp(argv[i], "--distance") && i + 1 < argc) distance = std::clamp(std::atof(argv[++i]), 0.5, 5.0);
|
||||
else {
|
||||
std::fprintf(stderr, "usage: %s [--watch-stdin] [--socket NAME] [--distance METRES]\n", argv[0]);
|
||||
return 2;
|
||||
}
|
||||
}
|
||||
std::signal(SIGINT, [](int) { g_stop = true; });
|
||||
std::signal(SIGTERM, [](int) { g_stop = true; });
|
||||
if (watchStdin)
|
||||
std::thread([] {
|
||||
char c[256];
|
||||
while (read(0, c, sizeof c) > 0) {
|
||||
}
|
||||
g_stop = true;
|
||||
}).detach();
|
||||
|
||||
int sock = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0);
|
||||
sockaddr_un addr{};
|
||||
addr.sun_family = AF_UNIX;
|
||||
std::memcpy(addr.sun_path + 1, sockName.data(), std::min(sockName.size(), sizeof addr.sun_path - 2));
|
||||
if (bind(sock, reinterpret_cast<sockaddr *>(&addr), socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sockName.size())) != 0) {
|
||||
std::fprintf(stderr, "ft-gazepanel: @%s is taken (another copy running?)\n", sockName.c_str());
|
||||
return 1;
|
||||
}
|
||||
|
||||
// As Frametop's other SteamVR clients: background first, so we never start vrserver.
|
||||
vr::EVRInitError err = vr::VRInitError_None;
|
||||
vr::VR_Init(&err, vr::VRApplication_Background);
|
||||
if (err == vr::VRInitError_None) {
|
||||
vr::VR_Shutdown();
|
||||
vr::VR_Init(&err, vr::VRApplication_Overlay);
|
||||
}
|
||||
if (err != vr::VRInitError_None) {
|
||||
std::fprintf(stderr, "ft-gazepanel: SteamVR: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
|
||||
return 1;
|
||||
}
|
||||
vr::IVROverlay *ov = vr::VROverlay();
|
||||
vr::VROverlayHandle_t h = vr::k_ulOverlayHandleInvalid;
|
||||
if (ov->CreateOverlay("frametop.gazepanel", "Frametop gaze calibration", &h) != vr::VROverlayError_None) {
|
||||
std::fprintf(stderr, "ft-gazepanel: can't create the overlay (another copy running?)\n");
|
||||
return 1;
|
||||
}
|
||||
ov->SetOverlaySortOrder(h, 250); // in front of Frametop's screens and the pointer's dot
|
||||
LoadFont();
|
||||
Buffers buffers;
|
||||
if (buffers.Make()) ov->SetOverlayFlag(h, vr::VROverlayFlags_IsPremultiplied, true);
|
||||
|
||||
Panel p;
|
||||
bool visible = false, dirty = false, shown = false; // shown: SteamVR shows it (after its first picture)
|
||||
auto place = [&] {
|
||||
const double half = std::tan(p.wDeg * M_PI / 360);
|
||||
vr::HmdMatrix34_t m{};
|
||||
m.m[0][0] = m.m[1][1] = m.m[2][2] = 1;
|
||||
m.m[2][3] = float(-distance);
|
||||
ov->SetOverlayTransformTrackedDeviceRelative(h, vr::k_unTrackedDeviceIndex_Hmd, &m);
|
||||
ov->SetOverlayWidthInMeters(h, float(2 * distance * half));
|
||||
};
|
||||
std::fprintf(stderr, "ft-gazepanel running: @%s, %.2f m\n", sockName.c_str(), distance);
|
||||
|
||||
while (!g_stop) {
|
||||
char buf[512];
|
||||
sockaddr_un from{};
|
||||
socklen_t fromLen = sizeof from;
|
||||
ssize_t n;
|
||||
while ((n = recvfrom(sock, buf, sizeof buf - 1, 0, reinterpret_cast<sockaddr *>(&from), &fromLen)) > 0) {
|
||||
buf[n] = 0;
|
||||
std::string reply = "ok";
|
||||
char word[16] = "", state[16] = "";
|
||||
double a = 0, b = 0, c = 0, d = 0, e = 0;
|
||||
int rest = 0;
|
||||
if (!std::strncmp(buf, "show ", 5)) {
|
||||
p.full = !std::strcmp(buf + 5, "full");
|
||||
p.fit = !std::strcmp(buf + 5, "fit");
|
||||
p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx;
|
||||
p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx;
|
||||
p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg;
|
||||
p.title.clear(), p.text.clear(), p.dotOn = false, p.state = "off";
|
||||
p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
|
||||
place();
|
||||
visible = dirty = true; // shown with its first picture
|
||||
} else if (!std::strcmp(buf, "hide")) {
|
||||
ov->HideOverlay(h);
|
||||
visible = shown = false;
|
||||
} else if (std::sscanf(buf, "bg %lf", &a) == 1) {
|
||||
p.bg = std::clamp(a, 0.0, 1.0), dirty = true;
|
||||
} else if (std::sscanf(buf, "dot %lf %lf %15s %lf", &a, &b, state, &c) >= 3) {
|
||||
p.dotYaw = a, p.dotPitch = b, p.state = state, p.progress = c;
|
||||
p.dotOn = std::strcmp(state, "off") != 0, dirty = true;
|
||||
} else if (int k; std::sscanf(buf, "eye %d %lf %lf %lf %lf %lf %n", &k, &a, &b, &c, &d, &e, &rest) >= 6 &&
|
||||
rest > 0 && (k == 0 || k == 1)) {
|
||||
p.eyes[k] = EyeCard{buf + rest, a, b, c, d, e}, dirty = true;
|
||||
} else if (!std::strncmp(buf, "hints", 5)) {
|
||||
p.hints.clear();
|
||||
std::string s = buf[5] == ' ' ? buf + 6 : "";
|
||||
for (size_t at = 0; !s.empty() && at <= s.size();) {
|
||||
const size_t bar = std::min(s.find('|', at), s.size());
|
||||
p.hints.push_back(s.substr(at, bar - at));
|
||||
at = bar + 1;
|
||||
}
|
||||
dirty = true;
|
||||
} else if (!std::strncmp(buf, "title", 5)) {
|
||||
p.title = buf[5] == ' ' ? buf + 6 : "", dirty = true;
|
||||
} else if (!std::strncmp(buf, "text", 4)) {
|
||||
p.text = buf[4] == ' ' ? buf + 5 : "", dirty = true;
|
||||
} else if (std::sscanf(buf, "%15s", word) == 1 && !std::strcmp(word, "ping")) {
|
||||
reply = visible ? "ok shown" : "ok hidden";
|
||||
} else {
|
||||
reply = "error unknown command";
|
||||
}
|
||||
if (fromLen > offsetof(sockaddr_un, sun_path))
|
||||
sendto(sock, reply.data(), reply.size(), 0, reinterpret_cast<sockaddr *>(&from), fromLen);
|
||||
fromLen = sizeof from;
|
||||
}
|
||||
vr::VREvent_t ev;
|
||||
while (vr::VRSystem()->PollNextEvent(&ev, sizeof ev))
|
||||
if (ev.eventType == vr::VREvent_Quit) {
|
||||
vr::VRSystem()->AcknowledgeQuit_Exiting();
|
||||
g_stop = true;
|
||||
}
|
||||
if (visible && dirty) {
|
||||
Draw(p);
|
||||
buffers.Present(ov, h, p);
|
||||
if (!shown) ov->ShowOverlay(h), shown = true;
|
||||
dirty = false;
|
||||
}
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(visible ? 10 : 50));
|
||||
}
|
||||
ov->DestroyOverlay(h);
|
||||
buffers.Drop();
|
||||
vr::VR_Shutdown();
|
||||
return 0;
|
||||
}
|
||||
@@ -1,9 +1,9 @@
|
||||
[Desktop Entry]
|
||||
Type=Application
|
||||
Name=Frametop Gaze Probe
|
||||
GenericName=Eye tracking playground
|
||||
Comment=How accurate the headset's eye tracking is on your screens, and gaze clicking with calibration
|
||||
Name=Frametop Gaze Probe (development)
|
||||
GenericName=Eye tracking development tool
|
||||
Comment=For developing Frametop's gaze tracking. Day to day, the calibration and checks are on the Gaze page of Frametop Input Settings
|
||||
Exec=@REPO@/gaze/probe/ft-gazeprobe
|
||||
Icon=view-visible
|
||||
Categories=Utility;Development;
|
||||
Categories=Development;
|
||||
Keywords=eye;gaze;tracking;calibration;pointer;steamvr;frametop;
|
||||
@@ -64,6 +64,9 @@ ACTION_LABELS = {
|
||||
"follow_toggle": "Head follow on/off (experimental)", "gaze_toggle": "Gaze pointer on/off (experimental)",
|
||||
"gaze_precision": "Gaze precision: hold to steer, release to click",
|
||||
"gaze_drag": "Gaze drag: press where you look, steer, release",
|
||||
"gaze_left": "Gaze left click: tap, or hold and turn your head to aim",
|
||||
"gaze_right": "Gaze right click: tap, or hold and turn your head to aim",
|
||||
"gaze_quickcal": "Gaze quick check (one dot)",
|
||||
"sens_up": "Faster pointer",
|
||||
"sens_down": "Slower pointer", "layout_reset": "Reset desktop screen layout",
|
||||
"screens_toggle": "Hide/show desktop screens", "keyboard_toggle": "Open/close keyboard",
|
||||
@@ -90,8 +93,12 @@ CONTROLLER_BUTTONS = {
|
||||
"right/bumper": "Right bumper", "right/trigger": "Right trigger", "right/grip": "Right grip",
|
||||
"right/thumbstick": "Right stick click",
|
||||
}
|
||||
# Not the gaze actions: gaze mode is a mouse feature (docs/gaze-controllers.md; the relay's GAZE_ACTIONS).
|
||||
CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none", "gaze_toggle", "gaze_precision", "gaze_drag")]
|
||||
# Gaze mode is a mouse and keyboard feature (docs/gaze-controllers.md; the relay's GAZE_ACTIONS).
|
||||
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
|
||||
CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none") + GAZE_ACTIONS]
|
||||
# Key combinations take any action but key and none; the keyboard clicks only work there.
|
||||
SHORTCUT_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none")]
|
||||
KEYBOARD_ONLY = ("gaze_left", "gaze_right")
|
||||
# Profiles (docs/profiles.md): "profile:NAME" opens one (the relay runs ft-layout use NAME).
|
||||
LAYOUT_PATH = os.path.expanduser("~/.config/frametop-layout.json")
|
||||
PROFILE = "profile:"
|
||||
@@ -109,13 +116,22 @@ def action_label(a):
|
||||
return ACTION_LABELS.get(a, a)
|
||||
|
||||
|
||||
def is_profile(a):
|
||||
return a.startswith(PROFILE) and len(a) > len(PROFILE)
|
||||
|
||||
|
||||
def mappable(a):
|
||||
"""An action a controller button or key combination can have."""
|
||||
return a in CONTROLLER_ACTIONS or (a.startswith(PROFILE) and len(a) > len(PROFILE))
|
||||
"""An action a controller button can have."""
|
||||
return a in CONTROLLER_ACTIONS or is_profile(a)
|
||||
|
||||
|
||||
def shortcut_mappable(a):
|
||||
"""An action a key combination can have: the gaze ones too."""
|
||||
return a in SHORTCUT_ACTIONS or is_profile(a)
|
||||
# Gaze mode settings (pointer helper), like POINTER_SETTINGS.
|
||||
GAZE_SETTINGS = [
|
||||
("POINTER_GAZE_RETAKE", "Look away to hand back", 5, 1, 45, 0.5, "°"),
|
||||
("POINTER_GAZE_NUDGE_MAX", "Largest nudge to learn", 8, 1, 30, 0.5, "°"),
|
||||
("POINTER_GAZE_NUDGE_MAX", "Largest correction to learn", 55, 1, 110, 1, "°"),
|
||||
("POINTER_GAZE_HOLD", "Hold still to drag", 0.5, 0.1, 2.0, 0.05, "s"),
|
||||
("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"),
|
||||
]
|
||||
@@ -127,8 +143,9 @@ POINTER_ROLES = {"right": "Right hand", "left": "Left hand", "stylus": "Stylus (
|
||||
# Key combinations ("key_bindings" in the rules): modifiers, either side folded into the left code.
|
||||
MODIFIER_CODES = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
|
||||
MODIFIER_NAMES = {29: "Ctrl", 42: "Shift", 56: "Alt", 125: "Meta"}
|
||||
# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): Meta+Shift+F.
|
||||
DEFAULT_KEY_BINDINGS = {"42+125+33": "float_toggle"}
|
||||
# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): Meta+J and
|
||||
# Meta+K click at the gaze, Meta+Shift+F floats a window.
|
||||
DEFAULT_KEY_BINDINGS = {"125+36": "gaze_left", "125+37": "gaze_right", "42+125+33": "float_toggle"}
|
||||
|
||||
|
||||
def key_bindings(rules):
|
||||
@@ -286,6 +303,7 @@ class Backend(QObject):
|
||||
self._gaze_prev = None # the status before, for rates
|
||||
self._gaze_at = 0.0
|
||||
self._gaze_mode = None # the helper's gaze mode: True, False, None (no answer)
|
||||
self._check_asked = 0.0 # when quickcal or calibrate went to the gaze service (its errors)
|
||||
self._driver_block = "" # set by _check_driver
|
||||
self._panels = None # SteamVR's overlays, from the helper; None until it answers
|
||||
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
@@ -349,6 +367,10 @@ class Backend(QObject):
|
||||
except BlockingIOError:
|
||||
return
|
||||
text = data.decode(errors="replace")
|
||||
if text.startswith("error ") and self._check_asked and time.monotonic() - self._check_asked < 3:
|
||||
self._check_asked = 0.0 # the gaze service's answer to quickcal or calibrate
|
||||
self.message.emit("Gaze check: " + text[6:], True)
|
||||
continue
|
||||
if text in ("ok on", "ok off"): # the helper's answer to "gaze ?" (from an unbound socket)
|
||||
self._gaze_mode = text == "ok on"
|
||||
self._gaze_at = time.monotonic()
|
||||
@@ -501,7 +523,7 @@ class Backend(QObject):
|
||||
|
||||
@Property("QVariantList", constant=True)
|
||||
def actions(self):
|
||||
return [{"value": k, "text": v} for k, v in ACTION_LABELS.items()] + \
|
||||
return [{"value": k, "text": v} for k, v in ACTION_LABELS.items() if k not in KEYBOARD_ONLY] + \
|
||||
[{"value": a, "text": action_label(a)} for a in profile_actions()]
|
||||
|
||||
@Slot(str, result="QVariantList")
|
||||
@@ -840,6 +862,18 @@ class Backend(QObject):
|
||||
self.pointerChanged.emit()
|
||||
self.message.emit(f"Mouse in gaze mode: {GAZE_MOUSE[mode].lower()}", False)
|
||||
|
||||
@Property(bool, notify=pointerChanged)
|
||||
def gazeMouseHeld(self):
|
||||
return read_conf().get("POINTER_GAZE_MOUSE_MOVE", "held") != "free"
|
||||
|
||||
@Slot(bool)
|
||||
def setGazeMouseHeld(self, on):
|
||||
write_conf_value("POINTER_GAZE_MOUSE_MOVE", "held" if on else "free")
|
||||
self.reload_timer.start()
|
||||
self.pointerChanged.emit()
|
||||
self.message.emit("Mouse in gaze mode: " + ("moves the pointer only while a button is held" if on
|
||||
else "moves the pointer any time"), False)
|
||||
|
||||
@Property(bool, notify=pointerChanged)
|
||||
def gazeDotAlways(self):
|
||||
return read_conf().get("POINTER_GAZE_DOT", "always") != "moving"
|
||||
@@ -881,7 +915,7 @@ class Backend(QObject):
|
||||
|
||||
@Property("QVariantList", constant=True)
|
||||
def shortcutActions(self):
|
||||
return [{"value": a, "text": action_label(a)} for a in CONTROLLER_ACTIONS + profile_actions()]
|
||||
return [{"value": a, "text": action_label(a)} for a in SHORTCUT_ACTIONS + profile_actions()]
|
||||
|
||||
@Property(bool, notify=shortcutCaptureChanged)
|
||||
def capturingShortcut(self):
|
||||
@@ -889,7 +923,7 @@ class Backend(QObject):
|
||||
|
||||
@Slot(str)
|
||||
def startShortcutCapture(self, action):
|
||||
if mappable(action):
|
||||
if shortcut_mappable(action):
|
||||
self._capture_combo = action
|
||||
self._combo_mods = set()
|
||||
self._send("watch 60")
|
||||
@@ -972,14 +1006,32 @@ class Backend(QObject):
|
||||
self.message.emit("The gaze service isn't running (frametop-gaze.service)", True)
|
||||
|
||||
@Slot()
|
||||
def openGazeProbe(self):
|
||||
"""Calibrate in ft-gazeprobe (a GTK app on the host, fullscreen on a Frametop screen)."""
|
||||
self._open_probe([], "Opening the gaze probe: calibrate there, then close it")
|
||||
def gazeQuickCheck(self):
|
||||
"""The gaze service's one-dot check, in the panel fixed to the headset."""
|
||||
self._gaze_check("quickcal", "Quick check: look at the dot in front of you")
|
||||
|
||||
@Slot()
|
||||
def openHeadsetFit(self):
|
||||
"""The probe's Headset fit mode: how well the tracker sees each eye, as you adjust."""
|
||||
self._open_probe(["--mode", "fit"], "Opening the headset fit check in the gaze probe")
|
||||
def gazeCalibrate(self):
|
||||
"""The full calibration in the panel fixed to the headset (gaze/gazecheck.py)."""
|
||||
self._gaze_check("calibrate", "Calibration: look at each dot in the headset; right click or Meta+K stops")
|
||||
|
||||
def _gaze_check(self, command, done):
|
||||
if self._send(command, GAZED):
|
||||
self._check_asked = time.monotonic()
|
||||
self.message.emit(done, False)
|
||||
else:
|
||||
self.message.emit("The gaze service isn't running (frametop-gaze.service)", True)
|
||||
|
||||
@Slot()
|
||||
def gazeFitCheck(self):
|
||||
"""The headset fit check, in the panel fixed to the headset (gaze/gazecheck.py)."""
|
||||
self._gaze_check("fitcheck", "Headset fit: in the headset, adjust it while you watch; right click or Meta+K closes it")
|
||||
|
||||
@Slot()
|
||||
def openGazeProbe(self):
|
||||
"""ft-gazeprobe, the gaze tracking's development tool (a GTK app on the host, fullscreen on
|
||||
a Frametop screen). Day to day, the checks and the calibration run in the headset panel."""
|
||||
self._open_probe([], "Opening the gaze probe (a development tool)")
|
||||
|
||||
def _open_probe(self, args, done):
|
||||
runner = ["distrobox-host-exec"] if shutil.which("distrobox-host-exec") else []
|
||||
|
||||
+51
-16
@@ -815,16 +815,25 @@ Kirigami.ApplicationWindow {
|
||||
property var status: backend.gazeStatus
|
||||
actions: [
|
||||
Kirigami.Action {
|
||||
text: "Calibrate…"
|
||||
text: "Quick check"
|
||||
icon.name: "crosshairs"
|
||||
tooltip: "Open the gaze probe to calibrate (fullscreen on a Frametop screen)"
|
||||
onTriggered: backend.openGazeProbe()
|
||||
enabled: backend.gazeServiceRunning
|
||||
tooltip: "One dot in front of you in the headset: look at it, and the gaze tracker relearns where it sits"
|
||||
onTriggered: backend.gazeQuickCheck()
|
||||
},
|
||||
Kirigami.Action {
|
||||
text: "Check headset fit…"
|
||||
text: "Calibrate"
|
||||
icon.name: "crosshairs"
|
||||
enabled: backend.gazeServiceRunning
|
||||
tooltip: "The full calibration in the headset: dots in three rounds, dark to bright"
|
||||
onTriggered: backend.gazeCalibrate()
|
||||
},
|
||||
Kirigami.Action {
|
||||
text: "Check headset fit"
|
||||
icon.name: "view-visible"
|
||||
tooltip: "How well the eye tracker sees each eye, and where it loses one, while you adjust the headset"
|
||||
onTriggered: backend.openHeadsetFit()
|
||||
enabled: backend.gazeServiceRunning
|
||||
tooltip: "In the headset: how well the eye tracker sees each eye, live, while you adjust it"
|
||||
onTriggered: backend.gazeFitCheck()
|
||||
},
|
||||
Kirigami.Action {
|
||||
text: "Reload calibration"
|
||||
@@ -838,6 +847,14 @@ Kirigami.ApplicationWindow {
|
||||
enabled: backend.gazeServiceRunning
|
||||
tooltip: "Drop what your mouse nudges taught; the calibration stays"
|
||||
onTriggered: backend.forgetGazeLessons()
|
||||
},
|
||||
Kirigami.Action {
|
||||
// A development tool: the checks and the calibration run in the headset panel.
|
||||
text: "Gaze probe (development)…"
|
||||
icon.name: "tools"
|
||||
displayHint: Kirigami.DisplayHint.AlwaysHide
|
||||
tooltip: "The gaze tracking's lab tool, for developing it: experiments, accuracy tests, practice"
|
||||
onTriggered: backend.openGazeProbe()
|
||||
}
|
||||
]
|
||||
|
||||
@@ -876,6 +893,22 @@ Kirigami.ApplicationWindow {
|
||||
}
|
||||
}
|
||||
}
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "Mouse movement:"
|
||||
text: "Only while a button is held (recommended)"
|
||||
checked: backend.gazeMouseHeld
|
||||
onToggled: backend.setGazeMouseHeld(checked)
|
||||
Controls.ToolTip.visible: hovered
|
||||
Controls.ToolTip.delay: Kirigami.Units.toolTipDelay
|
||||
Controls.ToolTip.text: checked
|
||||
? "Your eyes move the pointer, so moving the mouse on its own does nothing. Hold a button "
|
||||
+ "and the pointer stops where you look: move onto the target and let go to click there "
|
||||
+ "(left or right). To drag after moving, press right while holding left. It's on because a "
|
||||
+ "bumped mouse can't pull the pointer away, and every mouse move is a real correction the "
|
||||
+ "eye tracker learns from. If the eye tracker stops, the mouse works as usual."
|
||||
: "The mouse moves the pointer any time, as without gaze. Moves that weren't corrections can "
|
||||
+ "teach the eye tracker the wrong thing."
|
||||
}
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "Gaze dot:"
|
||||
text: "Always shown (off: only while the mouse moves it)"
|
||||
@@ -885,9 +918,11 @@ Kirigami.ApplicationWindow {
|
||||
Controls.Label {
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
|
||||
wrapMode: Text.WordWrap
|
||||
text: "Gaze works with the mouse. The Frame controllers don't take part, and moving one hands the "
|
||||
+ "pointer back to the controllers. Gaze pointer on/off, Gaze precision, and Gaze drag can go on "
|
||||
+ "a mouse button (Buttons page) or a key combination (Keyboard page)."
|
||||
text: "Gaze works with the mouse and the keyboard. Keyboard clicks (Meta+J left, Meta+K right by "
|
||||
+ "default): tap to click where you look, or hold, turn your head until the dot sits on the "
|
||||
+ "target, and let go; hold still to drag. The Frame controllers don't take part, and moving "
|
||||
+ "one hands the pointer back to them. Gaze pointer on/off, Gaze precision, and Gaze drag can "
|
||||
+ "also go on a mouse button (Buttons page) or another key combination (Keyboard page)."
|
||||
opacity: 0.7
|
||||
font: Kirigami.Theme.smallFont
|
||||
}
|
||||
@@ -911,7 +946,7 @@ Kirigami.ApplicationWindow {
|
||||
&& (!gpage.status.own_running || gpage.status.own_reseat || !gpage.status.calibration_samples)
|
||||
text: !gpage.status.eyegrab ? "Needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)"
|
||||
: !gpage.status.own_running ? "Starting…"
|
||||
: !gpage.status.calibration_samples ? "Not calibrated: use Calibrate… with Own tracker"
|
||||
: !gpage.status.calibration_samples ? "Not calibrated: use Calibrate"
|
||||
: "The headset was off: your first nudge and click resets where it sits"
|
||||
color: gpage.status.own_running ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor
|
||||
font: Kirigami.Theme.smallFont
|
||||
@@ -999,7 +1034,7 @@ Kirigami.ApplicationWindow {
|
||||
+ (gpage.status.one_eye_share > 0.5 ? " · only one eye tracked" : "")
|
||||
color: gpage.status.one_eye_share > 0.5 ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.textColor
|
||||
Controls.ToolTip.text: "Only one eye tracked: the gaze comes from the other eye, a little less "
|
||||
+ "precisely. Check headset fit… shows where the tracker loses it."
|
||||
+ "precisely. Check headset fit shows where the tracker loses it."
|
||||
Controls.ToolTip.visible: gpage.status.one_eye_share > 0.5 && ghover.hovered
|
||||
HoverHandler { id: ghover }
|
||||
}
|
||||
@@ -1023,7 +1058,7 @@ Kirigami.ApplicationWindow {
|
||||
? gpage.status.calibration_samples + " points ("
|
||||
+ (gpage.status.tracker === "own" ? "Own tracker, " + gpage.status.calibration_made
|
||||
: gpage.status.kind === "eyes" ? gpage.status.model + ", each eye" : gpage.status.model) + ")"
|
||||
: "none yet: use Calibrate…"
|
||||
: "none yet: use Calibrate"
|
||||
}
|
||||
Controls.Label {
|
||||
visible: backend.gazeServiceRunning
|
||||
@@ -1037,13 +1072,13 @@ Kirigami.ApplicationWindow {
|
||||
padding: Kirigami.Units.largeSpacing
|
||||
wrapMode: Text.Wrap
|
||||
opacity: 0.7
|
||||
text: "Map a mouse button (Buttons) or a controller button (Controllers) to \"Gaze pointer on/off\" "
|
||||
text: "Map a mouse button (Buttons) or a key combination (Keyboard) to \"Gaze pointer on/off\" "
|
||||
+ "to switch it on the fly. A click waits for the release: if the gaze is off, drag onto the target "
|
||||
+ "with the button held and let go there. Hold still to drag: hold a press this long without moving "
|
||||
+ "to drag something instead. Look away to hand back: how far from the pointer you look before the "
|
||||
+ "gaze takes it back from the mouse. Largest nudge to learn: bigger mouse moves before a click "
|
||||
+ "are treated as using the mouse, not correcting the gaze. Eye tracker: SteamVR's, or our own "
|
||||
+ "(gaze/tracker), which keeps its own calibration (Calibrate… with Own tracker). Eye bias: the gaze "
|
||||
+ "gaze takes it back from the mouse. Largest correction to learn: a click corrected further than "
|
||||
+ "this isn't learned; the tracker is that far off, so the quick check runs instead. Eye tracker: SteamVR's, or our own "
|
||||
+ "(gaze/tracker), which keeps its own calibration. Eye bias: the gaze "
|
||||
+ "combines both eyes, since their errors partly cancel; Left or Right counts that eye twice as "
|
||||
+ "much, and Auto weights each by how far off it was at your recent nudges."
|
||||
}
|
||||
|
||||
+94
-9
@@ -23,7 +23,11 @@ pointer_toggle, follow_toggle = head follow on or off, gaze_toggle = gaze mode o
|
||||
(the pointer goes where you look; see pointer/helper/ft-pointer.cpp), gaze_precision = while
|
||||
held, the pointer stops where you look and the mouse steers it, and the release clicks there,
|
||||
gaze_drag = the same, but pressed at once, so it drags ("precision|gazedrag mouse|keyboard 1|0"
|
||||
to the helper), sens_up, sens_down,
|
||||
to the helper), gaze_left and gaze_right = keyboard clicks at the gaze: a tap clicks where you
|
||||
look; held, the pointer stops there and your head steers it (it stays put in your view), and
|
||||
the release clicks; held still for half a second, it's a real press that your head drags
|
||||
("gazekey left|right 1|0" to the helper; by default Meta+J and Meta+K, DEFAULT_KEY_BINDINGS),
|
||||
gaze_quickcal = the gaze service's one-dot check ("quickcal" to @ft_gazed), sens_up, sens_down,
|
||||
layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens,
|
||||
keyboard_toggle = open or close Frametop's keyboard, float_toggle = float the desktop window under the
|
||||
pointer (else the active one) in VR, or put it back if it floats, dock_all = put every floating
|
||||
@@ -35,8 +39,12 @@ rules file; any action but key and the gaze ones, GAZE_ACTIONS: gaze mode is a m
|
||||
docs/gaze-controllers.md). So can key combinations on any keyboard ("key_bindings":
|
||||
{"29+56+34": action}, evdev codes joined by "+", modifiers first and left-hand codes for
|
||||
either side, here Ctrl+Alt+G): the combination does the action, and its last key isn't typed.
|
||||
A rules file without "key_bindings" gets DEFAULT_KEY_BINDINGS (Meta+Shift+F: float_toggle); one
|
||||
with its own, even an empty one, doesn't. The float actions work without pointer mode too. The controllers aren't input devices here, only SteamVR sees
|
||||
A rules file without "key_bindings" gets DEFAULT_KEY_BINDINGS (Meta+J: gaze_left, Meta+K:
|
||||
gaze_right, Meta+Shift+F: float_toggle); one with its own, even an empty one, doesn't. The float
|
||||
actions work without pointer mode too. A combination with Meta also sends the desktop an F24 press
|
||||
and Meta's release right away: so letting go of Meta doesn't open Plasma's launcher, and a gaze
|
||||
click isn't Meta+click (KWin's window move and resize). Another key while Meta is still held gives
|
||||
the desktop Meta back. The controllers aren't input devices here, only SteamVR sees
|
||||
them, so the pointer helper reads them with SteamVR input and sends "vrbtn <button> 1|0".
|
||||
It only takes the buttons the relay tells it to ("vrbind <button>..." to @ft_pointer_helper,
|
||||
sent on start, reload, and when the helper says "vrhello"), and only while no game runs,
|
||||
@@ -179,10 +187,15 @@ def eviocguniq(length):
|
||||
VIRTUAL_PREFIX = "frametop virtual"
|
||||
RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
|
||||
ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
|
||||
"pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "sens_up", "sens_down",
|
||||
"layout_reset", "screens_toggle", "keyboard_toggle", "float_toggle", "dock_all", "key", "none")
|
||||
"pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right",
|
||||
"gaze_quickcal", "sens_up", "sens_down", "layout_reset", "screens_toggle", "keyboard_toggle", "float_toggle",
|
||||
"dock_all", "key", "none")
|
||||
# Gaze mode is a mouse feature: these never come from a controller button (docs/gaze-controllers.md).
|
||||
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag")
|
||||
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
|
||||
# Key combinations a rules file without "key_bindings" gets: Meta+J and Meta+K click at the gaze
|
||||
# (free on the Frametop desktop, and apps don't use Meta), Meta+Shift+F floats a window.
|
||||
DEFAULT_KEY_BINDINGS = {"125+36": "gaze_left", "125+37": "gaze_right", "42+125+33": "float_toggle"}
|
||||
KEY_F24 = 194 # sent to the desktop with a Meta combination (see the top)
|
||||
# Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
|
||||
MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
|
||||
VR_KEYBOARD_MODES = ("always", "no_keyboard", "button", "never") # when Frametop's keyboard opens
|
||||
@@ -193,6 +206,7 @@ VR_BUTTONS = ("left/view", "left/dpad_up", "left/dpad_down", "left/dpad_left", "
|
||||
"right/bumper", "right/trigger", "right/grip", "right/thumbstick")
|
||||
VR_DEVICE = "frame_controller" # the id controller buttons have in watch events
|
||||
SCREENS = "\0ft_screens"
|
||||
GAZED = "\0ft_gazed"
|
||||
FLOAT = "\0frametop_float" # ft-floatd, floating windows in the Frametop desktop
|
||||
# Actions for ft-floatd ("float_toggle", "dock_all"): they don't need pointer mode.
|
||||
FLOAT_ACTIONS = {"float_toggle": b"float pointer", "dock_all": b"dock all"}
|
||||
@@ -205,8 +219,6 @@ def known_action(a):
|
||||
|
||||
def needs_pointer(a):
|
||||
return a not in FLOAT_ACTIONS and not a.startswith(PROFILE)
|
||||
# Key combinations a rules file without "key_bindings" gets: Meta+Shift+F floats a window.
|
||||
DEFAULT_KEY_BINDINGS = {"42+125+33": "float_toggle"}
|
||||
KEYS = "\0frametop_keys" # keys of keyboards grabbed for the desktop, for other readers
|
||||
FT_LAYOUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "layout", "ft-layout")
|
||||
DEFAULT_BUTTONS = {BTN_LEFT: "left", BTN_RIGHT: "right", BTN_MIDDLE: "middle",
|
||||
@@ -470,6 +482,12 @@ class Pointer:
|
||||
which take the mouse or the keyboard only."""
|
||||
if value == 2:
|
||||
return
|
||||
if name in ("gaze_left", "gaze_right"):
|
||||
if value == 1:
|
||||
self.wake(now)
|
||||
self.flush()
|
||||
self.send(f"gazekey {name[5:]} {value}")
|
||||
return
|
||||
if name in ("gaze_precision", "gaze_drag"):
|
||||
if source not in ("mouse", "keyboard"):
|
||||
return # gaze mode is a mouse feature (GAZE_ACTIONS)
|
||||
@@ -507,6 +525,11 @@ class Pointer:
|
||||
elif name == "gaze_toggle":
|
||||
self.send("gaze toggle") # until the next restart; the setting is POINTER_GAZE
|
||||
log("gaze mode toggled")
|
||||
elif name == "gaze_quickcal":
|
||||
try:
|
||||
self.sock.sendto(b"quickcal", GAZED)
|
||||
except OSError:
|
||||
pass # the gaze service isn't running
|
||||
elif name == "screens_toggle":
|
||||
try:
|
||||
self.sock.sendto(b"toggle", SCREENS)
|
||||
@@ -746,14 +769,28 @@ def main():
|
||||
state["pointer"].action(action, value, now, source)
|
||||
|
||||
held_modifiers = set() # on any keyboard, folded (MODIFIERS)
|
||||
held_meta = set() # the Meta keys held, as they are (KEY_LEFTMETA, KEY_RIGHTMETA)
|
||||
meta_hidden = set() # held Meta keys the desktop was told came up (a combination; key_binding)
|
||||
combos_down = {} # key code -> the action its combination started (released with it)
|
||||
|
||||
def key_binding(code, value, now):
|
||||
"""A key from a keyboard: does it complete a key combination ("key_bindings")? True if
|
||||
it was taken for one (then it isn't typed)."""
|
||||
nonlocal meta_down
|
||||
if code in MODIFIERS:
|
||||
(held_modifiers.add if value else held_modifiers.discard)(MODIFIERS[code])
|
||||
if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
|
||||
(held_meta.add if value else held_meta.discard)(code)
|
||||
if value == 0 and code in meta_hidden:
|
||||
meta_hidden.discard(code)
|
||||
return True # the desktop already had it come up (below)
|
||||
return False
|
||||
if value == 1 and code not in combos_down and meta_hidden:
|
||||
# Another key while Meta is still held after a combination: the desktop gets Meta
|
||||
# back first, so Meta+that key still works there.
|
||||
for c in sorted(meta_hidden):
|
||||
to_screens(c, 1)
|
||||
meta_hidden.clear()
|
||||
if value == 0 and code in combos_down:
|
||||
action = combos_down.pop(code)
|
||||
if state["pointer"] or not needs_pointer(action):
|
||||
@@ -766,19 +803,33 @@ def main():
|
||||
if not known_action(action) or action in ("key", "none"):
|
||||
return False
|
||||
combos_down[code] = action
|
||||
if held_meta - meta_hidden:
|
||||
# The desktop saw Meta go down. Another key in between keeps its release from
|
||||
# opening Plasma's launcher (and Meta from toggling the dashboard here), and Meta
|
||||
# comes up there now: KWin takes Meta with a mouse button for moving or resizing
|
||||
# windows, which would swallow a gaze click. Its real release is dropped (above).
|
||||
meta_down = False
|
||||
to_screens(KEY_F24, 1)
|
||||
to_screens(KEY_F24, 0)
|
||||
for c in sorted(held_meta - meta_hidden):
|
||||
to_screens(c, 0)
|
||||
meta_hidden.update(held_meta)
|
||||
if state["pointer"] or not needs_pointer(action):
|
||||
do_action(action, 1, now, "keyboard")
|
||||
log(f"key combination {combo}: {action}")
|
||||
return True
|
||||
|
||||
|
||||
screens_down = set() # keys the desktop was told went down and not yet up (see reconcile_desktop_keys)
|
||||
|
||||
def to_screens(code, value):
|
||||
"""A key for the desktop screens (ft-screens decides whether it types)."""
|
||||
if value in (0, 1) and code < BTN_MISC:
|
||||
try:
|
||||
screens_sock.sendto(f"key {code} {value}".encode(), SCREENS)
|
||||
except OSError:
|
||||
pass # ft-screens not running
|
||||
return # ft-screens not running
|
||||
(screens_down.add if value else screens_down.discard)(code)
|
||||
nodes = {} # fd -> Node
|
||||
# Nodes already probed (rejected or open): path -> inode. A device that disconnects and
|
||||
# reconnects between two scans often gets the same event numbers back, so the path alone
|
||||
@@ -860,6 +911,35 @@ def main():
|
||||
mouse.sync()
|
||||
keyboard.sync()
|
||||
|
||||
def physically_down():
|
||||
"""The keys down on every device read here, as the kernel has them (EVIOCGKEY)."""
|
||||
down = set()
|
||||
for node in nodes.values():
|
||||
buf = bytearray((KEY_MAX + 8) // 8)
|
||||
try:
|
||||
fcntl.ioctl(node.fd, EVIOCGKEY, buf)
|
||||
except OSError:
|
||||
continue
|
||||
down.update(i * 8 + bit for i, b in enumerate(buf) if b for bit in range(8) if b >> bit & 1)
|
||||
return down
|
||||
|
||||
def reconcile_desktop_keys():
|
||||
"""A key the desktop has down that no device holds comes up there, and the key
|
||||
combinations forget a Meta or modifier no device holds. A key can be left down when
|
||||
its device vanishes with it held (release_held only lets go of it here) or a release
|
||||
goes astray: on 2026-10-01, after a calibration, Meta stayed down in KWin, so typing
|
||||
opened the overview and clicks on the desktop did other things."""
|
||||
if not screens_down and not held_meta and not held_modifiers:
|
||||
return
|
||||
down = physically_down()
|
||||
for code in sorted(screens_down - down):
|
||||
to_screens(code, 0)
|
||||
log(f"key {code} released on the desktop: no keyboard holds it")
|
||||
for code in [c for c in held_meta if c not in down]:
|
||||
held_meta.discard(code)
|
||||
meta_hidden.discard(code)
|
||||
held_modifiers.intersection_update({MODIFIERS[c] for c in down if c in MODIFIERS})
|
||||
|
||||
def keys_down(node):
|
||||
buf = bytearray((KEY_MAX + 8) // 8)
|
||||
try:
|
||||
@@ -988,11 +1068,16 @@ def main():
|
||||
|
||||
vr_bind(time.monotonic()) # a helper that's already running keeps its buttons in step
|
||||
waiting = False # a keyboard's grab waits for its keys to come up
|
||||
# A relay that went away with a key down left it down on the desktop, where this one
|
||||
# never sent it: modifiers come up there now (a release of a key that isn't down is nothing).
|
||||
for code in sorted(MODIFIERS):
|
||||
to_screens(code, 0)
|
||||
while True:
|
||||
now = time.monotonic()
|
||||
pointer = state["pointer"]
|
||||
if now >= next_scan:
|
||||
next_scan = now + 1.0
|
||||
reconcile_desktop_keys()
|
||||
current = {}
|
||||
for name in os.listdir("/dev/input"):
|
||||
if name.startswith("event"):
|
||||
|
||||
+324
-31
@@ -160,9 +160,14 @@
|
||||
// you were looking at: from the raw gaze when the mouse took over to where you clicked is
|
||||
// the tracker's error there. The helper sends it to ft-gazed as a lesson ("lesson <raw yaw>
|
||||
// <raw pitch> <true yaw> <true pitch>", the true direction relative to the head as it was
|
||||
// when the mouse took over) if the mouse moved between 0.2 deg and POINTER_GAZE_NUDGE_MAX
|
||||
// (8 deg) and the click came within 10 s; more is using the mouse, not a nudge. A held
|
||||
// press dragged onto the target is the same: from the raw gaze at the press to the release. With no
|
||||
// when the mouse took over) if the mouse moved at least 0.2 deg, the click came within 10 s,
|
||||
// and the correction (raw gaze to click) is within POINTER_GAZE_NUDGE_MAX (55 deg, half what
|
||||
// the headset shows across: the Frame's eyes see 109 deg each). Past that, the tracker is far
|
||||
// off (or you went somewhere else with the mouse): it isn't learned, and ft-gazed is asked
|
||||
// for its quick check instead ("recheck <deg>"; it waits out its cooldown). Our tracker's
|
||||
// clicks since its calibration put a one-dot check within 15 deg for the next 2 minutes and
|
||||
// 25 for the next 10, so the check gets back under it (2026-10-01). A held press dragged onto
|
||||
// the target is the same: from the raw gaze at the press to the release. With no
|
||||
// fresh gaze (a blink, the service stopped, the headset off), the pointer stays put.
|
||||
//
|
||||
// Gaze precision (the relay's gaze_precision and gaze_drag actions, bound to a mouse button or
|
||||
@@ -174,7 +179,40 @@
|
||||
// title bars, grab bars, and selections. Without gaze mode both still work from wherever the
|
||||
// pointer is.
|
||||
// In gaze mode the mouse's left button is a gaze precision button (POINTER_GAZE_MOUSE =
|
||||
// precision, the default), or clicks at once where the pointer is (direct).
|
||||
// precision, the default), or clicks at once where the pointer is (direct). With precision the
|
||||
// mouse's buttons work like the keyboard clicks: the right button's press is held back the same
|
||||
// way, and the right click comes on the release, where the pointer is by then (gaze_right).
|
||||
// Pressing the right button while the left one's press is held back presses the left button
|
||||
// where the pointer is now, and the mouse drags (gaze_left then gaze_right): the drag lasts
|
||||
// while either button is held. So once you've moved the pointer, the left button alone only
|
||||
// clicks; to drag from there, press the right one. Pressing the right one again (a double right
|
||||
// click, with the left still held) tilts, as a right press does during any drag (see Tilt).
|
||||
// Held still for POINTER_GAZE_HOLD, either press is a real one.
|
||||
// POINTER_GAZE_MOUSE_MOVE: held (the default) or free. Held: while the gaze has the pointer,
|
||||
// moving the mouse does nothing; it moves the pointer only during a press (as a correction,
|
||||
// like a keyboard click's head). So a bumped or drifting mouse can't pull the pointer off what
|
||||
// you're looking at, and every mouse move is a correction worth learning. With the gaze stale
|
||||
// for a second (the tracker stopped, eyes closed), in a game, or the headset off, the mouse
|
||||
// moves the pointer as usual. Free: the mouse takes the pointer whenever it moves.
|
||||
// Keyboard clicks (the relay's gaze_left and gaze_right, Meta+J and Meta+K by default;
|
||||
// "gazekey left|right 1|0" here) work like gaze mode's mouse press, steered by the head: the
|
||||
// press stops the pointer where you look, and while the keys are held the pointer stays put in
|
||||
// your view, so turning your head carries it onto what you meant (past
|
||||
// POINTER_HEAD_DEADZONE, 0.5 deg, so a still head doesn't wobble it). The release clicks there
|
||||
// (left, or right for gaze_right), and a correction is a lesson for the gaze tracker, as with
|
||||
// the mouse. Held still for POINTER_GAZE_HOLD instead, it's a real press, and the head drags.
|
||||
// A quick tap (let go within POINTER_KEY_TAP, 0.25 s) clicks where the dot was at the press,
|
||||
// whatever the head did meanwhile, and tells the gaze tracker it was right there (a lesson
|
||||
// with no correction). Pressing gaze_right while gaze_left aims (Meta+K with Meta+J held)
|
||||
// presses the left button where the dot is now, so you can correct first and then drag; the
|
||||
// drag lasts while either key is held. gaze_right pressed during a gaze_left drag (held still
|
||||
// into one, or again after starting one with it: a double Meta+K) tilts while it's held (see
|
||||
// Tilt): turning the head turns the panel, and the mouse can too; let go of it and the head
|
||||
// drags again from there. Without gaze mode they work from wherever the pointer is.
|
||||
// The gaze calibration panel (gaze/panel/ft-gazepanel, run by the gaze service): while
|
||||
// ft-gazed says it's up ("calpanel 1", renewed every second; it lapses 3 s after the last),
|
||||
// the dot hides and a press answers the panel instead of clicking: a left click or gaze_left
|
||||
// sends "calaccept" to @ft_gazed (take this dot now), a right click or gaze_right "calquit".
|
||||
// POINTER_ROLE (right, left, or stylus): the hand role our device takes while connected. A
|
||||
// Frame controller in your hand counts as used through its touch sensors and takes its hand's
|
||||
// role back, and then no click lands (see "no hand role" in the main loop): with a controller
|
||||
@@ -252,12 +290,13 @@
|
||||
// panel, and a laser that starts behind them can't hit them. POINTER_FOLLOW (0) and
|
||||
// POINTER_LEASH_DEG (10), POINTER_LEASH_DELAY (0.2 s), POINTER_LEASH_RETURN (0.2 s),
|
||||
// POINTER_FOLLOW_REACH (70 deg): head follow, above. POINTER_GAZE (0), POINTER_GAZE_RETAKE
|
||||
// (5 deg), POINTER_GAZE_NUDGE_MAX (8 deg), POINTER_GAZE_HOLD (0.5 s), POINTER_GAZE_DOT
|
||||
// (5 deg), POINTER_GAZE_NUDGE_MAX (55 deg), POINTER_GAZE_HOLD (0.5 s), POINTER_GAZE_DOT
|
||||
// (always), POINTER_GAZE_SHOW (1 s): gaze mode, above. POINTER_CONTROLLER_PICKUP (1, 0.5 to 5): how hard a controller must
|
||||
// move to take the laser back, above. POINTER_IGNORE (empty): ignored panels, above.
|
||||
// POINTER_HANDS (0), POINTER_PINCH_GAIN (0.5), POINTER_PINCH_DEADZONE (1.5 deg),
|
||||
// POINTER_GRIP_GAIN (1), POINTER_GRIP_BELOW (0.35 m), POINTER_PINCH_TYPING (1 s): hands, above.
|
||||
// POINTER_GAZE_MOUSE (precision), POINTER_ROLE (right): gaze precision, above.
|
||||
// POINTER_GAZE_MOUSE (precision), POINTER_HEAD_DEADZONE (0.5 deg), POINTER_KEY_TAP (0.25 s),
|
||||
// POINTER_ROLE (right): gaze precision and keyboard clicks, above.
|
||||
#include <openvr.h>
|
||||
|
||||
#include "vrbuttons.h"
|
||||
@@ -520,6 +559,7 @@ private:
|
||||
if (rest.find("Thumbnail") != std::string::npos || rest.find("Subview") != std::string::npos) continue;
|
||||
if (key.rfind("system.pointer", 0) == 0 || key.rfind("system.cursor", 0) == 0 ||
|
||||
key.rfind("frametop.pointer", 0) == 0 || key.rfind("frametop.guide", 0) == 0 ||
|
||||
key == "frametop.gazepanel" || // the gaze calibration panel, fixed to the headset
|
||||
key == "frametop.catcher" || // ft-screens' release catcher: only on a laser mid-drag
|
||||
key == "system.HeadsetView" || key == "system.toast")
|
||||
continue;
|
||||
@@ -664,14 +704,16 @@ int main() {
|
||||
bool follow = false, followConf = false, followReset = true;
|
||||
// Gaze mode (see the top); gazeConf is POINTER_GAZE as last read, like followConf.
|
||||
bool gazeOn = false, gazeConf = false;
|
||||
double gazeRetake = 5, gazeNudgeMax = 8, gazeHold = 0.5, gazeShow = 1;
|
||||
double gazeRetake = 5, gazeNudgeMax = 55, gazeHold = 0.5, gazeShow = 1;
|
||||
bool gazeDotAlways = true; // POINTER_GAZE_DOT (see the top)
|
||||
double headDeadzone = 0.5, keyTap = 0.25; // POINTER_HEAD_DEADZONE, POINTER_KEY_TAP (keyboard clicks, see the top)
|
||||
// Hands (see the top): POINTER_HANDS, POINTER_PINCH_GAIN, POINTER_PINCH_DEADZONE, POINTER_GRIP_GAIN.
|
||||
bool handsOn = false;
|
||||
double pinchGain = 0.5, pinchDeadzone = 1.5, gripGain = 1.0, handBelow = 0.35, typingHold = 1.0;
|
||||
// Gaze precision (see the top): POINTER_GAZE_MOUSE (precision: the mouse's left button
|
||||
// holds back its press in gaze mode; direct: it clicks at once), POINTER_ROLE.
|
||||
bool gazeMousePrecision = true;
|
||||
bool gazeMouseHeld = true; // POINTER_GAZE_MOUSE_MOVE (see the top)
|
||||
std::string role = "right";
|
||||
double pickupScale = 1; // POINTER_CONTROLLER_PICKUP: scales the controller-moved limits
|
||||
std::vector<std::string> ignore; // POINTER_IGNORE (see ParseIgnore)
|
||||
@@ -692,9 +734,11 @@ int main() {
|
||||
const bool wantFollow = ConfDouble(conf, "POINTER_FOLLOW", 0) != 0;
|
||||
if (wantFollow != followConf) follow = followConf = wantFollow, followReset = true;
|
||||
gazeRetake = std::clamp(ConfDouble(conf, "POINTER_GAZE_RETAKE", 5), 1.0, 45.0);
|
||||
gazeNudgeMax = std::clamp(ConfDouble(conf, "POINTER_GAZE_NUDGE_MAX", 8), 1.0, 30.0);
|
||||
gazeNudgeMax = std::clamp(ConfDouble(conf, "POINTER_GAZE_NUDGE_MAX", 55), 1.0, 110.0);
|
||||
gazeHold = std::clamp(ConfDouble(conf, "POINTER_GAZE_HOLD", 0.5), 0.1, 5.0);
|
||||
gazeShow = std::clamp(ConfDouble(conf, "POINTER_GAZE_SHOW", 1), 0.0, 30.0);
|
||||
headDeadzone = std::clamp(ConfDouble(conf, "POINTER_HEAD_DEADZONE", 0.5), 0.0, 5.0);
|
||||
keyTap = std::clamp(ConfDouble(conf, "POINTER_KEY_TAP", 0.25), 0.0, 1.0);
|
||||
const auto gd = conf.find("POINTER_GAZE_DOT");
|
||||
gazeDotAlways = gd == conf.end() || gd->second != "moving";
|
||||
const bool wantGaze = ConfDouble(conf, "POINTER_GAZE", 0) != 0;
|
||||
@@ -707,6 +751,8 @@ int main() {
|
||||
typingHold = std::clamp(ConfDouble(conf, "POINTER_PINCH_TYPING", 1.0), 0.0, 5.0);
|
||||
const auto gm = conf.find("POINTER_GAZE_MOUSE");
|
||||
gazeMousePrecision = gm == conf.end() || gm->second != "direct";
|
||||
const auto mm = conf.find("POINTER_GAZE_MOUSE_MOVE");
|
||||
gazeMouseHeld = mm == conf.end() || mm->second != "free";
|
||||
const auto ro = conf.find("POINTER_ROLE");
|
||||
role = ro != conf.end() && (ro->second == "left" || ro->second == "stylus") ? ro->second : "right";
|
||||
pickupScale = std::clamp(ConfDouble(conf, "POINTER_CONTROLLER_PICKUP", 1), 0.5, 5.0);
|
||||
@@ -787,6 +833,15 @@ int main() {
|
||||
Clock::time_point movingSince[vr::k_unMaxTrackedDeviceCount] = {};
|
||||
// Tilt mode (see top of file).
|
||||
bool leftHeld = false, tilting = false, tiltStart = false, swallowedRight = false;
|
||||
// The mouse's buttons in gaze mode (see the top): leftDown, rightDown as the relay last said;
|
||||
// aimRight, the held-back press is the right button's; chordDrag, a drag the right button
|
||||
// began during the left's held-back press (the first release drops it, the other's is nothing).
|
||||
bool leftDown = false, rightDown = false, aimRight = false, chordDrag = false;
|
||||
bool ignoreLeftUp = false;
|
||||
// The keyboard clicks' keys as the relay last said, and a tilt gaze_right holds during a
|
||||
// keyboard drag (see the top): the head turns the panel, from where it was (keyTiltYaw, keyTiltPitch).
|
||||
bool keyLeftDown = false, keyRightDown = false, keyTilting = false;
|
||||
double keyTiltYaw = 0, keyTiltPitch = 0;
|
||||
double tiltYaw = 0, tiltPitch = 0;
|
||||
double dragDistance = 0, lastDistance = 1.5; // drag lock: distance from the anchor at the press
|
||||
bool onVrSettings = false; // the cursor is on the SteamVR Settings page (kept while dragging)
|
||||
@@ -844,13 +899,17 @@ int main() {
|
||||
// hand: seen from the eye then, in the room), and the pointer then.
|
||||
HandGestures handFile;
|
||||
PoseHistory poses;
|
||||
enum class Src { None, Hand, Mouse };
|
||||
enum class Src { None, Hand, Mouse, Head };
|
||||
struct Hold {
|
||||
Src src = Src::None;
|
||||
int side = -1; // a hand's side (0 left, 1 right)
|
||||
bool grip = false; // pressed at once and dragging (a grip, gaze drag), not a click on release
|
||||
bool engaged = false; // past the dead zone
|
||||
bool pressed = false; // a real press went out (a grip or gaze drag, or a pinch without gaze mode)
|
||||
bool promote = false; // held still for POINTER_GAZE_HOLD, it becomes a real press (keyboard clicks)
|
||||
bool right = false; // the right button (gaze_right)
|
||||
bool keyDrag = false; // gaze_right pressed while gaze_left aimed: a left press, either key ends it
|
||||
double pressYaw = 0, pressPitch = 0; // the pointer at the press (a quick tap clicks there)
|
||||
Vec3 origin;
|
||||
double refYaw = 0, refPitch = 0, startYaw = 0, startPitch = 0, lastYaw = 0, lastPitch = 0;
|
||||
} hold;
|
||||
@@ -860,6 +919,11 @@ int main() {
|
||||
bool drag, down;
|
||||
};
|
||||
std::vector<DevicePress> devicePresses;
|
||||
// "gazekey left|right 1|0" from the relay (keyboard clicks), done in the frame.
|
||||
struct KeyPress {
|
||||
bool right, down;
|
||||
};
|
||||
std::vector<KeyPress> keyPresses;
|
||||
uint32_t seenBegins[2][2] = {}, seenEnds[2][2] = {}; // [pinch, grip][side], as last read
|
||||
bool handBaseline = false;
|
||||
int handOpens = 0;
|
||||
@@ -870,23 +934,36 @@ int main() {
|
||||
bool inGame = false, gazeAwake = false;
|
||||
Clock::time_point inGameAt{}, gazeAwakeAt{};
|
||||
Clock::time_point lastMove{}, lastHeld{}, pulseAt{};
|
||||
// The left button, as sent to the driver.
|
||||
// The left button, as sent to the driver; pressRight: the next press is the right button
|
||||
// instead (gaze_right), heldButton: the one pressed.
|
||||
bool pressRight = false;
|
||||
std::string heldButton = "trigger";
|
||||
bool confirmLesson = false; // a keyboard click's quick tap: a lesson with no correction (see the top)
|
||||
// The gaze calibration panel is up until then ("calpanel 1"; see the top); calOpened: it just
|
||||
// came up, so a press in progress ends without a click.
|
||||
Clock::time_point calPanelUntil{};
|
||||
bool calOpened = false;
|
||||
auto pressLeft = [&] {
|
||||
// ft-screens sends the keyboard to the panel clicked last; it sees clicks on
|
||||
// its own screens, but only we know when one lands on another panel.
|
||||
SendTo(out, "ft_screens", "click " + (lastHit.empty() ? std::string("-") : lastHit));
|
||||
// A click after nudging the gaze-placed pointer: the nudge is a lesson (see the top).
|
||||
// A click after nudging the gaze-placed pointer: the nudge is a lesson, or past
|
||||
// POINTER_GAZE_NUDGE_MAX, a quick check (see the top).
|
||||
if (gazeOn && nudging && !gazeOwns && havePoint && Clock::now() - nudgeAt < std::chrono::seconds(10) &&
|
||||
nudgeMoved >= 0.2 && nudgeMoved <= gazeNudgeMax) {
|
||||
(confirmLesson || nudgeMoved >= 0.2)) {
|
||||
const Vec3 d = RotateInverse(nudgeHead, Normalize(lastPoint - Position(nudgeHead)));
|
||||
const double ty = std::atan2(-d.x, -d.z) * 180 / M_PI, tp = std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI;
|
||||
const double off = std::hypot(std::remainder(ty - nudgeRawHy, 360.0), tp - nudgeRawHp);
|
||||
char msg[160];
|
||||
std::snprintf(msg, sizeof msg, "lesson %.3f %.3f %.3f %.3f", nudgeRawHy, nudgeRawHp,
|
||||
std::atan2(-d.x, -d.z) * 180 / M_PI, std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI);
|
||||
if (off <= gazeNudgeMax)
|
||||
std::snprintf(msg, sizeof msg, "lesson %.3f %.3f %.3f %.3f", nudgeRawHy, nudgeRawHp, ty, tp);
|
||||
else
|
||||
std::snprintf(msg, sizeof msg, "recheck %.0f", off);
|
||||
SendTo(out, "ft_gazed", msg);
|
||||
if (debug) std::printf("gaze %s (nudged %.2f deg)\n", msg, nudgeMoved);
|
||||
if (debug) std::printf("gaze %s (nudged %.2f deg, off %.2f)\n", msg, nudgeMoved, off);
|
||||
if (debug) std::fflush(stdout);
|
||||
}
|
||||
nudging = false;
|
||||
nudging = confirmLesson = false;
|
||||
leftHeld = true;
|
||||
dragDistance = lastDistance;
|
||||
pressKey.clear();
|
||||
@@ -900,14 +977,16 @@ int main() {
|
||||
tiltYaw = tiltPitch = 0; // a new drag starts untilted
|
||||
dropHoldUntil = {};
|
||||
pulseAt = Clock::now();
|
||||
SendTo(out, "ft_pointer", "btn trigger 1");
|
||||
heldButton = pressRight ? "b" : "trigger";
|
||||
pressRight = false;
|
||||
SendTo(out, "ft_pointer", "btn " + heldButton + " 1");
|
||||
};
|
||||
auto releaseLeft = [&] {
|
||||
leftHeld = false;
|
||||
tilting = false;
|
||||
// Hold the drag pose (tilt, frozen distance) while SteamVR finishes the drop.
|
||||
dropHoldUntil = Clock::now() + std::chrono::milliseconds(500);
|
||||
SendTo(out, "ft_pointer", "btn trigger 0");
|
||||
SendTo(out, "ft_pointer", "btn " + heldButton + " 0");
|
||||
// ft-screens releases a button held on its screens in KWin even when SteamVR hands
|
||||
// the release to some other overlay (its catcher usually gets it; this is the backstop).
|
||||
SendTo(out, "ft_screens", "up");
|
||||
@@ -915,22 +994,48 @@ int main() {
|
||||
};
|
||||
// The left button, from the relay's "btn trigger", with gaze mode's held-back press (see
|
||||
// the top).
|
||||
auto canAim = [&] {
|
||||
return gazeOn && gazeMousePrecision && gazeOwns && !aimHeld && !clickPress && !clickRelease &&
|
||||
hold.src == Src::None;
|
||||
};
|
||||
// Hold the press back: the pointer stops where the gaze put it.
|
||||
auto aimStart = [&](bool right) {
|
||||
gazeOwns = false;
|
||||
nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
|
||||
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
|
||||
nudgeAt = aimSince = Clock::now(), nudgeMoved = 0;
|
||||
aimHeld = true, aimRight = right;
|
||||
};
|
||||
// A drag the right button began (chordDrag): it lasts while either button is held.
|
||||
auto chordDrop = [&] {
|
||||
chordDrag = false;
|
||||
if (leftHeld) releaseLeft();
|
||||
if (debug) std::printf("mouse drag dropped\n");
|
||||
if (debug) std::fflush(stdout);
|
||||
};
|
||||
auto leftButton = [&](bool down) {
|
||||
leftDown = down;
|
||||
if (down) {
|
||||
if (gazeOn && gazeMousePrecision && gazeOwns && !aimHeld && !clickPress && !clickRelease &&
|
||||
hold.src == Src::None) {
|
||||
// Hold the press back: the pointer stops where the gaze put it.
|
||||
gazeOwns = false;
|
||||
nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
|
||||
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
|
||||
nudgeAt = aimSince = Clock::now(), nudgeMoved = 0;
|
||||
aimHeld = true;
|
||||
if (aimHeld && aimRight && !aimHand) {
|
||||
ignoreLeftUp = true; // the right's press is held back: the left does nothing
|
||||
return;
|
||||
}
|
||||
if (canAim()) {
|
||||
aimStart(false);
|
||||
return;
|
||||
}
|
||||
pressLeft();
|
||||
return;
|
||||
}
|
||||
if (aimHeld && !aimHand) {
|
||||
if (ignoreLeftUp) {
|
||||
ignoreLeftUp = false;
|
||||
return;
|
||||
}
|
||||
if (chordDrag) {
|
||||
if (!rightDown) chordDrop(); // otherwise the right holds it (tilting, maybe)
|
||||
return;
|
||||
}
|
||||
if (aimHeld && !aimHand && !aimRight) {
|
||||
aimHeld = false;
|
||||
clickPress = true; // after this frame's pose, so it lands where the pointer was moved to
|
||||
gazeBack = nudgeMoved < 0.2;
|
||||
@@ -938,6 +1043,50 @@ int main() {
|
||||
}
|
||||
if (leftHeld) releaseLeft();
|
||||
};
|
||||
// The right button (see the top); false: not taken here (a tilt, or passed on as it is).
|
||||
auto rightButton = [&](bool down) {
|
||||
rightDown = down;
|
||||
if (down) {
|
||||
if (aimHeld && !aimHand && !aimRight) {
|
||||
// During the left's held-back press: press the left where the pointer is now (the
|
||||
// correction is a lesson), and the mouse drags.
|
||||
aimHeld = false;
|
||||
chordDrag = gazeBack = true;
|
||||
pressLeft();
|
||||
if (debug) std::printf("mouse drag began (right during left)\n");
|
||||
if (debug) std::fflush(stdout);
|
||||
return true;
|
||||
}
|
||||
if (canAim() && !leftHeld) {
|
||||
aimStart(true);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
if (chordDrag) {
|
||||
if (leftDown) return !swallowedRight; // the left holds it; a tilt's release ends the tilt
|
||||
tilting = swallowedRight = false;
|
||||
chordDrop();
|
||||
return true;
|
||||
}
|
||||
if (aimHeld && !aimHand && aimRight) {
|
||||
aimHeld = aimRight = false;
|
||||
pressRight = clickPress = true; // the right click, where the pointer was moved to
|
||||
gazeBack = nudgeMoved < 0.2;
|
||||
return true;
|
||||
}
|
||||
if (leftHeld && heldButton == "b") { // its press, held still into a real one
|
||||
releaseLeft();
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
// POINTER_GAZE_MOUSE_MOVE=held (see the top): the gaze is fresh and nothing is pressed, so a
|
||||
// mouse move doesn't move the pointer.
|
||||
auto mouseMoveHeld = [&] {
|
||||
return gazeOn && gazeMouseHeld && !inGame && !headsetOff && Clock::now() - gz.at < std::chrono::seconds(1) &&
|
||||
!aimHeld && !leftHeld && !tilting && hold.src == Src::None && !clickPress && !clickRelease;
|
||||
};
|
||||
auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10);
|
||||
|
||||
// --- Panel placement (see "Placement" at the top of the file) ---
|
||||
@@ -1157,6 +1306,27 @@ int main() {
|
||||
gz = {g[0], g[1], g[2], g[3], Clock::now()};
|
||||
continue;
|
||||
}
|
||||
// The gaze calibration panel (see the top): presses answer it instead of clicking.
|
||||
{
|
||||
int on;
|
||||
if (std::sscanf(buf, "calpanel %d", &on) == 1) {
|
||||
const bool was = Clock::now() < calPanelUntil;
|
||||
calPanelUntil = on ? Clock::now() + std::chrono::seconds(3) : Clock::time_point{};
|
||||
if (on && !was) calOpened = true;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (Clock::now() < calPanelUntil) {
|
||||
const bool accept = !std::strncmp(buf, "btn trigger 1", 13) || !std::strncmp(buf, "gazekey left 1", 14);
|
||||
const bool quit = !std::strncmp(buf, "btn b 1", 7) || !std::strncmp(buf, "gazekey right 1", 15);
|
||||
if (accept || quit) {
|
||||
SendTo(out, "ft_gazed", accept ? "calaccept" : "calquit");
|
||||
continue;
|
||||
}
|
||||
if (!std::strncmp(buf, "btn ", 4) || !std::strncmp(buf, "gazekey ", 8) ||
|
||||
!std::strncmp(buf, "precision ", 10) || !std::strncmp(buf, "gazedrag ", 9))
|
||||
continue; // their releases, and the other buttons: nothing to click now
|
||||
}
|
||||
{
|
||||
char kind[16], source[16];
|
||||
int v;
|
||||
@@ -1167,6 +1337,13 @@ int main() {
|
||||
devicePresses.push_back({source, !std::strcmp(kind, "gazedrag"), v != 0});
|
||||
continue;
|
||||
}
|
||||
if (std::sscanf(buf, "gazekey %15s %d", source, &v) == 2 &&
|
||||
(!std::strcmp(source, "left") || !std::strcmp(source, "right"))) {
|
||||
lastMouse = Clock::now();
|
||||
if (!active) wake(Clock::now());
|
||||
keyPresses.push_back({!std::strcmp(source, "right"), v != 0});
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (std::strcmp(buf, "typing") == 0) { // not mouse input: it never wakes the pointer
|
||||
lastTyping = Clock::now();
|
||||
@@ -1208,9 +1385,13 @@ int main() {
|
||||
}
|
||||
const bool mouseInput = std::strncmp(buf, "move", 4) == 0 || std::strncmp(buf, "btn", 3) == 0 ||
|
||||
std::strncmp(buf, "scroll", 6) == 0;
|
||||
if (mouseInput) lastMouse = Clock::now();
|
||||
// A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using
|
||||
// the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back.
|
||||
const bool moveHeldBack = std::strncmp(buf, "move", 4) == 0 && mouseMoveHeld();
|
||||
if (mouseInput && !moveHeldBack) lastMouse = Clock::now();
|
||||
// Any mouse input wakes the pointer (after a controller took over, or a helper restart).
|
||||
if (!active && mouseInput) wake(Clock::now());
|
||||
if (moveHeldBack) continue;
|
||||
double a, b;
|
||||
char key[128];
|
||||
double px, py, pz, pyaw, ppitch, proll = 0, pgrab = -1;
|
||||
@@ -1259,6 +1440,10 @@ int main() {
|
||||
} else if (std::strncmp(buf, "btn trigger 0", 13) == 0) {
|
||||
leftButton(false);
|
||||
continue;
|
||||
} else if (std::strncmp(buf, "btn b 1", 7) == 0 && rightButton(true)) {
|
||||
continue;
|
||||
} else if (std::strncmp(buf, "btn b 0", 7) == 0 && rightButton(false)) {
|
||||
continue;
|
||||
} else if (std::strncmp(buf, "btn b 1", 7) == 0 && leftHeld) {
|
||||
tilting = tiltStart = swallowedRight = true; // right press while dragging: tilt, no right-click
|
||||
continue;
|
||||
@@ -1454,11 +1639,14 @@ int main() {
|
||||
if (gazeOn) gazeOwns = true, nudging = false; // no click: the pointer goes back to the gaze
|
||||
} else {
|
||||
clickPress = true; // the click is on the release, where the pointer is now
|
||||
pressRight = hold.right;
|
||||
gazeBack = nudgeMoved < 0.2;
|
||||
}
|
||||
}
|
||||
if (debug)
|
||||
std::printf("hold %s %s%s\n", hold.src == Src::Hand ? (hold.grip ? "grip" : "pinch") : hold.grip ? "gaze drag" : "gaze precision",
|
||||
std::printf("hold %s %s%s\n", hold.src == Src::Hand ? (hold.grip ? "grip" : "pinch")
|
||||
: hold.src == Src::Head ? (hold.right ? "key right" : "key left")
|
||||
: hold.grip ? "gaze drag" : "gaze precision",
|
||||
lost ? "lost" : "released", hold.pressed ? "" : lost ? " (no click)" : " (click)");
|
||||
if (debug) std::fflush(stdout);
|
||||
hold = {};
|
||||
@@ -1493,7 +1681,12 @@ int main() {
|
||||
}
|
||||
yaw = std::remainder(hold.startYaw + gain * dy, 360.0);
|
||||
pitch = std::clamp(hold.startPitch + gain * dp, -85.0, 85.0);
|
||||
if (!hold.pressed) nudgeMoved += std::hypot(std::remainder(yaw - hold.lastYaw, 360.0), pitch - hold.lastPitch);
|
||||
// How far the nudge went: for a keyboard click, from where the dot was at the press
|
||||
// (a head wobbles on the way); otherwise the path, as with the mouse.
|
||||
if (!hold.pressed && hold.src == Src::Head)
|
||||
nudgeMoved = std::hypot(std::remainder(yaw - hold.pressYaw, 360.0), pitch - hold.pressPitch);
|
||||
else if (!hold.pressed)
|
||||
nudgeMoved += std::hypot(std::remainder(yaw - hold.lastYaw, 360.0), pitch - hold.lastPitch);
|
||||
hold.lastYaw = yaw, hold.lastPitch = pitch;
|
||||
lastMove = tnow; // the dot shows while it moves (gaze mode)
|
||||
};
|
||||
@@ -1551,6 +1744,85 @@ int main() {
|
||||
if (debug) std::fflush(stdout);
|
||||
}
|
||||
devicePresses.clear();
|
||||
// Keyboard clicks (see the top): held back at the gaze, steered by the head.
|
||||
auto headAngles = [&](double &hy, double &hp) {
|
||||
if (!hmd.bPoseIsValid) return false;
|
||||
const Vec3 f = Normalize({-hm[0][2], -hm[1][2], -hm[2][2]});
|
||||
hy = std::atan2(-f.x, -f.z) * 180 / M_PI;
|
||||
hp = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
|
||||
return true;
|
||||
};
|
||||
for (const KeyPress &k : keyPresses) {
|
||||
(k.right ? keyRightDown : keyLeftDown) = k.down;
|
||||
if (!k.down) {
|
||||
if (keyTilting && k.right) { // the tilt ends; the head drags again from here
|
||||
keyTilting = tilting = false;
|
||||
hold.engaged = false;
|
||||
if (debug) std::printf("hold key left: tilt ended\n");
|
||||
if (debug) std::fflush(stdout);
|
||||
}
|
||||
// The keys holding it: a gaze_left then gaze_right drag, either; otherwise its own.
|
||||
const bool held = hold.keyDrag ? keyLeftDown || keyRightDown : hold.right ? keyRightDown : keyLeftDown;
|
||||
if (hold.src == Src::Head && !held) {
|
||||
if (!hold.pressed && aimHeld && tnow - aimSince < std::chrono::duration<double>(keyTap)) {
|
||||
// A quick tap: a click where the dot was at the press, and the gaze was right.
|
||||
yaw = hold.pressYaw, pitch = hold.pressPitch;
|
||||
nudgeMoved = 0;
|
||||
confirmLesson = true;
|
||||
}
|
||||
endHold(false);
|
||||
keyTilting = false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (k.right && hold.src == Src::Head && !hold.right && hold.pressed && leftHeld) {
|
||||
// gaze_right during a gaze_left drag: tilt while it's held (see the top).
|
||||
tilting = tiltStart = keyTilting = true;
|
||||
headAngles(keyTiltYaw, keyTiltPitch);
|
||||
if (debug) std::printf("hold key left: tilt began\n");
|
||||
if (debug) std::fflush(stdout);
|
||||
continue;
|
||||
}
|
||||
if (k.right && hold.src == Src::Head && !hold.right && !hold.pressed && aimHeld) {
|
||||
// gaze_right while gaze_left aims: press the left button where the dot is now
|
||||
// (the correction is a lesson), then the head drags.
|
||||
aimHeld = aimHand = false;
|
||||
hold.pressed = hold.grip = hold.keyDrag = true, hold.promote = false, hold.engaged = false;
|
||||
headAngles(hold.refYaw, hold.refPitch);
|
||||
gazeBack = gazeOn;
|
||||
pressRight = false;
|
||||
pressLeft();
|
||||
if (debug) std::printf("hold key left: pressed (right key)\n");
|
||||
if (debug) std::fflush(stdout);
|
||||
continue;
|
||||
}
|
||||
if (hold.src != Src::None || leftHeld || aimHeld || clickPress || clickRelease) continue;
|
||||
Hold h;
|
||||
h.src = Src::Head, h.promote = true, h.right = k.right, h.pressYaw = yaw, h.pressPitch = pitch;
|
||||
if (!headAngles(h.refYaw, h.refPitch)) continue;
|
||||
hold = h;
|
||||
startHold(false);
|
||||
if (debug) std::printf("hold key %s began\n", k.right ? "right" : "left");
|
||||
if (debug) std::fflush(stdout);
|
||||
}
|
||||
keyPresses.clear();
|
||||
if (calOpened) { // the calibration panel came up: a press in progress ends, no click
|
||||
calOpened = false;
|
||||
if (hold.src != Src::None) endHold(true);
|
||||
if (leftHeld) releaseLeft();
|
||||
aimHeld = aimHand = clickPress = false;
|
||||
}
|
||||
if (hold.src == Src::Head) {
|
||||
double hy, hp;
|
||||
if (headAngles(hy, hp) && keyTilting) {
|
||||
// The head turns the panel, as the mouse does in a tilt; the pointer stays put.
|
||||
tiltYaw += std::remainder(hy - keyTiltYaw, 360.0);
|
||||
tiltPitch = std::clamp(tiltPitch + hp - keyTiltPitch, -80.0, 80.0);
|
||||
keyTiltYaw = hy, keyTiltPitch = hp;
|
||||
} else if (headAngles(hy, hp)) {
|
||||
steer(hy, hp, hold.pressed ? 0.3 : headDeadzone, 1.0);
|
||||
}
|
||||
}
|
||||
fh_gestures_t hg;
|
||||
const bool handOk = handsOn && handFile.Read(hg);
|
||||
if (handOk && (hg.seq != handSeq || handFile.opens != handOpens)) {
|
||||
@@ -1857,6 +2129,7 @@ int main() {
|
||||
auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
|
||||
alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
|
||||
}
|
||||
if (tnow < calPanelUntil) alpha = 0;
|
||||
overlay->SetOverlayAlpha(marker, float(alpha));
|
||||
overlay->SetOverlayWidthInMeters(marker, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360)));
|
||||
auto mm = Billboard(near, eye);
|
||||
@@ -1886,6 +2159,7 @@ int main() {
|
||||
alpha = std::max(alpha, std::clamp((0.6 - pulse) / 0.3, 0.0, 1.0));
|
||||
}
|
||||
}
|
||||
if (tnow < calPanelUntil) alpha = 0; // the calibration panel is up (see the top)
|
||||
overlay->SetOverlayAlpha(show, float(alpha));
|
||||
overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360)));
|
||||
auto m = Billboard(at, eye);
|
||||
@@ -1945,12 +2219,31 @@ int main() {
|
||||
|
||||
// A held-back press (see the top): held still long enough, it's a real press (a drag);
|
||||
// released, it's a click where the pointer is now (this frame's pose has gone out).
|
||||
if (!active) aimHeld = clickPress = aimHand = false; // released meanwhile: nothing to click
|
||||
if (!active) aimHeld = aimRight = clickPress = aimHand = confirmLesson = false; // released meanwhile: nothing to click
|
||||
if (aimHeld && !aimHand && nudgeMoved < 0.2 && tnow - aimSince >= std::chrono::duration<double>(gazeHold)) {
|
||||
aimHeld = false;
|
||||
gazeBack = true;
|
||||
pressRight = aimRight; // the right button's: a real right press (see the top)
|
||||
aimRight = false;
|
||||
pressLeft();
|
||||
}
|
||||
// A keyboard click held still for POINTER_GAZE_HOLD: a real press, then the head drags
|
||||
// (from where it is now).
|
||||
if (aimHeld && aimHand && hold.promote && !hold.engaged &&
|
||||
tnow - aimSince >= std::chrono::duration<double>(gazeHold)) {
|
||||
aimHeld = aimHand = false;
|
||||
hold.pressed = hold.grip = true;
|
||||
if (hmd.bPoseIsValid) {
|
||||
const Vec3 f = Normalize({-hm[0][2], -hm[1][2], -hm[2][2]});
|
||||
hold.refYaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
|
||||
hold.refPitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
|
||||
}
|
||||
gazeBack = gazeOn;
|
||||
pressRight = hold.right;
|
||||
pressLeft();
|
||||
if (debug) std::printf("hold key %s: pressed (held still)\n", hold.right ? "right" : "left");
|
||||
if (debug) std::fflush(stdout);
|
||||
}
|
||||
if (clickPress) {
|
||||
clickPress = false;
|
||||
pressLeft();
|
||||
|
||||
@@ -29,11 +29,14 @@ POINTER_LEASH_RETURN=0.2 # head follow: once it comes along, how quickly (s) t
|
||||
POINTER_FOLLOW_REACH=70 # head follow: how far (degrees) from the middle of your view the mouse can move the cursor
|
||||
POINTER_GAZE=0 # 1 = the pointer goes where you look, the mouse does the last bit (needs the gaze service: gaze/run.sh install)
|
||||
POINTER_GAZE_RETAKE=5 # gaze mode: how far (degrees) you look away from the pointer before the gaze takes it back from the mouse
|
||||
POINTER_GAZE_NUDGE_MAX=8 # gaze mode: a mouse nudge up to this far (degrees) before a click is learned as the eye tracker's error
|
||||
POINTER_GAZE_NUDGE_MAX=55 # gaze mode: a correction up to this far (degrees) before a click is learned as the eye tracker's error; a bigger one runs the quick check (55: half the headset's view)
|
||||
POINTER_GAZE_HOLD=0.5 # gaze mode: a press held this long (s) without moving becomes a real press (to drag); moved or released sooner, it clicks where you let go
|
||||
POINTER_GAZE_MOUSE_MOVE=held # gaze mode: the mouse moves the pointer only while a button is held, as a correction (held) | any time (free)
|
||||
POINTER_GAZE_DOT=always # gaze mode: the dot shows all the time (always) | only while the mouse moves it (moving)
|
||||
POINTER_GAZE_SHOW=1 # gaze mode, with POINTER_GAZE_DOT=moving: the dot shows this long (s) after the mouse moves it; also while a press is held, and a pulse per click
|
||||
POINTER_GAZE_MOUSE=precision # gaze mode: the left button holds back its press, the mouse steers, the release clicks (precision) | clicks right away (direct)
|
||||
POINTER_HEAD_DEADZONE=0.5 # keyboard clicks at the gaze (Meta+J, Meta+K): degrees the head turns before the held dot moves with it
|
||||
POINTER_KEY_TAP=0.25 # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
|
||||
GAZE_TRACKER=steam # gaze service: steam = SteamVR's eye tracker | own = our own (gaze/tracker: its frame grabber needs gaze/tracker/install.sh; calibrated in the gaze probe with Own tracker)
|
||||
GAZE_EYE=auto # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
|
||||
HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
|
||||
|
||||
Reference in new issue
Block a user