Input Settings: the Games page is Game optimization

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitignore

@@ -5,3 +5,9 @@ target/
 captures/
 __pycache__/
 frametop-report-*.txt
+# Eye-camera recordings (biometric) never go in the repo: they live in
+# ~/.local/share/frametop/eyes/captures. These catch strays (frame dumps, a lab venv).
+*.raw
+*.pgm
+.venv/
+.frame-job.d/

AGENTS.md

@@ -25,10 +25,14 @@ scripts/frame.sh --host '<cmd>'          # runs on the SteamOS host
 A Steam Frame is someone's personal headset, and they may be wearing it while you work.
 
 - Don't kill or restart `gamescope`, `steam`, `vrserver`, `vrcompositor`, the gamescope session, or the Frametop desktop without asking. Each one ends or disrupts whatever is happening in VR.
-- Don't run host `sudo`, `steamos-readonly disable`, `steamos-devmode` changes, pacman installs, or reboots without explicit approval. Only the Bluetooth fixes need host `sudo`, and they ask.
-- Write only inside the repo, `/tmp`, and the container unless told otherwise. The installers are the exception: they write the user services, launchers, and the SteamVR driver into the home folder.
+- Don't run host `sudo`, `steamos-readonly disable`, `steamos-devmode` changes, pacman installs, or reboots without explicit approval. Three installers need host `sudo`, and they ask for it: the Bluetooth fixes (`setup/bluetooth/install.sh`), hand tracking (`hands/run.sh install` and `caps`, which set ft-camd's file capabilities with `setcap`), and our own eye tracker's frame grabber (`gaze/tracker/install.sh`).
+- Write only inside the repo, `/tmp`, and the container unless told otherwise. The installers are the exception: they write the user services, launchers, and the SteamVR driver into the home folder. The Bluetooth fixes and the eye tracker's frame grabber also install root-owned files and system services under `/etc` (`/etc/steamframe`, `/etc/frametop`, `/etc/systemd/system`).
 - Never copy `.netrc`, SSH keys, or Steam config off the Frame or into this repo.
 
+## SteamOS updates
+
+A SteamOS update replaces SteamVR, KWin, and gamescope with the rest of the OS image. When a change starts depending on something from the image (a host file, an OpenVR interface outside the bundled header, an undocumented layout or output format, a SteamVR or KWin quirk), add a check for it to `scripts/update-check.py`, or a retest hint for its package there. [docs/design.md](docs/design.md) has the background.
+
 ## Names
 
 User-facing names are "Frametop", "Frametop Display Settings", and "Frametop Input Settings". Programs and files use the `ft-` / `ft_` prefix (`ft-screens`, `ft-pointer`, `ft-layout`, the `ft_pointer` driver); config, units, and overlay keys use `frametop`. Program names must stay within 15 characters: Linux truncates process names there, and the scripts find programs with `pgrep -x` / `pkill -x`.

README.md

@@ -1,34 +1,41 @@
 # Frametop
 
-Frametop puts a multi-monitor KDE Plasma desktop into SteamVR on the Valve Steam Frame, and lets a Bluetooth mouse drive all of SteamVR. It installs and runs on the headset itself.
+Frametop is a desktop for the Steam Frame that runs on the headset itself, with no PC. Put several monitors around you, pull app windows out to float on their own, and point with a mouse, or with your eyes.
 
-Each screen is its own monitor with its own resolution, so you can have an ultrawide in the middle and two portrait screens beside it, at whatever size and distance you like. The screens come back to your saved layout when the desktop starts. You can move, resize, curve, and roll them, pin one to your wrist, and put them all back with a shortcut.
+- **Screens that are real monitors.** Each KDE Plasma screen has its own resolution and shape: an ultrawide in front, portrait screens beside it. Move, resize, curve, and roll them, or pin one to your wrist or your head. They come back to your layout when the desktop starts.
+- **Windows that float on their own.** Take any app window off the screens into a panel of its own with Meta+Shift+F, its title bar button, or Launch as Standalone. It stays part of the desktop, so drag and drop and the clipboard still work between floating windows and the screens.
+- **Profiles.** Save where your screens are, which ones show, and which apps are open where. Switch to a profile from Display Settings, a key, or its launcher entry, or start the desktop in one.
+- **One mouse for all of SteamVR.** A Bluetooth mouse drives a small dot anchored in the room. It works the screens, the dashboard, Steam, and overlays, and hands the laser back when you pick up a controller.
+- **Look and click (experimental).** In gaze mode the pointer goes where you look. Meta+J and Meta+K, or the mouse buttons, click and fine-tune, and each correction teaches the tracker. Calibration runs in the headset.
+- **Made for long sessions.** The displays turn off when the headset isn't used, even on a stand that makes it seem worn. It stays awake on the charger, and you can reach the desktop remotely over VNC.
 
-The mouse shows up as a small dot anchored in the room. It works on the SteamVR dashboard, Steam, overlays, and the desktop, and it hands the laser back to your controllers when you pick one up.
-
-It comes with two settings apps, Frametop Display Settings for the screens and Frametop Input Settings for mice, keyboards, and button mappings, plus fixes that let Bluetooth LE mice and keyboards like the Swiftpoint Z3 reconnect after they sleep.
+Two settings apps come with it: Frametop Display Settings for the screens, profiles, and power, and Frametop Input Settings for mice, keyboards, gaze, and button mappings. Optional fixes let Bluetooth LE mice and keyboards like the Swiftpoint Z3 reconnect after they sleep. Install it with one command: see [Install on the headset](#install-on-the-headset).
 
 Frametop is an independent project, not made by or affiliated with Valve.
 
+Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions, ideas, and help with your setup.
+
 ## Install on the headset
 
+> **Frametop doesn't work on the SteamOS beta right now.** On the beta (SteamOS 0.4.3), gaze mode can't read the eye tracker, and the desktop has started without its taskbar ([#15](https://github.com/DeeJanuz/frametop/issues/15)). Use the stable SteamOS release until this note is gone.
+
 You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space.
 
 1. In the launcher, choose Launch a program → Desktop.
 2. In the application menu, open System → Konsole.
-3. Clone the repo and run the installer:
+3. Run:
 
    ```
-   git clone https://github.com/DeeJanuz/frametop.git ~/frametop
-   cd ~/frametop
-   ./install.sh
+   curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
    ```
 
-   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you two things along the way. The Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
+   It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/DeeJanuz/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
+
+   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
 
 After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings.
 
-If you work in the desktop for long stretches, stop Steam from putting the headset to sleep while it's plugged in: in Steam, open Settings → Power, and under When Plugged In and Idle set Sleep after to Never. By default Steam suspends the Frame after an hour without input, even while it charges. The displays still turn off a few seconds after you take the headset off.
+If you work in the desktop for long stretches, or leave the headset on a stand, open Frametop Display Settings → Power. Turn on Stay awake while plugged in: by default Steam puts the Frame to sleep after an hour without input, even while it charges. And choose when the displays turn off while the headset isn't used. SteamVR turns them off a few seconds after you take the headset off, but a stand or mount that covers the proximity sensor inside it makes the headset seem worn, and its displays stay on all night.
 
 ### Add a Bluetooth mouse or keyboard
 
@@ -38,37 +45,107 @@ If you work in the desktop for long stretches, stop Steam from putting the heads
 
 ## Use
 
+### Screens
+
 | Do this | To get this |
 | --- | --- |
-| Move the mouse | The dot moves around you and snaps onto whatever panel it's over |
-| Click, right-click, scroll | Acts on the panel under the dot |
-| Pick up a controller | The controller gets its laser back; move the mouse to take over again |
 | Point near the bottom of a screen | Its controls fade in: the bar, the curve and roll buttons, and the resize tab on the corner |
 | Drag the bar under a screen | Moves the screen; scroll while dragging to push it away or pull it closer. With the mouse, hold right while dragging to tilt it |
 | Drag the tab on a screen's bottom right corner | Resizes the screen |
 | Click the curve button (next to the bar) | Curves the screen around you, or flattens it |
 | Drag the roll button sideways, or scroll on it | Rolls the screen; it snaps level near straight |
 | While carrying a screen, sweep its laser across your other controller's ring, then let go | Pins it to that wrist, at its size and distance, as you hold it when you let go; it shows while you see its front. Grab its bar to adjust it (it stays pinned); sweep across the ring again to take it off |
+| Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head |
 | Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) |
-| Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & wrist tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
-| Play a VR game | The screens hide and your controllers stay in the game. Open the SteamVR dashboard, or press Meta+Shift+H, to see and use them. To keep them visible over games, change During VR games on the Visibility & wrist tab; the controllers still stay in the game, and you use the screens with the mouse or the dashboard |
-
-You can map the mouse's extra buttons to actions such as Toggle SteamVR dashboard, Recenter pointer, or Head follow on/off on the Buttons page of Frametop Input Settings, and the Frame controllers' buttons on its Controllers page. Pointer speed, dot size, and the rest are on its Pointer page and take effect immediately. Head follow, which is experimental and off by default, makes the pointer come along when you turn your head: it stays put until your head turns past the leash angle, then glides back to its place in your view, and a leash of 0 keeps it fixed in your view. It's only lightly tested and not polished; tuning its settings, or improving how it feels, is open to anyone who wants to take it further.
+| Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
+| Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own |
+| Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead |
+| Play a VR game | Frametop pauses so the game gets the headset to itself (see [Pause for VR games](#pause-for-vr-games)): the screens hide and your controllers stay in the game. Click both thumbsticks together twice to bring Frametop back. With the automatic pause off, the screens still hide, and the SteamVR dashboard or Meta+Shift+H shows them; to keep them visible over games, change During VR games on the Visibility & pins tab |
 
 Restarting the desktop (Restart desktop in Frametop Display Settings) closes its windows, but background work you started in it, such as servers, tmux sessions, or builds, keeps running.
 
+### Mouse and controllers
+
+| Do this | To get this |
+| --- | --- |
+| Move the mouse | The dot moves around you and snaps onto whatever panel it's over |
+| Click, right-click, scroll | Acts on the panel under the dot |
+| Pick up a controller | The controller gets its laser back; move the mouse to take over again |
+
+You can map the mouse's extra buttons to actions such as Toggle SteamVR dashboard, Recenter pointer, or Head follow on/off on the Buttons page of Frametop Input Settings, and the Frame controllers' buttons on its Controllers page. Pointer speed, dot size, and the rest are on its Pointer page and take effect immediately. If a panel you only look at, such as a performance overlay that follows your view, keeps catching the dot, tick it (or its whole app) on the Ignored panels page, and the pointer passes through it. Head follow, which is experimental and off by default, makes the pointer come along when you turn your head: it stays put until your head turns past the leash angle, then glides back to its place in your view, and a leash of 0 keeps it fixed in your view. It's only lightly tested and not polished; tuning its settings, or improving how it feels, is open to anyone who wants to take it further.
+
+### Floating windows
+
+Any desktop window can float in VR as a panel of its own. It stays a window of the same desktop, so drag and drop and the clipboard work between floating windows and the screens, and it's still in the taskbar and Alt+Tab.
+
+| Do this | To get this |
+| --- | --- |
+| Meta+Shift+F over a desktop window | Floats that window, or puts it back on its screen if it floats. It acts on the window under the pointer, or the active one if the pointer is over the wallpaper. Rebind it, or map it to a mouse or controller button, in Frametop Input Settings (Keyboard page, or Buttons and Controllers as Float window in VR) |
+| Click the float button, left of Close in a window's title bar | The same. Float in VR is also in every window's menu (Alt+F3). Apps that draw their own title bar, like Chromium and Electron apps, don't have the button: use Meta+Shift+F |
+| Right-click an app in the Application Launcher (or the taskbar) and pick Launch as Standalone | Starts the app with its window floating, where that app last floated, or in front of you the first time. From a terminal: `float/ft-float launch org.kde.dolphin`, or `float/ft-float run <command>` |
+| Meta+scroll over a floating window | Scales it up or down |
+
+### Profiles
+
+A profile is a named setup: where the screens are, with their sizes and pins, which ones are hidden, and which apps are open and where their windows are, on a screen or floating.
+
+| Do this | To get this |
+| --- | --- |
+| Save as profile… (Frametop Display Settings, Layout & profiles) | Saves the current setup under a name, or updates the profile you're in |
+| Pick a profile under Arrangement and press Open profile | Switches to it: the screens move, open windows of its apps go to their places, and the apps that aren't open start. Nothing closes |
+| Pick a profile under Start in profile, or run its entry (Frametop: NAME) from SteamVR's Launch a program list | The desktop starts in that profile, or switches to it if it's running. A profile can also go on a key combination, mouse button, or controller button in Frametop Input Settings |
+
+### Pause for VR games
+
+Frametop pauses while a VR game runs, so the game gets the headset's CPU and GPU, and comes back a few seconds after the game ends. Paused, the screens hide and the desktop nearly stops drawing, but its windows stay open. Gaze mode's eye tracking stops, and so do remote desktop and hand tracking if they run. The mouse works as a plain mouse in SteamVR.
+
+| Do this | To get this |
+| --- | --- |
+| Click both thumbsticks together, twice | Pauses Frametop, or brings it back, in a game or not. You hear a short sound. The game sees the clicks too |
+| Start a VR game | Frametop pauses, and comes back 5 seconds after the game ends. Bring it back during the game, and it stays on until that game ends |
+| Map Pause/resume Frametop to a mouse button, key combination, or controller button | The same, from that button (Frametop Input Settings) |
+
+The Game optimization page of Frametop Input Settings turns the automatic pause off, changes the gesture, closes the desktop instead of hiding it (more for the game, but its windows close), and turns the sound off. From a terminal: `input/ft-pause on`, `off`, or `status`.
+
+### Gaze mode (experimental)
+
+In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later), and then our own eye tracker for it, which is more accurate than SteamVR's (or run `gaze/tracker/install.sh` later; it needs `sudo`). Gaze mode uses ours once it's installed, and SteamVR's until then. Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
+
+| Do this | To get this |
+| --- | --- |
+| Tap Meta+J, or Meta+K | A left or right click where you look |
+| Hold Meta+J, turn your head onto what you meant, and let go | A click there. Held still for half a second, it becomes a press, and turning your head drags |
+| Hold the left mouse button, move the mouse onto what you meant, and let go | A click there. The right button does the same for a right click. Held still, the left button drags |
+| Double right click (or double Meta+K) while dragging a screen's bar | Pans and tilts the screen |
+| Put the headset on | A quick check: look at the dot for a moment, and the pointer lines up again |
+
+By default the mouse only corrects: while the gaze has the pointer, moving the mouse does nothing until you hold a button. Each correction before a click teaches the tracker where it was off. A correction past the learning limit (55 degrees by default, about half of what you can see) starts a quick check instead. Calibrate, on the Gaze page, runs a full calibration in a panel in front of you: look at each dot and click. Check headset fit shows how well the eye tracker sees your eyes. [gaze/README.md](gaze/README.md) has the details.
+
+### Leave the headset on a stand and reach it remotely
+
+To keep the Frame on and connected while you're not wearing it, for SSH, remote desktop, or anything else running on it, open the Power tab in Frametop Display Settings:
+
+- Turn off when unused for: how long the headset can go unused before its displays turn off (Never by default). Unused means the headset and controllers haven't moved and no mouse, keyboard, or button was used. SteamVR normally turns the displays off when its proximity sensor says the headset came off, but a stand or mount that covers the sensor makes the headset seem worn, so the displays stay on all night. This setting doesn't depend on the sensor. Pick the headset up or use any input, and the displays come back on.
+- Stay awake while plugged in: stops Steam from putting the Frame to sleep while it charges. By default Steam puts it to sleep after an hour without input, even on the charger, which ends remote sessions. This is Steam's own Settings → Power → When Plugged In and Idle setting, so the power button still puts the Frame to sleep, and Steam's battery setting still applies.
+
+With the displays off, the headset keeps tracking and rendering, so it uses about as much power as in use. Leave it on a charger that keeps up with that: a USB-C PD charger, not a 5 V one.
+
 ## Known limitations
 
 This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 20260922, SteamVR 2.17.10).
 
-- A SteamOS or SteamVR update can break parts of it until Frametop catches up. If something stops working after an update, please report it.
+- A SteamOS or SteamVR update can break parts of it until Frametop catches up. After an update, run `cd ~/frametop && scripts/doctor.sh` in a terminal. It checks what Frametop needs from SteamOS, and says what changed since the versions you last marked as working and what to try. Once everything works, `scripts/doctor.sh --mark-good` records the versions. If something stops working, please report it.
 - The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes.
-- During a VR game you can't show the screens with a controller button, because the game owns the buttons. Open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead.
-- Flatscreen games aren't detected as games. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & wrist tab).
+- During a VR game, mapped controller buttons belong to the game, so they can't bring the screens up. The pause gesture still works: Frametop reads it without taking the thumbsticks from the game. With the automatic pause off, open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead.
+- Flatscreen games aren't detected as games, so they don't pause Frametop by themselves: click both thumbsticks twice to pause it. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab).
 - Typing follows your last click. A controller click on a panel other than the screens (the dashboard, a Steam app) doesn't move typing there; click it with the mouse, or click a screen to bring typing back.
 - The screens don't draw a mouse cursor of their own. The 3D mouse's dot or SteamVR's laser shows where you're pointing.
+- Profiles reopen apps, not what the apps had open. Tabs, files, and folders are left to each app's own restore.
+- Dragging something from one panel to another (a screen and a floating window) works, but the dragged item's icon doesn't show while the pointer is between panels.
+- Gaze mode is only as good as its calibration, and that depends on how the headset sits on your face. If the pointer lands off after you adjust the headset, run Quick check or Calibrate on the Gaze page of Frametop Input Settings.
 - On SteamVR's Settings page, the 3D mouse shows a laser beam and a larger hit dot, like a controller. SteamVR doesn't tell other programs where that page is (unlike Steam's pages, such as Library), so the mouse used to miss most of it: clicks went through to a desktop screen behind, and the dot disappeared. As a workaround, on that page only, the laser starts near your eye and SteamVR finds the page itself. See docs/design.md.
-- Remote desktop over VNC (`./desktops.sh remote on`) needs Tailscale on the Frame.
+- Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart.
+- Turning the displays off on a stand only turns their backlight off. SteamVR has no way for other programs to put the headset in standby, so tracking and rendering keep running, and the headset draws nearly its full power.
 
 ## Reporting problems
 
@@ -78,40 +155,59 @@ In a terminal on the headset, run:
 cd ~/frametop && scripts/report.sh
 ```
 
-This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file.
+This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
 
 ## Update
 
+Run the same command again. It updates `~/frametop` to the latest of the version you have (or switches, if you pick the other one) and installs it:
+
 ```
-cd ~/frametop && git pull && ./install.sh
+curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
 ```
 
+Or by hand: `cd ~/frametop && git pull && ./install.sh`.
+
 ## Uninstall
 
 ```
 ./desktops.sh uninstall                # the launcher's Desktop entry goes back to the stock desktop
 ./desktops.sh relay uninstall
 pointer/helper/run.sh uninstall
+power/run.sh uninstall
 pointer/driver/install.sh uninstall    # then restart SteamVR
 input-settings/install.sh uninstall
 display-settings/install.sh uninstall
+remote/install.sh uninstall
 setup/bluetooth/install.sh uninstall   # if you installed the Bluetooth fixes
+hands/run.sh uninstall                 # if you installed hand tracking by hand
+gaze/run.sh uninstall                  # if you installed the gaze service
+gaze/tracker/install.sh uninstall      # if you installed our own eye tracker's frame grabber
+gaze/probe/install.sh uninstall        # if you installed the gaze probe
 ```
 
+Your settings stay: `~/.config/frametop.conf`, `frametop-input.json` (button maps and key combinations), `frametop-layout.json` (the layout and profiles), `frametop-float.json`, and `frametop-remote/` in `~/.config`, and the gaze calibration in `~/.local/state/frametop/gaze`. So does the desktop's own Plasma setup, in `~/.config/frametop`. Delete them too for a clean slate.
+
 ## How it works
 
-A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland compositor. KWin opens one window per screen, ft-screens sets each window's size, and each frame goes to SteamVR as an overlay without being copied. An input relay (`input/`) keeps Bluetooth mice working in SteamVR and feeds the mouse to the 3D pointer, which drives a virtual SteamVR controller (`pointer/`). [docs/reference.md](docs/reference.md) covers each piece, and [docs/design.md](docs/design.md) explains the design and what we learned about SteamVR on the Frame. [docs/hazards.md](docs/hazards.md) lists known ways the input handling can go wrong.
+A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland compositor. KWin opens one window per screen, ft-screens sets each window's size, and each frame goes to SteamVR as an overlay without being copied. An input relay (`input/`) keeps Bluetooth mice working in SteamVR and feeds the mouse to the 3D pointer, which drives a virtual SteamVR controller (`pointer/`). A power service (`power/`) turns the displays off while the headset isn't used. [docs/reference.md](docs/reference.md) covers each piece, and [docs/design.md](docs/design.md) explains the design and what we learned about SteamVR on the Frame. [docs/hazards.md](docs/hazards.md) lists known ways the input handling can go wrong.
 
 | Folder | What it is |
 | --- | --- |
+| `get.sh` | The one-line installer: picks stable or experimental, clones or updates the repo, and runs `install.sh`. |
 | `install.sh` | The one-step installer. Safe to re-run. |
 | `desktops.sh` | Start, stop, and configure the desktop, and install the input relay. |
 | `screens/` | ft-screens, the compositor (wlroots and OpenVR). |
 | `session/` | The desktop session script and its config example. |
 | `layout/` | ft-layout: where the screens float, and their sizes. |
+| `float/` | Floating windows: ft-floatd and the KWin script that float a desktop window in VR. |
+| `decoration/` | The desktop's window decoration: Breeze's look plus the float button. |
 | `input/` | The input relay (Bluetooth mice and keyboards, button maps). |
 | `pointer/` | The 3D mouse: SteamVR driver, helper service, and a probe tool. |
+| `power/` | ft-powerd: turns the displays off while the headset isn't used. |
+| `gaze/` | Gaze mode (experimental): the gaze service, its calibration panel, our own eye tracker, and the gaze probe. See [gaze/README.md](gaze/README.md). |
+| `hands/` | Hand tracking (experimental, deferred: the installer doesn't offer it). See [hands/README.md](hands/README.md). |
 | `display-settings/`, `input-settings/` | The two settings apps (Kirigami, Python). |
+| `remote/` | Frametop Remote Access, the app that turns remote desktop over VNC on and off. |
 | `setup/` | The build container and the Bluetooth fixes. See [setup/README.md](setup/README.md). |
 | `scripts/` | Helpers the installers use. They run commands locally on the Frame, or over SSH from a PC. |
 
@@ -129,16 +225,17 @@ The scripts also work from a Linux or WSL PC over SSH, which is easier for editi
        IdentityFile ~/.ssh/<your-key>
    ```
 
-3. The Bluetooth fixes need `sudo`, and there's no terminal on the Frame to type the password into, so put it in `.env` at the repo root. It's gitignored and never synced:
+3. The Bluetooth fixes need `sudo` on the Frame. The installer asks for the password in your terminal (over `ssh -t`). To skip the question, or to install with no terminal, put it in `.env` at the repo root instead. It's gitignored and never synced:
 
    ```
    steamos_root_pwd="<password>"
    ```
 
-Then run `./install.sh` from the PC. Daily use:
+Then run `./install.sh` from the PC. If SteamVR isn't running on the Frame, the services that need it start with it later. Daily use:
 
 ```
 scripts/doctor.sh                  # is the Frame reachable and ready?
+scripts/doctor.sh --mark-good      # and record the versions Frametop works with
 scripts/sync.sh                    # copy the repo to ~/dev/frametop on the Frame
 scripts/frame.sh '<cmd>'           # run in the dev container, in the Frame's copy
 scripts/frame.sh -C <dir> '<cmd>'  # same, in a folder of the repo

decoration/apply.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Switch the running Frametop desktop's window decoration without restarting it: Frametop's
+# own (this folder, with the float button) or back to Breeze. The session script does the
+# same at every desktop start (session/frametop-session.sh), so this is for trying changes.
+#   decoration/apply.sh        install this folder's copy and use it
+#   decoration/apply.sh --off  back to Breeze (until the next desktop start)
+set -euo pipefail
+here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+deco=kwin4_decoration_qml_frametop
+cfg=$HOME/.config/frametop
+kwinrc=$cfg/kwinrc
+
+# The desktop's D-Bus: from its Plasma shell, which uses Frametop's config folder (KWin's own
+# environment isn't readable: it runs with extra capabilities).
+bus=
+for pid in $(pgrep -x plasmashell); do
+  env=$( (tr '\0' '\n' < "/proc/$pid/environ") 2>/dev/null) || continue
+  grep -qx "XDG_CONFIG_HOME=$cfg" <<<"$env" || continue
+  bus=$(sed -n 's/^DBUS_SESSION_BUS_ADDRESS=//p' <<<"$env")
+done
+[ -n "$bus" ] || { echo "the Frametop desktop isn't running" >&2; exit 1; }
+
+if [ "${1:-}" = --off ]; then
+  kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key library --delete
+  kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme --delete
+else
+  # Under a new name each time: KWin keeps a decoration's QML (even a broken one) by name
+  # until it restarts. The next desktop start goes back to the plain name.
+  decos=${XDG_DATA_HOME:-$HOME/.local/share}/kwin/decorations
+  rm -rf "$decos/${deco}"_try*
+  deco=${deco}_try$(date +%s)
+  mkdir -p "$decos"
+  cp -r "$here" "$decos/$deco"
+  rm -f "$decos/$deco/apply.sh"
+  sed -i "s/\"Id\": \"[^\"]*\"/\"Id\": \"$deco\"/" "$decos/$deco/metadata.json"
+  kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key library org.kde.kwin.aurorae
+  kwriteconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme "$deco"
+fi
+DBUS_SESSION_BUS_ADDRESS=$bus gdbus call --session -d org.kde.KWin -o /KWin -m org.kde.KWin.reconfigure >/dev/null
+echo "decoration: $(kreadconfig6 --file "$kwinrc" --group org.kde.kdecoration2 --key theme --default Breeze)"

decoration/contents/ui/FtButton.qml

@@ -0,0 +1,123 @@
+/*
+    A title bar button drawn the way Breeze draws its own: a glyph in the title's colour,
+    a soft circle behind it on hover, and a red circle for Close.
+
+    The float button is KWin's Keep Below button with a glyph of its own: Frametop's KWin
+    script (float/frametop-float.js) floats a window when keep-below is set on it and docks
+    it when it's cleared, and keeps the flag set on every floating window, so "toggled" here
+    means "floating" and the glyph turns into "back to the desktop".
+
+    SPDX-License-Identifier: GPL-2.0-or-later
+*/
+import QtQuick
+import QtQuick.Shapes
+import org.kde.kwin.decoration
+
+DecorationButton {
+    id: button
+    property real size: 20
+    property color fg: "white"
+    property color bg: "black"
+    readonly property bool isClose: buttonType === DecorationOptions.DecorationButtonClose
+    readonly property bool isFloat: buttonType === DecorationOptions.DecorationButtonKeepBelow
+    // The glyph, as polylines in an 18 x 18 box (Breeze's own sizes).
+    readonly property var glyph: {
+        switch (buttonType) {
+        case DecorationOptions.DecorationButtonClose:
+            return [[[5, 5], [13, 13]], [[13, 5], [5, 13]]];
+        case DecorationOptions.DecorationButtonMaximizeRestore:
+            return decoration.client.maximized ? [[[4.5, 9], [9, 4.5], [13.5, 9], [9, 13.5], [4.5, 9]]]
+                                               : [[[4, 11.5], [9, 6.5], [14, 11.5]]];
+        case DecorationOptions.DecorationButtonMinimize:
+            return [[[4, 7], [9, 12], [14, 7]]];
+        case DecorationOptions.DecorationButtonKeepAbove:
+            return [[[4, 9], [9, 4], [14, 9]], [[4, 14], [9, 9], [14, 14]]];
+        case DecorationOptions.DecorationButtonShade:
+            return [[[4, 5], [14, 5]], [[4, 13], [9, 8], [14, 13]]];
+        case DecorationOptions.DecorationButtonApplicationMenu:
+            return [[[3.5, 5], [14.5, 5]], [[3.5, 9], [14.5, 9]], [[3.5, 13], [14.5, 13]]];
+        case DecorationOptions.DecorationButtonKeepBelow:
+            // A box with an arrow leaving it (float in VR), or coming back into it (put it back).
+            return button.toggled
+                ? [[[8, 3.5], [3.5, 3.5], [3.5, 14.5], [14.5, 14.5], [14.5, 10]], [[14.5, 3.5], [8.5, 9.5]],
+                   [[8.5, 5], [8.5, 9.5], [13, 9.5]]]
+                : [[[8, 3.5], [3.5, 3.5], [3.5, 14.5], [14.5, 14.5], [14.5, 10]], [[8.5, 9.5], [14.5, 3.5]],
+                   [[10, 3.5], [14.5, 3.5], [14.5, 8]]];
+        }
+        return [];
+    }
+    readonly property string tip: isFloat ? (toggled ? "Back to Desktop" : "Float in VR") : ""
+
+    width: size
+    height: size
+
+    Rectangle {
+        anchors.fill: parent
+        radius: width / 2
+        visible: button.hovered || button.pressed || (button.toggled && !button.isFloat)
+        color: button.isClose ? (button.pressed ? "#c0392b" : "#da4453")
+                              : Qt.rgba(button.fg.r, button.fg.g, button.fg.b,
+                                        button.pressed ? 0.35 : button.hovered ? 0.2 : 0.12)
+    }
+
+    // On all desktops: a dot, filled while the window is on all of them. Help: a question mark.
+    Rectangle {
+        visible: button.buttonType === DecorationOptions.DecorationButtonOnAllDesktops
+        anchors.centerIn: parent
+        width: button.size * 0.3
+        height: width
+        radius: width / 2
+        color: button.toggled ? button.fg : "transparent"
+        border.width: Math.max(1, button.size / 18)
+        border.color: button.fg
+    }
+    Text {
+        visible: button.buttonType === DecorationOptions.DecorationButtonQuickHelp
+        anchors.centerIn: parent
+        text: "?"
+        color: button.fg
+        font.pixelSize: button.size * 0.65
+        font.bold: true
+    }
+
+    // Up to three strokes per glyph.
+    function stroke(i) {
+        const k = button.size / 18;
+        return i < glyph.length ? glyph[i].map(p => Qt.point(p[0] * k, p[1] * k)) : [];
+    }
+    readonly property color strokeColor: isClose && (hovered || pressed) ? "white" : fg
+    readonly property real strokeWidth: Math.max(1, size / 18 * 1.1)
+    // (An inline component can't see the ids around it, so everything comes in as properties.)
+    component Stroke: ShapePath {
+        property var points: []
+        fillColor: "transparent"
+        capStyle: ShapePath.RoundCap
+        joinStyle: ShapePath.RoundJoin
+        PathPolyline { path: points }
+    }
+    Shape {
+        anchors.fill: parent
+        preferredRendererType: Shape.CurveRenderer
+        Stroke { points: button.stroke(0); strokeColor: button.strokeColor; strokeWidth: button.strokeWidth }
+        Stroke { points: button.stroke(1); strokeColor: button.strokeColor; strokeWidth: button.strokeWidth }
+        Stroke { points: button.stroke(2); strokeColor: button.strokeColor; strokeWidth: button.strokeWidth }
+    }
+
+    onHoveredChanged: {
+        if (!tip || typeof decoration.requestShowToolTip !== "function")
+            return;
+        if (hovered)
+            decoration.requestShowToolTip(tip);
+        else
+            decoration.requestHideToolTip();
+    }
+    Component.onCompleted: {
+        if (buttonType === DecorationOptions.DecorationButtonQuickHelp)
+            visible = Qt.binding(() => decoration.client.providesContextHelp);
+        if (buttonType === DecorationOptions.DecorationButtonApplicationMenu)
+            visible = Qt.binding(() => decoration.client.hasApplicationMenu);
+        // Like Breeze: no On All Desktops button with only one virtual desktop.
+        if (buttonType === DecorationOptions.DecorationButtonOnAllDesktops)
+            visible = Qt.binding(() => decorationSettings.onAllDesktopsAvailable);
+    }
+}

decoration/contents/ui/main.qml

@@ -0,0 +1,270 @@
+/*
+    Frametop's window decoration: Breeze's flat title bar, drawn in QML for KWin's Aurorae
+    engine, plus a button left of Close that floats the window in VR (docs/floating-windows.md,
+    decision 25). Aurorae loads QML without compiling anything, so this keeps working across
+    SteamOS's KWin updates, which a C++ decoration wouldn't.
+
+    The float button is the Keep Below button (FtButton.qml): Frametop's KWin script floats a
+    window when keep-below is set and docks it when it's cleared. A Keep Below button in the
+    configured button order is left out, since the float button stands in for it.
+
+    SPDX-License-Identifier: GPL-2.0-or-later
+*/
+import QtQuick
+import org.kde.kwin.decoration
+
+Decoration {
+    id: root
+    alpha: false
+
+    DecorationOptions {
+        id: options
+        deco: decoration
+    }
+    TextMetrics {
+        id: metrics
+        font: options.titleFont
+        text: "Mj"
+    }
+
+    readonly property bool maximized: decoration.client.maximized
+    readonly property int buttonSize: Math.max(16, Math.round(metrics.height * 1.25))
+    readonly property int titleHeight: buttonSize + 8
+    readonly property int borderSize: decorationSettings.borderSize
+    readonly property int side: {
+        switch (borderSize) {
+        case DecorationOptions.BorderNone:
+        case DecorationOptions.BorderNoSides: return 0;
+        case DecorationOptions.BorderTiny: return 2;
+        case DecorationOptions.BorderLarge: return 6;
+        case DecorationOptions.BorderVeryLarge: return 8;
+        case DecorationOptions.BorderHuge: return 12;
+        case DecorationOptions.BorderVeryHuge: return 18;
+        case DecorationOptions.BorderOversized: return 27;
+        default: return 4;
+        }
+    }
+    readonly property int bottomBorder: borderSize === DecorationOptions.BorderNone ? 0
+                                : borderSize === DecorationOptions.BorderNoSides ? 4 : side
+    readonly property color outline: Qt.tint(options.titleBarColor, Qt.rgba(0, 0, 0, 0.35))
+
+    function applyBorders() {
+        borders.left = side;
+        borders.right = side;
+        borders.bottom = bottomBorder;
+        borders.top = titleHeight;
+        maximizedBorders.top = titleHeight;
+        // Without visible side borders, keep a strip to grab for resizing.
+        extendedBorders.left = side ? 0 : 4;
+        extendedBorders.right = side ? 0 : 4;
+        extendedBorders.bottom = bottomBorder ? 0 : 4;
+    }
+    onTitleHeightChanged: applyBorders()
+    onSideChanged: applyBorders()
+    onBottomBorderChanged: applyBorders()
+    Component.onCompleted: applyBorders()
+
+    // The configured buttons, with the float button left of Close (or first on the right
+    // when there's no Close), and no Keep Below of their own.
+    function order(list, right) {
+        const out = [];
+        let placed = false;
+        for (let i = 0; i < list.length; ++i) {
+            const t = list[i];
+            if (t === DecorationOptions.DecorationButtonKeepBelow)
+                continue;
+            if (t === DecorationOptions.DecorationButtonClose && !placed) {
+                if (right) {
+                    out.push(DecorationOptions.DecorationButtonKeepBelow, t);
+                } else {
+                    out.push(t, DecorationOptions.DecorationButtonKeepBelow);
+                }
+                placed = true;
+                continue;
+            }
+            out.push(t);
+        }
+        return {buttons: out, placed: placed};
+    }
+    readonly property var leftOrder: order(options.titleButtonsLeft || [], false)
+    readonly property var rightOrder: {
+        const r = order(options.titleButtonsRight || [], true);
+        if (!r.placed && !leftOrder.placed)
+            r.buttons.unshift(DecorationOptions.DecorationButtonKeepBelow);
+        return r;
+    }
+
+    function componentFor(t) {
+        switch (t) {
+        case DecorationOptions.DecorationButtonMenu: return menuButton;
+        case DecorationOptions.DecorationButtonExplicitSpacer: return spacer;
+        case DecorationOptions.DecorationButtonClose: return closeButton;
+        case DecorationOptions.DecorationButtonMaximizeRestore: return maximizeButton;
+        case DecorationOptions.DecorationButtonMinimize: return minimizeButton;
+        case DecorationOptions.DecorationButtonKeepBelow: return floatButton;
+        case DecorationOptions.DecorationButtonKeepAbove: return keepAboveButton;
+        case DecorationOptions.DecorationButtonShade: return shadeButton;
+        case DecorationOptions.DecorationButtonOnAllDesktops: return allDesktopsButton;
+        case DecorationOptions.DecorationButtonQuickHelp: return helpButton;
+        case DecorationOptions.DecorationButtonApplicationMenu: return appMenuButton;
+        }
+        return null;
+    }
+
+    Rectangle {
+        anchors.fill: parent
+        color: options.titleBarColor
+        border.width: root.maximized ? 0 : 1
+        border.color: root.outline
+    }
+
+    Item {
+        id: titleBar
+        x: root.maximized ? 0 : Math.max(root.side, 1)
+        y: root.maximized ? 0 : 1
+        width: root.width - 2 * x
+        height: root.titleHeight - y
+
+        Row {
+            id: leftButtons
+            anchors.left: parent.left
+            anchors.leftMargin: 4
+            anchors.verticalCenter: parent.verticalCenter
+            spacing: 4
+            Repeater {
+                model: root.leftOrder.buttons
+                delegate: Loader {
+                    required property var modelData
+                    sourceComponent: root.componentFor(modelData)
+                }
+            }
+        }
+        Row {
+            id: rightButtons
+            anchors.right: parent.right
+            anchors.rightMargin: 4
+            anchors.verticalCenter: parent.verticalCenter
+            spacing: 4
+            layoutDirection: Qt.LeftToRight
+            Repeater {
+                model: root.rightOrder.buttons
+                delegate: Loader {
+                    required property var modelData
+                    sourceComponent: root.componentFor(modelData)
+                }
+            }
+        }
+        // Centred over the whole bar, like Breeze, but never under the buttons.
+        Text {
+            id: caption
+            readonly property real free: rightButtons.x - (leftButtons.x + leftButtons.width) - 16
+            width: Math.min(implicitWidth, free)
+            x: Math.max(leftButtons.x + leftButtons.width + 8,
+                        Math.min((parent.width - width) / 2, rightButtons.x - 8 - width))
+            anchors.verticalCenter: parent.verticalCenter
+            text: decoration.client.caption
+            textFormat: Text.PlainText
+            font: options.titleFont
+            color: options.fontColor
+            elide: Text.ElideRight
+            renderType: Text.NativeRendering
+        }
+        Component.onCompleted: decoration.installTitleItem(titleBar)
+    }
+
+    Component {
+        id: menuButton
+        MenuButton {
+            width: root.buttonSize
+            height: root.buttonSize
+        }
+    }
+    Component {
+        id: spacer
+        Item {
+            width: root.buttonSize
+            height: root.buttonSize
+        }
+    }
+    Component {
+        id: closeButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonClose
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+    Component {
+        id: maximizeButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonMaximizeRestore
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+    Component {
+        id: minimizeButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonMinimize
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+    Component {
+        id: floatButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonKeepBelow
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+    Component {
+        id: keepAboveButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonKeepAbove
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+    Component {
+        id: shadeButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonShade
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+    Component {
+        id: allDesktopsButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonOnAllDesktops
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+    Component {
+        id: helpButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonQuickHelp
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+    Component {
+        id: appMenuButton
+        FtButton {
+            buttonType: DecorationOptions.DecorationButtonApplicationMenu
+            size: root.buttonSize
+            fg: options.fontColor
+            bg: options.titleBarColor
+        }
+    }
+}

decoration/metadata.json

@@ -0,0 +1,14 @@
+{
+    "KPackageStructure": "KWin/Decoration",
+    "KPlugin": {
+        "Authors": [
+            {
+                "Name": "Frametop"
+            }
+        ],
+        "Description": "Breeze-style window decoration with a button that floats the window in VR (Frametop desktop)",
+        "Id": "kwin4_decoration_qml_frametop",
+        "License": "GPL",
+        "Name": "Frametop"
+    }
+}

desktops.sh

@@ -40,7 +40,12 @@ $running && echo 'started' || { echo 'failed:'; tail -20 $log; exit 1; }" ;;
 sed 's|@SESSION@|$session/frametop-session.sh|' $session/deckard-nested-desktop.desktop > ~/$override
 [ -f ~/.config/frametop.conf ] || cp $session/frametop.conf.example ~/.config/frametop.conf
 echo \"installed ~/$override\"; grep ^Exec= ~/$override; echo; cat ~/.config/frametop.conf" ;;
-  uninstall) "$frame" --host "rm -f ~/$override && echo 'removed; the launcher uses the stock desktop again'" ;;
+  uninstall)
+    # Also what the session puts in place at each start: Launch as Standalone's app copies and
+    # the title bar decoration (float/ft_apps.py, decoration/).
+    "$frame" --host "rm -f ~/$override
+rm -rf ~/.local/share/frametop/apps ~/.local/share/kwin/decorations/kwin4_decoration_qml_frametop
+rmdir ~/.local/share/frametop 2>/dev/null; echo 'removed; the launcher uses the stock desktop again'" ;;
   screens)
     [[ ${2:-} =~ ^[1-9]$ ]] || { echo "usage: $0 screens N   (1-9)" >&2; exit 2; }
     "$frame" --host "set -e; f=~/.config/frametop.conf

display-settings/ft_display_settings.py

@@ -10,10 +10,16 @@ the dev container:
     1920x1080 worth of pixels, rotation for portrait.)
   - Visibility (ft-screens): when the screens show (always, only with the SteamVR
     dashboard open, while you look at a controller, or only when toggled), the wrist
-    angle within which a pinned screen shows, and pin or unpin all screens.
-  - Layout: a preset (curved or flat, rows, distance, gap, height) or the arrangement
-    captured from where the screens are now, with a preview; arrange now; save the
-    current arrangement; arrange automatically when the desktop starts.
+    angle within which a pinned screen shows, and pinning each screen to a wrist or
+    your head.
+  - Layout: a preset (curved or flat, rows, distance, gap, height) or a named layout
+    saved from where the screens are, with a preview; arrange now; save the current
+    arrangement under a name; rename and delete; arrange automatically when the
+    desktop starts.
+  - Power: how long the headset can go unused before ft-powerd turns its displays off
+    (DISPLAY_OFF_MIN; the service's state comes from its control socket, @ft_powerd),
+    and whether the Frame stays awake while plugged in, which is Steam's own setting
+    (steam_settings.py; the value from before is kept as STEAM_SLEEP_AC_BEFORE).
 Settings go to ~/.config/frametop.conf and ~/.config/frametop-layout.json. Anything
 that touches SteamVR runs layout/ft-layout on the host.
 Launch with display-settings/ft-display-settings (host wrapper).
@@ -22,8 +28,9 @@ import os
 import shutil
 import socket
 import sys
+import threading
 
-from PySide6.QtCore import Property, QObject, QProcess, QTimer, QUrl, Signal, Slot
+from PySide6.QtCore import Property, QObject, QProcess, Qt, QTimer, QUrl, Signal, Slot
 from PySide6.QtGui import QGuiApplication, QIcon
 from PySide6.QtQml import QQmlApplicationEngine
 from PySide6.QtQuickControls2 import QQuickStyle
@@ -32,6 +39,7 @@ HERE = os.path.dirname(os.path.abspath(__file__))
 LAYOUT_DIR = os.path.join(HERE, "..", "layout")
 sys.path.insert(0, LAYOUT_DIR)
 import ft_layout  # noqa: E402  (pure Python: the same geometry ft-layout uses)
+import steam_settings  # noqa: E402
 
 FT_LAYOUT = os.path.join(LAYOUT_DIR, "ft-layout")
 DESKTOPS = os.path.join(HERE, "..", "desktops.sh")
@@ -46,6 +54,10 @@ SCREEN_RESOLUTIONS = [(1920, 1080, ""), (2560, 1440, ""), (3840, 2160, "4K"), (2
                       (2560, 1600, "16:10"), (1080, 1920, "portrait"), (1440, 2560, "portrait"),
                       (2160, 3840, "portrait 4K")]
 FT_SCREENS = "\0ft_screens"
+FT_POWERD = "\0ft_powerd"
+# Steam's default for "When Plugged In and Idle -> Sleep after", to go back to when
+# nothing was saved.
+STEAM_SLEEP_AC_DEFAULT = 3600
 SCALES = [0.75, 1.0, 1.25, 4 / 3, 1.5, 1.75, 2.0]
 ROTATIONS = [("normal", "Landscape"), ("left", "Portrait"), ("right", "Portrait (flipped)")]
 
@@ -83,7 +95,9 @@ def host_command(*cmd):
 class Backend(QObject):
     changed = Signal()
     busyChanged = Signal()
+    powerChanged = Signal()
     message = Signal(str, bool)  # text, is error
+    _steamDone = Signal(object, object, str)  # Steam's sleep settings or None, error or None, what was done
 
     def __init__(self):
         super().__init__()
@@ -95,6 +109,15 @@ class Backend(QObject):
         self._sock.bind("")  # an abstract address ft-screens can reply to
         self._sock.settimeout(1.0)
         self._started = {}  # conf values the running desktop started with
+        self._psock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+        self._psock.bind("")  # for ft-powerd's replies
+        self._psock.settimeout(0.5)
+        self._powerd = None  # ft-powerd's status: (state, seconds unused, timeout seconds); None: not running
+        self._steam = None  # Steam's sleep settings: {"ac": seconds, "battery": seconds}
+        self._steam_error = ""
+        self._steam_busy = False
+        self._steamDone.connect(self._steam_done, Qt.QueuedConnection)
+        self._pins = []     # each running screen's pin: none | left | right | head
         self.poll = QTimer(interval=3000, timeout=self._check_running)
         self.poll.start()
         self._check_running()
@@ -115,11 +138,23 @@ class Backend(QObject):
         # from the container, so ft_layout.nested_env() doesn't work here).
         running = os.path.exists(f"/run/user/{os.getuid()}/frametop/wayland-0")
         count = self._screens_running() if running and ft_layout.backend() == "screens" else 0
-        if running != self._running or count != self._running_count:
+        pins = self._read_pins(count)
+        if running != self._running or count != self._running_count or pins != self._pins:
+            if running != self._running or count != self._running_count:
+                self._started = self._conf() if running else {}
             self._running = running
             self._running_count = count
-            self._started = self._conf() if running else {}
+            self._pins = pins
             self.changed.emit()
+        self._check_powerd()
+
+    def _read_pins(self, count):
+        pins = []
+        for i in range(count):
+            reply = self._ask_screens(f"get {i + 1}")
+            f = reply.split() if reply and reply.startswith("ok") else []
+            pins.append(f[16] if len(f) > 16 else "none")
+        return pins
 
     def _ask_screens(self, text):
         """Request/reply to ft-screens; None if it isn't running."""
@@ -370,20 +405,139 @@ class Backend(QObject):
             else:
                 self._ask_screens(f"gesture {v['gesture_hand']} {float(v['gesture_angle']):.1f}")
 
-    @Slot(str)
-    def pinAll(self, hand):
-        reply = self._ask_screens(f"pin all {hand}") if self._running else None
-        if reply and reply.startswith("ok"):
-            self.message.emit(f"All screens ride on your {hand} wrist now; grab a screen's bar to take it off. "
-                              "Save current arrangement keeps it.", False)
+    @Property("QVariantList", notify=changed)
+    def pins(self):
+        return self._pins
+
+    @Property("QVariantList", notify=changed)
+    def screensShown(self):
+        """For each screen, whether it shows (False: hidden on its own, ft-layout hide N)."""
+        layout = ft_layout.load_layout()
+        return [not ft_layout.screen_entry(layout, i).get("hidden") for i in range(ft_layout.screen_count(layout))]
+
+    @Slot(int, bool)
+    def setScreenShown(self, index, shown):
+        """Hide screen `index` (0-based) on its own, whatever the visibility mode, or show it."""
+        layout = ft_layout.load_layout()
+        screens = layout.setdefault("screens", [])
+        while len(screens) <= index:
+            screens.append({})
+        if shown:
+            screens[index].pop("hidden", None)
         else:
-            self.message.emit(f"Couldn't pin: {reply or 'the desktop is not running'}", True)
+            screens[index]["hidden"] = True
+        ft_layout.save_layout(layout)
+        self.changed.emit()
+        if self._running:
+            reply = self._ask_screens(f"{'reveal' if shown else 'conceal'} {index + 1}")
+            if not (reply and reply.startswith("ok")):
+                self.message.emit("Saved; the desktop applies it when it next starts "
+                                  "(its compositor is older than hiding screens one at a time)", False)
+
+    @Slot(str, str)
+    def pin(self, which, where):
+        """Pin screen `which` (1-based, or "all") to "left", "right", or "head" as it is
+        now, or take it off ("none")."""
+        cmd = f"unpin {which}" if where == "none" else f"pin {which} {where}"
+        reply = self._ask_screens(cmd) if self._running else None
+        if not (reply and reply.startswith("ok")):
+            self.message.emit(f"Couldn't {'unpin' if where == 'none' else 'pin'}: "
+                              f"{reply or 'the desktop is not running'}", True)
+        elif which == "all" and where != "none":
+            place = "on your head" if where == "head" else f"on your {where} wrist"
+            self.message.emit(f"All screens ride {place} now. Save as profile… (Layout & profiles) keeps it.", False)
+        self._check_running()
+
+    # --- power: ft-powerd and Steam's sleep setting ---
+    def _check_powerd(self):
+        try:
+            self._psock.sendto(b"status", FT_POWERD)
+            reply = self._psock.recv(256).decode().split()
+            status = (reply[1], float(reply[2]), float(reply[3])) if reply[:1] == ["ok"] else None
+        except (OSError, IndexError, ValueError):
+            status = None
+        if status != self._powerd:
+            self._powerd = status
+            self.powerChanged.emit()
+
+    @Property("QVariantMap", notify=powerChanged)
+    def power(self):
+        try:
+            off_min = float(ft_layout.read_conf().get("DISPLAY_OFF_MIN") or 0)
+        except ValueError:
+            off_min = 0.0
+        state, unused, _ = self._powerd or ("", 0, 0)
+        return {"offMinutes": off_min, "service": self._powerd is not None, "state": state, "unused": unused,
+                "steam": self._steam is not None, "steamBusy": self._steam_busy, "steamError": self._steam_error,
+                "acSleep": self._steam["ac"] if self._steam else -1,
+                "batterySleep": self._steam["battery"] if self._steam else -1}
+
+    @Slot(float)
+    def setDisplayOffMinutes(self, minutes):
+        """ft-powerd re-reads frametop.conf within 2 s."""
+        write_conf_value("DISPLAY_OFF_MIN", f"{max(0.0, minutes):g}")
+        self.powerChanged.emit()
 
     @Slot()
-    def unpinAll(self):
-        reply = self._ask_screens("unpin all") if self._running else None
-        if not (reply and reply.startswith("ok")):
-            self.message.emit(f"Couldn't unpin: {reply or 'the desktop is not running'}", True)
+    def displaysOffNow(self):
+        try:
+            self._psock.sendto(b"off", FT_POWERD)
+            reply = self._psock.recv(256).decode()
+        except OSError:
+            reply = "error the power service isn't running"
+        if not reply.startswith("ok"):
+            self.message.emit(f"Couldn't turn the displays off: {reply.split(' ', 1)[-1]}", True)
+        self._check_powerd()
+
+    def _steam_call(self, what, fn):
+        """Runs fn, which talks to Steam (up to a few seconds), off the UI thread, then reads
+        Steam's sleep settings; _steam_done gets them on the UI thread."""
+        if self._steam_busy:
+            return
+        self._steam_busy = True
+        self.powerChanged.emit()
+
+        def work():
+            try:
+                fn()
+                self._steamDone.emit(steam_settings.sleep_settings(), None, what)
+            except (steam_settings.SteamUnreachable, OSError, ValueError) as e:
+                self._steamDone.emit(None, str(e), what)
+
+        threading.Thread(target=work, daemon=True).start()
+
+    def _steam_done(self, settings, error, what):
+        self._steam_busy = False
+        if settings is not None:
+            self._steam, self._steam_error = settings, ""
+        else:
+            self._steam, self._steam_error = None, error
+            if what:
+                self.message.emit(f"Couldn't change Steam's sleep setting: {error}", True)
+        self.powerChanged.emit()
+
+    @Slot()
+    def refreshPower(self):
+        self._check_powerd()
+        self._steam_call("", lambda: None)
+
+    @Slot(bool)
+    def setStayAwake(self, on):
+        """Steam's "When Plugged In and Idle -> Sleep after" is Never while this is on. The value
+        from before is kept in frametop.conf and goes back when it's turned off."""
+        def change():
+            ac = steam_settings.sleep_settings()["ac"]
+            if on:
+                if ac > 0:
+                    write_conf_value("STEAM_SLEEP_AC_BEFORE", str(ac))
+                steam_settings.set_sleep_setting("system_idle_suspend_ac_sec", 0)
+            elif ac == 0:
+                try:
+                    before = int(ft_layout.read_conf().get("STEAM_SLEEP_AC_BEFORE") or STEAM_SLEEP_AC_DEFAULT)
+                except ValueError:
+                    before = STEAM_SLEEP_AC_DEFAULT
+                steam_settings.set_sleep_setting("system_idle_suspend_ac_sec", before if before > 0 else STEAM_SLEEP_AC_DEFAULT)
+        self._steam_call("stay awake" if on else "sleep", change)
 
     @Slot()
     def restartDesktop(self):
@@ -401,7 +555,83 @@ class Backend(QObject):
 
     @Slot(str)
     def setMode(self, mode):
-        self._edit_layout(lambda l: l.__setitem__("mode", mode))
+        def edit(layout):
+            layout["mode"] = mode
+            layout.pop("active", None)
+        self._edit_layout(edit)
+
+    @Property("QVariantList", notify=changed)
+    def layoutNames(self):
+        return ft_layout.layout_names(ft_layout.load_layout())
+
+    @Slot(str)
+    def useLayout(self, name):
+        """A named layout as the arrangement (Arrange now puts the screens there)."""
+        try:
+            self._edit_layout(lambda l: ft_layout.use_named(l, name))
+        except RuntimeError as e:
+            self.message.emit(str(e), True)
+
+    @Slot(str)
+    def saveLayout(self, name):
+        try:
+            ft_layout.check_name(name)
+        except RuntimeError as e:
+            return self.message.emit(str(e), True)
+        self._run(f"Saving the arrangement as {' '.join(name.split())}", "save", name)
+
+    @Slot(str, str)
+    def renameLayout(self, old, new):
+        try:
+            self._edit_layout(lambda l: ft_layout.rename_named(l, old, new))
+            ft_layout.write_launchers(ft_layout.load_layout())
+        except (RuntimeError, OSError) as e:
+            self.message.emit(str(e), True)
+
+    @Slot(str)
+    def deleteLayout(self, name):
+        try:
+            self._edit_layout(lambda l: ft_layout.delete_named(l, name))
+            ft_layout.write_launchers(ft_layout.load_layout())
+        except (RuntimeError, OSError) as e:
+            self.message.emit(str(e), True)
+
+    # --- profiles (docs/profiles.md): a named layout's apps and hidden screens ---
+    @Slot(str, result="QVariantList")
+    def profileWindows(self, name):
+        """A profile's windows, as "app" and "where" for the list."""
+        out = []
+        for e in ft_layout.load_layout().get("profiles", {}).get(name, {}).get("windows", []):
+            app = e.get("app") or os.path.basename((e.get("cmd") or ["?"])[0])
+            app = app.rsplit(".", 1)[-1] if "." in app and not e.get("cmd") else app
+            where = "floating" if "float" in e else f"screen {e.get('screen', 1)}" + (", maximized" if e.get("maximized") else "")
+            out.append({"app": app, "where": where})
+        return out
+
+    @Slot(str, result="QVariantList")
+    def profileHidden(self, name):
+        return ft_layout.load_layout().get("profiles", {}).get(name, {}).get("hidden", [])
+
+    @Slot(str, int)
+    def removeProfileWindow(self, name, index):
+        def edit(layout):
+            windows = layout.get("profiles", {}).get(name, {}).get("windows", [])
+            if 0 <= index < len(windows):
+                windows.pop(index)
+        self._edit_layout(edit)
+
+    @Property(str, notify=changed)
+    def defaultProfile(self):
+        return ft_layout.load_layout().get("default_profile", "")
+
+    @Slot(str)
+    def setDefaultProfile(self, name):
+        def edit(layout):
+            if name:
+                layout["default_profile"] = name
+            else:
+                layout.pop("default_profile", None)
+        self._edit_layout(edit)
 
     @Slot(str, "QVariant")
     def setPreset(self, key, value):
@@ -416,7 +646,13 @@ class Backend(QObject):
 
     @Slot()
     def arrange(self):
-        self._run("Arranging the screens", "apply")
+        """Arrange the screens; in a profile, also open its apps (ft-layout use)."""
+        layout = ft_layout.load_layout()
+        name = layout.get("active")
+        if layout.get("mode") == "custom" and name in layout.get("layouts", {}):
+            self._run(f"Opening {name}", "use", name)
+        else:
+            self._run("Arranging the screens", "apply")
 
     @Slot()
     def capture(self):

display-settings/install.sh

@@ -17,10 +17,12 @@ case ${1:-install} in
     on_frame "chmod +x display-settings/ft-display-settings layout/ft-layout layout/ft_layout.py
 mkdir -p ~/.config/frametop
 kwriteconfig6 --file ~/$shortcuts --group services --group ft-layout-reset.desktop --key _launch 'Meta+Shift+R'
-kwriteconfig6 --file ~/$shortcuts --group services --group ft-screens-toggle.desktop --key _launch 'Meta+Shift+H'"
+kwriteconfig6 --file ~/$shortcuts --group services --group ft-screens-toggle.desktop --key _launch 'Meta+Shift+H'
+layout/ft-layout launchers  # each profile's entry (Frametop: NAME), if there are profiles"
     echo "installed: Frametop Display Settings, Reset Screen Layout (Meta+Shift+R), Hide/Show Screens (Meta+Shift+H)" ;;
   uninstall)
     on_frame "rm -f ~/$apps/ft-display-settings.desktop ~/$apps/ft-layout-reset.desktop ~/$apps/ft-screens-toggle.desktop
+rm -f ~/$apps/frametop-profile-*.desktop  # the profiles' entries (the profiles stay in ~/.config/frametop-layout.json)
 [ -f ~/$shortcuts ] && for f in ft-layout-reset ft-screens-toggle; do kwriteconfig6 --file ~/$shortcuts --group services --group \$f.desktop --key _launch --delete; done
 echo removed" ;;
   *) echo "usage: $0 [install|uninstall]" >&2; exit 2 ;;

display-settings/main.qml

@@ -12,11 +12,13 @@ Kirigami.ApplicationWindow {
 
     // Pages as tabs across the top (a side drawer was easy to miss).
     readonly property var pages: backend.backend === "screens"
-        ? [{ text: "Screens", icon: "video-display", page: screensPage },
-           { text: "Layout", icon: "view-grid", page: layoutPage },
-           { text: "Visibility & wrist", icon: "view-visible", page: visibilityPage }]
-        : [{ text: "Screens", icon: "video-display", page: screensPage },
-           { text: "Layout", icon: "view-grid", page: layoutPage }]
+        ? [{ name: "screens", text: "Screens", icon: "video-display", page: screensPage },
+           { name: "layout", text: "Layout", icon: "view-grid", page: layoutPage },
+           { name: "visibility", text: "Visibility & pins", icon: "view-visible", page: visibilityPage },
+           { name: "power", text: "Power", icon: "preferences-system-power-management", page: powerPage }]
+        : [{ name: "screens", text: "Screens", icon: "video-display", page: screensPage },
+           { name: "layout", text: "Layout", icon: "view-grid", page: layoutPage },
+           { name: "power", text: "Power", icon: "preferences-system-power-management", page: powerPage }]
 
     header: Controls.TabBar {
         id: tabs
@@ -29,7 +31,7 @@ Kirigami.ApplicationWindow {
                 onClicked: root.show(modelData.page)
             }
         }
-        Component.onCompleted: currentIndex = ({ layout: 1, visibility: 2 })[startPage] || 0
+        Component.onCompleted: currentIndex = Math.max(0, root.pages.findIndex(p => p.name === startPage))
     }
 
     function show(page) {
@@ -37,8 +39,16 @@ Kirigami.ApplicationWindow {
         pageStack.push(page)
     }
 
-    // FT_DISPLAY_PAGE=layout|visibility opens the app on that page.
-    pageStack.initialPage: ({ layout: layoutPage, visibility: visibilityPage })[startPage] || screensPage
+    // FT_DISPLAY_PAGE=layout|visibility|power opens the app on that page.
+    pageStack.initialPage: ({ layout: layoutPage, visibility: visibilityPage, power: powerPage })[startPage] || screensPage
+
+    // "1 hour", "15 minutes", "30 seconds".
+    function duration(seconds) {
+        const unit = (n, word) => n + " " + word + (n === 1 ? "" : "s")
+        if (seconds >= 3600 && seconds % 3600 === 0) return unit(seconds / 3600, "hour")
+        if (seconds >= 60 && seconds % 60 === 0) return unit(seconds / 60, "minute")
+        return unit(seconds, "second")
+    }
 
     Connections {
         target: backend
@@ -66,6 +76,89 @@ Kirigami.ApplicationWindow {
         ]
     }
 
+    // Save the arrangement under a name, or rename a saved layout.
+    Kirigami.PromptDialog {
+        id: nameDialog
+        property string mode: "save"  // save | rename
+        property string oldName: ""
+        readonly property var names: backend.layoutNames
+        readonly property string name: nameField.text.trim().split(/\s+/).join(" ")
+        readonly property bool taken: name !== oldName && names.indexOf(name) >= 0
+        readonly property bool ok: name !== "" && !(mode === "rename" && taken)
+        title: mode === "save" ? "Save the arrangement" : "Rename " + oldName
+        standardButtons: Kirigami.Dialog.NoButton
+
+        function openFor(m, text) {
+            mode = m
+            oldName = m === "rename" ? text : ""
+            nameField.text = text
+            open()
+            nameField.forceActiveFocus()
+            nameField.selectAll()
+        }
+        function accept() {
+            if (!ok) return
+            close()
+            if (mode === "save") backend.saveLayout(name)
+            else if (name !== oldName) backend.renameLayout(oldName, name)
+        }
+
+        ColumnLayout {
+            Controls.Label {
+                Layout.fillWidth: true
+                wrapMode: Text.Wrap
+                text: nameDialog.mode === "save"
+                      ? "Where the screens are now, with their sizes, curves, and pins, under this name:"
+                      : "New name:"
+            }
+            Controls.TextField {
+                id: nameField
+                Layout.fillWidth: true
+                maximumLength: 40
+                onAccepted: nameDialog.accept()
+            }
+            Controls.Label {
+                visible: nameDialog.taken
+                opacity: 0.7
+                text: nameDialog.mode === "save" ? "Replaces the saved layout with that name."
+                                                 : "There's already a layout with that name."
+            }
+        }
+        customFooterActions: [
+            Kirigami.Action {
+                text: nameDialog.mode === "save" ? "Save" : "Rename"
+                icon.name: nameDialog.mode === "save" ? "document-save" : "edit-rename"
+                enabled: nameDialog.ok
+                onTriggered: nameDialog.accept()
+            },
+            Kirigami.Action {
+                text: "Cancel"
+                icon.name: "dialog-cancel"
+                onTriggered: nameDialog.close()
+            }
+        ]
+    }
+
+    Kirigami.PromptDialog {
+        id: deleteDialog
+        property string name: ""
+        title: "Delete " + name + "?"
+        subtitle: "The screens stay where they are; only the saved layout goes."
+        standardButtons: Kirigami.Dialog.NoButton
+        customFooterActions: [
+            Kirigami.Action {
+                text: "Delete"
+                icon.name: "edit-delete"
+                onTriggered: { deleteDialog.close(); backend.deleteLayout(deleteDialog.name) }
+            },
+            Kirigami.Action {
+                text: "Cancel"
+                icon.name: "dialog-cancel"
+                onTriggered: deleteDialog.close()
+            }
+        ]
+    }
+
     // ---------------------------------------------------------------- Screens
     Component {
         id: screensPage
@@ -318,8 +411,8 @@ Kirigami.ApplicationWindow {
                 text: spage.md
                       ? "Each screen is a real monitor of its own: any resolution, portrait by choosing a tall one. Resolution, "
                         + "width, and curve apply at once. In VR: move a screen by the bar underneath, curve it with the round "
-                        + "button next to the bar, resize it by the tab on its bottom right corner; Save current arrangement on the "
-                        + "Layout page keeps all of it."
+                        + "button next to the bar, resize it by the tab on its bottom right corner; Save as profile… on the "
+                        + "Layout & profiles page keeps all of it."
                       : "gamescope draws every screen at the same resolution, at most 1920 × 1080 worth of pixels. Portrait turns "
                         + "a screen on its side. Rotation and scale apply at once; the rest when the desktop starts."
             }
@@ -331,25 +424,36 @@ Kirigami.ApplicationWindow {
         id: layoutPage
         Kirigami.ScrollablePage {
             id: lpage
-            title: "Layout"
+            title: "Layout & profiles"
             property var layout: backend.layout
             property var preset: layout.preset || {}
             property bool hasCustom: (layout.screens || []).some(s => s.pos !== undefined)
+            // Named layouts: the arrangement is one of them (named) when it came from it, and
+            // hasn't been placed by hand and saved without a name since.
+            property var names: backend.layoutNames
+            property bool fromNamed: names.indexOf(layout.active) >= 0
+            property bool named: layout.mode === "custom" && fromNamed
+            property bool unnamed: names.length === 0 || ((hasCustom || layout.mode === "custom") && !fromNamed)
+            property var choices: [{ text: "Curved around you", value: "arc" }, { text: "Flat wall", value: "flat" }]
+                .concat(names.map(n => ({ text: n, value: "layout:" + n })))
+                .concat(unnamed ? [{ text: names.length ? "Unnamed arrangement" : "Saved arrangement", value: "custom" }] : [])
 
             actions: [
                 Kirigami.Action {
-                    text: "Arrange now"
+                    text: lpage.named ? "Open profile" : "Arrange now"
                     icon.name: "view-restore"
-                    tooltip: "Float the screens out of the dashboard and put them in this layout, around where you're facing"
+                    tooltip: lpage.named ? "Put the screens in this profile's places, around where you're facing, and open its apps (windows already open move; nothing closes)"
+                                         : "Float the screens out of the dashboard and put them in this layout, around where you're facing"
                     enabled: backend.desktopRunning && backend.busy === ""
                     onTriggered: backend.arrange()
                 },
                 Kirigami.Action {
-                    text: "Save current arrangement"
+                    text: "Save as profile…"
                     icon.name: "document-save"
-                    tooltip: "Use where the screens are now (placed by hand) as the layout"
+                    tooltip: "Save where the screens are now, which ones are hidden, and the open apps and where their windows are, under a name"
                     enabled: backend.desktopRunning && backend.busy === ""
-                    onTriggered: backend.capture()
+                    onTriggered: nameDialog.openFor("save", lpage.named ? lpage.layout.active
+                                                                        : "Layout " + (lpage.names.length + 1))
                 }
             ]
 
@@ -366,32 +470,99 @@ Kirigami.ApplicationWindow {
                 Kirigami.FormLayout {
                     Layout.fillWidth: true
 
-                    Controls.ComboBox {
+                    RowLayout {
                         Kirigami.FormData.label: "Arrangement:"
-                        model: [
-                            { text: "Curved around you", value: "arc" },
-                            { text: "Flat wall", value: "flat" },
-                            { text: "Saved arrangement", value: "custom" }
-                        ]
-                        textRole: "text"
-                        valueRole: "value"
-                        currentIndex: lpage.layout.mode === "custom" ? 2 : (lpage.preset.kind === "flat" ? 1 : 0)
-                        onActivated: {
-                            if (currentValue === "custom") backend.setMode("custom")
-                            else backend.setPreset("kind", currentValue)
+                        Controls.ComboBox {
+                            model: lpage.choices
+                            textRole: "text"
+                            valueRole: "value"
+                            currentIndex: lpage.layout.mode !== "custom" ? (lpage.preset.kind === "flat" ? 1 : 0)
+                                        : lpage.named ? 2 + lpage.names.indexOf(lpage.layout.active)
+                                        : lpage.choices.length - 1
+                            onActivated: {
+                                if (currentValue === "custom") backend.setMode("custom")
+                                else if (currentValue.startsWith("layout:")) backend.useLayout(currentValue.slice(7))
+                                else backend.setPreset("kind", currentValue)
+                            }
+                        }
+                        Controls.ToolButton {
+                            visible: lpage.named
+                            icon.name: "edit-rename"
+                            text: "Rename…"
+                            display: Controls.AbstractButton.IconOnly
+                            Controls.ToolTip.text: text
+                            Controls.ToolTip.visible: hovered
+                            onClicked: nameDialog.openFor("rename", lpage.layout.active)
+                        }
+                        Controls.ToolButton {
+                            visible: lpage.named
+                            icon.name: "edit-delete"
+                            text: "Delete…"
+                            display: Controls.AbstractButton.IconOnly
+                            Controls.ToolTip.text: text
+                            Controls.ToolTip.visible: hovered
+                            onClicked: { deleteDialog.name = lpage.layout.active; deleteDialog.open() }
                         }
                     }
 
                     Controls.Label {
                         visible: lpage.layout.mode === "custom"
                         Kirigami.FormData.label: ""
-                        text: lpage.hasCustom ? "Where the screens were when you saved. Pick a preset to edit."
-                                              : "Nothing saved yet: place the screens by hand, then Save current arrangement."
+                        text: lpage.named ? "Where the screens were when you saved it, and the apps that were open. Open "
+                                            + "profile puts the screens there and opens the apps. Save as profile updates "
+                                            + "it or saves a new one."
+                              : lpage.hasCustom ? "Where the screens were when you saved. Save as profile "
+                                                  + "names it. Pick a preset to edit."
+                              : "Nothing saved yet: place the screens by hand, open your apps, then Save as profile."
                         opacity: 0.7
                         wrapMode: Text.Wrap
                         Layout.maximumWidth: Kirigami.Units.gridUnit * 20
                     }
 
+                    // The profile's apps (docs/profiles.md): each window and where it goes.
+                    ColumnLayout {
+                        id: profileApps
+                        visible: lpage.named
+                        Kirigami.FormData.label: "Apps:"
+                        property var windows: lpage.named ? backend.profileWindows(lpage.layout.active) : []
+                        property var hidden: lpage.named ? backend.profileHidden(lpage.layout.active) : []
+                        Connections {
+                            target: backend
+                            function onChanged() {
+                                profileApps.windows = lpage.named ? backend.profileWindows(lpage.layout.active) : []
+                                profileApps.hidden = lpage.named ? backend.profileHidden(lpage.layout.active) : []
+                            }
+                        }
+                        Controls.Label {
+                            visible: profileApps.windows.length === 0
+                            text: "None saved. Open the apps you want, place their windows, then Save as profile."
+                            opacity: 0.7
+                            wrapMode: Text.Wrap
+                            Layout.maximumWidth: Kirigami.Units.gridUnit * 20
+                        }
+                        Repeater {
+                            model: profileApps.windows
+                            delegate: RowLayout {
+                                required property var modelData
+                                required property int index
+                                Controls.Label { text: modelData.app + " (" + modelData.where + ")" }
+                                Controls.ToolButton {
+                                    icon.name: "list-remove"
+                                    text: "Leave out"
+                                    display: Controls.AbstractButton.IconOnly
+                                    Controls.ToolTip.text: "Leave this window out of the profile"
+                                    Controls.ToolTip.visible: hovered
+                                    onClicked: backend.removeProfileWindow(lpage.layout.active, index)
+                                }
+                            }
+                        }
+                        Controls.Label {
+                            visible: profileApps.hidden.length > 0
+                            text: "Hides screen" + (profileApps.hidden.length > 1 ? "s " : " ") + profileApps.hidden.join(", ")
+                            opacity: 0.7
+                        }
+                    }
+
                     Controls.SpinBox {
                         Kirigami.FormData.label: "Rows:"
                         visible: lpage.layout.mode !== "custom"
@@ -429,8 +600,19 @@ Kirigami.ApplicationWindow {
                         Kirigami.FormData.label: "When the desktop starts:"
                         text: "Float the screens and arrange them"
                         checked: lpage.layout.auto !== false
+                        enabled: backend.defaultProfile === ""
                         onToggled: backend.setAuto(checked)
                     }
+                    Controls.ComboBox {
+                        Kirigami.FormData.label: "Start in profile:"
+                        model: [{ text: "None", value: "" }].concat(lpage.names.map(n => ({ text: n, value: n })))
+                        textRole: "text"
+                        valueRole: "value"
+                        currentIndex: Math.max(0, indexOfValue(backend.defaultProfile))
+                        onActivated: backend.setDefaultProfile(currentValue)
+                        Controls.ToolTip.text: "The desktop starts in this profile: its screens, and its apps open. Each profile also has its own entry in SteamVR's Launch a program list"
+                        Controls.ToolTip.visible: hovered
+                    }
                 }
 
                 // Preview: from above (you at the bottom) and from the front.
@@ -620,7 +802,7 @@ Kirigami.ApplicationWindow {
                     Repeater {
                         model: [
                             { value: "hide", text: "Hide them unless the SteamVR dashboard is open", help: "The game has the view to itself; open the dashboard (or press Meta+Shift+H) to see the screens." },
-                            { value: "visible", text: "Keep them visible over the game", help: "They float over the game as they are outside it." }
+                            { value: "visible", text: "Keep them visible over the game", help: "They float over the game as they are outside it. Turn off Pause while a VR game runs in Frametop Input Settings (Game optimization), or Frametop pauses and hides them anyway." }
                         ]
                         delegate: ColumnLayout {
                             required property var modelData
@@ -677,10 +859,49 @@ Kirigami.ApplicationWindow {
                         }
                     }
 
-                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Screens on a wrist" }
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Screens shown" }
 
+                    Repeater {
+                        model: backend.screensShown
+                        delegate: Controls.Switch {
+                            required property var modelData
+                            required property int index
+                            Kirigami.FormData.label: "Screen " + (index + 1) + ":"
+                            text: modelData ? "Shown" : "Hidden"
+                            checked: modelData
+                            onToggled: backend.setScreenShown(index, checked)
+                        }
+                    }
+                    Controls.Label {
+                        text: "A hidden screen stays hidden whatever the choices above say, and Meta+Shift+H doesn't bring it back. Windows on it stay there; new ones that would open on it float instead."
+                        opacity: 0.7
+                        font: Kirigami.Theme.smallFont
+                        wrapMode: Text.Wrap
+                        Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                    }
+
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Pinned screens" }
+
+                    Repeater {
+                        model: backend.pins
+                        delegate: Controls.ComboBox {
+                            required property var modelData
+                            required property int index
+                            Kirigami.FormData.label: "Screen " + (index + 1) + ":"
+                            model: [
+                                { text: "In the room", value: "none" },
+                                { text: "On the left wrist", value: "left" },
+                                { text: "On the right wrist", value: "right" },
+                                { text: "On your head", value: "head" }
+                            ]
+                            textRole: "text"
+                            valueRole: "value"
+                            currentIndex: Math.max(0, ["none", "left", "right", "head"].indexOf(modelData))
+                            onActivated: backend.pin(String(index + 1), currentValue)
+                        }
+                    }
                     RowLayout {
-                        Kirigami.FormData.label: "Show while facing you within:"
+                        Kirigami.FormData.label: "Wrist screens show within:"
                         Controls.Slider {
                             id: wrist
                             from: 20; to: 120; stepSize: 1
@@ -695,17 +916,22 @@ Kirigami.ApplicationWindow {
                         Controls.Button {
                             text: "Pin to left wrist"
                             enabled: backend.desktopRunning
-                            onClicked: backend.pinAll("left")
+                            onClicked: backend.pin("all", "left")
                         }
                         Controls.Button {
                             text: "Pin to right wrist"
                             enabled: backend.desktopRunning
-                            onClicked: backend.pinAll("right")
+                            onClicked: backend.pin("all", "right")
+                        }
+                        Controls.Button {
+                            text: "Pin to head"
+                            enabled: backend.desktopRunning
+                            onClicked: backend.pin("all", "head")
                         }
                         Controls.Button {
                             text: "Unpin"
                             enabled: backend.desktopRunning
-                            onClicked: backend.unpinAll()
+                            onClicked: backend.pin("all", "none")
                         }
                     }
                 }
@@ -720,7 +946,126 @@ Kirigami.ApplicationWindow {
                           + "then let go: it rides on that wrist at that size and distance, however far away. To adjust a "
                           + "pinned screen, grab its bar, move it, and let go (it stays pinned); sweep across the ring to "
                           + "take it off. It shows while you see its front within the angle above, and fades out beyond "
-                          + "it. Save current arrangement (Layout) keeps pins."
+                          + "it.\n\nPin a screen to your head: choose On your head above. It rides on the headset where it "
+                          + "is now, like a HUD, and shows whenever the screens do. Grab its bar to move it; it stays on "
+                          + "your head where you let go. Choosing a pin above keeps the screen where it is now, so place "
+                          + "it first. Save as profile… (Layout) keeps pins."
+                }
+            }
+        }
+    }
+
+    // ---------------------------------------------------------------- Power
+    Component {
+        id: powerPage
+        Kirigami.ScrollablePage {
+            id: ppage
+            title: "Power"
+            property var p: backend.power
+            // The timeout choices, plus a value set by hand in frametop.conf.
+            property var offChoices: {
+                const list = [{ text: "Never", value: 0 }].concat([1, 2, 5, 10, 15, 30, 60].map(
+                    m => ({ text: root.duration(m * 60), value: m })))
+                if (!list.some(c => c.value === p.offMinutes))
+                    list.push({ text: root.duration(Math.round(p.offMinutes * 60)), value: p.offMinutes })
+                return list
+            }
+
+            Component.onCompleted: backend.refreshPower()
+
+            actions: [
+                Kirigami.Action {
+                    text: "Turn displays off now"
+                    icon.name: "system-suspend"
+                    tooltip: "To try it: they come back on when the headset moves or any input is used"
+                    enabled: ppage.p.service && ppage.p.state === "on"
+                    onTriggered: backend.displaysOffNow()
+                }
+            ]
+
+            header: Kirigami.InlineMessage {
+                position: Kirigami.InlineMessage.Position.Header
+                visible: !ppage.p.service
+                type: Kirigami.MessageType.Warning
+                text: "The power service (frametop-power) isn't running, so the displays won't turn off on their own. "
+                      + "It starts with SteamVR once it's installed: power/run.sh install, or run ./install.sh again."
+            }
+
+            ColumnLayout {
+                spacing: Kirigami.Units.largeSpacing
+
+                Kirigami.FormLayout {
+                    Layout.fillWidth: true
+
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Displays" }
+
+                    Controls.ComboBox {
+                        Kirigami.FormData.label: "Turn off when unused for:"
+                        model: ppage.offChoices
+                        textRole: "text"
+                        valueRole: "value"
+                        Component.onCompleted: currentIndex = Math.max(0, indexOfValue(ppage.p.offMinutes))
+                        onActivated: backend.setDisplayOffMinutes(currentValue)
+                    }
+                    Controls.Label {
+                        text: "Unused means the headset and controllers haven't moved and no mouse, keyboard, or button "
+                              + "was used. This works even when the headset seems to be worn, like on a display mount "
+                              + "that covers its proximity sensor. Moving the headset or using any input turns the "
+                              + "displays back on. Taking the headset off still turns them off within seconds."
+                        opacity: 0.7
+                        font: Kirigami.Theme.smallFont
+                        wrapMode: Text.Wrap
+                        Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                    }
+                    Controls.Label {
+                        Kirigami.FormData.label: "Now:"
+                        visible: ppage.p.service
+                        text: ppage.p.state === "off" ? "Off. Move the headset or use any input to turn them on."
+                              : ppage.p.state === "away" ? "Off. SteamVR turned them off because the headset isn't being worn."
+                              : ppage.p.offMinutes > 0
+                                ? "On, unused for " + (ppage.p.unused < 60 ? Math.floor(ppage.p.unused) + " s"
+                                    : Math.floor(ppage.p.unused / 60) + " min " + Math.floor(ppage.p.unused % 60) + " s")
+                                : "On"
+                    }
+
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Sleep" }
+
+                    Controls.Switch {
+                        id: awake
+                        Kirigami.FormData.label: "While plugged in:"
+                        text: "Stay awake"
+                        checked: ppage.p.acSleep === 0
+                        enabled: ppage.p.steam && !ppage.p.steamBusy
+                        onToggled: {
+                            backend.setStayAwake(checked)
+                            checked = Qt.binding(() => ppage.p.acSleep === 0)  // follow what Steam has
+                        }
+                    }
+                    Controls.Label {
+                        text: !ppage.p.steam
+                              ? (ppage.p.steamBusy ? "Checking Steam's setting…" : "Couldn't reach Steam: " + ppage.p.steamError)
+                              : "Keeps the Frame awake and connected while it charges, for remote access, downloads, and "
+                                + "anything else running. This is Steam's own setting (Settings → Power → When Plugged In "
+                                + "and Idle), so the power button still puts the Frame to sleep. "
+                                + (ppage.p.acSleep > 0 ? "Now Steam puts it to sleep after " + root.duration(ppage.p.acSleep)
+                                                         + " without input, even while it charges. " : "")
+                                + "On battery, Steam's battery setting still applies ("
+                                + (ppage.p.batterySleep > 0 ? "sleep after " + root.duration(ppage.p.batterySleep) : "never sleep")
+                                + ")."
+                        opacity: 0.7
+                        font: Kirigami.Theme.smallFont
+                        wrapMode: Text.Wrap
+                        Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                    }
+                }
+
+                Controls.Label {
+                    Layout.fillWidth: true
+                    wrapMode: Text.Wrap
+                    opacity: 0.7
+                    text: "With the displays off, the headset keeps tracking and drawing, so it can wake the moment "
+                          + "it moves. It still uses most of its power, so leave it on a charger that keeps up with it "
+                          + "in use."
                 }
             }
         }

display-settings/steam_settings.py

@@ -0,0 +1,49 @@
+"""Steam's sleep settings, read and written through Steam's own UI.
+
+Steam, not systemd, puts the Frame to sleep: after "When Plugged In and Idle -> Sleep after"
+(an hour by default) without input, even while it charges. That's a Steam client setting,
+`system_idle_suspend_ac_sec` (0 = never), with no file or command line to change it. Steam
+on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context
+(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080
+(steam/steamui.py). There `settingsStore.clientSettings` has the current values, and
+`SteamClient.Settings.SetSetting` takes a change as a serialized CMsgClientSettings protobuf,
+which is what Steam's own Settings -> Power page sends.
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "steam"))
+from steamui import SteamUnreachable, evaluate  # noqa: E402,F401  (callers catch SteamUnreachable here)
+
+# CMsgClientSettings field numbers (Steam's UI bundle maps the names to these).
+FIELDS = {"system_idle_suspend_ac_sec": 24004, "system_idle_suspend_battery_sec": 24003}
+
+
+def sleep_settings():
+    """{"ac": seconds, "battery": seconds}: when Steam puts the Frame to sleep without input,
+    plugged in and on battery (0 = never)."""
+    value = evaluate("(() => { const c = settingsStore.clientSettings; "
+                     "return {ac: c.system_idle_suspend_ac_sec, battery: c.system_idle_suspend_battery_sec}; })()")
+    if not isinstance(value, dict) or not all(isinstance(value.get(k), int) for k in ("ac", "battery")):
+        raise SteamUnreachable("Steam's settings don't have the sleep timeouts")
+    return value
+
+
+def set_sleep_setting(name, seconds):
+    """Sets one of FIELDS to a whole number of seconds and checks that Steam took it."""
+    field, seconds = FIELDS[name], int(seconds)
+    if seconds < 0:
+        raise ValueError("seconds must be 0 (never) or more")
+    ok = evaluate(f"""(async () => {{
+        const bytes = [];
+        const varint = n => {{ while (n > 127) {{ bytes.push((n & 127) | 128); n = Math.floor(n / 128); }} bytes.push(n); }};
+        varint({field} * 8); varint({seconds});
+        await SteamClient.Settings.SetSetting(btoa(String.fromCharCode(...bytes)));
+        for (let i = 0; i < 40; i++) {{
+            if (settingsStore.clientSettings.{name} === {seconds}) return true;
+            await new Promise(r => setTimeout(r, 50));
+        }}
+        return false;
+    }})()""")
+    if ok is not True:
+        raise SteamUnreachable(f"Steam didn't take {name} = {seconds}")

docs/controller-click-stability.md

@@ -0,0 +1,34 @@
+# Controller desktop click stability
+
+Trigger presses reach KDE immediately, but controller motion within 8 logical
+pixels of the press stays at that position until release. Releasing without a motion outside this
+zone delivers the click at the original position, even if the hand moved during
+release. Moving outside the zone begins a normal drag immediately; returning to
+the zone does not turn it back into a click. There is no hold-duration timer.
+
+This filters overlay pointer content events on desktop monitors only, and only
+presses from hand controllers start it. The 3D mouse (whose laser comes from the
+`ft_pointer` virtual controller), SteamVR UI, separate screen grab bars and
+floating-app title-bar carrying are unaffected. Multi-button gestures keep their
+existing behavior. A motion onto another desktop monitor starts a drag;
+cross-monitor motion is not stabilized.
+
+CLI (runtime preferences, reset to 8 on desktop restart):
+
+```sh
+input/ft-clickctl status
+input/ft-clickctl threshold 8
+input/ft-clickctl threshold 0  # disable without a restart
+```
+
+Thresholds are 0–64 logical pixels, normalized to each panel's KDE scale.
+Status reports held state, suppressed motions, stabilized clicks and drags.
+Changing the threshold while a controller button is held is refused.
+
+This is a separate contribution from desktop mouse/controller ownership. Its
+hardware validation must check small controls, intentional text selection,
+long presses, cross-monitor dragging and simultaneous mouse use. The existing
+renderer laser remains tracked; this change stabilizes desktop input rather
+than smoothing the visual laser. Default threshold is a starting point to test.
+
+Run `scripts/test-controller-click.sh` for the isolated gesture-state tests.

docs/design.md

@@ -42,18 +42,64 @@ Wherever ft-screens needs to know where a laser points (showing the controls, th
 
 `ComputeOverlayIntersection` ignores `SetOverlayIntersectionMask`, and a control can't be allowed to cover part of its screen, so the resize tab sits entirely outside the corner.
 
-### Wrist pinning
+### Pinning
 
 Pinning started as "bring the screen to your wrist", which doesn't work for big screens, because their centre is far from the edge you bring close. It became aiming: while a screen is carried, the line from the carrying device to its bar is tested against the other hand controllers. Crossing a controller's 6 cm ring arms the pin (leaving past 9 cm, so it doesn't flicker), and crossing it again disarms it. The pin happens on release, with the screen's pose at that moment, so you can arm it and then turn the screen. An earlier version pinned the moment the laser touched the wrist, which left the screen at whatever angle the carrying hand had while pointing there.
 
 A pinned screen's alpha follows the angle between its front and the direction to your head, fully visible inside the wrist angle and fading over the last 10°.
 
+A head pin is the same pin on the headset (device index 0): the screen's transform is relative to the headset, so SteamVR keeps it rigidly in your view with no lag from us. It skips the facing rule, since a screen on your head always faces you the way it did when pinned. There's no aiming gesture for it: the line from the carrying device can't sensibly pass through your own head, and a ring in front of your face would be in the way. So it's set from Frametop Display Settings or `ft-layout`, and it pins the screen where it is. Carrying a head-pinned screen re-pins it on release, like a wrist pin, so it can be adjusted in VR.
+
+### Named layouts
+
+Named layouts are now profiles, which also hold which screens are hidden and which apps to open, with where their windows go. See [profiles.md](profiles.md).
+
+A profile's screen part is the custom arrangement under a name: each screen's pose relative to your head, width, curve, and pin, but not its resolution or scale, which need a desktop restart or belong to KWin. Using one copies it into the custom arrangement, so everything that applies the layout (desktop start, Meta+Shift+R, Arrange now) works unchanged, and `active` remembers which name it came from. Saving without a name (`ft-layout capture`) clears `active`, because the screens have been placed by hand since. Layouts are kept per screen number, so one saved with a different screen count still applies: missing screens keep their last saved place or the preset's.
+
 ### Visibility and VR games
 
 `VROverlayFlags_MakeOverlaysInteractiveIfVisible` keeps SteamVR's laser mouse on while an overlay with that flag is visible. Without it, the laser is off whenever the dashboard is closed: the first click on a panel only turns it on, and the laser turns off again as soon as it leaves every panel. With it, controllers work the screens normally, but the laser also takes the controllers away from a VR game.
 
 `IVRApplications::GetCurrentSceneProcessId()` is 0 when no game is running (the Frame's home environment isn't a scene app) and the game's process ID while one is. ft-screens checks it twice a second, turns the flag off while a game runs, and by default hides the screens unless the dashboard is open. Flatscreen games run inside Steam's gamescope overlay and aren't scene apps, which is why "only with the dashboard open" is offered as a controller setting.
 
+## Floating windows
+
+[floating-windows.md](floating-windows.md) describes the feature and its parts. Drag and drop and the clipboard only work between windows of one compositor, so a floating window stays a KWin window and gets a KWin output of its own: one of the spare outputs KWin opens after the screens, shown by ft-screens as a panel cropped to the window. What follows is how KWin 6.2.5 behaves underneath that, from its source (`src/backends/wayland/`) and from trying it on the Frametop desktop.
+
+### KWin's nested outputs
+
+- Disabling a nested output keeps its host window. `Output::applyChanges` only flips `enabled`, KWin stops rendering it (no more commits), and Plasma drops its desktop view. So ft-screens keeps the same toplevel, and its screen numbers stay put. Enabling the output again resumes on the same toplevel.
+- Each output's host window is titled `KDE Wayland Compositor WL-<n>`, with `- Output disabled` appended while it's disabled (`WaylandOutput::updateWindowTitle`, on every `enabledChanged`). ft-screens reads the title to tell screens (`WL-0` to `WL-<SCREENS-1>`) from spares, and to see a spare turn on and off.
+- A spare resized while it's disabled comes up at the new size on its first frame, so floating a window needn't blink. Outputs with gaps between them are accepted, so ft-floatd places spares apart from the screens and from each other, within Xwayland's 32767-pixel limit.
+- KWin keeps a Wayland popup inside its parent's output (`XdgPopupWindow::updateRelativePlacement` uses the output's placement area), and X11 apps place their menus within the monitor. That's why a floating window's output has a margin around the window: menus and dropdowns open past the window's edges, into the margin.
+- KWin makes a nested output the size it's configured to times its scale, rounded (at 1.5 it lays out 1067 × 667 on a 1600 × 1000 buffer), and gives the buffer a whole buffer scale (1.2 becomes 2). A buffer whose size isn't a multiple of that is a protocol error that disconnects KWin, so ft-floatd sizes spares in multiples of it. After a scale change, ft-floatd asks for the output's size again in the new scale's terms, or the next configure would make it the old size times the scale.
+- **Virtual outputs don't work.** `createVirtualOutput` makes an output window but never adds it to the backend's `m_outputs`, so `findOutput()` returns null when the pointer enters it, and the next line dereferences it (`Q_ASSERT` is compiled out). A click on such a panel would crash KWin. This rules out virtual outputs (`stream_virtual_output`) for floating windows without a patched KWin.
+
+### The pointer
+
+- Pointer positions reach KWin only through motion events: the output's position in the layout plus the position on its window. When ft-screens stops sending motion, KWin's pointer stays put.
+- KWin starts an interactive move on the press itself, before any motion. So ft-screens stops sending motion as soon as a press lands in a floating window's title bar (from the frame and client rectangles ft-floatd sends it), with no round trip, and carries the panel instead. KWin gets the release at the press point, and the window moves by nothing on its output.
+- KWin's nested backend ignores the position in `wl_pointer.enter`, and wlroots drops a motion to the position it entered at, so the first click after crossing onto another panel landed where KWin's pointer had been. ft-screens enters one unit off.
+
+### The KWin script
+
+The KWin side is a script (`float/frametop-float.js`), not a C++ effect, because a script keeps working across KWin updates and an effect would have to match the host's exact KWin build. KWin scripts can call D-Bus but can't serve it, so ft-floatd's commands come back through a long poll: the script calls `NextCommand`, which answers when a command is ready, or empty after 20 seconds, under KWin's 25-second D-Bus timeout. A few things about KWin's script engine:
+
+- `windowAdded` reports popups as windows of their own (`popupWindow` true, `transient` true) with their geometry.
+- Setting `frameGeometry` applies asynchronously: the app has to answer the new size first.
+- A script can't read a window's maximize mode, so the script counts a window as maximized when it fills its output's maximize area.
+- `globalThis` isn't defined. `print` goes to the journal unless `QT_FORCE_STDERR_LOGGING=1`.
+
+The title bar's float button is Frametop's own window decoration (`decoration/`), written in QML for KWin's Aurorae engine, which loads it without compiling. A C++ fork of Breeze would have to match SteamOS's exact KDecoration build. A decoration can only make the window requests KWin offers it, so the button toggles keep-below, which has no visible effect on a window alone on its own output, and the script treats keep-below as the floating flag.
+
+### KWin's placement memory
+
+KWin keeps each window's geometry, full screen, and maximized state for each layout of the outputs (its `PlacementTracker`, keyed by every enabled output's name and geometry). When the outputs come back to a layout it has seen, it puts every window back as it was in it. That's for plugging monitors in and out, and it does harm here. A spare output changes size after its window does, so what KWin keeps for a spare's size is the window's next size. Resizing a floating window back to an earlier size made the window and its output flip between two sizes for good (Dolphin went between 1187 and 1424 logical pixels wide, its output between 1687 and 1925). Full screen flipped the same way, and floating or docking one window could move others, even onto a spare or off one.
+
+So the KWin script keeps where each window belongs: where ft-floatd put it, or where it went outside an output change. While KWin changes the outputs, it reports nothing to ft-floatd. Once KWin is done (`screensChanged` comes after its restore), it puts floating windows back, and the screens' windows too when only spares changed. KWin's resize request hasn't reached the app by then, so the app never sees it. A size asked for is held for a second, since an app can still answer an older request, and then the script takes the size the window has. If `screensChanged` doesn't come within 2 seconds, the script stops waiting for it.
+
+KWin also ends an interactive move or resize whenever the outputs change. So during a resize by a floating window's edge, ft-floatd only crops the panel to the window, and resizes the output when the drag ends. The margin is the room to grow until then.
+
 ## The 3D mouse
 
 The mouse works like the pointer on the Apple Vision Pro: a small cursor floats in the room, lands on whatever panel it meets, and acts on it like a controller's laser.
@@ -64,11 +110,13 @@ SteamVR's dashboard and every overlay it hosts are driven by the vrcompositor `l
 
 Driver poses are in SteamVR's raw tracking space, and client programs work in the standing universe, which on the Frame is about 1.6 m above raw. Mixing them up put the laser's origin 1.6 m above your head. The helper converts using the headset's pose in both spaces every frame.
 
+Frametop's SteamVR clients (ft-pointer, ft-screens, ft-gaze) connect as a background app first and switch to an overlay app only once that works. `VR_Init` as an overlay app starts vrserver itself when none is running, and one started that way from the dev container never finds the headset. At a boot where the gamescope session timed out, systemd dropped `steamvr.service`'s start job, the pointer service (ordered only `After=` it) started anyway, and its vrserver made every SteamVR launch fail with `HmdNotFound`. SteamOS's health check then kept resetting the Steam client and tried to fall back to the previous OS slot. The units also say `Requisite=steamvr.service`, so they don't start at all when SteamVR's start fails.
+
 The driver starts disconnected, because holding the right-hand role while SteamVR starts leaves the Steam UI stuck on its loading icon. It connects when the mouse is used and claims the right hand. SteamVR keeps a hand role reserved for a disconnected device that still asks for it, so the driver switches its role hint between right hand (connected) and opt-out (not connected).
 
 ### The cursor
 
-Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it.
+Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either.
 
 OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately.
 
@@ -83,7 +131,7 @@ A few overlays need special handling:
 
 Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
 
-Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse or controller button mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz, and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze; the mouse is gaze mode's only pointer device for now. The dot shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. The gaze moving the pointer doesn't show it: you know where you're looking. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured.
+Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still up to the click; a right click or MetLine truncated
 
 Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
 
@@ -91,7 +139,7 @@ Replacing a loaded driver's files, as re-running the installer used to do, leave
 
 ### Handing the laser back and forth
 
-The dashboard follows whichever device summoned it or last pressed its trigger. Frametop adds "last used wins": moving a real controller releases the pointer at once, and the next mouse movement takes the laser back. Small movements don't count; waking needs `POINTER_WAKE_COUNTS` of mouse motion within a second, so desk jitter doesn't steal the laser. While the pointer is awake, a tiny transparent overlay with `MakeOverlaysInteractiveIfVisible` keeps SteamVR's laser mouse on, since otherwise the first click would only switch the laser on.
+The dashboard follows whichever device summoned it or last pressed its trigger. Frametop adds "last used wins": moving a real controller releases the pointer, and the next mouse movement takes the laser back. Moving means faster than 0.35 m/s or 2 rad/s (both times `POINTER_CONTROLLER_PICKUP`, 1 by default) for 100 ms in a row, while the controller is tracked normally. A single sample over the limit used to be enough, and controllers resting on a desk took the laser back on a knock or a tracking jump while the mouse was in use. Small movements don't count; waking needs `POINTER_WAKE_COUNTS` of mouse motion within a second, so desk jitter doesn't steal the laser. While the pointer is awake, a tiny transparent overlay with `MakeOverlaysInteractiveIfVisible` keeps SteamVR's laser mouse on, since otherwise the first click would only switch the laser on.
 
 When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off.
 
@@ -109,6 +157,10 @@ An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every i
 
 Volume keys must never reach gamescope. With the openvr backend, gamescope sends volume up and down to Steam by moving keyboard focus to Steam for the key and then back to the previously focused surface. When nothing had focus, the one it moves back to is null, and wlroots aborts on a null focus surface (`wlr_seat_keyboard_notify_enter: Assertion 'surface' failed`), which ends the whole VR session. Keyboard focus is often empty while you work in VR, so one press of the headset's volume button could take everything down. gamescope reads the headset's buttons and every keyboard itself (`InputStealer`), as do SteamVR's processes, so the relay has to stop volume keys at the device. Grabbing `gpio-keys` would also take the headset's click button, so the relay remaps the volume entries in each device's keymap (`EVIOCSKEYCODE`) and handles the stand-in codes itself. That fix covers every device at once, including keyboards that aren't grabbed.
 
+Frametop's keyboard opens by itself for a text field on the desktop. The apps run inside the nested KWin, so only KWin knows when a text field has focus, and the way it tells anyone is its input method protocol (`zwp_input_method_v1`): KWin starts one input method program and activates it whenever the focused app turns on text input. `input/ft-textinput` is that program, speaking the Wayland wire protocol directly so it needs nothing but Python on the host. It only reports focus. The gamescope session puts `QT_IM_MODULE=xim` and `GTK_IM_MODULE=xim` in the systemd user environment; with those, Qt and GTK apps use X input methods and never turn on Wayland text input, so the session script drops them.
+
+The keyboard itself is ft-screens' own panel (`screens/keyboard.cpp`). We tried SteamVR's first (`ShowKeyboardForOverlay`), and on the Frame it doesn't fit a desktop. It's Steam's own panel (`valve.steam.gamepadui.keyboard`), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (`SetKeyboardTransformAbsolute`) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, like `@frametop_keys`), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU and uploaded with `SetOverlayRaw` when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.
+
 The Frame controllers can be mapped like mouse buttons, but they aren't input devices on the host: they reach SteamVR over the headset's own radio, and no evdev or hidraw node exists for them. So only a SteamVR client can read them. Overlay apps normally get controller input only while they have input focus, which a background helper never has. SteamVR's experimental global action set priority (`steamvr/globalActionSetPriority`, "Enable global input from overlays") lets an overlay's action set receive input anyway, and takes the inputs it binds from the scene app. Binding every button would take them all from games, so the helper's action manifest puts each button in an action set of its own, and it activates only the sets of mapped buttons. The mapping itself stays in the relay, which does the action, so mice and controllers share one list of actions.
 
 ## The desktop session
@@ -117,14 +169,41 @@ The session is modeled on SteamOS's `steamos-nested-desktop` and runs beside it.
 
 The VR launcher starts the session from the Steam client, and the client's environment came along: `LD_LIBRARY_PATH` pointing at Steam's own runtime, whose `libavcodec` has no H.264 decoder, so VLC in the desktop couldn't play most videos, plus the client's overlay and launch settings. The session script drops the client's variables before it starts anything. SteamOS's global Mesa settings (`/usr/share/deckard/mesavars.sh`) stay, and the gamescope session's Vulkan layer (`ENABLE_GAMESCOPE_WSI`) is only kept for the gamescope backend.
 
-Steam, not systemd, suspends the Frame: after `system_idle_suspend_ac_sec` (an hour by default) without input on AC power, it logs `Switching to power state: k_ESystemPowerState_Sleep` and suspends, even while charging. It's a Steam setting (Settings → Power → When Plugged In and Idle → Sleep after), so the README recommends setting it to Never. SteamVR's standby, which turns the displays off when the headset comes off, is separate.
+Steam, not systemd, suspends the Frame: after `system_idle_suspend_ac_sec` (an hour by default) without input on AC power, it logs `Switching to power state: k_ESystemPowerState_Sleep` and suspends, even while charging. It's a Steam setting (Settings → Power → When Plugged In and Idle → Sleep after), which the Stay awake while plugged in switch in Frametop Display Settings sets to Never. SteamVR's standby, which turns the displays off when the headset comes off, is separate; see below.
 
 Flatpak apps need `XDG_DATA_DIRS` to include Flatpak's exports, or Plasma opens Discover instead of launching them, so the session sources `/etc/profile.d/flatpak.sh`.
 
-A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it.
+The private runtime directory also moves the session's document portal to `$XDG_RUNTIME_DIR/frametop/doc`, and that broke saving and uploading in Flatpak apps. The file picker (xdg-desktop-portal 1.18.4 on SteamOS) gives a sandboxed app the host path of the file it picked, `/run/user/1000/frametop/doc/ID/NAME`. Inside the sandbox the portal is at `/run/flatpak/doc`, and `/run/user/1000` is a private per-app folder (`.flatpak/APP/xdg-run` in the runtime directory). So Brave created the missing folder there, "finished" the download into it, and the file vanished when the session cleaned up. The session script now links that path to `/run/flatpak/doc` in each installed app's folder before Plasma starts. Upstream xdg-desktop-portal fixed this after 1.22.1 (commit `69ba5e1`) by handing Flatpak apps `/run/flatpak/doc` paths, after which the links go unused.
+
+A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. It then waits for distrobox-init to log `container_setup_done`, as `distrobox enter` does only for containers it starts itself. A new container's first start takes a minute or more (it installs distrobox's dependencies and sets up passwordless sudo), and an install that entered right away met a sudo password prompt with no terminal to answer it ([#9](https://github.com/DeeJanuz/frametop/issues/9)).
 
 Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`.
 
+## Displays off on a stand
+
+SteamVR decides the headset is off from its proximity sensor, which the driver reads through the DSP, and turns the displays off 5 seconds later. On a display mount that covered the sensor, that never happened: SteamVR kept the headset in use all night (no `entering standby` for device 0 in vrserver.txt, and XRService's user presence stayed at 1), and Steam didn't sleep either, because its idle count treats a present user as active. The battery went from 100% to 12% overnight on a 5 V, 3 A charger, with the headset drawing about 17 W.
+
+There's no client call that puts the headset in standby. The cv driver's `teststandby` debug request (`IVRDebug::DriverDebugRequest`) only answers "Standby unknown hmd" on the Frame. But what the driver does for the displays in standby is write `/sys/class/backlight/ae94000.dsi.0/brightness` ("cv: Set displays off" writes 0, "Set displays on" the old value), and the `video` group can write that file, from the container too. So `ft-powerd` goes by use instead of the sensor and turns the backlight off itself. Tracking and rendering keep running. Turning the backlight off moved the battery current by only about 75 mA (0.5 W), so they're most of the load, but they're also why the displays can wake the moment the headset moves.
+
+Movement is judged within 10-second windows. On the mount, the head pose jittered within 0.5 mm and 0.1 degrees over 20 seconds, and its position drifted 1.7 mm (0.16 degrees) in 4 minutes. Compared with a fixed reference, that drift would count as movement sooner or later and keep the displays on; within 10 seconds it never reaches the 5 mm and 0.5 degree thresholds, and anyone wearing the headset passes them now and then.
+
+Staying awake while charging uses Steam's own setting rather than a logind sleep inhibitor. Steam suspends with `dbus-send ... login1.Manager.Suspend boolean:true`, and a block inhibitor does stop that (`CanSuspend` answers "challenge" while one is held), but it stops the power button too. `system_idle_suspend_ac_sec` is field 24004 of Steam's CMsgClientSettings. In Steam's SharedJSContext, reachable over CDP on port 8080 because Steam runs with `-cef-enable-debugging`, `SteamClient.Settings.SetSetting` takes a change as a base64 protobuf, the way Steam's Power page sends it (0 is never), and `settingsStore.clientSettings` has the current values.
+
+## Pausing for VR games
+
+Hiding the screens during a game kept them out of view, but Frametop kept using the headset. Measured on 2026-10-02 with gaze mode off and no game running, in shares of one core: our eye tracker (ft-eyes) about 60%, ft-eyegrab, ft-gaze and ft-gazed about 3 to 4% each; remote desktop (krdpserver, FreeRDP, Xvnc) about 2 cores while it ran; KWin about 13%, ft-screens about 4%. The gaze service ran at full rate whether gaze mode was on or not; now it idles while the gaze isn't used (gaze/README.md), and pausing stops it outright. Reading SteamVR's eye tracking 90 times a second also made it restart every 10 to 13 seconds during Beat Saber, and each restart took input focus from the game, which paused it (PR #13; since then ft-gaze skips SteamVR's gaze action during games, but our own tracker kept running). So pausing stops what costs the most and leaves windows where they are.
+
+- A hidden screen still cost as much as a visible one. ft-screens sent every committed screen its frame callback at 90 Hz whether its overlay showed or not, so KWin kept drawing, and its apps with it. Paused, ft-screens sends the callbacks once a second. A Wayland client draws again only after its last frame's callback, so KWin's output stalls, KWin's own clients stop getting theirs, and the whole desktop idles, without anything losing its connection. A second's pace, rather than none, keeps any client that waits on a callback from waiting forever. Stopping KWin or the apps with SIGSTOP would free the same, but a Wayland peer that stops reading overflows the other side's 4 KB socket buffer, which ends the connection: that's how the live desktop died once when its KWin stalled (`Data too big for buffer`). They also sit in different cgroups (KWin under steam.service when the VR launcher starts it, ft-screens in the dev container's), so no single freeze stops them together.
+- The relay does the pausing because it's the one part that always runs, and the pointer helper keeps running because stopping it leaves its virtual controller connected with its last pose (the driver has no staleness timeout), maybe holding a hand role, with the 3D mouse dead. Releasing it does the job. The helper already checks for a scene app twice a second, so it's what tells the relay a game started.
+- The gesture has to work during a game, but SteamVR input reaches only the app with input focus, and an overlay with global input (`steamvr/globalActionSetPriority`) takes the buttons it binds from the game. vrserver's web socket on 127.0.0.1:27062, which its controller binding page uses for the live view, reports every controller component whatever has focus, and reading it takes nothing. The game sees the clicks too, so the default is a gesture games hardly use: both thumbsticks, together, twice. "Together" means within 0.3 seconds of each other, so a stick held down to sprint while the other clicks doesn't count. The stream is about 160 messages a second, nearly all capacitive sensing, so the reader parses only the few that mention a gesture's button. A controller's root path changes while the 3D mouse holds its hand role (`/devices/cv/<serial>` instead of `/user/hand/right`), so the reader looks the controllers up again every 3 seconds.
+- Resuming starts remote desktop through `systemd-run --scope`: started straight from the relay, it would join the relay's cgroup and end with the next relay restart.
+
+## SteamOS updates
+
+On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-steamvr-rel` package), next to KWin, gamescope, and the kernel, so every SteamOS update can bring a new SteamVR too. Frametop survives updates: it lives in the home folder and the `dev` container, the Bluetooth fixes are in `/etc`, which SteamOS keeps across updates, and nothing goes into `/usr`. What an update can break is what Frametop uses from the image. The public OpenVR API is versioned and stays put. The rest is less certain: `IVRIPCResourceManagerClient`, which is newer than the header SteamVR ships; the text `vrcmd --overlays` prints; the eye tracker's shared memory layout; XRService's camera buffers; KWin's nested backend; and behavior Frametop works around, such as the SteamVR Settings page that `ComputeOverlayIntersection` can't find or the scale KWin's nested backend doesn't undo.
+
+`scripts/update-check.py`, which `scripts/doctor.sh` runs, checks what it can directly: that SteamVR still serves every OpenVR interface version the installed programs were built against (read from the binaries), that `vrcmd`'s format still parses, that the eye tracker's sample timestamp is still at the offset ft-gaze reads, and the host files, services, sockets, and driver registration. Behavior can't be checked without someone in the headset, so it records the versions of the packages that matter once things work (`--mark-good`), and after an update names what changed and what to try by hand.
+
 ## Approaches we dropped
 
 - WayVR, an existing Wayland desktop for VR. It built and connected to SteamVR on the Frame, but nothing showed in the headset. It has no bindings for the Frame's controllers, and its KDE screen capture needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship.
@@ -133,8 +212,8 @@ Program names stay within 15 characters, because Linux truncates process names t
 
 ## Open questions
 
-- A head-locked screen, like a HUD.
 - A controller button that shows the screens during a game. Games own the controllers, so this needs SteamVR input actions for ft-screens.
 - Drawing KWin's cursor on the screens.
 - Plasma can lose its panels when the number of screens goes down, because they're saved against a screen that no longer exists. Removing `plasma-org.kde.plasma.desktop-appletsrc` and `plasmashellrc` from `~/.config/frametop` brings the default panels back.
 - Frame pacing and GPU cost with several busy screens haven't been measured.
+- Real standby on a stand, with rendering and tracking paused, not just the backlight off. SteamVR has no call for it, and its activity level follows the proximity sensor.

docs/floating-windows.md

@@ -0,0 +1,170 @@
+# Floating windows
+
+Any desktop app can float in VR in a panel of its own, like SteamVR's floating windows, while it stays part of the Frametop desktop. Drag and drop, the clipboard, and focus keep working between floating windows and the screens.
+
+- **Float a window** with "Float in VR" in its window menu (Alt+F3), the float button left of Close in its title bar, or the float key (Meta+Shift+F by default). Start an app floating with "Launch as Standalone" in its right-click menu in the Application Launcher or the taskbar, with `ft-float launch` or `ft-float run`, or from a profile ([profiles.md](profiles.md)).
+- **Put it back** with the dock button under its panel, the title bar button, the float key, or "Back to Desktop" in the window menu. It returns to the screen, position, and size it came from. The `dock_all` action docks every floating window.
+- **Move it** by its title bar or the bar under its panel, **resize it** by its edges or the corner tab, and **change its scale** with Meta+scroll over it. Each app's last floating place, size, and scale are remembered.
+- Files, text, and images drag between any two floating windows, and between floating windows and the screens.
+
+How KWin behaves underneath all this, and what the KWin script does about it, is in [design.md](design.md#floating-windows).
+
+## Not built
+
+These were decided (see the table) but aren't built:
+
+- Tearing a window off a screen by dragging its title bar into the air, with a ghost of it on the laser (decision 3; see "Tearing a window off a screen").
+- Docking by pushing a floating window flush against a screen, with the landing spot highlighted (decisions 4 and 17).
+- New windows of a floating app placed where that app's windows went last time, or to the parent's right (decision 15).
+- +/- scale buttons on the floating panel's bar (decision 18). Meta+scroll changes the scale.
+- The glow at the edge of your view toward a floating window activated out of sight, and the setting that brings it in front of you instead (decision 19).
+- A Floating windows section in Frametop Display Settings (decision 1). `FLOAT_SLOTS` and `FLOAT_MARGIN` are set in `~/.config/frametop.conf`.
+- A drag proxy on the catcher, so a drag's icon shows while the laser is between panels.
+- Keeping floating windows out of Show Desktop (Meta+D) (decision 9). Nothing handles it yet.
+
+## Decisions
+
+The numbers are cited in the code, so they stay as they are. Struck-out text was replaced by a later decision.
+
+| # | Question | Decision |
+|---|---|---|
+| 1 | How many windows can float at once | 8 spare outputs by default, configurable with `FLOAT_SLOTS` (at most 16); a change needs a desktop restart. A Display Settings control isn't built |
+| 2 | Menus and dropdowns | Each floating output has a margin around the window. The panel shows only the window, and each open popup gets a small overlay of its own, cut from the same buffer |
+| 3 | Tearing off | Not built. Drag the title bar past a screen's edge and let go in the air, with a small dead zone past the edge |
+| 4 | Docking by dragging | Not built. Push the window flush against a screen (within about 10 cm), with the landing spot highlighted, and let go |
+| 5 | Visibility | Floating windows follow the same rules as the screens: the hide hotkey, the visibility modes, and the games rule |
+| 6 | Windows a floating app opens | They float too |
+| 7 | Launching floating from the headset | ~~One "Frametop Apps" launcher entry with a picker~~ Replaced by 26 and profiles (27) |
+| 8 | Build order | Not kept here: it only set the order of the work |
+| 9 | Show Desktop (Meta+D) | Floating windows stay. Not built |
+| 10 | Frametop Apps and visibility | ~~The entry starts the desktop with each screen hidden on its own, so only floating windows show~~ Replaced: a profile can hide screens (27) |
+| 11 | Window frame | KWin's title bar and border stay. Frametop's bar, close, and "back to desktop" are extras |
+| 12 | Resizing | The window's own edges and Frametop's corner tab both change the size in pixels at the same density; the output follows |
+| 13 | Margin | 300 px on each side, configurable (`FLOAT_MARGIN`) |
+| 14 | All spares in use | The window stays on the screens, with a notification |
+| 15 | Where a floating app's new windows go | Not built: where that app's windows went last time, otherwise to the parent's right, curving around you. For now, a new window that opens on a floating window's output floats a little in front of it |
+| 16 | Where the code is written | Not kept here: it was about the work, not about Frametop |
+| 17 | Size when docked by dragging | Not built (see 4): the current floating size in pixels, shrunk to fit the screen |
+| 18 | Bigger text | A scale for each window (KWin's output scale): Meta+scroll over the window. Remembered for each app. +/- buttons on its bar aren't built |
+| 19 | Switching to a window you can't see | It's focused. Not built: a glow at the edge of your view that points to it, and a setting that moves it in front of you |
+| 20 | Full screen | The window fills its own panel. The margin drops to zero while it's full screen, and the panel keeps its size and place |
+| 21 | Named layouts | ~~They cover the screens only~~ Replaced by profiles (27). Outside a profile, floating windows use the placement remembered for each app |
+| 22 | The float key | One toggle: it floats a window, or docks it if it already floats. The input relay owns it (`float_toggle`), Meta+Shift+F by default, rebindable in Frametop Input Settings and mappable to mouse and controller buttons. KWin has no shortcut of its own for it, so one press can't fire twice |
+| 23 | Which window the key acts on | The window under the desktop's pointer; the active window if there's none there (the wallpaper, the taskbar) |
+| 24 | Docking everything | A `dock_all` action, with no default binding |
+| 25 | A button on every window | A float button left of Close in the title bar, from Frametop's own QML window decoration, made to look like Breeze. It shows a dock icon on floating windows. Apps that draw their own title bar (Chromium, Electron, GTK) use the key |
+| 26 | Launching one app floating | "Launch as Standalone" in the right-click menu of every app in the Application Launcher and the taskbar, from copies of the apps' desktop files that only the Frametop desktop reads. It replaces the Frametop Apps entry (7, 10) |
+| 27 | Profiles | Named layouts become profiles: the screens' places, which screens show, and the apps and their windows, floating or not. See `docs/profiles.md` |
+
+Also: floating windows get the wrist pin, the head pin, and pass-through (`pointer-ignore`) like screens. Every gesture works with the controllers as well as the 3D mouse. A window launched floating uses the primary screen's density. VNC shows only the primary screen.
+
+## The approach: each floating window gets a KWin output of its own
+
+Drag and drop and the clipboard only work between windows of the same compositor. A Wayland window can't move from one compositor to another. So a floating window has to stay a KWin window.
+
+ft-screens already shows each KWin output as a panel. It sets the output's size with an `xdg_toplevel` configure, and KWin resizes the output to match. So a floating window gets an output of its own, sized to fit it, and ft-screens shows that output as a panel with its own controls. To KWin this is an ordinary desktop with more monitors. Dragging between two floating windows is the same as dragging between two monitors, which KWin already handles. ft-screens moves the pointer between panels in the middle of a drag: `handle_vr_event` moves pointer focus to another KWin window even while a button is held.
+
+Alternatives considered:
+
+- **Run floating apps directly on ft-screens.** It's a wlroots compositor, so apps could connect to it and get a panel per window. But they would get no drag and drop or clipboard with desktop apps without a bridge. Also, a window that's already on the desktop could never float, because a Wayland client can't change compositors. Rejected.
+- **One large hidden "canvas" output.** Every floating window would sit on one big output, and each panel would show a crop of it (`SetOverlayTextureBounds`). That needs only one extra output, with no copies. But an 8K canvas uses about 128 MB per buffer, with two or three buffers in KWin's swapchain. It would also have to repack windows whenever one resized, full screen would fill the whole canvas, and every window would share one scale. It was the fallback in case per-window outputs didn't work.
+- **Screencast single windows** (`zkde_screencast` `stream_window`, over PipeWire). This adds copies and latency, and the window still needs a real place in KWin's layout to receive input. Rejected.
+- **SteamOS's own floating windows** (Launch a program from the dashboard). Those apps run in gamescope, outside KWin, so they can't drag and drop with the desktop.
+
+### Where the extra outputs come from: spare outputs
+
+KWin's nested backend opens its outputs at start (`--output-count`). The session starts KWin with the screen count plus `FLOAT_SLOTS` outputs (default 8, at most 16). ft-floatd turns off the spares nothing floats on with `kscreen-doctor` once it starts. Floating a window enables a spare, and docking the window disables it again. `FLOAT_SLOTS` limits how many windows can float at once, and changing it means restarting the desktop. How KWin's nested backend treats disabled outputs, and why its virtual outputs can't be used instead, is in [design.md](design.md#floating-windows).
+
+ft-screens creates a `screen` for each toplevel in the order they appear, and indexes its settings by that order. Spares come after the screens, so they get indices `SCREENS` and up. Their panels are hidden while their output is disabled.
+
+## How the parts fit together
+
+```
+  KWin script "frametop-float"                ft-floatd (host, Python)                 ft-screens
+  window events, moves, menus  ── D-Bus ──▶   window ↔ output ↔ panel table   ── @ft_screens ──▶   panels, controls,
+  runs commands                ◀─ long poll ─  spare outputs (kscreen-doctor)  ◀─ @frametop_float ─  lasers, catcher
+```
+
+- **KWin script `frametop-float`** (`float/frametop-float.js`). ft-floatd loads it into the desktop's KWin over D-Bus (`org.kde.kwin.Scripting`). A script keeps working across KWin updates. A C++ effect would have to match the host's exact KWin build, and the build container is Fedora, not SteamOS. The script watches windows (`windowAdded`/`windowRemoved`, `frameGeometryChanged`, `outputChanged`, `interactiveMoveResizeStarted`/`Finished`, `fullScreenChanged`, `maximizedChanged`, `minimizedChanged`, `keepBelowChanged`, `windowActivated`) and the outputs (`screensChanged`). It runs commands: move a window to an output, set its geometry, put it on all virtual desktops, and restore it. It adds "Float in VR" ("Back to Desktop" on a floating window) to the window menu (`registerUserActionsMenu`). It registers no shortcut: the float key belongs to the input relay. KWin scripts can call D-Bus but can't serve it, so commands come back through a long poll. The script calls ft-floatd's `NextCommand`, which answers when a command is ready, and then the script calls it again. It also keeps KWin's placement memory from moving windows (see design.md).
+- **ft-floatd** (`float/ft-floatd`, Python). The host has dbus-python and PyGObject. It runs inside the desktop's Plasma session, started from its autostart. It owns `org.frametop.Float` on the session's private bus, and it keeps the table of which window is on which output and panel. It enables and disables spare outputs and sets their scale and position with `kscreen-doctor`, and their size through ft-screens. It tells ft-screens where each floating window goes and tells the script which window goes where. It launches apps floating, opens profiles' apps, and remembers each app's placement and scale, keyed by desktop file name. Commands come in on `@frametop_float`, from `ft-float`, the input relay, and ft-screens.
+- **ft-screens.** A spare output's panel is a floating window's. Floating panels get the same bar, curve, roll, resize tab, and wrist and head pins as screens, plus dock and close buttons left of the bar. Other parts: the catcher, popup and dialog overlays, and carrying a panel during a KWin move. `MAX_SCREENS` (screens and spares together) is 24. Commands arrive on `@ft_screens`. Events go out to `@frametop_float` from an unbound socket, the same way ft-screens talks to the input relay.
+- **Session script.** Adds `FLOAT_SLOTS` to KWin's output count, starts ft-floatd from the desktop's autostart, installs Frametop's window decoration and chooses it in the session's `kwinrc`, and writes the Launch as Standalone copies of the apps' desktop files.
+- **ft-layout.** Arranges only the screens' outputs, and leaves the spares (`WL-<SCREENS>` and up) to ft-floatd, enabled or not.
+- **ft-pointer.** The drag lock crosses onto other Frametop panels (see "Drag and drop between panels"), and a left release also goes to ft-screens as a backstop for the catcher.
+- **Input relay.** Owns the float key: `float_toggle` and `dock_all` send `float pointer` and `dock all` to ft-floatd.
+
+Program names stay within 15 characters (`ft-floatd`). Overlay keys are `frametop.float.N` and `frametop.float.N.bar`, and so on; a floating window's popups and dialogs are `frametop.float.N.sub.K`.
+
+## A floating window
+
+- **Output and margin.** Its output is the window's frame plus a margin on each side (`FLOAT_MARGIN`, default 300 px). KWin keeps a Wayland popup inside its parent's output, so the margin gives menus and dropdowns room past the window's edges. X11 apps place their own menus within the monitor, so the same applies. Enabled spares sit apart from the screens and from each other in KWin's layout, so nothing spills from one to the next. Memory: a 1600 × 1000 window with a 300 px margin is about 14 MB per buffer, 42 MB for three.
+- **What the panel shows.** Only the window's frame: ft-screens crops the output's buffer with `SetOverlayTextureBounds` and maps mouse positions through the crop. Each open popup or dialog gets a small overlay of its own, cut from the same buffer and placed a few millimetres in front of the window, so the main panel never changes size. KWin tells scripts about popups as windows of their own (`windowAdded` with `popupWindow`), so the script reports their rectangles.
+- **Where it appears.** Floated from a screen, the panel starts where the window was on that screen, 30 cm in front of it. Launched floating, it goes where that app last floated, or in front of you, 0.8 to 2 m away.
+- **Size and scale.** The panel's width is the window's pixel width times the source screen's metres per pixel, so text stays the same size in VR. A window launched floating uses the primary screen's density. Each window also has a scale (KWin's output scale), changed with Meta+scroll over the window in steps of 10% and remembered for each app. A bigger scale makes the content bigger at the same panel size.
+- **Window state.** An ordinary window, not maximized, placed inside its output with the margin around it, and set to show on all virtual desktops. It keeps its title bar and border. Apps that draw their own title bar (GTK, Chromium) keep theirs.
+- **Moving.** Press the title bar. KWin starts an interactive move on the press itself, before any motion, so ft-screens stops forwarding pointer motion to KWin as soon as a press lands in a floating window's title bar (from the frame and client rectangles ft-floatd sends it). KWin's pointer stays at the press point and the window moves by nothing. For apps that draw their own title bar, the script reports the move and ft-screens stops then (`carry`); ft-floatd puts back any few pixels the window slipped before that. Meanwhile ft-screens carries the panel with the pressing device, the same way the bar does: it follows rigidly, scroll pushes and pulls, and the 3D mouse's right-drag tilts. When the button comes up, KWin gets the release at the press point. The bar under the panel works too.
+- **Resizing.** The window's own edges (inside the margin, so KWin's resize works as on the desktop) and Frametop's corner tab both change the window's size in pixels at the same density, so the app lays itself out again. ft-floatd resizes the output to keep the margin, and the panel grows or shrinks around the window's top-left corner. KWin ends a resize by the window's edge whenever an output changes, so during one the panel follows the window and the output follows only when the drag ends: the margin is the room to grow until then. A screen's tab only scales the panel. Resizing is throttled to about 20 updates a second, with a minimum of 320 × 200, like screens.
+- **KWin's placement memory.** KWin puts windows back where they were for each layout of the outputs it has seen, which fights spare outputs that follow their windows' sizes. The KWin script undoes it ([design.md](design.md#kwins-placement-memory)).
+- **Full screen.** The window fills its own panel: the margin drops to zero while it's full screen, and the output is the panel's size in pixels. The panel keeps its size and place. On leaving full screen, the margin comes back.
+- **Buttons.** The close button closes the window. The dock button docks it where it came from.
+- **Minimize.** Minimizing, from the title bar or the taskbar, hides the panel, and restoring it shows the panel again. Floating windows stay in the desktop's taskbar and in Alt+Tab.
+- **New windows.** Popups and dialogs of a floating window (`transientFor`) show as small overlays over it. Another window of a floating app that opens on its output floats too, a little in front of it. Any other window that opens on a floating window's output goes to the first screen that shows. When every spare is in use, the window stays on the screens and a notification says so.
+- **On a hidden screen.** A new window that opens on a screen hidden on its own floats instead, where that app last floated or in front of you.
+
+## Tearing a window off a screen (not built)
+
+The design for decision 3:
+
+1. Press a desktop window's title bar and drag it. KWin starts a move, and the script tells ft-floatd, which tells ft-screens: `move-start <output> <window> <rect>`.
+2. While the button is held, the laser leaves every Frametop panel by more than a small dead zone (a few centimetres past the edge). Letting go inside the dead zone is an ordinary drop.
+3. ft-screens shows a ghost: an overlay showing the screen's live buffer cropped to the window (`SetOverlayTextureBounds`, no copy). It's at the screen's pixel density and distance, on the laser, facing you, with the point you grabbed under the laser. The ghost takes mouse input, so SteamVR's laser lands on it and the release comes to ft-screens.
+4. Go back onto a screen before letting go, and the ghost disappears. It's an ordinary move again.
+5. Let go on the ghost, and ft-screens releases the button in KWin, which ends the move. It then reports the tear-off to ft-floatd, with the window, the ghost's pose, and the density. ft-floatd enables a spare output and has ft-screens size it to the window plus the margin and put its panel at the ghost's pose. Then it has the script move the window onto that output. The ghost stays until the new panel's first frame at the right size arrives, so nothing blinks.
+
+## Putting it back
+
+- **Button.** The dock button returns the window to the screen, position, and size it had before it floated. If that screen is hidden now, it goes onto the first screen that shows.
+- **Dragging (not built).** Carry the floating window, by its title bar or its bar, until the spot you're pointing at is on a screen. Then push it flush with the screen, within about 10 cm of its surface: scroll away with the mouse, or move the controller forward. The screen shows where the window will land, and letting go docks it there at its current size in pixels, shrunk to fit if the screen is smaller. A carried panel keeps its distance, so moving a floating window in front of a screen never docks it by accident.
+- Docking disables the output and hides the panel.
+
+## Getting at it: the float key and the title bar button
+
+Decisions 22 to 25.
+
+- **The float key.** The input relay owns it: the action `float_toggle`, bound to Meta+Shift+F unless the rules file says otherwise (a rules file with no `key_bindings` gets that default; one with its own list, even an empty one, doesn't). It can be rebound or removed in Frametop Input Settings, and mapped to a mouse button or a Frame controller button like any other action. The relay takes the combination before it reaches the desktop and sends `float pointer` to ft-floatd, which asks the script for the window under KWin's pointer (`workspace.cursorPos`, top of `workspace.stackingOrder`, popups and dialogs counting as their parent). With none there, the wallpaper or the taskbar, it's the active window. KWin's pointer is where the 3D mouse or a laser last was on a Frametop panel. A window that floats docks; any other floats. The KWin script has no shortcut of its own (ft-floatd removes one that an older script registered), so one press can't float a window and dock it again.
+- **Docking everything.** `dock_all` (no default binding) sends `dock all`, which docks every floating window where it came from.
+- **The title bar button.** Breeze can't take a button of its own, and a C++ fork of it would have to match SteamOS's exact KDecoration build (Plasma 6.3 replaces KDecoration2 with KDecoration3). So the Frametop desktop gets its own window decoration, written in QML for KWin's Aurorae engine, which loads it without compiling (`decoration/`, installed to `~/.local/share/kwin/decorations/kwin4_decoration_qml_frametop`, chosen in the session's `kwinrc` only, so Desktop Mode keeps Breeze; `decoration/apply.sh` switches the running desktop to it or back to Breeze). It's drawn to look like Breeze, with a float button left of Close. The button calls `requestToggleKeepBelow()`, the one window request a decoration can make that has no visible effect here, and the KWin script reads the change: keep-below set on a window on the screens floats it, cleared on a floating window docks it. The script keeps keep-below set on every floating window, however it was floated, so the button shows its dock icon there. A window alone on its own output loses nothing by being kept below (only the wallpaper is under it). If the window can't float (every spare is in use), the script clears the flag again. Keep Below Others in a window's menu does the same as the button.
+- **Apps that draw their own title bar** (Chromium and Electron apps, GTK apps) never show KWin's decoration, so they don't get the button. They use the key, or the window menu (Alt+F3).
+
+## Launching an app floating
+
+- **In the desktop.** Use "Float in VR" in any window's menu, its title bar button, or the float key.
+- **From the menu.** Right-click an app in the Application Launcher, or in the taskbar (where it starts another window of that app), and pick "Launch as Standalone" (decision 26). The launcher has no way to add an entry to every app's menu, but its menu shows each app's own desktop actions. So the Frametop desktop reads copies of the apps' desktop files with one more action added (`float/ft_apps.py`). They're written to `~/.local/share/frametop/apps/applications` from every desktop file in `XDG_DATA_DIRS`: by the session script before Plasma starts, and by ft-floatd whenever an app is installed, changed, or removed. The session puts `~/.local/share/frametop/apps` first in `XDG_DATA_DIRS`. Plasma's app cache is keyed by those directories, so Desktop Mode never sees the copies. Desktop files in `~/.local/share/applications` come before every data dir, so an app you've customized there keeps your copy and has no Launch as Standalone. The action runs `ft-float launch <desktop file name>`.
+- **From a command.** `ft-float run <command>` and `ft-float launch <app.desktop>` start an app and float its first window. ft-floatd records the process it started, and new windows are matched by PID, including child processes. Some single-instance apps (Firefox, D-Bus-activated apps) open the window from a process that was already running. Those are matched by desktop file name. Either way, the window has to show up within 30 seconds.
+- **From the headset with the desktop off.** A profile's launcher entry starts the desktop in that profile, and a profile can hide every screen and hold only floating apps (`docs/profiles.md`). There's no separate Frametop Apps entry or picker.
+- **Remembered placement.** Each app's last floating pose (relative to the primary screen's panel, so it moves with the screens' layout), size in pixels, and scale, keyed by desktop file name, in `~/.config/frametop-float.json`. It's kept whenever one of the app's windows stops floating. With nothing remembered, the window opens in front of you, at the primary screen's density, 0.8 to 2 m away. A profile's own placement wins when the profile opens the app.
+
+## Drag and drop between panels
+
+KWin handles the protocols: Wayland, X11 through Xwayland, and the portal's file transfer. Frametop has to get the pointer right between panels.
+
+- **Crossing panels.** When the laser moves onto another panel mid-drag, ft-screens gives that panel's KWin window pointer focus. KWin puts its cursor at that output's position, and the drop target gets enter and motion events. Floating windows add nothing new here, but they make gaps between panels the normal case.
+- **Gaps (the catcher).** While the laser is between panels, none of Frametop's overlays get its events, and ft-screens clears pointer focus on `FT_LEAVE` even with a button held. A release in empty space would never reach KWin, and the drag or move would stay stuck until the next click. So while a button is held on a Frametop panel and the laser leaves all of them, ft-screens puts an invisible catcher overlay on the laser. A release on the catcher releases in KWin wherever the pointer last was. Dropping in a gap cancels, just as dropping outside any window does. The pointer helper also tells ft-screens when the mouse's left button comes up ("up"), in case the catcher misses it. This also covers window moves and drags on the screens that end off a panel.
+- **The 3D mouse's drag lock.** While the button is held, the drag lock keeps the cursor at its distance and stops hit tests, so a drag onto a nearer panel would pass behind it. So while the button is held, the helper keeps testing the other Frametop panels (not the one pressed on, and not while carrying one) and moves onto a panel the ray meets. Off the edge of the pressed panel, it keeps that panel's plane, so moves and resizes past the edge still work.
+- **Drag icon.** KWin 6 draws the drag icon as part of its scene, on the output its pointer is on. In a gap it stays at the source panel's edge.
+- **Flatpak apps.** Dropping files into a sandboxed app goes through the document portal, the same path that the session script's file-picker fix covers ([design.md](design.md#the-desktop-session)).
+
+## Things that must keep working
+
+- **Typing follows the last click.** A click on a floating panel counts as a click on the desktop, since the window is a KWin window.
+- **Visibility.** Floating windows follow the screens' rules: the hide hotkey, the visibility modes, and hiding during a VR game unless the dashboard is open. Controllers' lasers are off in games.
+- **Headset standby.** Nothing new may poll SteamVR with new clients, so no new `vrcmd` loops.
+- **The pointer helper's overlay list.** The helper learns about overlays by running `vrcmd --overlays` in the background, so a new floating panel appears in its next listing.
+- **Plasma.** An enabled floating output gets a desktop view (wallpaper) under its window, hidden by the crop. Plasma doesn't add panels to new outputs by default. A floating output must never become primary. With spare outputs, the output count stays the same, which avoids the lost-taskbar problem in design.md's open questions.
+- **Restarting the desktop** closes every window, floating ones included. Each app's placement is remembered, so an app launched floating again comes back where it was.
+
+## Risks
+
+- A SteamOS update can change KWin's script API or its nested backend. The script and the output handling are the parts to recheck after one, and KWin's placement memory with them (the script undoes it, see [design.md](design.md#kwins-placement-memory)).
+- GPU memory: each floating output has its own swapchain of two or three buffers, including the margin. The Frame has 16 GB shared, with about 4 GB free in normal use (2026-09-29).
+- Frame pacing with many panels hasn't been measured (already an open question in design.md). Each output is a separate render pass in KWin.

docs/gaze-controllers.md

@@ -0,0 +1,28 @@
+# Gaze with the controllers: a dead end
+
+Gaze mode is a mouse and keyboard feature. On 2026-09-30 we tried to make the Frame controllers its buttons: with gaze mode on and no game running, either controller's trigger would click where you look (a tap clicks, moving the hand steers the pointer, holding still drags), the controllers' lasers would be muted, and SteamVR's dashboard would follow the gaze too. It can't be done cleanly, for the reasons below. The work wasn't merged, and it's kept outside the published history.
+
+## What worked
+
+- Our `ft_pointer` device can hold SteamVR's laser without a hand role, in the treadmill role. It has to hint that role when SteamVR activates it; a hint changed later never gets the `/user/treadmill` path.
+- A trimmed copy of the Frame controller's compositor binding mutes the controllers' laser buttons. It's chosen with `POST /input/selectconfig.action` on vrserver's port 27062 (a JSON body; a form-encoded one gets "Parse failed"). The helper's global action sets can't do it, because SteamVR marks them inactive while its laser mouse has focus.
+- vrserver's web socket on 127.0.0.1:27062 reports every controller component without taking it from anyone.
+- Steam's UI can be kept from acting on the controllers by wrapping its gamepad input source (webpack module 17900) through Steam's CEF debugger.
+- Our device takes the laser back 10 to 15 ms after a controller takes it.
+
+## What broke it
+
+- Steam reads the Frame controllers itself. They aren't devices on the host: vrserver owns their radio and passes their raw reports to Steam through SteamVR's private Steam interface. Steam's client library turns them into a virtual device ("SteamFrameVirtual", at `/steamvr/virtual`), outside every SteamVR binding. Steam's own SteamVR action manifest asks only for haptics.
+- Every press and every release that Steam sees takes SteamVR's dashboard, and Frametop's panels, out of laser mode about 40 ms later. That happens whatever the bindings say and whatever Steam's UI does with the event. None of these stopped it: dropping the events in Steam's UI, removing the controller's `dualanalog` bindings, binding every button to a harmless compositor action, or setting `dashboard.modalGamepadAndLaser` to false.
+- Taking the laser back after each switch leaves a gap of 20 to 40 ms, and panels treat it as the pointer leaving, so clicks and drags break.
+- Steam can't be told to ignore the controllers. It won't save a controller layout for the virtual controller ("Saved Binding Selection Failed - No Identity"), and its menus don't go through the layout anyway: a live preview of the empty layout for Steam's UI (app 769) changed nothing.
+- SteamVR hands the controllers to a VR app instead of Steam only while that app has the input focus, as games do. A dashboard overlay with overlay flag `1 << 4` is given the focus only in gamepad mode, and only for gamepad input.
+
+## Options not taken
+
+- Frametop as a transparent VR app (a scene application using OpenXR's alpha blend mode) whenever gaze mode is on. That would cut Steam off while the dashboard is closed, but Steam's dashboard pages would still take the controllers, and it costs a scene layer all the time.
+- Patching Steam's running process, with an eBPF probe that writes its memory or an injected hook, to drop the controller reports while gaze mode is on. It would cover everything, but it changes Valve's software, needs Steam's client library reverse-engineered, and breaks with Steam updates.
+
+## What was built
+
+The attempt had a plan and a test log, probes for the laser, input focus, and SteamVR settings, a reader for vrserver's web socket, a filter for Steam's UI, and controller code in the relay and the helper. None of it is in this repo. Only the gaze dot setting (`POINTER_GAZE_DOT`) came over.

docs/hazards.md

@@ -16,8 +16,9 @@ The input relay takes the volume keys from every device that has them, so gamesc
 
 ft-screens drops keys while no screen has focus or the SteamVR dashboard is open, but always lets through the release of a key the desktop saw pressed, so a modifier held as the dashboard opens doesn't stay down.
 
-- **A release that never arrives leaves the key held in the desktop.** KWin repeats held keys itself, so a stuck letter repeats and a stuck modifier changes every later key (Ctrl+Alt held turns T into Konsole). Pressing and releasing the key again clears it.
-- **A keyboard that disconnects mid-press is one way to get there.** The relay forgets the held key without telling ft-screens. The same goes for the relay restarting while a key is down.
+- **A key whose release never arrives stays held in the desktop until the relay clears it, within about a second.** KWin repeats held keys itself, so a stuck letter repeats and a stuck modifier changes every later key (Ctrl+Alt held turns T into Konsole). The relay remembers which keys it told the desktop went down, and once a second it releases any that no keyboard holds (`reconcile_desktop_keys`, which asks the kernel with `EVIOCGKEY`). Pressing and releasing the key again also clears it.
+- **A keyboard that disconnects mid-press, or a relay restart with a key down, is how it happens.** The once-a-second check catches the first. A relay that starts doesn't know what an earlier one left down, so it releases the modifiers on the desktop; another key left down that way stays until it's pressed and released again.
+- **To see where a key went,** run `scripts/keys-report.py` and reproduce the problem while it records. It logs the modifiers, Tab, and Esc (no other keys) as the relay reads them and as its virtual keyboard sends them on, with the device roles and grabs, which programs have each keyboard open, and the relay's and desktop's logs.
 - **Switching where typing goes waits for keys to come up.** The relay changes a keyboard's grab only while none of its keys are down, so a press and its release go to the same side. A key held for a long time delays the switch until it's let go.
 
 ## Typing and grabbed keyboards

docs/profiles.md

@@ -0,0 +1,54 @@
+# Profiles
+
+Profiles are built: Display Settings, `ft-layout`, each profile's launcher entry, and the input relay's `profile:NAME` action open them, and the desktop can start in one.
+
+A profile is a named layout that also opens apps. It holds:
+
+- where each screen goes, with its size in metres, curve, roll, and pin (what a named layout held before profiles);
+- which screens show and which are hidden;
+- the apps, one entry per window: on a screen at a place and size, or floating at a pose, size, and scale.
+
+So a "Work" profile can put three screens around you with a browser, two terminals, and an editor on them, and a "Couch" profile can hide every screen and float one video player in front of you.
+
+## Decisions
+
+| # | Question | Decision |
+|---|---|---|
+| 1 | Profiles and named layouts | One list. Named layouts grow into profiles; a layout saved before profiles existed is a profile with no apps and every screen shown |
+| 2 | Making one | Capture what's open: the screens and every app's windows. Display Settings lists a profile's apps, so one can be removed. No editor beyond that |
+| 3 | Apps with several windows | One entry per window. The app is launched once; when its first window shows up, it's launched again for each window still missing. A browser that restores its own windows gets launched once, a terminal twice |
+| 4 | What's recorded of an app | Its desktop file name, or its command line if it has none. Not what it had open: tabs, files, and folders are left to the app's own restore |
+| 5 | Switching while apps are open | Additive: launch what's missing, move the windows that match into place, leave the rest alone. Nothing is ever closed |
+| 6 | Saving changes | Only on an explicit save. Moving things after switching doesn't change the profile |
+| 7 | Starting one | Four ways: a default profile when the desktop starts, Display Settings, a launcher entry for each profile, and a mappable action |
+| 8 | Plasma's session restore | Off in the Frametop session, so a profile is the only thing that reopens apps |
+| 9 | Screen count and resolution | Not part of a profile. They're global, because changing them restarts the desktop, which closes every window |
+
+## Where profiles live
+
+`~/.config/frametop-layout.json` keeps its `layouts` as they are (`{"Work": [screen places]}`), so older copies of ft-layout still read it. What a profile adds goes in a parallel `profiles` map under the same names:
+
+```
+"layouts":  {"Work": [{"pos": ..., "face": ..., "roll": ..., "metres": ..., "curve": ..., "pin": ...}, ...]},
+"profiles": {"Work": {"hidden": [3],
+                      "windows": [
+                        {"app": "org.kde.konsole", "screen": 2, "rect": [40, 60, 1200, 800], "maximized": false},
+                        {"app": "com.brave.Browser", "screen": 1, "maximized": true},
+                        {"cmd": ["/opt/tool/run"], "class": "tool", "screen": 1, "rect": [...]},
+                        {"app": "org.kde.dolphin", "float": {"rel": [12 numbers], "pixels": [1400, 900],
+                                                             "scale": 1.2, "mpp": 0.00097}}]}},
+"default_profile": "Work"
+```
+
+- `screen` is 1-based, as everywhere in Frametop. `rect` is the window's frame in KWin's logical units, relative to its screen's output, so it survives the screens being arranged differently.
+- A floating window's place (`rel`) is its panel's centre and axes in the frame of the primary screen's panel, the same way ft-floatd remembers each app's place. The screens go relative to your head when the profile is applied, and the floating windows follow them. `mpp` is its density in metres per pixel, and `scale` its scale, put back with the rest.
+- Renaming or deleting a layout renames or deletes its profile entry with it.
+
+## How it works
+
+- **Capture** (`ft-layout save NAME`, and Save as profile… in Display Settings). ft-layout captures the screens as before, then asks ft-floatd for the windows (`windows` on @frametop_float). ft-floatd has the KWin script report every window as it is now (`report-all`), then answers with every normal window: its desktop file name, the screen it's on, its rectangle there, and whether it's maximized. For floating windows it gives their panel's place, their size in pixels, and their scale. A window whose app id has no desktop file (a Flatpak app's X11 window can give its own: RustDesk's says `com.carriez.flutter_hbb`, its desktop file is `com.rustdesk.RustDesk`) is kept by the desktop file whose `StartupWMClass` names its window class. Windows with no desktop file name are kept by their process's command line (`/proc/<pid>/cmdline`) and window class. Windows of Plasma itself, the Frametop settings apps, and dialogs aren't recorded. If ft-floatd doesn't answer, the profile keeps the apps it had.
+- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name (found the same way as at capture): single-instance and D-Bus-activated apps open their windows from a process that was already running.
+- **Default at start.** The session script runs `ft-layout start --wait 90`. That opens the profile in `FT_PROFILE` or `default_profile` (screens, then the apps once ft-floatd is up), or runs `apply --wait` if there's none. Start in profile on the Layout & profiles page sets `default_profile` (`ft-layout default NAME|none`). Plasma's session restore is turned off in the session (`ksmserverrc`: `loginMode=emptySession`).
+- **Launcher entries.** Each profile gets `~/.local/share/applications/frametop-profile-<name>.desktop` ("Frametop: Work"), written when it's saved and removed when it's deleted. They show in SteamVR's Launch a program list, the Application Launcher, and KRunner. Running one (`ft-layout open NAME`) switches to that profile if the desktop runs. Otherwise it starts the desktop with `FT_PROFILE` set (`systemd-run`, as `desktops.sh start` does), which overrides `default_profile` for that start. That needs SteamVR to be running.
+- **The action.** `profile:NAME` in the input relay (it runs `ft-layout use NAME`) for key combinations, mouse buttons, and controller buttons, with or without pointer mode. Input Settings lists one "Open profile NAME" action per profile.
+- **Display Settings.** On the Layout & profiles page, the arrangement list has the profiles, which can be renamed and deleted. Open profile and Save as profile… are the page's actions. A profile's apps are listed with where each goes and a button to leave one out, plus which screens it hides. Start in profile picks the one the desktop starts with. The Visibility tab's Screens shown switches hide screens one at a time.

docs/reference.md

@@ -39,30 +39,40 @@ The controls are sized from both the screen's width and its distance from you, f
 
 To pin a screen to a wrist, carry it by its bar and sweep the laser across your other controller. A ring around that controller marks the target, and a dot shows where the laser passes. Crossing the ring arms the pin, and the ring and bar turn blue; crossing it again disarms it. When you let go while armed, the screen rides on that controller at the size, distance, and angle it had, so you can arm the pin first and then turn the screen the way you want. Grab a pinned screen's bar to adjust it; it goes back to the same wrist when you let go unless you disarm it. A pinned screen shows only while you're looking at its front, within the wrist angle, and fades out over the last 10°.
 
-The Visibility & wrist tab of Frametop Display Settings decides when the screens show:
+To pin a screen to your head, like a HUD, set it to On your head on the Visibility & pins tab of Frametop Display Settings (or `ft-layout pin N head`). It rides on the headset where it is at that moment, so place it first, and it shows whenever the screens do. Grab its bar to move it; it goes back on your head where you let go. Sweeping across a wrist ring while you carry it moves it to that wrist, and sweeping across again leaves it in the room. The 3D mouse's dot stays in the room, so a head-pinned screen moves away from it when you turn your head, unless head follow is on.
+
+The Visibility & pins tab of Frametop Display Settings decides when the screens show:
 
 - Always. Meta+Shift+H, the Hide/Show Screens menu entry, or a mapped mouse button hides them.
 - Only while the SteamVR dashboard is open.
 - While you look at a chosen controller (the wrist gesture).
 - Only after you show them with the hotkey.
 
-In the last three modes the hotkey shows the screens anyway. Two more settings on the same tab cover VR games, which ft-screens detects as SteamVR scene apps:
+In the last three modes the hotkey shows the screens anyway. A screen can also be hidden on its own (Screens shown on the same tab, or `ft-layout hide N`): it stays hidden whatever the mode or the hotkey says, until it's shown again there. Windows on it stay put, and a new window that would open on it floats instead (ft-floatd). Profiles use this to show only some screens. Two more settings on the same tab cover VR games, which ft-screens detects as SteamVR scene apps:
 
 - During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up.
 - Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.
 
 Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay, from pass-through keyboards and any key a pointer device passes through. Typing follows your last click: after a click on a screen it goes to the desktop, even with the SteamVR dashboard open, and after a mouse click on any other panel (the dashboard, Steam, an app like Spotify) it goes there instead. While it goes to the desktop, the relay grabs pass-through keyboards so gamescope, which reads every keyboard itself, doesn't type them into the Steam app too. A program that watches every keyboard for a hotkey loses a grabbed one; with `SHARE_KEYS=1` in `~/.config/frametop.conf`, their keys also go to `@frametop_keys` for it. That's off by default, since any local process that binds the name first would get everything typed into the desktop. Hidden screens don't take typing.
 
+Frametop's keyboard opens by itself when a text field on the desktop gets focus, and stays open until its Close key, a layout reset, or a mapped button closes it (or, with Keep it open off in Frametop Input Settings, until the text field loses focus). While the Steam menu (the dashboard) or Steam's own keyboard is up, it steps aside, and it comes back where it was when they're gone; one asked for meanwhile appears then. In the "only with the dashboard" visibility mode, the dashboard doesn't count. It doesn't open without a head pose (the headset in standby). It's a panel of keys (a US laptop layout, with Esc where Caps Lock would be, arrows, and a Close key) that ft-screens shows 0.7 m in front of you and below your eyes, facing you. It stays where it opened, and its grab bar (the pill along the top) moves it like a screen's. Type on it with a controller's laser or the 3D mouse. Shift, Ctrl and Alt latch for the next key, and a held key repeats. KWin starts `input/ft-textinput` as the desktop's input method, and KWin activates it whenever the focused app turns on text input for a field. It tells the relay (`textfield 1` or `0`), the relay decides by the Keyboard setting in Frametop Input Settings, and ft-screens opens the keyboard for the screen that has keyboard focus (`vrkeyboard show`, `hide`, or `toggle` from a mapped button). Its keys reach the focused screen as key presses, so it works in every app, but only apps that use Wayland text input (Qt, GTK, Firefox) open it by themselves; Chromium, Electron and X11 apps need the button. The session drops the `QT_IM_MODULE=xim` and `GTK_IM_MODULE=xim` that the gamescope session sets, or Qt and GTK apps wouldn't use Wayland text input either.
+
+KWin's nested backend doesn't undo a screen's scale on pointer input, so ft-screens divides panel positions (in pixels) by it. `ft-layout` sends it each screen's scale as KWin reports it (`scale N s`) whenever it applies scales: at desktop start and from Frametop Display Settings. A scale changed only in Plasma's own display settings is put back to the Frametop layout's the next time `ft-layout` runs.
+
 ft-screens listens for datagrams on the abstract socket `@ft_screens` and replies to the sender:
 
 ```
 place N x y z yaw pitch roll     width N metres          curve N radius|on|off
-pin N|all left|right [matrix]    unpin N|all             size N w h
-get N    screens    head    state    key code value
+pin N|all left|right|head [matrix]    unpin N|all        size N w h
+get N    screens    head    state    key code value    scale N s    vrkeyboard show|hide|toggle|close
 visibility always|dashboard|gesture|toggle    wrist degrees    gesture left|right degrees
-hide | show | toggle    controllers always|outside_games|dashboard    ingames hide|visible
+hide | show | toggle    controllers always|outside_games|dashboard    ingames hide|visible    pause on|off|state
+conceal N|all    reveal N|all    concealed    cutouts on|off|state    cutouts predict on|off    cutouts lead ms
+float N mpp x y w h title    unfloat N    pose N matrix    sub N k x y w h | sub N k off    minimized N 0|1    carry N
 ```
 
+`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `pause on` (from the input relay, when Frametop pauses for a VR game) hides every screen and floating window whatever else says, and slows the desktop down; `pause off` undoes it. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar.
+
 ## Input relay
 
 SteamVR opens input devices only when it starts. A Bluetooth mouse that sleeps and reconnects gets new device nodes, SteamVR keeps reading the dead ones, and the mouse stops working until SteamVR restarts. `input/input-relay.py` avoids this. It creates two virtual devices, `frametop virtual mouse` and `frametop virtual keyboard`, through `/dev/uinput` before SteamVR starts. It then grabs USB and Bluetooth mice and keyboards as they come and go and forwards their events, so SteamVR only ever sees the virtual devices, which never go away.
@@ -75,6 +85,8 @@ The relay also owns the volume keys, on every device that has them, the headset'
 desktops.sh relay install     # enable it (starts with the next reboot or SteamVR start)
 desktops.sh relay status | log | uninstall
 input/input-relay.py --no-grab   # try it without taking devices from SteamVR
+input/test/keys-test.py          # key combinations and modifier taps, against fake devices (safe next to the live relay)
+steam/ft-steam menu              # what Open Steam menu does; ft-steam check: Steam's UI still has the calls
 ```
 
 The first time, the relay has to start before SteamVR, so reboot or restart SteamVR after installing it. After that it's safe to restart on its own: systemd keeps the virtual devices open in its file descriptor store (`FileDescriptorStorePreserve=yes`), so SteamVR keeps the same devices.
@@ -89,7 +101,7 @@ A mouse drives SteamVR the way a controller's laser does, but shows up as a smal
 
 Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again.
 
-To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button (or a Meta tap, with `META_DASHBOARD=1`) wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant.
+To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant. Open Steam menu / close dashboard (`steam_menu`) needs no pointer: `steam/ft-steam menu` asks Steam's UI, over its debugging port (`steam/steamui.py`), to show its dashboard overlay and focus the Steam frame's menu, or to hide the dashboard if it's up.
 
 ```
 pointer/driver/build.sh && pointer/driver/install.sh install   # then restart SteamVR
@@ -98,17 +110,20 @@ pointer/helper/run.sh status | log | restart
 pointer/driver/install.sh probe     # devices, hand roles, who owns the dashboard pointer
 ```
 
-The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, and `POINTER_GAZE_SHOW`. The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper.
+The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
 
 ## Frametop Input Settings
 
-A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has six pages:
+A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has nine pages:
 
-- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, where a Meta tap toggles the dashboard if `META_DASHBOARD=1` is in `~/.config/frametop.conf`), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
-- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, faster or slower, pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
-- Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through. Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
+- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
+- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, pause or resume Frametop ([Pausing for VR games](#pausing-for-vr-games)), Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
+- Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
+- Game optimization has the pause for VR games: its state with Pause now or Resume, whether VR games pause Frametop by themselves, the controller gesture (one or two buttons, pressed once or twice; one button always takes two presses), what happens to the desktop, and the sound. They're saved as `pause_auto`, `pause_gesture`, `pause_desktop`, and `pause_sound` in `~/.config/frametop-input.json`. See [Pausing for VR games](#pausing-for-vr-games).
+- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
 - Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
-- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), the gaze mode sliders, the gaze service's state (headset, samples per second, whether only one eye is tracked, the calibration, the nudges learned), and Calibrate (opens the gaze probe), Reload calibration, and Forget nudges.
+- Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
+- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
 - Bluetooth lists paired devices and has Apply Bluetooth fixes, which runs `/etc/steamframe/bt-fixups.sh` through `pkexec`. Pair new devices in Steam.
 
 Device rules are saved in `~/.config/frametop-input.json`. `input-settings/install.sh` installs the menu entry. Its launcher hands podman the real `XDG_RUNTIME_DIR` and user bus and gives the app the session's Wayland socket, because the desktop session runs on a private D-Bus and podman fails on it.
@@ -117,40 +132,137 @@ Device rules are saved in `~/.config/frametop-input.json`. `input-settings/insta
 
 When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout.
 
-The desktop's own screen arrangement follows where the screens are around you, whatever their numbers: a screen you see to the left of another is to its left in Plasma too, so the pointer and dragged windows cross straight to it. Screens one above the other stack, and screens pinned to a wrist come last. It's updated at startup, after arranging or saving the layout, and half a second after you let go of a screen you moved. With the headset off there's no head pose to go by, and the arrangement stays as it was.
+The desktop's own screen arrangement follows where the screens are around you, whatever their numbers: a screen you see to the left of another is to its left in Plasma too, so the pointer and dragged windows cross straight to it. Screens one above the other stack, and screens pinned to a wrist or your head come last. It's updated at startup, after arranging or saving the layout, and half a second after you let go of a screen you moved. With the headset off there's no head pose to go by, and the arrangement stays as it was.
 
-Frametop Display Settings has three tabs:
+Frametop Display Settings has four tabs (three with the gamescope backend, which has no Visibility & pins):
 
 - Screens: add and remove screens, and set each one's resolution (presets from 1080p to 4K, ultrawide, super ultrawide, portrait, or custom), its width in VR (0.5 to 6 m), its scale, whether it's curved, and whether it has the taskbar. Resolution, width, and curve apply at once. Adding or removing a screen takes a desktop restart, which the app offers.
-- Layout: a curve around you, with the screens hinged edge to edge like monitors on a desk and each turned to face you, or a flat wall. Both take rows, distance, gap, and height. Save current arrangement keeps the positions and sizes you set by hand instead. A preview shows the layout from above and from the front, and a switch turns auto-arrange at startup on or off.
-- Visibility & wrist: the visibility, game, and controller settings described above, the wrist angle, and buttons to pin all screens to a wrist or unpin them.
+- Layout (the Layout & profiles page): the Arrangement list starts with two presets: Curved around you, with the screens hinged edge to edge like monitors on a desk and each turned to face you, and Flat wall. Both take rows, distance, gap, and height, and Arrange now applies them. Save as profile… saves where the screens are now (positions, sizes, curves, and pins), which ones are hidden, and the open apps and where their windows are, under a name. Profiles are listed in Arrangement after the presets: pick one and Open profile switches to it, and the buttons next to the list rename or delete it. Below it, Apps lists the profile's windows and where they go, and Leave out drops one. A profile saved with fewer screens than you have now leaves the others where they were saved last, or where the preset would put them. Start in profile picks the profile the desktop starts in; with None, a switch turns auto-arrange at startup on or off. A preview shows the layout from above and from the front. See [profiles.md](profiles.md).
+- Visibility & pins: the visibility, game, and controller settings described above, the wrist angle, where each screen is pinned (in the room, a wrist, or your head), and buttons to pin all screens or unpin them.
+- Power: when the displays turn off while the headset isn't used, their state now, Turn displays off now (to try it), and Stay awake while plugged in. See [Displays off and sleep](#displays-off-and-sleep).
 
 `layout/ft-layout` does the arranging. It's a Python script that uses only the standard library and runs on the host:
 
 ```
 layout/ft-layout apply      # arrange every screen
 layout/ft-layout capture    # save the current arrangement and sizes as the layout
+layout/ft-layout save NAME  # ...under a name too, with the open apps and hidden screens (a profile, docs/profiles.md), and use it
+layout/ft-layout use NAME   # switch to a profile: arrange the screens in it and open its apps
+layout/ft-layout open NAME  # a profile's launcher entry: use it, or start the desktop in it
+layout/ft-layout default NAME|none  # the profile the desktop starts with (start --wait runs it at desktop start)
+layout/ft-layout layouts    # list the named layouts (* = in use); rename OLD NEW, delete NAME
+layout/ft-layout pin N|all left|right|head   # pin as they are now; unpin N|all
 layout/ft-layout plan       # print the arrangement as JSON (no VR needed)
 layout/ft-layout scale      # per-screen scale, positions (as the screens are around you), and taskbar screen, to KWin
 layout/ft-layout toggle     # hide or show all screens
+layout/ft-layout hide N|all # hide a screen on its own, whatever the visibility mode; show N|all brings it back, hidden lists them
 display-settings/install.sh # menu entries and the Meta+Shift+R and Meta+Shift+H shortcuts
 ```
 
 The layout is stored relative to your head when it's applied. `/tmp/frametop-layout.log` has the run from the last desktop start.
 
+## Floating windows
+
+A desktop window can float in VR as a panel of its own, away from the screens. Meta+Shift+F floats the window under the pointer (or the active one, over the wallpaper), or puts it back on its screen if it floats. So do Float in VR in every window's menu (Alt+F3; Back to Desktop on a floating one), the button left of Close in its title bar, and a mouse button, controller button, or key combination mapped to Float window in VR in Frametop Input Settings; Put all floating windows back is mappable too. Launch as Standalone, in an app's right-click menu in the Application Launcher or the taskbar, starts the app with its first window floating, where that app last floated or in front of you. [floating-windows.md](floating-windows.md) explains how it works.
+
+`float/ft-floatd` does this. It runs inside the desktop's Plasma session (log: `/tmp/frametop-floatd.log`), loads the KWin script `float/frametop-float.js`, and moves each floating window to a spare KWin output of its own, which ft-screens shows as a panel cropped to the window. The settings are in `~/.config/frametop.conf`: `FLOAT_SLOTS` is how many windows can float at once (8, at most 16; 0 turns floating off; restart the desktop after a change), and `FLOAT_MARGIN` the pixels around each window on its output, so menus have room past its edges (300). Each app's last floating place, size, and scale are kept in `~/.config/frametop-float.json` by desktop file name, relative to the primary screen, so they move with the screens. `FT_FLOAT_DEBUG=1` in ft-floatd's environment logs every event from the KWin script.
+
+With floating on, the session gives the desktop's windows Frametop's own decoration (`decoration/`): Breeze's look plus the float button. Apps that draw their own title bar, like Chromium and Electron apps, don't have the button. `decoration/apply.sh` puts a changed copy into the running desktop, and `decoration/apply.sh --off` goes back to Breeze until the next desktop start.
+
+```
+float/ft-float float [ID|active]      # float a window (default: the active one, as a toggle)
+float/ft-float float pointer          # the float key: the window under the pointer, floated or put back
+float/ft-float dock [ID|active|all]   # put a floating window back (default: the active one), or all of them
+float/ft-float launch org.kde.dolphin # start an app (its desktop file name) floating
+float/ft-float run COMMAND [ARG...]   # the same for a command
+float/ft-float close ID               # close a window
+float/ft-float list                   # the spare outputs and what floats on them
+```
+
+## Displays off and sleep
+
+SteamVR turns the displays off a few seconds after the headset's proximity sensor says it came off. A stand or display mount that covers the sensor makes the headset seem worn, so its displays stay on, and Steam, which then counts someone as present, never puts it to sleep either.
+
+`power/ft-powerd` goes by use instead. It runs in the `dev` container as `frametop-power.service` and starts with SteamVR. Once the headset has gone unused for `DISPLAY_OFF_MIN` minutes (0, the default, is never), it turns the displays' backlight off, and it turns it back on at the next use. Use is any of these:
+
+- The headset, a Frame controller, or the 3D mouse's virtual controller moving more than `DISPLAY_MOVE_MM` (5 mm) or turning more than `DISPLAY_MOVE_DEG` (0.5 degrees) within 10 seconds.
+- A key, button, or mouse motion on any input device on the host, including the headset's own buttons and the input relay's virtual mouse and keyboard.
+- The headset going back on after SteamVR's own standby, or something else turning the backlight back on.
+
+While SteamVR has the headset in standby, SteamVR owns the displays and ft-powerd waits. The backlight is `/sys/class/backlight/ae94000.dsi.0/brightness`, the same file SteamVR's driver writes for standby. With the backlight off, tracking and rendering keep running, which lets the displays wake the moment the headset moves, but the headset still uses most of its power. ft-powerd puts the backlight back when it stops, and if it was killed with the displays off, the next start does (the value is kept in `~/.cache/frametop/powerd-brightness` meanwhile).
+
+Stay awake while plugged in is Steam's own setting, When Plugged In and Idle → Sleep after (`system_idle_suspend_ac_sec`), set to Never. Frametop Display Settings changes it the way Steam's Settings → Power page does, through Steam's UI on its debugging port (`display-settings/steam_settings.py`), and keeps the value from before in `STEAM_SLEEP_AC_BEFORE` to put back when the switch goes off. The power button still puts the Frame to sleep, and Steam's battery setting still applies.
+
+```
+power/build.sh && power/run.sh install
+power/run.sh status        # "ok on|off|away <seconds unused> <timeout seconds>"
+power/run.sh off | on      # the displays off now, or back on
+power/run.sh log
+```
+
+## Pausing for VR games
+
+Paused, Frametop leaves the headset's CPU and GPU to a VR game. The input relay does it (`input/game_pause.py`), since it's the one part that always runs:
+
+- The gaze service stops (`frametop-gaze`: ft-gazed, ft-gaze, our own eye tracker, the gaze panel), so nothing reads SteamVR's eye tracking. Our frame grabber, the root service `ft-eyegrab`, goes idle by itself 3 seconds after our eye tracker stops asking it for frames.
+- Hand tracking stops if it runs (`frametop-camd`, `frametop-hands`).
+- The desktop, as the Game optimization page of Frametop Input Settings says (`pause_desktop`): hidden (the default) or closed. Hidden, ft-screens hides every screen and floating window whatever the visibility mode, the hotkey, or the dashboard says, and gives KWin a frame callback once a second instead of 90 times. KWin draws a screen only after its frame callback, and its apps wait for theirs, so the desktop hardly draws, but its windows stay open. Remote desktop stops if it runs (`session/remote-ctl.sh`). Closed, `desktops.sh stop` closes the desktop and its windows, and resuming starts it again (about 12 seconds), in its start profile if it has one.
+- The relay lets go of the 3D mouse and feeds pointer devices to its virtual mouse and keyboard, as with `POINTER=0`. Typing goes to Steam. Mapped buttons and key combinations do nothing but pausing, the Steam menu, and commands; a key combination that does nothing is typed as usual.
+
+Resuming starts again only what pausing stopped, and plays a second sound. The pointer helper and ft-powerd keep running: they cost little, the helper is what says a game started, and stopping it would leave its virtual controller connected with its last pose.
+
+Ways to pause and resume:
+
+- The controller gesture, by default both thumbsticks clicked together twice: both go down within 0.3 seconds of each other, and the second time within 0.7 seconds of the first. The relay reads it from vrserver's web socket (`input/vrws.py`), which works whatever has input focus and takes nothing from the game, so the game sees the clicks too. The Game optimization page changes it: one or two of the buttons the Controllers page lists, pressed once or twice (one button always takes two), or none.
+- The Pause/resume Frametop action, on a mouse button, a key combination, or a controller button (outside games, like every mapped controller button).
+- VR games, with Pause while a VR game runs on (`pause_auto`, the default). The pointer helper tells the relay when a scene app starts and ends (`vrgame 1|0`, repeated every 5 seconds). A game starting pauses Frametop. A pause that starts while a game runs ends 5 seconds after the game does, unless another game starts first. Resumed during a game, Frametop stays on until that game ends. A pause that starts outside a game lasts until you resume. Flatscreen games aren't scene apps, so they don't pause it.
+- From a terminal or a script:
+
+```
+input/ft-pause on | off | toggle   # pause or resume
+input/ft-pause status              # the state as JSON (the relay's "pause ?")
+input/vrws.py 10                   # the controllers' buttons from vrserver's web socket, for 10 s
+input/test/pause-test.py           # the gesture and the automatic pause, offline
+```
+
+The state outlives a relay restart, in `/run/user/UID/frametop-pause.json`. A SteamVR restart while paused starts the gaze service with it, and the relay stops it again when the pointer helper comes back.
+
+## Gaze pointer (experimental)
+
+In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. The service idles while the gaze isn't used: its eye tracker reader and our own eye tracker run only while gaze mode is on and someone wears the headset, while a check or the calibration runs, or while the Gaze page of Frametop Input Settings is open, and stop 30 seconds after ([gaze/README.md](../gaze/README.md)). Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off.
+
+- Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held.
+- The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual.
+- A correction before a click teaches the gaze service the tracker's error there. A correction bigger than `POINTER_GAZE_NUDGE_MAX` (55 degrees) isn't learned; it opens a quick check instead.
+- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it.
+
+[gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool.
+
+## Hand tracking (experimental, deferred)
+
+Deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it. It still builds and runs, installed by hand, for working on it.
+
+Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode a pinch clicks where you look when it opens; hold it and move the hand to correct the pointer first. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`.
+
+- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`.
+- `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring.
+- The install leaves both off, and they don't start with SteamVR. `ft-handsctl on` starts them while SteamVR runs, and `ft-handsctl off` stops them; they also stop with SteamVR. The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live.
+- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (after some SteamVR restarts the side cameras' names come out swapped, and hands land beside the holes; `hands/tools/check_sides.py --ring` tells), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, and `POINTER_PINCH_TYPING`. The example config explains each.
+
+Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md).
+
 ## Remote desktop over VNC
 
-With `REMOTE=1` in the config (`desktops.sh remote on`), the desktop is also served over VNC, for RealVNC Viewer or macOS Screen Sharing. `desktops.sh remote info` prints the address and password.
+With `REMOTE=1` in the config (`desktops.sh remote on`), the desktop's primary screen (the one with the taskbar) is also served over VNC, at that screen's resolution, for RealVNC Viewer or macOS Screen Sharing. `desktops.sh remote info` prints the address and password.
 
 It listens on port 5900 on the Frame's Tailscale address only, not the LAN, so it needs Tailscale on the Frame ([deck-tailscale](https://github.com/tailscale-dev/deck-tailscale)). VNC authentication has no encryption of its own, so viewers warn about it, but the tailnet encrypts the traffic. The password is in `~/.config/frametop-remote/vnc-password` and VNC limits it to 8 characters. To change it, delete that folder and restart the desktop.
 
-No VNC server can capture KWin on SteamOS directly: `krfb` needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship, and `wayvnc` only works with wlroots compositors. So `session/remote-desktop.sh` captures the desktop with KDE's `krdpserver --plasma` on `127.0.0.1:3390`, and `session/vnc-bridge.sh` runs TigerVNC's `Xvnc` on display `:20` with a full-screen FreeRDP client inside it and serves that. Both run in the `dev` container, and the extra hop adds a little latency.
+No VNC server can capture KWin on SteamOS directly: `krfb` needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship, and `wayvnc` only works with wlroots compositors. So `session/remote-desktop.sh` captures the desktop with KDE's `krdpserver --plasma` on `127.0.0.1:3390`, and `session/vnc-bridge.sh` runs TigerVNC's `Xvnc` on display `:20` with a FreeRDP client inside it and serves that. Both run in the `dev` container, and the extra hop adds a little latency. krdp streams every screen; the VNC screen is the primary's size, and the FreeRDP window is shifted so the primary fills it (`ft-layout remote-view` gives the offset). krdp's own `--monitor` would stream just one screen, but it maps the pointer as if that screen sat at 0,0, so clicks would miss. When the layout changes, the VNC screen resizes and FreeRDP reconnects within a few seconds.
 
-With remote access on, the nested KWin runs with `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1`, so any app in the Frametop desktop could capture its screen or inject input. This applies only to that desktop, not the stock one. Port 3389 is SteamOS's own `xrdp`, which starts a separate X11 session rather than showing the VR desktop.
+With remote access on, the nested KWin runs with `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1` and `KWIN_SCREENSHOT_NO_PERMISSION_CHECKS=1`, so any app in the Frametop desktop could capture its screens or inject input. The second one lets scripts take screenshots through KWin's `org.kde.KWin.ScreenShot2` D-Bus interface. This applies only to that desktop, not the stock one. Port 3389 is SteamOS's own `xrdp`, which starts a separate X11 session rather than showing the VR desktop.
 
 ## Limits
 
-- There's no way yet to pin a screen to your head like a HUD.
 - A controller button can't show hidden screens; a mapped mouse or keyboard button can.
 - KWin's cursor isn't drawn on the screens, because KWin draws it as a host cursor, which ft-screens doesn't render. The 3D mouse's dot and SteamVR's laser dot show where you're pointing.
 - The old gamescope backend (`BACKEND=gamescope`) still works, but it gives every screen the same resolution, at most 1920×1080 pixels' worth, and arranging screens borrows the pointer for a few seconds.

float/frametop-float.js

@@ -0,0 +1,420 @@
+// frametop-float: the KWin side of floating windows (see docs/floating-windows.md). ft-floatd
+// loads it into the desktop's KWin over D-Bus (org.kde.kwin.Scripting) and talks to it:
+//   - events go to ft-floatd as JSON strings (org.frametop.Float.Event), for the windows it
+//     cares about: floating windows (the ones on a spare output, WL-<screens> and up), their
+//     popups and dialogs, new windows, and requests to float or dock one;
+//   - commands come back through a long poll: the script calls NextCommand, ft-floatd
+//     answers when it has one (or after a while with nothing), and the script calls again.
+// KWin scripts can call D-Bus but can't serve it, hence the poll. Window ids are KWin's
+// internalId (a UUID string).
+
+const SERVICE = "org.frametop.Float", PATH = "/Float", IFACE = "org.frametop.Float";
+let screens = 0;       // outputs WL-0 .. WL-<screens - 1> are screens; the rest are spares
+let polling = false;
+const watched = {};    // id -> true once its signals are connected
+let marking = false;   // the script itself is setting keep-below (see mark)
+const settled = {};    // id -> {output, frame, fullScreen, maximized}: where a window belongs (see putBack)
+const held = {};       // id -> {w, h, until, asked}: a size asked for, for a second (see hold)
+const moved = {};      // id -> true, or "back" once put back: it moved while KWin changed the outputs
+let layout = "", layoutOutputs = {}, layoutSince = 0;  // the outputs at the last screensChanged
+
+function send(ev) {
+    callDBus(SERVICE, PATH, IFACE, "Event", JSON.stringify(ev));
+}
+
+function outputIndex(o) {
+    const m = o ? /^WL-(\d+)$/.exec(o.name) : null;
+    return m ? parseInt(m[1]) : -1;
+}
+function isSpare(o) {
+    return screens > 0 && outputIndex(o) >= screens;
+}
+function rect(g) {
+    return {x: g.x, y: g.y, w: g.width, h: g.height};
+}
+function byId(id) {
+    const all = workspace.windowList();
+    for (let i = 0; i < all.length; ++i)
+        if (String(all[i].internalId) === id) return all[i];
+    return null;
+}
+function outputByName(name) {
+    const all = workspace.screens;
+    for (let i = 0; i < all.length; ++i)
+        if (all[i].name === name) return all[i];
+    return null;
+}
+function info(w) {
+    const o = w.output;
+    return {
+        id: String(w.internalId), pid: w.pid, cls: String(w.resourceClass), app: String(w.desktopFileName),
+        caption: String(w.caption), output: o ? o.name : "", outputRect: o ? rect(o.geometry) : null,
+        frame: rect(w.frameGeometry), client: rect(w.clientGeometry), popup: w.popupWindow,
+        transient: w.transient, parent: w.transientFor ? String(w.transientFor.internalId) : "",
+        normal: w.normalWindow, dialog: w.dialog, fullScreen: w.fullScreen, minimized: w.minimized,
+        onAllDesktops: w.onAllDesktops, maximized: isMaximized(w)
+    };
+}
+
+// KWin 6.2's scripts have no maximize mode to read: a window is maximized when it fills its
+// output's maximize area.
+function isMaximized(w) {
+    if (!w.normalWindow || !w.output) return false;
+    const a = workspace.clientArea(KWin.MaximizeArea, w), g = w.frameGeometry;
+    return g.x === a.x && g.y === a.y && g.width === a.width && g.height === a.height;
+}
+
+function report(type, w) {
+    if (w.deleted) return;  // a window on its way out still changes output and size
+    const ev = info(w);
+    ev.ev = type;
+    send(ev);
+}
+
+// KWin's placement memory (its PlacementTracker) keeps each window's geometry for each layout of
+// the outputs (every enabled output's name and geometry), and when the outputs come back to a
+// layout it has seen, it puts the windows back where they were in it. That's for plugging monitors
+// in and out, and it does harm here. A spare output changes size after its window does, so what
+// KWin keeps for a spare's size is the window's next size: resizing a floating window back to a
+// size it had set off an endless flip between two sizes. And floating or docking one window could
+// move others, even onto a spare or off one. So the script keeps where each window belongs
+// (settled), tells ft-floatd nothing while KWin changes the outputs, and once KWin is done
+// (screensChanged comes after its restore) puts the floating windows back, and the screens' windows
+// too when only spares changed.
+function outputsNow() {
+    const all = workspace.screens, out = {};
+    for (let i = 0; i < all.length; ++i) {
+        const g = all[i].geometry;
+        out[all[i].name] = g.x + "," + g.y + " " + g.width + "x" + g.height;
+    }
+    return out;
+}
+function keyOf(outputs) {
+    return Object.keys(outputs).sort().map(n => n + "=" + outputs[n]).join(" ");
+}
+function takeLayout() {
+    layoutOutputs = outputsNow();
+    layout = keyOf(layoutOutputs);
+    layoutSince = 0;
+}
+// KWin is changing the outputs: they differ from the last screensChanged.
+function changingOutputs() {
+    if (keyOf(outputsNow()) === layout) {
+        layoutSince = 0;
+        return false;
+    }
+    if (!layoutSince) {
+        layoutSince = Date.now();
+    } else if (Date.now() - layoutSince > 2000) {
+        takeLayout();  // screensChanged should have come by now: don't stay quiet for good
+        return false;
+    }
+    return true;
+}
+function settle(w) {
+    if (w.output) {
+        settled[String(w.internalId)] = {output: w.output.name, frame: rect(w.frameGeometry), fullScreen: w.fullScreen};
+    }
+}
+// ft-floatd put the window here: it's where it belongs now.
+function expect(w, output, c, fullScreen) {
+    settled[String(w.internalId)] = {output: output, frame: {x: c.x, y: c.y, w: c.w, h: c.h}, fullScreen: fullScreen};
+    hold(w, c.w, c.h);
+}
+
+// A size asked for (by ft-floatd, or by the script putting a window back) comes in when the app
+// answers, and until then the app can still answer older requests: one KWin's restore made, or,
+// just after it opened, its own. For a second, the script asks again instead of taking those;
+// then it takes the size the window has (an app can refuse a size, below its minimum).
+const holdTimer = new QTimer();
+holdTimer.singleShot = true;
+holdTimer.timeout.connect(() => {
+    const now = Date.now();
+    Object.keys(held).forEach(id => {
+        const h = held[id];
+        if (h.until > now) return;
+        delete held[id];
+        const w = byId(id);
+        if (!h.asked || !w || w.deleted) return;  // (nothing held back: nothing to tell)
+        settle(w);
+        if (isSpare(w.output)) report("geometry", w);
+    });
+    if (Object.keys(held).length) holdTimer.start();
+});
+function hold(w, width, height) {
+    held[String(w.internalId)] = {w: width, h: height, until: Date.now() + 1000};
+    holdTimer.interval = 1100;
+    holdTimer.start();
+}
+// A size change while a size is held: true when it isn't that size (the script asked again).
+// (ft-floatd's sizes can be fractional, the window's are whole: within a pixel is the same.)
+function holding(w) {
+    const id = String(w.internalId), h = held[id], s = settled[id];
+    if (!h) return false;
+    const g = w.frameGeometry;
+    if (!s || h.until < Date.now() || w.move || w.resize || (Math.abs(g.width - h.w) < 1 && Math.abs(g.height - h.h) < 1)) {
+        delete held[id];
+        return false;
+    }
+    w.frameGeometry = {x: s.frame.x, y: s.frame.y, width: h.w, height: h.h};
+    h.asked = true;
+    return true;
+}
+
+// Runs while KWin's output change still counts as going on (nothing reported), so the steps on
+// the way don't reach ft-floatd: told only where the window ends up (see reportMoved).
+function putBack(w, screensChanged) {
+    const id = String(w.internalId), s = settled[id];
+    if (w.deleted || !s || screens === 0 || !marked(w)) return;
+    const o = outputByName(s.output);
+    // KWin's restore sets full screen (and maximized) as it was in that layout too: with a
+    // floating window that flipped forever, its output changing size with it. Ask for the
+    // state it had: KWin's request hasn't reached the app yet, so it never sees it.
+    w.fullScreen = s.fullScreen;
+    if (o && isSpare(o) && !s.fullScreen) w.setMaximize(false, false);
+    // Not where KWin had to move it: its output went, a screen changed, or it's full screen or
+    // maximized (KWin fits those to their output).
+    const back = o && !s.fullScreen && !w.fullScreen && !w.move && !w.resize && !s.maximized
+        && (isSpare(o) || (!screensChanged && !isMaximized(w)));
+    if (!back) return;
+    if (!w.output || w.output.name !== s.output) workspace.sendClientToScreen(w, o);
+    w.frameGeometry = {x: s.frame.x, y: s.frame.y, width: s.frame.w, height: s.frame.h};
+    if (isSpare(o)) hold(w, s.frame.w, s.frame.h);
+    // Moved during the change (by KWin, by the lines above, or the size ft-floatd asked for came
+    // in): report where it is, and keep settled as it is.
+    if (moved[id]) moved[id] = "back";
+}
+// After an output change: tell ft-floatd where the windows that moved during it are now.
+function reportMoved(w) {
+    const id = String(w.internalId), s = settled[id], how = moved[id];
+    if (!how) return;
+    delete moved[id];
+    if (w.deleted) return;
+    if (how !== "back") settle(w);
+    if (!w.output || !s || w.output.name !== s.output) {
+        report("output", w);
+        mark(w);
+    } else if (isSpare(w.output)) {
+        report("geometry", w);
+    }
+}
+workspace.screensChanged.connect(() => {
+    const before = layoutOutputs, now = outputsNow();
+    if (keyOf(now) === layout) return;
+    let screensChanged = screens === 0;
+    Object.keys(Object.assign({}, before, now)).forEach(name => {
+        const m = /^WL-(\d+)$/.exec(name);
+        if (before[name] !== now[name] && !(m && parseInt(m[1]) >= screens)) screensChanged = true;
+    });
+    const all = workspace.windowList();
+    all.forEach(w => putBack(w, screensChanged));
+    takeLayout();
+    all.forEach(reportMoved);
+});
+
+// Floating windows, and popups and dialogs on a spare output: tell ft-floatd about changes.
+function watch(w) {
+    const id = String(w.internalId);
+    if (watched[id]) return;
+    watched[id] = true;
+    const onSpare = () => isSpare(w.output);
+    w.frameGeometryChanged.connect(() => {
+        if (changingOutputs()) {
+            moved[id] = true;
+            return;
+        }
+        if (holding(w)) return;
+        settle(w);
+        if (onSpare()) report("geometry", w);
+    });
+    w.outputChanged.connect(() => {
+        if (changingOutputs()) {
+            moved[id] = true;
+            return;
+        }
+        if (!held[id]) settle(w);  // (held: the place asked for is settled already)
+        report("output", w);
+        mark(w);
+    });
+    w.keepBelowChanged.connect(() => keepBelowChanged(w));
+    w.interactiveMoveResizeStarted.connect(() => {
+        if (onSpare()) send({ev: "move-start", id: id, move: w.move, resize: w.resize, frame: rect(w.frameGeometry)});
+    });
+    w.interactiveMoveResizeFinished.connect(() => { if (onSpare()) report("move-end", w); });
+    w.fullScreenChanged.connect(() => {
+        if (changingOutputs()) {
+            moved[id] = true;
+            return;
+        }
+        if (settled[id]) settled[id].fullScreen = w.fullScreen;
+        if (onSpare()) report("fullscreen", w);
+    });
+    w.minimizedChanged.connect(() => { if (onSpare()) report("minimized", w); });
+    w.maximizedChanged.connect(() => {
+        // A floating window stays an ordinary window: its output is its size plus a margin.
+        if (onSpare() && w.normalWindow && !w.fullScreen) w.setMaximize(false, false);
+    });
+}
+
+workspace.windowAdded.connect(w => {
+    watch(w);
+    settle(w);
+    report("added", w);
+});
+workspace.windowRemoved.connect(w => {
+    const id = String(w.internalId);
+    send({ev: "removed", id: id});
+    delete watched[id];
+    delete settled[id];
+    delete held[id];
+    delete moved[id];
+});
+workspace.windowActivated.connect(w => {
+    if (w && isSpare(w.output)) send({ev: "activated", id: String(w.internalId)});
+});
+takeLayout();
+workspace.windowList().forEach(w => {
+    watch(w);
+    settle(w);
+});
+
+// Keep-below means "floating" in the Frametop desktop. The title bar's float button (Frametop's
+// window decoration, decoration/) is the Keep Below button, so setting the flag on a window on
+// the screens floats it, and clearing it on a floating one docks it. The script keeps the flag
+// set on every floating window, its dialogs included, and cleared everywhere else, however the
+// window got there. Kept below, a window alone on its own output only has the wallpaper under it.
+function marked(w) {
+    return w.managed && !w.deleted && !w.specialWindow && !w.popupWindow;
+}
+function topOf(w) {
+    let top = w;
+    for (let n = 0; top.transientFor && n < 10; ++n) top = top.transientFor;
+    return top;
+}
+function mark(w) {
+    if (screens === 0 || !marked(w)) return;
+    const want = isSpare(w.output);
+    if (w.keepBelow === want) return;
+    marking = true;
+    w.keepBelow = want;
+    marking = false;
+}
+function keepBelowChanged(w) {
+    if (marking || screens === 0 || !marked(w)) return;
+    const top = topOf(w);
+    if (w.keepBelow !== isSpare(top.output)) requestFloat(top);
+}
+
+function requestFloat(w) {
+    if (!w || !w.normalWindow || w.popupWindow) return;
+    report(isSpare(w.output) ? "dock-request" : "float-request", w);
+}
+
+registerUserActionsMenu(w => {
+    if (!w.normalWindow || w.popupWindow) return null;
+    const floating = isSpare(w.output);
+    return {
+        text: floating ? "Back to Desktop" : "Float in VR",
+        icon: floating ? "window-restore" : "window-new",
+        triggered: () => requestFloat(w)
+    };
+});
+// The float key is the input relay's (float_toggle, Meta+Shift+F by default): it reaches us as
+// "request-pointer". No shortcut of KWin's own, so one press can't float a window and dock it again.
+
+// The window under KWin's pointer (where the 3D mouse or a laser last was on a panel): the top
+// one there, a popup or dialog standing for the window it belongs to. Null over the wallpaper
+// or the taskbar.
+function underPointer() {
+    const p = workspace.cursorPos;
+    const order = workspace.stackingOrder;
+    for (let i = order.length - 1; i >= 0; --i) {
+        const w = order[i];
+        if (w.deleted || w.minimized || w.hidden || !w.managed) continue;
+        const g = w.frameGeometry;
+        if (p.x < g.x || p.y < g.y || p.x >= g.x + g.width || p.y >= g.y + g.height) continue;
+        const top = topOf(w);
+        return top.normalWindow && !top.popupWindow ? top : null;
+    }
+    return null;
+}
+
+function run(c) {
+    const w = c.id ? byId(c.id) : null;
+    switch (c.cmd) {
+        case "config":
+            screens = c.screens;
+            workspace.windowList().forEach(w => { report("window", w); mark(w); });
+            break;
+        case "mark":  // after a float that didn't happen: keep-below back as it was
+            if (w) mark(w);
+            break;
+        case "place": {  // onto an output, at a frame rectangle (logical, global)
+            if (!w) break;
+            const o = outputByName(c.output);
+            if (!o) break;
+            if (w.fullScreen && !c.keepFullScreen) w.fullScreen = false;
+            w.setMaximize(false, false);
+            expect(w, o.name, c, w.fullScreen && !!c.keepFullScreen);
+            workspace.sendClientToScreen(w, o);
+            w.frameGeometry = {x: c.x, y: c.y, width: c.w, height: c.h};
+            if (c.onAllDesktops !== undefined) w.onAllDesktops = c.onAllDesktops;
+            if (c.maximized) {
+                // Maximized: KWin picks the size, and the place above is only where it goes.
+                delete held[c.id];
+                settled[c.id].maximized = true;
+                w.setMaximize(true, true);
+            }
+            break;
+        }
+        case "geometry":
+            if (!w) break;
+            expect(w, settled[c.id] ? settled[c.id].output : (w.output ? w.output.name : ""), c, w.fullScreen);
+            w.frameGeometry = {x: c.x, y: c.y, width: c.w, height: c.h};
+            break;
+        case "close":
+            if (w) w.closeWindow();
+            break;
+        case "activate":
+            if (w) workspace.activeWindow = w;
+            break;
+        case "minimize":
+            if (w) w.minimized = c.on;
+            break;
+        case "info":
+            if (w) report("window", w);
+            break;
+        case "report-all":  // a profile's capture: every window as it is now, then a marker
+            workspace.windowList().forEach(w => report("window", w));
+            send({ev: "reported", token: c.token});
+            break;
+        case "request-float":  // ft-float float ID: float it, if it isn't floating
+            if (w && !isSpare(w.output)) requestFloat(w);
+            break;
+        case "request-active":  // ft-float float|dock active
+            requestFloat(workspace.activeWindow);
+            break;
+        case "request-pointer":  // the float key: the window under the pointer, else the active one
+            requestFloat(underPointer() || workspace.activeWindow);
+            break;
+    }
+}
+
+function poll() {
+    if (polling) return;
+    polling = true;
+    callDBus(SERVICE, PATH, IFACE, "NextCommand", reply => {
+        polling = false;
+        if (reply) {
+            try {
+                JSON.parse(reply).forEach(run);
+            } catch (e) {
+                print("frametop-float: bad command " + reply + ": " + e);
+            }
+        }
+        poll();
+    });
+}
+
+send({ev: "hello"});
+poll();

float/ft-float

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+"""ft-float: talk to ft-floatd (floating windows in the Frametop desktop).
+
+  ft-float float [ID|active]   float a window (default: the active one; for it, a toggle)
+  ft-float float pointer       the float key: float the window under the pointer (else the
+                               active one), or put it back if it floats
+  ft-float dock [ID|active]    put a floating window back on the desktop
+  ft-float dock all            put every floating window back
+  ft-float launch APP          start an app (its desktop file name, e.g. org.kde.dolphin) and
+                               float its first window where that app last floated
+  ft-float run COMMAND [ARG...]  the same for a command
+  ft-float close ID            close a window
+  ft-float list                the spare outputs and what floats on them
+FT_FLOAT_SOCKET names ft-floatd's socket (default frametop_float).
+"""
+import json
+import os
+import socket
+import sys
+
+if len(sys.argv) < 2 or sys.argv[1] in ("-h", "--help"):
+    sys.exit(__doc__)
+s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+s.bind("")
+s.settimeout(5)
+try:
+    text = "run " + json.dumps(sys.argv[2:]) if sys.argv[1] == "run" else " ".join(sys.argv[1:])
+    s.sendto(text.encode(), "\0" + os.environ.get("FT_FLOAT_SOCKET", "frametop_float"))
+    reply = s.recv(8192).decode()
+except OSError as e:
+    sys.exit(f"ft-floatd didn't answer ({e}); is the Frametop desktop running?")
+print(reply)
+sys.exit(0 if reply.startswith("ok") else 1)

float/ft-float-apps

@@ -0,0 +1,3 @@
+#!/bin/sh
+# Launch as Standalone: write the apps' desktop file copies (see ft_apps.py).
+exec python3 "$(dirname "$(readlink -f "$0")")/ft_apps.py" "$@"

float/ft-floatd

@@ -0,0 +1,3 @@
+#!/bin/bash
+# ft-floatd on the Frame host, inside the Frametop desktop's session (see ft_floatd.py).
+exec python3 "$(dirname "$(readlink -f "$0")")/ft_floatd.py" "$@"

float/ft_apps.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+"""Launch as Standalone: an action on every app in the Frametop desktop's menus.
+
+Plasma's Application Launcher has no way to add an entry to every app's right-click menu,
+but its menu (and the taskbar's) shows each app's own desktop actions. So the Frametop
+desktop reads copies of the apps' desktop files with one more action, "Launch as
+Standalone", which runs `ft-float launch <desktop file name>`: the app starts and its first
+window floats in VR (docs/floating-windows.md, decision 26).
+
+The copies go in OUT (~/.local/share/frametop/apps/applications), and the session puts
+OUT_ROOT first in XDG_DATA_DIRS, so a copy wins over the app's own file. Plasma's app cache
+is keyed by those directories, so Desktop Mode never sees the copies. Files in
+XDG_DATA_HOME/applications come before every data dir, so an app customized there keeps
+its own file and has no Launch as Standalone. A copy drops DBusActivatable: a D-Bus
+activated app would be asked to run the action itself, and it doesn't know ours.
+
+  ft-float-apps        write the copies (the session script runs it before Plasma starts;
+                       ft-floatd again whenever an app's desktop file changes)
+"""
+import os
+import sys
+
+from gi.repository import GLib
+
+ACTION = "frametop-standalone"
+OUT_ROOT = os.path.expanduser("~/.local/share/frametop/apps")
+OUT = os.path.join(OUT_ROOT, "applications")
+FT_FLOAT = os.path.join(os.path.dirname(os.path.realpath(__file__)), "ft-float")
+GROUP = "Desktop Entry"
+
+
+def data_dirs():
+    """(XDG_DATA_HOME, the XDG_DATA_DIRS other than ours)."""
+    home = os.environ.get("XDG_DATA_HOME") or os.path.expanduser("~/.local/share")
+    ours = os.path.realpath(OUT_ROOT)
+    dirs = [d for d in os.environ.get("XDG_DATA_DIRS", "/usr/local/share:/usr/share").split(":")
+            if d and os.path.realpath(d) != ours]
+    return home, dirs
+
+
+def app_dirs():
+    """Every applications folder that holds the apps' own desktop files, home first."""
+    home, dirs = data_dirs()
+    return [os.path.join(d, "applications") for d in [home] + dirs]
+
+
+def scan():
+    """Desktop file name -> (its path, whether it's in XDG_DATA_HOME), the first one found
+    winning, as the desktop spec has it."""
+    found = {}
+    home_apps = app_dirs()[0]
+    for root in app_dirs():
+        for dirpath, _dirs, files in os.walk(root, followlinks=True):
+            for name in files:
+                if name.endswith(".desktop"):
+                    path = os.path.join(dirpath, name)
+                    found.setdefault(os.path.relpath(path, root).replace("/", "-"), (path, root == home_apps))
+    return found
+
+
+def standalone(path, desktop_id):
+    """The desktop file with Launch as Standalone added, or None for one that isn't a
+    visible app."""
+    kf = GLib.KeyFile()
+    try:
+        kf.load_from_file(path, GLib.KeyFileFlags.KEEP_TRANSLATIONS | GLib.KeyFileFlags.KEEP_COMMENTS)
+    except GLib.Error:
+        return None
+
+    def get(key):
+        try:
+            return kf.get_string(GROUP, key)
+        except GLib.Error:
+            return None
+    if get("Type") != "Application" or not get("Exec"):
+        return None
+    if (get("NoDisplay") or "").lower() == "true" or (get("Hidden") or "").lower() == "true":
+        return None
+    actions = [a for a in (get("Actions") or "").split(";") if a and a != ACTION]
+    kf.set_string(GROUP, "Actions", ";".join(actions + [ACTION]) + ";")
+    try:
+        kf.remove_key(GROUP, "DBusActivatable")
+    except GLib.Error:
+        pass
+    group = "Desktop Action " + ACTION
+    kf.set_string(group, "Name", "Launch as Standalone")
+    kf.set_string(group, "Icon", "window-new")
+    exe = FT_FLOAT if " " not in FT_FLOAT else '"' + FT_FLOAT + '"'
+    kf.set_string(group, "Exec", f"{exe} launch {desktop_id}")
+    return kf.to_data()[0]
+
+
+def write_all():
+    """Write the copies, and remove the ones whose app is gone. Returns how many there are."""
+    os.makedirs(OUT, exist_ok=True)
+    keep = set()
+    for desktop_id, (path, in_home) in sorted(scan().items()):
+        if in_home:
+            continue  # XDG_DATA_HOME's own file wins over a copy anyway
+        data = standalone(path, desktop_id)
+        if data is None:
+            continue
+        keep.add(desktop_id)
+        out = os.path.join(OUT, desktop_id)
+        try:
+            with open(out) as f:
+                if f.read() == data:
+                    continue
+        except OSError:
+            pass
+        with open(out + ".tmp", "w") as f:
+            f.write(data)
+        os.replace(out + ".tmp", out)
+    for name in os.listdir(OUT):
+        if name.endswith(".desktop") and name not in keep:
+            os.remove(os.path.join(OUT, name))
+    return len(keep)
+
+
+if __name__ == "__main__":
+    if len(sys.argv) > 1:
+        sys.exit(__doc__)
+    print(f"{write_all()} apps with Launch as Standalone in {OUT}")

gaze/README.md

@@ -4,24 +4,38 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (t
 
 - `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on.
 - `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log.
-- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground. It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
+- `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
+- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
+
+Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`) and then our own tracker (`gaze/tracker/install.sh`), which gaze mode uses once it's installed; the probe is installed by hand.
 
 ```
-gaze/build.sh                 # build ft-gaze
-gaze/probe/install.sh         # build, and add Frametop Gaze Probe to the app menu
-gaze/probe/ft-gazeprobe --screen 1
-gaze/run.sh install           # the gaze service, with SteamVR
+gaze/run.sh install           # the gaze service: builds ft-gaze and the panel, starts with SteamVR
 gaze/ft-gazectl on            # the pointer follows your gaze (off: the mouse alone)
+gaze/tracker/install.sh       # our own eye tracker's frame grabber (asks for sudo)
+gaze/build.sh                 # build ft-gaze and the panel by hand
+gaze/probe/install.sh         # development: build, and add Frametop Gaze Probe to the app menu
+gaze/probe/ft-gazeprobe --screen 1
 ```
 
 ## Gaze pointer
 
 Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:
 
-- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. It uses SteamVR's combined gaze (mmap set 1), which keeps going when the tracker loses one eye; set 2's combined direction is off by half of whatever the lost eye reads, and the tracker does lose an eye for minutes at a time. It drops blinks (both eyes closing), smooths with a fixation lock, and applies the calibration from the probe (`calibration.json`, reloaded when the probe changes it) plus what the pointer has learned since (`pointer-lessons.json`). It sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
-- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse or controller button mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on. The dot only shows while the mouse moves it, while a press is held, and as a pulse when you click.
-- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
+- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes:
+  - **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
+  - **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
+
+  With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
+
+  Looks down past the screens (under 20 degrees down, on no Frametop screen: a glance at the keyboard) aren't sent, so the pointer stays where it was instead of following you down, and eyes lost there aren't counted. It drops blinks (both eyes closing or lost), smooths with a fixation lock, and sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
+- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse button or key combination mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look, and the mouse does the last bit. By default the mouse moves it only while a button is held (see "The mouse only corrects" below). With `POINTER_GAZE_MOUSE_MOVE=free`, moving the mouse takes the pointer, from where the gaze put it, and looking well away (5 degrees) gives it back to the gaze. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on, until a controller is picked up. The dot shows all the time (`POINTER_GAZE_DOT=moving`: only while the mouse moves it, while a press is held, and as a pulse when you click). Gaze mode works with the mouse and the keyboard, not the controllers ([docs/gaze-controllers.md](../docs/gaze-controllers.md) explains why).
+- **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
+- **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
+- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
+- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
 - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
+- **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there.
 
 Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.
 
@@ -30,13 +44,38 @@ Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction,
 | Source | Where it comes from |
 | --- | --- |
 | SteamVR action | An `eyetracking` action bound to `/user/head/eyetracking` (`actions/`), read with `IVRInput::GetEyeTrackingDataRelativeToNow`. This is the supported way. |
-| mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. |
+| mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space: set 1 is filtered, and its two eyes always share one pitch; set 2 is each eye's own reading. After each set come the tracker's variances for each eye, and at the end each eye's raw measurement and its variance (the tracker's confidence in that frame), which ft-gaze passes on for the fit check. |
+| Left eye, right eye | Each eye alone, from set 2: calibrate and test them to see what one eye is worth against both. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. |
+| Own tracker | Our own tracker (`tracker/`, experimental; see "Our own eye tracker"). It keeps its own calibration, not SteamVR's: Calibrate on the Gaze page fits it while Eye tracker is Own tracker (eight dots to a ring instead of six, on a slight oval, since the fit goes wrong past its dots; the probe's calibration with its tracker toggle on Own tracker does the same), and clicks teach it how far the headset has moved on your face since. After the headset was off, one look at a centre dot (the quick check, or the probe's first dot) resets that. With Own tracker on, the probe hides SteamVR's gaze and draws a red dot where each eye alone puts it, and asks the gaze service to keep the tracker running. The gaze pointer can use it too (Eye tracker: Own tracker, on the Gaze page of Frametop Input Settings). ft-gaze reports it as `own` while it's running, and as `{"ok":0}` otherwise. |
 
-The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`.
+The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`. When the tracker loses an eye, the same log says `CEyePoseUKF L: Large dt` (or `R`) as it starts that eye over.
+
+## Our own eye tracker
+
+`gaze/tracker/` is an eye tracker of our own, because SteamVR's is about 1.5 degrees off after the best correction the gaze service can learn, and what's left is mostly look-to-look noise that no correction on top of its output can remove. Ours processes the eye cameras itself: 0.59 degrees (median) in its best live session against 0.83 for SteamVR's with the probe's correction, and after the headset was taken off and put back without recalibrating, 0.58 once your first clicks had taught it where the headset sat (`tracker/findings.md` has the measurements).
+
+- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too).
+- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`.
+- `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones are meant for a PC: if you have `frame-job` (a personal tool, not in this repo), `gaze/tracker/.frame-job` sends them there. `lab/py` runs them with that Python (in the dev container on the Frame; on a PC, the same venv from `requirements.txt`, which frame-job's setup makes).
+
+Ground rules, for anyone changing it:
+
+- **Clean room.** Nothing of Valve's goes in: we don't decompile, disassemble, or patch the `eyetracking` binary or its network weights, and we don't copy their code or weights. Its public output (eye-server.mmap, read-only) is fair game as a baseline and as labels, and so are published papers and openly licensed pupil detectors (check each one's license: PuRe, PuReST, ElSe, and ExCuSe are non-commercial only).
+- **Root only reads.** ft-eyegrab never writes to, stops, or signals the `eyetracking` process, vrserver, or vrcompositor, never opens `/dev/adsp`, `/dev/cdsp`, or `/dev/spidev0.1`, and never writes to `/dev/shm/eye-server.mmap` (it also carries calibration clicks into SteamVR's tracker), `/opt`, or `/persist`.
+- **Eye images are biometric data.** Recordings live outside the repo, in `~/.local/share/frametop/eyes/captures` (0700), and `.gitignore` catches stray frame dumps. They go nowhere but the machine that runs your offline jobs.
+- **Mind the headset's budget.** Finding a pupil takes about 0.4 ms a frame while ft-eyes follows it, and 1.4-2.1 ms when it searches the whole frame. Replays, scoring, and training go to a PC.
+
+## Headset fit
+
+Check headset fit on the Gaze page opens it in the headset panel: a card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and the hints, live while you adjust the headset. A left click or Meta+J runs the guided check (dots, then looks down, up, left and right), and a right click or Meta+K closes it. The probe's Headset fit mode (`ft-gazeprobe --mode fit`, for development) has the same check with maps: it shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it.
+
+Losing an eye is usually about where you look, not the tracker. On this Frame the left eye was lost 57 to 64 % of the time looking 30 to 50 degrees down (at the keyboard) and the right eye never; at screen height both were seen over 98 % of the time. Looking down, the lids come down over the eyes. That's harmless, since the gaze service ignores looks down past the screens: they show on the maps, but not in the counts or as a problem.
 
 ## Probe
 
-The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The keys do the same (Tab, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place.
+The probe is a development tool (in the Gaze page's overflow menu): calibration experiments, accuracy tests, and practice modes. Users calibrate and check in the headset panel instead.
+
+The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The arrow in the panel's title bar collapses it to just that bar, so the dot and targets behind it stay visible; the collapsed bar stays through tests. The keys do the same (Tab, C, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place.
 
 - **Run calibration (start here):** the initial calibration, modeled on Apple Vision Pro's eye setup. Face the centre and keep your head still. Look at one dot and press the trigger, then at each of six dots in a circle. That happens in three rounds, and the screen goes dark, then medium, then bright, because pupil size changes with brightness and the tracker's error with it. Each round turns the ring 20 degrees, and the middle round's ring is half the size, so the 21 dots cover the middle, halfway out, and the edge of your view. The ring's size is `Calibration ring` (degrees, 20 by default, less if the window is too small). Error grows toward the edge, and the calibration can only correct as far out as it has seen dots. The current dot is a bright pulsing dot with a point in the middle; finished dots fade to specks, so your eyes don't go back to them. Samples from blinks and from moments when the tracker lost an eye are dropped: openness under half of what it was during that look (not a fixed level, because your lids come down when you look down, and you squint in the bright round), or the angle between the eyes jumping more than 1.5 degrees from its median (that angle depends on how far away you're looking, so only a jump counts). Each dot is measured with medians, so one bad sample can't fail it. A look that lands where the gaze was for another dot of the round is refused as a look at the wrong dot. Mouse clicks don't count during a calibration run or a test: use Enter or Space. If a dot still fails, the message says why and the next try listens longer. After two failures, S (or the menu) skips the dot. Every attempt is logged to `calibration-attempts.jsonl`. At the end it fits every source's calibration from all the dots, replacing what it had learned (quadratic if the model was none). Esc cancels. The run is saved as `calibration-*.json`. With "Test after calibration" on (the default), the accuracy test starts right after, on new spots.
 - **Free look:** the gaze dot. The trigger calibrates wherever you're looking (see below).

gaze/build.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
-# Build ft-gaze in the dev container on the Frame (gaze/build/ft-gaze; it also runs there).
+# Build ft-gaze and the calibration panel ft-gazepanel in the dev container on the Frame
+# (gaze/build/; they also run there). The panel draws its text with stb_truetype (public
+# domain, one header, pinned as in screens/build.sh).
 # The eye tracking API (IVRInput::GetEyeTrackingDataRelativeToNow) is newer than the header
 # shipped with SteamVR's samples, so this uses the pinned public header ft-screens fetches.
 set -euo pipefail
@@ -10,4 +12,9 @@ openvr=v2.15.6
 [ -f build/include/openvr-$openvr ] || { curl -fsSL "https://raw.githubusercontent.com/ValveSoftware/openvr/$openvr/headers/openvr.h" -o build/include/openvr.h && touch build/include/openvr-$openvr; }
 g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include -I../pointer/common \
   -o build/ft-gaze ft-gaze.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
-echo "built build/ft-gaze"'
+stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
+[ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
+g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
+  -o build/ft-gazepanel panel/ft-gazepanel.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 \
+  $(pkg-config --libs gbm libdrm) -lpthread
+echo "built build/ft-gaze build/ft-gazepanel"'

gaze/fitcheck.py

@@ -0,0 +1,367 @@
+"""fitcheck: how well the eye tracker sees each eye, for fitting the headset (the headset
+panel's fit check, gazecheck.py, and ft-gazeprobe's Headset fit mode).
+
+From each ft-gaze sample it takes, per eye, whether the tracker has that eye (its variance
+for the eye's direction, "unc", under EYE_LOST; see gazecal), how open the eye is, and the
+tracker's own confidence in its latest measurement of it ("eye" "q": the measurement's
+variance, about 2e-5 on a clear view). It keeps that per direction you look in (10 degree
+cells, and a few named regions), so a map shows where each eye gets lost, and turns it into
+hints.
+
+On the Frame this was written for, the left eye was lost 57-63 % of the time looking 30-50
+degrees down (at the keyboard) and the right never; at screen height both were seen over
+98 % of the time. Looking down, the lids come down over the eyes, and a glance at the
+keyboard isn't where the gaze pointer matters: ft-gazed ignores looks down past the
+screens. So those are on the maps, but not in the cards' counts or the hints' warnings.
+
+Directions are head-relative degrees (yaw +left, pitch +up), the combined gaze's.
+"""
+
+import math
+import statistics
+from collections import deque
+
+from gazecal import EYE_FOUND, EYE_LOST
+
+EYES = ("Left eye", "Right eye")
+CELL = 10.0
+YAW = (-40, 40)
+PITCH = (-50, 30)
+CLOSED = 0.12      # openness under this: closed (a blink, or squeezed shut)
+MIN_REGION = 60    # samples in a region before it's judged (two thirds of a second)
+
+# Named regions, for the hints: (key, words, test on yaw and pitch).
+REGIONS = [
+    ("down", "down (at a keyboard or desk)", lambda y, p: p < -20),
+    ("up", "up", lambda y, p: p > 15),
+    ("left", "to the left", lambda y, p: y > 20 and -20 <= p <= 15),
+    ("right", "to the right", lambda y, p: y < -20 and -20 <= p <= 15),
+    ("centre", "straight ahead (screen height)", lambda y, p: abs(y) <= 20 and -20 <= p <= 15),
+]
+
+# The guided check: dots on the screen (fractions of its size; the corners stay clear of the
+# probe's title bar and toolbar), then prompts to look past it. Seconds each.
+GUIDE = [
+    ("dot", (0.5, 0.5), 2.0), ("dot", (0.12, 0.2), 2.0), ("dot", (0.88, 0.2), 2.0),
+    ("dot", (0.88, 0.92), 2.0), ("dot", (0.12, 0.92), 2.0), ("dot", (0.5, 0.92), 2.0),
+    ("look", "Look down at your keyboard", 4.0), ("look", "Look up, above the screen", 3.0),
+    ("look", "Look far to the left", 3.0), ("look", "Look far to the right", 3.0),
+    ("dot", (0.5, 0.5), 2.0),
+]
+
+
+class FitCheck:
+    def __init__(self):
+        self.reset()
+
+    def reset(self):
+        self.lost = [False, False]
+        self.lost_since = [None, None]
+        self.losses = [0, 0]                      # times each eye was lost
+        self.durations = [[], []]                 # how long each loss lasted (s)
+        self.cells = [{}, {}]                     # per eye: (i, j) -> [samples, lost]
+        self.regions = [{k: [0, 0] for k, _, _ in REGIONS} for _ in EYES]
+        self.recent = [deque(maxlen=900), deque(maxlen=900)]  # (t, lost) for the last 10 s
+        self.q = [deque(maxlen=180), deque(maxlen=180)]       # recent fresh measurement variances
+        self.open = [0.0, 0.0]
+        self.unc = [0.0, 0.0]
+        self.gaze = None
+        self.samples = 0
+        self.guide = None                          # {"start": t, "step": i, "results": [...]}
+        self.have_eye_data = False
+
+    # --- Samples ---
+
+    def feed(self, s, now):
+        m1 = s["src"].get("mmap1") or {}
+        unc, opens = m1.get("unc"), m1.get("open")
+        if "hy" not in m1 or not unc or not opens:
+            return
+        self.have_eye_data = True
+        self.samples += 1
+        eye = s.get("eye") or {}
+        hy, hp = m1["hy"], m1["hp"]
+        self.gaze = (hy, hp)
+        self.unc = list(unc)
+        for k in (0, 1):
+            self.open[k] += 0.2 * (opens[k] - self.open[k])
+            was = self.lost[k]
+            self.lost[k] = unc[k] > (EYE_FOUND if was else EYE_LOST)
+            if self.lost[k] and not was:
+                if not looking_down(hy, hp):
+                    self.losses[k] += 1
+                self.lost_since[k] = now
+            elif was and not self.lost[k] and self.lost_since[k] is not None:
+                if not looking_down(hy, hp):
+                    self.durations[k].append(now - self.lost_since[k])
+                self.lost_since[k] = None
+            q = eye.get("q")
+            if q and (eye.get("new") or [1, 1])[k]:
+                self.q[k].append(q[k])
+        closed = [opens[k] < CLOSED for k in (0, 1)]
+        if all(closed) or all(self.lost):
+            return  # a blink: says nothing about the fit
+        key = (math.floor(hy / CELL), math.floor(hp / CELL))
+        for k in (0, 1):
+            c = self.cells[k].setdefault(key, [0, 0])
+            c[0] += 1
+            c[1] += self.lost[k]
+            for rk, _, test in REGIONS:
+                if test(hy, hp):
+                    r = self.regions[k][rk]
+                    r[0] += 1
+                    r[1] += self.lost[k]
+            if not looking_down(hy, hp):
+                self.recent[k].append((now, self.lost[k]))
+        g = self.guide
+        if g and g["step"] < len(GUIDE):
+            res = g["results"][g["step"]]
+            res[0] += 1
+            res[1] += self.lost[0]
+            res[2] += self.lost[1]
+
+    # --- The guided check ---
+
+    def toggle_guide(self, now):
+        if self.guide and self.guide["step"] < len(GUIDE):
+            self.guide = None
+        else:
+            self.guide = {"start": now, "step": 0, "step_start": now, "results": [[0, 0, 0] for _ in GUIDE]}
+
+    def guide_step(self, now):
+        """The current step (kind, what, seconds left), or None when there's no check running."""
+        g = self.guide
+        if not g or g["step"] >= len(GUIDE):
+            return None
+        kind, what, secs = GUIDE[g["step"]]
+        if now - g["step_start"] >= secs:
+            g["step"] += 1
+            g["step_start"] = now
+            return self.guide_step(now)
+        return kind, what, secs - (now - g["step_start"])
+
+    # --- Summaries ---
+
+    def status(self, k):
+        if not self.samples:
+            return "no data", (0.6, 0.6, 0.6)
+        if self.lost[k]:
+            return "LOST", (1.0, 0.35, 0.3)
+        if self.open[k] < CLOSED:
+            return "closed", (0.8, 0.8, 0.8)
+        return "tracking", (0.35, 1.0, 0.5)
+
+    def tracked_share(self, k, now, window=10.0):
+        pts = [lost for t, lost in self.recent[k] if now - t <= window]
+        return (1 - sum(pts) / len(pts)) if pts else None
+
+    def signal(self, k):
+        """The tracker's recent confidence in this eye, 0..1 (from its measurement variance:
+        2e-5 or less is 1, 1e-3 or more is 0), or None."""
+        if len(self.q[k]) < 10:
+            return None
+        q = statistics.median(self.q[k])
+        return min(1.0, max(0.0, (math.log10(1e-3) - math.log10(max(q, 1e-9))) / (math.log10(1e-3) - math.log10(2e-5))))
+
+    def region_share(self, k, key):
+        n, lost = self.regions[k][key]
+        return (lost / n) if n >= MIN_REGION else None
+
+    def hints(self):
+        if not self.have_eye_data:
+            return ["No per-eye data from ft-gaze (it needs SteamVR's eye-server.mmap, and a current build)."]
+        if self.samples < 3 * MIN_REGION:
+            return ["Look around slowly (the screen's corners, then down at your keyboard, up, left and right) "
+                    "or press Enter for a guided check."]
+        out = []
+        bad = {}
+        for k in (0, 1):
+            for key, words, _ in REGIONS:
+                share = self.region_share(k, key)
+                if share is not None and share >= 0.15:
+                    bad.setdefault(key, {})[k] = share
+        for key, words, _ in REGIONS:
+            if key not in bad:
+                continue
+            eyes = bad[key]
+            if len(eyes) == 2:
+                if key == "down":
+                    out.append("Both eyes get lost looking down at the keyboard. That's fine: the gaze service "
+                               "ignores looks down past the screens.")
+                elif key == "centre":
+                    out.append(f"Both eyes get lost looking {words} ({eyes[0]:.0%} and {eyes[1]:.0%} of the time): "
+                               "check the lenses are clean and the headset is on as usual; if it stays like this, "
+                               "the tracker isn't getting a clear view of either eye.")
+                else:
+                    out.append(f"Both eyes get lost looking {words}: that's past what the tracker covers for your "
+                               "face, not one eye's fit.")
+                continue
+            k = next(iter(eyes))
+            other = self.region_share(1 - k, key)
+            vs = f", the {EYES[1 - k].lower()} {other:.0%}" if other is not None else ""
+            line = f"{EYES[k]}: lost {eyes[k]:.0%} of the time looking {words}{vs}."
+            if key == "down":
+                line += (" That's fine: glancing at the keyboard, the lids come down over the eyes, and the gaze "
+                         "service ignores looks down past the screens, so the pointer stays put.")
+            elif key == "centre":
+                line += (" Even at screen height: clean that lens, and check its distance from your eye and the "
+                         "IPD. Lashes that touch the lens get in the camera's way too.")
+            else:
+                line += (" At the edge of your view: try the IPD setting, and centring the headset between your "
+                         "eyes.")
+            out.append(line)
+        s0, s1 = self.signal(0), self.signal(1)
+        if s0 is not None and s1 is not None and abs(s0 - s1) > 0.25:
+            k = 0 if s0 < s1 else 1
+            out.append(f"The tracker is less sure of your {EYES[k].lower()} even when it has it "
+                       f"(signal {min(s0, s1):.0%} against {max(s0, s1):.0%}).")
+        if not out:
+            out.append("Both eyes are tracked everywhere you've looked so far.")
+        return out
+
+    # --- Drawing (cairo) ---
+
+    def draw(self, cr, w, h, text, now):
+        # Right of the probe's collapsed title bar, under its toolbar (top right).
+        left = 300
+        top = 190
+        text(cr, left, top - 60, "Headset fit", (1, 1, 1), 30)
+        text(cr, left, top - 28, "Adjust the headset and watch each eye. Enter: guided check. R: start over.",
+             (0.8, 0.8, 0.8), 18)
+        card_w = min(560, (w - left - 80) / 2)
+        mh = max(0, min(card_w * 0.8, h - top - 280 - 200))
+        for k in (0, 1):
+            x = left + k * (card_w + 40)
+            self.draw_card(cr, x, top, card_w, text, now, k)
+            self.draw_map(cr, x, top + 280, card_w, mh, text, k)
+        y = top + 280 + (mh + 60 if mh >= 80 else 0)
+        for line in self.hints()[:4]:
+            for part in wrap(line, max(40, int((w - left - 40) / 10))):
+                if y > h - 30:
+                    break
+                text(cr, left, y, part, (1, 0.95, 0.75), 18)
+                y += 26
+            y += 8
+        step = self.guide_step(now)
+        g = self.guide
+        if step:
+            kind, what, left_s = step
+            if kind == "dot":
+                fx, fy = what
+                x, y = fx * w, fy * h
+                cr.set_source_rgba(1, 0.85, 0.2, 0.95)
+                cr.arc(x, y, 14 + 4 * math.sin(now * 6), 0, 2 * math.pi)
+                cr.fill()
+            else:
+                text(cr, w / 2 - 260, h / 2, f"{what} ({left_s:.0f})", (1, 0.85, 0.2), 34)
+        elif g and g["step"] >= len(GUIDE):
+            self.draw_guide_results(cr, w, h, text)
+
+    def draw_card(self, cr, x, y, cw, text, now, k):
+        cr.set_source_rgba(1, 1, 1, 0.06)
+        cr.rectangle(x, y, cw, 230)
+        cr.fill()
+        word, col = self.status(k)
+        text(cr, x + 16, y + 38, EYES[k], (1, 1, 1), 26)
+        cr.select_font_face("sans")
+        cr.set_font_size(26)
+        text(cr, x + cw - 16 - cr.text_extents(word).x_advance, y + 38, word, col, 26)
+        rows = [("Open", self.open[k]), ("Signal", self.signal(k)), ("Seen, last 10 s", self.tracked_share(k, now))]
+        yy = y + 70
+        for label, v in rows:
+            text(cr, x + 16, yy + 16, label, (0.85, 0.85, 0.85), 17)
+            bx, bw = x + 170, cw - 250
+            cr.set_source_rgba(1, 1, 1, 0.12)
+            cr.rectangle(bx, yy, bw, 20)
+            cr.fill()
+            if v is not None:
+                v = min(1.0, max(0.0, v))
+                cr.set_source_rgba(*bar_colour(v), 0.9)
+                cr.rectangle(bx, yy, bw * v, 20)
+                cr.fill()
+                text(cr, bx + bw + 10, yy + 16, f"{v:.0%}", (0.9, 0.9, 0.9), 17)
+            yy += 36
+        d = self.durations[k]
+        longest = max(d) if d else 0
+        n = self.losses[k]
+        text(cr, x + 16, yy + 22, f"Lost {n} time{'' if n == 1 else 's'}" + (f", longest {longest:.1f} s" if longest >= 0.05 else ""),
+             (0.85, 0.85, 0.85), 17)
+
+    def draw_map(self, cr, x, y, mw, mh, text, k):
+        """Where you looked (yaw across, pitch up), each cell coloured by how often this eye
+        was lost there: green never, red always, dark: not looked there yet."""
+        if mh < 80:
+            return
+        cols = int((YAW[1] - YAW[0]) / CELL)
+        rows = int((PITCH[1] - PITCH[0]) / CELL)
+        cw, ch = mw / cols, mh / rows
+        text(cr, x, y - 8, f"Where the {EYES[k].lower()} gets lost", (0.85, 0.85, 0.85), 17)
+        for i in range(cols):
+            yaw_i = math.floor(YAW[1] / CELL) - 1 - i       # left of the map is your left (+yaw)
+            for j in range(rows):
+                pitch_j = math.floor(PITCH[1] / CELL) - 1 - j
+                c = self.cells[k].get((yaw_i, pitch_j))
+                cx, cy = x + i * cw, y + j * ch
+                if c and c[0] >= 10:
+                    share = c[1] / c[0]
+                    cr.set_source_rgba(*bar_colour(1 - share), 0.75)
+                else:
+                    cr.set_source_rgba(1, 1, 1, 0.05)
+                cr.rectangle(cx + 1, cy + 1, cw - 2, ch - 2)
+                cr.fill()
+        # Straight ahead, and the gaze now.
+        def at(yaw, pitch):
+            return x + (YAW[1] - yaw) / (YAW[1] - YAW[0]) * mw, y + (PITCH[1] - pitch) / (PITCH[1] - PITCH[0]) * mh
+        cr.set_source_rgba(1, 1, 1, 0.35)
+        cr.set_line_width(1)
+        ox, oy = at(0, 0)
+        cr.move_to(ox - 10, oy)
+        cr.line_to(ox + 10, oy)
+        cr.move_to(ox, oy - 10)
+        cr.line_to(ox, oy + 10)
+        cr.stroke()
+        text(cr, x, y + mh + 20, "+ ahead, bottom rows: keyboard", (0.6, 0.6, 0.6), 14)
+        if self.gaze:
+            gx, gy = at(max(YAW[0], min(YAW[1], self.gaze[0])), max(PITCH[0], min(PITCH[1], self.gaze[1])))
+            cr.set_source_rgba(1, 1, 1, 0.95)
+            cr.arc(gx, gy, 5, 0, 2 * math.pi)
+            cr.fill()
+
+    def draw_guide_results(self, cr, w, h, text):
+        res = self.guide["results"]
+        lines = []
+        for (kind, what, _), (n, l0, l1) in zip(GUIDE, res):
+            if not n:
+                continue
+            name = what if kind == "look" else "dot at {:.0%}, {:.0%}".format(*what)
+            lines.append(f"{name}: left lost {l0 / n:.0%}, right {l1 / n:.0%}")
+        y = h / 2 - 20 * len(lines)
+        text(cr, w / 2 - 300, y - 40, "Guided check", (1, 0.85, 0.2), 26)
+        for line in lines:
+            text(cr, w / 2 - 300, y, line, (1, 1, 1), 19)
+            y += 30
+
+
+def looking_down(yaw, pitch):
+    """A look down at the keyboard: the "down" region, which ft-gazed doesn't send on."""
+    return pitch < -20
+
+
+def bar_colour(v):
+    """Red (0) through amber to green (1)."""
+    if v < 0.5:
+        return 1.0, 0.3 + 0.9 * v, 0.3
+    return 1.0 - 1.3 * (v - 0.5), 0.75 + 0.25 * (v - 0.5) * 2, 0.35
+
+
+def wrap(s, width):
+    words, lines, cur = s.split(), [], ""
+    for wd in words:
+        if cur and len(cur) + 1 + len(wd) > width:
+            lines.append(cur)
+            cur = wd
+        else:
+            cur = f"{cur} {wd}".strip()
+    if cur:
+        lines.append(cur)
+    return lines

gaze/frametop-gaze.service

@@ -5,6 +5,7 @@ Documentation=file://@REPO@/gaze/README.md
 # Needs SteamVR's IPC (ft-gaze is an overlay client); it starts and stops with SteamVR.
 After=steamvr.service frametop-pointer.service
 PartOf=steamvr.service
+Requisite=steamvr.service
 
 [Service]
 # Host Python; it runs ft-gaze in the dev container (distrobox enter), which quits when

gaze/ft-gaze.cpp

@@ -1,5 +1,5 @@
 // ft-gaze: the headset's eye tracking as rays and Frametop screen pixels (OpenVR overlay
-// client, runs in the dev container). An experiment for gaze input; ft-gazeprobe reads it.
+// client, runs in the dev container). The gaze service (ft-gazed) and the gaze probe run it.
 //
 // Every eye tracker sample (90 Hz) becomes one JSON line on stdout with each gaze source
 // hit-tested against the Frametop screens:
@@ -8,13 +8,30 @@
 //
 //   {"t":<sample time, CLOCK_MONOTONIC_RAW s>,"age":<ms old when read>,"n":<sample counter>,
 //    "head":{"yaw":..,"pitch":..,"hit":HIT},          head forward ray (for head nudging)
-//    "src":{"action":SRC,"mmap1":SRC,"mmap2":SRC}}
+//    "src":{"action":SRC,"mmap1":SRC,"mmap2":SRC,"left":SRC,"right":SRC,"own":SRC},"eye":EYE}
 //   SRC = {"hy":..,"hp":..,"hit":HIT} or {"ok":0}    hy/hp: gaze direction relative to the
 //                                                     head, degrees (yaw +left, pitch +up)
 //         mmap1 adds "open":[l,r] (probably eye openness, 0 in a blink) and "dist" (vergence
 //         distance, m); both mmap sets add "lr", the angle between the eyes (deg), which
 //         jumps when the tracker loses an eye, and "eyes":[[hy,hp],[hy,hp]], each eye's own
-//         direction (left, right), for calibrating the eyes separately.
+//         direction (left, right), for calibrating the eyes separately, and "unc":[l,r],
+//         the tracker's uncertainty about each eye's direction (its filter's variance):
+//         about 0.0005-0.002 while it sees the eye, 0.015-0.03 once it's lost it.
+//    "left":SRC,"right":SRC                           each eye's own direction from set 2
+//         (set 1's eyes always share one pitch, and while it's lost an eye it keeps that
+//         eye's yaw where it was: set 2 is each eye's own reading). From the head's origin,
+//         not the eye's.
+//    "own":SRC                                        our own tracker (gaze/tracker/ft-eyes), from
+//         /dev/shm/frametop-eyes-gaze; adds "age" (ms since its frame), "eyes":[[hy,hp],[hy,hp]]
+//         (left, right; null for an eye it doesn't see), "ehit":[HIT,HIT] where each of those
+//         lands, and "slip":[[x,y],[x,y]] (left, right: each eye's shift in its camera image
+//         since the calibration, pixels; null until a click has measured it). {"ok":0}
+//         without the file or when it's over 100 ms old.
+//   EYE = {"q":[l,r],"m":[[x,y],[x,y]],"new":[l,r]}  the tracker's latest measurement of
+//         each eye before filtering: "m" (camera-relative, undocumented units), "q" its
+//         variance (about 2e-5 on a clear view of the eye, rising as the lid or lashes get
+//         in the way), "new" whether it changed since the last sample (it freezes while the
+//         tracker can't see that eye, and in blinks). "eye" is null without the mmap.
 //   HIT = {"s":<screen>,"x":..,"y":..,"j":[dx/dhy,dy/dhy,dx/dhp,dy/dhp],"dpp":<deg per px>}
 //         or null. x, y are pixels on that screen; j is pixels per degree of head-relative
 //         yaw and pitch there, so a correction in degrees can be turned into pixels and back.
@@ -78,7 +95,13 @@ constexpr size_t kLeft1 = 0x15f, kRight1 = 0x16b;  // set 1: unit vectors, head
 constexpr size_t kFix1 = 0x18f;    // set 1 fixation point: length is the vergence distance (m)
 constexpr size_t kLeft2 = 0x19b, kRight2 = 0x1a7;  // set 2
 constexpr size_t kOpen = 0x1cb;    // two floats, 0..1: probably eye openness or confidence
-constexpr size_t kNeed = 0x1d3;
+// After each set's two directions, six floats: the left eye's variance (three), the
+// right's (three; the middle one of each is shared). They jump when an eye is lost.
+constexpr size_t kVar1 = 0x177, kVar2 = 0x1b3;
+// The measurements the filter is fed: left x, y, right x, y, then the variance of each (left
+// x, y, right x, y). An eye's pair stops changing while the tracker can't see it.
+constexpr size_t kMeas = 0x1d3;
+constexpr size_t kNeed = 0x1f3;
 
 struct EyeFile {
     const uint8_t *p = nullptr;
@@ -115,6 +138,7 @@ struct EyeSample {
     double t = 0;
     Vec3 left1, right1, fix1, left2, right2;
     float open[2] = {0, 0};
+    float var1[6] = {}, var2[6] = {}, meas[8] = {};
 };
 
 // A consistent copy: the writer has no seqlock we can use, so read until the counter and
@@ -127,6 +151,9 @@ bool ReadSample(const EyeFile &f, EyeSample &s) {
         s.left1 = f.V(kLeft1), s.right1 = f.V(kRight1), s.fix1 = f.V(kFix1);
         s.left2 = f.V(kLeft2), s.right2 = f.V(kRight2);
         std::memcpy(s.open, f.p + kOpen, sizeof s.open);
+        std::memcpy(s.var1, f.p + kVar1, sizeof s.var1);
+        std::memcpy(s.var2, f.p + kVar2, sizeof s.var2);
+        std::memcpy(s.meas, f.p + kMeas, sizeof s.meas);
         std::atomic_thread_fence(std::memory_order_acquire);
         if (f.Get<uint32_t>(kCounter) == n0 && f.Get<double>(kTime) == t0) {
             s.n = n0, s.t = t0;
@@ -136,6 +163,69 @@ bool ReadSample(const EyeFile &f, EyeSample &s) {
     return false;
 }
 
+// --- Our own tracker: /dev/shm/frametop-eyes-gaze, written by gaze/tracker/ft-eyes ---
+// Layout (ft-eyes' docstring): u32 seq (odd while written), u32 version, f64 t, f32 yaw,
+// pitch, u32 flags (bit 0 right eye, 1 left, 2 right slip known, 3 left), u32 n, then f32
+// right yaw, pitch, left yaw, pitch; slip right x, y, left x, y; pupils (unused here).
+struct OwnSample {
+    double t = 0;
+    float yaw = 0, pitch = 0;
+    uint32_t flags = 0, n = 0;
+    float eyes[4] = {}, slip[4] = {};
+};
+
+class OwnFile {
+public:
+    // Reopened when it appears or is replaced, since ft-eyes may start after us.
+    bool Read(OwnSample &o) {
+        const double now = NowRaw();
+        if (!p_ || now - checked_ > 2.0) Reopen(now);
+        if (!p_) return false;
+        for (int attempt = 0; attempt < 4; ++attempt) {
+            uint32_t s0, s1, version;
+            std::memcpy(&s0, p_, 4);
+            if (s0 & 1) continue;
+            std::atomic_thread_fence(std::memory_order_acquire);
+            std::memcpy(&version, p_ + 4, 4);
+            std::memcpy(&o.t, p_ + 8, 8);
+            std::memcpy(&o.yaw, p_ + 16, 4);
+            std::memcpy(&o.pitch, p_ + 20, 4);
+            std::memcpy(&o.flags, p_ + 24, 4);
+            std::memcpy(&o.n, p_ + 28, 4);
+            std::memcpy(o.eyes, p_ + 32, sizeof o.eyes);
+            std::memcpy(o.slip, p_ + 48, sizeof o.slip);
+            std::atomic_thread_fence(std::memory_order_acquire);
+            std::memcpy(&s1, p_, 4);
+            if (s0 == s1) return version == 1;
+        }
+        return false;
+    }
+
+private:
+    static constexpr size_t kSize = 128;
+    void Reopen(double now) {
+        checked_ = now;
+        struct stat st {};
+        if (stat("/dev/shm/frametop-eyes-gaze", &st) != 0) return Close();
+        if (p_ && st.st_ino == ino_) return;
+        Close();
+        const int fd = open("/dev/shm/frametop-eyes-gaze", O_RDONLY | O_CLOEXEC);
+        if (fd < 0) return;
+        if (fstat(fd, &st) == 0 && size_t(st.st_size) >= kSize) {
+            void *m = mmap(nullptr, kSize, PROT_READ, MAP_SHARED, fd, 0);
+            if (m != MAP_FAILED) p_ = static_cast<const uint8_t *>(m), ino_ = st.st_ino;
+        }
+        close(fd);
+    }
+    void Close() {
+        if (p_) munmap(const_cast<uint8_t *>(p_), kSize);
+        p_ = nullptr;
+    }
+    const uint8_t *p_ = nullptr;
+    ino_t ino_ = 0;
+    double checked_ = -1e9;
+};
+
 // --- Screens from ft-screens ---
 struct Screen {
     int index = 0;
@@ -351,7 +441,11 @@ int main(int argc, char **argv) {
             stdinClosed = true;
         }).detach();
     vr::EVRInitError err = vr::VRInitError_None;
-    vr::VR_Init(&err, vr::VRApplication_Overlay);
+    vr::VR_Init(&err, vr::VRApplication_Background);
+    if (err == vr::VRInitError_None) {
+        vr::VR_Shutdown();
+        vr::VR_Init(&err, vr::VRApplication_Overlay);
+    }
     if (err != vr::VRInitError_None) {
         std::fprintf(stderr, "ft-gaze: SteamVR: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
         return 1;
@@ -367,22 +461,42 @@ int main(int argc, char **argv) {
     vr::VRActionSetHandle_t set = vr::k_ulInvalidActionSetHandle;
     input->GetActionHandle("/actions/gaze/in/gaze", &gaze);
     input->GetActionSetHandle("/actions/gaze", &set);
-    std::fprintf(stderr, "ft-gaze: action manifest %s: error %d\n", manifest.c_str(), int(me));
+    if (me == vr::VRInputError_None)
+        std::fprintf(stderr, "ft-gaze: action manifest %s: ok\n", manifest.c_str());
+    else
+        std::fprintf(stderr, "ft-gaze: action manifest %s: error %d\n", manifest.c_str(), int(me));
 
     EyeFile eyes;
     const bool haveMmap = eyes.Open();
     std::fprintf(stderr, "ft-gaze: eye-server.mmap %s\n", haveMmap ? "open" : "not available");
 
+    OwnFile ownFile;
     Screens screens;
     screens.Start();
     PoseHistory history;
     uint32_t lastN = 0;
+    float lastMeas[8] = {};
     double lastEmit = 0;
     int actionErrors = 0;
     vr::EVRInputError lastActionError = vr::VRInputError_None;
+    // During a VR game, SteamVR's gaze action is left alone. With ft-gaze reading the eyes, SteamVR
+    // restarted its eye tracker every 10 s or so in a game, as if the headset came off, and each
+    // restart took input focus from the game: Beat Saber paused (PR #13). Of what ft-gaze reads,
+    // only the action reaches SteamVR (the mmap and our tracker are files), so gaze still works
+    // over the dashboard. Games are told apart the way ft-screens does it, by the scene app.
+    bool inGame = false;
+    double nextGameCheck = 0;
 
     while (true) {
         const double now = NowRaw();
+        if (now >= nextGameCheck) {
+            nextGameCheck = now + 0.5;
+            const bool game = vr::VRApplications()->GetCurrentSceneProcessId() != 0;
+            if (game != inGame)
+                std::fprintf(stderr, "ft-gaze: %s\n",
+                             game ? "a VR game is running: SteamVR's gaze action left alone" : "the VR game ended");
+            inGame = game;
+        }
         vr::TrackedDevicePose_t hp;
         sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &hp, 1);
         if (hp.bPoseIsValid) history.Add(now, hp.mDeviceToAbsoluteTracking);
@@ -403,27 +517,29 @@ int main(int argc, char **argv) {
             // SteamVR's action: a room-space origin and fixation point, turned into the head
             // frame so every source reports the same kind of angles.
             std::string action = "{\"ok\":0}";
-            vr::VRActiveActionSet_t active{};
-            active.ulActionSet = set;
-            active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin;
-            input->UpdateActionState(&active, sizeof active, 1);
-            vr::VREyeTrackingData_t e{};
-            const vr::EVRInputError ae =
-                input->GetEyeTrackingDataRelativeToNow(gaze, vr::TrackingUniverseStanding, 0, &e, sizeof e);
-            if (ae == vr::VRInputError_None && e.bActive && e.bValid) {
-                const Vec3 o{e.vGazeOrigin.v[0], e.vGazeOrigin.v[1], e.vGazeOrigin.v[2]};
-                const Vec3 t{e.vGazeTarget.v[0], e.vGazeTarget.v[1], e.vGazeTarget.v[2]};
-                const Vec3 dHead = RotateInverse(headNow, Normalize(t - o));
-                char extra[96];
-                std::snprintf(extra, sizeof extra, "\"tracked\":%d,\"dist\":%.3f,", int(e.bTracked), Length(t - o));
-                action = SrcJson(list, headNow, dHead, extra);
-            } else if (ae != lastActionError || (verbose && ++actionErrors % 90 == 1)) {
-                std::fprintf(stderr, "ft-gaze: action: error %d active %d valid %d\n", int(ae), int(e.bActive),
-                             int(e.bValid));
-                lastActionError = ae;
+            if (!inGame) {
+                vr::VRActiveActionSet_t active{};
+                active.ulActionSet = set;
+                active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin;
+                input->UpdateActionState(&active, sizeof active, 1);
+                vr::VREyeTrackingData_t e{};
+                const vr::EVRInputError ae =
+                    input->GetEyeTrackingDataRelativeToNow(gaze, vr::TrackingUniverseStanding, 0, &e, sizeof e);
+                if (ae == vr::VRInputError_None && e.bActive && e.bValid) {
+                    const Vec3 o{e.vGazeOrigin.v[0], e.vGazeOrigin.v[1], e.vGazeOrigin.v[2]};
+                    const Vec3 t{e.vGazeTarget.v[0], e.vGazeTarget.v[1], e.vGazeTarget.v[2]};
+                    const Vec3 dHead = RotateInverse(headNow, Normalize(t - o));
+                    char extra[96];
+                    std::snprintf(extra, sizeof extra, "\"tracked\":%d,\"dist\":%.3f,", int(e.bTracked), Length(t - o));
+                    action = SrcJson(list, headNow, dHead, extra);
+                } else if (ae != lastActionError || (verbose && ++actionErrors % 90 == 1)) {
+                    std::fprintf(stderr, "ft-gaze: action: error %d active %d valid %d\n", int(ae), int(e.bActive),
+                                 int(e.bValid));
+                    lastActionError = ae;
+                }
             }
 
-            std::string m1 = "{\"ok\":0}", m2 = m1;
+            std::string m1 = "{\"ok\":0}", m2 = m1, left = m1, right = m1, eye = "null";
             if (haveMmap) {
                 // lr: the angle between the two eyes' directions. It's a fraction of a degree
                 // normally; when the tracker loses one eye (or during a blink) it jumps.
@@ -438,12 +554,53 @@ int main(int argc, char **argv) {
                     std::snprintf(b, sizeof b, "\"eyes\":[[%.4f,%.4f],[%.4f,%.4f]],", ly, lp, ry, rp);
                     return std::string(b);
                 };
-                char extra[128];
+                auto unc = [](const float *v) {
+                    char b[64];
+                    std::snprintf(b, sizeof b, "\"unc\":[%.5f,%.5f],", std::max(v[0], v[2]), std::max(v[3], v[5]));
+                    return std::string(b);
+                };
+                char extra[256];
                 std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1),
                               s.open[0], s.open[1], lr(s.left1, s.right1));
-                m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1));
+                m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1) + unc(s.var1));
                 std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2));
-                m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2));
+                m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2) + unc(s.var2));
+                left = SrcJson(list, headThen, s.left2);
+                right = SrcJson(list, headThen, s.right2);
+                const float *m = s.meas;
+                const bool newL = m[0] != lastMeas[0] || m[1] != lastMeas[1];
+                const bool newR = m[2] != lastMeas[2] || m[3] != lastMeas[3];
+                std::memcpy(lastMeas, m, sizeof lastMeas);
+                std::snprintf(extra, sizeof extra, "{\"q\":[%.3g,%.3g],\"m\":[[%.4f,%.4f],[%.4f,%.4f]],\"new\":[%d,%d]}",
+                              (m[4] + m[5]) / 2, (m[6] + m[7]) / 2, m[0], m[1], m[2], m[3], int(newL), int(newR));
+                eye = extra;
+            }
+
+            // Our tracker: its own sample time picks the head pose, like the mmap's.
+            std::string own = "{\"ok\":0}";
+            OwnSample o;
+            if (ownFile.Read(o) && now - o.t < 0.1) {
+                vr::HmdMatrix34_t headOwn = headNow;
+                history.At(o.t, headOwn);
+                auto pair = [](bool ok, float a, float b) {
+                    char p[48];
+                    if (!ok) return std::string("null");
+                    std::snprintf(p, sizeof p, "[%.4f,%.4f]", a, b);
+                    return std::string(p);
+                };
+                // Stored right eye first; reported left first, like the other sources.
+                const std::string extra = "\"age\":" + std::to_string(int((now - o.t) * 1000)) +
+                                          ",\"eyes\":[" + pair(o.flags & 2, o.eyes[2], o.eyes[3]) + "," +
+                                          pair(o.flags & 1, o.eyes[0], o.eyes[1]) + "],\"slip\":[" +
+                                          pair(o.flags & 8, o.slip[2], o.slip[3]) + "," +
+                                          pair(o.flags & 4, o.slip[0], o.slip[1]) + "],";
+                // Where each eye's own gaze lands (left, right), for drawing them apart.
+                auto eyeHit = [&](bool ok, float y, float p) {
+                    return ok ? HitJson(list, headOwn, y, p) : std::string("null");
+                };
+                const std::string hits = "\"ehit\":[" + eyeHit(o.flags & 2, o.eyes[2], o.eyes[3]) + "," +
+                                         eyeHit(o.flags & 1, o.eyes[0], o.eyes[1]) + "],";
+                own = SrcJson(list, headOwn, Direction(o.yaw, o.pitch), extra + hits);
             }
 
             double yaw, pitch;
@@ -451,9 +608,10 @@ int main(int argc, char **argv) {
             yaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
             pitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
             std::printf("{\"t\":%.5f,\"age\":%.1f,\"n\":%u,\"head\":{\"yaw\":%.4f,\"pitch\":%.4f,\"hit\":%s},"
-                        "\"src\":{\"action\":%s,\"mmap1\":%s,\"mmap2\":%s}}\n",
+                        "\"src\":{\"action\":%s,\"mmap1\":%s,\"mmap2\":%s,\"left\":%s,\"right\":%s,\"own\":%s},"
+                        "\"eye\":%s}\n",
                         s.t, (now - s.t) * 1000, s.n, yaw, pitch, HitJson(list, headNow, 0, 0).c_str(), action.c_str(),
-                        m1.c_str(), m2.c_str());
+                        m1.c_str(), m2.c_str(), left.c_str(), right.c_str(), own.c_str(), eye.c_str());
             if (std::fflush(stdout) != 0) break;  // the reader went away
         }
 

gaze/ft-gazectl

@@ -36,8 +36,17 @@ def main():
         if r is None:
             sys.exit("the pointer helper isn't running")
         print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)")
-        if ask("ft_gazed", "status", 0.3) is None:
+        st = ask("ft_gazed", "status", 0.3)
+        if st is None:
             print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow")
+        elif r == "ok on":
+            try:
+                calibrated = (json.loads(st).get("checks") or {}).get("calibrated")
+            except ValueError:
+                calibrated = None
+            if calibrated is False:
+                print("note: no calibration for this eye tracker yet; it opens in the headset "
+                      "(if it can't, ft-gazectl status says why: checks.problem)")
     elif cmd in ("status", "forget", "reload"):
         r = ask("ft_gazed", cmd)
         if r is None:

gaze/ft-gazed

@@ -1,49 +1,115 @@
 #!/usr/bin/python3
 """ft-gazed: the gaze service. The headset's eye tracking, corrected, for the pointer.
 
+Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read
+again when the file changes:
+  GAZE_TRACKER=auto|own|steam
+                           our own eye tracker (gaze/tracker/ft-eyes, ft-gaze's source "own";
+                           this service runs it, see below) or SteamVR's. auto (the default)
+                           is ours when it's installed (gaze/tracker/install.sh: the frame
+                           grabber, and ft-eyes' Python in this checkout), else SteamVR's; it
+                           switches when ours is installed or removed.
+  GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far
+                           off it was at your recent nudges; left or right counts that eye twice
+                           as much as the other. Either eye alone carries the gaze when the
+                           other isn't seen.
+
 Runs ft-gaze (in the dev container), and for every eye tracker sample (90 Hz):
 
-  1. drops blinks: both eyes' openness under half its running median (each eye its own).
-     With the default source, mmap set 1, that's all: set 1 is SteamVR's combined gaze,
-     which keeps going when the tracker loses one eye (its two directions stay together).
-     Set 2's combined direction is the mean of the eyes' own, so with one eye lost it's
-     off by half of whatever that eye reads (9 to 14 degrees apart were seen): with set 2,
-     samples with an eye under its floor, or the angle between the eyes jumping more than
-     1.5 degrees from its median, are dropped too;
-  2. smooths it with a fixation lock (the running mean of the current fixation, 1 degree);
-  3. corrects it: the calibration from ft-gazeprobe (calibration.json, reloaded when the
-     probe changes it) plus what the pointer's corrections have taught since (LiveCorrection,
-     saved in pointer-lessons.json);
+  1. drops blinks: both eyes' openness under half its running median (each eye its own),
+     or both lost (the tracker's variance for them, ft-gaze's "unc", over EYE_LOST);
+     Looks down past the screens (pitch under KEYBOARD_PITCH, on no Frametop screen: at the
+     keyboard, through the gap by the nose) aren't sent, so the pointer stays where it was
+     instead of following you down; the tracker often loses an eye there (the lids come
+     down), and that isn't counted as a lost eye either;
+  2. combines the eyes, each corrected on its own. With SteamVR, that's each eye's own
+     reading (set 2, ft-gaze's "left" and "right"), corrected by its calibration from
+     ft-gazeprobe (calibration.json, reloaded when the probe changes it) plus what the
+     pointer's corrections have taught that eye since (LiveCorrection, saved in
+     pointer-lessons.json), then weighted by the eye bias. A lost or closed eye drops out.
+     Our own tracker keeps its own calibration, so its eyes are used as they come;
+  3. smooths it with a fixation lock (the running mean of the current fixation, 1 degree);
   4. sends it to the pointer helper: "gz <yaw> <pitch> <raw yaw> <raw pitch>", head-relative
      degrees (yaw +left, pitch +up). The helper uses it only in gaze mode.
 
+Without per-eye calibrations (a calibration from before the probe had the eyes as sources),
+or with --source, it's the older path: one source, SteamVR's combined gaze (mmap set 1) by
+default, corrected as a whole. There, with one eye lost or closed, the gaze comes from the
+other (EyeFallback: that eye's own reading from set 2, plus what it usually reads against
+the combined gaze, learned while both are seen). SteamVR's combined gaze (set 1) keeps going
+on one eye too, but holds the lost eye's yaw, so it moves half as far sideways as the eyes
+do. Before the fallback has learned an eye, set 1 is used as it is; set 2's combined
+direction is the mean of the eyes' own (off by half of whatever the lost eye reads), so with
+set 2 that sample is dropped, as is one where the angle between the eyes jumps more than 1.5
+degrees from its median.
+
 Lessons come back from the helper: when you nudge the gaze-placed pointer with the mouse and
 click, it sends "lesson <raw yaw> <raw pitch> <true yaw> <true pitch>": where the raw gaze
 was when the mouse took over, and where the pointer was when you clicked (you were looking
-there). The gap is the tracker's error there, and it's learned, unless it's more than
-LESSON_MAX degrees past the correction (then it wasn't a nudge onto what you looked at).
+there). The gap is the tracker's error there. The raw gaze is the one sent, so it also says
+when that look was (the history of what was sent), and so what each eye read then:
+  - SteamVR: each eye learns its own error, unless the gaze was more than
+    POINTER_GAZE_NUDGE_MAX degrees (frametop.conf, 55 by default: the helper's limit too)
+    past the correction (then it wasn't a nudge onto what you looked at);
+  - our tracker: unless it was more than POINTER_GAZE_NUDGE_MAX off, the look goes to it as
+    a click ("click T YAW PITCH" on @ft_eyes), as the
+    probe's clicks do, and it learns how far the headset has moved on your face. That's
+    what it gets wrong, and after the headset was off, the first click resets it;
+  - either way, how far off each eye was (before the lesson taught it anything) goes to the
+    eye bias, for auto.
 
 SteamVR's eye tracking log is followed for the headset going on (its eye model starts over,
 and the error moves): lessons from before count less then, so the first few after it
 relearn the offset.
 
+Our own tracker (gaze/tracker/ft-eyes) runs here too, in the dev container, while
+GAZE_TRACKER=own and the gaze is in use (below), or the gaze probe asks for it ("eyes SECONDS",
+a lease the probe renews). It reads the eye-camera frames the root service frametop-eyegrab
+copies (gaze/tracker/install.sh), which copies them only while ft-eyes runs.
+
+Idle: ft-gaze and our own tracker run only while the gaze is in use: gaze mode on with someone
+wearing the headset (the pointer helper says both: "gaze ? headset" -> "ok on|off worn|away"), a
+check or calibration open or asked for, or a "wake" lease (the Gaze page of Frametop Input
+Settings renews one while it's open). IDLE_AFTER after the last use they stop, and the frame
+grabber goes idle with our tracker; with our tracker, that was over half a core with gaze mode
+off. A check asked for while idle waits for the tracker to start (at most WAKE_SETTLE). Our
+tracker's next click after it starts again re-seats it, as after the headset was off, so turning
+gaze mode on after a while opens the quick check.
+
+Checks and calibration (gaze/gazecheck.py): a one-dot quick check when the headset goes on or
+our tracker asks for a click, and the full calibration when gaze mode comes on without one,
+both in a panel fixed to the headset (gaze/panel/ft-gazepanel, which this service runs too).
+
 Nothing here writes to SteamVR, its eye tracker, or its files: ft-gaze reads the eye
 tracker's shared memory read-only.
 
 Control socket: abstract unix datagram "@ft_gazed":
   lesson <rhy> <rhp> <thy> <thp>   from the pointer helper (see above)
+  recheck <deg>                    from the pointer helper: a click's correction was past
+                                   POINTER_GAZE_NUDGE_MAX, so the quick check (gazecheck.py)
   status                           reply: one JSON object
   forget                           drop what the lessons taught (the calibration stays)
-  reload                           read calibration.json again
+  reload                           read calibration.json and the settings again
+  eyes <seconds>                   keep our own tracker running that much longer (at most 120),
+                                   whatever GAZE_TRACKER says: the probe's lease. Reply: "ok"
+  wake <seconds>                   keep the gaze running (not idle) that much longer (at most 120),
+                                   whatever gaze mode says: Input Settings' Gaze page. Reply: "ok"
+  quickcal                         the one-dot check now (the calibration if there's none)
+  calibrate                        the full calibration in the panel
+  fitcheck                         the headset fit check in the panel
+  calaccept | calquit              from the pointer helper while the panel is up: take this dot
+                                   now (a left click, Meta+J) | close it (a right click, Meta+K)
 
-Options: --source action|mmap1|mmap2 (default mmap1; set 2 was a little quieter in the probe,
-but loses the pointer whenever the tracker loses an eye), -v (a status line every
-5 s on stderr), --to NAME (send the gaze to the abstract socket @NAME instead of the pointer
-helper; for testing: a helper without gaze mode forwards what it doesn't know to its driver).
+Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever
+the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever
+the tracker loses an eye), -v (a status line every 5 s on stderr), --to NAME (send the gaze
+to the abstract socket @NAME instead of the pointer helper; for testing: a helper without
+gaze mode forwards what it doesn't know to its driver).
 """
 
 import argparse
 import json
+import math
 import os
 import selectors
 import signal
@@ -56,17 +122,39 @@ from collections import deque
 from pathlib import Path
 
 sys.path.insert(0, str(Path(__file__).resolve().parent))
-from gazecal import DEFAULT_MODEL, MODELS, STATE, Correction, Fixation, LiveCorrection, SteamEyeLog  # noqa: E402
+from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correction, EyeFallback,  # noqa: E402
+                     EyeWeights, Fixation, LiveCorrection, SteamEyeLog)
+from gazecheck import Checks  # noqa: E402
 
 REPO = Path(__file__).resolve().parents[1]
 HELPER = REPO / "gaze" / "build" / "ft-gaze"
 ME = "\0ft_gazed"
 POINTER = "\0ft_pointer_helper"
+EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes"  # our own tracker
+EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python"  # numpy, OpenCV (build.sh)
+EYES_SOCKET = "\0ft_eyes"                      # its control socket
+EYES_CAMS = Path("/dev/shm/frametop-eyes-cams")  # the frames it reads (frametop-eyegrab.service)
+EYEGRAB = (Path("/etc/frametop/ft-eyegrab"), Path("/etc/systemd/system/frametop-eyegrab.service"))  # tracker/install.sh
+CONF = Path.home() / ".config" / "frametop.conf"
 CALIBRATION = STATE / "calibration.json"
 LESSONS = STATE / "pointer-lessons.json"
 LESSON_LOG = STATE / "pointer-lessons.jsonl"
-LESSON_MAX = 8.0  # degrees past the correction
+SOURCES = ("action", "mmap1", "mmap2", "left", "right")  # the ones with a calibration here
+SIDES = ("left", "right")  # ft-gaze's order, and the sources for each eye alone
+TRACKERS = ("steam", "own")
+TRACKER_SETTINGS = ("auto",) + TRACKERS
+BIASES = ("auto", "left", "right")
+NUDGE_MAX = 55.0  # degrees: POINTER_GAZE_NUDGE_MAX's default, the largest lesson taken
+HISTORY = 12.0    # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later)
+LOOK = 0.3        # seconds of samples before that moment make the look (the probe's fixation)
 RETRY = 3.0       # seconds before starting ft-gaze again
+EYES_RETRY = 10.0  # seconds before starting ft-eyes again after it stopped on its own
+EYES_LEASE_MAX = 120.0
+SETTLE = 0.3      # seconds after an eye is found again before the fallback learns from it
+IDLE_AFTER = 30.0  # seconds after the gaze was last in use before ft-gaze and our tracker stop
+WAKE_SETTLE = 15.0  # seconds after waking that the tracker may take to start sending
+WAKE_MAX = 120.0
+KEYBOARD_PITCH = -20.0  # degrees: gaze under this, on no screen, is a look at the keyboard
 
 
 class PointerLessons(LiveCorrection):
@@ -85,85 +173,241 @@ def log(msg):
     print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
 
 
+def own_installed():
+    """Is our own tracker installed: its frame grabber, and ft-eyes' Python here?"""
+    return all(p.exists() for p in EYEGRAB) and EYES_PYTHON.exists()
+
+
+def read_settings():
+    """(tracker, GAZE_TRACKER, eye bias, nudge max) from frametop.conf, defaults for anything
+    missing or unknown. The tracker is "steam" or "own": auto picks ours when it's installed."""
+    conf = {}
+    try:
+        for line in CONF.read_text().splitlines():
+            line = line.split("#", 1)[0].strip()
+            if "=" in line:
+                k, v = line.split("=", 1)
+                conf[k.strip()] = v.strip().lower()
+    except OSError:
+        pass
+    setting = conf.get("GAZE_TRACKER", "auto")
+    setting = setting if setting in TRACKER_SETTINGS else "auto"
+    tracker = setting if setting != "auto" else "own" if own_installed() else "steam"
+    bias = conf.get("GAZE_EYE", "auto")
+    try:
+        nudge = min(max(float(conf.get("POINTER_GAZE_NUDGE_MAX", NUDGE_MAX)), 1.0), 110.0)  # the helper's range
+    except ValueError:
+        nudge = NUDGE_MAX
+    return tracker, setting, bias if bias in BIASES else "auto", nudge
+
+
+def mtime(path):
+    try:
+        return path.stat().st_mtime
+    except OSError:
+        return None
+
+
 class Service:
     def __init__(self, source, verbose, to=POINTER):
-        self.source, self.verbose, self.to = source, verbose, to
+        self.override, self.verbose, self.to = source, verbose, to
+        self.source = source or "mmap1"  # the older path's source
         STATE.mkdir(parents=True, exist_ok=True)
-        self.base = Correction()
+        self.tracker, self.tracker_setting, self.bias, self.nudge_max = read_settings()
+        self.conf_mtime = mtime(CONF)
+        self.models = {name: Correction() for name in SOURCES}
         self.mode = DEFAULT_MODEL
         self.cal_mtime = None
-        self.live = PointerLessons()
+        self.lives = {name: PointerLessons() for name in SOURCES}
+        self.weights = {t: EyeWeights(self.bias) for t in TRACKERS}
         self.dirty = False
         self.load_calibration()
         self.load_lessons()
         self.steam = SteamEyeLog()
         self.steam.poll()
-        self.live.wear_time = self.steam.worn()
-        self.live.refit(self.base, self.mode)
+        self.refit()
         self.fix = Fixation(radius=1.0)
         self.opens = (deque(maxlen=90), deque(maxlen=90))  # left, right
         self.vergence = deque(maxlen=90)
-        self.counts = {"samples": 0, "sent": 0, "blinks": 0, "one_eye": 0, "dropped": 0, "lessons_taken": 0, "refused": 0}
+        self.fallback = EyeFallback()
+        self.lost = [False, False]
+        self.bad_at = [0.0, 0.0]    # sample time an eye was last lost or closed
+        self.counts = {"samples": 0, "sent": 0, "blinks": 0, "one_eye": 0, "one_eye_used": 0, "lost_left": 0,
+                       "lost_right": 0, "looking_down": 0, "dropped": 0, "lessons_taken": 0, "refused": 0}
         self.last_sample = 0.0
         self.last = None
+        self.last_kind = None
+        # What was sent, for finding a lesson's look: (sample time, raw as sent, each eye's reading).
+        self.history = deque()
+        self.own = {}        # our tracker's last status reply
+        self.own_at = 0.0
+        self.eyes_proc = None  # ft-eyes, while it runs
+        self.eyes_until = 0.0  # the probe's lease (monotonic time)
+        self.eyes_restart_at = 0.0
+        self.awake = False      # ft-gaze (and our tracker) run: the gaze is in use (see the top)
+        self.idle_at = 0.0      # idle from then, unless it's in use again first
+        self.woke_at = 0.0
+        self.wake_until = 0.0   # a "wake" lease
 
         self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
         self.sock.bind(ME)
         self.out = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
+        # To our tracker, with an address of its own, so its replies don't land on @ft_gazed.
+        self.eyes_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
+        self.eyes_sock.bind("")
         self.sel = selectors.DefaultSelector()
         self.sel.register(self.sock, selectors.EVENT_READ, "control")
+        self.sel.register(self.eyes_sock, selectors.EVENT_READ, "own")
+        self.checks = Checks(self, self.sel)
         self.proc = None
         self.buf = b""
         self.restart_at = 0.0
         self.running = True
 
-    # --- Calibration and lessons ---
+    @property
+    def kind(self):
+        """"own" (our tracker), "eyes" (SteamVR's eyes, each corrected), or "source" (the
+        older path: one SteamVR source, corrected as a whole)."""
+        if self.override:
+            return "source"
+        if self.tracker == "own":
+            return "own"
+        return "eyes" if all(self.models[e].samples for e in SIDES) else "source"
+
+    # --- Settings, calibration and lessons ---
+
+    def load_settings(self):
+        self.conf_mtime = mtime(CONF)
+        tracker, self.tracker_setting, bias, self.nudge_max = read_settings()
+        if (tracker, bias) != (self.tracker, self.bias):
+            auto = ""
+            if self.tracker_setting == "auto":
+                auto = " (auto: ours is installed)" if tracker == "own" else " (auto: ours isn't installed)"
+            log(f"tracker {tracker}{auto}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
+            self.tracker, self.bias = tracker, bias
+            for w in self.weights.values():
+                w.bias = bias
+            self.fix.reset()
 
     def load_calibration(self):
         try:
-            mtime = CALIBRATION.stat().st_mtime
+            mt = CALIBRATION.stat().st_mtime
             d = json.loads(CALIBRATION.read_text())
         except (OSError, ValueError):
             return
-        self.cal_mtime = mtime
-        if self.source in d:
-            self.base.from_json(d[self.source])
+        self.cal_mtime = mt
+        for name in SOURCES:
+            if name in d:
+                self.models[name].from_json(d[name])
         mode = d.get("_meta", {}).get("model")
         self.mode = mode if mode in MODELS else DEFAULT_MODEL
-        log(f"calibration: {self.mode}, {self.base.samples} samples")
+        log(f"calibration: {self.mode}, " + ", ".join(f"{n} {self.models[n].samples}" for n in (self.source,) + SIDES)
+            + " samples")
 
     def load_lessons(self):
         try:
             d = json.loads(LESSONS.read_text())
-            if d.get("source") == self.source:
-                self.live.samples = d.get("samples", [])[-PointerLessons.KEEP:]
         except (OSError, ValueError):
-            pass
-        log(f"{len(self.live.samples)} lessons")
+            d = {}
+        # The first version kept one source's: {"source": NAME, "samples": [...]}.
+        sources = d.get("sources") or ({d["source"]: d.get("samples", [])} if "source" in d else {})
+        for name, samples in sources.items():
+            if name in self.lives:
+                self.lives[name].samples = samples[-PointerLessons.KEEP:]
+        for t, misses in (d.get("misses") or {}).items():
+            if t in self.weights:
+                self.weights[t] = EyeWeights(self.bias, misses)
+        log(", ".join(f"{n} {len(self.lives[n].samples)}" for n in (self.source,) + SIDES) + " lessons")
 
     def save_lessons(self):
         tmp = LESSONS.with_suffix(".tmp")
-        tmp.write_text(json.dumps({"source": self.source, "samples": self.live.samples}))
+        tmp.write_text(json.dumps({"version": 2, "sources": {n: lv.samples for n, lv in self.lives.items() if lv.samples},
+                                   "misses": {t: w.misses for t, w in self.weights.items()}}))
         tmp.replace(LESSONS)
         self.dirty = False
 
-    def correction(self, hy, hp):
-        by, bp = self.base.get(hy, hp, self.mode)
-        ly, lp = self.live.get(hy, hp)
+    def forget_lessons(self):
+        """Drop what the lessons taught (the calibration stays)."""
+        self.lives = {name: PointerLessons() for name in SOURCES}
+        self.weights = {t: EyeWeights(self.bias) for t in TRACKERS}
+        self.refit()
+        self.save_lessons()
+
+    def refit(self):
+        for name, live in self.lives.items():
+            live.wear_time = self.steam.worn()
+            live.refit(self.models[name], self.mode)
+
+    def correction(self, name, hy, hp):
+        by, bp = self.models[name].get(hy, hp, self.mode)
+        ly, lp = self.lives[name].get(hy, hp)
         return by + ly, bp + lp
 
+    def look(self, ry, rp):
+        """When the gaze sent as raw (ry, rp) was last sent, and each eye's median reading over
+        the LOOK before it: (t, [(yaw, pitch) or None] * 2), or (None, None)."""
+        key = f"{ry:.3f} {rp:.3f}"
+        t = next((h[0] for h in reversed(self.history) if h[1] == key), None)
+        if t is None:
+            return None, None
+        eyes = []
+        for k in (0, 1):
+            seen = [h[2][k] for h in self.history if t - LOOK <= h[0] <= t and h[2] and h[2][k]]
+            eyes.append((statistics.median(e[0] for e in seen), statistics.median(e[1] for e in seen)) if seen else None)
+        return t, eyes
+
     def lesson(self, rhy, rhp, thy, thp):
-        dy, dp = thy - rhy, thp - rhp  # the whole error there
-        cy, cp = self.correction(rhy, rhp)
-        left = ((dy - cy) ** 2 + (dp - cp) ** 2) ** 0.5
-        rec = {"time": time.time(), "source": self.source, "model": self.mode, "raw": [rhy, rhp],
-               "true": [thy, thp], "correction": [cy, cp], "lesson_deg": left, "wear": self.steam.worn()}
-        if left > LESSON_MAX:
-            rec["refused"] = f"more than {LESSON_MAX} deg past the correction"
+        kind = self.kind
+        rec = {"time": time.time(), "kind": kind, "raw": [rhy, rhp], "true": [thy, thp], "wear": self.steam.worn()}
+        if kind == "source":
+            dy, dp = thy - rhy, thp - rhp  # the whole error there
+            cy, cp = self.correction(self.source, rhy, rhp)
+            left = math.hypot(dy - cy, dp - cp)
+            rec.update(source=self.source, model=self.mode, correction=[cy, cp], lesson_deg=left)
+            if left > self.nudge_max:
+                rec["refused"] = f"more than {self.nudge_max:g} deg past the correction"
+            else:
+                self.lives[self.source].add({"time": rec["time"], "hy": rhy, "hp": rhp, "dy": dy, "dp": dp,
+                                             "wy": 1.0, "wp": 1.0, "how": "pointer"}, self.models[self.source], self.mode)
+            return self.taken(rec)
+        # The raw gaze sent here is the corrected, combined one: its whole error is left.
+        left = math.hypot(thy - rhy, thp - rhp)
+        t, eyes = self.look(rhy, rhp)
+        weights = self.weights[self.tracker if kind == "own" else "steam"]
+        rec.update(tracker=self.tracker if kind == "own" else "steam", bias=self.bias, lesson_deg=left, look_t=t,
+                   eyes=eyes, weights=[round(w, 3) for w in weights.weights()])
+        if t is None:
+            rec["refused"] = "that gaze isn't in the last few seconds sent"
+        elif left > self.nudge_max:
+            rec["refused"] = f"more than {self.nudge_max:g} deg off"
+        if "refused" in rec:
+            return self.taken(rec)
+        if kind == "own":
+            # Its eyes come calibrated: how far off each was is its miss. The click goes to it.
+            miss = [math.hypot(thy - e[0], thp - e[1]) if e else None for e in eyes]
+            try:
+                self.eyes_sock.sendto(f"click {t:.6f} {thy:.4f} {thp:.4f}".encode(), EYES_SOCKET)
+            except OSError as e:
+                rec["refused"] = f"our tracker isn't running ({e})"
+                return self.taken(rec)
+        else:
+            miss = []
+            for name, e in zip(SIDES, eyes):
+                if not e:
+                    miss.append(None)
+                    continue
+                cy, cp = self.correction(name, *e)
+                miss.append(math.hypot(thy - e[0] - cy, thp - e[1] - cp))
+                self.lives[name].add({"time": rec["time"], "hy": e[0], "hp": e[1], "dy": thy - e[0], "dp": thp - e[1],
+                                      "wy": 1.0, "wp": 1.0, "how": "pointer"}, self.models[name], self.mode)
+        rec["miss"] = miss
+        weights.add(miss)
+        return self.taken(rec)
+
+    def taken(self, rec):
+        if "refused" in rec:
             self.counts["refused"] += 1
         else:
-            self.live.add({"time": rec["time"], "hy": rhy, "hp": rhp, "dy": dy, "dp": dp, "wy": 1.0, "wp": 1.0,
-                           "how": "pointer"}, self.base, self.mode)
             self.counts["lessons_taken"] += 1
             self.dirty = True
         try:
@@ -173,6 +417,46 @@ class Service:
             log(f"lesson log: {e}")
         return rec
 
+    # --- Idle (see the top) ---
+
+    def use_reason(self):
+        """Why the gaze is in use now, or None."""
+        c = self.checks
+        if c.gaze_on and c.headset is not False:
+            return "gaze mode is on"
+        if c.active or c.pending:
+            return "a check"
+        if time.monotonic() < self.wake_until:
+            return "asked to stay awake"
+        return None
+
+    def idle_reason(self):
+        c = self.checks
+        if c.gaze_on is None:
+            return "the pointer helper isn't answering (SteamVR not running?)"
+        if c.gaze_on and c.headset is False:
+            return "nobody is wearing the headset"
+        return "gaze mode is off"
+
+    def waking(self):
+        """Idle, or awake too briefly for the tracker to be sending yet."""
+        return not self.awake or time.monotonic() - self.woke_at < WAKE_SETTLE
+
+    def update_awake(self):
+        now = time.monotonic()
+        why = self.use_reason()
+        if why:
+            self.idle_at = now + IDLE_AFTER
+        if why and not self.awake:
+            self.awake, self.woke_at, self.restart_at = True, now, 0.0
+            log(f"awake: {why}")
+        elif not why and self.awake and now >= self.idle_at:
+            self.awake = False
+            log(f"idle: {self.idle_reason()}; ft-gaze and our tracker stop until the gaze is used again")
+            self.stop_helper()
+            if self.eyes_proc and not self.eyes_wanted():
+                self.stop_eyes()
+
     # --- ft-gaze ---
 
     def start_helper(self):
@@ -214,6 +498,61 @@ class Service:
                 pass
         self.proc = None
 
+    def eyes_wanted(self):
+        return (self.tracker == "own" and not self.override and self.awake) or time.monotonic() < self.eyes_until
+
+    def start_eyes(self):
+        """Our own tracker, in the dev container, with build/venv's numpy and OpenCV. Like
+        ft-gaze, it quits when its stdin closes."""
+        if not EYES_PYTHON.exists():
+            log(f"ft-eyes isn't built: run {REPO}/gaze/tracker/build.sh")
+            self.eyes_restart_at = time.monotonic() + 30
+            return
+        env = dict(os.environ)
+        env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
+        subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
+        distrobox = Path.home() / ".local" / "bin" / "distrobox"
+        self.eyes_proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(EYES_PYTHON), str(EYES_PROG), "-v",
+                                           "--watch-stdin"], env=env, stdin=subprocess.PIPE,
+                                          stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, start_new_session=True)
+        os.set_blocking(self.eyes_proc.stderr.fileno(), False)
+        self.sel.register(self.eyes_proc.stderr, selectors.EVENT_READ, "eyes")
+        log("ft-eyes started" + ("" if EYES_CAMS.exists() else
+                                 f": no {EYES_CAMS} yet (the frame grabber: gaze/tracker/install.sh)"))
+
+    def stop_eyes(self):
+        if not self.eyes_proc:
+            return
+        try:
+            self.sel.unregister(self.eyes_proc.stderr)
+        except (KeyError, ValueError):
+            pass
+        if self.eyes_proc.stdin and not self.eyes_proc.stdin.closed:
+            self.eyes_proc.stdin.close()
+        try:
+            self.eyes_proc.wait(timeout=3)
+        except subprocess.TimeoutExpired:
+            try:
+                os.killpg(self.eyes_proc.pid, signal.SIGTERM)
+            except ProcessLookupError:
+                pass
+        self.eyes_proc = None
+        self.own = {}
+
+    def read_eyes(self):
+        try:
+            data = os.read(self.eyes_proc.stderr.fileno(), 65536)
+        except BlockingIOError:
+            return
+        if not data:
+            log(f"ft-eyes stopped (exit {self.eyes_proc.poll()}); again in {EYES_RETRY:.0f} s if still wanted")
+            self.stop_eyes()
+            self.eyes_restart_at = time.monotonic() + EYES_RETRY
+            return
+        for line in data.decode("utf-8", "replace").splitlines():
+            if line.strip() and (self.verbose or "fps" not in line):
+                log(line)
+
     def read_stdout(self):
         try:
             data = os.read(self.proc.stdout.fileno(), 65536)
@@ -241,19 +580,13 @@ class Service:
             if line.strip():
                 log(line)
 
-    def on_sample(self, s):
-        src = s["src"].get(self.source) or {}
-        if "hy" not in src:
-            return
-        self.counts["samples"] += 1
-        self.last_sample = time.monotonic()
-        m1 = s["src"].get("mmap1") or {}
-        o = m1.get("open")
-        lr = src.get("lr", m1.get("lr"))
-        # Blinks and lost eyes, judged against the last second (see steady_samples: relative,
-        # because the lids come down looking down, and the vergence depends on distance).
-        # An eye's floor comes from its good readings, so a lost eye doesn't drag it to 0.
+    def judge_eyes(self, m1, down):
+        """Which eyes (left, right) are closed, from SteamVR's openness (set 1); updates
+        self.lost from its variances. Blinks and lost eyes are judged against the last second
+        (see steady_samples: relative, because the lids come down looking down). An eye's
+        floor comes from its good readings, so a lost eye doesn't drag it to 0."""
         low = [False, False]
+        o = m1.get("open")
         if o and len(o) == 2:
             for k in (0, 1):
                 hist = self.opens[k]
@@ -261,25 +594,131 @@ class Service:
                 floor = max(0.12, 0.5 * statistics.median(good)) if len(good) >= 30 else 0.12
                 low[k] = o[k] < floor
                 hist.append(o[k])
-        if all(low):
+        unc = m1.get("unc")
+        if unc and len(unc) == 2:
+            for k in (0, 1):
+                self.lost[k] = unc[k] > (EYE_FOUND if self.lost[k] else EYE_LOST)
+            if not down:
+                self.counts["lost_left"] += self.lost[0]
+                self.counts["lost_right"] += self.lost[1]
+        return low
+
+    def on_sample(self, s):
+        self.checks.on_sample(s)
+        kind = self.kind
+        if kind != self.last_kind:
+            log({"own": "our own tracker", "eyes": "SteamVR's eyes, each calibrated",
+                 "source": f"SteamVR's {self.source}, calibrated as a whole"}[kind]
+                + (f", eye bias {self.bias}" if kind != "source" else ""))
+            self.last_kind = kind
+            self.fix.reset()
+        if kind == "source":
+            self.on_source_sample(s)
+        else:
+            self.on_eyes_sample(s, kind == "own")
+
+    def on_eyes_sample(self, s, own):
+        m1 = s["src"].get("mmap1") or {}
+        if own:
+            src = s["src"].get("own") or {}
+            if "hy" not in src:
+                return
+            eyes = [tuple(e) if e else None for e in (src.get("eyes") or [None, None])]
+            hp, hit = src["hp"], src.get("hit")
+        else:
+            per = [s["src"].get(name) or {} for name in SIDES]
+            eyes = [(p["hy"], p["hp"]) if "hy" in p else None for p in per]
+            if not any(eyes):
+                return
+            hp, hit = next(e[1] for e in eyes if e), m1.get("hit")
+        self.counts["samples"] += 1
+        self.last_sample = time.monotonic()
+        down = hp < KEYBOARD_PITCH and not hit
+        low = self.judge_eyes(m1, down)
+        if down:
+            self.counts["looking_down"] += 1
+            return
+        # Our tracker finds the pupils itself; SteamVR's openness still marks the blinks.
+        bad = [eyes[k] is None or low[k] or (not own and self.lost[k]) for k in (0, 1)]
+        if all(bad):
             self.counts["blinks"] += 1
             return
-        if any(low):
+        if any(bad):
             self.counts["one_eye"] += 1
-        if self.source == "mmap2":
-            jump = (lr is not None and len(self.vergence) >= 30
-                    and abs(lr - statistics.median(self.vergence)) > 1.5)
-            if lr is not None and not any(low):
-                self.vergence.append(lr)
-            if any(low) or jump:
+            self.counts["one_eye_used"] += 1
+        seen = [None if bad[k] else eyes[k] for k in (0, 1)]
+        if own:
+            corrected = seen
+        else:
+            corrected = []
+            for name, e in zip(SIDES, seen):
+                c = self.correction(name, *e) if e else None
+                corrected.append((e[0] + c[0], e[1] + c[1]) if e else None)
+        gy, gp = self.weights["own" if own else "steam"].combine(corrected)
+        fy, fp = self.fix(gy, gp, s["t"], 1.0)
+        self.send(s["t"], fy, fp, fy, fp, seen)
+
+    def on_source_sample(self, s):
+        src = s["src"].get(self.source) or {}
+        if "hy" not in src:
+            return
+        self.counts["samples"] += 1
+        self.last_sample = time.monotonic()
+        m1 = s["src"].get("mmap1") or {}
+        lr = src.get("lr", m1.get("lr"))
+        down = src["hp"] < KEYBOARD_PITCH and not src.get("hit")
+        low = self.judge_eyes(m1, down)
+        if down:
+            self.counts["looking_down"] += 1
+            for k in (0, 1):
+                self.bad_at[k] = s["t"]  # the fallback doesn't learn from these either
+            return
+        bad = [low[k] or self.lost[k] for k in (0, 1)]
+        if all(bad):
+            self.counts["blinks"] += 1
+            return
+        hy, hp = src["hy"], src["hp"]
+        eyes = (s["src"].get("mmap2") or {}).get("eyes")
+        for k in (0, 1):
+            if bad[k]:
+                self.bad_at[k] = s["t"]
+        if any(bad):
+            self.counts["one_eye"] += 1
+            seen = 1 if bad[0] else 0
+            est = self.fallback.get(seen, eyes[seen][0], eyes[seen][1]) if eyes else None
+            if est:
+                hy, hp = est
+                self.counts["one_eye_used"] += 1
+            elif self.source == "mmap2":
                 self.counts["dropped"] += 1
                 return
+        else:
+            if self.source == "mmap2":
+                jump = (lr is not None and len(self.vergence) >= 30
+                        and abs(lr - statistics.median(self.vergence)) > 1.5)
+                if lr is not None:
+                    self.vergence.append(lr)
+                if jump:
+                    self.counts["dropped"] += 1
+                    return
+            # Learn only once both have been seen for a moment: the tracker's filter starts
+            # an eye over when it finds it again.
+            if eyes and s["t"] - max(self.bad_at) > SETTLE:
+                for k in (0, 1):
+                    self.fallback.update(k, eyes[k][0], eyes[k][1], hy, hp)
         # The fixation lock works in degrees here (1 degree per "pixel").
-        fy, fp = self.fix(src["hy"], src["hp"], s["t"], 1.0)
-        cy, cp = self.correction(fy, fp)
-        self.last = (fy + cy, fp + cp, fy, fp)
+        fy, fp = self.fix(hy, hp, s["t"], 1.0)
+        cy, cp = self.correction(self.source, fy, fp)
+        self.send(s["t"], fy + cy, fp + cp, fy, fp, None)
+
+    def send(self, t, hy, hp, rhy, rhp, eyes):
+        self.last = (hy, hp, rhy, rhp)
+        raw = f"{rhy:.3f} {rhp:.3f}"
+        self.history.append((t, raw, eyes))
+        while self.history and self.history[0][0] < t - HISTORY:
+            self.history.popleft()
         try:
-            self.out.sendto(f"gz {fy + cy:.3f} {fp + cp:.3f} {fy:.3f} {fp:.3f}".encode(), self.to)
+            self.out.sendto(f"gz {hy:.3f} {hp:.3f} {raw}".encode(), self.to)
             self.counts["sent"] += 1
         except OSError:
             pass  # the pointer helper isn't running
@@ -297,21 +736,40 @@ class Service:
             if words[:1] == ["lesson"] and len(words) == 5:
                 try:
                     rec = self.lesson(*map(float, words[1:]))
+                    self.checks.after_lesson(rec)
                     reply = "refused" if "refused" in rec else f"ok {rec['lesson_deg']:.2f}"
                     log(f"lesson {rec['lesson_deg']:.2f} deg at {rec['raw'][0]:+.1f},{rec['raw'][1]:+.1f}"
+                        + (f", eyes off {', '.join('-' if m is None else f'{m:.2f}' for m in rec['miss'])}"
+                           if rec.get("miss") else "")
                         + (f": {rec['refused']}" if "refused" in rec else ""))
                 except ValueError:
                     reply = "error bad lesson"
             elif words[:1] == ["status"]:
                 reply = json.dumps(self.status())
             elif words[:1] == ["forget"]:
-                self.live = PointerLessons()
-                self.live.wear_time = self.steam.worn()
-                self.save_lessons()
+                self.forget_lessons()
                 reply = "ok"
+            elif words[:1] and words[0] in ("quickcal", "calibrate", "fitcheck", "calaccept", "calquit", "recheck"):
+                reply = self.checks.command(words)
+            elif words[:1] == ["eyes"] and len(words) == 2:
+                try:
+                    secs = min(max(float(words[1]), 0.0), EYES_LEASE_MAX)
+                    self.eyes_until = max(self.eyes_until, time.monotonic() + secs)
+                    reply = "ok"
+                except ValueError:
+                    reply = "error bad seconds"
+            elif words[:1] == ["wake"] and len(words) == 2:
+                try:
+                    secs = min(max(float(words[1]), 0.0), WAKE_MAX)
+                    self.wake_until = max(self.wake_until, time.monotonic() + secs)
+                    self.update_awake()
+                    reply = "ok"
+                except ValueError:
+                    reply = "error bad seconds"
             elif words[:1] == ["reload"]:
+                self.load_settings()
                 self.load_calibration()
-                self.live.refit(self.base, self.mode)
+                self.refit()
                 reply = "ok"
             else:
                 reply = "error unknown command"
@@ -321,27 +779,77 @@ class Service:
                 except OSError:
                     pass
 
+    def on_own(self):
+        """Replies from our tracker: its status (JSON), or a click's "ok ..."/"fail ..."."""
+        while True:
+            try:
+                data = self.eyes_sock.recv(4096).decode("utf-8", "replace")
+            except (BlockingIOError, OSError):
+                return
+            if data.startswith("{"):
+                try:
+                    self.own, self.own_at = json.loads(data), time.monotonic()
+                except ValueError:
+                    pass
+            else:
+                log(f"our tracker: {data}")
+
     def status(self):
-        ly, lp = self.live.offset()
-        return {"source": self.source, "model": self.mode, "calibration_samples": self.base.samples,
-                "lessons": len(self.live.samples), "lesson_offset": [round(ly, 3), round(lp, 3)],
-                "ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
-                if self.last_sample else None, "headset_on": self.steam.wearing(), "headset_on_since": self.steam.worn(),
-                "last": [round(v, 2) for v in self.last] if self.last else None, **self.counts}
+        kind = self.kind
+        tracker = "own" if kind == "own" else "steam"
+        w = self.weights[tracker]
+        st = {"tracker": tracker, "kind": kind, "source": "own" if kind == "own" else self.source if kind == "source"
+              else "left+right", "model": self.mode, "eye_bias": self.bias}
+        if kind == "source":
+            ly, lp = self.lives[self.source].offset()
+            st.update(calibration_samples=self.models[self.source].samples, lessons=len(self.lives[self.source].samples),
+                      lesson_offset=[round(ly, 3), round(lp, 3)])
+        else:
+            st.update(eye_weights=[round(v, 3) for v in w.weights()], eye_misses=[len(m) for m in w.misses],
+                      eye_rms=[None if r is None else round(r, 2) for r in w.rms()])
+        if kind == "eyes":
+            st.update(calibration_samples=min(self.models[e].samples for e in SIDES),
+                      lessons=max(len(self.lives[e].samples) for e in SIDES))
+        if kind == "own":
+            own = self.own if time.monotonic() - self.own_at < 5 else {}
+            cal = own.get("calibration") or {}
+            st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
+                      lessons=max(len(m) for m in w.misses), own_running=bool(own),
+                      own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
+        st.update(awake=self.awake, idle=None if self.awake else self.idle_reason())
+        st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists(),
+                  tracker_setting=self.tracker_setting, own_installed=own_installed())
+        st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
+                   if self.last_sample else None, "headset_on": self.steam.wearing(),
+                   "headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None,
+                   "eyes_lost": self.lost, "fallback_ready": [self.fallback.ready(0), self.fallback.ready(1)],
+                   "checks": self.checks.status(), **self.counts})
+        return st
 
     def periodic(self):
-        if self.steam.poll() or self.steam.worn() != self.live.wear_time:
-            if self.steam.worn() != self.live.wear_time:
+        if self.steam.poll() or any(lv.wear_time != self.steam.worn() for lv in self.lives.values()):
+            if any(lv.wear_time != self.steam.worn() for lv in self.lives.values()):
                 log("headset on again: older lessons count less until new ones come in")
-            self.live.wear_time = self.steam.worn()
-            self.live.refit(self.base, self.mode)
-        try:
-            mtime = CALIBRATION.stat().st_mtime
-        except OSError:
-            mtime = None
-        if mtime != self.cal_mtime:
+            self.refit()
+        if mtime(CALIBRATION) != self.cal_mtime:
             self.load_calibration()
-            self.live.refit(self.base, self.mode)
+            self.refit()
+        if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
+                                              and own_installed() != (self.tracker == "own")):
+            self.load_settings()
+        self.update_awake()
+        want = self.eyes_wanted()
+        if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:
+            self.start_eyes()
+        elif not want and self.eyes_proc:
+            log("ft-eyes no longer wanted: stopping it")
+            self.stop_eyes()
+        if self.eyes_proc:
+            try:
+                self.eyes_sock.sendto(b"status", EYES_SOCKET)
+            except OSError:
+                pass  # not up yet: status() says so once the last answer is old
+        self.checks.periodic()
         if self.dirty:
             self.save_lessons()
 
@@ -350,29 +858,41 @@ class Service:
         next_verbose = time.monotonic() + 5
         while self.running:
             now = time.monotonic()
-            if not self.proc and now >= self.restart_at:
+            if self.awake and not self.proc and now >= self.restart_at:
                 self.start_helper()
-            for key, _ in self.sel.select(timeout=0.5):
+            for key, _ in self.sel.select(timeout=0.05 if self.checks.active else 0.5):
                 if key.data == "control":
                     self.on_control()
+                elif key.data == "checks":
+                    self.checks.on_readable()
+                elif key.data == "panel" and self.checks.panel_proc:
+                    self.checks.read_panel()
+                elif key.data == "own":
+                    self.on_own()
+                elif key.data == "eyes" and self.eyes_proc:
+                    self.read_eyes()
                 elif key.data == "stdout" and self.proc:
                     self.read_stdout()
                 elif key.data == "stderr" and self.proc:
                     self.read_stderr()
+            self.checks.tick()
             if now >= next_periodic:
                 self.periodic()
                 next_periodic = now + 1.0
             if self.verbose and now >= next_verbose:
                 log(json.dumps(self.status()))
                 next_verbose = now + 5
+        self.checks.stop()
         self.stop_helper()
+        self.stop_eyes()
         if self.dirty:
             self.save_lessons()
 
 
 def main():
     ap = argparse.ArgumentParser(description="The gaze service: corrected eye tracking for the pointer")
-    ap.add_argument("--source", choices=["action", "mmap1", "mmap2"], default="mmap1")
+    ap.add_argument("--source", choices=["action", "mmap1", "mmap2"],
+                    help="the older one-source path with this SteamVR source, whatever the settings say")
     ap.add_argument("-v", "--verbose", action="store_true")
     ap.add_argument("--to", default="ft_pointer_helper", help="abstract socket to send the gaze to")
     args = ap.parse_args()

gaze/gazecal.py

@@ -2,8 +2,8 @@
 
 The correction models (Correction: the calibration fitted from calibration dots;
 LiveCorrection: what clicks teach on the fly, on top of it), the smoothing filters, the
-blink and dropout filter for one look at a spot, and SteamEyeLog, which follows SteamVR's
-eye tracking log. Angles are head-relative degrees (yaw +left, pitch +up), as ft-gaze
+blink and dropout filter for one look at a spot, EyeFallback (the gaze from one eye while
+the tracker has lost the other), EyeWeights (how much each eye counts), and SteamEyeLog, which follows SteamVR's eye tracking log. Angles are head-relative degrees (yaw +left, pitch +up), as ft-gaze
 reports them.
 """
 
@@ -391,6 +391,146 @@ class LiveCorrection:
         return self.cy[0], self.cp[0]
 
 
+# The tracker's variance for an eye's direction (ft-gaze's "unc"): 0.0005-0.002 while it
+# sees the eye, 0.015-0.03 once it's lost it, falling back through 0.008-0.002 in the 0.1 s
+# after it finds it again.
+EYE_LOST = 0.004
+EYE_FOUND = 0.0025
+
+
+class EyeFallback:
+    """The gaze from one eye, while the tracker has lost the other.
+
+    SteamVR's combined gaze (mmap set 1) keeps going with one eye lost, but badly: it holds
+    the lost eye's yaw where it was and gives it the other eye's pitch, so the gaze moves
+    half as far sideways as the eyes do (seen: the right eye swung 5 degrees, the combined
+    gaze 2.5). Set 2's eyes are each eye's own reading. While both are seen, this learns what
+    each eye reads against the combined gaze (an offset: half the angle between the eyes,
+    plus how differently the tracker reads each), in 10 degree cells of where that eye
+    looks, blended over the four nearest; while one is lost, the other eye plus its offset
+    stands in for the combined gaze. So the rest (fixation lock, calibration, lessons)
+    carries on as if nothing happened.
+
+    On a recording, one eye alone came out 1.1 degrees (median) from both eyes' gaze, 0.8
+    over a tenth of a second of a steady look, and a little more jittery (0.31-0.37 degrees
+    against 0.28). Carrying on the offset from just before a loss did no better: what's
+    left is fast noise, not something particular to that look.
+
+    `update` and `get` take head-relative degrees (yaw, pitch)."""
+
+    CELL = 10.0
+    GLOBAL_RATE = 0.01   # per sample: about a second at 90 Hz
+    CELL_RATE = 0.02     # the least a cell learns per sample, once it has CELL_FULL
+    CELL_FULL = 30       # samples before a cell counts fully
+    READY = 45           # samples of both eyes before an eye can stand in
+
+    def __init__(self):
+        self.glob = [None, None]      # per eye: [oy, op]
+        self.seen = [0, 0]
+        self.cells = [{}, {}]         # per eye: (i, j) -> [oy, op, n]
+
+    def ready(self, eye):
+        return self.seen[eye] >= self.READY
+
+    def update(self, eye, ey, ep, cy, cp):
+        oy, op = cy - ey, cp - ep
+        g = self.glob[eye]
+        if g is None:
+            self.glob[eye] = [oy, op]
+        else:
+            g[0] += self.GLOBAL_RATE * (oy - g[0])
+            g[1] += self.GLOBAL_RATE * (op - g[1])
+        self.seen[eye] += 1
+        key = (math.floor(ey / self.CELL), math.floor(ep / self.CELL))
+        c = self.cells[eye].setdefault(key, [oy, op, 0])
+        c[2] += 1
+        a = max(1.0 / c[2], self.CELL_RATE)
+        c[0] += a * (oy - c[0])
+        c[1] += a * (op - c[1])
+
+    def offset(self, eye, ey, ep):
+        g = self.glob[eye]
+        if g is None:
+            return None
+        # Bilinear over the four cells whose centres surround the point.
+        fy, fp = ey / self.CELL - 0.5, ep / self.CELL - 0.5
+        i0, j0 = math.floor(fy), math.floor(fp)
+        ty, tp = fy - i0, fp - j0
+        sy = sp = used = 0.0
+        for di, wi in ((0, 1 - ty), (1, ty)):
+            for dj, wj in ((0, 1 - tp), (1, tp)):
+                c = self.cells[eye].get((i0 + di, j0 + dj))
+                if c:
+                    w = wi * wj * min(1.0, c[2] / self.CELL_FULL)
+                    sy += w * c[0]
+                    sp += w * c[1]
+                    used += w
+        return sy + (1 - used) * g[0], sp + (1 - used) * g[1]
+
+    def get(self, eye, ey, ep):
+        """The combined gaze from this eye's reading, or None before it has learned enough."""
+        if not self.ready(eye):
+            return None
+        oy, op = self.offset(eye, ey, ep)
+        return ey + oy, ep + op
+
+
+class EyeWeights:
+    """How much each eye (0 left, 1 right) counts in the gaze, for ft-gazed's eye bias.
+
+    Two eyes beat either one: their errors partly cancel. On 306 live clicks with our own
+    tracker (gaze/tracker, 2026-09-29) the eyes' sideways errors were correlated -0.37, and
+    the mean of both was 0.65 degrees off (median), the left eye alone 0.96, the right 1.11.
+    So a bias leans instead of choosing: "left" or "right" counts that eye LEAN times the
+    other (on those clicks, 2:1 toward the better eye cost about 0.03 degrees, toward the
+    worse one about 0.13). "auto" weights each
+    by the inverse square of its RMS miss at the last KEEP lessons, once both have MIN, and
+    alike until then. Each miss is measured before its lesson teaches anything, so each is a
+    fresh test. On SteamVR's own test (2026-09-29) its calibration dots said the left eye was
+    the better one and new spots said the right, so the misses come from lessons, not the fit.
+    An eye that isn't seen (None) drops out, and the other carries the gaze alone."""
+
+    LEAN = 2.0
+    KEEP = 20
+    MIN = 5
+    FLOOR = 0.3   # degrees: so one lucky run can't give an eye all the weight
+    STALE = 8.0   # degrees: a miss this big is the headset moved, not the eye's accuracy
+
+    def __init__(self, bias="auto", misses=None):
+        self.bias = bias
+        self.misses = [list(m) for m in (misses or ([], []))]
+
+    def add(self, miss):
+        """One lesson's miss per eye (degrees, None where it wasn't seen)."""
+        if any(m is not None and m > self.STALE for m in miss):
+            return
+        for k, m in enumerate(miss):
+            if m is not None:
+                self.misses[k] = (self.misses[k] + [m])[-self.KEEP:]
+
+    def rms(self):
+        return [math.sqrt(sum(m * m for m in ms) / len(ms)) if ms else None for ms in self.misses]
+
+    def weights(self):
+        """(left, right), summing to 1."""
+        if self.bias in ("left", "right"):
+            w = [self.LEAN, 1.0] if self.bias == "left" else [1.0, self.LEAN]
+        elif all(len(ms) >= self.MIN for ms in self.misses):
+            w = [1.0 / max(r, self.FLOOR) ** 2 for r in self.rms()]
+        else:
+            w = [1.0, 1.0]
+        return w[0] / sum(w), w[1] / sum(w)
+
+    def combine(self, eyes):
+        """The weighted gaze from [(yaw, pitch) or None, (yaw, pitch) or None], or None."""
+        w = [wk for wk, e in zip(self.weights(), eyes) if e is not None]
+        seen = [e for e in eyes if e is not None]
+        if not seen:
+            return None
+        total = sum(w)
+        return (sum(wk * e[0] for wk, e in zip(w, seen)) / total, sum(wk * e[1] for wk, e in zip(w, seen)) / total)
+
+
 class SteamEyeLog:
     """Follows SteamVR's eye tracking log (read only) for what moves the raw gaze under a
     calibration.
@@ -404,6 +544,9 @@ class SteamEyeLog:
     each time the headset goes on ("HMD on"): the eye model starts over then too."""
 
     PATH = Path.home() / ".local" / "share" / "Steam" / "logs" / "eyetracking.txt"
+    # It writes "HMD on" again every minute or so while on, and flickers off for 0.01-0.3 s:
+    # only an on after an off of BLIP or longer counts.
+    BLIP = 1.5
 
     def __init__(self):
         self.pos = 0
@@ -455,9 +598,14 @@ class SteamEyeLog:
                 self.starts.append(t)
                 restarted = not first
             elif "HMD on" in line:
-                self.wears.append(t)
+                off = bool(self.offs) and (not self.wears or self.offs[-1] > self.wears[-1])
+                if off and self.wears and t - self.offs[-1] < self.BLIP:
+                    self.offs.pop()  # the sensor flickering: it never came off
+                elif off or not self.wears:
+                    self.wears.append(t)
             elif "HMD off" in line:
-                self.offs.append(t)
+                if not self.offs or (self.wears and self.wears[-1] > self.offs[-1]):
+                    self.offs.append(t)
             elif "Accept usercal" in line:
                 self.accepts.append(t)
             elif "Reject usercal" in line:
@@ -492,29 +640,44 @@ def cross_validate(points, mode):
     return errs
 
 
-def steady_samples(samples, vergence_jump=1.5):
+def steady_samples(samples, vergence_jump=1.5, why=None):
     """The samples of one look at one spot where the tracker had both eyes: none in a blink
-    (openness under half its median over the samples), and none where the angle between the eyes' directions (`lr`, the
+    (openness under half its median over the samples), none where it had lost an eye (its
+    variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the
     vergence) is more than `vergence_jump` degrees from its median over the samples. The
     vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
     fraction of one far off), so only a jump away from what it was during this look means
-    the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept."""
+    the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.
+    `why`, a dict, gets how many were dropped for each reason: "lost_left", "lost_right",
+    "lost_both", "blink" and "vergence" (each sample once, for the first that applies)."""
+    if why is None:
+        why = {}
     # Openness: a blink is a sharp drop from what it was during this look. Not a fixed
     # level: looking down, the upper lids come down with the eyes, and in bright light you
     # squint, so the reading can stay under 0.5 for the whole look while the tracker follows
     # the eyes fine (a calibration dot at the bottom of the bright round failed that way).
     opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
     floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12
-    opened = []
+    seen = []
     for smp in samples:
-        o = (smp["src"].get("mmap1") or {}).get("open")
-        if not o or min(o) >= floor:
-            opened.append(smp)
+        m1 = smp["src"].get("mmap1") or {}
+        o = m1.get("open")
+        lost = [u > EYE_LOST for u in m1.get("unc") or [0, 0]]
+        # A lost eye's openness reads 0 too, so a lost eye is named before a blink.
+        key = ("lost_both" if all(lost) else "lost_left" if lost[0] else "lost_right") if any(lost) else \
+            "blink" if o and min(o) < floor else None
+        if key:
+            why[key] = why.get(key, 0) + 1
+        else:
+            seen.append(smp)
 
     def vergence(smp):
         return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
-    have = [v for v in map(vergence, opened) if v is not None]
+    have = [v for v in map(vergence, seen) if v is not None]
     if len(have) < 5:
-        return opened
+        return seen
     med = statistics.median(have)
-    return [smp for smp in opened if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
+    kept = [smp for smp in seen if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
+    if len(kept) < len(seen):
+        why["vergence"] = why.get("vergence", 0) + len(seen) - len(kept)
+    return kept

gaze/gazecheck.py

@@ -0,0 +1,896 @@
+"""gazecheck: the gaze service's checks and calibration, in the panel fixed to the headset
+(gaze/panel/ft-gazepanel; ft-gazed runs it). Every kind is made of dots shown at head-relative
+directions: look at each one.
+
+  quick   one dot in the middle of your view. It opens when the headset goes on: eyes seen
+          for DON_DELAY after none for AWAY_MIN (SteamVR's tracker's variance for an eye under
+          EYE_LOST). SteamVR's "HMD on" can't say: it repeats every minute or so, and it can
+          stay on for hours with nobody in the headset. It also opens when our own tracker asks
+          for a click (its "reseat": the headset may sit differently on your face now), at most
+          once every QUICK_COOLDOWN, and on "quickcal" (Frametop Input Settings, or a mouse
+          button or key combination mapped to Gaze quick check), and when a click's correction
+          was past POINTER_GAZE_NUDGE_MAX (55 degrees; the helper's "recheck"): the tracker is
+          far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing. The headset going
+          on is seen only while the gaze service is awake (gaze mode on, someone wearing it: see
+          ft-gazed), so it also opens when gaze mode comes on after the service idled.
+  five    the middle and four around it, when the first FIVE_COUNT lessons after a quick check
+          were all over FIVE_LIMIT degrees off: the quick check didn't fix it.
+  full    the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil
+          size, and the tracker's error with it, changes with brightness), each the middle and
+          a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots)
+          RING degrees out, half that in the middle round, turned 20 degrees a round. It opens
+          whenever gaze mode is on without a calibration for the tracker in use and someone's
+          in the headset, and on "calibrate". One that closes unfinished (ignored, too few
+          dots) opens again only once the headset comes off and on, or gaze mode off and on.
+          Frametop's screens hide while it runs. Quitting it while there's still no
+          calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
+          Why gaze mode can't work yet goes in the status ("problem"), for Input Settings.
+
+  fit     the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a
+          card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was
+          seen) and hints, from the gaze probe's Headset fit (gaze/fitcheck.py), while you
+          adjust the headset. A left click or Meta+J runs its guided check (dots, then looks
+          down, up, left and right); a right click or Meta+K closes it, as does FIT_TIMEOUT.
+
+A check asked for while the gaze service idles (quickcal, calibrate, fitcheck) wakes it and
+waits until the tracker sends, at most ft-gazed's WAKE_SETTLE; then it opens, or logs why not.
+
+The quick check's dot captures itself: from CHECK_SETTLE after it shows (the eyes getting
+there), once the gaze has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and
+capture time). It's the gaze holding still that counts, not where the tracker puts it, so it
+works however far off the tracker is; a left click or Meta+J (the pointer helper's
+"calaccept") takes it now. The full calibration's and five's dots wait for that click: you
+click when you're looking at the dot (the user asked for that: a steady gaze isn't always on
+the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as
+it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason:
+gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a
+click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common
+reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
+renewed every second; the helper shows it again by itself when that stops).
+
+What a capture teaches:
+  our tracker   quick and five: a click ("click T YAW PITCH", like a pointer lesson); full:
+                calib-start, a calib-point for each dot, calib-fit (its calibration)
+  SteamVR's     quick and five: a lesson for each eye, like the pointer's; full: each source's
+                calibration fitted from the dots (gazecal.Correction, the probe's way), saved
+                to calibration.json, and the lessons start over on top of it
+Either way, how far off each eye was goes to the eye bias, and each capture to the lesson log.
+"""
+
+import json
+import math
+import os
+import selectors
+import signal
+import socket
+import statistics
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+from fitcheck import MIN_REGION, FitCheck, wrap
+from gazecal import DEFAULT_MODEL, EYE_LOST, STATE, steady_samples
+
+REPO = Path(__file__).resolve().parents[1]
+PANEL_PROG = REPO / "gaze" / "build" / "ft-gazepanel"
+PANEL = "\0ft_gazepanel"
+POINTER = "\0ft_pointer_helper"
+SCREENS = "\0ft_screens"
+EYES = "\0ft_eyes"
+CONF = Path.home() / ".config" / "frametop.conf"
+CALIBRATION = STATE / "calibration.json"
+POINTS = STATE / "points.jsonl"   # the probe's record of calibration dots, for its refine
+CHECK_LOG = STATE / "checks.jsonl"
+
+CHECK_SETTLE = 0.45
+CHECK_WINDOW = 0.6
+CHECK_SPREAD = 1.0     # degrees
+ACCEPT_SPREAD = 2.5    # degrees: a capture asked for (calaccept) takes this much
+CLICK_IDLE = 120.0     # seconds a dot of five or full waits for its click; then the check closes
+QUICK_TIMEOUT = 6.0
+QUICK_COOLDOWN = 120.0
+DON_DELAY = 3.0        # seconds of eyes after AWAY_MIN without: the headset went on
+AWAY_MIN = 3.0
+EYES_GONE = 2.0        # seconds without eyes that close a check: the headset came off
+FIVE_LIMIT = 2.0
+FIVE_COUNT = 3
+DONE_PAUSE = 0.35      # seconds the filled dot shows before the next
+RING = 20.0
+ROUND_BG = (0.03, 0.33, 0.8)
+ROUND_NAMES = ("dark", "medium", "bright")
+RING_SCALE = (1.0, 0.5, 1.0)
+PANEL_RETRY = 10.0
+FULL_RETRY = 10.0      # seconds before an automatic calibration that failed to start tries again
+FIT_TIMEOUT = 300.0    # seconds the fit check stays up
+FIT_EVERY = 0.5        # seconds between its cards' updates (each is a new picture for the panel)
+FIT_HINT_WIDTH = 95    # characters a hint line holds in the panel
+
+
+def log(msg):
+    print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
+
+
+def check_dots(kind, own):
+    """(yaw, pitch, round): head-relative degrees, yaw +left, pitch +up."""
+    if kind == "quick":
+        return [(0.0, 0.0, 0)]
+    if kind == "five":
+        return [(0.0, 0.0, 0), (12.0, 0.0, 0), (-12.0, 0.0, 0), (0.0, 9.0, 0), (0.0, -9.0, 0)]
+    out = []
+    n = 8 if own else 6
+    for rnd in range(3):
+        out.append((0.0, 0.0, rnd))
+        r = RING * RING_SCALE[rnd]
+        for i in range(n):
+            a = math.radians(-90 + rnd * 20 + i * 360 / n)  # as the probe: x right, y down
+            x, y = r * math.cos(a), r * math.sin(a) * (0.9 if own else 1.0)
+            out.append((-x, -y, rnd))
+    return out
+
+
+# Why SteamVR's samples for a look were dropped (gazecal.steady_samples' counts) -> (short, long):
+# short for the small panel, long for the calibration's. FIT_HINT goes after the ones the
+# headset's fit causes, in the calibration's panel.
+STEAM_REASONS = {"lost_left": ("left eye lost", "SteamVR lost your left eye"),
+                 "lost_right": ("right eye lost", "SteamVR lost your right eye"),
+                 "lost_both": ("both eyes lost", "SteamVR lost both eyes"),
+                 "blink": ("blinked", "you blinked"),
+                 "vergence": ("eyes disagreed", "SteamVR's two eyes disagreed")}
+FIT_HINT = "check the headset fit"
+ACCEPT_WAIT = 1.5      # seconds after a click with no capture before the panel says what it waits for
+
+
+def reject_reason(reply, why):
+    """Why a dot wasn't taken -> (short, long, fit): our tracker's reply (`reply`), or SteamVR's
+    drop counts (`why`, when `reply` is None). `fit`: the headset's fit is the likely cause."""
+    if reply is None:
+        if not why:
+            return "no reading", "SteamVR sent no reading for that look", False
+        key = max(why, key=why.get)
+        return *STEAM_REASONS[key], key.startswith("lost")
+    words = reply.removeprefix("fail ").split()
+    # ft-eyes: "fail the left eye was seen in only 3 frames", "fail the left eye moved (4.2 px)"
+    if len(words) >= 3 and words[0] == "the" and words[2] == "eye":
+        eye = words[1]
+        if "seen" in words:
+            return f"{eye} eye not seen", f"our tracker saw your {eye} eye in only {words[-2]} frames", True
+        if "moved" in words:
+            return f"{eye} eye moved", f"your {eye} eye moved while you looked", False
+    if not reply:
+        return "no answer", "our tracker didn't answer", False
+    if reply.startswith("our tracker isn't running"):
+        return "tracker not running", "our tracker isn't running", False
+    text = reply.removeprefix("fail ")
+    return text[:24], f"our tracker said: {text}", False
+
+
+def spread(points):
+    """The median point and the spread around it (1.4826 x the median distance: a standard
+    deviation that one stray sample can't move far)."""
+    mx = statistics.median(p[0] for p in points)
+    my = statistics.median(p[1] for p in points)
+    return 1.4826 * statistics.median(math.hypot(p[0] - mx, p[1] - my) for p in points), (mx, my)
+
+
+def write_gaze(on, path=CONF):
+    """POINTER_GAZE=1|0 in the config, every other line kept."""
+    try:
+        lines = path.read_text().splitlines()
+    except OSError:
+        lines = []
+    value = f"POINTER_GAZE={1 if on else 0}"
+    for i, line in enumerate(lines):
+        if line.split("#", 1)[0].split("=", 1)[0].strip() == "POINTER_GAZE":
+            lines[i] = value
+            break
+    else:
+        lines.append(value)
+    tmp = path.with_suffix(".tmp")
+    tmp.write_text("\n".join(lines) + "\n")
+    tmp.replace(path)
+
+
+def ask(addr, command, timeout=1.0):
+    """A command to a local datagram socket, and its reply ("" without one)."""
+    s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC)
+    try:
+        s.bind("")
+        s.settimeout(timeout)
+        s.sendto(command.encode(), addr)
+        return s.recv(4096).decode("utf-8", "replace")
+    except OSError:
+        return ""
+    finally:
+        s.close()
+
+
+class Checks:
+    def __init__(self, svc, sel):
+        self.svc = svc
+        self.sel = sel
+        self.out = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
+        self.out.bind("")  # the panel's and the helper's replies come back here
+        sel.register(self.out, selectors.EVENT_READ, "checks")
+        self.check = None
+        self.gaze_on = None
+        self.headset = None      # someone wears it (the helper's "worn"), False "away", None unknown
+        self.gaze_heard = 0.0
+        self.pending = None      # (command words, when): asked for while the gaze service idled
+        self.full_armed = True    # gaze mode on without a calibration opens the full one (need_full)
+        self.full_blocked = None  # why it can't open now
+        self.full_retry_at = 0.0
+        self.full_failed = None   # why the last calibration failed (too few dots), for problem()
+        self.last_quick = 0.0
+        self.sample_at = 0.0     # the tracker last sent anything
+        self.seen_at = 0.0       # eyes last seen (SteamVR's tracker's variance for them, "unc")
+        self.away = True         # no eyes for AWAY_MIN: their coming back is the headset going on
+        self.back_since = None
+        self.reseat_seen = False
+        self.after_quick = None
+        self.panel_proc = None
+        self.panel_restart_at = 0.0
+        self.screens_shown = None
+        self.last_progress = 0.0
+
+    @property
+    def active(self):
+        return self.check is not None
+
+    # --- The panel process ---
+
+    def start_panel(self):
+        if not PANEL_PROG.exists():
+            log(f"ft-gazepanel isn't built: run {REPO}/gaze/build.sh")
+            self.panel_restart_at = time.monotonic() + 60
+            return
+        env = dict(os.environ)
+        env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
+        distrobox = Path.home() / ".local" / "bin" / "distrobox"
+        self.panel_proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(PANEL_PROG), "--watch-stdin"],
+                                           env=env, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL,
+                                           stderr=subprocess.PIPE, start_new_session=True)
+        os.set_blocking(self.panel_proc.stderr.fileno(), False)
+        self.sel.register(self.panel_proc.stderr, selectors.EVENT_READ, "panel")
+        log("ft-gazepanel started")
+
+    def read_panel(self):
+        try:
+            data = os.read(self.panel_proc.stderr.fileno(), 65536)
+        except BlockingIOError:
+            return
+        if not data:
+            log(f"ft-gazepanel stopped (exit {self.panel_proc.poll()}); again in {PANEL_RETRY:.0f} s")
+            self.stop_panel()
+            self.panel_restart_at = time.monotonic() + PANEL_RETRY
+            if self.check:
+                self.close("the panel stopped")
+            return
+        for line in data.decode("utf-8", "replace").splitlines():
+            if line.strip():
+                log(line)
+
+    def stop_panel(self):
+        if not self.panel_proc:
+            return
+        try:
+            self.sel.unregister(self.panel_proc.stderr)
+        except (KeyError, ValueError):
+            pass
+        if self.panel_proc.stdin and not self.panel_proc.stdin.closed:
+            self.panel_proc.stdin.close()
+        try:
+            self.panel_proc.wait(timeout=2)
+        except subprocess.TimeoutExpired:
+            try:
+                os.killpg(self.panel_proc.pid, signal.SIGTERM)
+            except ProcessLookupError:
+                pass
+        self.panel_proc = None
+
+    def to_panel(self, command):
+        try:
+            self.out.sendto(command.encode(), PANEL)
+        except OSError:
+            pass
+
+    def to_helper(self, command):
+        try:
+            self.out.sendto(command.encode(), POINTER)
+        except OSError:
+            pass
+
+    def on_readable(self):
+        """Replies on our socket: the helper's "ok on|off [worn|away]" to "gaze ? headset" (an
+        older helper leaves the headset out); the panel's are dropped."""
+        while True:
+            try:
+                words = self.out.recv(4096).decode("utf-8", "replace").split()
+            except (BlockingIOError, OSError):
+                return
+            if words[:1] == ["ok"] and len(words) in (2, 3) and words[1] in ("on", "off"):
+                on = words[1] == "on"
+                headset = None if len(words) == 2 else words[2] == "worn"
+                was = (self.gaze_on, self.headset)
+                self.gaze_on, self.headset, self.gaze_heard = on, headset, time.monotonic()
+                if on and was[0] is False:
+                    self.on_gaze_on()
+                if was != (on, headset):
+                    self.svc.update_awake()
+
+    # --- State ---
+
+    def calibrated(self):
+        """Does the tracker in use have a calibration? None: our tracker hasn't said yet."""
+        svc = self.svc
+        kind = svc.kind
+        if kind == "own":
+            if time.monotonic() - svc.own_at > 5 or not svc.own:
+                return None
+            return bool((svc.own.get("calibration") or {}).get("dots"))
+        if kind == "eyes":
+            return True
+        return svc.models[svc.source].samples > 0
+
+    def eyes_seen(self, within=1.0):
+        return time.monotonic() - self.seen_at < within
+
+    def can_run(self):
+        """Someone's in the headset and the tracker is sending."""
+        return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2
+
+    def on_gaze_on(self):
+        self.full_armed = True
+        self.need_full("gaze mode came on without a calibration")
+
+    def need_full(self, reason):
+        """Gaze mode is on without a calibration: open the full one, once per arming (see the top),
+        or note why it can't open."""
+        cal = self.calibrated()
+        if cal:
+            self.full_armed = True  # missing one later (the other tracker picked) is news again
+        if not self.gaze_on or self.check or not self.full_armed or cal is not False:
+            self.full_blocked = None
+            return
+        now = time.monotonic()
+        if not self.can_run() and self.svc.waking():
+            return  # the tracker is still starting (the service idled)
+        if not self.can_run():
+            why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2
+                   else "no eyes seen (is the headset on?)")
+        elif now < self.full_retry_at:
+            return
+        else:
+            reply = self.start("full", reason)
+            why = None if reply == "ok" else reply.removeprefix("error ")
+            if why:
+                self.full_retry_at = now + FULL_RETRY
+        if why and why != self.full_blocked:
+            log(f"the calibration can't open: {why}")
+        self.full_blocked = why
+        if not why:
+            self.full_armed = False
+
+    def problem(self):
+        """Why gaze mode, on, can't follow your eyes yet, or None. Our tracker not having said
+        yet is None: Input Settings has its own line for our tracker."""
+        if not self.gaze_on or self.calibrated() is not False:
+            return None
+        if self.check and self.check["kind"] == "full":
+            return "Not calibrated yet: the calibration is open in the headset"
+        if self.full_blocked:
+            return f"Not calibrated, and the calibration can't open: {self.full_blocked}"
+        if not self.full_armed:
+            if self.full_failed:
+                return f"Not calibrated: the calibration failed (most dots: {self.full_failed}). Use Calibrate"
+            return "Not calibrated: the calibration closed unfinished. Use Calibrate"
+        return "Not calibrated: the calibration opens in the headset"
+
+    def auto_quick(self, reason):
+        now = time.monotonic()
+        if self.check or not self.gaze_on or not self.can_run() or self.calibrated() is not True:
+            return
+        if now - self.last_quick < QUICK_COOLDOWN:
+            log(f"quick check skipped ({reason}): one ran {now - self.last_quick:.0f} s ago")
+            return
+        self.start("quick", reason)
+
+    # --- Running a check ---
+
+    def start(self, kind, reason):
+        svc = self.svc
+        if self.check:
+            return "error a check is running"
+        if not self.panel_proc:
+            return "error the panel isn't running (gaze/build.sh builds it)"
+        if kind == "fit":
+            return self.start_fit(reason)
+        if not self.can_run():
+            return "error the headset is off or the tracker isn't sending"
+        own = svc.kind == "own"
+        if kind == "full" and own:
+            reply = ask(EYES, "calib-start", 3.0)
+            if not reply.startswith("ok"):
+                log(f"calibration not started: our tracker says {reply or 'nothing'}")
+                return f"error our tracker: {reply or 'no reply'}"
+        now = time.monotonic()
+        self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
+                      "started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
+                      "skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""}
+        log(f"{kind} check: {reason}")
+        if kind == "full":
+            st = ask(SCREENS, "state", 0.5).split()
+            if len(st) >= 3 and st[0] == "ok":
+                self.screens_shown = (st[2] == "0") if st[1] == "always" else (st[2] == "1")
+                ask(SCREENS, "hide", 0.5)
+        self.to_helper("calpanel 1")
+        self.to_panel(f"show {'full' if kind == 'full' else 'quick'}")
+        self.show_dot()
+        if kind == "quick":
+            self.last_quick = now
+        return "ok"
+
+    def start_fit(self, reason):
+        # Eyes lost are what it's there to show, so it needs only the tracker sending.
+        if time.monotonic() - self.sample_at > 2:
+            return "error the headset is off or the eye tracker isn't sending"
+        now = time.monotonic()
+        self.check = {"kind": "fit", "reason": reason, "own": False, "dots": [], "i": 0, "started": now, "shown": now,
+                      "run": [], "accept": False, "done_at": None, "tries": 0, "skipped": 0, "captured": 0,
+                      "points": {}, "fit": FitCheck(), "drawn": {}, "drawn_at": 0.0, "step": None}
+        log(f"fit check: {reason}")
+        self.to_helper("calpanel 1")
+        self.to_panel("show fit")
+        self.to_panel("title Headset fit: adjust the headset while you watch")
+        self.to_panel("text Left click or Meta+J: guided check · Right click or Meta+K: done")
+        return "ok"
+
+    def fit_tick(self, now):
+        c = self.check
+        fit = c["fit"]
+        # The guided check: its dots at head-relative directions in the panel (the probe's
+        # screen fractions, spread over the middle of the panel), and its looks as the title.
+        step = fit.guide_step(now)
+        key = None if step is None else (step[0], step[1])
+        if key != c["step"]:
+            c["step"] = key
+            if step is None:
+                self.to_panel("dot 0 0 off")
+                self.to_panel("title Headset fit: adjust the headset while you watch")
+            elif step[0] == "dot":
+                fx, fy = step[1]
+                self.to_panel(f"dot {(0.5 - fx) * 32:.2f} {(0.5 - fy) * 24:.2f} look")
+                self.to_panel("title Look at the dot")
+            else:
+                self.to_panel("dot 0 0 off")
+                self.to_panel(f"title {step[1]}")
+        if now - c["drawn_at"] < FIT_EVERY:
+            return
+        c["drawn_at"] = now
+        drawn = c["drawn"]
+        for k in (0, 1):
+            word, (r, g, b) = fit.status(k)
+            sig, seen = fit.signal(k), fit.tracked_share(k, now)
+            cmd = (f"eye {k} {r:.2f} {g:.2f} {b:.2f} {-1 if sig is None else round(sig, 1):g} "
+                   f"{-1 if seen is None else round(seen * 20) / 20:g} {word.capitalize()}")
+            if drawn.get(k) != cmd:
+                drawn[k] = cmd
+                self.to_panel(cmd)
+        if fit.have_eye_data and fit.samples < 3 * MIN_REGION:
+            hints = ["Look around slowly: up, down, left and right. Or left click (Meta+J) for a guided check."]
+        else:
+            hints = fit.hints()
+        lines = [line for h in hints for line in wrap(h, FIT_HINT_WIDTH)][:8]
+        cmd = "hints " + "|".join(lines)
+        if drawn.get("hints") != cmd:
+            drawn["hints"] = cmd
+            self.to_panel(cmd)
+
+    def show_dot(self):
+        c = self.check
+        yaw, pitch, rnd = c["dots"][c["i"]]
+        if c["kind"] == "full":
+            self.to_panel(f"bg {ROUND_BG[rnd]}")
+            self.to_panel(f"title Gaze calibration: {ROUND_NAMES[rnd]} round, {rnd + 1} of 3")
+            self.to_panel("text Look at the dot and click (left click or Meta+J). Right click or Meta+K: stop")
+        elif c["kind"] == "five":
+            self.to_panel(f"text Look at the dot and click ({c['i'] + 1} of {len(c['dots'])})")
+        else:
+            self.to_panel("text Look at the dot")
+        self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look")
+        self.last_progress = None
+        c["shown"] = time.monotonic()
+        c["run"], c["accept"], c["done_at"], c["accept_at"] = [], False, None, None
+
+    def on_sample(self, s):
+        self.sample_at = time.monotonic()
+        if self.pending:
+            self.run_pending()
+        unc = (s["src"].get("mmap1") or {}).get("unc")
+        if unc and min(unc) <= EYE_LOST:
+            self.seen_at = time.monotonic()
+            if self.away and self.back_since is None:
+                self.back_since = self.seen_at
+        c = self.check
+        if c and c["kind"] == "fit":
+            c["fit"].feed(s, self.sample_at)
+            return
+        if not c or c["done_at"]:
+            return
+        if c["own"]:
+            src = s["src"].get("own") or {}
+        else:
+            src = s["src"].get("mmap1") or {}
+            if "hy" not in src:
+                per = [s["src"].get(n) or {} for n in ("left", "right")]
+                per = [p for p in per if "hy" in p]
+                src = {"hy": statistics.fmean(p["hy"] for p in per), "hp": statistics.fmean(p["hp"] for p in per)} if per else {}
+        if "hy" not in src:
+            return
+        now = time.monotonic()
+        c["gaze_at"] = now
+        if now < c["shown"] + CHECK_SETTLE:
+            return
+        g = (src["hy"], src["hp"])
+        run = c["run"]
+        if run:
+            recent = run[-30:]
+            my, mp = statistics.median(p[2] for p in recent), statistics.median(p[3] for p in recent)
+            if math.hypot(g[0] - my, g[1] - mp) > 2.5 * CHECK_SPREAD:
+                run.clear()  # the eyes moved on (a saccade, a blink): start over
+        run.append((now, s, g[0], g[1]))
+        window = [p for p in run if p[0] >= now - CHECK_WINDOW]
+        held = now - run[0][0]
+        # The quick check's ring fills in quarters: each step is a new picture for the panel, so
+        # few of them keep it solid. The others wait for the click (see the top): no ring.
+        progress = math.floor(min(1.0, held / CHECK_WINDOW) * 4) / 4
+        if c["kind"] == "quick" and progress != self.last_progress:
+            yaw, pitch, _ = c["dots"][c["i"]]
+            self.to_panel(f"dot {yaw:.3f} {pitch:.3f} capture {progress:.2f}")
+            self.last_progress = progress
+        if c["accept"] and held >= 0.3 and len(window) >= 10:
+            sd, _ = spread([(p[2], p[3]) for p in window])
+            if sd <= ACCEPT_SPREAD:
+                self.capture(window)
+            return
+        if c["kind"] == "quick" and held >= CHECK_WINDOW and len(window) >= 20:
+            sd, _ = spread([(p[2], p[3]) for p in window])
+            if sd <= CHECK_SPREAD:
+                self.capture(window)
+            else:
+                del run[:len(run) // 2]  # not steady enough yet: keep trying with the newer half
+
+    def capture(self, window):
+        svc = self.svc
+        c = self.check
+        yaw, pitch, rnd = c["dots"][c["i"]]
+        samples = [p[1] for p in window]
+        rec = {"time": time.time(), "check": c["kind"], "dot": c["i"], "round": rnd, "true": [yaw, pitch],
+               "samples": len(samples), "own": c["own"]}
+        ok = True
+        if c["own"]:
+            eyes = []
+            for k in (0, 1):
+                seen = [smp["src"]["own"]["eyes"][k] for smp in samples
+                        if (smp["src"].get("own") or {}).get("eyes") and smp["src"]["own"]["eyes"][k]]
+                eyes.append((statistics.median(e[0] for e in seen), statistics.median(e[1] for e in seen)) if seen else None)
+            miss = [math.hypot(yaw - e[0], pitch - e[1]) if e else None for e in eyes]
+            rec.update(eyes=eyes, miss=miss)
+            t0, t1 = samples[0]["t"], samples[-1]["t"]
+            if c["kind"] == "full":
+                reply = ask(EYES, f"calib-point {t0:.6f} {t1:.6f} {yaw:.4f} {pitch:.4f}", 3.0)
+                rec["reply"] = reply
+                ok = reply.startswith("ok")
+            else:
+                try:
+                    svc.eyes_sock.sendto(f"click {t1:.6f} {yaw:.4f} {pitch:.4f}".encode(), EYES)
+                except OSError as e:
+                    rec["reply"], ok = f"our tracker isn't running ({e})", False
+            if ok:
+                svc.weights["own"].add(miss)
+        else:
+            why = {}
+            steady = steady_samples(samples, why=why)
+            rec["dropped"] = why
+            reads = {}
+            for name in ("action", "mmap1", "mmap2", "left", "right"):
+                pts = [(smp["src"][name]["hy"], smp["src"][name]["hp"]) for smp in steady
+                       if "hy" in (smp["src"].get(name) or {})]
+                if len(pts) >= 15:
+                    reads[name] = (statistics.median(p[0] for p in pts), statistics.median(p[1] for p in pts))
+            rec["reads"] = reads
+            main = ("left", "right") if svc.kind == "eyes" else (svc.source,)
+            if not all(n in reads for n in main):
+                ok = False
+                rec["reply"] = f"only {len(steady)} of {len(samples)} samples had both eyes"
+            elif c["kind"] == "full":
+                for name, (hy, hp) in reads.items():
+                    c["points"].setdefault(name, []).append((hy, hp, yaw - hy, pitch - hp))
+                try:
+                    with open(POINTS, "a") as f:
+                        for name, (hy, hp) in reads.items():
+                            f.write(json.dumps({"time": time.time(), "source": name, "hy": hy, "hp": hp,
+                                                "off": [yaw - hy, pitch - hp], "layout": None, "how": "panel"}) + "\n")
+                except OSError:
+                    pass
+            else:
+                miss = []
+                for name in main:
+                    hy, hp = reads[name]
+                    cy, cp = svc.correction(name, hy, hp)
+                    miss.append(math.hypot(yaw - hy - cy, pitch - hp - cp))
+                    svc.lives[name].add({"time": time.time(), "hy": hy, "hp": hp, "dy": yaw - hy, "dp": pitch - hp,
+                                         "wy": 1.0, "wp": 1.0, "how": "check"}, svc.models[name], svc.mode)
+                rec["miss"] = miss
+                if svc.kind == "eyes":
+                    svc.weights["steam"].add(miss)
+                svc.dirty = True
+        if not ok:
+            short, long, fit = reject_reason(rec.get("reply", "") if c["own"] else None, rec.get("dropped"))
+            rec["reason"] = long
+            c["reasons"][long] = c["reasons"].get(long, 0) + 1
+            if fit:
+                c["fit_reasons"].add(long)
+        rec["accepted"] = ok
+        self.log_check(rec)
+        if not ok:
+            c["tries"] += 1
+            log(f"{c['kind']} check dot {c['i'] + 1}: not taken: {rec['reason']} ({rec.get('reply', '')})")
+            full = c["kind"] == "full"
+            if c["tries"] >= 2 or not full:
+                self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if full else f"Skipped: {short}")
+                self.skip()
+            else:
+                self.note(f"Not taken: {long}. Look at the dot and click again")
+                self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
+                c["run"], c["accept"], c["accept_at"] = [], False, None
+                c["shown"] = time.monotonic()  # settle again, then retry
+            return
+        c["captured"] += 1
+        c["tries"] = 0
+        self.note("")
+        self.to_panel(f"dot {yaw:.3f} {pitch:.3f} done")
+        c["done_at"] = time.monotonic() + DONE_PAUSE
+
+    def note(self, text):
+        """The panel's warning line, over the instructions (empty: none). It stays until the
+        next dot is taken, so a skipped dot's reason is still there at the one after it."""
+        c = self.check
+        if c.get("note") != text:
+            c["note"] = text
+            self.to_panel(f"note {text}".rstrip())
+
+    def skip(self):
+        c = self.check
+        yaw, pitch, _ = c["dots"][c["i"]]
+        c["skipped"] += 1
+        c["tries"] = 0
+        self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
+        c["done_at"] = time.monotonic() + DONE_PAUSE
+
+    def advance(self):
+        c = self.check
+        c["i"] += 1
+        if c["i"] < len(c["dots"]):
+            self.show_dot()
+            return
+        self.finish()
+
+    def finish(self):
+        svc = self.svc
+        c = self.check
+        kind = c["kind"]
+        if kind in ("quick", "five"):
+            if c["captured"]:
+                log(f"{kind} check done ({c['captured']} of {len(c['dots'])} dots)")
+                self.after_quick = [] if kind == "quick" else None
+            self.close()
+            return
+        n = len(c["dots"])
+        if c["captured"] < n * 2 / 3:
+            reasons = c["reasons"]
+            main = max(reasons, key=reasons.get) if reasons else None
+            self.full_failed = main
+            log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays"
+                + (f". Not taken: {', '.join(f'{r} ({k})' for r, k in reasons.items())}" if reasons else ""))
+            self.to_panel(f"text Calibration failed: only {c['captured']} of {n} dots. Try again from Input Settings")
+            if main:
+                fit = main in c["fit_reasons"]
+                self.note(f"Most dots: {main}" + (". Check headset fit on the Gaze page" if fit else ""))
+            self.to_panel("dot 0 0 off")
+            c["done_at"] = time.monotonic() + (8.0 if main else 3.0)  # time to read why
+            c["closing"] = True
+            return
+        self.full_failed = None
+        if c["own"]:
+            reply = ask(EYES, "calib-fit", 10.0)
+            log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
+        else:
+            mode = svc.mode if svc.mode != "none" else DEFAULT_MODEL
+            for name, pts in c["points"].items():
+                if name in svc.models and pts:
+                    svc.models[name].fit(pts, mode)
+            d = {name: m.to_json() for name, m in svc.models.items()}
+            d["_meta"] = {"calibrated_at": time.time(), "model": mode, "how": "panel"}
+            d["_live"] = {}
+            tmp = CALIBRATION.with_suffix(".tmp")
+            tmp.write_text(json.dumps(d, indent=1))
+            tmp.replace(CALIBRATION)
+            svc.forget_lessons()  # a new calibration: the pointer's lessons start over on top of it
+            svc.load_calibration()
+            svc.refit()
+            log(f"calibration ({c['captured']} of {n} dots): {mode}, saved")
+        self.close()
+
+    def close(self, why=None):
+        if not self.check:
+            return
+        if why:
+            log(f"{self.check['kind']} check closed: {why}")
+        self.to_panel("hide")
+        self.to_helper("calpanel 0")
+        if self.check["kind"] == "full" and self.screens_shown:
+            ask(SCREENS, "show", 0.5)
+        self.screens_shown = None
+        self.check = None
+
+    def quit(self):
+        c = self.check
+        if not c:
+            return
+        self.close("quit")
+        if c["kind"] == "full" and self.calibrated() is False:
+            # No calibration still: gaze mode can't work, so it goes off until it's turned on again.
+            try:
+                write_gaze(False)
+            except OSError as e:
+                log(f"can't write {CONF}: {e}")
+            self.to_helper("gaze off")
+            self.gaze_on = False
+            log("gaze mode off: no calibration")
+
+    def log_check(self, rec):
+        try:
+            with open(CHECK_LOG, "a") as f:
+                f.write(json.dumps(rec) + "\n")
+        except OSError:
+            pass
+
+    # --- From the service ---
+
+    def run_pending(self):
+        """A check asked for while the service idled: once the tracker sends (and our tracker has
+        said whether it's calibrated), or WAKE_SETTLE after waking, when its error is the real one."""
+        svc = self.svc
+        words, _ = self.pending
+        ready = (time.monotonic() - svc.last_sample < 2 if words[0] == "fitcheck" else self.can_run()) \
+            and (svc.kind != "own" or self.calibrated() is not None)
+        if not ready and svc.waking():
+            return
+        self.pending = None
+        reply = self.command(words, queue=False)
+        if reply != "ok":
+            log(f"{words[0]}, asked for while idle: {reply.removeprefix('error ')}")
+
+    def command(self, words, queue=True):
+        """quickcal, calibrate, calaccept, calquit -> a reply."""
+        cmd = words[0]
+        if cmd in ("quickcal", "calibrate", "fitcheck") and queue and self.svc.waking() and not self.check:
+            # The tracker isn't running (or only just started): wake it, and do this once it sends.
+            self.pending = (words, time.monotonic())
+            self.svc.update_awake()
+            return "ok waking the eye tracker first"
+        if cmd == "quickcal":
+            return self.start("full" if self.calibrated() is False else "quick", "asked for")
+        if cmd == "calibrate":
+            return self.start("full", "asked for")
+        if cmd == "fitcheck":
+            return self.start("fit", "asked for")
+        if cmd == "calaccept":
+            if self.check and self.check["kind"] == "fit":
+                self.check["fit"].toggle_guide(time.monotonic())
+            elif self.check:
+                if not self.check["accept"]:
+                    self.check["accept_at"] = time.monotonic()
+                self.check["accept"] = True
+            return "ok"
+        if cmd == "calquit":
+            self.quit()
+            return "ok"
+        if cmd == "recheck" and len(words) == 2:
+            self.auto_quick(f"a click was corrected {words[1]} deg")
+            return "ok"
+        return "error unknown command"
+
+    def after_lesson(self, rec):
+        if self.after_quick is None or "refused" in rec:
+            return
+        self.after_quick.append(rec.get("lesson_deg", 0.0))
+        if len(self.after_quick) < FIVE_COUNT:
+            return
+        off = self.after_quick
+        self.after_quick = None
+        if all(d > FIVE_LIMIT for d in off):
+            log(f"the {FIVE_COUNT} lessons after the quick check were {', '.join(f'{d:.1f}' for d in off)} deg off")
+            if self.gaze_on and self.can_run() and not self.check:
+                self.start("five", "the quick check didn't fix it")
+
+    def tick(self):
+        c = self.check
+        if not c:
+            return
+        now = time.monotonic()
+        if c["kind"] == "fit":
+            # Not closed when the eyes go: adjusting the headset loses them.
+            if now - c["started"] > FIT_TIMEOUT:
+                self.close("timed out")
+            else:
+                self.fit_tick(now)
+            return
+        if c["done_at"] and now >= c["done_at"]:
+            if c.get("closing"):
+                self.close()
+            else:
+                self.advance()
+            return
+        if not self.eyes_seen(EYES_GONE):
+            self.close("the headset came off")
+            return
+        if c["done_at"]:
+            return
+        if c["accept"] and c["accept_at"] and now - c["accept_at"] > ACCEPT_WAIT:
+            # Clicked, but no capture yet (see on_sample): say what it's waiting for.
+            full = c["kind"] == "full"
+            if now - c.get("gaze_at", 0.0) > 0.5:
+                self.note("Waiting: the eye tracker isn't sending a gaze" if full else "Waiting: no gaze")
+            else:
+                self.note("Waiting for your gaze to hold still on the dot" if full else "Hold your look still")
+        if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
+            self.close("ignored")
+        elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
+            self.close(f"no click on dot {c['i'] + 1} for {CLICK_IDLE:.0f} s")
+
+    def periodic(self):
+        svc = self.svc
+        now = time.monotonic()
+        if not self.panel_proc and now >= self.panel_restart_at:
+            self.start_panel()
+        self.to_helper("gaze ? headset")
+        if now - self.gaze_heard > 5:
+            self.gaze_on = self.headset = None  # the helper isn't answering
+        if self.pending:
+            self.run_pending()
+        if self.check:
+            self.to_helper("calpanel 1")
+        if not self.eyes_seen(AWAY_MIN):
+            self.away, self.back_since = True, None
+        elif self.back_since is not None and not self.eyes_seen():
+            self.back_since = None  # gone again before DON_DELAY
+        elif self.back_since is not None and now - self.back_since >= DON_DELAY:
+            self.away, self.back_since = False, None
+            self.full_armed = True  # a calibration that closed unfinished opens again
+            self.auto_quick("the headset went on")
+        if svc.kind == "own" and now - svc.own_at < 5:
+            reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values())
+            if not reseat:
+                self.reseat_seen = False
+            elif not self.reseat_seen and self.can_run():
+                self.reseat_seen = True
+                self.auto_quick("our tracker asked for a click")
+        self.need_full("gaze mode is on without a calibration")
+
+    def status(self):
+        c = self.check
+        st = {"check": None, "gaze_mode": self.gaze_on, "headset_worn": self.headset, "pending": self.pending[0][0]
+              if self.pending else None, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
+              "problem": self.problem(),
+              "panel": self.panel_proc is not None,
+              "last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None}
+        if c:
+            st["check"] = {"kind": c["kind"], "dot": c["i"] + 1, "dots": len(c["dots"]), "captured": c["captured"],
+                           "skipped": c["skipped"], "reason": c["reason"]}
+        return st
+
+    def stop(self):
+        self.close("the gaze service is stopping")
+        self.stop_panel()

gaze/panel/ft-gazepanel.cpp

@@ -0,0 +1,533 @@
+// ft-gazepanel: the gaze calibration panel (docs/design.md, gaze/README.md). A SteamVR overlay
+// fixed to the headset, so wherever you turn your head it stays in the same place in your
+// view: a dot drawn at a head-relative direction is exactly that direction from the headset,
+// which is what the gaze service needs to know where you were looking. It's drawn on the CPU
+// and takes no input: the gaze service (gaze/ft-gazed) drives it, and the pointer helper
+// passes it your presses (calaccept, calquit).
+//
+// The panel sits POINTER-like at --distance (1.5 m, about where Frametop's screens are, so
+// the eyes converge as they do in use). "quick" is a small square, QUICK_DEG across, for the
+// one-dot check; "full" is FULL_DEG across (4:3), with a solid background whose brightness the
+// service sets per round (pupil size changes with it, and the tracker's error with it); "fit"
+// is FIT_DEG across (4:3), see-through like quick, for the headset fit check: a card per eye
+// (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and hints.
+//
+// Control socket: abstract unix datagram "@ft_gazepanel" (--socket NAME); a sender with an
+// address gets "ok" or "error ...":
+//   show quick|full|fit          the panel, empty, in front of you
+//   hide
+//   bg <0..1>                    the background's brightness (full)
+//   dot <yaw> <pitch> <state> [<progress 0..1>]
+//                                the dot, head-relative degrees (yaw +left, pitch +up); state:
+//                                look, capture (a ring filling to progress), done,
+//                                fail, off
+//   title <text> / text <text>   a line at the top / at the bottom (empty to clear)
+//   note <text>                  a warning line just above the bottom one, in orange (red on
+//                                the bright round): why a dot wasn't taken (empty to clear)
+//   eye <0|1> <r> <g> <b> <signal 0..1|-1> <seen 0..1|-1> <word>
+//                                fit: an eye's card (0 left): its state in that colour, the
+//                                tracker's signal, the share of the last 10 s it was seen
+//   hints <line>|<line>|...      fit: lines under the cards (empty to clear)
+//   ping
+//
+// Each picture goes into the next of three shared buffers (linear DMA-BUFs SteamVR imported
+// once, the size of the biggest panel; the texture bounds show the part in use), and the panel
+// switches to it, as screens/keyboard.cpp does. SetOverlayRaw, which uploads a new texture each
+// time, flickered on every change of the full calibration's 1024x768 picture, and in a live
+// test the headset kept showing an old picture after the panel had drawn new ones (2026-10-01).
+// It's only the fallback. A "show" makes the panel visible once its first picture is in.
+//
+// Options: --watch-stdin (quit when stdin closes: the service runs it), --socket NAME,
+// --distance METRES. Runs in the dev container (gaze/build.sh builds it into gaze/build).
+#include <openvr.h>
+
+#define STB_TRUETYPE_IMPLEMENTATION
+#include "stb_truetype.h"
+
+#include <drm_fourcc.h>
+#include <fcntl.h>
+#include <gbm.h>
+#include <sys/socket.h>
+#include <sys/un.h>
+#include <unistd.h>
+
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <cmath>
+#include <csignal>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <map>
+#include <string>
+#include <thread>
+#include <vector>
+
+namespace {
+
+using Clock = std::chrono::steady_clock;
+constexpr double kQuickDeg = 16;      // QUICK_DEG: the one-dot check's square
+constexpr double kFullDeg = 64;       // FULL_DEG: the full calibration's width (4:3)
+constexpr double kFitDeg = 40;        // FIT_DEG: the headset fit check's width (4:3)
+constexpr int kQuickPx = 320, kFullW = 1024, kFullH = 768, kFitW = 800, kFitH = 600;
+std::atomic<bool> g_stop{false};
+
+// ---------------------------------------------------------------- text (as screens/keyboard.cpp)
+
+stbtt_fontinfo g_font;
+std::vector<unsigned char> g_fontData;
+bool g_fontOk = false;
+constexpr int kMaxW = kFullW, kMaxH = kFullH;  // the buffers' size: the biggest panel
+struct Glyph {
+    std::vector<unsigned char> bitmap;
+    int w = 0, h = 0, xoff = 0, yoff = 0, advance = 0;
+};
+std::map<std::pair<uint32_t, int>, Glyph> g_glyphs;
+
+void LoadFont() {
+    std::string path;
+    if (FILE *p = popen("fc-match -f '%{file}' 'Noto Sans' 2>/dev/null", "r")) {
+        char buf[512];
+        if (std::fgets(buf, sizeof buf, p)) path = buf;
+        pclose(p);
+    }
+    if (path.empty()) path = "/usr/share/fonts/google-noto-vf/NotoSans[wght].ttf";
+    if (FILE *f = std::fopen(path.c_str(), "rb")) {
+        std::fseek(f, 0, SEEK_END);
+        g_fontData.resize(size_t(std::ftell(f)));
+        std::fseek(f, 0, SEEK_SET);
+        g_fontOk = std::fread(g_fontData.data(), 1, g_fontData.size(), f) == g_fontData.size() &&
+                   stbtt_InitFont(&g_font, g_fontData.data(), stbtt_GetFontOffsetForIndex(g_fontData.data(), 0));
+        std::fclose(f);
+    }
+    if (!g_fontOk) std::fprintf(stderr, "ft-gazepanel: no font (%s); no text\n", path.c_str());
+}
+
+const Glyph &GetGlyph(uint32_t cp, int size) {
+    auto [it, fresh] = g_glyphs.try_emplace({cp, size});
+    Glyph &g = it->second;
+    if (fresh) {
+        const float scale = stbtt_ScaleForPixelHeight(&g_font, float(size));
+        unsigned char *b = stbtt_GetCodepointBitmap(&g_font, 0, scale, int(cp), &g.w, &g.h, &g.xoff, &g.yoff);
+        if (b) g.bitmap.assign(b, b + size_t(g.w) * g.h), stbtt_FreeBitmap(b, nullptr);
+        int adv, lsb;
+        stbtt_GetCodepointHMetrics(&g_font, int(cp), &adv, &lsb);
+        g.advance = int(std::lround(adv * scale));
+    }
+    return g;
+}
+
+std::vector<uint32_t> Codepoints(const std::string &s) {
+    std::vector<uint32_t> out;
+    for (const unsigned char *p = (const unsigned char *)s.c_str(); *p;) {
+        uint32_t c = *p++;
+        int more = c >= 0xF0 ? 3 : c >= 0xE0 ? 2 : c >= 0xC0 ? 1 : 0;
+        if (more) c &= 0x3Fu >> more;
+        for (; more && (*p & 0xC0) == 0x80; --more) c = c << 6 | (*p++ & 0x3F);
+        out.push_back(c);
+    }
+    return out;
+}
+
+// ---------------------------------------------------------------- the picture
+
+struct EyeCard {
+    std::string word = "no data";
+    double r = 0.6, g = 0.6, b = 0.6, signal = -1, seen = -1;
+};
+
+struct Panel {
+    bool full = false, fit = false;
+    EyeCard eyes[2];
+    std::vector<std::string> hints;
+    int w = kQuickPx, h = kQuickPx;
+    double wDeg = kQuickDeg;  // across
+    std::vector<uint8_t> px;
+    double bg = 0.05;
+    std::string title, text, note;
+    bool dotOn = false;
+    double dotYaw = 0, dotPitch = 0, progress = 0;
+    std::string state = "off";
+};
+
+void Blend(Panel &p, int x, int y, double r, double g, double b, double a) {
+    if (x < 0 || y < 0 || x >= p.w || y >= p.h || a <= 0) return;
+    uint8_t *q = &p.px[(size_t(y) * p.w + x) * 4];
+    a = std::min(a, 1.0);
+    q[0] = uint8_t(std::lround(q[0] + (r * 255 - q[0]) * a));
+    q[1] = uint8_t(std::lround(q[1] + (g * 255 - q[1]) * a));
+    q[2] = uint8_t(std::lround(q[2] + (b * 255 - q[2]) * a));
+    q[3] = uint8_t(std::lround(q[3] + (255 - q[3]) * a));
+}
+
+// A filled disc, or (inner > 0) a ring from inner to outer radius, anti-aliased; with sweep < 1,
+// only that share of the ring, clockwise from the top.
+void Disc(Panel &p, double cx, double cy, double outer, double inner, double r, double g, double b, double a,
+          double sweep = 1) {
+    const int x0 = int(cx - outer - 1), x1 = int(cx + outer + 1), y0 = int(cy - outer - 1), y1 = int(cy + outer + 1);
+    for (int y = y0; y <= y1; ++y)
+        for (int x = x0; x <= x1; ++x) {
+            const double dx = x + 0.5 - cx, dy = y + 0.5 - cy, d = std::hypot(dx, dy);
+            double cover = std::clamp(outer - d + 0.5, 0.0, 1.0);
+            if (inner > 0) cover = std::min(cover, std::clamp(d - inner + 0.5, 0.0, 1.0));
+            if (sweep < 1) {
+                double ang = std::atan2(dx, -dy) / (2 * M_PI);  // 0 at the top, clockwise
+                if (ang < 0) ang += 1;
+                if (ang > sweep) continue;
+            }
+            Blend(p, x, y, r, g, b, a * cover);
+        }
+}
+
+void Rect(Panel &p, int x0, int y0, int x1, int y1, double r, double g, double b, double a) {
+    for (int y = std::max(y0, 0); y < std::min(y1, p.h); ++y)
+        for (int x = std::max(x0, 0); x < std::min(x1, p.w); ++x) Blend(p, x, y, r, g, b, a);
+}
+
+// Text centred on (x, cy), or starting at x (left).
+void Text(Panel &p, const std::string &s, int size, int x, int cy, double r, double g, double b, bool left = false) {
+    if (!g_fontOk || s.empty()) return;
+    const auto cps = Codepoints(s);
+    int width = 0;
+    for (uint32_t cp : cps) width += GetGlyph(cp, size).advance;
+    int ascent, descent, gap;
+    stbtt_GetFontVMetrics(&g_font, &ascent, &descent, &gap);
+    const float scale = stbtt_ScaleForPixelHeight(&g_font, float(size));
+    if (!left) x -= width / 2;
+    const int baseline = cy + int(std::lround((ascent + descent) * scale / 2));
+    for (uint32_t cp : cps) {
+        const Glyph &gl = GetGlyph(cp, size);
+        for (int gy = 0; gy < gl.h; ++gy)
+            for (int gx = 0; gx < gl.w; ++gx)
+                Blend(p, x + gl.xoff + gx, baseline + gl.yoff + gy, r, g, b, gl.bitmap[size_t(gy) * gl.w + gx] / 255.0);
+        x += gl.advance;
+    }
+}
+
+void Text(Panel &p, const std::string &s, int size, int x, int cy, double lum, bool left = false) {
+    Text(p, s, size, x, cy, lum, lum, lum, left);
+}
+
+// The headset fit check (see the top): a card per eye, then the hints.
+void DrawFit(Panel &p, double pxPerDeg, int textSize, double faint) {
+    const int margin = int(p.w * 0.06), cw = int(p.w * 0.41), ch = int(p.h * 0.32), top = int(p.h * 0.12);
+    const int pad = int(pxPerDeg * 0.9), big = int(pxPerDeg * 1.6), small = int(pxPerDeg * 0.8);
+    for (int k = 0; k < 2; ++k) {
+        const EyeCard &e = p.eyes[k];
+        const int x0 = k == 0 ? margin : p.w - margin - cw;
+        Rect(p, x0, top, x0 + cw, top + ch, 1, 1, 1, 0.07);
+        Text(p, k ? "Right eye" : "Left eye", textSize, x0 + pad, top + pad + textSize / 2, faint, true);
+        Text(p, e.word, big, x0 + pad, top + int(ch * 0.40), e.r, e.g, e.b, true);
+        // The tracker's signal: a bar, red to green.
+        const int by = top + int(ch * 0.62), bh = std::max(4, int(pxPerDeg * 0.35)), bw = cw - 2 * pad;
+        Text(p, "Signal", small, x0 + pad, by - small, faint, true);
+        Rect(p, x0 + pad, by, x0 + pad + bw, by + bh, 1, 1, 1, 0.15);
+        if (e.signal >= 0) {
+            const double v = std::clamp(e.signal, 0.0, 1.0);
+            const double r = v < 0.5 ? 1.0 : 1.0 - 1.3 * (v - 0.5), g = v < 0.5 ? 0.3 + 0.9 * v : 0.75 + 0.5 * (v - 0.5);
+            Rect(p, x0 + pad, by, x0 + pad + int(bw * v), by + bh, r, g, 0.35, 0.95);
+        }
+        char seen[64] = "Seen: not yet";
+        if (e.seen >= 0) std::snprintf(seen, sizeof seen, "Seen %d%% of the last 10 s", int(std::lround(e.seen * 100)));
+        Text(p, seen, small, x0 + pad, top + ch - pad, faint, true);
+    }
+    int y = top + ch + pad * 2;
+    for (const std::string &line : p.hints) {
+        Text(p, line, textSize, margin, y, faint, true);
+        y += int(textSize * 1.4);
+    }
+}
+
+// Head-relative direction -> panel pixel: the panel is a plane `d` in front of the headset.
+void ToPixel(const Panel &p, double yaw, double pitch, double &x, double &y) {
+    const double yr = yaw * M_PI / 180, pr = pitch * M_PI / 180;
+    const double half = std::tan(p.wDeg * M_PI / 360);  // half the width, per unit of distance
+    const double X = -std::tan(yr), Y = std::tan(pr) / std::cos(yr);
+    x = (X / (2 * half) + 0.5) * p.w;
+    y = (0.5 - Y / (2 * half) * p.w / p.h) * p.h;
+}
+
+void Draw(Panel &p) {
+    p.px.assign(size_t(p.w) * p.h * 4, 0);
+    const double pxPerDeg = p.w / p.wDeg;
+    if (p.full) {
+        for (size_t i = 0; i < p.px.size(); i += 4)
+            p.px[i] = p.px[i + 1] = p.px[i + 2] = uint8_t(std::lround(p.bg * 255)), p.px[i + 3] = 255;
+    } else {
+        // The quick check and the fit check: a dim rounded panel, see-through, so it's clear of what's behind.
+        const double r = std::min(p.w, p.h) * 0.12;
+        for (int y = 0; y < p.h; ++y)
+            for (int x = 0; x < p.w; ++x) {
+                const double dx = std::max({r - x - 0.5, x + 0.5 - (p.w - r), 0.0});
+                const double dy = std::max({r - y - 0.5, y + 0.5 - (p.h - r), 0.0});
+                const double cover = std::clamp(r - std::hypot(dx, dy) + 0.5, 0.0, 1.0);
+                Blend(p, x, y, 0.06, 0.06, 0.07, 0.82 * cover);
+            }
+    }
+    const bool light = p.full && p.bg > 0.5;  // a dark dot on the bright round
+    const double ink = light ? 0.0 : 1.0, faint = light ? 0.2 : 0.75;
+    const int titleSize = int(pxPerDeg * (p.full ? 1.5 : 1.1)), textSize = int(pxPerDeg * (p.full ? 1.2 : 0.9));
+    Text(p, p.title, titleSize, p.w / 2, int(titleSize * 1.2), faint);
+    Text(p, p.text, textSize, p.w / 2, p.h - int(textSize * 1.3), faint);
+    // Under the full calibration's lowest dots (RING degrees down) and over the text.
+    if (light) Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 0.7, 0.12, 0.05);
+    else Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 1.0, 0.62, 0.3);
+    if (p.fit) DrawFit(p, pxPerDeg, textSize, faint);
+    if (!p.dotOn || p.state == "off") return;
+    double x, y;
+    ToPixel(p, p.dotYaw, p.dotPitch, x, y);
+    const double core = 0.22 * pxPerDeg;
+    if (p.state == "look") {
+        // Still, so the panel looks solid and is drawn again only when something changes.
+        const double rr = 0.75 * pxPerDeg;
+        Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, ink, ink, ink, 0.6);
+        Disc(p, x, y, core, 0, ink, ink, ink, 1);
+    } else if (p.state == "capture") {
+        const double rr = 0.75 * pxPerDeg;
+        Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, ink, ink, ink, 0.25);
+        Disc(p, x, y, rr, rr - 0.12 * pxPerDeg, 0.3, 0.85, 1.0, 1, std::clamp(p.progress, 0.0, 1.0));
+        Disc(p, x, y, core, 0, ink, ink, ink, 1);
+    } else if (p.state == "done") {
+        Disc(p, x, y, 0.75 * pxPerDeg, 0, 0.25, 0.85, 0.4, 0.9);
+        Disc(p, x, y, core, 0, 1, 1, 1, 1);
+    } else if (p.state == "fail") {
+        Disc(p, x, y, 0.75 * pxPerDeg, 0.6 * pxPerDeg, 0.95, 0.35, 0.3, 0.9);
+        Disc(p, x, y, core, 0, ink, ink, ink, 1);
+    }
+}
+
+
+// ---------------------------------------------------------------- the buffers (see the top)
+
+struct Buffer {
+    gbm_bo *bo = nullptr;
+    int fd = -1;
+    vr::SharedTextureHandle_t handle = 0;
+};
+
+struct Buffers {
+    int drm = -1;
+    gbm_device *gbm = nullptr;
+    Buffer b[3];
+    int next = 0;
+    bool ok = false;
+
+    bool Make() {
+        drm = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC);
+        if (drm >= 0) gbm = gbm_create_device(drm);
+        for (Buffer &x : b) {
+            // ABGR8888 is R, G, B, A in memory, like Panel::px.
+            if (gbm) x.bo = gbm_bo_create(gbm, kMaxW, kMaxH, GBM_FORMAT_ABGR8888, GBM_BO_USE_RENDERING | GBM_BO_USE_LINEAR);
+            if (!x.bo || (x.fd = gbm_bo_get_fd(x.bo)) < 0) break;
+            vr::DmabufAttributes_t a{};
+            a.unWidth = kMaxW, a.unHeight = kMaxH;
+            a.unDepth = a.unMipLevels = a.unArrayLayers = a.unSampleCount = 1;
+            a.unFormat = DRM_FORMAT_ABGR8888;
+            a.ulModifier = DRM_FORMAT_MOD_LINEAR;
+            a.unPlaneCount = 1;
+            a.plane[0].unOffset = gbm_bo_get_offset(x.bo, 0);
+            a.plane[0].unStride = gbm_bo_get_stride(x.bo);
+            a.plane[0].nFd = x.fd;
+            if (!vr::VRIPCResourceManager()->ImportDmabuf(vr::VRApplication_Overlay, &a, &x.handle)) x.handle = 0;
+            if (!x.handle) break;
+        }
+        ok = b[2].handle != 0;
+        if (!ok) {
+            std::fprintf(stderr, "ft-gazepanel: no shared buffers; falling back to SetOverlayRaw (it flickers)\n");
+            Drop();
+        }
+        return ok;
+    }
+
+    void Drop() {
+        for (Buffer &x : b) {
+            if (x.handle) vr::VRIPCResourceManager()->UnrefResource(x.handle);
+            if (x.fd >= 0) close(x.fd);
+            if (x.bo) gbm_bo_destroy(x.bo);
+            x = Buffer{};
+        }
+        if (gbm) gbm_device_destroy(gbm);
+        if (drm >= 0) close(drm);
+        gbm = nullptr, drm = -1, ok = false;
+    }
+
+    // p's picture to the overlay: into the next buffer, premultiplied (the overlay's flag says
+    // so), then the overlay switches to it.
+    void Present(vr::IVROverlay *ov, vr::VROverlayHandle_t h, const Panel &p) {
+        vr::EVROverlayError e;
+        if (!ok) {
+            e = ov->SetOverlayRaw(h, const_cast<uint8_t *>(p.px.data()), uint32_t(p.w), uint32_t(p.h), 4);
+        } else {
+            Buffer &x = b[next];
+            next = (next + 1) % 3;
+            uint32_t stride = 0;
+            void *mapping = nullptr;
+            auto *dst = static_cast<uint8_t *>(gbm_bo_map(x.bo, 0, 0, p.w, p.h, GBM_BO_TRANSFER_WRITE, &stride, &mapping));
+            if (!dst) {
+                std::fprintf(stderr, "ft-gazepanel: can't map a buffer\n");
+                return;
+            }
+            for (int y = 0; y < p.h; ++y) {
+                const uint8_t *src = &p.px[size_t(y) * p.w * 4];
+                uint8_t *row = dst + size_t(y) * stride;
+                for (int i = 0; i < p.w * 4; i += 4) {
+                    const unsigned a = src[i + 3];
+                    row[i] = uint8_t(src[i] * a / 255), row[i + 1] = uint8_t(src[i + 1] * a / 255);
+                    row[i + 2] = uint8_t(src[i + 2] * a / 255), row[i + 3] = uint8_t(a);
+                }
+            }
+            gbm_bo_unmap(x.bo, mapping);
+            const vr::VRTextureBounds_t bounds{0, 0, float(p.w) / kMaxW, float(p.h) / kMaxH};
+            ov->SetOverlayTextureBounds(h, &bounds);
+            vr::Texture_t tex = {&x.handle, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
+            e = ov->SetOverlayTexture(h, &tex);
+        }
+        if (e != vr::VROverlayError_None)
+            std::fprintf(stderr, "ft-gazepanel: the picture didn't go to SteamVR: %s\n", ov->GetOverlayErrorNameFromEnum(e));
+    }
+};
+
+}  // namespace
+
+int main(int argc, char **argv) {
+    bool watchStdin = false;
+    std::string sockName = "ft_gazepanel";
+    double distance = 1.5;
+    for (int i = 1; i < argc; ++i) {
+        if (!std::strcmp(argv[i], "--watch-stdin")) watchStdin = true;
+        else if (!std::strcmp(argv[i], "--socket") && i + 1 < argc) sockName = argv[++i];
+        else if (!std::strcmp(argv[i], "--distance") && i + 1 < argc) distance = std::clamp(std::atof(argv[++i]), 0.5, 5.0);
+        else {
+            std::fprintf(stderr, "usage: %s [--watch-stdin] [--socket NAME] [--distance METRES]\n", argv[0]);
+            return 2;
+        }
+    }
+    std::signal(SIGINT, [](int) { g_stop = true; });
+    std::signal(SIGTERM, [](int) { g_stop = true; });
+    if (watchStdin)
+        std::thread([] {
+            char c[256];
+            while (read(0, c, sizeof c) > 0) {
+            }
+            g_stop = true;
+        }).detach();
+
+    int sock = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0);
+    sockaddr_un addr{};
+    addr.sun_family = AF_UNIX;
+    std::memcpy(addr.sun_path + 1, sockName.data(), std::min(sockName.size(), sizeof addr.sun_path - 2));
+    if (bind(sock, reinterpret_cast<sockaddr *>(&addr), socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sockName.size())) != 0) {
+        std::fprintf(stderr, "ft-gazepanel: @%s is taken (another copy running?)\n", sockName.c_str());
+        return 1;
+    }
+
+    // As Frametop's other SteamVR clients: background first, so we never start vrserver.
+    vr::EVRInitError err = vr::VRInitError_None;
+    vr::VR_Init(&err, vr::VRApplication_Background);
+    if (err == vr::VRInitError_None) {
+        vr::VR_Shutdown();
+        vr::VR_Init(&err, vr::VRApplication_Overlay);
+    }
+    if (err != vr::VRInitError_None) {
+        std::fprintf(stderr, "ft-gazepanel: SteamVR: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
+        return 1;
+    }
+    vr::IVROverlay *ov = vr::VROverlay();
+    vr::VROverlayHandle_t h = vr::k_ulOverlayHandleInvalid;
+    if (ov->CreateOverlay("frametop.gazepanel", "Frametop gaze calibration", &h) != vr::VROverlayError_None) {
+        std::fprintf(stderr, "ft-gazepanel: can't create the overlay (another copy running?)\n");
+        return 1;
+    }
+    ov->SetOverlaySortOrder(h, 250);  // in front of Frametop's screens and the pointer's dot
+    LoadFont();
+    Buffers buffers;
+    if (buffers.Make()) ov->SetOverlayFlag(h, vr::VROverlayFlags_IsPremultiplied, true);
+
+    Panel p;
+    bool visible = false, dirty = false, shown = false;  // shown: SteamVR shows it (after its first picture)
+    auto place = [&] {
+        const double half = std::tan(p.wDeg * M_PI / 360);
+        vr::HmdMatrix34_t m{};
+        m.m[0][0] = m.m[1][1] = m.m[2][2] = 1;
+        m.m[2][3] = float(-distance);
+        ov->SetOverlayTransformTrackedDeviceRelative(h, vr::k_unTrackedDeviceIndex_Hmd, &m);
+        ov->SetOverlayWidthInMeters(h, float(2 * distance * half));
+    };
+    std::fprintf(stderr, "ft-gazepanel running: @%s, %.2f m\n", sockName.c_str(), distance);
+
+    while (!g_stop) {
+        char buf[512];
+        sockaddr_un from{};
+        socklen_t fromLen = sizeof from;
+        ssize_t n;
+        while ((n = recvfrom(sock, buf, sizeof buf - 1, 0, reinterpret_cast<sockaddr *>(&from), &fromLen)) > 0) {
+            buf[n] = 0;
+            std::string reply = "ok";
+            char word[16] = "", state[16] = "";
+            double a = 0, b = 0, c = 0, d = 0, e = 0;
+            int rest = 0;
+            if (!std::strncmp(buf, "show ", 5)) {
+                p.full = !std::strcmp(buf + 5, "full");
+                p.fit = !std::strcmp(buf + 5, "fit");
+                p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx;
+                p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx;
+                p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg;
+                p.title.clear(), p.text.clear(), p.note.clear(), p.dotOn = false, p.state = "off";
+                p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
+                place();
+                visible = dirty = true;  // shown with its first picture
+            } else if (!std::strcmp(buf, "hide")) {
+                ov->HideOverlay(h);
+                visible = shown = false;
+            } else if (std::sscanf(buf, "bg %lf", &a) == 1) {
+                p.bg = std::clamp(a, 0.0, 1.0), dirty = true;
+            } else if (std::sscanf(buf, "dot %lf %lf %15s %lf", &a, &b, state, &c) >= 3) {
+                p.dotYaw = a, p.dotPitch = b, p.state = state, p.progress = c;
+                p.dotOn = std::strcmp(state, "off") != 0, dirty = true;
+            } else if (int k; std::sscanf(buf, "eye %d %lf %lf %lf %lf %lf %n", &k, &a, &b, &c, &d, &e, &rest) >= 6 &&
+                       rest > 0 && (k == 0 || k == 1)) {
+                p.eyes[k] = EyeCard{buf + rest, a, b, c, d, e}, dirty = true;
+            } else if (!std::strncmp(buf, "hints", 5)) {
+                p.hints.clear();
+                std::string s = buf[5] == ' ' ? buf + 6 : "";
+                for (size_t at = 0; !s.empty() && at <= s.size();) {
+                    const size_t bar = std::min(s.find('|', at), s.size());
+                    p.hints.push_back(s.substr(at, bar - at));
+                    at = bar + 1;
+                }
+                dirty = true;
+            } else if (!std::strncmp(buf, "title", 5)) {
+                p.title = buf[5] == ' ' ? buf + 6 : "", dirty = true;
+            } else if (!std::strncmp(buf, "text", 4)) {
+                p.text = buf[4] == ' ' ? buf + 5 : "", dirty = true;
+            } else if (!std::strncmp(buf, "note", 4)) {
+                p.note = buf[4] == ' ' ? buf + 5 : "", dirty = true;
+            } else if (std::sscanf(buf, "%15s", word) == 1 && !std::strcmp(word, "ping")) {
+                reply = visible ? "ok shown" : "ok hidden";
+            } else {
+                reply = "error unknown command";
+            }
+            if (fromLen > offsetof(sockaddr_un, sun_path))
+                sendto(sock, reply.data(), reply.size(), 0, reinterpret_cast<sockaddr *>(&from), fromLen);
+            fromLen = sizeof from;
+        }
+        vr::VREvent_t ev;
+        while (vr::VRSystem()->PollNextEvent(&ev, sizeof ev))
+            if (ev.eventType == vr::VREvent_Quit) {
+                vr::VRSystem()->AcknowledgeQuit_Exiting();
+                g_stop = true;
+            }
+        if (visible && dirty) {
+            Draw(p);
+            buffers.Present(ov, h, p);
+            if (!shown) ov->ShowOverlay(h), shown = true;
+            dirty = false;
+        }
+        std::this_thread::sleep_for(std::chrono::milliseconds(visible ? 10 : 50));
+    }
+    ov->DestroyOverlay(h);
+    buffers.Drop();
+    vr::VR_Shutdown();
+    return 0;
+}

gaze/probe/ft-gazeprobe

@@ -2,8 +2,8 @@
 """ft-gazeprobe: a playground for eye tracking as pointer input on the Frametop desktop.
 
 Opens fullscreen on one Frametop screen and shows where the headset's eye tracker says
-you're looking, from the three sources ft-gaze reads (SteamVR's eye tracking action and
-the two gaze sets in eye-server.mmap). Three modes:
+you're looking, from the sources ft-gaze reads (SteamVR's eye tracking action, the two
+gaze sets in eye-server.mmap, and each eye alone). The modes:
 
   Free look       the gaze dot; the trigger calibrates wherever you are looking.
   Accuracy test   look at each target and tap the trigger; measures every source's error
@@ -15,6 +15,11 @@ the two gaze sets in eye-server.mmap). Three modes:
                   it; if it's the wrong one, hold, then glance toward the right one or
                   move the mouse, and let go on it. Each click teaches the click
                   corrections (LiveCorrection).
+  Headset fit     how well the tracker sees each eye (fitcheck.py): live per eye, whether
+                  it's tracked, how open it is, and the tracker's confidence, and maps of
+                  where you looked and where each eye got lost, with hints. Enter runs a
+                  guided check (dots around the screen, then down, up, left and right),
+                  R starts over. Adjust the headset while you watch it.
 
 Run calibration: the initial calibration, after Apple Vision Pro's eye setup. One dot,
 then six in a circle, in three rounds that go from a dark to a bright screen (pupil size
@@ -22,8 +27,8 @@ changes with brightness, and the tracker's error with it). Look at each highligh
 and press the trigger. At the end, each source's calibration is fitted from all 21 dots;
 freeze and look refines it on demand after that.
 
-Trigger: Enter, Space, or a mouse button. Tab shows and hides the panel, F11 toggles
-fullscreen, Esc quits.
+Trigger: Enter, Space, or a mouse button. Tab shows and hides the panel, C collapses it to
+its title bar (or its arrow button does), F11 toggles fullscreen, Esc quits.
 
 Freeze and look (the default trigger): the press freezes the dot where the tracker says
 you're looking. Look at the frozen dot. After a moment to settle, the probe averages where
@@ -55,6 +60,7 @@ import math
 import os
 import random
 import signal
+import socket
 import statistics
 import subprocess
 import sys
@@ -72,16 +78,61 @@ from gi.repository import Adw, Gdk, Gio, GLib, Gtk  # noqa: E402
 REPO = Path(__file__).resolve().parents[2]
 HELPER = REPO / "gaze" / "build" / "ft-gaze"
 STATE = Path.home() / ".local" / "state" / "frametop" / "gaze"
-SOURCES = ["action", "mmap1", "mmap2"]
-SOURCE_NAMES = {"action": "SteamVR action", "mmap1": "mmap set 1", "mmap2": "mmap set 2"}
-SOURCE_COLORS = {"action": (0.2, 0.8, 1.0), "mmap1": (1.0, 0.6, 0.1), "mmap2": (0.9, 0.3, 0.9)}
+# left and right: each eye alone (set 2's own reading of that eye), calibrated and tested
+# like the rest, to see what one eye is worth against both. own: our own tracker
+# (gaze/tracker/ft-eyes, which the gaze service runs); it has its own calibration.
+SOURCES = ["action", "mmap1", "mmap2", "left", "right", "own"]
+SOURCE_NAMES = {"action": "SteamVR action", "mmap1": "mmap set 1", "mmap2": "mmap set 2", "left": "Left eye",
+                "right": "Right eye", "own": "Own tracker"}
+SOURCE_COLORS = {"action": (0.2, 0.8, 1.0), "mmap1": (1.0, 0.6, 0.1), "mmap2": (0.9, 0.3, 0.9),
+                 "left": (0.4, 1.0, 0.6), "right": (1.0, 1.0, 0.4), "own": (1.0, 0.35, 0.35)}
 TRIGGER_KEYS = {Gdk.KEY_Return, Gdk.KEY_KP_Enter, Gdk.KEY_space}
 
 
+class OwnTracker:
+    """Talks to our own tracker (gaze/tracker/ft-eyes) over its control socket (@ft_eyes).
+    It keeps its own calibration: the probe sends it calibration dots and clicks, each with
+    when you looked and where (head-relative degrees), and it learns from its own frames.
+    The gaze service (ft-gazed) runs it: `lease` asks it to keep it running a while longer,
+    whatever the Eye tracker setting says."""
+
+    def __init__(self):
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+        self.sock.bind(f"\0ft_gazeprobe.{os.getpid()}")
+        self.sock.settimeout(1.0)
+
+    def ask(self, command):
+        """The reply line, or "fail ..." if the tracker isn't running or doesn't answer."""
+        try:
+            while True:  # drop a late reply to an earlier command
+                self.sock.setblocking(False)
+                self.sock.recv(4096)
+        except (BlockingIOError, OSError):
+            pass
+        self.sock.settimeout(1.0)
+        try:
+            self.sock.sendto(command.encode(), "\0ft_eyes")
+            return self.sock.recv(4096).decode()
+        except (ConnectionRefusedError, FileNotFoundError):
+            return "fail the Own tracker isn't running yet (the gaze service starts it)"
+        except (socket.timeout, OSError) as e:
+            return f"fail no answer from the Own tracker ({e})"
+
+    def lease(self, seconds=30):
+        """Ask the gaze service to keep the Own tracker running for `seconds` more. False if
+        the gaze service isn't running."""
+        try:
+            self.sock.sendto(f"eyes {seconds}".encode(), "\0ft_gazed")
+            return True
+        except OSError:
+            return False
+
+
 # The math, filters, correction models, and SteamVR log reader are shared with ft-gazed.
 sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 from gazecal import (DEFAULT_MODEL, MODELS, Correction, Fixation, LiveCorrection, OneEuro,  # noqa: E402
                      SteamEyeLog, cross_validate, deg_from_px, px_from_deg, steady_samples)
+from fitcheck import FitCheck  # noqa: E402
 
 
 # --- Gaze from ft-gaze ----------------------------------------------------------------
@@ -414,8 +465,12 @@ class Probe(Adw.ApplicationWindow):
         self.pointer = None
         self.practice = None   # click practice: the press being held (see practice_press)
         self.recent = deque(maxlen=60)  # the last samples, for where you looked at a press
+        self.fitcheck = FitCheck()      # Headset fit: how well the tracker sees each eye
 
         self.models = {s: Correction() for s in SOURCES}
+        self.own = OwnTracker()
+        self.reseat_check = False    # showing the one-dot check after the headset was off
+        self.reseat_skipped = False  # S skipped this one
         self.live = {s: LiveCorrection() for s in SOURCES}
         self.steam = SteamEyeLog()
         self.steam.poll()
@@ -471,8 +526,8 @@ class Probe(Adw.ApplicationWindow):
 
     # --- The context menu (right-click, the Menu key, or Shift+F10) ---
 
-    MODE_KEYS = ["free", "test", "practice", "snap"]
-    MODE_NAMES = ["Free look", "Accuracy test", "Click practice", "Snap practice"]
+    MODE_KEYS = ["free", "test", "practice", "snap", "fit"]
+    MODE_NAMES = ["Free look", "Accuracy test", "Click practice", "Snap practice", "Headset fit"]
 
     def build_menu(self):
         def action(name, fn, state=None):
@@ -501,6 +556,8 @@ class Probe(Adw.ApplicationWindow):
         self.mode_action.connect("activate", lambda a, v: self.set_mode(v.get_string()))
         self.add_action(self.mode_action)
         self.panel_action = action("panel", self.on_panel_toggle, GLib.Variant("b", True))
+        self.collapse_action = action("collapse", lambda a, v: self.set_collapsed(v.get_boolean()),
+                                      GLib.Variant("b", False))
         self.full_action = action("fullscreen", lambda a, v: self.toggle_fullscreen(), GLib.Variant("b", True))
         action("quit", lambda *_: self.close())
 
@@ -527,6 +584,7 @@ class Probe(Adw.ApplicationWindow):
         menu.append_submenu("Correction model", models)
         view = Gio.Menu()
         view.append("Panel", "win.panel")
+        view.append("Collapse panel", "win.collapse")
         view.append("Fullscreen", "win.fullscreen")
         view.append("Quit", "win.quit")
         menu.append_section(None, view)
@@ -655,16 +713,29 @@ class Probe(Adw.ApplicationWindow):
         return s
 
     def build_panel(self):
-        box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=8, halign=Gtk.Align.START,
-                      valign=Gtk.Align.START, margin_start=40, margin_top=40)
-        box.add_css_class("probe-panel")
-        box.set_size_request(460, -1)
+        panel = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, halign=Gtk.Align.START,
+                        valign=Gtk.Align.START, margin_start=40, margin_top=40)
+        panel.add_css_class("probe-panel")
 
-        title = Gtk.Label(label="Gaze probe", xalign=0)
+        # The title bar stays when the panel is collapsed, so the gaze dot isn't lost behind it.
+        head = Gtk.Box(spacing=12)
+        title = Gtk.Label(label="Gaze probe", xalign=0, hexpand=True)
         title.add_css_class("title-2")
-        box.append(title)
-        hint = Gtk.Label(label="Trigger: Enter, Space, or click. Right-click for the menu. Tab hides this panel, Esc quits. "
-                                     "Click the window once so the keys reach it.", xalign=0, wrap=True)
+        head.append(title)
+        self.w_collapse = Gtk.Button(icon_name="pan-up-symbolic", tooltip_text="Collapse the panel (C)")
+        self.w_collapse.connect("clicked", lambda *_: self.set_collapsed(not self.collapsed))
+        head.append(self.w_collapse)
+        panel.append(head)
+
+        box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=8, margin_top=8)
+        box.set_size_request(460, -1)
+        self.panel_body = box
+        panel.append(box)
+        # Wrapped labels ask for their whole text on one line, which made the panel as wide
+        # as the screen; max_width_chars keeps it to about the grid's width.
+        hint = Gtk.Label(label="Trigger: Enter, Space, or click. Right-click for the menu. Tab hides this panel, "
+                               "C collapses it, Esc quits. Click the window once so the keys reach it.",
+                         xalign=0, wrap=True, max_width_chars=45)
         hint.add_css_class("dim-label")
         box.append(hint)
 
@@ -677,12 +748,22 @@ class Probe(Adw.ApplicationWindow):
             grid.attach(widget, 1, row, 1, 1)
             row += 1
 
+        # Which tracker drives the dot. SteamVR's gaze is still logged in the background with
+        # the Own tracker (for comparing), but not shown.
+        tracker = Gtk.Box(css_classes=["linked"])
+        self.w_steamvr = Gtk.ToggleButton(label="SteamVR", active=True)
+        self.w_own = Gtk.ToggleButton(label="Own tracker", group=self.w_steamvr)
+        tracker.append(self.w_steamvr)
+        tracker.append(self.w_own)
+        self.w_own.connect("toggled", lambda b: self.on_tracker())
+        add("Tracker", tracker)
         self.w_screen = self.dropdown(["Screen 1"])
         add("Screen", self.w_screen)
         self.w_mode = self.dropdown(self.MODE_NAMES)
         add("Mode", self.w_mode)
         # mmap set 2 had the least jitter (0.25 deg vs 0.30 for the action, sitting still).
         self.w_source = self.dropdown([SOURCE_NAMES[s] for s in SOURCES], 2)
+        self.w_source.connect("notify::selected", lambda *_: self.sync_tracker())
         add("Source", self.w_source)
         self.w_filter = self.dropdown(["Raw", "One Euro", "Fixation lock"], 2)
         add("Smoothing", self.w_filter)
@@ -740,10 +821,31 @@ class Probe(Adw.ApplicationWindow):
             buttons.append(b)
         box.append(buttons)
 
-        self.w_stats = Gtk.Label(xalign=0, yalign=0, wrap=True, selectable=False)
+        self.w_stats = Gtk.Label(xalign=0, yalign=0, wrap=True, selectable=False, max_width_chars=50)
         self.w_stats.add_css_class("probe-stats")
         box.append(self.w_stats)
-        return box
+        return panel
+
+    @property
+    def collapsed(self):
+        return not self.panel_body.get_visible()
+
+    def set_collapsed(self, collapsed):
+        self.panel_body.set_visible(not collapsed)
+        self.w_collapse.set_icon_name("pan-down-symbolic" if collapsed else "pan-up-symbolic")
+        self.w_collapse.set_tooltip_text("Expand the panel (C)" if collapsed else "Collapse the panel (C)")
+        self.collapse_action.set_state(GLib.Variant("b", collapsed))
+
+    def under_panel(self, x0, y0, x1, y1, margin=20):
+        """Whether a box on the canvas overlaps the panel, collapsed or not."""
+        if not self.panel.get_visible():
+            return False
+        # Collapsed: the title bar. Before the first layout: about the full panel.
+        px, py, pw, ph = (40, 40, 240, 70) if self.collapsed else (40, 40, 520, 960)
+        ok, r = self.panel.compute_bounds(self.area)
+        if ok and r.get_width() > 0:
+            px, py, pw, ph = r.get_x(), r.get_y(), r.get_width(), r.get_height()
+        return x0 < px + pw + margin and px - margin < x1 and y0 < py + ph + margin and py - margin < y1
 
     def place(self):
         """Find the Frametop screens and go fullscreen on the chosen one."""
@@ -818,6 +920,30 @@ class Probe(Adw.ApplicationWindow):
         n = len(self.live[self.source].samples)
         return self.model_mode + (f" + {n} clicks" if n and self.w_snaps.get_active() else "")
 
+    def on_tracker(self):
+        """The tracker toggle: the Own tracker, or SteamVR's set 2 (its quietest source)."""
+        own = self.w_own.get_active()
+        if own != (self.source == "own"):
+            self.w_source.set_selected(SOURCES.index("own") if own else SOURCES.index("mmap2"))
+        if own:
+            if not self.own.lease():
+                self.on_status("The gaze service isn't running, and it runs the Own tracker "
+                               "(frametop-gaze.service, gaze/run.sh install)")
+                return
+            self.lease_at = time.monotonic()
+            reply = self.own.ask("status")
+            if reply.startswith("fail"):
+                self.on_status("Starting the Own tracker (a few seconds)…")
+            else:
+                st = json.loads(reply)
+                self.on_status("Own tracker: " + ("calibrated " + st["calibration"].get("made", "")
+                                                  if st.get("calibration") else "not calibrated yet: Run calibration"))
+
+    def sync_tracker(self):
+        own = self.source == "own"
+        if self.w_own.get_active() != own:
+            (self.w_own if own else self.w_steamvr).set_active(True)
+
     def on_setting(self):
         model = self.model_mode
         if model != getattr(self, "last_model", model):
@@ -843,6 +969,9 @@ class Probe(Adw.ApplicationWindow):
             self.snap = None
         elif not self.snap:
             self.snap_layout()
+        if self.mode == "fit" and getattr(self, "last_mode", None) != "fit" and hasattr(self, "collapse_action"):
+            self.set_collapsed(True)  # the fit check needs the room
+        self.last_mode = self.mode
         if hasattr(self, "mode_action"):
             self.mode_action.set_state(GLib.Variant("s", self.mode))
         self.update_stats()
@@ -860,15 +989,27 @@ class Probe(Adw.ApplicationWindow):
         if keyval == Gdk.KEY_Tab:
             self.panel.set_visible(not self.panel.get_visible())
             return True
+        if keyval in (Gdk.KEY_c, Gdk.KEY_C):
+            self.panel.set_visible(True)
+            self.set_collapsed(not self.collapsed)
+            return True
         if keyval == Gdk.KEY_F11:
             self.toggle_fullscreen()
             return True
+        if keyval in (Gdk.KEY_r, Gdk.KEY_R) and self.mode == "fit":
+            self.fitcheck.reset()
+            self.area.queue_draw()
+            return True
         if keyval == Gdk.KEY_BackSpace and self.mode == "snap":
             self.snap_undo()
             return True
         if keyval in (Gdk.KEY_s, Gdk.KEY_S) and self.calib:
             self.calib_skip()
             return True
+        if keyval in (Gdk.KEY_s, Gdk.KEY_S) and self.reseat_check:
+            self.reseat_check, self.reseat_skipped = False, True
+            self.area.queue_draw()
+            return True
         if keyval == Gdk.KEY_Escape:
             if self.calib or self.test:
                 self.calib = self.test = None
@@ -910,12 +1051,69 @@ class Probe(Adw.ApplicationWindow):
         if self.steam.poll():
             self.on_status("SteamVR's eye tracker just started again: its own calibration started over, "
                            "so the raw gaze may have moved")
+        self.poll_own()
         self.update_stats()
         return True
 
+    def poll_own(self):
+        """With the Own tracker: after the headset was off (or the tracker restarted), its
+        next click starts each eye's shift over, and until then the first clicks can be
+        10-17 degrees off (gaze/tracker/findings.md, session 5). So ask for one look at a centre dot first,
+        as Varjo's headsets do each time they're put on. Also keeps the lease on the tracker."""
+        if self.source != "own":
+            return
+        if time.monotonic() - getattr(self, "lease_at", 0.0) > 10:
+            self.lease_at = time.monotonic()
+            self.own.lease()
+        if self.calib or self.capture:
+            return
+        reply = self.own.ask("status")
+        if reply.startswith("fail"):
+            return
+        st = json.loads(reply)
+        pending = bool(st.get("calibration")) and any(e.get("reseat") for e in st["eyes"].values())
+        if not pending:
+            self.reseat_skipped = False
+        check = pending and not self.reseat_skipped
+        if check != self.reseat_check:
+            self.reseat_check = check
+            self.area.queue_draw()
+
+    def reseat_press(self):
+        """The press on the check dot: a click there teaches both eyes' shifts at once."""
+        if not self.recent:
+            return
+        t_end = self.recent[-1]["t"]
+        w, h = self.canvas_size()
+        d = self.screen_direction([smp for smp in self.recent if smp["t"] >= t_end - 0.6], (w / 2, h / 2))
+        if d is None:
+            self.on_status("no gaze samples on this screen: look at the dot and press again")
+            return
+        reply = self.own.ask(f"click {t_end:.6f} {d[0]:.4f} {d[1]:.4f}")
+        if reply.startswith("ok"):
+            self.reseat_check = False
+            self.on_status("Own tracker: checked after the headset was off")
+        else:
+            self.on_status(reply[5:] if reply.startswith("fail") else reply)
+        self.area.queue_draw()
+
+    def draw_reseat(self, cr, w, h):
+        cx, cy = w / 2, h / 2
+        cr.set_source_rgb(1, 1, 1)
+        cr.arc(cx, cy, 10, 0, 2 * math.pi)
+        cr.fill()
+        cr.set_source_rgb(0.1, 0.1, 0.1)
+        cr.arc(cx, cy, 3, 0, 2 * math.pi)
+        cr.fill()
+        ink = (0.9, 0.9, 0.9)
+        self.text(cr, 60, 70, "Own tracker: the headset was off, so it may sit differently now", ink, 26)
+        self.text(cr, 60, 108, self.status if self.status.startswith("no gaze") else
+                  "Look at the dot and press (Enter, Space, or click). S skips.", ink, 20)
+
     def on_sample(self, s):
         self.sample = s
         self.recent.append(s)
+        self.fitcheck.feed(s, time.monotonic())
         self.rate_count += 1
         self.last_arrival = time.monotonic()
         t = s["t"]
@@ -931,6 +1129,9 @@ class Probe(Adw.ApplicationWindow):
         head = s["head"].get("hit")
         ox, oy = self.origin or (0, 0)
         self.head = (head["x"] - ox, head["y"] - oy) if head and head["s"] == self.screen else None
+        # The Own tracker's eyes, each where it alone puts your gaze (left, right).
+        self.own_eyes = [(h["x"] - ox, h["y"] - oy) if h and h["s"] == self.screen else None
+                         for h in ((s["src"].get("own") or {}).get("ehit") or [])]
 
         r = self.raw.get(self.source)
         if r is None:
@@ -965,7 +1166,10 @@ class Probe(Adw.ApplicationWindow):
 
     def correction(self, name, hy, hp, snaps=None):
         """The whole correction at (hy, hp): the calibration, plus what snap and practice
-        clicks have taught since (when "Learn from clicks" is on)."""
+        clicks have taught since (when "Learn from clicks" is on). None for the Own tracker:
+        it learns from the clicks itself (own_click)."""
+        if name == "own":
+            return 0.0, 0.0
         cy, cp = self.models[name].get(hy, hp, self.model_mode)
         if snaps if snaps is not None else self.w_snaps.get_active():
             ly, lp = self.live[name].get(hy, hp)
@@ -1094,6 +1298,9 @@ class Probe(Adw.ApplicationWindow):
         if self.calib:
             self.calib_press()
             return
+        if self.reseat_check:
+            self.reseat_press()
+            return
         if self.mode == "test":
             self.test_press()
             return
@@ -1103,6 +1310,10 @@ class Probe(Adw.ApplicationWindow):
         if self.mode == "practice":
             self.practice_press()
             return
+        if self.mode == "fit":
+            self.fitcheck.toggle_guide(time.monotonic())
+            self.area.queue_draw()
+            return
         if self.action == "freeze":
             if not self.capture:
                 self.start_capture()
@@ -1131,7 +1342,7 @@ class Probe(Adw.ApplicationWindow):
         return self.cursor
 
     def on_release(self):
-        if self.mode == "test":
+        if self.mode in ("test", "fit"):
             return
         if self.mode == "snap":
             self.snap_release()
@@ -1199,7 +1410,7 @@ class Probe(Adw.ApplicationWindow):
             self.on_status("no gaze on this screen at the press")
             return
         self.practice = {"t": time.monotonic(), "F": self.gaze, "fix": fix, "pointer0": self.pointer,
-                         "target": self.targets[0]}
+                         "target": self.targets[0], "t_raw": self.recent[-1]["t"] if self.recent else None}
         self.area.queue_draw()
 
     def practice_point(self):
@@ -1240,6 +1451,13 @@ class Probe(Adw.ApplicationWindow):
             # You were looking at the release point when you pressed: from the raw gaze to
             # it is the whole error there.
             oy, op = deg_from_px(fix["j"], px - fix["x"], py - fix["y"])
+            if name == "own":
+                rec["sources"][name] = {"hy": fix["hy"], "hp": fix["hp"], "raw": [fix["x"], fix["y"]],
+                                        "off": [oy, op], "n": fix["n"]}
+                if self.source == "own":
+                    rec["own"] = self.own_click(p, fix["hy"] + oy, fix["hp"] + op, math.hypot(oy, op))
+                    verdict = rec["own"]
+                continue
             c = self.correction(name, fix["hy"], fix["hp"], snaps=True)
             left = math.hypot(oy - c[0], op - c[1])
             rec["sources"][name] = {"hy": fix["hy"], "hp": fix["hp"], "raw": [fix["x"], fix["y"]], "off": [oy, op],
@@ -1256,7 +1474,7 @@ class Probe(Adw.ApplicationWindow):
         if learn:
             self.save_calibration()
         rec["verdict"] = verdict
-        rec["learned"] = verdict == "learned"
+        rec["learned"] = verdict == "learned" or verdict.startswith("taught")
         self.attempts.append(rec)
         self.log("practice.jsonl", rec)
         # Drawn for a moment: the drag, from where the gaze put the pointer to where you let go.
@@ -1266,6 +1484,19 @@ class Probe(Adw.ApplicationWindow):
         self.new_target()
         self.update_stats()
 
+    def own_click(self, p, yaw, pitch, off):
+        """Teach the Own tracker: you were looking at (yaw, pitch) just before the press.
+        Every click counts, dragged or not (a click that needed no drag says so too), unless
+        the drag was too far to be the tracker's error."""
+        if not self.w_snaps.get_active():
+            return "clicked (learning is off)"
+        if off > self.OWN_LEARN_MAX:
+            return f"dragged {off:.1f} deg: too far to be the tracker's error, not learned"
+        if p.get("t_raw") is None:
+            return "no sample time at the press"
+        reply = self.own.ask(f"click {p['t_raw']:.6f} {yaw:.4f} {pitch:.4f}")
+        return "taught the Own tracker" if reply.startswith("ok") else reply
+
     def on_motion(self, x, y):
         self.pointer = (x, y)
         if self.practice:
@@ -1286,8 +1517,8 @@ class Probe(Adw.ApplicationWindow):
         last = self.targets[0] if self.targets else None
         for _ in range(50):
             x, y = random.uniform(margin, w - margin), random.uniform(margin, h - margin)
-            if self.panel.get_visible() and x < 560 and y < 1000:
-                continue  # not under the panel
+            if self.under_panel(x - r, y - r, x + r, y + r):
+                continue
             if not last or math.hypot(x - last[0], y - last[1]) > min(w, h) * 0.25:
                 break
         self.targets = [(x, y, r)]
@@ -1323,6 +1554,11 @@ class Probe(Adw.ApplicationWindow):
     FLICK = 1.5       # degrees the gaze has to move from where it was at the press to step
     REARM = 0.9       # ... and come back within, before the next glance steps again
     SNAP_LEARN_MAX = 6.0  # degrees: a lesson bigger than this (after the correction) is a wrong element
+    # The Own tracker takes bigger ones: after the headset is taken off and put back on, its
+    # first clicks were 11-17 degrees off, and a 6-degree limit kept them from teaching it
+    # (gaze/tracker/findings.md, session 5). It keeps the median of an eye's last 5 clicks, so one click
+    # where you changed your mind does little harm.
+    OWN_LEARN_MAX = 25.0
 
     def snap_layout(self):
         w, h = self.canvas_size()
@@ -1368,8 +1604,8 @@ class Probe(Adw.ApplicationWindow):
                 corners = [(ox, oy), (ox + gw, oy), (ox, oy + gh), (ox + gw, oy + gh)]
                 if any(math.hypot(x - cx, y - cy) > radius * 1.05 for x, y in corners):
                     continue
-                if self.panel.get_visible() and ox < 560 and oy < 1000:
-                    continue  # not under the panel
+                if self.under_panel(ox, oy, ox + gw, oy + gh):
+                    continue
                 if any(ox < b[2] + margin and b[0] < ox + gw + margin and oy < b[3] + margin and b[1] < oy + gh + margin
                        for b in boxes):
                     continue
@@ -1487,6 +1723,13 @@ class Probe(Adw.ApplicationWindow):
                 best, best_cost = k, cost
         return best
 
+    @staticmethod
+    def steady_for(name, samples):
+        """steady_samples, but for the Own tracker its own view of the eyes, not SteamVR's."""
+        if name != "own":
+            return steady_samples(samples)
+        return [smp for smp in samples if all((smp["src"].get("own") or {}).get("eyes") or [None])]
+
     def press_fixation(self, name):
         """Where `name` put your gaze just before the press: the median of the last 300 ms,
         without blinks, dropouts, or samples from before an eye movement in that time."""
@@ -1495,7 +1738,7 @@ class Probe(Adw.ApplicationWindow):
         t_end = self.recent[-1]["t"]
         pts = []
         ox, oy = self.origin or (0, 0)
-        for smp in steady_samples([smp for smp in self.recent if smp["t"] >= t_end - 0.3]):
+        for smp in self.steady_for(name, [smp for smp in self.recent if smp["t"] >= t_end - 0.3]):
             src = smp["src"].get(name) or {}
             hit = src.get("hit")
             if hit and hit["s"] == self.screen:
@@ -1669,12 +1912,28 @@ class Probe(Adw.ApplicationWindow):
     def calib_points(self, rnd):
         """The centre, then six on a ring of `Calibration ring` degrees (as much of it as fits
         in the window), turned 20 degrees per round, and half the size in the middle round,
-        so the fit sees the middle, halfway out, and the edge of your view."""
+        so the fit sees the middle, halfway out, and the edge of your view.
+
+        For the Own tracker: the centre, then eight directions on an oval out to `Calibration
+        ring` degrees each way (as much as fits across and up the window), turned 20 degrees
+        per round, and half the size in the middle round. Its fit is quadratic and goes
+        wrong past its dots, and the ring (limited by the window's height) left the sides
+        out: in practice2 (gaze/tracker/findings.md) the worst clicks were all past the ring."""
         w, h = self.canvas_size()
         cx, cy = w / 2, h / 2
         r = self.raw.get(self.source)
         px_per_deg = 1 / r[5] if r and r[5] else 32.0
-        radius = min(self.w_ring.get_value() * px_per_deg, w / 2 - 60, h / 2 - 60) * self.RING_SCALE[rnd]
+        reach = self.w_ring.get_value() * px_per_deg
+        if self.source == "own":
+            k = self.RING_SCALE[rnd]
+            rx = min(reach, cx - 80) * k
+            up, down = min(reach, cy - 150) * k, min(reach, cy - 80) * k  # clear of the text at the top
+            pts = [(cx, cy)]
+            for i in range(8):
+                a = math.radians(-90 + rnd * 20 + i * 45)
+                pts.append((cx + rx * math.cos(a), cy + (up if math.sin(a) < 0 else down) * math.sin(a)))
+            return pts
+        radius = min(reach, w / 2 - 60, h / 2 - 60) * self.RING_SCALE[rnd]
         pts = [(cx, cy)]
         for k in range(6):
             a = math.radians(-90 + rnd * 20 + k * 60)
@@ -1682,6 +1941,13 @@ class Probe(Adw.ApplicationWindow):
         return pts
 
     def start_calibration(self):
+        if self.source == "own":
+            # A fresh calibration for the Own tracker: it forgets the old one's clicks and
+            # shifts when this one is fitted.
+            reply = self.own.ask("calib-start")
+            if reply.startswith("fail"):
+                self.on_status(reply[5:])
+                return
         self.test = None
         self.results = None
         self.capture = None
@@ -1744,7 +2010,9 @@ class Probe(Adw.ApplicationWindow):
                    "sources": {n: {k: g[k] for k in ("err_deg", "sd_deg", "off_deg", "mean", "n")}
                                for n, g in entry["sources"].items()}}
         verdict = None
-        if not mine:
+        if self.source == "own":
+            verdict = self.own_calib_point(c["samples"], target)
+        elif not mine:
             if len(steady) < 20:
                 verdict = (f"only {len(steady)} of {len(c['samples'])} samples had both eyes tracked "
                            "(blinks, or the tracker lost an eye): open your eyes wide and press again")
@@ -1766,11 +2034,44 @@ class Probe(Adw.ApplicationWindow):
             c["retry"] = verdict + (" (S skips this dot)" if c["tries"] >= 2 else "")
             return
         c["tries"] = 0
-        c.setdefault("seen", {})[c["index"]] = tuple(mine["mean"])
+        if mine:
+            c.setdefault("seen", {})[c["index"]] = tuple(mine["mean"])
         c["data"].append(entry)
         c["done_at"] = now  # a short pause on the filled dot, then the next one
         GLib.timeout_add(350, self.calib_next)
 
+    def own_calib_point(self, samples, target):
+        """Send one calibration dot to the Own tracker: the time you looked at it and its
+        direction. None if accepted, else why not."""
+        d = self.screen_direction(samples, target)
+        if d is None or len(samples) < 2:
+            return "no gaze samples on this screen: look at the dot and press again"
+        reply = self.own.ask(f"calib-point {samples[0]['t']:.6f} {samples[-1]['t']:.6f} {d[0]:.4f} {d[1]:.4f}")
+        if reply.startswith("ok"):
+            return None
+        return reply[5:] + ": look at the dot and press again"
+
+    def screen_direction(self, samples, target):
+        """The head-relative direction (yaw, pitch) of a point on the canvas, from the screen
+        geometry around it: SteamVR's nearby gaze and its pixels-per-degree there (only the
+        geometry is used, not where SteamVR thinks you looked). None without samples."""
+        pts = []
+        ox, oy = self.origin or (0, 0)
+        for smp in samples:
+            for name in ("mmap1", "mmap2", "action"):
+                src = smp["src"].get(name) or {}
+                hit = src.get("hit")
+                if hit and hit["s"] == self.screen:
+                    pts.append((hit["x"] - ox, hit["y"] - oy, hit["j"], src["hy"], src["hp"]))
+                    break
+        if len(pts) < 5:
+            return None
+        x = statistics.median(q[0] for q in pts)
+        y = statistics.median(q[1] for q in pts)
+        j = [statistics.fmean(q[2][k] for q in pts) for k in range(4)]
+        oy_, op_ = deg_from_px(j, target[0] - x, target[1] - y)
+        return statistics.median(q[3] for q in pts) + oy_, statistics.median(q[4] for q in pts) + op_
+
     def other_dot(self, c, g):
         """Was the gaze on another dot of this round rather than the current one?
 
@@ -1835,17 +2136,22 @@ class Probe(Adw.ApplicationWindow):
         self.test_history = []
         self.log_points("calibration", data)
         self.save_calibration()
+        own_note = ""
+        if self.source == "own":
+            reply = self.own.ask("calib-fit")
+            own_note = ("Own tracker: " + reply[3:]) if reply.startswith("ok") else ("Own tracker: " + reply)
+            print(f"calibration: {own_note}", file=sys.stderr, flush=True)
         if self.model_mode == "none":
             self.last_model = mode  # just fitted
             self.w_model.set_selected(MODELS.index(mode))
         if self.w_autotest.get_active():
             # The check on spots the calibration hasn't seen, right away: same head
             # position, same session of SteamVR's own calibration.
-            self.on_status(f"calibrated ({mode}) from {len(data)} dots; now testing it on new spots")
+            self.on_status(f"calibrated ({mode}) from {len(data)} dots; now testing it on new spots. {own_note}")
             GLib.timeout_add(1500, lambda: (self.start_test(), False)[1])
         else:
             self.panel.set_visible(True)
-            self.on_status(f"calibrated ({mode}) from {len(data)} dots; Start test to check it")
+            self.on_status(f"calibrated ({mode}) from {len(data)} dots; Start test to check it. {own_note}")
 
     def draw_calib(self, cr, w, h):
         c = self.calib
@@ -1885,9 +2191,9 @@ class Probe(Adw.ApplicationWindow):
                 cr.arc(x, y, 7, 0, 2 * math.pi)
                 cr.stroke()
         GLib.idle_add(self.area.queue_draw)  # the pulse
-        total = len(self.ROUND_BG) * 7
-        n = c["round"] * 7 + c["index"] + 1
-        self.text(cr, 60, 70, f"Calibration: round {c['round'] + 1} of 3 ({self.ROUND_NAMES[c['round']]}), dot {n} of {total}",
+        total = len(self.ROUND_BG) * len(c["points"])
+        n = c["round"] * len(c["points"]) + c["index"] + 1
+        self.text(cr, 60, 70, f"Calibration ({'Own tracker' if self.source == 'own' else 'SteamVR'}): round {c['round'] + 1} of 3 ({self.ROUND_NAMES[c['round']]}), dot {n} of {total}",
                   ink, 26)
         hint = "Face the centre dot, look at the highlighted dot, and press. Esc cancels, S skips a dot."
         self.text(cr, 60, 108, c.get("retry") or hint, (0.8, 0.2, 0.1) if c.get("retry") and bright else
@@ -2268,6 +2574,9 @@ class Probe(Adw.ApplicationWindow):
         if self.calib:
             self.draw_calib(cr, w, h)
             return
+        if self.reseat_check:
+            self.draw_reseat(cr, w, h)
+            return
         m = self.monitors.get(self.screen)
         if m and self.is_fullscreen():
             g = m.get_geometry()
@@ -2281,9 +2590,13 @@ class Probe(Adw.ApplicationWindow):
         if self.w_cells.get_active() and r and r[5]:
             self.degree_grid(cr, w, h, 1 / r[5])
 
+        if self.mode == "fit":
+            self.fitcheck.draw(cr, w, h, self.text, time.monotonic())
+            if self.fitcheck.guide_step(time.monotonic()):
+                GLib.idle_add(self.area.queue_draw)
         if self.test:
             self.draw_test(cr)
-        if self.results and not self.test and not self.snap:
+        if self.results and not self.test and not self.snap and self.mode != "fit":
             self.draw_results(cr)
         if self.snap:
             self.draw_snap(cr)
@@ -2318,7 +2631,8 @@ class Probe(Adw.ApplicationWindow):
 
         frozen = self.capture is not None
         live = not frozen or self.w_live.get_active()
-        if self.w_all.get_active() and live:
+        own = self.source == "own"
+        if self.w_all.get_active() and live and not own:
             for name in SOURCES:
                 rr = self.raw.get(name)
                 if rr:
@@ -2335,7 +2649,14 @@ class Probe(Adw.ApplicationWindow):
                 cr.move_to(x, y - 12)
                 cr.line_to(x, y + 12)
                 cr.stroke()
-        if self.w_rawdot.get_active() and r and live:
+        if self.w_rawdot.get_active() and own and live:
+            # Each eye alone: they should meet, with a little jitter, where you look.
+            for e in getattr(self, "own_eyes", []):
+                if e:
+                    cr.set_source_rgba(1, 0.25, 0.25, 0.85)
+                    cr.arc(e[0], e[1], 5, 0, 2 * math.pi)
+                    cr.fill()
+        elif self.w_rawdot.get_active() and r and live:
             cr.set_source_rgba(1, 0.3, 0.3, 0.8)
             cr.arc(r[0], r[1], 5, 0, 2 * math.pi)
             cr.fill()
@@ -2357,7 +2678,7 @@ class Probe(Adw.ApplicationWindow):
         res = self.result
         if res and now - res["shown"] < 2.5 and "L" in res:
             a = max(0.0, 1 - (now - res["shown"]) / 2.5)
-            ok = res["verdict"] == "learned" or res["verdict"] == "measured"
+            ok = res["verdict"] in ("learned", "measured") or res["verdict"].startswith("taught")
             (fx, fy), (lx, ly) = res["F"], res["L"]
             cr.set_source_rgba(*((1, 0.85, 0.2) if ok else (1, 0.35, 0.3)), a)
             cr.set_line_width(2)
@@ -2534,12 +2855,22 @@ def region_errors(targets, source, key="cerr_deg"):
 def main():
     ap = argparse.ArgumentParser(description="Eye tracking playground for the Frametop desktop")
     ap.add_argument("--screen", type=int, default=0, help="Frametop screen to open on (default: where it opens)")
+    ap.add_argument("--mode", choices=Probe.MODE_KEYS, help="start in this mode (fit: Headset fit)")
+    ap.add_argument("--source", choices=SOURCES,
+                    help="gaze source to start with (default: own if our tracker is running, else mmap2)")
     args, rest = ap.parse_known_args()
     app = Adw.Application(application_id="dev.frametop.GazeProbe", flags=Gio.ApplicationFlags.NON_UNIQUE)
     windows = []
 
     def activate(a):
         win = Probe(a, args.screen)
+        if args.source:
+            win.w_source.set_selected(SOURCES.index(args.source))
+        elif not win.own.ask("status").startswith("fail"):
+            # Our own tracker is running: that's what you're here to test.
+            win.w_source.set_selected(SOURCES.index("own"))
+        if args.mode:
+            win.set_mode(args.mode)
         windows.append(win)
         win.present()
 

gaze/probe/ft-gazeprobe.desktop

@@ -1,9 +1,9 @@
 [Desktop Entry]
 Type=Application
-Name=Frametop Gaze Probe
-GenericName=Eye tracking playground
-Comment=How accurate the headset's eye tracking is on your screens, and gaze clicking with calibration
+Name=Frametop Gaze Probe (development)
+GenericName=Eye tracking development tool
+Comment=For developing Frametop's gaze tracking. Day to day, the calibration and checks are on the Gaze page of Frametop Input Settings
 Exec=@REPO@/gaze/probe/ft-gazeprobe
 Icon=view-visible
-Categories=Utility;Development;
+Categories=Development;
 Keywords=eye;gaze;tracking;calibration;pointer;steamvr;frametop;

gaze/run.sh

@@ -12,8 +12,8 @@ case ${1:-status} in
     "$root/gaze/build.sh"
     fill_template "$root/gaze/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit"
     on_frame "chmod +x gaze/ft-gazed gaze/ft-gazectl"
-    "$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable --now $unit
-sleep 2; echo \"$unit: \$(systemctl --user is-active $unit)\"; journalctl --user -u $unit --no-pager -o cat -n 5" ;;
+    "$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable $unit
+$(start_with_steamvr $unit)" ;;
   uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;;
   start|stop|restart) "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit" ;;
   status) "$frame" --host "systemctl --user is-active $unit" || true; on_frame "gaze/ft-gazectl status" || true ;;

gaze/test/idle-test.py

@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+"""Offline test of the gaze service idling (gaze/ft-gazed, gaze/gazecheck.py): ft-gaze runs only
+while the gaze is in use, and a check asked for while it idles waits for the tracker.
+
+Runs ft-gazed's Service with its sockets renamed, a fake pointer helper (answers "gaze ?
+headset" as the test says), and a fake ft-gaze (prints samples, quits when its stdin closes).
+SteamVR's tracker is the one in use, so our own isn't started; the panel isn't either. Nothing
+reaches the live gaze service, the pointer helper, or SteamVR, so it's safe next to them.
+
+  gaze/test/idle-test.py
+"""
+import importlib.machinery
+import importlib.util
+import os
+import socket
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+GAZE = os.path.join(HERE, "..")
+sys.path.insert(0, GAZE)
+loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(GAZE, "ft-gazed"))
+gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
+loader.exec_module(gazed)
+import gazecheck  # noqa: E402  (the module ft-gazed imported)
+
+tag = f"ft_gaze_idle_test_{os.getpid()}"
+gazed.ME = f"\0{tag}_gazed"
+gazed.POINTER = gazecheck.POINTER = f"\0{tag}_helper"
+gazecheck.SCREENS = f"\0{tag}_screens"
+gazecheck.PANEL_PROG = gazed.REPO / "nonexistent-panel"  # "isn't built": no panel
+gazed.IDLE_AFTER, gazed.WAKE_SETTLE = 1.0, 3.0
+gazed.read_settings = lambda: ("steam", "steam", "auto", 55.0)
+logs = []
+gazed.log = gazecheck.log = lambda msg: logs.append(msg)
+
+# The fake ft-gaze: 90 samples a second, both eyes seen, until its stdin closes.
+tmp = tempfile.mkdtemp(prefix="ft-gaze-idle-test-")
+FAKE = os.path.join(tmp, "ft-gaze")
+with open(FAKE, "w") as f:
+    f.write('''import json, select, sys, time
+while True:
+    if select.select([sys.stdin], [], [], 1 / 90)[0] and not sys.stdin.read(1):
+        break
+    eye = {"hy": 1.0, "hp": 2.0}
+    print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001],
+          "open": [0.8, 0.8]}, "left": eye, "right": eye}}), flush=True)
+''')
+started = []
+
+
+def start_helper(self):
+    """ft-gaze, straight from here instead of the dev container."""
+    import selectors
+    self.proc = subprocess.Popen([sys.executable, FAKE], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+                                 stderr=subprocess.PIPE)
+    os.set_blocking(self.proc.stdout.fileno(), False)
+    os.set_blocking(self.proc.stderr.fileno(), False)
+    self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
+    self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr")
+    self.buf = b""
+    started.append(time.monotonic())
+
+
+gazed.Service.start_helper = start_helper
+
+# The fake pointer helper.
+helper_state = {"reply": "ok off worn"}
+helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+helper.bind(gazed.POINTER)
+helper.settimeout(0.2)
+
+
+def answer():
+    while True:
+        try:
+            data, addr = helper.recvfrom(512)
+        except socket.timeout:
+            continue
+        except OSError:
+            return
+        if data.startswith(b"gaze ?") and addr:
+            helper.sendto(helper_state["reply"].encode(), addr)
+
+
+threading.Thread(target=answer, daemon=True).start()
+svc = gazed.Service(None, False, gazed.POINTER)
+threading.Thread(target=svc.run, daemon=True).start()
+ctl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+ctl.bind("")
+ctl.settimeout(2)
+
+
+def ask(cmd):
+    ctl.sendto(cmd.encode(), gazed.ME)
+    return ctl.recv(4096).decode()
+
+
+failures = []
+
+
+def check(label, got, want):
+    ok = got == want
+    print(("ok    " if ok else "FAIL  ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(label)
+
+
+def wait(cond, seconds):
+    end = time.monotonic() + seconds
+    while time.monotonic() < end:
+        if cond():
+            return True
+        time.sleep(0.05)
+    return cond()
+
+
+time.sleep(2.5)
+check("gaze mode off: idle, no ft-gaze", (svc.awake, svc.proc is None), (False, True))
+check("status says why", ask("status").count('"idle": "gaze mode is off"'), 1)
+
+helper_state["reply"] = "ok on worn"
+check("gaze mode on, headset worn: awake within 2 s", wait(lambda: svc.awake and svc.proc is not None, 2.5), True)
+check("samples come in", wait(lambda: svc.counts["samples"] > 20, 2), True)
+
+helper_state["reply"] = "ok on away"
+check("headset off: still awake for IDLE_AFTER", wait(lambda: not svc.awake, 0.5), False)
+check("then idle, ft-gaze stopped", wait(lambda: not svc.awake and svc.proc is None, 3.5), True)
+check("why: nobody wears it", ask("status").count('"idle": "nobody is wearing the headset"'), 1)
+
+helper_state["reply"] = "ok on"
+check("an older helper (no headset word): awake with gaze mode on", wait(lambda: svc.awake, 2.5), True)
+helper_state["reply"] = "ok off worn"
+check("gaze mode off again: idle", wait(lambda: not svc.awake and svc.proc is None, 4.5), True)
+
+check("wake lease", ask("wake 2"), "ok")
+check("wake: awake at once", (svc.awake, wait(lambda: svc.proc is not None, 1)), (True, True))
+check("lease over (2 s + IDLE_AFTER): idle", wait(lambda: not svc.awake, 4.5), True)
+
+time.sleep(0.5)
+logs.clear()
+count = len(started)
+check("quick check while idle: queued, waking", ask("quickcal"), "ok waking the eye tracker first")
+check("it woke", wait(lambda: svc.awake and len(started) > count, 1.5), True)
+check("once the tracker sends, it ran (and says why it couldn't open)",
+      wait(lambda: any("quickcal, asked for while idle: the panel isn't running" in m for m in logs), 3), True)
+check("nothing left pending", svc.checks.pending, None)
+check("idle again after", wait(lambda: not svc.awake, 3), True)
+
+print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+svc.running = False
+time.sleep(0.7)
+os.remove(FAKE)
+os.rmdir(tmp)
+os._exit(1 if failures else 0)

gaze/tracker/.frame-job

@@ -0,0 +1,14 @@
+# frame-job settings (see `frame-job --help`): offline lab jobs (ft-eyes-score, ft-eyes-e2e)
+# run on the 7i. Eye recordings may go there and nowhere else, and never into the repo: they
+# live outside it, in ~/.local/share/frametop/eyes/captures, and on the 7i in
+# ~/frame-compute/frametop-eyes/data/captures.
+NAME=frametop-eyes
+RUN_ON=7i
+DATA="$HOME/.local/share/frametop/eyes/captures"
+RESULTS=""
+EXCLUDE=""
+# The lab's Python on the 7i: build/venv from requirements.txt, as build.sh makes it on the Frame.
+SETUP='cmp -s requirements.txt build/venv/requirements.done || { rm -rf build/venv && python3 -m venv build/venv && build/venv/bin/pip install -q --disable-pip-version-check -r requirements.txt && cp requirements.txt build/venv/requirements.done; }'
+VENV=
+# Live tools: they read the Frame's eye cameras.
+LOCAL_ONLY="ft-eyes ft-eyes-record ft-eyes-session"

gaze/tracker/build.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Build our own eye tracker on the Frame, in the dev container:
+#   build/ft-eyegrab   the frame grabber. It runs on the host, as root
+#                      (frametop-eyegrab.service, gaze/tracker/install.sh), so this checks it
+#                      only needs glibc symbols the SteamOS host has (2.39; the container has 2.43).
+#   build/venv         Python with numpy and OpenCV (requirements.txt) for ft-eyes and lab/,
+#                      remade when requirements.txt changes.
+# Usage: gaze/tracker/build.sh
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+"$root/scripts/sync.sh" >/dev/null
+exec "$root/scripts/frame.sh" -C gaze/tracker 'set -e; mkdir -p build
+gcc -std=gnu11 -O2 -Wall -Wextra -pthread -o build/ft-eyegrab ft-eyegrab.c
+max=$(objdump -T build/ft-eyegrab | grep -oE "GLIBC_[0-9.]+" | sort -uV | tail -1)
+echo "built build/ft-eyegrab, newest glibc symbol: $max"
+[ "$(printf "%s\n" "$max" GLIBC_2.39 | sort -V | tail -1)" = GLIBC_2.39 ] || { echo "needs newer glibc than the host has" >&2; exit 1; }
+if ! cmp -s requirements.txt build/venv/requirements.done; then
+  rm -rf build/venv
+  python3 -m venv build/venv
+  build/venv/bin/pip install -q --disable-pip-version-check -r requirements.txt
+  cp requirements.txt build/venv/requirements.done
+fi
+echo "build/venv: $(build/venv/bin/python -c "import numpy, cv2; print(\"numpy\", numpy.__version__, \"opencv\", cv2.__version__)")"'

gaze/tracker/eyes_model.py

@@ -0,0 +1,243 @@
+"""The gaze calibration: from pupil and glint positions to head-relative gaze angles.
+
+Shared by ft-eyes (live) and the lab tools (fitting and scoring on recordings). Gaze angles are
+ft-gaze's: degrees relative to the head, yaw positive to the left, pitch positive up.
+
+Per eye (0 = right, camera 0; 1 = left, camera 1), three quadratic fits:
+  pupil   pupil centre -> gaze. The one used for output, after the slip correction.
+  glint   pupil minus the glint pair's midpoint -> gaze. Slip moves both alike, so this
+          holds up when the headset shifts, but it's noisier, and the pair is often gone.
+  where   gaze -> pupil centre: where the pupil sits for a gaze, with no slip.
+Slip: wherever the pair is seen, `glint` gives the gaze, `where` says where the pupil should
+be, and the difference is how far the eye has moved in the image. Slip changes slowly, so
+the median over the last SLIP_WINDOW seconds shifts every frame, glints or not.
+
+That glint estimate is only good to 1-4 px (1-3 degrees), though. Clicks are better: each one
+says where the pupil should have been for a known gaze (`where`), so pupil minus that is the
+shift. `Shift` keeps the median of the last few, and uses the glints only to notice a sudden
+jump (the headset nudged or put back on), until clicks catch up. On practice2 with
+practice1's calibration: 1.2 degrees median, against 2.7 with the glints alone (findings.md).
+"""
+import json
+import time
+from collections import deque
+
+import numpy as np
+
+SLIP_WINDOW = 30.0
+SLIP_MIN = 10        # pair sightings needed before trusting a slip estimate
+MIN_CLICKS = 12
+SPREAD_MIN = 0.3     # degrees: floor for an eye's fit spread, so one eye can't take all the weight
+SHIFT_KEEP = 5       # clicks in the shift estimate
+SHIFT_JUMP = 3.0     # a glint slip change this big (px) since the last click is a nudge
+JUMP_HOLD = 1.0      # s: ...if it holds this long (a bad glint pair gives a jump that snaps back)
+JUMP_MAX = 40.0      # px: bigger is a bad glint pair, not the headset (a re-seat moved 20-30)
+JUMP_WINDOW = 10.0   # seconds of glint sightings for noticing a jump
+
+
+class Quad:
+    """Ridged quadratic least squares from 2-D inputs, normalised on the training set."""
+
+    def __init__(self, X=None, Y=None, ridge=1e-3):
+        if X is None:
+            return
+        X, Y = np.asarray(X, float), np.asarray(Y, float)
+        self.m, self.s = X.mean(0), X.std(0) + 1e-9
+        A = self.terms(X)
+        R = ridge * np.eye(A.shape[1])
+        R[0, 0] = 0
+        self.w = np.linalg.solve(A.T @ A + R, A.T @ Y)
+
+    def terms(self, X):
+        P = (np.atleast_2d(np.asarray(X, float)) - self.m) / self.s
+        return np.c_[np.ones(len(P)), P, P ** 2, P[:, 0] * P[:, 1]]
+
+    def __call__(self, X):
+        return self.terms(X) @ self.w
+
+    def one(self, x, y):
+        """Faster for a single point (the live path)."""
+        px, py = (x - self.m[0]) / self.s[0], (y - self.m[1]) / self.s[1]
+        t = np.array([1.0, px, py, px * px, py * py, px * py])
+        return t @ self.w
+
+    def to_json(self):
+        return {"m": self.m.tolist(), "s": self.s.tolist(), "w": self.w.tolist()}
+
+    @classmethod
+    def from_json(cls, d):
+        q = cls()
+        q.m, q.s, q.w = (np.array(d[k]) for k in ("m", "s", "w"))
+        return q
+
+
+def pair_mid(pair):
+    return ((pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2)
+
+
+class Calibration:
+    """The three fits per eye. Build from clicks (ft-eyes-score's features) or load from JSON.
+
+    `spread` is each eye's RMS miss (degrees) on the clicks its pupil fit was made from.
+    `combine` weights the eyes by its inverse square: on practice2 (one headset position,
+    leave-one-out) that gave 0.75 median against 0.84 for the plain average, since one eye
+    is usually much better than the other (left 0.73, right 1.32 there)."""
+
+    def __init__(self, fits=None, info=None, spread=None):
+        self.fits = fits or {}   # (name, eye) -> Quad
+        self.info = info or {}
+        self.spread = spread or {}  # eye -> degrees
+
+    @classmethod
+    def fit(cls, clicks, info=None):
+        truth = lambda cs: np.array([k["truth"] for k in cs])  # noqa: E731
+        fits, spread = {}, {}
+        for c in (0, 1):
+            cs = [k for k in clicks if k["eye"][c] is not None]
+            if len(cs) < MIN_CLICKS:
+                continue
+            fits["pupil", c] = Quad([k["eye"][c]["pupil"] for k in cs], truth(cs))
+            miss = fits["pupil", c]([k["eye"][c]["pupil"] for k in cs]) - truth(cs)
+            spread[c] = float(np.sqrt(np.mean(np.sum(miss ** 2, axis=1))))
+            fits["where", c] = Quad(truth(cs), [k["eye"][c]["pupil"] for k in cs])
+            gs = [k for k in cs if k["eye"][c]["mid"] is not None]
+            if len(gs) >= MIN_CLICKS:
+                fits["glint", c] = Quad([k["eye"][c]["pupil"] - k["eye"][c]["mid"] for k in gs], truth(gs))
+        return cls(fits, info, spread)
+
+    def has(self, name, eye):
+        return (name, eye) in self.fits
+
+    def slip(self, eye, rows):
+        """Median slip in pixels from pair sightings `rows` (t, px, py, mx, my), or None."""
+        if not self.has("glint", eye) or len(rows) < SLIP_MIN:
+            return None
+        rows = np.asarray(rows, float)
+        gaze = self.fits["glint", eye](rows[:, 1:3] - rows[:, 3:5])
+        return np.median(rows[:, 1:3] - self.fits["where", eye](gaze), axis=0)
+
+    def click_shift(self, eye, pupil, truth):
+        """The shift a click measures: the pupil, less where it sits for that gaze."""
+        return np.asarray(pupil, float) - self.fits["where", eye].one(*truth)
+
+    def gaze(self, eye, x, y, slip=None):
+        """Gaze (yaw, pitch) for a pupil centre, less a slip if there is one."""
+        if slip is not None:
+            x, y = x - slip[0], y - slip[1]
+        return self.fits["pupil", eye].one(x, y)
+
+    def weight(self, eye):
+        s = self.spread.get(eye)
+        return 1.0 if s is None else 1.0 / max(s, SPREAD_MIN) ** 2
+
+    def combine(self, gazes):
+        """The weighted mean of {eye: (yaw, pitch)}, or None if empty."""
+        if not gazes:
+            return None
+        w = {c: self.weight(c) for c in gazes}
+        return sum(w[c] * np.asarray(g, float) for c, g in gazes.items()) / sum(w.values())
+
+    def save(self, path):
+        d = {"version": 1, "info": self.info, "spread": {str(e): s for e, s in self.spread.items()},
+             "fits": [{"name": n, "eye": e, **q.to_json()} for (n, e), q in self.fits.items()]}
+        path.parent.mkdir(parents=True, exist_ok=True)
+        tmp = path.with_suffix(".tmp")
+        tmp.write_text(json.dumps(d, indent=1))
+        tmp.replace(path)
+
+    @classmethod
+    def load(cls, path):
+        d = json.loads(path.read_text())
+        return cls({(f["name"], f["eye"]): Quad.from_json(f) for f in d["fits"]}, d.get("info"),
+                   {int(e): s for e, s in d.get("spread", {}).items()})
+
+
+class Shift:
+    """Where one eye sits in the image now, relative to the calibration (pixels).
+
+    `base` is the median shift the last SHIFT_KEEP clicks measured. `ref` is the glint slip
+    estimate at the last click; if the glint estimate has since moved more than SHIFT_JUMP
+    (and less than JUMP_MAX) and stayed there for JUMP_HOLD seconds, the headset moved, and
+    the change is added until the next click. That click then starts the history over,
+    since the older ones describe the old position. Live, bad glint pairs made the estimate
+    leap by up to 68 px for under a second (practice2), hence the hold. `reseat` does the
+    same for the next click without the glints: the frames stopped (the headset was off),
+    so the headset may be anywhere now."""
+
+    def __init__(self, base=(0.0, 0.0), ref=None):
+        self.meas = []
+        self.base = np.asarray(base, float)
+        self.ref = None if ref is None else np.asarray(ref, float)
+        self.jump = np.zeros(2)
+        self.held = None  # (change, since when) while a jump waits out JUMP_HOLD
+        self.reseated = False
+
+    def glint(self, g, t):
+        """The latest glint slip estimate (or None), at time t (s)."""
+        if g is None:
+            return
+        if self.ref is None:
+            self.ref = np.asarray(g, float)
+        d = np.asarray(g, float) - self.ref
+        size = np.hypot(*d)
+        if size > JUMP_MAX:
+            return
+        if size <= SHIFT_JUMP:
+            self.jump, self.held = np.zeros(2), None
+            return
+        if self.held is None or np.hypot(*(d - self.held[0])) > SHIFT_JUMP:
+            self.held = (d, t)
+        elif t - self.held[1] >= JUMP_HOLD:
+            self.jump = d
+
+    def reseat(self):
+        self.reseated = True
+
+    def click(self, d, g=None):
+        """A click measured the shift d; g is the glint estimate then."""
+        if np.any(self.jump) or self.reseated:
+            self.meas = []
+            self.reseated = False
+        self.meas = (self.meas + [np.asarray(d, float)])[-SHIFT_KEEP:]
+        self.base = np.median(self.meas, axis=0)
+        self.jump, self.held = np.zeros(2), None
+        if g is not None:
+            self.ref = np.asarray(g, float)
+
+    @property
+    def value(self):
+        return self.base + self.jump
+
+    def to_json(self):
+        return {"base": self.base.tolist(), "ref": None if self.ref is None else self.ref.tolist(),
+                "meas": [m.tolist() for m in self.meas]}
+
+    @classmethod
+    def from_json(cls, d):
+        s = cls(d.get("base", (0, 0)), d.get("ref"))
+        s.meas = [np.asarray(m, float) for m in d.get("meas", [])]
+        return s
+
+
+class SlipTracker:
+    """Live slip estimate for one eye: pair sightings over the last SLIP_WINDOW seconds,
+    re-estimated at most every `every` seconds."""
+
+    def __init__(self, cal, eye, every=0.5, window=SLIP_WINDOW):
+        self.cal, self.eye, self.every, self.window = cal, eye, every, window
+        self.rows = deque()
+        self.value, self.at = None, 0.0
+
+    def add(self, t, pupil, mid):
+        self.rows.append((t, pupil[0], pupil[1], mid[0], mid[1]))
+        while self.rows and self.rows[0][0] < t - self.window:
+            self.rows.popleft()
+
+    def get(self, now=None):
+        now = time.monotonic() if now is None else now
+        if now - self.at >= self.every:
+            self.at = now
+            s = self.cal.slip(self.eye, list(self.rows))
+            if s is not None:
+                self.value = s
+        return self.value

gaze/tracker/eyes_pupil.py

@@ -0,0 +1,139 @@
+"""Classic pupil and glint finder for one 512x400 eye-camera frame.
+
+The pupil is a dark blob enclosed by brighter iris and skin. The lens rim and the unlit
+background are just as dark, but they touch the image edge, so any dark region that reaches
+the edge is dropped. Closing the glints' holes can join the pupil to that background (the
+left camera's, when you look more than about 20 degrees left), so when nothing is found
+the search runs again with a smaller closing.
+"""
+import cv2
+import numpy as np
+
+DARK = 30          # pupil pixels are below this (the face around it is 40-180)
+MIN_AREA = 150     # pupil area range in pixels
+MAX_AREA = 20000
+MIN_FILL = 0.75    # blob area / fitted-ellipse area
+MAX_ASPECT = 3.0   # long / short axis; the steep camera sees a squashed pupil
+GLINT = 200        # glints are near-saturated spots on or by the pupil
+CLOSE = 7          # px: closes the glints' holes in the pupil (the right eye's need this much)
+CLOSE_TIGHT = 3    # px: the retry, keeps a pupil near the dark background apart from it
+
+
+NEAR = 70          # the windowed search: this many pixels, or 3 pupil radii, around a hint
+
+
+def find_pupil(frame, near=None):
+    """Return dict(x, y, a, b, angle, area, fill, glints) or None.
+
+    `near` (a previous result) searches a window around it first (0.4 ms, not 1.4-2.1);
+    if the pupil isn't wholly inside the window it falls back to the whole frame."""
+    if near is not None:
+        r = int(max(NEAR, 3 * near['a']))
+        x0, y0 = max(int(near['x']) - r, 0), max(int(near['y']) - r, 0)
+        p = _find_either(frame[y0:int(near['y']) + r, x0:int(near['x']) + r], x0, y0)
+        if p is not None:
+            p['glints'] = find_glints(frame, p)
+            return p
+    p = _find_either(frame, 0, 0)
+    if p is not None:
+        p['glints'] = find_glints(frame, p)
+    return p
+
+
+def _find_either(img, ox, oy):
+    p = _find(img, ox, oy)
+    return p if p is not None else _find(img, ox, oy, CLOSE_TIGHT)
+
+
+def _find(img, ox, oy, close=CLOSE):
+    """The pupil in `img` (a window at ox, oy of the frame), with frame coordinates. A dark
+    region touching the window's edge doesn't count: it's background, rim, or cut off."""
+    f = cv2.GaussianBlur(img, (5, 5), 0)
+    dark = (f < DARK).astype(np.uint8)
+    # Glints punch bright holes in the pupil; close them so the blob stays whole.
+    dark = cv2.morphologyEx(dark, cv2.MORPH_CLOSE, np.ones((close, close), np.uint8))
+    dark = cv2.morphologyEx(dark, cv2.MORPH_OPEN, np.ones((3, 3), np.uint8))
+    n, lab, stats, _ = cv2.connectedComponentsWithStats(dark, connectivity=8)
+    h, w = img.shape
+    best = None
+    for i in range(1, n):
+        x, y, bw, bh, area = stats[i]
+        if area < MIN_AREA or area > MAX_AREA:
+            continue
+        if x <= 1 or y <= 1 or x + bw >= w - 1 or y + bh >= h - 1:
+            continue
+        blob = lab[y:y + bh, x:x + bw] == i
+        cs, _ = cv2.findContours(blob.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
+        c = max(cs, key=len)
+        if len(c) < 5:
+            continue
+        (ex, ey), (d1, d2), ang = cv2.fitEllipse(c)
+        a, b = max(d1, d2) / 2, min(d1, d2) / 2
+        if b < 3 or a / b > MAX_ASPECT:
+            continue
+        fill = area / (np.pi * a * b)
+        if fill < MIN_FILL or fill > 1.25:
+            continue
+        # Prefer the darkest, fullest blob.
+        score = fill - img[y:y + bh, x:x + bw][blob].mean() / 255
+        if best is None or score > best[0]:
+            # fitEllipse's angle is the direction of its first axis (d1); the long axis is
+            # that one or the one at right angles.
+            major = np.radians(ang if d1 >= d2 else ang + 90)
+            best = (score, dict(x=ex + x + ox, y=ey + y + oy, a=a, b=b, angle=ang, major=major,
+                                area=int(area), fill=fill, box=(x + ox, y + oy, bw, bh)))
+    return best[1] if best else None
+
+
+GLINT_RING = 140   # a glint sits on dark iris or pupil: its surroundings are below this
+GLINT_PAIR = (5, 45)  # the two LEDs' reflections: this far apart, in pixels, mostly vertical
+
+
+def find_glints(frame, p):
+    """Small bright spots on dark iris or pupil within 2.5 pupil radii, as (x, y) list.
+    Bright skin has noise speckle above GLINT too, so a spot counts only if the ring
+    around it is dark."""
+    r = int(p['a'] * 2.5) + 6
+    x0, y0 = max(int(p['x']) - r, 0), max(int(p['y']) - r, 0)
+    roi = frame[y0:int(p['y']) + r, x0:int(p['x']) + r]
+    n, lab, stats, cents = cv2.connectedComponentsWithStats((roi >= GLINT).astype(np.uint8))
+    out = []
+    for i in range(1, n):
+        x, y, w, h, area = stats[i]
+        if not 2 <= area <= 80:
+            continue
+        ya, yb, xa, xb = max(y - 4, 0), y + h + 4, max(x - 4, 0), x + w + 4
+        ring = roi[ya:yb, xa:xb][lab[ya:yb, xa:xb] != i]
+        if ring.size and np.median(ring) < GLINT_RING:
+            out.append((cents[i][0] + x0, cents[i][1] + y0))
+    return out
+
+
+def glint_pair(p):
+    """The two LED reflections as ((x, y) upper, (x, y) lower), or None. Picks the
+    vertical-ish pair nearest the pupil centre."""
+    g = p['glints']
+    best = None
+    for i in range(len(g)):
+        for j in range(i + 1, len(g)):
+            dx, dy = g[j][0] - g[i][0], g[j][1] - g[i][1]
+            d = np.hypot(dx, dy)
+            if not GLINT_PAIR[0] <= d <= GLINT_PAIR[1] or abs(dy) < 2 * abs(dx):
+                continue
+            mx, my = (g[i][0] + g[j][0]) / 2, (g[i][1] + g[j][1]) / 2
+            cost = np.hypot(mx - p['x'], my - p['y'])
+            if best is None or cost < best[0]:
+                best = (cost, (g[i], g[j]) if dy > 0 else (g[j], g[i]))
+    return best[1] if best else None
+
+
+def draw(frame, p, scale=1.0):
+    img = cv2.cvtColor(cv2.convertScaleAbs(frame, alpha=2.0), cv2.COLOR_GRAY2BGR)
+    if p:
+        cv2.ellipse(img, ((p['x'], p['y']), (2 * p['a'], 2 * p['b']), np.degrees(p['major'])),
+                    (0, 255, 0), 1)
+        for gx, gy in p['glints']:
+            cv2.circle(img, (int(gx), int(gy)), 3, (0, 0, 255), 1)
+    if scale != 1.0:
+        img = cv2.resize(img, None, fx=scale, fy=scale)
+    return img

gaze/tracker/findings.md

@@ -0,0 +1,465 @@
+# Findings so far
+
+Our own eye tracker's research notes, newest sections last. They were written while it was a
+separate project (frame-eyes), so they use its names: `fe-trackd` is now `ft-eyes`,
+`fe-bufprobe` is `ft-eyegrab`, `fe_model`/`fe_pupil` are `eyes_model`/`eyes_pupil`, the
+tools are in `lab/` (`fe-score` is `ft-eyes-score`, `fe-replaytest` is `ft-eyes-e2e`,
+`fe-record` and `fe-replay` are `ft-eyes-record` and `ft-eyes-replay`), `fe-live` is the
+gaze service running ft-eyes, and `captures/NAME` is `~/.local/share/frametop/eyes/captures/NAME`.
+
+These were measured on the Frame on 2026-09-28 (SteamVR eyetracking 2.17.10), and all by
+reading only.
+
+## SteamVR's tracker process
+
+- `eyetracking -b CDSP -w .../et_dsp_20250610_03136.weights` runs as the user (steamos),
+  started by SteamVR. The user is in the `cdsp` and `spidev` groups. `ptrace_scope` is 1,
+  so reading another process's fds or memory needs root (pidfd_getfd).
+- Log: `~/.local/share/Steam/logs/eyetracking.txt`. Component names: `CStereoAdspCams`
+  (the eye cameras come in through the audio DSP; "Set framerate 72/90"),
+  `CGazeEstimatorCdsp` / `CDSPGazenet` (the neural net on the compute DSP), and
+  `CEyePoseUKF L/R` (a filter per eye; "Large dt" means it had no measurement for over
+  0.4 s and starts that eye over).
+  - "Failed to grab cdsp input buffer" came up 5,924 times in 6 hours (about 0.3 % of
+    frames at 90 Hz).
+  - "Accept usercal" is its passive calibration from quick mouse clicks.
+- The eye cameras aren't V4L2 devices, and there's no fastrpc node.
+- Open fds that matter:
+  - `/dev/spidev0.1`: modalias `spi:hid-over-spi`, role unknown.
+  - Six udmabufs, all `exp_name: udmabuf`: three of 16 MiB (16777216 B) and three of
+    32 MiB (33554432 B), fds 50, 51, 53, 54, 159 and 169 at the time.
+  - `/dev/shm/eye-server.mmap`: its output.
+  - `/dev/input/event0-7`.
+  - The frames most likely arrive in the udmabufs, which it shares with the DSPs.
+
+## The eye-camera frames (found 2026-09-28, `tools/fe-bufprobe --scan`)
+
+- **The buffers.** The six udmabuf fds are really two buffers, three fds each: a 16 MiB one
+  (inode 1) and a 32 MiB one (inode 2).
+- **The frames.** In the 16 MiB buffer, eight slots sit 0x40000 apart from 0x230000, four
+  per camera: slots 0-3 are camera 0 and slots 4-7 camera 1. Each slot starts with a small
+  block (slot 0's holds a table of floats such as 0.00125, 4.655, 90.0, 1.0, possibly
+  exposure, gain, and frame rate; the others were zero), then a 512x400 8-bit grayscale
+  frame, row stride 512. The frame starts at slot base + 0x40c0 + 0x40 per slot index,
+  plus one more 0x40 for camera 1's slots. Found by the dark lens-rim column lining up;
+  `slot_start()` in fe-bufprobe.
+- **What they show.** Infrared images, one camera per eye, dim (mean about 25-40), with a
+  dark band on one side (the lens rim). One camera saw its eye at a steep angle (squashed
+  pupil near the image edge, big reflections on the white); the other nearly head-on (a
+  round pupil with two small glints in it). Which camera is which eye isn't known yet.
+- **Timing.**
+  - About 90 frames a second per camera, the two within about 0.6 ms of each other.
+  - A frame lands over several milliseconds, in bursts, so a copy taken when the slot
+    "stops changing" can be half old. The reliable rule: a slot is complete when its camera
+    starts writing another slot.
+  - Camera 0 fills its slots in turn (3, 0, 1, 2); camera 1 in a repeating order of eight
+    (7, 5, 4, 6, 5, 7, 6, 4), never the same slot twice in a row.
+  - `--rec` stamps each frame when its slot first changed, polling every 0.3 ms, so times
+    are only good to a few ms.
+  - The cameras run at 90 fps ("Set framerate 90" in the log). The first recorder copied
+    a frame as soon as its camera started the next one and got 94 a second in fit1 and 106
+    in a streaming test. The extras were half-written frames: a frame's last writes can land
+    after the next frame starts, and a slow poll saw both at once. The recorder now copies a
+    frame when its camera starts the frame after next (no slot is rewritten sooner than three
+    frames), ignores late writes to the frame just finished, and saves from a separate
+    thread. fit1 may hold a few percent of torn frames.
+  - Eye tracking stops when the headset is off ("HMD off, stopping eye tracking"), so
+    recordings are empty then.
+- **Which camera is which eye** (capture fit1, 2026-09-29, closing one eye at a time):
+  camera 0 (slots 0-3) is the **right** eye, seen at a steep angle; camera 1 (slots 4-7) is
+  the **left** eye, seen nearly head-on. Both images have the lens rim dark on the left and
+  the lit face on the right.
+- **Why the left eye is lost looking down:** at the keyboard, camera 1 sees only the upper
+  lid and lashes. Camera 0 still catches part of the right eye. It's the camera angle, not
+  the net.
+- **The 32 MiB buffer.** Two 48 KiB regions (0x1522000, 0x1532000) that change every frame,
+  mean bytes about 148, 99 % nonzero. Probably the net's input per eye (crops, maybe not 8-bit
+  pixels). Not decoded.
+- **`tools/fe-session NAME SECONDS`.** Records frames and ft-gaze's samples together, on the
+  same clock (CLOCK_MONOTONIC_RAW). Each frame's nearest SteamVR sample is a median 3.8 ms
+  away.
+
+## Our first pupil finder (`tools/fe_pupil.py`, 2026-09-29)
+
+- Threshold dark (< 30), close glint holes, drop dark regions that touch the image edge
+  (lens rim, background), keep the fullest, darkest ellipse-shaped blob. Glints: spots
+  >= 200 within 1.5 pupil radii. 3.1 ms a frame on the CPU, unoptimised.
+- On fit1 (20 s: open, each eye closed, keyboard, up), pupil found vs SteamVR seeing the
+  eye:
+
+  | Gaze pitch | Right, SteamVR | Right, ours | Left, SteamVR | Left, ours |
+  | --- | --- | --- | --- | --- |
+  | below -20 (keyboard) | 79 % | 35 % | 22 % | 24 % |
+  | -20 to 15 (screens, includes closed-eye time) | 91 % | 89 % | 74 % | 72 % |
+  | above 15 | 100 % | 100 % | 98 % | 99 % |
+
+  It misses the right eye looking down, where the lower lid cuts the pupil. False finds
+  on closed eyes: 0.4 % right, 2.8 % left.
+- A quadratic fit from pupil centre to SteamVR's per-eye gaze, held out by time block:
+  median 4.8 degrees right, 3.0 left. That isn't an accuracy figure yet. fit1 has few
+  distinct gaze points, it uses no glints, and SteamVR's per-eye gaze is itself off by
+  several degrees. It needs a recording against known targets.
+
+## eye-server.mmap (its output)
+
+The file is 324,122 bytes. Only bytes 0x0-0x1f3 are used; the rest is zero. It's packed and
+unaligned, so read it with memcpy. Offsets are also in `~/frametop/gaze/ft-gaze.cpp`.
+
+| Offset | What |
+| --- | --- |
+| 0x38 | u32 sample counter |
+| 0x157 | f64 sample time, CLOCK_MONOTONIC_RAW |
+| 0x15f, 0x16b | set 1: left and right eye direction (3 f32, head space, -Z forward). Filtered; both eyes always share one pitch; a lost eye keeps its yaw |
+| 0x177 | set 1: 6 f32 variances (left 3, right 3; the middle one of each is shared). About 0.0005-0.002 when the eye is seen, 0.015-0.03 when it's lost |
+| 0x18f | set 1 fixation point (3 f32; its length is the vergence distance) |
+| 0x19b, 0x1a7 | set 2: each eye's own direction |
+| 0x1b3 | set 2: 6 f32 variances |
+| 0x1cb | 2 f32, 0..1: openness (0 in a blink) |
+| 0x1d3 | 8 f32: left measurement x, y; right x, y (camera-relative, freezes while that eye isn't seen); then variance of left x, y, right x, y (about 2e-5 on a clear view, rising as lids or lashes get in the way) |
+| 0x0-0x157 | header, plus records that look like the calibration-click channel into the tracker. Never write |
+
+## Accuracy of SteamVR's gaze (this user, this headset)
+
+- **Tonight's practice (71 clicks, 45 minutes):**
+  - raw error: median 5.1 degrees (3.0 in the 21:23 test; it varies by session);
+  - corrected at the press: median 1.5, with 1 in 10 past 3.3;
+  - best smooth correction fitted to the same clicks, each predicted from the rest: 1.7-1.9;
+  - weighting recent clicks more (half-lives from 20 minutes down to 1) didn't help, so
+    there's no slow drift to follow.
+- **Look-to-look:** two looks within 3 degrees of each other, under 5 minutes apart,
+  differ by a median 1.15 degrees (3.0 when 5 or more minutes apart). Jitter within one
+  look is 0.25-0.3.
+- **One eye alone (set 2), against both eyes' gaze:** median 0.8 degrees over a steady
+  look. Per-eye raw errors are large and opposite in yaw: at one spot, left (+8.2, +8.9)
+  and right (-5.0, +5.2) degrees, both (+1.6, +7.1).
+- **Losses:** the left eye was lost 57-64 % of the time looking 30-50 degrees down (at
+  the keyboard, through the gap by the nose), and the right eye never. At screen height
+  both were seen over 98 % of the time. Openness looking down: left 0.45, right 0.65.
+  Harmless for the pointer: ft-gazed ignores looks down past the screens.
+
+## First accuracy test against known targets (practice1, 2026-09-29)
+
+5 minutes, 98 gaze-probe practice clicks, gaze yaw -26..25 and pitch -14..20 degrees. Truth
+is SteamVR's raw gaze at the press plus the angle to the release point. `tools/fe-score.py`
+fits a quadratic per method and scores each click leave-one-out. Pupil = median centre over
+the frames 250-20 ms before the press; no glints yet.
+
+| Method (85 clicks with both pupils found) | Median | 90 % |
+| --- | --- | --- |
+| SteamVR raw | 6.51 | 11.04 |
+| SteamVR + quadratic fit | 1.52 | 3.10 |
+| Ours, right pupil only | 0.74 | 1.48 |
+| Ours, left pupil only | 0.62 | 1.48 |
+| Ours, both pupils averaged | 0.61 | 1.13 |
+
+SteamVR with the probe's live correction: 1.44 median over all 98. The pupil was found
+before 88/98 clicks (right) and 90/98 (left). Caveats: one session with the headset
+never moved (pupil-only mapping breaks when the headset slips; glints should fix that),
+the truth includes the user's own drag precision, and it's offline only.
+
+## Glints and slip (2026-09-29, practice1)
+
+- Two IR LED reflections, a vertical pair 12-19 px apart, sit on the cornea near the pupil.
+  There's no alternating illumination: the pair is in every frame the geometry allows.
+  Before a click: right eye 45/88, left 54/90 (the steep right camera loses the pair on
+  the white when the eye looks across). Bright skin has noise speckle above 200, so a glint
+  only counts if the ring around it is dark (`find_glints`, `glint_pair` in fe_pupil.py).
+- Pupil minus pair midpoint as the feature: 0.84 median, noisier than the pupil alone
+  (0.61), because the pair's position is noisy.
+- Slip method (`tools/fe-score.py`): where the pair is seen, the glint fit gives the gaze,
+  a fit of gaze to pupil position says where the pupil should be, and the difference is
+  the slip. The median over the last 30 s shifts every frame, glints or not. In-session:
+  0.61 median, same as the pupil alone. The estimate stayed within 1-3 px all session.
+- Simulated slip (fit on the first 49 clicks, test on the rest shifted 10 px): pupil alone
+  0.62 -> 2.7-3.1; glint 0.83 unchanged; slip 0.67 unchanged. That checks the math only,
+  for a pure image shift. A real re-seat also tilts and changes the distance.
+- Next test: a second session after taking the headset off and on, scored with
+  `fe-score.py captures/practice1 captures/practice2`.
+
+## Live tracker (2026-09-29)
+
+- `fe-bufprobe --share` (root) copies each finished frame into `/dev/shm/frame-eyes-cams`
+  (0600, the user's); `fe-trackd` (user) finds pupils and glints and writes
+  `/dev/shm/frame-eyes-gaze`; ft-gaze reads that as the source `own`. `tools/fe-live` runs
+  both. The user side never touches SteamVR's buffers.
+- The windowed pupil search gives the same results as the full frame (0.000 px apart on
+  2000 frames per eye), at 0.4 ms instead of 1.4-2.1; with glints, about 1 ms a frame, 90 fps
+  per eye.
+- Replaying practice1's first minute through fe-trackd: 0.64 median at the 14 clicks
+  (0.61 offline with the same calibration; in-sample, so a pipeline check, not accuracy).
+  Sample-to-sample jitter 0.08 degrees; SteamVR's is 0.25-0.3.
+- The calibration covers yaw -26..25 and pitch -14..20 degrees (practice1's clicks). Beyond
+  that the quadratic extrapolates.
+
+## Test 2: a second session after re-seating (practice2, 2026-09-29 15:18)
+
+124 practice clicks over about 4 minutes, headset nudged at about 100 s. The probe stayed on
+SteamVR's mmap2 as its source, but recorded our live gaze at every press. Scored with
+`fe-score.py captures/practice1 captures/practice2` (median degrees):
+
+| Method | Fit on practice1 | Fit within practice2 (leave-one-out) |
+| --- | --- | --- |
+| SteamVR raw | 2.86 | 2.86 |
+| SteamVR + the probe's live correction | 1.37 | 1.37 |
+| SteamVR + quadratic fit | 3.84 | 1.17 |
+| Ours, pupil only | 14.31 | 2.86 |
+| Ours, glint | 3.31 | 1.54 |
+| Ours, slip | 2.68 (live: 2.87) | 1.36 |
+
+- The re-seat moved the eyes 20-30 px in the images: pupil-only goes 14 degrees off. The
+  slip correction takes that to 2.7, but no further. The rest is partly one offset (yaw
+  -1.6, pitch +1.2; removing it leaves 1.43), and an offset from the previous 5 clicks
+  gives 1.31, the same as SteamVR's live correction (1.37).
+- Within one headset position (before the nudge, 45 clicks; after it, 68): pupil only
+  1.01 and 0.99, slip 1.18 and 1.86, SteamVR with the same fit 0.91 and 1.19. So today our
+  tracker is level with SteamVR within a position, not ahead of it as in practice1 (0.61
+  against 1.69). One session was not enough to claim a lead.
+- The slip estimate adds noise: it's worse than no correction within a position, and it
+  wandered (the right eye's jumped 18 px near the end, after the clicks). It comes from the
+  glint fit, which is itself only 1.5-3 degrees good, and the left eye's pair was seen
+  before only 21 of 124 clicks.
+- Live: fe-trackd ran 81-90 fps per eye at 2-3.6 ms a frame alongside VR (1 ms in replay);
+  ft-gaze's `own` came through on every line, about 37 ms old.
+- What would help: a geometric eye model (the eyeball centre from how the pupil ellipse
+  changes shape, as Swirski's method and Pupil Labs' pye3d do) instead of 2-D regression,
+  so that headset movement is modelled and not fitted around. practice1 and practice2
+  together (re-seat plus a nudge) are the benchmark for it.
+
+## The geometric model, and a shift taught by clicks (2026-09-29, practice1 -> practice2)
+
+All offline: calibrate on practice1, score practice2's clicks.
+
+- **Eyeball centre from the pupil ellipses (Swirski-style, weak perspective): worse.** The
+  centre it finds is steady within a session (a few px per 50 s) and moves between the
+  sessions about as the slip does, with a rotation radius of about 60 px (10-12 mm). But
+  it's off from the true shift by up to 8 px (6-7 degrees) on the right eye. Correcting
+  with it gave 4.3-5.8 median, against 2.7 for the glint slip. The cornea's refraction and
+  where you happened to look in the window likely bias it.
+- **The calibration itself carries over.** The best possible pixel shift per eye, fitted
+  on practice2's own clicks with one shift per headset position (before and after the
+  nudge), gives 1.18 held out (shift+scale 1.11, affine 1.07). So practice1's fit is fine
+  if we know the shift; the problem was only estimating it. One shift for the whole
+  session gets only 2.7-2.8, because the nudge moved the eyes again.
+- **How well each estimate finds that shift (px, right x/y, before the nudge):** true
+  (-0.7,-28.6), glints (+2.0,-25.6), eyeball centre (-6.9,-22.2). The glints are off by
+  1-4 px, which is 1-3 degrees; the eyeball centre is worse.
+- **Clicks estimate it best.** Each click says where the pupil should have been for a
+  known gaze, so pupil minus that is the shift. Scored in time order with only earlier
+  clicks:
+
+  | Shift from | Median | 90% |
+  | --- | --- | --- |
+  | Glints only, last 10 or 30 s | 2.68-2.69 | 3.78-4.13 |
+  | Last 3 clicks | 1.29 | 2.80 |
+  | **Last 5 clicks** | **1.18** | 3.09 |
+  | Last 5 clicks + glint slip since | 1.30-1.34 | 3.34-3.55 |
+  | **Last 5 clicks, glints only to catch a jump over 3 px** | **1.22** | **2.33** |
+  | SteamVR + the probe's live correction (same clicks) | 1.37 | 2.71 |
+
+  Adding the glint slip to the clicks' shift adds its noise. Using it only to notice a
+  nudge (then the shift follows the glints until clicks catch up) keeps the median and
+  cuts the tail after a nudge. That's `fe_model.Shift`, and `fe-score`'s `clicks` method
+  reproduces it (1.22 median, 2.33 90%).
+- So on this pair of sessions, ours with click correction is slightly ahead of SteamVR
+  with the probe's click correction: 1.22 against 1.37 median, 2.33 against 2.71 for the
+  worst tenth. One pair of sessions, so not yet a lead (see Test 2).
+- fe-trackd now works this way: the probe's calibration with the source "Own tracker"
+  sends each dot to fe-trackd (`calib-point`), which fits from its own pupil history
+  (`calib-fit`, which replaces the old calibration and clears the shifts), and each
+  practice release sends a `click` that teaches the shift. See fe-trackd's docstring.
+- **The live path reproduces it** (`tools/fe-replaytest captures/practice1 captures/practice2`
+  on the 7i: a scratch fe-trackd, calibrated through `calib-point` from practice1's clicks,
+  then fed practice2's clicks at the recorded pace and scored on what it published in the
+  300 ms before each press). 84 of 98 dots accepted (14 had an eye in under 15 frames),
+  fit 0.59 median. practice2: median 1.21 and 1.23 over two runs (offline 1.22), but 90%
+  3.02 and 2.86 (offline 2.33). The tail: the first click (19.7, nothing learned yet after
+  the re-seat), and two clicks at 192 s and 213 s (7.6 and 10.3) with a settled 5-click
+  shift and no jump. (Offline has the same two, 6.8 and 9.3: see the next section.) The
+  glint jump restarted the right eye's shift 9 times and the left's 4.
+
+## Wide gaze, the left pupil, and false glint jumps (2026-09-29, practice2)
+
+- **The bad clicks were all past the calibration, or right eye only.** practice1's clicks
+  reach yaw 25 and pitch 20; practice2's reach 30 and 25. Offline (median / 90%):
+
+  | Clicks | Ours | SteamVR + probe |
+  | --- | --- | --- |
+  | Inside practice1's range (104) | 1.09 / 1.97 | 1.37 / 2.71 |
+  | Outside it (18-20) | 1.80 / 5.01 | 1.19 / 2.67 |
+  | Both eyes (99) | 1.09 / 1.98 | 1.40 / 2.83 |
+  | Right eye only (23) | 1.66 / 4.05 | 0.99 / 2.47 |
+
+  A fit that goes linear past its data, more ridge, or a linear fit didn't help outside.
+- **The left eye was lost at every click past about 19 degrees left.** The pupil is still
+  mid-image (x 293-310 of 512), but the left camera's image is dark from x 0 to about 360,
+  and the 7 px closing (which heals glint holes) joined the pupil to that background, which
+  touches the edge, so it was dropped. `fe_pupil` now retries with a 3 px closing when
+  nothing is found: left eye found in 97% of the frames before practice2's clicks (was
+  79%; 181 of 217 frames at 20+ degrees, was 16), right eye unchanged, centres moved at
+  most 0.16 px. The right eye still needs the 7 px (3 px alone: 96% against 98%).
+- **Those pupils are accurate.** Within one headset position (practice2 after the nudge,
+  68 clicks, leave-one-out, so calibrated out there too): left eye alone 0.70 / 1.39 below
+  19 degrees left and 0.87 / 1.14 beyond; right eye 1.27 / 2.13 and 2.09 / 6.42; both
+  averaged 0.85 / 1.30 and 1.18 / 5.14. Calibrated where you look, the left eye is our best.
+- **But practice1's calibration has 4 clicks per eye beyond 19 degrees**, so the newly seen
+  left eye extrapolates there (3.64 median alone, right 1.78), and practice1 -> practice2
+  got a worse tail: 1.20 / 3.20 offline (1.22 / 2.33 when the left eye was simply lost
+  there). Weighting the eyes, or leaving out an eye or a click past the calibrated range,
+  didn't recover it. The fix is coverage: the probe's calibration for the Own tracker now
+  puts its dots on an oval out to the `Calibration ring` angle each way (the ring was
+  limited by the window's height and never reached the sides). A first try put them at
+  the practice area's corners, which in a large window were too far to look at while
+  facing the centre.
+- **The glint jumps.** Offline (checked once per click) there were 2 per eye, 3 of 4 real
+  (the next click found the shift the glints claimed). Live, bad glint pairs made the right
+  eye's estimate leap by up to 68 px for under a second, 20 times between clicks, and the
+  shift restarted 8-9 times. `Shift` now takes a jump only once it has held for 1 s
+  (JUMP_HOLD) and ignores ones over 40 px (JUMP_MAX). Live replay: shift restarts 1 (right)
+  and 0 (left), biggest leap between clicks 4.9 px, output steps over 10 degrees 86 -> 32.
+  Offline with the hold: 1.19 / 3.35.
+- Live replay with both changes: 94 of 98 calibration dots taken (83-84 before), practice2
+  1.28 / 3.16. The tail stays until a calibration covers the practice area.
+- `fe-score` now reads each capture's own `practice.jsonl` (cut from the probe's log by
+  `fe-score.py --clicks`, or the first scoring on the Frame), so the 7i can rebuild features.
+
+## Session 3 (2026-09-29 22:04-22:10, live, SteamVR driving)
+
+The calibration and 84 practice clicks went to SteamVR (the probe's tracker toggle was
+left on SteamVR), so fe-trackd got no dots or clicks and kept practice1's calibration. The
+probe still logged our gaze at 79 presses. SteamVR + the probe's correction: 1.19 median,
+2.61 90% (raw 2.18 / 4.30). Ours with practice1's calibration and no clicks: 12.5 median;
+with a stand-in for the click shift (the median offset of the previous 5 clicks, in gaze
+angles rather than per eye in pixels): 1.38 / 2.38. No frames were recorded.
+
+## Session 4: the Own tracker driving (2026-09-29 22:12-22:22, live)
+
+Fresh calibration from the probe with the Own tracker: 27 dots on an oval out to 20
+degrees (30 of 31 attempts accepted; one had no right eye). Then 136 practice clicks, all
+with both eyes, each teaching the shift. The probe logged SteamVR at 114 of the presses,
+and its correction learned from the same drags, so the comparison is fair:
+
+| At the same 114 presses, to where you let go | Median | 90% |
+| --- | --- | --- |
+| **Ours** | **0.59** | **1.50** |
+| SteamVR + the probe's correction | 0.83 | 2.02 |
+| SteamVR raw | 3.49 | 5.21 |
+
+Ours was closer on 76 of 114. Ours at the press (what the dot showed, all 136): 0.67
+median, 1.37 90%, and steady from the first 10 clicks (0.71) on; SteamVR's correction
+took about 30 clicks to get under 1 degree. No click changed an eye's shift by more than
+6 px. The user: "MUCH improved". No frames were recorded, and the headset wasn't nudged or
+re-seated, so this is within one position; the cross-session question is still open.
+
+## Session 5: off and on again, no recalibration (2026-09-29 22:26-22:31, live)
+
+Session 4's calibration and shifts, headset taken off and put back on, then 132 practice
+clicks with the Own tracker driving. The re-seat moved the eyes about 15 px (right) and
+33 px (left) in the images.
+
+- Before the first taught click, the glints had moved the right eye's shift to within
+  about 4 px and the left's about two thirds of the way. Clicks 1-4 were still 11-17
+  degrees off, and the probe refused to teach them (its 6-degree limit on a lesson), so
+  the first taught click was the 5th (5.4 degrees). It restarted each eye's history as
+  designed; clicks 6, 7, 8: 2.8, 1.3, 0.4.
+- After that (clicks 6 on, 127, to where you let go): ours 0.58 median, 1.42 90%, the
+  same as within one position (session 4: 0.59); SteamVR + the probe's correction 2.94 /
+  5.73 (SteamVR raw drifted from 3.5 to 4.7 through the session). Ours closer on 117 of 132.
+- Changes: the probe lets the Own tracker learn from drags up to 25 degrees
+  (OWN_LEARN_MAX), and starts on the Own tracker when fe-trackd answers; its calibration
+  header names the tracker. fe-trackd restarts an eye's shift history at the next click
+  after frames stop for 3 s (the headset off) and after its own restart, glints or not.
+  Replay regression (fe-replaytest practice1 practice2): 1.28 / 3.16, as before.
+
+## Weighting the eyes (2026-09-29, practice2 after the nudge)
+
+One headset position, 64 clicks with both eyes, leave-one-out: plain average 0.84 / 1.59;
+left eye alone 0.73 / 1.36; right alone 1.32 / 3.05; weighted by each eye's inverse
+residual variance on its own calibration 0.75 / 1.26. Not in fe-trackd yet.
+
+## Next steps (2026-09-29, after a literature search; sources in the session report)
+
+Ranked by expected gain for the effort, checked against our own numbers:
+1. Weight the eyes by each one's calibration residuals (above: 0.84 -> 0.75 median). S.
+2. A one-dot re-seat check when frames come back after a gap (Varjo recalibrates with one
+   dot at every put-on): one look and press teaches both shifts before the first real
+   click, instead of 11-17 degree first clicks. S.
+3. Our tracker as a source for the Frametop pointer (ft-gazed): session 5 beat SteamVR
+   across a re-seat. M. Done 2026-09-30 (below).
+4. Record frames during live tests (fe-session), so each can be replayed. S (disk: about
+   2 GB a minute).
+5. A less biased glint slip estimate: ours is off by 1-4 px even over hundreds of frames,
+   so it's bias, not noise; try taking out the part of the glint midpoint that follows the
+   pupil (regressed on calibration data) before using it. S-M, offline first.
+6. Smooth-pursuit calibration (a moving dot): dense labels out to the edge in about 20 s,
+   for wider coverage. M.
+7. Sub-pixel edge ellipse refit with RANSAC for the steep right eye (our weaker eye,
+   1.32 against 0.73). S-M.
+8. Later, if needed: learned pupil segmentation (EllSeg, RITnet: MIT) on the GPU through
+   ncnn, a 3-D cornea model from the two glints, or a per-user network trained on the
+   residuals across re-seats. L. Not recommended: the eyeball-centre model (tried, and our
+   steep camera and +-20 degree range are outside its published conditions). PuRe,
+   PuReST, ElSe, and ExCuSe are licensed for non-commercial use only.
+
+## Quick wins from the next steps (2026-09-29, late)
+
+- Eye weighting (1): `Calibration.spread` is each eye's RMS miss on its own calibration
+  dots, and `combine` weights by its inverse square (floor 0.3 degrees). fe-trackd, fe-score,
+  and so fe-replaytest use it; older calibrations get it from their saved dots. The
+  22:16 calibration: right 1.48, left 1.08, so the left eye counts about twice as much.
+  practice1 -> practice2 offline is unchanged (1.20 / 3.35: that tail is extrapolation).
+- Re-seat check (2): fe-trackd's status says when the next click will start an eye's shift
+  over; the probe then shows one centre dot, and a press on it sends that click. Checked
+  on the 7i (a pending re-seat at start on both eyes, cleared by one click); the probe's
+  screen for it wasn't seen (the web view didn't connect).
+- Recording (4): `tools/fe-record` copies every shared frame (9 s of replay: 1620 frames,
+  none dropped, all identical to the source); `fe-live --record NAME` runs it alongside.
+
+## The Frametop pointer, and the eyes on live clicks (2026-09-30)
+
+ft-gazed (`~/frametop/gaze`) can now use our tracker: `GAZE_TRACKER=own`, the Eye tracker
+setting on the Gaze page of Frametop Input Settings. A mouse nudge before a click reaches
+fe-trackd as a click. The nudge's raw gaze is one ft-gazed sent, so ft-gazed finds when
+that look was, and fe-trackd keeps 12 s of pupils instead of 5, because the helper sends a
+nudge up to 10 s after the look.
+
+`GAZE_EYE` (auto, left, right) weights the eyes there, from each eye's own gaze. Replayed on
+the 306 live clicks of sessions 4 and 5 (`clicks.jsonl`: each eye's pupil and shift just
+before the click, so each is a fresh test):
+- Each eye alone: left 0.96 median (mean 1.17), right 1.11 (1.26). The eyes' RMS misses
+  were about equal (1.43, 1.46), unlike practice2's leave-one-out (0.73, 1.32).
+- Both eyes: 0.65 (0.77) evenly. By the calibration's spread (the 22:16 one: left counts
+  about twice): 0.63 (0.81). By each eye's RMS miss at its last 5, 10, or 20 clicks: 0.66
+  (0.79-0.81).
+- By share of the right eye: 0.3 gives 0.68, 0.5 gives 0.65, 0.7 gives 0.81.
+- The eyes' yaw errors are correlated -0.37: they partly cancel, which is why two eyes
+  beat either one by a third.
+
+So a bias leans instead of choosing: Left or Right counts that eye twice. Auto starts
+even and weights by each eye's RMS miss at its last 20 nudges, once each has 5. On
+SteamVR's side the calibration's own fit picked the wrong eye (its dots: left 1.78, right
+1.88; new spots: left 2.50, right 1.63), so auto learns from nudges, not the fit.
+fe-trackd's own `combine` still uses the spread, for the probe.
+
+## Valve's tracker
+
+It can't be the starting point, legally or practically:
+
+- **No source.** `/opt/steamvr/tools/eyetracking/bin/linuxarm64/eyetracking` is a
+  stripped aarch64 binary. The paths left in it (`/data/src/eyetracking/eyetracklib/...`)
+  are Valve's build machine's.
+- **The net is just numbers.** `et_dsp_20250610_03136.weights` is 393,600 bytes of raw
+  floats (about 98,000 parameters), with no header or architecture. The layer layout
+  lives in the binary and in the program it loads onto the compute DSP (`CDSPGazenet`).
+  Rebuilding it would mean reverse engineering both.
+- **License.** SteamVR is Valve's proprietary software, used under the Steam Subscriber
+  Agreement. That agreement doesn't allow reverse engineering, decompiling, modifying, or
+  redistributing it, except where the law allows. `third_party_legal_notices.txt`
+  covers only the open libraries it uses (Ceres, protobuf, ...), not the tracker. Putting
+  their code or weights in a GitHub repo would be redistribution. (Not legal advice.)
+- **Not much to gain.** Their net is small and tuned to their cameras. Improving it would
+  need the same thing our own tracker needs: your eye images with known gaze, for
+  training.
+
+What we can use: its public output (the mmap, read-only), as a baseline and as labels.
+Anything published and openly licensed is also fair game: papers and open-source pupil
+detectors (check each one's license before using its code).

gaze/tracker/frametop-eyegrab.service

@@ -0,0 +1,36 @@
+# Template: gaze/tracker/install.sh fills in @UID@ and @GID@ (the Frametop user) and installs
+# it to /etc/systemd/system.
+[Unit]
+Description=Frametop eye-camera frames for our own eye tracker (read-only copies from SteamVR's eyetracking)
+Documentation=file://@REPO@/gaze/README.md
+
+[Service]
+# Idle (no frames copied, none of the tracker's buffers held) until ft-eyes or
+# ft-eyes-record touches the want file; see ft-eyegrab.c.
+ExecStart=/etc/frametop/ft-eyegrab --share /dev/shm/frametop-eyes-cams --owner @UID@:@GID@ --want /dev/shm/frametop-eyes-want
+Restart=on-failure
+RestartSec=5
+Nice=5
+# Root only for what reading another process's buffers needs: CAP_SYS_PTRACE (pidfd_getfd),
+# CAP_DAC_READ_SEARCH (its /proc/PID/fd), and CAP_CHOWN (the shared file goes to the user).
+CapabilityBoundingSet=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_CHOWN
+AmbientCapabilities=
+NoNewPrivileges=yes
+ProtectSystem=strict
+ProtectHome=yes
+ReadWritePaths=/dev/shm
+PrivateNetwork=yes
+RestrictAddressFamilies=AF_UNIX
+ProtectKernelModules=yes
+ProtectKernelTunables=yes
+ProtectControlGroups=yes
+ProtectClock=yes
+ProtectHostname=yes
+RestrictNamespaces=yes
+RestrictRealtime=yes
+LockPersonality=yes
+MemoryDenyWriteExecute=yes
+SystemCallArchitectures=native
+
+[Install]
+WantedBy=multi-user.target

gaze/tracker/ft-eyegrab.c

@@ -0,0 +1,605 @@
+/*
+ * ft-eyegrab: the eye-camera frames for our own eye tracker (ft-eyes), copied read-only out
+ * of the DMA-BUFs SteamVR's eyetracking process holds. Runs as root (pidfd_getfd needs
+ * CAP_SYS_PTRACE; ptrace_scope is 1): the system service frametop-eyegrab.service runs
+ * --share, installed by gaze/tracker/install.sh. The other modes are for finding the frames
+ * again after a SteamVR update (run them with sudo).
+ *
+ *   ft-eyegrab --share PATH [--owner UID:GID] [--want FILE]
+ *                              keep the latest frames of both cameras in PATH (shared memory,
+ *                              0600, owned by UID:GID, or the sudo user) for ft-eyes; follows
+ *                              the tracker through SteamVR restarts. With --want, only while
+ *                              FILE (a regular file owned by that user) was touched in the
+ *                              last WANT_FRESH seconds: ft-eyes and ft-eyes-record touch it
+ *                              every second, so nothing is copied, and none of the tracker's
+ *                              buffers are held, while nobody reads the frames
+ *   ft-eyegrab                 list the buffers
+ *   ft-eyegrab --scan [N]      N snapshots (default 40) about 11 ms apart: which 4 KiB pages
+ *                              change, merged into regions, with byte statistics for each
+ *   ft-eyegrab --dump I OFF LEN FILE
+ *                              copy LEN bytes at OFF of buffer I (from the list) to FILE
+ *   ft-eyegrab --seq I OFF LEN FRAMES DIR
+ *                              FRAMES copies of that region, one each time it changes, to
+ *                              DIR/NNNN.raw, with DIR/times.txt (CLOCK_MONOTONIC_RAW)
+ *   ft-eyegrab --rec SECONDS DIR
+ *                              every new eye-camera frame for SECONDS: DIR/frames.raw (512x400
+ *                              8-bit frames back to back) and DIR/index.txt, one line per frame:
+ *                              "<n> <slot> <camera 0|1> <CLOCK_MONOTONIC_RAW time seen>"
+ *                              (lab/ft-eyes-record does the same from the shared frames,
+ *                              without root)
+ *
+ * The eye frames (found with --scan): in the 16 MiB buffer, eight slots 0x40000 apart from
+ * 0x230000, four per camera (slots 0-3, 4-7). Each slot starts with a small block, then a
+ * 512x400 8-bit image at 0x40c0 + 0x40 per slot, and one more 0x40 for the second camera's.
+ *
+ * Only reads the tracker's buffers. They are borrowed with pidfd_getfd and mapped PROT_READ; nothing is
+ * written, and the process isn't stopped or signalled. Reads can tear while the DSP writes.
+ */
+#define _GNU_SOURCE
+#include <dirent.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <pthread.h>
+#include <signal.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+#include <time.h>
+#include <unistd.h>
+
+#define MAXBUF 32
+#define PAGE 4096
+
+typedef struct {
+    int xfd, fd;
+    size_t size;
+    unsigned long ino;
+    const uint8_t *p;
+} buf_t;
+
+static buf_t bufs[MAXBUF];
+static int nbufs;
+
+static double now(void) {
+    struct timespec ts;
+    clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
+    return ts.tv_sec + ts.tv_nsec * 1e-9;
+}
+
+static int find_tracker(void) {
+    DIR *d = opendir("/proc");
+    struct dirent *e;
+    int pid = -1;
+    while (d && (e = readdir(d))) {
+        char path[300], cmd[512];
+        if (e->d_name[0] < '0' || e->d_name[0] > '9') continue;
+        snprintf(path, sizeof path, "/proc/%s/cmdline", e->d_name);
+        FILE *f = fopen(path, "r");
+        if (!f) continue;
+        size_t n = fread(cmd, 1, sizeof cmd - 1, f);
+        fclose(f);
+        cmd[n] = 0;
+        if (strstr(cmd, "tools/eyetracking/bin/") && strstr(cmd, "/eyetracking")) {
+            pid = atoi(e->d_name);
+            break;
+        }
+    }
+    if (d) closedir(d);
+    return pid;
+}
+
+static int open_bufs(int pid) {
+    int pidfd = syscall(SYS_pidfd_open, pid, 0);
+    if (pidfd < 0) {
+        perror("pidfd_open");
+        return -1;
+    }
+    char dir[64];
+    snprintf(dir, sizeof dir, "/proc/%d/fd", pid);
+    DIR *d = opendir(dir);
+    struct dirent *e;
+    while (d && (e = readdir(d)) && nbufs < MAXBUF) {
+        char link[320], target[256];
+        if (e->d_name[0] == '.') continue;
+        snprintf(link, sizeof link, "%s/%s", dir, e->d_name);
+        ssize_t n = readlink(link, target, sizeof target - 1);
+        if (n <= 0) continue;
+        target[n] = 0;
+        if (strncmp(target, "/dmabuf:", 8) != 0) continue;
+        int xfd = atoi(e->d_name);
+        int fd = syscall(SYS_pidfd_getfd, pidfd, xfd, 0);
+        if (fd < 0) {
+            fprintf(stderr, "pidfd_getfd %d: %s\n", xfd, strerror(errno));
+            continue;
+        }
+        struct stat st;
+        fstat(fd, &st);
+        off_t size = lseek(fd, 0, SEEK_END);
+        void *p = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
+        if (p == MAP_FAILED) {
+            fprintf(stderr, "mmap fd %d (%lld bytes): %s\n", xfd, (long long)size, strerror(errno));
+            close(fd);
+            continue;
+        }
+        bufs[nbufs++] = (buf_t){xfd, fd, (size_t)size, (unsigned long)st.st_ino, p};
+    }
+    if (d) closedir(d);
+    close(pidfd);
+    return nbufs;
+}
+
+static uint64_t page_hash(const uint8_t *p) {
+    const uint64_t *q = (const uint64_t *)p;
+    uint64_t h = 1469598103934665603ull;
+    for (int i = 0; i < PAGE / 8; i += 4) h = (h ^ q[i]) * 1099511628211ull;  // every 4th word
+    return h;
+}
+
+static void stats(const uint8_t *p, size_t n, double *mean, int *lo, int *hi, double *nonzero) {
+    uint64_t sum = 0, nz = 0;
+    int a = 255, b = 0;
+    for (size_t i = 0; i < n; i++) {
+        sum += p[i];
+        nz += p[i] != 0;
+        if (p[i] < a) a = p[i];
+        if (p[i] > b) b = p[i];
+    }
+    *mean = n ? (double)sum / n : 0;
+    *lo = a, *hi = b, *nonzero = n ? (double)nz / n : 0;
+}
+
+static void scan(int snaps) {
+    for (int b = 0; b < nbufs; b++) {
+        size_t pages = bufs[b].size / PAGE;
+        uint64_t *prev = calloc(pages, 8), *cur = calloc(pages, 8);
+        int *changes = calloc(pages, sizeof(int));
+        for (size_t i = 0; i < pages; i++) prev[i] = page_hash(bufs[b].p + i * PAGE);
+        double t0 = now();
+        for (int s = 1; s < snaps; s++) {
+            usleep(11000);
+            for (size_t i = 0; i < pages; i++) {
+                cur[i] = page_hash(bufs[b].p + i * PAGE);
+                if (cur[i] != prev[i]) changes[i]++;
+                prev[i] = cur[i];
+            }
+        }
+        double dt = now() - t0;
+        printf("buffer %d (fd %d, %zu bytes, ino %lu): %d snapshots over %.2f s\n", b, bufs[b].xfd, bufs[b].size,
+               bufs[b].ino, snaps, dt);
+        // Regions: runs of pages that changed at least once (gaps of up to 2 pages merged).
+        size_t i = 0;
+        int regions = 0;
+        while (i < pages) {
+            if (!changes[i]) {
+                i++;
+                continue;
+            }
+            size_t start = i, end = i, gap = 0;
+            int most = 0;
+            long total = 0;
+            for (; i < pages; i++) {
+                if (changes[i]) {
+                    end = i, gap = 0;
+                    total += changes[i];
+                    if (changes[i] > most) most = changes[i];
+                } else if (++gap > 2) {
+                    break;
+                }
+            }
+            size_t off = start * PAGE, len = (end - start + 1) * PAGE;
+            double mean, nz;
+            int lo, hi;
+            stats(bufs[b].p + off, len, &mean, &lo, &hi, &nz);
+            printf("  region 0x%08zx +0x%zx (%zu KiB): changed in up to %d of %d intervals (avg %.1f); "
+                   "bytes mean %.1f min %d max %d nonzero %.0f%%\n",
+                   off, len, len / 1024, most, snaps - 1, (double)total / (end - start + 1), mean, lo, hi, nz * 100);
+            regions++;
+        }
+        if (!regions) {
+            double mean, nz;
+            int lo, hi;
+            stats(bufs[b].p, bufs[b].size, &mean, &lo, &hi, &nz);
+            printf("  no change; bytes mean %.1f min %d max %d nonzero %.1f%%\n", mean, lo, hi, nz * 100);
+        }
+        free(prev), free(cur), free(changes);
+    }
+}
+
+#define EYE_W 512
+#define EYE_H 400
+#define EYE_SLOTS 8
+
+static size_t slot_start(int k) {
+    return 0x230000 + (size_t)k * 0x40000 + 0x40c0 + (size_t)k * 0x40 + (k >= 4 ? 0x40 : 0);
+}
+
+// A cheap fingerprint of a frame: 256 words spread over it (a new frame changes nearly all).
+static uint64_t frame_sig(const uint8_t *p) {
+    uint64_t h = 1469598103934665603ull, w;
+    for (int i = 0; i < 256; i++) {
+        memcpy(&w, p + (size_t)i * (EYE_W * EYE_H / 256), 8);
+        h = (h ^ w) * 1099511628211ull;
+    }
+    return h;
+}
+
+static volatile sig_atomic_t stop_rec;
+static void on_stop(int sig) { (void)sig; stop_rec = 1; }
+
+// The poller copies finished frames into a ring; a writer thread saves them, so a slow disk
+// write never delays the polling.
+#define RING 128
+static struct {
+    uint8_t *frames;
+    int slot[RING];
+    double time[RING];
+    size_t head, tail, dropped;  // head: next to fill (poller); tail: next to save (writer)
+    int done;
+    pthread_mutex_t mu;
+    pthread_cond_t cv;
+    FILE *f, *ix;
+} ring = {.mu = PTHREAD_MUTEX_INITIALIZER, .cv = PTHREAD_COND_INITIALIZER};
+
+static void *ring_writer(void *arg) {
+    (void)arg;
+    size_t fsize = EYE_W * EYE_H, n = 0;
+    pthread_mutex_lock(&ring.mu);
+    for (;;) {
+        while (ring.tail == ring.head && !ring.done) pthread_cond_wait(&ring.cv, &ring.mu);
+        if (ring.tail == ring.head) break;
+        size_t i = ring.tail % RING;
+        pthread_mutex_unlock(&ring.mu);
+        fwrite(ring.frames + i * fsize, 1, fsize, ring.f);
+        fprintf(ring.ix, "%zu %d %d %.6f\n", n++, ring.slot[i], ring.slot[i] >= 4, ring.time[i]);
+        pthread_mutex_lock(&ring.mu);
+        ring.tail++;
+    }
+    pthread_mutex_unlock(&ring.mu);
+    return NULL;
+}
+
+static void ring_put(const uint8_t *frame, int slot, double t) {
+    size_t fsize = EYE_W * EYE_H;
+    pthread_mutex_lock(&ring.mu);
+    int full = ring.head - ring.tail >= RING;
+    pthread_mutex_unlock(&ring.mu);
+    if (full) {
+        ring.dropped++;
+        return;
+    }
+    size_t i = ring.head % RING;
+    memcpy(ring.frames + i * fsize, frame, fsize);
+    ring.slot[i] = slot, ring.time[i] = t;
+    pthread_mutex_lock(&ring.mu);
+    ring.head++;
+    pthread_cond_signal(&ring.cv);
+    pthread_mutex_unlock(&ring.mu);
+}
+
+static int eye_buffer(void) {
+    for (int i = 0; i < nbufs; i++)
+        if (bufs[i].size == 16777216) return i;
+    return -1;
+}
+
+// Calls done(frame, slot, time) for every complete eye-camera frame until `seconds` pass
+// (forever if negative), a stop signal comes, the tracker process goes away, or keep()
+// (checked about every 0.25 s, when given) says to stop.
+//
+// A frame lands over several milliseconds, in bursts, and its last bursts can come after
+// the camera has started its next frame. A slot isn't rewritten until at least three frames
+// later (camera 0 cycles 3,0,1,2; camera 1 7,5,4,6,5,7,6,4), so a frame is passed on when
+// its camera starts the frame after next. Changes to the slot just finished are late bursts,
+// not a new frame. A frame's time is when its slot first changed.
+static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8_t *, int, double),
+                        int (*keep)(void)) {
+    uint64_t sig[EYE_SLOTS];
+    double first[EYE_SLOTS];
+    int cur[2] = {-1, -1}, prev[2] = {-1, -1};
+    for (int k = 0; k < EYE_SLOTS; k++) sig[k] = frame_sig(bufs[b].p + slot_start(k)), first[k] = 0;
+    double start = now(), checked = start, kept = start;
+    char proc[64];
+    snprintf(proc, sizeof proc, "/proc/%d", pid);
+    while ((seconds < 0 || now() - start < seconds) && !stop_rec) {
+        double t = now();
+        if (t - checked > 1.0) {  // the tracker restarted: its buffers are stale
+            struct stat st;
+            if (stat(proc, &st) != 0) return;
+            checked = t;
+        }
+        if (keep && t - kept > 0.25) {
+            if (!keep()) return;
+            kept = t;
+        }
+        for (int k = 0; k < EYE_SLOTS; k++) {
+            uint64_t s = frame_sig(bufs[b].p + slot_start(k));
+            if (s == sig[k]) continue;
+            sig[k] = s;
+            int cam = k >= 4;
+            if (k == cur[cam] || k == prev[cam]) continue;  // landing, or a late burst
+            if (prev[cam] >= 0) done(bufs[b].p + slot_start(prev[cam]), prev[cam], first[prev[cam]]);
+            prev[cam] = cur[cam];
+            cur[cam] = k;
+            first[k] = t;
+        }
+        usleep(300);
+    }
+}
+
+static void rec_frame(const uint8_t *frame, int slot, double t) { ring_put(frame, slot, t); }
+
+static int rec(double seconds, const char *dir, int pid) {
+    int b = eye_buffer();
+    if (b < 0) {
+        fprintf(stderr, "no 16 MiB buffer\n");
+        return 1;
+    }
+    // Frames stream to disk (about 37 MB/s), so a long recording doesn't fill memory.
+    // Ctrl-C or SIGTERM ends it early and keeps what was recorded.
+    char path[512];
+    snprintf(path, sizeof path, "%s/frames.raw", dir);
+    ring.f = fopen(path, "wb");
+    snprintf(path, sizeof path, "%s/index.txt", dir);
+    ring.ix = fopen(path, "w");
+    ring.frames = malloc((size_t)RING * EYE_W * EYE_H);
+    if (!ring.f || !ring.ix || !ring.frames) {
+        perror(dir);
+        return 1;
+    }
+    setvbuf(ring.f, NULL, _IOFBF, 4 << 20);
+    signal(SIGINT, on_stop);
+    signal(SIGTERM, on_stop);
+    pthread_t writer;
+    pthread_create(&writer, NULL, ring_writer, NULL);
+    double start = now();
+    poll_frames(b, seconds, pid, rec_frame, NULL);
+    pthread_mutex_lock(&ring.mu);
+    ring.done = 1;
+    pthread_cond_signal(&ring.cv);
+    pthread_mutex_unlock(&ring.mu);
+    pthread_join(writer, NULL);
+    fclose(ring.f), fclose(ring.ix);
+    printf("%zu frames in %.1f s to %s", ring.head, now() - start, dir);
+    if (ring.dropped) printf(" (%zu dropped: disk too slow)", ring.dropped);
+    printf("\n");
+    free(ring.frames);
+    return 0;
+}
+
+// --- --share: the latest frames in shared memory for the live tracker ---
+//
+// The file (SHARE_PATH, mode 0600, owned by the --owner user) is a header, then SHARE_SLOTS
+// entries per camera. Each entry is a 64-byte head and one 512x400 frame. Frame n of camera
+// c goes in entry c * SHARE_SLOTS + n % SHARE_SLOTS. The head's `seq` is odd while it's
+// written (read it before and after copying, and retry if it changed or was odd), and
+// count[c] is how many frames camera c has published. tracker_pid is 0 while no frames
+// come (nobody wants them, or SteamVR's tracker isn't running).
+//
+// This keeps the tracker's own buffers behind root: the user side only ever sees copies.
+#define SHARE_SLOTS 8
+#define SHARE_MAGIC 0x31434546u  // "FEC1"
+#define WANT_FRESH 3.0           // seconds a touch of the --want file lasts
+
+typedef struct {
+    uint32_t magic, version, width, height, slots, entry_size;
+    volatile uint64_t count[2];
+    uint32_t tracker_pid, pad0;
+    uint8_t pad[16];
+} share_head_t;
+
+typedef struct {
+    volatile uint64_t seq;
+    double t;
+    uint64_t n;
+    uint32_t cam, slot;
+    uint8_t pad[32];
+} share_entry_t;
+
+_Static_assert(sizeof(share_head_t) == 64, "share header");
+_Static_assert(sizeof(share_entry_t) == 64, "share entry");
+
+static uint8_t *share;
+static const char *want_path;
+static uid_t owner_uid = (uid_t)-1;
+static gid_t owner_gid = (gid_t)-1;
+
+static void share_frame(const uint8_t *frame, int slot, double t) {
+    share_head_t *h = (share_head_t *)share;
+    int cam = slot >= 4;
+    uint64_t n = h->count[cam];
+    size_t esize = sizeof(share_entry_t) + EYE_W * EYE_H;
+    share_entry_t *e = (share_entry_t *)(share + sizeof *h + (cam * SHARE_SLOTS + n % SHARE_SLOTS) * esize);
+    e->seq++;
+    __atomic_thread_fence(__ATOMIC_RELEASE);
+    memcpy((uint8_t *)(e + 1), frame, EYE_W * EYE_H);
+    e->t = t, e->n = n, e->cam = cam, e->slot = slot;
+    __atomic_thread_fence(__ATOMIC_RELEASE);
+    e->seq++;
+    __atomic_thread_fence(__ATOMIC_RELEASE);
+    h->count[cam] = n + 1;
+}
+
+// Someone reads the frames: the want file was touched lately. It must be a regular file
+// (lstat: a link isn't followed) owned by the frames' owner, so no one else can turn this on.
+static int wanted(void) {
+    if (!want_path) return 1;
+    struct stat st;
+    if (lstat(want_path, &st) != 0 || !S_ISREG(st.st_mode)) return 0;
+    if (owner_uid != (uid_t)-1 && st.st_uid != owner_uid) return 0;
+    struct timespec ts;
+    clock_gettime(CLOCK_REALTIME, &ts);
+    double age = (ts.tv_sec - st.st_mtim.tv_sec) + (ts.tv_nsec - st.st_mtim.tv_nsec) * 1e-9;
+    return age < WANT_FRESH;
+}
+
+static void close_bufs(void) {
+    for (int i = 0; i < nbufs; i++) munmap((void *)bufs[i].p, bufs[i].size), close(bufs[i].fd);
+    nbufs = 0;
+}
+
+static int share_loop(const char *path) {
+    size_t esize = sizeof(share_entry_t) + EYE_W * EYE_H;
+    size_t size = sizeof(share_head_t) + 2 * SHARE_SLOTS * esize;
+    unlink(path);
+    int fd = open(path, O_RDWR | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC, 0600);
+    if (fd < 0 || ftruncate(fd, size) != 0) {
+        perror(path);
+        return 1;
+    }
+    if (owner_uid != (uid_t)-1 && fchown(fd, owner_uid, owner_gid) != 0) perror("fchown");
+    share = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+    close(fd);
+    if (share == MAP_FAILED) {
+        perror("mmap");
+        return 1;
+    }
+    share_head_t *h = (share_head_t *)share;
+    *h = (share_head_t){.magic = SHARE_MAGIC, .version = 1, .width = EYE_W, .height = EYE_H,
+                        .slots = SHARE_SLOTS, .entry_size = (uint32_t)esize};
+    signal(SIGINT, on_stop);
+    signal(SIGTERM, on_stop);
+    fprintf(stderr, "ft-eyegrab: sharing frames in %s%s%s\n", path, want_path ? " while wanted by " : "",
+            want_path ? want_path : "");
+    int pid = -1, idle = -1, missing = 0;
+    while (!stop_rec) {
+        if (!wanted()) {
+            // Nobody reads the frames: copy nothing, and let go of the tracker's buffers.
+            if (idle != 1) fprintf(stderr, "ft-eyegrab: idle (nobody wants frames)\n"), idle = 1;
+            close_bufs();
+            pid = -1;
+            h->tracker_pid = 0;
+            usleep(250000);
+            continue;
+        }
+        if (nbufs == 0 && ((pid = find_tracker()) < 0 || open_bufs(pid) <= 0 || eye_buffer() < 0)) {
+            // SteamVR's tracker isn't running (yet, or again).
+            if (!missing) fprintf(stderr, "ft-eyegrab: waiting for SteamVR's eyetracking\n"), missing = 1;
+            close_bufs();
+            h->tracker_pid = 0;
+            sleep(2);
+            continue;
+        }
+        if (idle != 0 || missing) fprintf(stderr, "ft-eyegrab: copying frames from eyetracking %d\n", pid);
+        idle = 0, missing = 0;
+        h->tracker_pid = pid;
+        poll_frames(eye_buffer(), -1, pid, share_frame, wanted);
+        struct stat st;
+        char proc[64];
+        snprintf(proc, sizeof proc, "/proc/%d", pid);
+        if (!stop_rec && stat(proc, &st) != 0) {
+            fprintf(stderr, "ft-eyegrab: eyetracking %d went away; waiting for it\n", pid);
+            close_bufs();
+            h->tracker_pid = 0;
+        }
+    }
+    close_bufs();
+    unlink(path);
+    return 0;
+}
+
+static int dump(int b, size_t off, size_t len, const char *file) {
+    if (b < 0 || b >= nbufs || off + len > bufs[b].size) {
+        fprintf(stderr, "out of range\n");
+        return 1;
+    }
+    FILE *f = fopen(file, "wb");
+    if (!f) {
+        perror(file);
+        return 1;
+    }
+    fwrite(bufs[b].p + off, 1, len, f);
+    fclose(f);
+    printf("wrote %zu bytes to %s\n", len, file);
+    return 0;
+}
+
+static int seq(int b, size_t off, size_t len, int frames, const char *dir) {
+    if (b < 0 || b >= nbufs || off + len > bufs[b].size) {
+        fprintf(stderr, "out of range\n");
+        return 1;
+    }
+    char path[512];
+    snprintf(path, sizeof path, "%s/times.txt", dir);
+    FILE *times = fopen(path, "w");
+    if (!times) {
+        perror(path);
+        return 1;
+    }
+    uint8_t *copy = malloc(len);
+    uint64_t last = 0;
+    int got = 0;
+    double start = now();
+    while (got < frames && now() - start < 30) {
+        uint64_t h = 0;
+        for (size_t i = 0; i + PAGE <= len; i += PAGE * 8) h ^= page_hash(bufs[b].p + off + i) + i;
+        if (h != last) {
+            last = h;
+            double t = now();
+            memcpy(copy, bufs[b].p + off, len);
+            snprintf(path, sizeof path, "%s/%04d.raw", dir, got);
+            FILE *f = fopen(path, "wb");
+            if (f) fwrite(copy, 1, len, f), fclose(f);
+            fprintf(times, "%d %.6f\n", got, t);
+            got++;
+        }
+        usleep(1000);
+    }
+    fclose(times);
+    free(copy);
+    printf("%d frames in %s\n", got, dir);
+    return 0;
+}
+
+static void usage(void) {
+    fprintf(stderr, "usage: ft-eyegrab [--share PATH [--owner UID:GID] [--want FILE] | --scan [N] | --dump I OFF LEN FILE |\n"
+                    "                   --seq I OFF LEN FRAMES DIR | --rec SECONDS DIR]\n");
+}
+
+int main(int argc, char **argv) {
+    if (argc >= 3 && strcmp(argv[1], "--share") == 0) {
+        const char *uid = getenv("SUDO_UID"), *gid = getenv("SUDO_GID");
+        if (uid && gid) owner_uid = (uid_t)atoi(uid), owner_gid = (gid_t)atoi(gid);
+        for (int i = 3; i < argc; i++) {
+            unsigned u, g;
+            if (strcmp(argv[i], "--owner") == 0 && i + 1 < argc && sscanf(argv[i + 1], "%u:%u", &u, &g) == 2) {
+                owner_uid = u, owner_gid = g, i++;
+            } else if (strcmp(argv[i], "--want") == 0 && i + 1 < argc) {
+                want_path = argv[++i];
+            } else {
+                usage();
+                return 2;
+            }
+        }
+        return share_loop(argv[2]);
+    }
+    int pid = find_tracker();
+    if (pid < 0) {
+        fprintf(stderr, "SteamVR's eyetracking process isn't running\n");
+        return 1;
+    }
+    if (open_bufs(pid) <= 0) {
+        fprintf(stderr, "no buffers (run as root)\n");
+        return 1;
+    }
+    if (argc >= 2 && strcmp(argv[1], "--scan") == 0) {
+        scan(argc >= 3 ? atoi(argv[2]) : 40);
+    } else if (argc == 6 && strcmp(argv[1], "--dump") == 0) {
+        return dump(atoi(argv[2]), strtoul(argv[3], NULL, 0), strtoul(argv[4], NULL, 0), argv[5]);
+    } else if (argc == 4 && strcmp(argv[1], "--rec") == 0) {
+        return rec(atof(argv[2]), argv[3], pid);
+    } else if (argc == 7 && strcmp(argv[1], "--seq") == 0) {
+        return seq(atoi(argv[2]), strtoul(argv[3], NULL, 0), strtoul(argv[4], NULL, 0), atoi(argv[5]), argv[6]);
+    } else if (argc == 1) {
+        printf("eyetracking pid %d\n", pid);
+        for (int b = 0; b < nbufs; b++)
+            printf("buffer %d: fd %d, %zu bytes, ino %lu\n", b, bufs[b].xfd, bufs[b].size, bufs[b].ino);
+    } else {
+        usage();
+        return 2;
+    }
+    return 0;
+}

gaze/tracker/ft-eyes

@@ -0,0 +1,453 @@
+#!/usr/bin/env python3
+"""ft-eyes: our own eye tracker, live.
+
+Reads the eye-camera frames that ft-eyegrab (the root service frametop-eyegrab, installed by
+gaze/tracker/install.sh) keeps in /dev/shm/frametop-eyes-cams, finds each eye's pupil and
+glint pair (eyes_pupil.py), turns them into a gaze with the saved calibration and a running
+slip estimate (eyes_model.py), and publishes the result in /dev/shm/frametop-eyes-gaze,
+where ft-gaze reads it as the source "own". It touches /dev/shm/frametop-eyes-want every
+second, and ft-eyegrab copies frames only while someone does.
+
+The gaze service (gaze/ft-gazed) runs it in the dev container, with --watch-stdin (it quits
+when its stdin closes), while Eye tracker is Own tracker or the gaze probe uses it. The
+probe calibrates and teaches it; the gaze pointer's nudges reach it as clicks, from ft-gazed.
+By hand: distrobox enter dev -- python3 gaze/tracker/ft-eyes -v
+
+Calibration: ~/.local/state/frametop/gaze/eyes/calibration.json, made by the gaze probe's
+calibration with the tracker toggle on Own tracker (or lab/ft-eyes-score --save CAPTURE).
+Each eye's shift since the calibration (eyes_model.Shift, taught by clicks) is kept in
+state.json next to it. After a restart, or frames stopping for GAP seconds (the headset
+off), the next click starts that eye's shift over, since the headset may sit differently now.
+
+Control socket: abstract datagram "@ft_eyes"; each command gets one reply line.
+  status                       JSON: calibration, and per eye its shift, clicks, whether a
+                               glint jump is applied, and "reseat" (the next click starts
+                               the shift over: the probe asks for a one-dot check then)
+  calib-start                  a new calibration: collect dots from now on
+  calib-point T0 T1 YAW PITCH  a dot you looked at from T0 to T1 (CLOCK_MONOTONIC_RAW) in
+                               that direction (head-relative degrees): "ok N0 N1 SD0 SD1"
+                               (frames and spread in px per eye, right first) or "fail WHY"
+  calib-fit                    fit the dots, save, and start the shifts and clicks over
+  click T YAW PITCH            a click: you were looking there just before T; teaches the
+                               shift: "ok DX0 DY0 DX1 DY1" (the shift each eye measured)
+
+Environment, for replays (lab/ft-eyes-e2e): FT_EYES_CAMS, FT_EYES_GAZE, FT_EYES_STATE (the
+state folder), FT_EYES_SOCKET (the socket's name). With FT_EYES_CAMS set, the want file isn't
+touched.
+
+/dev/shm/frametop-eyes-gaze, 128 bytes, little-endian (mirrored in ft-gaze.cpp):
+   0 u32 seq      odd while it's being written: read it before and after, retry if it moved
+   4 u32 version  1
+   8 f64 t        the newest frame's time, CLOCK_MONOTONIC_RAW seconds
+  16 f32 yaw, pitch  the gaze, head-relative degrees (yaw +left, pitch +up), eyes averaged
+  24 u32 flags    bit 0: right eye in it, 1: left eye in it, 2: right shift from clicks, 3: left
+  28 u32 n        samples published
+  32 f32 x4       right yaw, pitch, left yaw, pitch (NaN when that eye isn't seen)
+  48 f32 x4       each eye's shift since the calibration, pixels: right x, y, left x, y
+  64 f32 x4       pupil centre, pixels: right x, y, left x, y
+"""
+import json
+import math
+import mmap
+import os
+import socket
+import struct
+import sys
+import threading
+import time
+from collections import deque
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import eyes_model  # noqa: E402
+import eyes_pupil  # noqa: E402
+
+CAMS = os.environ.get("FT_EYES_CAMS", "/dev/shm/frametop-eyes-cams")  # overrides for replays
+OUT = os.environ.get("FT_EYES_GAZE", "/dev/shm/frametop-eyes-gaze")
+WANT = None if "FT_EYES_CAMS" in os.environ else Path("/dev/shm/frametop-eyes-want")
+OUT_SIZE = 128
+CALIBRATION = Path(os.environ.get("FT_EYES_STATE", Path.home() / ".local/state/frametop/gaze/eyes")) / "calibration.json"
+W, H = 512, 400
+FRESH = 0.03          # an eye's reading counts toward the output for this long (s)
+LOST_EVERY = 3        # while an eye is lost, search the whole frame only every 3rd frame
+EYES = ("right", "left")
+STATE = CALIBRATION.parent / "state.json"
+CLICKS = CALIBRATION.parent / "clicks.jsonl"
+SOCKET = "\0" + os.environ.get("FT_EYES_SOCKET", "ft_eyes")
+HISTORY = 12.0        # seconds of pupil positions kept per eye, for dots and clicks (the pointer's
+                      # clicks come from ft-gazed up to 10 s after the look)
+GAP = 3.0             # s without frames: the headset was off, and may sit differently now
+CLICK_BEFORE = 0.3    # a click's frames: the 300 ms before it (like the probe's fixation)
+CALIB_MIN = 15        # frames an eye needs in a calibration dot's window
+CALIB_SPREAD = 4.0    # px: more than this and the eye moved during the dot
+
+
+class Cams:
+    """The shared frames. Header and entry layout: ft-eyegrab.c, share_head_t/share_entry_t."""
+
+    def __init__(self):
+        fd = os.open(CAMS, os.O_RDONLY)
+        try:
+            self.ino = os.fstat(fd).st_ino
+            self.mm = mmap.mmap(fd, 0, prot=mmap.PROT_READ)
+        finally:
+            os.close(fd)
+        magic, version, w, h, self.slots, self.esize = struct.unpack_from("<6I", self.mm, 0)
+        if magic != 0x31434546 or version != 1 or (w, h) != (W, H):
+            raise RuntimeError(f"{CAMS}: unexpected header")
+
+    def replaced(self):
+        """ft-eyegrab restarted: it makes a new file, and this one is stale."""
+        try:
+            return os.stat(CAMS).st_ino != self.ino
+        except OSError:
+            return True
+
+    def count(self, cam):
+        return struct.unpack_from("<Q", self.mm, 24 + 8 * cam)[0]
+
+    def tracker(self):
+        return struct.unpack_from("<I", self.mm, 40)[0]
+
+    def frame(self, cam, n):
+        """Frame n of a camera as (time, array), or None if it was overwritten meanwhile."""
+        off = 64 + (cam * self.slots + n % self.slots) * self.esize
+        for _ in range(3):
+            seq = struct.unpack_from("<Q", self.mm, off)[0]
+            if seq & 1:
+                continue
+            img = np.frombuffer(self.mm, np.uint8, W * H, off + 64).reshape(H, W).copy()
+            t, got = struct.unpack_from("<dQ", self.mm, off + 8)
+            if struct.unpack_from("<Q", self.mm, off)[0] == seq and got == n:
+                return t, img
+        return None
+
+
+class Out:
+    def __init__(self):
+        fd = os.open(OUT, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600)
+        try:
+            os.ftruncate(fd, OUT_SIZE)
+            self.mm = mmap.mmap(fd, OUT_SIZE)
+        finally:
+            os.close(fd)
+        self.seq = (struct.unpack_from("<I", self.mm, 0)[0] + 1) & ~1  # even: at rest
+        self.n = 0
+
+    def write(self, t, gaze, flags, eyes, slips, pupils):
+        struct.pack_into("<I", self.mm, 0, self.seq + 1)  # odd while writing
+        self.n += 1
+        struct.pack_into("<IdffII12f", self.mm, 4, 1, t, gaze[0], gaze[1], flags, self.n,
+                         *eyes, *slips, *pupils)
+        self.seq = (self.seq + 2) & 0xFFFFFFFE
+        struct.pack_into("<I", self.mm, 0, self.seq)
+
+
+class Eye:
+    def __init__(self, eye):
+        self.eye = eye
+        self.cal = None
+        self.slip = None
+        self.shift = eyes_model.Shift()
+        self.history = deque()  # (t, x, y, glint mid x, y)
+        self.last = None        # the previous pupil (window hint)
+        self.gaze = None        # (t, yaw, pitch, x, y)
+        self.lost = 0
+        self.frames = self.found = 0
+        self.work = 0.0
+        self.last_t = None      # the previous frame's time
+
+    def use(self, cal, shift=None):
+        self.cal = cal if cal is not None and cal.has("pupil", self.eye) else None
+        self.slip = eyes_model.SlipTracker(cal, self.eye, window=eyes_model.JUMP_WINDOW) if self.cal else None
+        self.shift = shift or eyes_model.Shift()
+        self.gaze = None
+
+    def feed(self, t, img):
+        self.frames += 1
+        if self.last_t is not None and t - self.last_t > GAP:
+            self.shift.reseat()
+        self.last_t = t
+        if self.last is None:
+            self.lost += 1
+            if self.lost % LOST_EVERY:
+                return
+        t0 = time.perf_counter()
+        p = eyes_pupil.find_pupil(img, self.last)
+        self.last = p
+        if p is not None:
+            self.found += 1
+            self.lost = 0
+            pair = eyes_pupil.glint_pair(p)
+            mid = eyes_model.pair_mid(pair) if pair else (math.nan, math.nan)
+            self.history.append((t, p["x"], p["y"], mid[0], mid[1]))
+            while self.history and self.history[0][0] < t - HISTORY:
+                self.history.popleft()
+            if self.cal:
+                if pair:
+                    self.slip.add(t, (p["x"], p["y"]), mid)
+                self.shift.glint(self.slip.get(), t)
+                g = self.cal.gaze(self.eye, p["x"], p["y"], self.shift.value)
+                self.gaze = (t, float(g[0]), float(g[1]), p["x"], p["y"])
+        self.work += time.perf_counter() - t0
+
+    def window(self, t0, t1):
+        """Median pupil and glint midpoint over [t0, t1], the frame count, and the spread."""
+        rows = np.array([r for r in self.history if t0 <= r[0] <= t1]).reshape(-1, 5)
+        if len(rows) == 0:
+            return None
+        pupil = np.median(rows[:, 1:3], axis=0)
+        spread = float(np.median(np.hypot(*(rows[:, 1:3] - pupil).T)))
+        mids = rows[~np.isnan(rows[:, 3]), 3:5]
+        mid = np.median(mids, axis=0) if len(mids) >= 3 else None
+        return dict(pupil=pupil, mid=mid, n=len(rows), spread=spread)
+
+
+class Tracker:
+    def __init__(self):
+        self.eyes = [Eye(0), Eye(1)]
+        self.cal = None
+        self.dots = []  # calibration dots so far, in ft-eyes-score's click form
+        self.calibrating = False
+        if CALIBRATION.exists():
+            self.cal = eyes_model.Calibration.load(CALIBRATION)
+            if not self.cal.spread:
+                self.cal.spread = spread_from_dots(self.cal)
+        shifts = {}
+        try:
+            d = json.loads(STATE.read_text())
+            if self.cal and d.get("calibration") == self.cal.info.get("made"):
+                shifts = {int(k): eyes_model.Shift.from_json(v) for k, v in d.get("shift", {}).items()}
+        except (OSError, ValueError):
+            pass
+        for e in self.eyes:
+            e.use(self.cal, shifts.get(e.eye))
+            # Kept from the last run, but the headset may have been off since: the first
+            # click starts the history over (the saved shift is used until then).
+            e.shift.reseat()
+
+    def save_state(self):
+        d = {"calibration": self.cal.info.get("made") if self.cal else None,
+             "shift": {e.eye: e.shift.to_json() for e in self.eyes}}
+        tmp = STATE.with_suffix(".tmp")
+        tmp.write_text(json.dumps(d))
+        tmp.replace(STATE)
+
+    def command(self, line):
+        w = line.split()
+        if not w:
+            return "fail empty"
+        if w[0] == "status":
+            return json.dumps(self.status())
+        if w[0] == "calib-start":
+            self.dots, self.calibrating = [], True
+            return "ok"
+        if w[0] == "calib-point" and len(w) == 5:
+            if not self.calibrating:
+                return "fail no calibration started"
+            t0, t1, yaw, pitch = map(float, w[1:])
+            got = {e.eye: e.window(t0, t1) for e in self.eyes}
+            for c, g in got.items():
+                if g is None or g["n"] < CALIB_MIN:
+                    return f"fail the {EYES[c]} eye was seen in only {0 if g is None else g['n']} frames"
+                if g["spread"] > CALIB_SPREAD:
+                    return f"fail the {EYES[c]} eye moved ({g['spread']:.1f} px)"
+            self.dots.append(dict(truth=(yaw, pitch), eye={c: dict(pupil=g["pupil"], mid=g["mid"]) for c, g in got.items()}))
+            return "ok {} {} {:.2f} {:.2f}".format(got[0]["n"], got[1]["n"], got[0]["spread"], got[1]["spread"])
+        if w[0] == "calib-fit":
+            if len(self.dots) < eyes_model.MIN_CLICKS:
+                return f"fail only {len(self.dots)} dots (need {eyes_model.MIN_CLICKS})"
+            cal = eyes_model.Calibration.fit(self.dots, {"made": time.strftime("%Y-%m-%d %H:%M:%S"),
+                                                       "dots": len(self.dots), "from": "probe calibration"})
+            if not all(cal.has(n, c) for n in ("pupil", "where") for c in (0, 1)):
+                return "fail not enough dots with both eyes"
+            errs = [float(np.hypot(*(np.mean([cal.gaze(c, *k["eye"][c]["pupil"]) for c in (0, 1)], axis=0)
+                                     - k["truth"]))) for k in self.dots]
+            if CALIBRATION.exists():
+                CALIBRATION.replace(CALIBRATION.with_name(time.strftime("calibration-%Y%m%d-%H%M%S.json")))
+            cal.save(CALIBRATION)
+            self.cal, self.calibrating = cal, False
+            for e in self.eyes:
+                e.use(cal)
+            self.save_state()
+            with open(CALIBRATION.with_name("calibration-dots.jsonl"), "a") as f:
+                for k in self.dots:
+                    f.write(json.dumps({"made": cal.info["made"], "truth": k["truth"],
+                                        "eye": {c: {"pupil": v["pupil"].tolist(),
+                                                    "mid": None if v["mid"] is None else v["mid"].tolist()}
+                                                for c, v in k["eye"].items()}}) + "\n")
+            return (f"ok {len(self.dots)} dots, fit median {np.median(errs):.2f} deg, eyes "
+                    + ", ".join(f"{EYES[c]} {cal.spread[c]:.2f}" for c in sorted(cal.spread)))
+        if w[0] == "click" and len(w) == 4:
+            if not self.cal:
+                return "fail not calibrated"
+            t, yaw, pitch = map(float, w[1:])
+            out, rec = [], {"time": time.time(), "t": t, "truth": [yaw, pitch], "eyes": {}}
+            for e in self.eyes:
+                g = e.window(t - CLICK_BEFORE, t)
+                if g is None or g["n"] < 5 or not e.cal:
+                    out += ["nan", "nan"]
+                    continue
+                d = self.cal.click_shift(e.eye, g["pupil"], (yaw, pitch))
+                before = e.shift.value.tolist()
+                e.shift.click(d, e.slip.value if e.slip else None)
+                rec["eyes"][e.eye] = {"pupil": g["pupil"].tolist(), "measured": d.tolist(), "before": before,
+                                      "after": e.shift.value.tolist()}
+                out += [f"{d[0]:.2f}", f"{d[1]:.2f}"]
+            self.save_state()
+            with open(CLICKS, "a") as f:
+                f.write(json.dumps(rec) + "\n")
+            return "ok " + " ".join(out)
+        return f"fail unknown command {w[0]}"
+
+    def status(self):
+        return {"calibration": self.cal.info if self.cal else None, "calibrating": self.calibrating,
+                "dots": len(self.dots),
+                "eyes": {EYES[e.eye]: {"shift": e.shift.value.tolist(), "clicks": len(e.shift.meas),
+                                       "jump": bool(np.any(e.shift.jump)),
+                                       "reseat": e.shift.reseated} for e in self.eyes}}
+
+
+def spread_from_dots(cal):
+    """The eyes' fit spreads for a calibration saved without them, from its dots in
+    calibration-dots.jsonl ({} if they aren't there: the eyes are then weighted alike)."""
+    try:
+        lines = CALIBRATION.with_name("calibration-dots.jsonl").read_text().splitlines()
+    except OSError:
+        return {}
+    dots = []
+    for line in lines:
+        d = json.loads(line)
+        if d.get("made") == cal.info.get("made"):
+            dots.append(dict(truth=d["truth"], eye={
+                int(c): dict(pupil=np.array(v["pupil"]), mid=None if v["mid"] is None else np.array(v["mid"]))
+                for c, v in d["eye"].items()}))
+    return eyes_model.Calibration.fit(dots).spread if dots else {}
+
+
+class Want:
+    """Touches the want file every second, so ft-eyegrab keeps copying frames."""
+
+    def __init__(self):
+        self.at = 0.0
+
+    def __call__(self):
+        if WANT is None or time.monotonic() - self.at < 1.0:
+            return
+        self.at = time.monotonic()
+        try:
+            fd = os.open(WANT, os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600)
+            os.utime(fd)
+            os.close(fd)
+        except OSError as e:
+            print(f"ft-eyes: {WANT}: {e}", file=sys.stderr, flush=True)
+
+
+def wait_for_cams(sock, tracker, want):
+    while True:
+        want()
+        try:
+            return Cams()
+        except (OSError, ValueError, RuntimeError):
+            serve(sock, tracker)
+            time.sleep(0.2)
+
+
+def serve(sock, tracker):
+    while True:
+        try:
+            data, addr = sock.recvfrom(512)
+        except BlockingIOError:
+            return
+        try:
+            reply = tracker.command(data.decode(errors="replace").strip())
+        except Exception as ex:  # a bad command must not take the tracker down
+            reply = f"fail {type(ex).__name__}: {ex}"
+        if addr:
+            try:
+                sock.sendto(reply.encode(), addr)
+            except OSError:
+                pass
+
+
+def main():
+    verbose = "-v" in sys.argv
+    if "--watch-stdin" in sys.argv:
+        # Run by ft-gazed through distrobox, which doesn't pass a stop on: quit when our
+        # stdin (its pipe) closes.
+        def watch():
+            while sys.stdin.buffer.read(4096):
+                pass
+            os._exit(0)
+        threading.Thread(target=watch, daemon=True).start()
+    want = Want()
+    CALIBRATION.parent.mkdir(parents=True, exist_ok=True)
+    tracker = Tracker()
+    eyes = tracker.eyes
+    out = Out()
+    sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    sock.bind(SOCKET)
+    sock.setblocking(False)
+    cal = tracker.cal
+    print("ft-eyes: " + (f"calibration from {cal.info.get('made')} ({cal.info.get('from', cal.info.get('capture'))})"
+                           if cal else "not calibrated: run the probe's calibration with the Own tracker")
+          + f"; waiting for {CAMS}", file=sys.stderr, flush=True)
+    cams = wait_for_cams(sock, tracker, want)
+    print("ft-eyes: frames found, tracking", file=sys.stderr, flush=True)
+    seen = [cams.count(0), cams.count(1)]
+    report = time.monotonic()
+    while True:
+        serve(sock, tracker)
+        want()
+        new = False
+        for c in (0, 1):
+            n = cams.count(c)
+            if n == seen[c]:
+                continue
+            seen[c] = n
+            got = cams.frame(c, n - 1)  # only the newest: never fall behind
+            if got:
+                eyes[c].feed(*got)
+                new = True
+        if new:
+            latest = max((e.gaze[0] for e in eyes if e.gaze), default=None)
+            use = [e for e in eyes if e.gaze and latest - e.gaze[0] < FRESH]
+            if use:
+                yaw, pitch = tracker.cal.combine({e.eye: e.gaze[1:3] for e in use})
+                flags, per, shifts, pupils = 0, [], [], []
+                for i, e in enumerate(eyes):
+                    fresh = e in use
+                    flags |= (1 << i) if fresh else 0
+                    flags |= (4 << i) if e.shift.meas else 0
+                    per += [e.gaze[1], e.gaze[2]] if fresh else [math.nan, math.nan]
+                    shifts += [float(v) for v in e.shift.value]
+                    pupils += [e.gaze[3], e.gaze[4]] if fresh else [math.nan, math.nan]
+                out.write(latest, (yaw, pitch), flags, per, shifts, pupils)
+        else:
+            time.sleep(0.001)
+        now = time.monotonic()
+        if now - report >= 5:
+            if verbose:
+                parts = []
+                for e in eyes:
+                    s = e.shift.value
+                    parts.append(f"{EYES[e.eye]} {e.frames / 5:.0f} fps, found {e.found / max(e.frames, 1):.0%}, "
+                                 f"{e.work / max(e.frames, 1) * 1000:.2f} ms/frame, shift ({s[0]:+.1f},{s[1]:+.1f})"
+                                 f" from {len(e.shift.meas)} clicks" + (", jump" if np.any(e.shift.jump) else ""))
+                    e.frames = e.found = 0
+                    e.work = 0.0
+                print("ft-eyes: " + "; ".join(parts), file=sys.stderr, flush=True)
+            report = now
+            if cams.replaced():
+                print("ft-eyes: frames went away; waiting", file=sys.stderr, flush=True)
+                cams = wait_for_cams(sock, tracker, want)
+                seen = [cams.count(0), cams.count(1)]
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt:
+        pass

gaze/tracker/install.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# Install (or remove) the frame grabber our own eye tracker needs: ft-eyegrab, as the system
+# service frametop-eyegrab.service. It copies the eye-camera frames, read-only, out of
+# SteamVR's eyetracking process into /dev/shm/frametop-eyes-cams for ft-eyes, and only while
+# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when ours is the
+# tracker in use (GAZE_TRACKER=auto, the default, picks it once this is installed) or the gaze
+# probe uses it. install.sh offers this after gaze mode.
+# Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit: it asks for
+# the password in the terminal, on the Frame or from a PC, or runs SUDO_ASKPASS when that's set
+# (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env).
+# Usage: gaze/tracker/install.sh [install|uninstall|status|log [lines]]
+set -euo pipefail
+
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+. "$root/scripts/_env.sh"
+src=$FRAME_REPO/gaze/tracker
+unit=frametop-eyegrab.service
+
+sudo_run() { frame_sudo "$1"; }
+
+case ${1:-install} in
+  install)
+    "$root/gaze/tracker/build.sh"
+    ids=$(on_frame 'echo "$(id -u):$(id -g)"')
+    fill_template "$root/gaze/tracker/$unit" | sed "s|@UID@|${ids%:*}|g; s|@GID@|${ids#*:}|g" |
+      on_frame "cat > /tmp/$unit"
+    sudo_run "set -e
+install -D -m 0755 -o root -g root $src/build/ft-eyegrab /etc/frametop/ft-eyegrab
+install -D -m 0644 -o root -g root /tmp/$unit /etc/systemd/system/$unit
+rm -f /tmp/$unit
+systemctl daemon-reload
+systemctl enable $unit
+systemctl restart $unit
+sleep 1
+echo \"$unit: \$(systemctl is-active $unit)\""
+    ;;
+  uninstall)
+    sudo_run "systemctl disable --now $unit 2>/dev/null
+rm -f /etc/systemd/system/$unit /etc/frametop/ft-eyegrab
+rmdir /etc/frametop 2>/dev/null; systemctl daemon-reload; echo removed" ;;
+  status) on_frame "systemctl is-active $unit; ls -l /dev/shm/frametop-eyes-cams 2>/dev/null" || true ;;
+  log) on_frame "journalctl -u $unit --no-pager -o cat -n ${2:-20}" ;;
+  *) echo "usage: $0 [install|uninstall|status|log [lines]]" >&2; exit 2 ;;
+esac

gaze/tracker/lab/eyes_lab.py

@@ -0,0 +1,31 @@
+"""What the lab tools share: where recordings are kept, and the tracker's modules.
+
+Recordings of the eye cameras are biometric data. They're kept outside the repo, in
+~/.local/share/frametop/eyes/captures (FT_EYES_CAPTURES overrides), one folder each, 0700,
+and never leave the Frame except for the 7i's copies frame-job makes for offline jobs.
+"""
+import os
+import sys
+from pathlib import Path
+
+TRACKER = Path(__file__).resolve().parents[1]  # gaze/tracker: ft-eyes, eyes_model, eyes_pupil
+LAB = Path(__file__).resolve().parent
+CAPTURES = Path(os.environ.get("FT_EYES_CAPTURES", Path.home() / ".local/share/frametop/eyes/captures"))
+sys.path.insert(0, str(TRACKER))
+
+
+def capture(arg):
+    """A recording's folder: a path as given, or a bare name in CAPTURES."""
+    p = Path(arg).expanduser()
+    return p if p.exists() or os.sep in arg else CAPTURES / arg
+
+
+def new_capture(name):
+    """A new, private recording folder in CAPTURES; exits if it's already there."""
+    out = CAPTURES / name
+    if out.exists():
+        sys.exit(f"{out} exists")
+    out.mkdir(parents=True)
+    for d in (CAPTURES, out):
+        d.chmod(0o700)
+    return out

gaze/tracker/lab/ft-eyes-e2e

@@ -0,0 +1,183 @@
+#!/usr/bin/env python3
+"""ft-eyes-e2e: the live path end to end on two recordings, without the headset.
+
+Starts a scratch ft-eyes (its own shared memory, socket, and state folder, so the real
+calibration is untouched), then:
+  1. plays CALIB into it and sends each of its practice clicks as a calibration dot (the
+     300 ms before the press), as the probe's calibration would, and fits;
+  2. plays TEST into it and, at each of its practice clicks, scores what ft-eyes was
+     publishing in the 300 ms before the press, then sends the click, as the probe does.
+So every TEST click is scored with only earlier data, like ft-eyes-score's `clicks` method.
+
+Usage: frame-job -- lab/py lab/ft-eyes-e2e CALIB TEST [--for S] [--dump FILE]
+CALIB and TEST are recordings (a bare name is in eyes_lab.CAPTURES; give full paths for
+frame-job to copy them to the 7i). Runs at the recorded pace (about the two recordings'
+length). --dump saves everything ft-eyes published during TEST (OUT_FIELDS per row) and each
+click's score, as a pickle, for looking into the bad clicks.
+"""
+import json
+import mmap
+import os
+import pickle
+import re
+import shutil
+import socket
+import struct
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from eyes_lab import LAB, TRACKER, capture  # noqa: E402
+
+BEFORE = 0.3  # the probe's fixation window before a press (s)
+# ft-eyes' output after its seq and version (ft-eyes' docstring): one row per sample.
+OUT_FORMAT = "<dffII12f"
+OUT_FIELDS = ("t", "yaw", "pitch", "flags", "n", "r_yaw", "r_pitch", "l_yaw", "l_pitch",
+              "r_shift_x", "r_shift_y", "l_shift_x", "l_shift_y", "r_pupil_x", "r_pupil_y", "l_pupil_x", "l_pupil_y")
+
+
+class Run:
+    def __init__(self):
+        tag = f"ft-eyes-e2e-{os.getpid()}"
+        self.state = Path(tempfile.mkdtemp(prefix=tag + "-"))
+        self.env = dict(os.environ, FT_EYES_CAMS=f"/dev/shm/{tag}-cams", FT_EYES_GAZE=f"/dev/shm/{tag}-gaze",
+                        FT_EYES_STATE=str(self.state), FT_EYES_SOCKET=tag)
+        self.log = self.state / "ft-eyes.log"
+        self.trackd = subprocess.Popen([sys.executable, str(TRACKER / "ft-eyes"), "-v"], env=self.env,
+                                       stdout=subprocess.DEVNULL, stderr=open(self.log, "w"))
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+        self.sock.bind("\0" + tag + "-client")
+        self.sock.settimeout(2)
+        self.to = "\0" + tag
+        self.gaze = None
+
+    def cmd(self, line):
+        for _ in range(50):  # ft-eyes may not have bound its socket yet
+            try:
+                self.sock.sendto(line.encode(), self.to)
+                return self.sock.recv(4096).decode()
+            except ConnectionRefusedError:
+                time.sleep(0.2)
+        raise RuntimeError("ft-eyes never answered")
+
+    def latest_frame_t(self):
+        """The newest replayed frame's time, or None before the replay starts."""
+        try:
+            with open(self.env["FT_EYES_CAMS"], "rb") as f:
+                mm = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
+        except (OSError, ValueError):
+            return None
+        slots, esize = struct.unpack_from("<2I", mm, 16)
+        best = None
+        for cam in (0, 1):
+            n = struct.unpack_from("<Q", mm, 24 + 8 * cam)[0]
+            if n:
+                t = struct.unpack_from("<d", mm, 64 + (cam * slots + (n - 1) % slots) * esize + 8)[0]
+                best = t if best is None else max(best, t)
+        return best
+
+    def published(self):
+        """ft-eyes' newest output (OUT_FIELDS), or None."""
+        if self.gaze is None:
+            try:
+                with open(self.env["FT_EYES_GAZE"], "rb") as f:
+                    self.gaze = mmap.mmap(f.fileno(), 128, prot=mmap.PROT_READ)
+            except (OSError, ValueError):
+                return None
+        for _ in range(3):
+            seq = struct.unpack_from("<I", self.gaze, 0)[0]
+            v = struct.unpack_from(OUT_FORMAT, self.gaze, 8)
+            if not seq & 1 and struct.unpack_from("<I", self.gaze, 0)[0] == seq:
+                return v
+        return None
+
+    def stage(self, cap, secs, handle):
+        """Play `cap` and call handle(click, samples) as each click's press time goes by.
+        Returns everything ft-eyes published meanwhile."""
+        clicks = pickle.load(open(cap / "features.pkl", "rb"))["clicks"]
+        replay = subprocess.Popen([sys.executable, str(LAB / "ft-eyes-replay"), str(cap), self.env["FT_EYES_CAMS"],
+                                   "--for", str(secs)], stdout=subprocess.DEVNULL)
+        todo, samples = list(clicks), []
+        while replay.poll() is None:
+            t = self.latest_frame_t()
+            v = self.published()
+            if v and v[0] > 0 and (not samples or v[0] != samples[-1][0]):
+                samples.append(v)
+            while t and todo and t > todo[0]["t"] + 0.05:
+                handle(todo.pop(0), samples)
+            time.sleep(0.003)
+        return samples
+
+    def close(self):
+        self.trackd.terminate()
+        self.trackd.wait()
+        for p in (self.env["FT_EYES_CAMS"], self.env["FT_EYES_GAZE"]):
+            if os.path.exists(p):
+                os.unlink(p)
+        shutil.rmtree(self.state, ignore_errors=True)
+
+
+def main(argv):
+    args = [a for i, a in enumerate(argv) if not a.startswith("--") and (i == 0 or argv[i - 1] not in ("--for", "--dump"))]
+    if len(args) != 2:
+        sys.exit(__doc__)
+    calib, test = map(capture, args)
+    secs = argv[argv.index("--for") + 1] if "--for" in argv else "1e9"
+    dump = Path(argv[argv.index("--dump") + 1]) if "--dump" in argv else None
+    run = Run()
+    try:
+        print("calib-start:", run.cmd("calib-start"), flush=True)
+        dots = []
+        run.stage(calib, secs, lambda k, _s: dots.append(
+            run.cmd(f"calib-point {k['t'] - BEFORE} {k['t']} {k['truth'][0]} {k['truth'][1]}")))
+        fails = [d for d in dots if not d.startswith("ok")]
+        print(f"{calib.name}: {len(dots) - len(fails)} of {len(dots)} clicks taken as dots", flush=True)
+        whys = [re.sub(r"[\d.]+", "N", f) for f in fails]
+        for why in sorted(set(whys)):
+            print(f"  {whys.count(why)} x {why}")
+        print("calib-fit:", run.cmd("calib-fit"), flush=True)
+        # ft-eyes-replay removes its file at the end; ft-eyes notices within 5 s and waits for the next.
+        time.sleep(6)
+        scored = []
+
+        def click(k, samples):
+            s = np.array([v for v in samples if k["t"] - BEFORE <= v[0] <= k["t"]]).reshape(-1, len(OUT_FIELDS))
+            err = float(np.hypot(*(np.median(s[:, 1:3], axis=0) - k["truth"]))) if len(s) >= 5 else None
+            reply = run.cmd(f"click {k['t']} {k['truth'][0]} {k['truth'][1]}")
+            st = json.loads(run.cmd("status"))["eyes"]
+            scored.append((k["t"], err, reply, [(v["clicks"], v["jump"]) for v in st.values()]))
+
+        test_samples = run.stage(test, secs, click)
+        if dump:
+            with open(dump, "wb") as f:
+                pickle.dump({"fields": OUT_FIELDS, "samples": np.array(test_samples, float),
+                             "clicks": [dict(t=x[0], err=x[1], reply=x[2], eyes=x[3]) for x in scored]}, f)
+            print("dumped to", dump)
+        e = np.array([x[1] for x in scored if x[1] is not None])
+        print(f"{test.name}: {len(e)} of {len(scored)} clicks scored live", flush=True)
+        if len(e):
+            print(f"  median {np.median(e):.2f} deg, 90% {np.percentile(e, 90):.2f}, "
+                  f"after the first 5: median {np.median(e[5:]):.2f}")
+        print("  clicks ft-eyes refused:", sum(not x[2].startswith("ok") for x in scored))
+        t0 = scored[0][0] if scored else 0
+        print("  clicks that started an eye's shift over after a jump: right {}, left {}".format(
+            *(sum(x[3][c][0] == 1 for x in scored[1:]) for c in (0, 1))))
+        print("  by time (s): " + ", ".join(
+            f"{lo}-{lo + 30}: {np.median(b):.2f}" for lo in range(0, 300, 30)
+            if len(b := [x[1] for x in scored if x[1] is not None and lo <= x[0] - t0 < lo + 30])))
+        print("  worst: " + ", ".join(
+            f"{x[0] - t0:.0f}s {x[1]:.1f} (clicks/jump R {x[3][0][0]}/{x[3][0][1]:d} L {x[3][1][0]}/{x[3][1][1]:d})"
+            for x in sorted((x for x in scored if x[1] is not None), key=lambda x: -x[1])[:10]))
+        print("status:", run.cmd("status"))
+        print("ft-eyes' last report:", run.log.read_text().strip().splitlines()[-1:])
+    finally:
+        run.close()
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

gaze/tracker/lab/ft-eyes-record

@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+"""ft-eyes-record: record a live session from the shared frames, so it can be replayed and
+scored later (ft-eyes-score, ft-eyes-e2e), without root and alongside ft-eyes.
+
+Reads /dev/shm/frametop-eyes-cams (ft-eyegrab, frametop-eyegrab.service) and writes every
+frame of both cameras to a new recording NAME in eyes_lab.CAPTURES: frames.raw, index.txt
+("<n> <slot> <camera> <time>", as ft-eyegrab --rec), and clocks.txt (wall clock and
+CLOCK_MONOTONIC_RAW, read together). It touches /dev/shm/frametop-eyes-want every second, so
+ft-eyegrab copies frames even without ft-eyes. About 2 GB a minute. Stops after SECONDS
+(default 900), on Ctrl-C or SIGTERM, or when the disk gets below MIN_FREE_GB. The probe's
+clicks are added later, on the Frame: `lab/py lab/ft-eyes-score --clicks NAME`.
+
+Usage: lab/ft-eyes-record NAME [SECONDS]   (host Python is enough: no numpy)
+"""
+import mmap
+import os
+import shutil
+import signal
+import struct
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from eyes_lab import new_capture  # noqa: E402
+
+CAMS = os.environ.get("FT_EYES_CAMS", "/dev/shm/frametop-eyes-cams")
+WANT = "/dev/shm/frametop-eyes-want"
+W, H = 512, 400
+MIN_FREE_GB = 20
+
+
+class Share:
+    """The shared frames. Layout: ft-eyegrab.c, share_head_t/share_entry_t."""
+
+    def __init__(self):
+        fd = os.open(CAMS, os.O_RDONLY)
+        try:
+            self.ino = os.fstat(fd).st_ino
+            self.mm = mmap.mmap(fd, 0, prot=mmap.PROT_READ)
+        finally:
+            os.close(fd)
+        magic, version, w, h, self.slots, self.esize = struct.unpack_from("<6I", self.mm, 0)
+        if magic != 0x31434546 or version != 1 or (w, h) != (W, H):
+            raise RuntimeError(f"{CAMS}: unexpected header")
+
+    def replaced(self):
+        try:
+            return os.stat(CAMS).st_ino != self.ino
+        except OSError:
+            return True
+
+    def count(self, cam):
+        return struct.unpack_from("<Q", self.mm, 24 + 8 * cam)[0]
+
+    def frame(self, cam, n):
+        """(time, slot, bytes) of frame n of a camera, or None if it was overwritten."""
+        off = 64 + (cam * self.slots + n % self.slots) * self.esize
+        for _ in range(3):
+            seq = struct.unpack_from("<Q", self.mm, off)[0]
+            if seq & 1:
+                continue
+            data = self.mm[off + 64:off + 64 + W * H]
+            t, got, _cam, slot = struct.unpack_from("<dQII", self.mm, off + 8)
+            if struct.unpack_from("<Q", self.mm, off)[0] == seq and got == n:
+                return t, slot, data
+        return None
+
+
+def touch_want():
+    """Tell ft-eyegrab someone wants frames (it idles otherwise)."""
+    if "FT_EYES_CAMS" in os.environ:
+        return
+    try:
+        fd = os.open(WANT, os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600)
+        os.utime(fd)
+        os.close(fd)
+    except OSError:
+        pass
+
+
+def open_share(deadline):
+    while time.monotonic() < deadline:
+        touch_want()
+        try:
+            return Share()
+        except (OSError, ValueError, RuntimeError):
+            time.sleep(0.5)
+    sys.exit(f"ft-eyes-record: no {CAMS} (is frametop-eyegrab.service running? gaze/tracker/install.sh)")
+
+
+def main(argv):
+    if not argv or argv[0].startswith("-"):
+        sys.exit(__doc__)
+    secs = float(argv[1]) if len(argv) > 1 else 900.0
+    out = new_capture(argv[0])
+    stop = []
+    for sig in (signal.SIGINT, signal.SIGTERM):
+        signal.signal(sig, lambda *_: stop.append(1))
+    share = open_share(time.monotonic() + 10)
+    (out / "clocks.txt").write_text(f"{time.time()} {time.clock_gettime(time.CLOCK_MONOTONIC_RAW)}\n")
+    seen = [share.count(0), share.count(1)]
+    written = dropped = 0
+    end = time.monotonic() + secs
+    check = touched = time.monotonic()
+    with open(out / "frames.raw", "wb") as frames, open(out / "index.txt", "w") as index:
+        while not stop and time.monotonic() < end:
+            new = []
+            for c in (0, 1):
+                n = share.count(c)
+                if n - seen[c] > share.slots:  # fell behind: those frames are gone
+                    dropped += n - seen[c] - share.slots
+                    seen[c] = n - share.slots
+                for i in range(seen[c], n):
+                    f = share.frame(c, i)
+                    if f is None:
+                        dropped += 1
+                    else:
+                        new.append((f[0], f[1], c, f[2]))
+                seen[c] = n
+            for t, slot, c, data in sorted(new, key=lambda f: f[0]):
+                frames.write(data)
+                index.write(f"{written} {slot} {c} {t:.6f}\n")
+                written += 1
+            if not new:
+                time.sleep(0.002)
+            if time.monotonic() - touched > 1:
+                touched = time.monotonic()
+                touch_want()
+            if time.monotonic() - check > 5:
+                check = time.monotonic()
+                if shutil.disk_usage(out).free < MIN_FREE_GB * 1e9:
+                    print(f"ft-eyes-record: under {MIN_FREE_GB} GB free, stopping", file=sys.stderr)
+                    break
+                if share.replaced():
+                    print("ft-eyes-record: the frame share was restarted; following it", file=sys.stderr)
+                    share = open_share(time.monotonic() + 10)
+                    seen = [share.count(0), share.count(1)]
+    print(f"ft-eyes-record: {written} frames ({dropped} dropped) in {out}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

gaze/tracker/lab/ft-eyes-replay

@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+"""ft-eyes-replay: play a recording into the shared-frame layout, as ft-eyegrab --share
+would, at the recorded pace, to test ft-eyes without the headset.
+
+Usage: lab/py lab/ft-eyes-replay NAME [PATH] [--from S] [--for S]
+NAME is a recording (a bare name is in eyes_lab.CAPTURES). PATH defaults to
+/dev/shm/frametop-eyes-cams-replay; run ft-eyes with FT_EYES_CAMS=PATH (and FT_EYES_GAZE=...
+so it doesn't overwrite the live output).
+"""
+import mmap
+import os
+import struct
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from eyes_lab import capture  # noqa: E402
+
+W, H, SLOTS = 512, 400, 8
+ESIZE = 64 + W * H
+
+
+def main(argv):
+    cap = capture(argv[0])
+    path = argv[1] if len(argv) > 1 and not argv[1].startswith("--") else "/dev/shm/frametop-eyes-cams-replay"
+    start = float(argv[argv.index("--from") + 1]) if "--from" in argv else 0.0
+    length = float(argv[argv.index("--for") + 1]) if "--for" in argv else 1e9
+    idx = np.array([[float(v) for v in l.split()] for l in (cap / "index.txt").read_text().splitlines()
+                    if len(l.split()) == 4])
+    frames = np.memmap(cap / "frames.raw", dtype=np.uint8, mode="r").reshape(-1, H, W)
+    size = 64 + 2 * SLOTS * ESIZE
+    if os.path.exists(path):
+        os.unlink(path)
+    fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_EXCL, 0o600)
+    os.ftruncate(fd, size)
+    mm = mmap.mmap(fd, size)
+    os.close(fd)
+    struct.pack_into("<6I2QI", mm, 0, 0x31434546, 1, W, H, SLOTS, ESIZE, 0, 0, os.getpid())
+    count = [0, 0]
+    t0 = idx[0, 3] + start
+    wall0 = time.monotonic()
+    try:
+        for i in range(len(frames)):
+            t = idx[i, 3]
+            if t < t0:
+                continue
+            if t - t0 > length:
+                break
+            delay = (t - t0) - (time.monotonic() - wall0)
+            if delay > 0:
+                time.sleep(delay)
+            cam = int(idx[i, 2])
+            n = count[cam]
+            off = 64 + (cam * SLOTS + n % SLOTS) * ESIZE
+            seq = struct.unpack_from("<Q", mm, off)[0]
+            struct.pack_into("<Q", mm, off, seq + 1)
+            mm[off + 64:off + 64 + W * H] = frames[i].tobytes()
+            struct.pack_into("<dQII", mm, off + 8, t, n, cam, int(idx[i, 1]))
+            struct.pack_into("<Q", mm, off, seq + 2)
+            count[cam] = n + 1
+            struct.pack_into("<Q", mm, 24 + 8 * cam, n + 1)
+    finally:
+        os.unlink(path)
+    print(f"replayed {sum(count)} frames")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

gaze/tracker/lab/ft-eyes-score

@@ -0,0 +1,291 @@
+#!/usr/bin/env python3
+"""ft-eyes-score: score our pupil tracker and SteamVR against the gaze probe's practice clicks.
+
+Usage (from gaze/tracker; A and B are recordings: a bare name is in eyes_lab.CAPTURES):
+  frame-job -- lab/py lab/ft-eyes-score A        fit and test on A, leave-one-out
+  frame-job -- lab/py lab/ft-eyes-score A B      fit on A, test on B
+  lab/py lab/ft-eyes-score --save A              fit on A, save it for ft-eyes (on the Frame)
+  lab/py lab/ft-eyes-score --clicks A...         on the Frame: copy each capture's practice
+      clicks from the probe's log into it (the first scoring does it too)
+For frame-job to copy a recording to the 7i, give its full path, e.g.
+~/.local/share/frametop/eyes/captures/A.
+
+Each practice click gives a known gaze direction: SteamVR's raw gaze at the press plus the
+angle from it to where you released (you were looking there). For each click we take the
+frames from just before the press and find each eye's pupil and glint pair.
+
+Methods, each a quadratic fit per eye, both eyes averaged when both are seen:
+  pupil   the pupil centre alone. Breaks when the headset slips on the face.
+  glint   pupil minus the glint pair's midpoint. Slip moves both alike, so this holds up,
+          but the right eye's pair is often off the cornea.
+  clicks  the pupil centre minus the shift the earlier clicks measured, with the glints
+          only noticing a sudden jump (eyes_model.Shift). The one ft-eyes uses.
+  slip    the pupil centre minus a slip estimate. Wherever the pair is seen, the glint
+          method gives the gaze, the fit says where the pupil should be for that gaze, and
+          the difference is the slip. Slip changes slowly, so the median over the last
+          30 seconds applies to every frame, with or without glints (see eyes_model.py).
+SteamVR gets the same quadratic fit on its raw gaze.
+"""
+import json
+import pickle
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from eyes_lab import capture  # noqa: E402
+import eyes_model  # noqa: E402
+import eyes_pupil  # noqa: E402
+
+PRACTICE = Path.home() / ".local/state/frametop/gaze/practice.jsonl"
+BEFORE = (0.25, 0.02)   # frames from 250 ms to 20 ms before the press
+EYES = {0: "right", 1: "left"}
+MIN_FRAMES = 5
+TRACK_EVERY = 9         # the slip track uses every 9th frame per camera (10 a second)
+VERSION = 4             # bump when the features change, to rebuild the caches
+
+
+# --- Features -------------------------------------------------------------------------
+
+def load_index(cap):
+    # Skip a half-written last line (the recorder may still be running).
+    return np.array([[float(v) for v in l.split()]
+                     for l in (cap / "index.txt").read_text().splitlines() if len(l.split()) == 4])
+
+
+def eye_features(frames):
+    """Median pupil centre and glint-pair midpoint over some frames of one eye."""
+    ps = [p for p in map(eyes_pupil.find_pupil, frames) if p]
+    if len(ps) < MIN_FRAMES:
+        return None
+    pupil = np.median([(p["x"], p["y"]) for p in ps], axis=0)
+    mids = []
+    for p in ps:
+        pair = eyes_pupil.glint_pair(p)
+        if pair:
+            mids.append(((pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2))
+    mid = np.median(mids, axis=0) if len(mids) >= 3 else None
+    return dict(pupil=pupil, mid=mid)
+
+
+def practice_log(cap, t0, t1, wall, mono):
+    """The probe's practice records during the capture. The first time (on the Frame), cut
+    from the probe's log into the capture's practice.jsonl, so the capture carries its own
+    clicks (the 7i has no probe log)."""
+    own = cap / "practice.jsonl"
+    if not own.exists():
+        if not PRACTICE.exists():
+            sys.exit(f"{own} is missing: run `lab/py lab/ft-eyes-score --clicks {cap.name}` once on the Frame")
+        keep = [line for line in open(PRACTICE)
+                if t0 - 5 < json.loads(line)["time"] - wall + mono < t1 + 60]
+        own.write_text("".join(keep))
+    return [json.loads(line) for line in open(own)]
+
+
+def features(cap):
+    """Per-click and per-session features, cached in the capture (numbers only)."""
+    cache = cap / "features.pkl"
+    if cache.exists():
+        f = pickle.loads(cache.read_bytes())
+        if f.get("version") == VERSION:
+            return f
+    wall, mono = map(float, (cap / "clocks.txt").read_text().split())
+    idx = load_index(cap)
+    frames = np.memmap(cap / "frames.raw", dtype=np.uint8, mode="r").reshape(-1, 400, 512)
+    idx = idx[:len(frames)]
+    t0, t1 = idx[0, 3], idx[-1, 3]
+
+    clicks = []
+    for r in practice_log(cap, t0, t1, wall, mono):
+        src = r.get("sources", {})
+        s = src.get("mmap1")
+        if r.get("mode") != "practice" or not s or "off" not in s:
+            continue
+        press = r["time"] - r["held_s"] - wall + mono
+        if not t0 + BEFORE[0] < press < t1:
+            continue
+        k = np.where((idx[:, 3] > press - BEFORE[0]) & (idx[:, 3] < press - BEFORE[1]))[0]
+        eye = {c: eye_features([frames[i] for i in k if idx[i, 2] == c]) for c in (0, 1)}
+        # The truth: a source's gaze at the press plus the angle from it to the release
+        # point. The angle is converted with a local linear fit of the screen, so the
+        # closer the source, the better: ours when the live tracker was running.
+        own = src.get("own") if src.get("own", {}).get("off") else None
+        base = own or s
+        truth = (base["hy"] + base["off"][0], base["hp"] + base["off"][1])
+        clicks.append(dict(t=press, truth=truth, truth_from="own" if own else "mmap1",
+                           steam=(s["hy"], s["hp"]),
+                           steam_err=float(np.hypot(s["hy"] - truth[0], s["hp"] - truth[1])),
+                           live_err=float(np.hypot(*own["off"])) if own else None,
+                           press_err=r.get("press_err_deg"), press_source=r.get("source"), eye=eye))
+
+    # The slip track: pupil and pair midpoint on a sample of frames through the session.
+    track = {}
+    for c in (0, 1):
+        rows = []
+        for i in np.where(idx[:, 2] == c)[0][::TRACK_EVERY]:
+            p = eyes_pupil.find_pupil(frames[i])
+            pair = p and eyes_pupil.glint_pair(p)
+            if pair:
+                rows.append((idx[i, 3], p["x"], p["y"],
+                             (pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2))
+        track[c] = np.array(rows).reshape(-1, 5)
+    f = dict(version=VERSION, clicks=clicks, track=track, span=(t0, t1))
+    cache.write_bytes(pickle.dumps(f))
+    return f
+
+
+# --- Fitting --------------------------------------------------------------------------
+
+class Model:
+    """A calibration fitted on some clicks, plus SteamVR's quadratic fit on the same."""
+
+    def __init__(self, clicks):
+        self.cal = eyes_model.Calibration.fit(clicks)
+        self.steam = eyes_model.Quad([k["steam"] for k in clicks], [k["truth"] for k in clicks])
+
+    def slip(self, c, track, t):
+        """Median slip (pixels) over the track in the SLIP_WINDOW seconds before t."""
+        tr = track[c]
+        if len(tr) == 0:
+            return None
+        return self.cal.slip(c, tr[(tr[:, 0] < t) & (tr[:, 0] > t - eyes_model.SLIP_WINDOW)])
+
+    def predict(self, method, k, track):
+        """Gaze for one click by a method, combining the eyes it has; None if neither."""
+        cal, out = self.cal, {}
+        for c in (0, 1):
+            e = k["eye"][c]
+            if e is None or not cal.has("pupil", c):
+                continue
+            if method == "pupil":
+                out[c] = cal.gaze(c, *e["pupil"])
+            elif method == "glint" and cal.has("glint", c) and e["mid"] is not None:
+                out[c] = cal.fits["glint", c].one(*(e["pupil"] - e["mid"]))
+            elif method == "slip":
+                s = self.slip(c, track, k["t"])
+                if s is not None:
+                    out[c] = cal.gaze(c, *e["pupil"], slip=s)
+        return cal.combine(out)
+
+
+# --- Scoring --------------------------------------------------------------------------
+
+METHODS = ("pupil", "glint", "slip")
+
+
+def report(name, err, total):
+    err = np.asarray([e for e in err if e is not None])
+    if len(err) == 0:
+        print(f"  {name:36s} no clicks")
+        return
+    print(f"  {name:36s} {len(err):3d}/{total}  median {np.median(err):5.2f}  "
+          f"mean {err.mean():5.2f}  90% {np.percentile(err, 90):5.2f} deg")
+
+
+def score(train, test, track, same):
+    """Errors per click for each method; leave-one-out when train and test are the same.
+    "clicks" goes through the test clicks in order, as live: each is predicted with the
+    shift the earlier ones measured (eyes_model.Shift), then teaches it."""
+    errs = {m: [] for m in METHODS + ("clicks", "steam")}
+    model = None if same else Model(train)
+    shifts = {c: eyes_model.Shift() for c in (0, 1)}
+    for i, k in enumerate(test):
+        mdl = Model(train[:i] + train[i + 1:]) if same else model
+        for m in METHODS:
+            g = mdl.predict(m, k, track)
+            errs[m].append(None if g is None else float(np.hypot(*(g - k["truth"]))))
+        errs["steam"].append(float(np.hypot(*(mdl.steam(k["steam"])[0] - k["truth"]))))
+        out = {}
+        for c in (0, 1):
+            e = k["eye"][c]
+            if e is None or not mdl.cal.has("pupil", c):
+                continue
+            tr = track[c]
+            # The glint estimate as live would have had it over the last second, for the hold.
+            for back in (eyes_model.JUMP_HOLD, eyes_model.JUMP_HOLD / 2, 0.0):
+                te = k["t"] - back
+                g = mdl.cal.slip(c, tr[(tr[:, 0] < te) & (tr[:, 0] > te - eyes_model.JUMP_WINDOW)]) if len(tr) else None
+                shifts[c].glint(g, te)
+            out[c] = mdl.cal.gaze(c, *e["pupil"], slip=shifts[c].value)
+            shifts[c].click(mdl.cal.click_shift(c, e["pupil"], k["truth"]), g)
+        errs["clicks"].append(float(np.hypot(*(mdl.cal.combine(out) - k["truth"]))) if out else None)
+    return errs
+
+
+def summary(f, label):
+    cl = f["clicks"]
+    T = np.array([k["truth"] for k in cl])
+    print(f"{label}: {len(cl)} clicks over {f['span'][1] - f['span'][0]:.0f} s, gaze yaw "
+          f"{T[:, 0].min():.0f}..{T[:, 0].max():.0f}, pitch {T[:, 1].min():.0f}..{T[:, 1].max():.0f}")
+    for c in (0, 1):
+        n = sum(k["eye"][c] is not None for k in cl)
+        g = sum(k["eye"][c] is not None and k["eye"][c]["mid"] is not None for k in cl)
+        print(f"  {EYES[c]} eye: pupil before {n} clicks, glint pair before {g}; "
+              f"slip track {len(f['track'][c])} frames with the pair")
+
+
+CALIBRATION = Path.home() / ".local/state/frametop/gaze/eyes/calibration.json"
+
+
+def main(args):
+    if args and args[0] == "--clicks":
+        for cap in map(capture, args[1:]):
+            wall, mono = map(float, (cap / "clocks.txt").read_text().split())
+            lines = (cap / "index.txt").read_text().splitlines()
+            ts = [float(l.split()[3]) for l in (lines[0], lines[-1])]
+            print(f"{cap}: {len(practice_log(cap, *ts, wall, mono))} practice records")
+        return
+    if args and args[0] == "--save":
+        cap = capture(args[1])
+        f = features(cap)
+        cal = eyes_model.Calibration.fit(f["clicks"], {"capture": cap.name, "clicks": len(f["clicks"]),
+                                                     "made": time.strftime("%Y-%m-%d %H:%M")})
+        cal.save(CALIBRATION)
+        print(f"saved {CALIBRATION}: {sorted(f'{n} {EYES[e]}' for n, e in cal.fits)}")
+        return
+    ca = capture(args[0])
+    a = features(ca)
+    summary(a, ca.name)
+    if len(args) > 1:
+        cb = capture(args[1])
+        b = features(cb)
+        summary(b, cb.name)
+        print(f"\nFit on {ca.name}, tested on {cb.name}:")
+        test, track, errs = b["clicks"], b["track"], score(a["clicks"], b["clicks"], b["track"], False)
+    else:
+        print("\nLeave-one-out within the session:")
+        test, track, errs = a["clicks"], a["track"], score(a["clicks"], a["clicks"], a["track"], True)
+    n = len(test)
+    report("SteamVR raw", [k["steam_err"] for k in test], n)
+    steam_press = [k["press_err"] for k in test if k["press_source"] != "own"]
+    report("SteamVR + probe's live correction", steam_press, len(steam_press))
+    live = [k["live_err"] for k in test if k["live_err"] is not None]
+    if live:
+        report("ours live (ft-eyes, as the probe saw it)", live, n)
+        own_press = [k["press_err"] for k in test if k["press_source"] == "own"]
+        report("ours live + probe's live correction", own_press, len(own_press))
+    report("SteamVR + quadratic fit", errs["steam"], n)
+    for m in METHODS:
+        report(f"ours, {m}", errs[m], n)
+    report("ours, clicks (shift from earlier clicks)", errs["clicks"], n)
+    # Like for like: the clicks every method scored.
+    common = [i for i in range(n) if all(errs[m][i] is not None for m in METHODS)]
+    print(f"\nSame {len(common)} clicks for every method:")
+    report("SteamVR + quadratic fit", [errs["steam"][i] for i in common], len(common))
+    for m in METHODS:
+        report(f"ours, {m}", [errs[m][i] for i in common], len(common))
+    if len(args) == 1:
+        for c in (0, 1):
+            tr = track[c]
+            if len(tr) < 20:
+                continue
+            mdl = Model(a["clicks"])
+            ss = [mdl.slip(c, track, t) for t in np.linspace(tr[0, 0] + eyes_model.SLIP_WINDOW, tr[-1, 0], 6)]
+            print(f"  {EYES[c]} eye slip estimate through the session (px): "
+                  + " ".join(f"({s[0]:+.1f},{s[1]:+.1f})" for s in ss if s is not None))
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:] or ["practice1"])

gaze/tracker/lab/ft-eyes-session

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# ft-eyes-session: record the eye cameras and SteamVR's gaze together, for SECONDS (default 10),
+# as a new recording NAME in ~/.local/share/frametop/eyes/captures (FT_EYES_CAPTURES).
+# Usage: gaze/tracker/lab/ft-eyes-session NAME [SECONDS]
+#
+#   frames.raw, index.txt, clocks.txt   every eye-camera frame (ft-eyes-record, from the shared
+#                                       frames of frametop-eyegrab.service; no root)
+#   gaze.jsonl                          ft-gaze's samples (gaze/build/ft-gaze)
+# Both are timed on CLOCK_MONOTONIC_RAW ("t" in gaze.jsonl, the last column of index.txt).
+set -euo pipefail
+lab=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+repo=$(cd "$lab/../../.." && pwd)
+name=${1:?usage: ft-eyes-session NAME [SECONDS]}
+secs=${2:-10}
+out=${FT_EYES_CAPTURES:-$HOME/.local/share/frametop/eyes/captures}/$name
+[ -e "$out" ] && { echo "$out exists" >&2; exit 1; }
+
+# ft-gaze runs in the dev container; start it first, it takes a moment to connect. It quits
+# when its stdin closes (--watch-stdin): killing distrobox doesn't reach it in the container.
+# The recorder makes the folder; ft-gaze's output waits for it.
+tmp=$(mktemp -d)
+sleep $((secs + 3)) | "$HOME/.local/bin/distrobox" enter dev -- "$repo/gaze/build/ft-gaze" \
+  --watch-stdin > "$tmp/gaze.jsonl" 2> "$tmp/gaze.log" &
+gaze=$!
+sleep 2
+python3 "$lab/ft-eyes-record" "$name" "$secs"
+wait $gaze || true
+mv "$tmp/gaze.jsonl" "$tmp/gaze.log" "$out/"
+rmdir "$tmp"
+echo "$(wc -l < "$out/index.txt") frames, $(wc -l < "$out/gaze.jsonl") gaze samples in $out"

gaze/tracker/lab/py

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+# Python with numpy and OpenCV for the lab tools: gaze/tracker/build/venv (gaze/tracker/build.sh
+# makes it in the dev container on the Frame; frame-job's SETUP makes it on the PC). On the
+# Frame's host it runs in the dev container, where it was made.
+# Usage: lab/py lab/TOOL [args]   (from gaze/tracker)
+here=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+py=$here/build/venv/bin/python
+if grep -qx 'ID=steamos' /etc/os-release 2>/dev/null; then
+  exec "$HOME/.local/bin/distrobox" enter dev -- "$py" "$@"
+fi
+[ -x "$py" ] || { echo "no $py: run gaze/tracker/build.sh (or a frame-job job, whose setup makes it)" >&2; exit 1; }
+exec "$py" "$@"

gaze/tracker/requirements.txt

@@ -0,0 +1,5 @@
+# ft-eyes and the lab tools (gaze/tracker/build.sh puts them in build/venv, in the dev
+# container; frame-job's SETUP does the same on the PC). Fedora's python3-opencv would pull in
+# VTK, GDAL, and over a gigabyte of map data; these wheels are about 165 MB.
+numpy==2.5.3
+opencv-python-headless==5.0.0.93

get.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Frametop's one-line installer. In a terminal on the Steam Frame (Konsole in the desktop, or
+# over SSH):
+#
+#   curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash
+#
+# It asks which version to install, clones the repo into ~/frametop (or updates the clone
+# that's there), and runs its install.sh. Run it again to update, or to switch versions.
+# Options (piped, they go after "bash -s --"):
+#   --stable        the main branch: tested releases (the default for a new install)
+#   --experimental  the experimental branch: the newest features, less tested
+#   --dir DIR       where the repo goes (default ~/frametop)
+#   --clone-only    get or update the repo, but don't run install.sh
+#   --yes, --no-bluetooth   passed to install.sh (--yes also answers this script's question:
+#                   the version already there, or stable)
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+usage: get.sh [--stable | --experimental] [--dir DIR] [--clone-only] [--yes] [--no-bluetooth]
+piped: curl -fsSL https://deejanuz.github.io/frametop/get.sh | bash -s -- [options]
+EOF
+}
+
+# Everything happens in main, called on the last line, so a download cut short runs nothing.
+main() {
+  local repo=https://github.com/DeeJanuz/frametop.git dir=$HOME/frametop branch= clone_only=0
+  local yes=0 tty=0 current= def answer
+  local pass=()
+  while [ $# -gt 0 ]; do
+    case $1 in
+      --stable) branch=main ;;
+      --experimental) branch=experimental ;;
+      --dir) dir=${2:?--dir needs a folder}; shift ;;
+      --clone-only) clone_only=1 ;;
+      --yes) yes=1; pass+=("$1") ;;
+      --no-bluetooth) pass+=("$1") ;;
+      -h|--help) usage; return 0 ;;
+      *) echo "unknown option: $1" >&2; usage >&2; return 2 ;;
+    esac
+    shift
+  done
+
+  if ! { grep -qx 'ID=steamos' /etc/os-release && grep -qE '^VARIANT_ID="?vr"?$' /etc/os-release; } 2>/dev/null; then
+    echo "Frametop installs on a Steam Frame (SteamOS, VR variant). Run this in a terminal on the headset." >&2
+    return 1
+  fi
+  # Piped into bash, stdin is this script: the questions (here and install.sh's) read the terminal.
+  { : </dev/tty; } 2>/dev/null && tty=1
+  if [ "$tty" = 0 ] && [ "$yes" = 0 ]; then
+    echo "This asks questions, and there's no terminal to ask in: run it in one, or add --yes." >&2
+    return 1
+  fi
+
+  if [ -e "$dir/.git" ]; then
+    git -C "$dir" remote get-url origin 2>/dev/null | grep -qi 'frametop' ||
+      { echo "$dir is a git repo, but not Frametop's. Pick another folder with --dir." >&2; return 1; }
+    current=$(git -C "$dir" branch --show-current)
+  elif [ -e "$dir" ]; then
+    echo "$dir is there and isn't Frametop's repo. Move it, or pick another folder with --dir." >&2
+    return 1
+  fi
+
+  if [ -z "$branch" ]; then
+    def=main
+    [ "$current" = experimental ] && def=experimental
+    if [ "$yes" = 1 ]; then
+      branch=$def
+    else
+      echo "Which version of Frametop?"
+      echo "  1) stable: the main branch, tested releases"
+      echo "  2) experimental: the newest features, less tested"
+      [ -n "$current" ] && echo "(installed now: $current)"
+      read -r -p "Choose 1 or 2 [$([ "$def" = main ] && echo 1 || echo 2)]: " answer </dev/tty || answer=
+      case ${answer:-$def} in
+        1|main|s*) branch=main ;;
+        2|experimental|e*) branch=experimental ;;
+        *) echo "not 1 or 2: $answer" >&2; return 2 ;;
+      esac
+    fi
+  fi
+
+  if [ ! -e "$dir" ]; then
+    echo "Cloning Frametop ($branch) into $dir"
+    git clone --branch "$branch" "$repo" "$dir"
+  else
+    if ! git -C "$dir" diff --quiet || ! git -C "$dir" diff --cached --quiet; then
+      echo "$dir has changes of its own. Commit or stash them first (git -C $dir status)." >&2
+      return 1
+    fi
+    echo "Updating $dir to the latest $branch"
+    git -C "$dir" fetch --quiet origin
+    if [ "$current" != "$branch" ]; then
+      if git -C "$dir" show-ref --verify --quiet "refs/heads/$branch"; then
+        git -C "$dir" switch --quiet "$branch"
+      else
+        git -C "$dir" switch --quiet --track -c "$branch" "origin/$branch"
+      fi
+    fi
+    git -C "$dir" merge --ff-only --quiet "origin/$branch" ||
+      { echo "$dir has commits of its own on $branch, so it can't just move to the latest. Update it by hand." >&2; return 1; }
+  fi
+  echo "Frametop $branch: $(git -C "$dir" log -1 --format='%h %s')"
+
+  if [ "$clone_only" = 1 ]; then
+    echo "Install with: cd $dir && ./install.sh"
+    return 0
+  fi
+  cd "$dir"
+  if [ "$tty" = 1 ]; then
+    ./install.sh "${pass[@]}" </dev/tty
+  else
+    ./install.sh "${pass[@]}" </dev/null
+  fi
+}
+
+main "$@"

hands/.gitignore

@@ -0,0 +1,5 @@
+# Model sources that tools/convert_models.py downloads; the converted ncnn models are kept
+models/onnx/
+models/*.task
+# frame-job settings (where replays run, and the lab's capture folder): personal, not shipped
+.frame-job

hands/Makefile

@@ -0,0 +1,49 @@
+# Hand tracking, built into build/ (hands/build.sh runs this in the dev container):
+#   make          ft-camd (camd/: runs on the host, so linked statically) and ft-hands (track/)
+#   make tools    ft-handreplay and ft-ringplay, for recordings
+# The first build fetches ncnn (NCNN_TAG) and builds it into build/ncnn, which takes a few
+# minutes. NCNN=DIR uses an ncnn install already built instead.
+NCNN_TAG  = 20260526
+NCNN     ?= build/ncnn/install
+CFLAGS   ?= -O2 -g -Wall -Wextra -Wno-unused-parameter
+CXXFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter -Wno-psabi
+CXXFLAGS += -std=c++17 -fopenmp -I$(NCNN)/include/ncnn
+LDLIBS    = $(NCNN)/lib/libncnn.a -ljsoncpp -fopenmp -lpthread
+
+CAMD  = camd/camd.c camd/tp.c camd/xrcams.c
+TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp
+HDR   = $(wildcard track/*.h) camd/fhring.h include/fh_hands.h include/fh_gestures.h
+
+all: build/ft-camd build/ft-hands
+tools: build/ft-handreplay build/ft-ringplay
+
+build/ft-camd: $(CAMD) camd/tp.h camd/xrcams.h camd/fhring.h
+	@mkdir -p build
+	$(CC) $(CFLAGS) -static -o $@ $(CAMD) -lm
+
+build/ft-hands: track/main.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
+	@mkdir -p build
+	$(CXX) $(CXXFLAGS) -o $@ track/main.cpp $(TRACK) $(LDLIBS)
+
+build/ft-handreplay: track/replay.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
+	@mkdir -p build
+	$(CXX) $(CXXFLAGS) -o $@ track/replay.cpp $(TRACK) $(LDLIBS)
+
+build/ft-ringplay: track/ringplay.cpp track/record.h camd/fhring.h
+	@mkdir -p build
+	$(CXX) $(CXXFLAGS) -o $@ track/ringplay.cpp
+
+# ncnn as frame-hands built it (the models were converted and quantized for it), minus its tools
+build/ncnn/install/lib/libncnn.a:
+	rm -rf build/ncnn && mkdir -p build/ncnn
+	git clone -q --depth 1 --branch $(NCNN_TAG) -c advice.detachedHead=false https://github.com/Tencent/ncnn.git build/ncnn/src
+	cmake -S build/ncnn/src -B build/ncnn/build -G Ninja -Wno-dev -DCMAKE_BUILD_TYPE=Release \
+	  -DCMAKE_INSTALL_PREFIX=$(CURDIR)/build/ncnn/install -DCMAKE_INSTALL_LIBDIR=lib -DNCNN_VULKAN=OFF \
+	  -DNCNN_OPENMP=ON -DNCNN_INT8=ON -DNCNN_SIMPLEOCV=ON -DNCNN_BUILD_TOOLS=OFF -DNCNN_BUILD_EXAMPLES=OFF \
+	  -DNCNN_BUILD_BENCHMARK=OFF -DNCNN_BUILD_TESTS=OFF -DNCNN_PYTHON=OFF > build/ncnn/cmake.log
+	cmake --build build/ncnn/build --target install > build/ncnn/build.log
+
+clean:
+	rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay
+
+.PHONY: all tools clean

hands/README.md

@@ -0,0 +1,221 @@
+# Hands (experimental, deferred)
+
+Hand tracking from the headset's own cameras. It's deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it and the README doesn't list it. It still builds and runs, installed by hand (below), for working on it.
+
+It serves two things in Frametop:
+
+- **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro.
+- **Pinches and grips:** with `POINTER_HANDS=1`, the pointer helper takes them as clicks and drags. Look at something and pinch to click it, with the eye tracker doing the looking (`gaze/`), or close your hand to press and drag what the pointer is on. See "Pinches and grips in the pointer" below.
+
+Two programs, each a user service that stops when SteamVR does:
+
+- `ft-camd` (`camd/`, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host.
+- `ft-hands` (`track/`, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container.
+
+They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` doesn't install it.
+
+For the cutouts alone, `hands/ft-cutouts on` starts the same two programs with ft-hands' `--no-gestures`: your hands show through the screens, and no pinch or grip is detected, so nothing clicks or drags. It runs this checkout's build as transient user units, so it needs `hands/build.sh` and ft-camd's capabilities (`hands/run.sh caps`) but not `hands/run.sh install`. It and `ft-handsctl on` stop each other's services, and it stops with SteamVR too.
+
+```
+ft-handsctl on | off            # on the Frame: start or stop hand tracking (SteamVR must be running)
+ft-handsctl status              # the services, and ft-hands' last status lines
+ft-handsctl log [lines]
+ft-handsctl cutouts on|off|state  # ft-screens' hand cutouts, without stopping tracking
+ft-handsctl gestures            # pinches and grips, live (tools/watch_gestures.py --distance)
+
+hands/ft-cutouts on | off | status   # the cutouts only: no pinches or grips
+
+hands/run.sh install            # build, give ft-camd its capabilities (sudo, once per build), install disabled
+hands/run.sh start|stop         # start or stop the services
+hands/run.sh restart            # after changing a setting
+hands/run.sh status
+hands/run.sh log [lines]
+hands/run.sh caps               # after rebuilding ft-camd (a rebuild clears its capabilities)
+hands/run.sh uninstall
+```
+
+Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides them), read when ft-camd and ft-hands start:
+
+- `HANDS_SWAP_SIDES=1`: the two side cameras' names are swapped (see ft-camd below). Check with `tools/check_sides.py --ring`.
+- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
+- `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
+- `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
+
+The pointer helper's `POINTER_HANDS` and `POINTER_PINCH_*`/`POINTER_GRIP_*` settings are in "Pinches and grips in the pointer" below.
+
+Files, all in `/run/user/UID/frametop-hands/` (private to the user; not `/run/user/UID/frametop/`, which the desktop session deletes whenever it starts):
+
+| File | Written by | Layout | Read by |
+| --- | --- | --- | --- |
+| `cam-ring` | ft-camd | `camd/fhring.h` | ft-hands, `tools/ring.py` |
+| `hands` | ft-hands | `include/fh_hands.h` | ft-screens (`screens/handcut.cpp`) |
+| `gestures` | ft-hands | `include/fh_gestures.h` | the pointer helper (`pointer/helper/ft-pointer.cpp`), `tools/watch_gestures.py` |
+
+The source keeps the `fh_` names and magic strings of frame-hands, the project it started as, so recordings made with it still work.
+
+## ft-camd
+
+XRService owns the headset cameras. ft-camd borrows its DMA-BUFs read-only with `pidfd_getfd`, the same way FrameEyeCameraFeed does. It never touches XRService's V4L2 descriptors. `discovery` in `camd/xrcams.c` is adapted from FrameEyeCameraFeed (MIT, see `camd/LICENSE.FrameEyeCameraFeed`).
+
+Polling buffers for changes can catch a frame while the camera is still writing it. Instead, ft-camd listens to the `v4l2:v4l2_dqbuf` tracepoint, which fires when XRService takes a buffer. It gives the buffer index, the sequence number and the capture timestamp. ft-camd learns which DMA-BUF holds each V4L2 index by watching which buffer changes at each dequeue:
+
+- Right after XRService allocates its buffers, the mapping is allocation order.
+- After XRService restarts streaming, the order is shuffled, and the mapping is learned index by index.
+- The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart.
+- It also re-maps an index on the fly when its buffer holds no new frame.
+
+**Privileges.** Setting up needs three things. `pidfd_getfd` on XRService needs `CAP_SYS_PTRACE`, because the Frame has `ptrace_scope=1`. The system-wide tracepoint needs `CAP_PERFMON`, because `perf_event_paranoid` is 2. Its format files are root-only, which needs `CAP_DAC_READ_SEARCH`. `hands/run.sh install` gives the binary those capabilities with `sudo setcap`. File capabilities need a filesystem mounted without `nosuid`. The Frame's `/home` (ext4) has no `nosuid`. ft-camd drops them all once it has set up, before it reads a frame, and then runs as you. XRService runs as you too. It also runs under `sudo`, for trying it by hand, and then drops to the user who ran sudo. It reads nothing from the ring's readers.
+
+The ring is mode 0600, in a folder only you can write. Frame handling:
+
+- Only complete, bright frames are published. The cameras alternate a normal exposure with a near-black one, so each camera gets 30 of its 60 fps.
+- A copy torn by the camera overwriting the buffer is dropped.
+- Each copy takes about 0.1 ms, and a cache sync about 0.15 ms.
+
+Options:
+
+- `--dark R`: a frame dimmer than R times the camera's recent brightest counts as near-black. Default 0.4.
+- `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands.
+- `--with-color`: also publish the two Arcturus colour cameras, flagged `FH_CAM_COLOR`. The service leaves it off and runs the mono cameras only (see "Known issues"). To try the colour cameras, add it to `ExecStart` in `hands/frametop-camd.service` and run `hands/run.sh install` again; ft-hands then picks the cameras by the light. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232). They run at `--color-idle` (2 fps), enough for ft-hands to tell how bright it is, until a reader asks for more in `/run/user/UID/frametop-hands/color-fps` (ft-hands writes 30 while it tracks or records with them), up to `--color-fps` (30; the cameras run at 60). `HANDS_CAMERAS=mono` leaves them out. Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core.
+- Each mono camera's latest near-black frame's mean goes in the ring (`dark_mean`): a short fixed exposure, so it follows the room's IR light, sunlight above all.
+- The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 colour cameras.
+- Colour isn't reliable yet. In the lit-room test of 2026-09-30, the colour cameras kept losing their buffer mapping while the headset was worn: 30 frames in a row looked unchanged, the camera relearned, and after 5 relearns ft-camd exited. Each relearn probed all 32 colour buffers, a whole-buffer cache sync each, which also made the mono cameras miss frames. Runs with the headset idle had none of this. So the passthrough compositor may be writing into the colour buffers while Room View shows. Since then a colour camera never takes the mono ones down: it probes at most 4 buffers a frame, and one that goes stale twice in a row is paused (10 s, doubling up to 160 s) and learned again, without ft-camd exiting. Whether a frame is new is judged on the luma rows only: the chroma after them hardly changes in a lit room.
+- `FT_CAMD_DEBUG=1` in the environment: at each stale colour frame, ft-camd logs to stderr the camera, the frame's time, V4L2 index and sequence number, and, for every candidate buffer, how many of its sampled words changed, in all and in the last eighth of the samples.
+- `--sensor S`: only the mono cameras whose sensor name contains S.
+- `--status S`: a status line every S seconds (0: never).
+
+It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. The service starts it again, and it attaches to the new buffers.
+
+**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. With the headset on, looking at a room with some texture, `tools/check_sides.py --ring` says whether the names are right (exit 0), swapped (exit 3), or it can't tell (exit 2). When they're swapped, set `HANDS_SWAP_SIDES=1`. The colour cameras are video3 (`arcimx616 0-0010`) and video0 (`0-001a`); which of them is `passthrough_left` in the module's calibration is for `tools/check_color.py` to settle, on a recording with texture in view.
+
+## ft-hands
+
+```
+hands/build/ft-hands                  # status every 5 s; Ctrl+C to stop
+hands/build/ft-hands --int8           # the 8-bit models (models/ncnn/*-int8.ncnn.*)
+```
+
+Run it in the dev container (`distrobox enter dev -- ...`). It reads the factory calibration from `/persist` (`/run/host/persist` in the container).
+
+Options:
+
+- `--threads N`: model threads, pinned to the `--cpus` list. Default 3.
+- `--cpus LIST`: CPUs for the model threads and the main loop. Default `5,6,7` (`HANDS_CPUS`). SteamOS starts user processes on CPUs 0-4, and XRService's head tracking runs on 2-3. With the headset on, a step took 8.4 ms on 5-7 against 13.2 ms on 2-4, and SteamVR's frame timing didn't change (2026-09-29, three rounds of the same replayed frames).
+- `--contrast MODE` or `PALM/HAND`: how crops are equalized before the models see them: `clahe[:CLIP]`, `none`, or `stretch` (1st-99th percentile). Default `clahe:2/none`. In the dim recording, CLAHE let the palm search find about 10% more hands, but it made the landmarks jitter more (published median 6.9 mm, against 6.0 mm with plain landmark crops).
+- `--swap-sides`: swap the two side cameras (`HANDS_SWAP_SIDES`, see ft-camd).
+- `--seconds N`: stop after N seconds.
+- `--status S`: how often to print status, in seconds.
+- `--models DIR`: where the models are.
+- `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
+- `--no-publish`: don't write the hands and gestures files.
+- `--no-gestures`: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. `ft-cutouts` runs it this way.
+- `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
+- `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
+- `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.
+- `--ring PATH`: read frames from another ring, such as `ft-ringplay`'s.
+- `--cams auto|mono|color|all` (`HANDS_CAMERAS`, default `auto`): which cameras to track with. The mono IR cameras light the hands themselves and track well in dim rooms, but in bright light they expose for the room and the hands come out dark. The colour pair is the other way round. `auto` goes by the colour frames' mean brightness: at `--bright-on` (`HANDS_BRIGHT_ON`, 40) or over for 2 s it tracks with `--bright` (`HANDS_BRIGHT`: `all`, every camera, the default, or `color`), and under `--bright-off` (`HANDS_BRIGHT_OFF`, 25) for 2 s with the mono cameras again. A dim evening room read 9. The switch is logged (`cameras: mono -> all (...)`), and the status line gives the colour level, the mono cameras' ambient IR, and how many steps had colour frames. Colour frames arrive on their own schedule, so a step holds the mono set, the colour pair, or both, and views wait in their camera for its next frame. With no colour cameras in the ring (ft-camd without `--with-color`, as the service runs it), ft-hands tracks with the mono cameras whatever this says.
+- `--color-left NODE` (`HANDS_COLOR_LEFT`, `color_video0`) and `--color-crop subtract|none` (`HANDS_COLOR_CROP`, `subtract`): how the colour module's calibration maps onto the images. Not settled yet: `tools/check_color.py` on a recording with a lit, textured view tells.
+- `--grip-begin R`, `--grip-end R`: the grip detector (below). Defaults 1.2 and 1.45.
+- `--gesture-log`: print what the pinch and grip detectors measure, 10 times a second: each hand's thumb-to-index distance (world and triangulated), its palm-down reading and its finger curl.
+
+**Gestures** (`/run/user/UID/frametop-hands/gestures`, `include/fh_gestures.h`), for the pointer helper:
+
+- A pinch: the thumb and index tips within 2 cm, ending past 3.5 cm (see "Pinch" below).
+- A grip, a closed hand: every finger's tip nearer the wrist than 1.2 times its knuckle is (from the model's 3D hand, so hand size doesn't matter), ending when they open past 1.45 on average. It begins only on a hand seen open within the last second (closing it is the gesture), with the palm at most 35 degrees below straight ahead and at least 15 cm in front of the eyes, and not with the thumb within 3 cm of the index tip (that's a pinch with the other fingers curled). A grip ends a pinch on the same hand, as lost. In the 2026-09-30 lit recording (no deliberate fists), the checks cut false grips from 14 to 6, all with the hands on the desk while looking down at it. The pointer helper ignores grips that begin more than `POINTER_GRIP_BELOW` (0.35 m) below the eyes, which it can tell and ft-hands can't.
+- `tools/watch_gestures.py --distance` shows both live; `ft-handreplay --timeline` logs them and each hand's finger curl.
+
+The status line also says how often a hand was on each side (by where the wrist is), and why views and hands came and went: views lost (the landmark model stopped seeing the hand), handoff misses (a crop projected from the hand's 3D position found nothing), duplicates, splits (two views disagreed in 3D), and hands created, merged and forgotten.
+
+### Scheduling
+
+- Each hand is tracked in its best two cameras, the way MediaPipe tracks: the landmark model runs on a crop placed from the previous landmarks, with no palm detection.
+- A hand seen in too few cameras is projected into the others through the calibration. Where it lands well inside a camera, that camera gets a crop to try. This is how a hand raised out of the side cameras reaches the upper ones.
+- The palm detector runs only while fewer than two hands are tracked, at most 5 times a second, on a few zoomed tiles per search. Tiles are picked in proportion to how likely hands are there. Each tile is turned so the expected shoulder-to-hand direction points up.
+- Frame sets are processed at 30 Hz while a hand moves faster than 0.25 m/s (or a pinch is down or closing), at 15 Hz otherwise, and at 5 Hz while no hand is in view.
+
+### 3D
+
+- **Two or more views:** each landmark is triangulated from the camera rays, weighted by the model's presence score. The median ray distance is reported as the residual.
+- **Pairing views across cameras.** The side cameras sit side by side, so two hands next to each other at the same height fall on the same epipolar lines, and rays to two different hands can nearly meet close to the cameras. That made phantom hands 12-15 cm in front of the eyes, which tore holes through the screens. Each step now scores every way of pairing the views in two cameras and keeps the best. A pair scores well when its rays meet, when each view's apparent size matches the triangulated distance, and when the model calls both the same hand. The size check uses a fixed prior: with the model's average hand, clean pairs measure 0.71-1.51 times the one-view distance, and mismatched pairs mostly far less.
+- **One view:** depth comes from the model's metric world landmarks, their spread across the palm against the angle it covers in the image, scaled by the user's hand size (learned while two views are available). That distance is off by 10-30% and wanders about 10% between frames, so a hand that drops to one camera keeps its last distance and drifts toward the one-view guess by 10% a frame.
+- **Smoothing.** The published landmarks go through a One Euro filter: it smooths hard while the hand is still (tracking noise is several mm per frame) and hardly at all while it moves fast. The palm speed that sets the update rate is the filtered one; the raw speed read about 0.25 m/s from noise alone.
+- **Capsules.** Forearms follow the hand's own axis, and nothing within 12 cm in front of the eyes is published.
+
+How good the depth is, measured from recordings (2026-09-30, `--depth` below): the two lower cameras see the hands about 77% of the time, a lower and an upper camera 7-12%, and one camera 12-15%. Depth is the noisy direction. With the lower pair, it jitters 4-6 times as much as sideways position (published: 3-7 mm against 1-2 mm). The one-camera guess is a median 2-6 cm off. When a camera drops out, drifting 10% a frame toward that guess is worse than keeping the last distance (after 0.5 s a median 23-30 mm off, against 11-12 mm).
+
+## Pinch
+
+ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gestures file. The layout, and how to read it without missing quick taps, is in `include/fh_gestures.h`.
+
+- A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost).
+- The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. `--pinch-triangulated` uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. In the dim recording, typing still gave 2 pinches a minute (see the next point).
+- `--pinch-palm-down MAX` holds back pinches begun with the palm facing down (MAX is the palm normal's share of the head's up axis). The default, 1, turns it off. A close held back that way has to open again before a pinch can begin. Typing curls the thumb onto the index: in the lit recording of 2026-09-30, typing on a keyboard in the lap began 23 pinches in about 2 minutes, all with the palm facing down (0.69-1.00), while the 26 deliberate ones read 0.00-0.50. But in the headset, deliberate pinches with the hand raised in front read 0.90-0.99 too, so the limit is off. Typing is caught by the pointer helper instead: the input relay tells it when you press a key, and no pinch begins within `POINTER_PINCH_TYPING` of one.
+- A hand a pinch is down on stays with that side until the pinch ends. The left/right call is a running average of the model's, and when it flipped mid-pinch, the other side took the same hand and both sides pinched at once.
+- The pinch point is between the index and middle knuckles, which hold still while the fingers open and close. The tips' midpoint moved 1-2 cm as a pinch opened, which dragged every release off its press. A drag is the pinch point now, minus where it was when the pinch began, both turned into the room with the HMD pose at their capture times.
+- `tools/watch_gestures.py` prints begins, ends and drag offsets live, and `--distance` prints each hand's distance.
+
+## Pinches and grips in the pointer
+
+With `POINTER_HANDS=1`, the pointer helper (`pointer/helper/ft-pointer.cpp`) reads the gestures file every frame. It's off by default.
+
+- **Pinch to click.** In gaze mode a pinch works like the mouse's press: the pointer stops where the gaze put it, and the click comes when the pinch opens, where the pointer is then. A quick tap clicks where you looked. Held, the pinching hand moves the pointer to correct the gaze, and the correction is a lesson for the gaze tracker, as with the mouse.
+- **Without gaze mode,** a pinch is a real press, like the mouse's button: pressed when it closes, released when it opens, and while it's held the hand drags the pointer. A tap is still a click where the pointer is.
+- **Grip to drag.** Closing the hand presses where the pointer is, the hand moves the pointer, and opening the hand releases. So a title bar moves its window, a panel's grab bar carries the panel, and text gets selected.
+- A pinch ended by losing the hand, or by a grip taking over, doesn't click.
+- The hand's movement is taken in the room, from where the eye was when the gesture began, so turning your head doesn't move the pointer. The first gesture while the pointer is off only wakes it. Gestures are ignored in a VR game (unless the dashboard is up), with the headset off, and while the mouse's button is held.
+
+Settings in `~/.config/frametop.conf`:
+
+- `POINTER_HANDS` (0): 1 turns pinches and grips on.
+- `POINTER_PINCH_GAIN` (0.5): a held pinch moves the pointer this many times the hand's angle, seen from the eye. Under 1 gives precision.
+- `POINTER_PINCH_DEADZONE` (1.5): how many degrees the pinching hand moves before the pointer does, so a tap's jitter and the pinch point shifting as the fingers close don't move it.
+- `POINTER_GRIP_GAIN` (1): a grip moves the pointer this many times the hand's angle.
+- `POINTER_GRIP_BELOW` (0.35): grips that begin more than this many metres below the eyes are ignored, because hands resting on a desk curl like a loose fist. Pinches have no such limit: deliberate ones sat 0.35-0.45 m below the eyes with the elbow resting.
+- `POINTER_PINCH_TYPING` (1): no pinch begins within this many seconds of a key press, because typing touches thumb to index.
+
+## Recordings
+
+`hands/build.sh --tools` also builds the offline tools.
+
+`ft-handreplay DIR` runs a recording through the tracker with the live scheduling and reports how well it kept the hands: hands per set, left and right coverage, track lengths, pinches, jitter, and the same reasons as the status line.
+
+```
+hands/build/ft-handreplay ~/.local/share/frametop/hands/rec-20260929-120000 --cost --oracle 10 --timeline /tmp/tl.txt
+```
+
+- `--cost`: instead of timing the steps, charge each round of model calls what it typically costs live (10 ms landmarks, 18 ms palms), so results repeat exactly.
+- `--oracle N`: every N-th set, also search every tile of every camera, and report how often the tracker had the hands that full search could find.
+- `--slow F`: live, the tracker skips sets that arrive while it's busy. Replay counts each step's time times F as busy (default 1; the headset is busier live).
+- `--timeline FILE`: a line per processed set and hand, with pinch events and distances.
+- `--cams mono|color|all`: which cameras to track with (default `mono`). `color` tracks with the Arcturus pair alone, for comparing it with the IR cameras on the same recording. It needs a recording made with `ft-camd --with-color`. `--color-left NODE` (`color_video0` or `color_video3`) and `--color-crop subtract|none` say how the module's calibration maps onto the images; `tools/check_color.py` finds out.
+- `--depth FILE`: a line per hand per processed set for `tools/depth_report.py`, which measures the depth without ground truth: how the hands were seen, the noise along the line of sight against across it, each camera's one-view distance against the triangulated one, and what a camera dropping out would do.
+- The pinch, contrast and presence options are ft-hands'.
+
+`ft-ringplay DIR --ring PATH [--from S] [--to S] [--loop]` publishes a recording into a ring file in real time, as ft-camd would, so `ft-hands --ring PATH --no-publish` runs the same frames run after run. It needs no privileges, and it skips the dark frames.
+
+## Tools
+
+Python, with NumPy and OpenCV. `setup/dev-container.sh` doesn't install them, because Fedora's `python3-opencv` pulls in over a gigabyte; in the dev container, run `sudo dnf install python3-numpy python3-opencv` once. Off the Frame, `FRAME_JOB_DEVICE_ROOT` can point at a folder with copies of the headset's calibration files.
+
+- `tools/check_sides.py --ring` (or a recording): are the side cameras named right?
+- `tools/check_color.py REC`: how the colour module's calibration maps onto its images.
+- `tools/show_set.py REC`: a recording's frame sets as images.
+- `tools/watch_gestures.py [--distance]`: pinches and grips, live.
+- `tools/depth_report.py DEPTH`: the depth measures above.
+- `tools/cut_sets.py REC OUT [--sets N | --at I,J,...]`: copies a few frame sets (by default 8, spread evenly) out of a recording into a small one, to look at or check elsewhere without moving gigabytes. Plain Python, so it also runs on the Frame's host.
+- `tools/convert_models.py`: how `models/ncnn` was made from the OpenCV Zoo ONNX ports of MediaPipe's models (see `models/NOTICE`).
+
+To try the hand cutouts without restarting the desktop, `screens/build/ft-handtest [--distance m] [--width m] [--seconds s]` (built by `screens/build.sh`, run in the dev container, with hand tracking on) shows a test panel of its own, a light grid 1 m wide and 0.8 m ahead by default, and cuts your hands out of it the way ft-screens cuts them out of the screens.
+
+## Build
+
+`hands/build.sh` builds in the dev container on the Frame, into `hands/build/`, with `hands/Makefile`. The first build fetches ncnn at a pinned tag (`NCNN_TAG` in the Makefile) and builds it into `hands/build/ncnn`, which takes a few minutes; `NCNN=DIR` points at an ncnn install already built instead. ft-camd is linked statically, because it runs on the host, which has an older glibc than the container.
+
+## Known issues
+
+- **The side cameras can come out swapped.** ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and some XRService restarts reverse it. For now it's caught by hand: `tools/check_sides.py --ring`, then `HANDS_SWAP_SIDES=1`. It needs a fix in ft-camd, or at least an automatic check when it starts.
+- **The colour cameras can't be used while the headset is worn.** The colour module then writes only a half-size image into the top-left quarter of its buffers, and ft-camd drops those frames. So the service runs the mono cameras only, and tracking in bright light, where the mono cameras see dark hands, doesn't get the colour pair's help.
+- **The colour calibration mapping isn't settled.** Which colour camera is `passthrough_left` (`HANDS_COLOR_LEFT`) and how the module's crop applies (`HANDS_COLOR_CROP`) still need `tools/check_color.py` on a recording with a lit, textured view.
+- **Depth when one camera loses the hand.** A hand seen in one camera drifts 10% per update toward the one-camera depth guess (`kMonoDepthGain`, 0.1, in `track/tracker.cpp`). In the 2026-09-30 replays that was worse than keeping the last distance (see "3D" above). A smaller gain, such as 0.02, is the next thing to try.
+- **Pinches aren't reliable enough for everyday use yet.** That's why hand tracking stays off until `ft-handsctl on`, and `POINTER_HANDS` is 0 by default.
+- **Floating windows don't get hand cutouts.** Their panels show crops of the client buffer, which the cutouts' side-by-side buffer doesn't match (`screens/vr.cpp`, `UpdateCutouts`).

hands/build.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Build hand tracking in the dev container on the Frame, into hands/build/: ft-camd and ft-hands,
+# and with --tools also ft-handreplay and ft-ringplay. The first build fetches ncnn and builds
+# it (a few minutes); NCNN=DIR, an ncnn install already on the Frame, skips that.
+# A rebuilt ft-camd has lost its capabilities: hands/run.sh install sets them again.
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+targets=all
+[ "${1:-}" = --tools ] && targets="all tools"
+"$root/scripts/sync.sh" >/dev/null
+exec "$root/scripts/frame.sh" -C hands "make -s ${NCNN:+NCNN=$NCNN} $targets && echo built \$(ls build/ft-* | tr '\n' ' ')"

hands/camd/LICENSE.FrameEyeCameraFeed

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Curtis English
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hands/camd/fhring.h

@@ -0,0 +1,91 @@
+/*
+ * fhring - the shared-memory frame ring ft-camd writes and trackers read.
+ *
+ * One file, /run/user/UID/frametop-hands/cam-ring (FH_RING_NAME in the user's runtime
+ * folder; the folder is private to the user), holds a header, then for each camera
+ * a few slots, each a slot header followed by the image rows packed tightly
+ * (stride == width for 8-bit mono). Only complete, bright frames are published.
+ *
+ * Writer, for frame n of a camera:  slot = n % nslots
+ *   slot.seq = 2n+1; write slot fields and pixels; slot.seq = 2n+2; cam.latest = n
+ * Reader:
+ *   n = cam.latest; read slot.seq, expect 2n+2; copy; re-read slot.seq; if it
+ *   changed the copy is torn, retry with the new latest.
+ *
+ * All multi-byte fields are little-endian; offsets are fixed so Python can read
+ * them with struct (tools/ring.py mirrors this file).
+ */
+
+#pragma once
+
+#include <assert.h>
+#include <stdint.h>
+
+#define FH_RING_MAGIC     "FHRING01"
+#define FH_RING_VERSION   1
+#define FH_RING_MAX_CAMS  8
+#define FH_RING_SLOTS     4
+#define FH_RING_NAME      "frametop-hands/cam-ring"    /* in /run/user/UID */
+
+enum {
+    FH_FMT_GREY8 = 0,
+};
+
+enum {
+    FH_CAM_DARK = 1u << 0,          /* the near-black exposures between this node's */
+                                    /* normal frames (ft-camd --with-dark)          */
+    FH_CAM_COLOR = 1u << 1,         /* an Arcturus color camera's luma, downscaled  */
+                                    /* (ft-camd --with-color). Not synced with the  */
+                                    /* mono cameras, and capture_ns is on its own   */
+                                    /* clock: line it up with them by dqbuf_ns      */
+};
+
+typedef struct {
+    char              sensor[32];   /* media entity, e.g. "og01a1b 4-0060"          */
+    char              name[32];     /* calibration name if known, else sensor slug  */
+    int32_t           node;         /* N of /dev/videoN                             */
+    uint32_t          format;       /* FH_FMT_*                                     */
+    uint32_t          width;
+    uint32_t          height;
+    uint32_t          stride;       /* bytes per row in the ring                    */
+    uint32_t          nslots;
+    uint64_t          slot_offset;  /* file offset of slot 0                        */
+    uint64_t          slot_bytes;   /* slot header + image, 64-byte aligned         */
+    volatile uint64_t latest;       /* newest published frame number, 0 = none yet  */
+    uint64_t          published;    /* frames published                             */
+    uint64_t          dropped;      /* dark, stale or torn frames not published     */
+    uint32_t          flags;        /* FH_CAM_*                                     */
+    float             dark_mean;    /* mono: mean luma of its latest near-black     */
+                                    /* frame (a short fixed exposure, so it follows */
+                                    /* the room's IR light, sunlight above all);    */
+                                    /* 0 before the first                           */
+    uint8_t           reserved[24];
+} fh_ring_cam_t;                    /* 160 bytes */
+
+typedef struct {
+    volatile uint64_t seq;          /* 2n+1 while frame n is written, 2n+2 when done */
+    uint64_t          frame;        /* n                                            */
+    uint64_t          capture_ns;   /* V4L2 timestamp (camera clock)                */
+    uint64_t          dqbuf_ns;     /* CLOCK_MONOTONIC when XRService dequeued it   */
+    uint64_t          publish_ns;   /* CLOCK_MONOTONIC when the copy finished       */
+    uint32_t          v4l2_seq;     /* V4L2 sequence number                         */
+    float             mean;         /* mean luma on a sparse grid                   */
+    uint8_t           reserved[16];
+} fh_ring_slot_t;                   /* 64 bytes, image follows */
+
+typedef struct {
+    char              magic[8];     /* FH_RING_MAGIC                                */
+    uint32_t          version;
+    uint32_t          header_bytes; /* sizeof(fh_ring_hdr_t)                        */
+    uint32_t          ncams;
+    uint32_t          reserved0;
+    uint64_t          file_bytes;
+    int64_t           writer_pid;
+    volatile uint64_t heartbeat_ns; /* CLOCK_MONOTONIC, refreshed at least every 0.2 s */
+    uint8_t           reserved[16];
+    fh_ring_cam_t     cams[FH_RING_MAX_CAMS];
+} fh_ring_hdr_t;
+
+static_assert(sizeof(fh_ring_cam_t) == 160, "fh_ring_cam_t layout");
+static_assert(sizeof(fh_ring_slot_t) == 64, "fh_ring_slot_t layout");
+static_assert(sizeof(fh_ring_hdr_t) == 64 + 160 * FH_RING_MAX_CAMS, "fh_ring_hdr_t layout");

hands/camd/tp.c

@@ -0,0 +1,427 @@
+/*
+ * tp - read kernel tracepoints system-wide through perf_event_open.
+ */
+
+#define _GNU_SOURCE
+
+#include "tp.h"
+
+#include <errno.h>
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/epoll.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <time.h>
+#include <unistd.h>
+
+#include <linux/perf_event.h>
+
+#ifndef TRACEFS
+#define TRACEFS "/sys/kernel/tracing/events"
+#endif
+#define RING_DATA_PAGES 16
+
+static void set_err(char *err, size_t n, const char *fmt, ...)
+{
+    va_list ap;
+
+    va_start(ap, fmt);
+    vsnprintf(err, n, fmt, ap);
+    va_end(ap);
+}
+
+bool tp_event_load(tp_event_t *ev, const char *system, const char *name, char *err, size_t errn)
+{
+    memset(ev, 0, sizeof(*ev));
+    snprintf(ev->system, sizeof(ev->system), "%s", system);
+    snprintf(ev->name, sizeof(ev->name), "%s", name);
+    ev->id = -1;
+
+    char path[256];
+    snprintf(path, sizeof(path), TRACEFS "/%s/%s/format", system, name);
+
+    FILE *f = fopen(path, "r");
+
+    if (!f) {
+        set_err(err, errn, "%s: %s", path, strerror(errno));
+        return false;
+    }
+
+    char line[512];
+
+    while (fgets(line, sizeof(line), f)) {
+
+        int id;
+
+        if (sscanf(line, "ID: %d", &id) == 1) {
+            ev->id = id;
+            continue;
+        }
+
+        char *fp = line;
+
+        while (*fp == ' ' || *fp == '\t')
+            fp++;
+
+        if (strncmp(fp, "field:", 6) || ev->nfields >= TP_MAX_FIELDS)
+            continue;
+
+        char *semi = strchr(fp, ';');
+
+        if (!semi)
+            continue;
+
+        /* the field name is the last identifier in the declaration */
+        char decl[256];
+        size_t dl = (size_t)(semi - (fp + 6));
+
+        if (dl >= sizeof(decl))
+            dl = sizeof(decl) - 1;
+
+        memcpy(decl, fp + 6, dl);
+        decl[dl] = 0;
+
+        char *br = strchr(decl, '[');
+
+        if (br)
+            *br = 0;
+
+        char *end = decl + strlen(decl);
+
+        while (end > decl && (end[-1] == ' ' || end[-1] == '\t'))
+            *--end = 0;
+
+        char *start = end;
+
+        while (start > decl && start[-1] != ' ' && start[-1] != '\t' && start[-1] != '*')
+            start--;
+
+        tp_field_t *fd = &ev->fields[ev->nfields];
+        const char *o = strstr(semi, "offset:");
+        const char *s = strstr(semi, "size:");
+        const char *g = strstr(semi, "signed:");
+
+        if (!o || !s)
+            continue;
+
+        snprintf(fd->name, sizeof(fd->name), "%s", start);
+        fd->offset    = atoi(o + 7);
+        fd->size      = atoi(s + 5);
+        fd->is_signed = g ? atoi(g + 7) != 0 : false;
+        ev->nfields++;
+    }
+
+    fclose(f);
+
+    if (ev->id < 0) {
+        set_err(err, errn, "%s: no ID line", path);
+        return false;
+    }
+
+    return true;
+}
+
+int tp_field(const tp_event_t *ev, const char *name)
+{
+    for (int i = 0; i < ev->nfields; i++)
+        if (!strcmp(ev->fields[i].name, name))
+            return i;
+
+    return -1;
+}
+
+int64_t tp_get(const tp_event_t *ev, int field, const uint8_t *raw, uint32_t rawlen)
+{
+    if (field < 0 || field >= ev->nfields)
+        return 0;
+
+    const tp_field_t *f = &ev->fields[field];
+
+    if (f->offset < 0 || (uint32_t)(f->offset + f->size) > rawlen)
+        return 0;
+
+    const uint8_t *p = raw + f->offset;
+
+    switch (f->size) {
+    case 1: { uint8_t  v; memcpy(&v, p, 1); return f->is_signed ? (int64_t)(int8_t)v  : (int64_t)v; }
+    case 2: { uint16_t v; memcpy(&v, p, 2); return f->is_signed ? (int64_t)(int16_t)v : (int64_t)v; }
+    case 4: { uint32_t v; memcpy(&v, p, 4); return f->is_signed ? (int64_t)(int32_t)v : (int64_t)v; }
+    case 8: { uint64_t v; memcpy(&v, p, 8); return (int64_t)v; }
+    default: return 0;
+    }
+}
+
+static int online_cpus(int *cpus, int max)
+{
+    FILE *f = fopen("/sys/devices/system/cpu/online", "r");
+    int n = 0;
+
+    if (!f)
+        return 0;
+
+    char buf[256] = {0};
+
+    if (!fgets(buf, sizeof(buf), f))
+        buf[0] = 0;
+
+    fclose(f);
+
+    for (char *tok = strtok(buf, ",\n"); tok && n < max; tok = strtok(NULL, ",\n")) {
+
+        int a, b;
+
+        if (sscanf(tok, "%d-%d", &a, &b) == 2) {
+            for (int c = a; c <= b && n < max; c++)
+                cpus[n++] = c;
+        } else if (sscanf(tok, "%d", &a) == 1) {
+            cpus[n++] = a;
+        }
+    }
+
+    return n;
+}
+
+bool tp_open(tp_t *tp, tp_event_t **events, int nevents, char *err, size_t errn)
+{
+    memset(tp, 0, sizeof(*tp));
+    tp->epfd = -1;
+
+    if (nevents <= 0 || nevents > TP_MAX_EVENTS) {
+        set_err(err, errn, "bad event count %d", nevents);
+        return false;
+    }
+
+    for (int i = 0; i < nevents; i++)
+        tp->events[i] = events[i];
+
+    tp->nevents = nevents;
+
+    int cpus[TP_MAX_CPUS];
+    tp->ncpu = online_cpus(cpus, TP_MAX_CPUS);
+
+    if (tp->ncpu <= 0) {
+        set_err(err, errn, "no online CPUs found");
+        return false;
+    }
+
+    long page = sysconf(_SC_PAGESIZE);
+    tp->map_len = (size_t)page * (1 + RING_DATA_PAGES);
+
+    tp->epfd = epoll_create1(EPOLL_CLOEXEC);
+
+    if (tp->epfd < 0) {
+        set_err(err, errn, "epoll_create1: %s", strerror(errno));
+        return false;
+    }
+
+    for (int c = 0; c < tp->ncpu; c++) {
+
+        tp->ring_fd[c] = -1;
+
+        for (int e = 0; e < nevents; e++) {
+
+            struct perf_event_attr a;
+            memset(&a, 0, sizeof(a));
+
+            a.size          = sizeof(a);
+            a.type          = PERF_TYPE_TRACEPOINT;
+            a.config        = (uint64_t)events[e]->id;
+            a.sample_period = 1;
+            a.sample_type   = PERF_SAMPLE_TID | PERF_SAMPLE_TIME | PERF_SAMPLE_CPU | PERF_SAMPLE_RAW;
+            a.wakeup_events = 1;
+            a.use_clockid   = 1;
+            a.clockid       = CLOCK_MONOTONIC;
+            a.disabled      = 1;
+
+            int fd = (int)syscall(SYS_perf_event_open, &a, -1, cpus[c], -1, PERF_FLAG_FD_CLOEXEC);
+
+            if (fd < 0) {
+                set_err(err, errn, "perf_event_open(%s:%s, cpu %d): %s",
+                        events[e]->system, events[e]->name, cpus[c], strerror(errno));
+                tp_close(tp);
+                return false;
+            }
+
+            tp->fds[tp->nfds++] = fd;
+
+            if (tp->ring_fd[c] < 0) {
+
+                void *m = mmap(NULL, tp->map_len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+
+                if (m == MAP_FAILED) {
+                    set_err(err, errn, "mmap perf ring (cpu %d): %s", cpus[c], strerror(errno));
+                    tp_close(tp);
+                    return false;
+                }
+
+                tp->ring[c]    = m;
+                tp->ring_fd[c] = fd;
+
+                struct epoll_event ee = { .events = EPOLLIN, .data.u32 = (uint32_t)c };
+                epoll_ctl(tp->epfd, EPOLL_CTL_ADD, fd, &ee);
+
+            } else if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT, tp->ring_fd[c]) < 0) {
+                set_err(err, errn, "PERF_EVENT_IOC_SET_OUTPUT: %s", strerror(errno));
+                tp_close(tp);
+                return false;
+            }
+        }
+    }
+
+    for (int i = 0; i < tp->nfds; i++)
+        ioctl(tp->fds[i], PERF_EVENT_IOC_ENABLE, 0);
+
+    return true;
+}
+
+static void ring_copy(uint8_t *dst, const uint8_t *base, uint64_t size, uint64_t pos, size_t len)
+{
+    uint64_t off   = pos % size;
+    size_t   first = (size_t)(size - off);
+
+    if (first >= len) {
+        memcpy(dst, base + off, len);
+    } else {
+        memcpy(dst, base + off, first);
+        memcpy(dst + first, base, len - first);
+    }
+}
+
+static int cmp_sample(const void *a, const void *b)
+{
+    const tp_sample_t *x = a, *y = b;
+
+    return (x->time > y->time) - (x->time < y->time);
+}
+
+static void dispatch(tp_t *tp, tp_cb cb, void *ctx)
+{
+    qsort(tp->pend, tp->npend, sizeof(tp->pend[0]), cmp_sample);
+
+    for (int i = 0; i < tp->npend; i++)
+        cb(ctx, &tp->pend[i]);
+
+    tp->npend = 0;
+}
+
+static int drain_ring(tp_t *tp, int c, tp_cb cb, void *ctx)
+{
+    struct perf_event_mmap_page *pg = tp->ring[c];
+    long     page = sysconf(_SC_PAGESIZE);
+    uint64_t off  = pg->data_offset ? pg->data_offset : (uint64_t)page;
+    uint64_t size = pg->data_size ? pg->data_size : (uint64_t)page * RING_DATA_PAGES;
+    const uint8_t *base = (const uint8_t *)pg + off;
+
+    uint64_t head = __atomic_load_n(&pg->data_head, __ATOMIC_ACQUIRE);
+    uint64_t tail = pg->data_tail;
+    int n = 0;
+
+    while (tail < head) {
+
+        struct perf_event_header hdr;
+        ring_copy((uint8_t *)&hdr, base, size, tail, sizeof(hdr));
+
+        if (hdr.size < sizeof(hdr))
+            break;
+
+        ring_copy(tp->scratch, base, size, tail, hdr.size);
+
+        const uint8_t *p   = tp->scratch + sizeof(hdr);
+        const uint8_t *end = tp->scratch + hdr.size;
+
+        if (hdr.type == PERF_RECORD_LOST && end - p >= 16) {
+
+            uint64_t lost;
+            memcpy(&lost, p + 8, 8);
+            tp->lost += lost;
+
+        } else if (hdr.type == PERF_RECORD_SAMPLE && end - p >= 28) {
+
+            tp_sample_t s;
+            uint32_t v32[2];
+
+            memcpy(v32, p, 8);      p += 8;
+            s.pid = v32[0];
+            s.tid = v32[1];
+            memcpy(&s.time, p, 8);  p += 8;
+            memcpy(v32, p, 8);      p += 8;
+            s.cpu = v32[0];
+            memcpy(&s.rawlen, p, 4); p += 4;
+            s.raw = p;
+
+            if (s.rawlen >= 2 && p + s.rawlen <= end) {
+
+                uint16_t type;
+                memcpy(&type, s.raw, 2);
+                s.ev = NULL;
+
+                for (int e = 0; e < tp->nevents; e++)
+                    if (tp->events[e]->id == type)
+                        s.ev = tp->events[e];
+
+                if (s.ev && s.rawlen <= TP_MAX_RAW) {
+
+                    if (tp->npend == TP_MAX_PENDING)
+                        dispatch(tp, cb, ctx);
+
+                    memcpy(tp->pend_raw[tp->npend], s.raw, s.rawlen);
+                    s.raw = tp->pend_raw[tp->npend];
+                    tp->pend[tp->npend++] = s;
+                    n++;
+                }
+            }
+        }
+
+        tail += hdr.size;
+    }
+
+    __atomic_store_n(&pg->data_tail, tail, __ATOMIC_RELEASE);
+
+    return n;
+}
+
+int tp_poll(tp_t *tp, int timeout_ms, tp_cb cb, void *ctx)
+{
+    struct epoll_event ev[TP_MAX_CPUS];
+
+    if (epoll_wait(tp->epfd, ev, TP_MAX_CPUS, timeout_ms) < 0 && errno != EINTR)
+        return -1;
+
+    /*
+     * Drain every ring, not just the ones that woke us: samples from several
+     * CPUs need to be handled together to keep per-camera order sane.
+     */
+    int n = 0;
+
+    for (int c = 0; c < tp->ncpu; c++)
+        if (tp->ring[c])
+            n += drain_ring(tp, c, cb, ctx);
+
+    dispatch(tp, cb, ctx);
+
+    return n;
+}
+
+void tp_close(tp_t *tp)
+{
+    for (int i = 0; i < tp->nfds; i++) {
+        ioctl(tp->fds[i], PERF_EVENT_IOC_DISABLE, 0);
+    }
+
+    for (int c = 0; c < tp->ncpu; c++)
+        if (tp->ring[c])
+            munmap(tp->ring[c], tp->map_len);
+
+    for (int i = 0; i < tp->nfds; i++)
+        close(tp->fds[i]);
+
+    if (tp->epfd >= 0)
+        close(tp->epfd);
+
+    tp->nfds  = 0;
+    tp->epfd  = -1;
+}

hands/camd/tp.h

@@ -0,0 +1,76 @@
+/*
+ * tp - read kernel tracepoints system-wide through perf_event_open.
+ *
+ * One perf ring per CPU; every event on that CPU writes into it. Field
+ * offsets come from the tracefs format files, so kernel layout changes don't
+ * silently break parsing. Needs root (or CAP_PERFMON plus tracefs access).
+ */
+
+#pragma once
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#define TP_MAX_FIELDS 40
+#define TP_MAX_EVENTS  8
+#define TP_MAX_CPUS   64
+#define TP_MAX_PENDING 2048
+#define TP_MAX_RAW    256
+
+typedef struct {
+    char name[48];
+    int  offset;
+    int  size;
+    bool is_signed;
+} tp_field_t;
+
+typedef struct {
+    char       system[32];
+    char       name[48];
+    int        id;
+    tp_field_t fields[TP_MAX_FIELDS];
+    int        nfields;
+} tp_event_t;
+
+typedef struct {
+    const tp_event_t *ev;
+    const uint8_t    *raw;
+    uint32_t          rawlen;
+    uint64_t          time;     /* CLOCK_MONOTONIC ns */
+    uint32_t          cpu;
+    uint32_t          pid;
+    uint32_t          tid;
+} tp_sample_t;
+
+typedef void (*tp_cb)(void *ctx, const tp_sample_t *s);
+
+typedef struct {
+    int         ncpu;
+    int         ring_fd[TP_MAX_CPUS];
+    void       *ring[TP_MAX_CPUS];
+    size_t      map_len;
+    int         fds[TP_MAX_CPUS * TP_MAX_EVENTS];
+    int         nfds;
+    int         epfd;
+    tp_event_t *events[TP_MAX_EVENTS];
+    int         nevents;
+    uint64_t    lost;
+    uint8_t     scratch[65536];
+    /* samples drained from all rings, sorted by time before dispatch */
+    tp_sample_t pend[TP_MAX_PENDING];
+    uint8_t     pend_raw[TP_MAX_PENDING][TP_MAX_RAW];
+    int         npend;
+} tp_t;
+
+bool    tp_event_load(tp_event_t *ev, const char *system, const char *name, char *err, size_t errn);
+int     tp_field(const tp_event_t *ev, const char *name);
+int64_t tp_get(const tp_event_t *ev, int field, const uint8_t *raw, uint32_t rawlen);
+
+bool tp_open(tp_t *tp, tp_event_t **events, int nevents, char *err, size_t errn);
+/*
+ * Wait up to timeout_ms, then hand every pending sample to cb in time order,
+ * across all CPUs. Returns samples read, -1 on error.
+ */
+int  tp_poll(tp_t *tp, int timeout_ms, tp_cb cb, void *ctx);
+void tp_close(tp_t *tp);

hands/camd/xrcams.c

@@ -0,0 +1,783 @@
+/*
+ * xrcams - find the headset cameras and the DMA-BUF queues XRService feeds them.
+ *
+ * Adapted from framecap.c in FrameEyeCameraFeed (vendor/FrameEyeCameraFeed),
+ * MIT License, Copyright (c) 2026 Curtis English. See LICENSE.FrameEyeCameraFeed.
+ *
+ * Everything is discovered rather than hardcoded:
+ *   - XRService is found by scanning /proc for its cmdline.
+ *   - The V4L2 nodes and sensor subdevs it holds open come from /proc/<pid>/fd.
+ *   - Each node's geometry comes from VIDIOC_G_FMT on our own handle.
+ *   - Each node is traced back to its sensor through MEDIA_IOC_G_TOPOLOGY.
+ *   - Buffers are split into queues by allocation order: XRService opens a
+ *     sensor subdev, then allocates that camera's buffers.
+ */
+
+#define _GNU_SOURCE
+
+#include "xrcams.h"
+
+#include <dirent.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdarg.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/stat.h>
+#include <sys/sysmacros.h>
+#include <unistd.h>
+
+#include <linux/media.h>
+
+#ifndef MEDIA_ENT_F_CAM_SENSOR
+#define MEDIA_ENT_F_CAM_SENSOR 0x00020001
+#endif
+
+#define MAX_FDENTS 4096
+#define MAX_TOPOS     8
+
+enum fdkind { FD_DMABUF, FD_SUBDEV_SENSOR, FD_VIDEO };
+
+typedef struct {
+    int           xfd;
+    enum fdkind   kind;
+    size_t        size;
+    unsigned long ino;
+    char          sensor[XR_SENSOR_LEN];
+    char          path[64];
+} fdent_t;
+
+typedef struct {
+    struct media_v2_entity    *ents;
+    struct media_v2_interface *intfs;
+    struct media_v2_pad       *pads;
+    struct media_v2_link      *links;
+    __u32 nents, nintfs, npads, nlinks;
+} topo_t;
+
+static fdent_t fdents[MAX_FDENTS];
+static int     nfdents;
+static topo_t  topos[MAX_TOPOS];
+static int     ntopos;
+
+static void set_err(char *err, size_t n, const char *fmt, ...)
+{
+    va_list ap;
+
+    va_start(ap, fmt);
+    vsnprintf(err, n, fmt, ap);
+    va_end(ap);
+}
+
+void xr_slugify(const char *in, char *out, size_t n)
+{
+    size_t i = 0;
+
+    for (; in[i] && i + 1 < n; i++)
+        out[i] = (in[i] == ' ' || in[i] == '/') ? '_' : in[i];
+
+    out[i] = 0;
+}
+
+/* --------------------------------------------------- media graph handling */
+
+static void topo_free_all(void)
+{
+    for (int i = 0; i < ntopos; i++) {
+        free(topos[i].ents);
+        free(topos[i].intfs);
+        free(topos[i].pads);
+        free(topos[i].links);
+    }
+
+    ntopos = 0;
+}
+
+static void topo_load_all(void)
+{
+    for (int mi = 0; mi < MAX_TOPOS; mi++) {
+
+        char mpath[32];
+        snprintf(mpath, sizeof(mpath), "/dev/media%d", mi);
+
+        int mfd = open(mpath, O_RDWR | O_CLOEXEC);
+
+        if (mfd < 0)
+            continue;
+
+        struct media_v2_topology t;
+        memset(&t, 0, sizeof(t));
+
+        if (ioctl(mfd, MEDIA_IOC_G_TOPOLOGY, &t) < 0) {
+            close(mfd);
+            continue;
+        }
+
+        topo_t *o = &topos[ntopos];
+        memset(o, 0, sizeof(*o));
+
+        o->nents  = t.num_entities;
+        o->nintfs = t.num_interfaces;
+        o->npads  = t.num_pads;
+        o->nlinks = t.num_links;
+
+        o->ents  = calloc(o->nents  ? o->nents  : 1, sizeof(*o->ents));
+        o->intfs = calloc(o->nintfs ? o->nintfs : 1, sizeof(*o->intfs));
+        o->pads  = calloc(o->npads  ? o->npads  : 1, sizeof(*o->pads));
+        o->links = calloc(o->nlinks ? o->nlinks : 1, sizeof(*o->links));
+
+        t.ptr_entities   = (__u64)(uintptr_t)o->ents;
+        t.ptr_interfaces = (__u64)(uintptr_t)o->intfs;
+        t.ptr_pads       = (__u64)(uintptr_t)o->pads;
+        t.ptr_links      = (__u64)(uintptr_t)o->links;
+
+        bool ok = o->ents && o->intfs && o->pads && o->links &&
+                  ioctl(mfd, MEDIA_IOC_G_TOPOLOGY, &t) == 0;
+        close(mfd);
+
+        if (!ok) {
+            free(o->ents); free(o->intfs); free(o->pads); free(o->links);
+            continue;
+        }
+
+        ntopos++;
+    }
+}
+
+static struct media_v2_entity *topo_entity(topo_t *t, __u32 id)
+{
+    for (__u32 i = 0; i < t->nents; i++)
+        if (t->ents[i].id == id)
+            return &t->ents[i];
+
+    return NULL;
+}
+
+static struct media_v2_pad *topo_pad(topo_t *t, __u32 id)
+{
+    for (__u32 i = 0; i < t->npads; i++)
+        if (t->pads[i].id == id)
+            return &t->pads[i];
+
+    return NULL;
+}
+
+static __u32 topo_entity_for_devnode(topo_t *t, dev_t rdev)
+{
+    __u32 intf_id = 0;
+
+    for (__u32 i = 0; i < t->nintfs; i++)
+        if (t->intfs[i].devnode.major == major(rdev) &&
+            t->intfs[i].devnode.minor == minor(rdev)) {
+            intf_id = t->intfs[i].id;
+            break;
+        }
+
+    if (!intf_id)
+        return 0;
+
+    for (__u32 i = 0; i < t->nlinks; i++)
+        if ((t->links[i].flags & MEDIA_LNK_FL_LINK_TYPE) == MEDIA_LNK_FL_INTERFACE_LINK &&
+            t->links[i].source_id == intf_id)
+            return t->links[i].sink_id;
+
+    return 0;
+}
+
+/*
+ * Walk upstream across enabled data links until a sensor is reached. A CSIPHY
+ * carries two sensors on separate (sink, source) pad pairs, so re-enter on the
+ * sink pad paired with the source pad we left through.
+ */
+static bool topo_walk_to_sensor(topo_t *t, __u32 ent_id, char *out, size_t outn)
+{
+    int exit_pad_index = -1;
+
+    for (int hop = 0; hop < 32 && ent_id; hop++) {
+
+        struct media_v2_entity *e = topo_entity(t, ent_id);
+
+        if (!e)
+            return false;
+
+        if (e->function == MEDIA_ENT_F_CAM_SENSOR) {
+            snprintf(out, outn, "%s", e->name);
+            return true;
+        }
+
+        __u32 first_sink = 0, paired = 0;
+        int   nsinks     = 0;
+
+        for (__u32 p = 0; p < t->npads; p++) {
+
+            if (t->pads[p].entity_id != ent_id || !(t->pads[p].flags & MEDIA_PAD_FL_SINK))
+                continue;
+
+            nsinks++;
+
+            if (!first_sink)
+                first_sink = t->pads[p].id;
+
+            if (exit_pad_index >= 1 && (int)t->pads[p].index == exit_pad_index - 1)
+                paired = t->pads[p].id;
+        }
+
+        __u32 sink_pad = (nsinks == 1) ? first_sink : (paired ? paired : first_sink);
+
+        if (!sink_pad)
+            return false;
+
+        __u32 src_pad = 0;
+
+        for (__u32 i = 0; i < t->nlinks; i++) {
+
+            if ((t->links[i].flags & MEDIA_LNK_FL_LINK_TYPE) != MEDIA_LNK_FL_DATA_LINK)
+                continue;
+
+            if (!(t->links[i].flags & MEDIA_LNK_FL_ENABLED))
+                continue;
+
+            if (t->links[i].sink_id == sink_pad) {
+                src_pad = t->links[i].source_id;
+                break;
+            }
+        }
+
+        struct media_v2_pad *sp = src_pad ? topo_pad(t, src_pad) : NULL;
+
+        if (!sp)
+            return false;
+
+        ent_id         = sp->entity_id;
+        exit_pad_index = (int)sp->index;
+    }
+
+    return false;
+}
+
+static bool sensor_for_video(dev_t rdev, char *out, size_t outn)
+{
+    for (int i = 0; i < ntopos; i++) {
+
+        __u32 ent = topo_entity_for_devnode(&topos[i], rdev);
+
+        if (ent && topo_walk_to_sensor(&topos[i], ent, out, outn))
+            return true;
+    }
+
+    return false;
+}
+
+static bool sensor_for_subdev(dev_t rdev, char *out, size_t outn)
+{
+    for (int i = 0; i < ntopos; i++) {
+
+        __u32 id = topo_entity_for_devnode(&topos[i], rdev);
+        struct media_v2_entity *e = id ? topo_entity(&topos[i], id) : NULL;
+
+        if (e && e->function == MEDIA_ENT_F_CAM_SENSOR) {
+            snprintf(out, outn, "%s", e->name);
+            return true;
+        }
+    }
+
+    return false;
+}
+
+static const char *role_for_sensor(const char *sensor)
+{
+    if (strstr(sensor, "og01a1b"))
+        return "tracking";      /* 1056x1024 side fisheye      */
+
+    if (strstr(sensor, "og0ve10"))
+        return "tracking";      /* 640x480 upper               */
+
+    if (strstr(sensor, "imx616"))
+        return "passthrough";   /* 2464x2464 Arcturus color    */
+
+    return "unknown";
+}
+
+/* ------------------------------------------------- XRService / proc scan */
+
+static pid_t find_process(const char *needle)
+{
+    DIR *d = opendir("/proc");
+
+    if (!d)
+        return 0;
+
+    struct dirent *e;
+    pid_t found = 0;
+
+    while ((e = readdir(d))) {
+
+        if (e->d_name[0] < '0' || e->d_name[0] > '9')
+            continue;
+
+        char path[288];
+        snprintf(path, sizeof(path), "/proc/%s/cmdline", e->d_name);
+
+        FILE *f = fopen(path, "rb");
+
+        if (!f)
+            continue;
+
+        char buf[512] = {0};
+        size_t got = fread(buf, 1, sizeof(buf) - 1, f);
+        fclose(f);
+
+        if (got == 0)
+            continue;
+
+        const char *base = strrchr(buf, '/');
+        base = base ? base + 1 : buf;
+
+        if (strstr(base, needle)) {
+            found = (pid_t)atoi(e->d_name);
+            break;
+        }
+    }
+
+    closedir(d);
+
+    return found;
+}
+
+static bool read_dmabuf_size(pid_t pid, int fd, size_t *size, unsigned long *ino)
+{
+    char path[64];
+    snprintf(path, sizeof(path), "/proc/%d/fdinfo/%d", pid, fd);
+
+    FILE *f = fopen(path, "r");
+
+    if (!f)
+        return false;
+
+    bool have = false;
+    char line[256];
+
+    *ino = 0;
+
+    while (fgets(line, sizeof(line), f)) {
+
+        unsigned long long v;
+
+        if (sscanf(line, "size: %llu", &v) == 1) {
+            *size = (size_t)v;
+            have  = true;
+        } else if (sscanf(line, "ino: %llu", &v) == 1) {
+            *ino = (unsigned long)v;
+        }
+    }
+
+    fclose(f);
+
+    return have;
+}
+
+static int cmp_int(const void *a, const void *b)
+{
+    return *(const int *)a - *(const int *)b;
+}
+
+static bool scan_xr_fds(pid_t pid, char *err, size_t errn)
+{
+    char dirpath[64];
+    snprintf(dirpath, sizeof(dirpath), "/proc/%d/fd", pid);
+
+    DIR *d = opendir(dirpath);
+
+    if (!d) {
+        set_err(err, errn, "opendir(%s): %s (are you root?)", dirpath, strerror(errno));
+        return false;
+    }
+
+    static int fds[8192];
+    int nfds = 0;
+    struct dirent *e;
+
+    while ((e = readdir(d)) && nfds < (int)(sizeof(fds) / sizeof(fds[0])))
+        if (e->d_name[0] >= '0' && e->d_name[0] <= '9')
+            fds[nfds++] = atoi(e->d_name);
+
+    closedir(d);
+
+    qsort(fds, nfds, sizeof(int), cmp_int);
+
+    nfdents = 0;
+
+    for (int i = 0; i < nfds && nfdents < MAX_FDENTS; i++) {
+
+        char link[64], target[256];
+        snprintf(link, sizeof(link), "/proc/%d/fd/%d", pid, fds[i]);
+
+        ssize_t n = readlink(link, target, sizeof(target) - 1);
+
+        if (n < 0)
+            continue;
+
+        target[n] = 0;
+
+        fdent_t ent;
+        memset(&ent, 0, sizeof(ent));
+        ent.xfd = fds[i];
+
+        if (strstr(target, "dmabuf")) {
+
+            if (!read_dmabuf_size(pid, fds[i], &ent.size, &ent.ino))
+                continue;
+
+            ent.kind = FD_DMABUF;
+
+        } else if (strncmp(target, "/dev/video", 10) == 0) {
+
+            ent.kind = FD_VIDEO;
+            snprintf(ent.path, sizeof(ent.path), "%.63s", target);
+
+        } else if (strncmp(target, "/dev/v4l-subdev", 15) == 0) {
+
+            struct stat st;
+
+            if (stat(target, &st) < 0 || !sensor_for_subdev(st.st_rdev, ent.sensor, sizeof(ent.sensor)))
+                continue;
+
+            ent.kind = FD_SUBDEV_SENSOR;
+
+        } else {
+            continue;
+        }
+
+        fdents[nfdents++] = ent;
+    }
+
+    return true;
+}
+
+/* ------------------------------------------------------ camera discovery */
+
+static void probe_cameras(xr_state_t *st)
+{
+    int seen[64];
+    int nseen = 0;
+
+    for (int i = 0; i < nfdents; i++) {
+
+        if (fdents[i].kind != FD_VIDEO)
+            continue;
+
+        const char *path = fdents[i].path;
+        int         node = atoi(path + 10);
+        bool        dup  = false;
+
+        for (int k = 0; k < nseen; k++)
+            if (seen[k] == node)
+                dup = true;
+
+        if (dup || st->ncameras >= XR_MAX_CAMERAS || nseen >= 64)
+            continue;
+
+        seen[nseen++] = node;
+
+        int fd = open(path, O_RDWR | O_CLOEXEC);
+
+        if (fd < 0)
+            continue;
+
+        struct v4l2_format fmt;
+        memset(&fmt, 0, sizeof(fmt));
+        fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+
+        xr_camera_t *c = &st->cameras[st->ncameras];
+        memset(c, 0, sizeof(*c));
+
+        if (ioctl(fd, VIDIOC_G_FMT, &fmt) == 0) {
+
+            c->width        = fmt.fmt.pix_mp.width;
+            c->height       = fmt.fmt.pix_mp.height;
+            c->pixfmt       = fmt.fmt.pix_mp.pixelformat;
+            c->nplanes      = fmt.fmt.pix_mp.num_planes;
+            c->bytesperline = fmt.fmt.pix_mp.plane_fmt[0].bytesperline;
+
+            for (unsigned p = 0; p < c->nplanes && p < VIDEO_MAX_PLANES; p++)
+                c->planesize[p] = fmt.fmt.pix_mp.plane_fmt[p].sizeimage;
+
+        } else {
+
+            memset(&fmt, 0, sizeof(fmt));
+            fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+
+            if (ioctl(fd, VIDIOC_G_FMT, &fmt) < 0) {
+                close(fd);
+                continue;
+            }
+
+            c->width        = fmt.fmt.pix.width;
+            c->height       = fmt.fmt.pix.height;
+            c->pixfmt       = fmt.fmt.pix.pixelformat;
+            c->nplanes      = 1;
+            c->bytesperline = fmt.fmt.pix.bytesperline;
+            c->planesize[0] = fmt.fmt.pix.sizeimage;
+        }
+
+        struct stat sb;
+
+        if (fstat(fd, &sb) == 0) {
+            c->minor = minor(sb.st_rdev);
+            sensor_for_video(sb.st_rdev, c->sensor, sizeof(c->sensor));
+        }
+
+        close(fd);
+
+        if (!c->sensor[0])
+            snprintf(c->sensor, sizeof(c->sensor), "unknown");
+
+        c->node = node;
+        snprintf(c->path, sizeof(c->path), "%s", path);
+        c->role = role_for_sensor(c->sensor);
+
+        st->ncameras++;
+    }
+}
+
+/*
+ * qcom-camss can report bytesperline as the visible width while the VFE
+ * writes a larger aligned pitch. sizeimage is right, so derive the pitch.
+ */
+unsigned xr_camera_stride(const xr_camera_t *c)
+{
+    if (!c->height || !c->planesize[0])
+        return c->bytesperline ? c->bytesperline : c->width;
+
+    double bpp = 1.0;
+
+    if (c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21)
+        bpp = 1.5;
+
+    unsigned s = (unsigned)((double)c->planesize[0] / ((double)c->height * bpp));
+
+    if (s >= c->width && s <= c->width * 4)
+        return s;
+
+    return c->bytesperline ? c->bytesperline : c->width;
+}
+
+/*
+ * The Arcturus color cameras (arcimx616) claim 2464x2464 NV12, but measured on
+ * 2026-09-28 their plane 0 holds 10-bit MIPI-packed YUV 4:2:0: 2464 luma rows
+ * then 1232 rows of interleaved UV, each row 2464 packed pixels (3080 bytes)
+ * padded to a 256-byte pitch (3328). Only the first 1972 pixels of a row carry
+ * image; the rest are zero.
+ */
+#define IMX616_VALID_WIDTH 1972
+
+void xr_camera_layout(const xr_camera_t *c, xr_layout_t *l)
+{
+    memset(l, 0, sizeof(*l));
+    l->height = c->height;
+
+    if (c->pixfmt == V4L2_PIX_FMT_NV12 && strstr(c->sensor, "imx616")) {
+
+        unsigned packed = (c->width * 5 + 3) / 4;
+
+        l->fmt   = XR_FMT_YUV420_10P;
+        l->pitch = (packed + 255) & ~255u;
+        l->rows  = c->height + c->height / 2;
+        l->width = IMX616_VALID_WIDTH < c->width ? IMX616_VALID_WIDTH : c->width;
+        return;
+    }
+
+    l->pitch = xr_camera_stride(c);
+    l->width = c->width < l->pitch ? c->width : l->pitch;
+
+    if (c->pixfmt == V4L2_PIX_FMT_NV12 || c->pixfmt == V4L2_PIX_FMT_NV21) {
+        l->fmt  = XR_FMT_NV12;
+        l->rows = c->height + c->height / 2;
+    } else {
+        l->fmt  = XR_FMT_GREY8;
+        l->rows = c->height;
+    }
+}
+
+const char *xr_fmt_name(xr_fmt_t f)
+{
+    switch (f) {
+    case XR_FMT_GREY8:      return "grey8";
+    case XR_FMT_NV12:       return "nv12";
+    case XR_FMT_YUV420_10P: return "yuv420_10p";
+    }
+
+    return "?";
+}
+
+/* ------------------------------------------------------- buffer grouping */
+
+/*
+ * XRService allocates one udmabuf per plane, plane 0 then plane 1, a whole
+ * queue at a time right after opening the sensor's subdev. Plane 1 matches
+ * VIDIOC_G_FMT exactly; plane 0 has slack, so it is matched with >=.
+ */
+static void build_groups(xr_state_t *st)
+{
+    char current_sensor[XR_SENSOR_LEN] = "";
+
+    for (int i = 0; i < nfdents; i++) {
+
+        if (fdents[i].kind == FD_SUBDEV_SENSOR) {
+            snprintf(current_sensor, sizeof(current_sensor), "%s", fdents[i].sensor);
+            continue;
+        }
+
+        if (fdents[i].kind != FD_DMABUF)
+            continue;
+
+        if (i + 1 >= nfdents || fdents[i + 1].kind != FD_DMABUF)
+            continue;
+
+        size_t s0 = fdents[i].size;
+        size_t s1 = fdents[i + 1].size;
+        bool match = false;
+
+        for (int c = 0; c < st->ncameras; c++) {
+
+            xr_camera_t *cam = &st->cameras[c];
+
+            if (cam->nplanes >= 2 && s1 == cam->planesize[1] && s0 >= cam->planesize[0]) {
+                match = true;
+                break;
+            }
+        }
+
+        if (!match)
+            continue;
+
+        xr_group_t *g = NULL;
+
+        if (st->ngroups > 0) {
+
+            xr_group_t *last = &st->groups[st->ngroups - 1];
+
+            if (last->planesize[0] == s0 && last->planesize[1] == s1 &&
+                !strcmp(last->sensor, current_sensor))
+                g = last;
+        }
+
+        if (!g) {
+
+            if (st->ngroups >= XR_MAX_GROUPS)
+                break;
+
+            g = &st->groups[st->ngroups++];
+            memset(g, 0, sizeof(*g));
+            g->planesize[0] = s0;
+            g->planesize[1] = s1;
+            snprintf(g->sensor, sizeof(g->sensor), "%s", current_sensor);
+        }
+
+        if (g->nbufs < XR_MAX_RUNBUFS) {
+            g->buf[g->nbufs].xfd   = fdents[i].xfd;
+            g->buf[g->nbufs].xfd1  = fdents[i + 1].xfd;
+            g->buf[g->nbufs].size  = s0;
+            g->buf[g->nbufs].size1 = s1;
+            g->nbufs++;
+        }
+
+        i++;    /* consume the plane 1 descriptor */
+    }
+
+    int keep = 0;
+
+    for (int i = 0; i < st->ngroups; i++)
+        if (st->groups[i].nbufs >= 4)
+            st->groups[keep++] = st->groups[i];
+
+    st->ngroups = keep;
+
+    /*
+     * Bind each run to a camera. The sensor marker alone can be wrong: XRService
+     * sometimes opens another sensor's subdev (e.g. the idle color camera)
+     * between an upper camera's subdev and its buffers, and two upper cameras
+     * can resolve to the same sensor name. So a marker match must also fit the
+     * camera's plane sizes, and each camera takes at most one run.
+     */
+    for (int pass = 0; pass < 2; pass++)
+        for (int i = 0; i < st->ngroups; i++) {
+
+            xr_group_t *g = &st->groups[i];
+
+            for (int c = 0; c < st->ncameras && !g->cam; c++) {
+
+                xr_camera_t *cam = &st->cameras[c];
+
+                if (pass == 0 && (!g->sensor[0] || strcmp(cam->sensor, g->sensor)))
+                    continue;
+
+                if (cam->nplanes < 2 || g->planesize[1] != cam->planesize[1] ||
+                    g->planesize[0] < cam->planesize[0])
+                    continue;
+
+                bool taken = false;
+
+                for (int k = 0; k < st->ngroups; k++)
+                    if (k != i && st->groups[k].cam == cam)
+                        taken = true;
+
+                if (!taken)
+                    g->cam = cam;
+            }
+        }
+}
+
+bool xr_discover(xr_state_t *st, const char *process, char *err, size_t errn)
+{
+    memset(st, 0, sizeof(*st));
+
+    st->pid = find_process(process);
+
+    if (!st->pid) {
+        set_err(err, errn, "%s is not running; start SteamVR on the headset first", process);
+        return false;
+    }
+
+    topo_load_all();
+
+    bool ok = scan_xr_fds(st->pid, err, errn);
+
+    if (ok) {
+        probe_cameras(st);
+        build_groups(st);
+    }
+
+    topo_free_all();
+
+    return ok;
+}
+
+void xr_print(const xr_state_t *st, FILE *f)
+{
+    fprintf(f, "XRService pid %d\n", st->pid);
+
+    for (int i = 0; i < st->ncameras; i++) {
+
+        const xr_camera_t *c = &st->cameras[i];
+        char fcc[5] = {
+            (char)(c->pixfmt & 0xff), (char)((c->pixfmt >> 8) & 0xff),
+            (char)((c->pixfmt >> 16) & 0xff), (char)((c->pixfmt >> 24) & 0xff), 0
+        };
+
+        fprintf(f, "  camera %-12s minor %-3u %-16s %ux%u %s pitch %u planes %zu %zu  role=%s\n",
+                c->path, c->minor, c->sensor, c->width, c->height, fcc,
+                xr_camera_stride(c), c->planesize[0], c->planesize[1], c->role);
+    }
+
+    for (int i = 0; i < st->ngroups; i++) {
+
+        const xr_group_t *g = &st->groups[i];
+
+        fprintf(f, "  queue %d: %d buffers plane0=%zu plane1=%zu fds %d..%d sensor '%s' -> %s\n",
+                i, g->nbufs, g->planesize[0], g->planesize[1],
+                g->buf[0].xfd, g->buf[g->nbufs - 1].xfd1, g->sensor,
+                g->cam ? g->cam->path : "(unbound)");
+    }
+}

hands/camd/xrcams.h

@@ -0,0 +1,82 @@
+/*
+ * xrcams - find the headset cameras and the DMA-BUF queues XRService feeds them.
+ *
+ * Adapted from framecap.c in FrameEyeCameraFeed (vendor/FrameEyeCameraFeed),
+ * MIT License, Copyright (c) 2026 Curtis English. See LICENSE.FrameEyeCameraFeed.
+ */
+
+#pragma once
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <sys/types.h>
+
+#include <linux/videodev2.h>
+
+#define XR_MAX_CAMERAS  16
+#define XR_MAX_GROUPS   32
+#define XR_MAX_RUNBUFS 128
+#define XR_SENSOR_LEN   64
+
+typedef struct {
+    int         node;                   /* N from /dev/videoN                */
+    unsigned    minor;                  /* char device minor, as tracepoints report it */
+    char        path[64];
+    unsigned    width;
+    unsigned    height;
+    unsigned    bytesperline;
+    unsigned    nplanes;
+    size_t      planesize[VIDEO_MAX_PLANES];
+    uint32_t    pixfmt;
+    char        sensor[XR_SENSOR_LEN];  /* media entity name of the sensor   */
+    const char *role;
+} xr_camera_t;
+
+typedef struct {
+    int    xfd;                         /* plane 0 descriptor in XRService   */
+    int    xfd1;                        /* plane 1 descriptor in XRService   */
+    size_t size;
+    size_t size1;
+} xr_bufref_t;
+
+/* One run of buffers XRService allocated for a camera queue, in allocation order. */
+typedef struct {
+    size_t       planesize[2];
+    int          nbufs;
+    xr_bufref_t  buf[XR_MAX_RUNBUFS];
+    char         sensor[XR_SENSOR_LEN]; /* from the preceding sensor subdev  */
+    xr_camera_t *cam;
+} xr_group_t;
+
+typedef struct {
+    pid_t       pid;
+    xr_camera_t cameras[XR_MAX_CAMERAS];
+    int         ncameras;
+    xr_group_t  groups[XR_MAX_GROUPS];
+    int         ngroups;
+} xr_state_t;
+
+typedef enum {
+    XR_FMT_GREY8,       /* 8-bit mono                                           */
+    XR_FMT_NV12,        /* 8-bit Y plane then interleaved UV, same pitch        */
+    XR_FMT_YUV420_10P   /* like NV12, but 10-bit MIPI-packed (4 px in 5 bytes)  */
+} xr_fmt_t;
+
+/* Where the image really sits in plane 0; V4L2's numbers can be misleading. */
+typedef struct {
+    xr_fmt_t fmt;
+    unsigned pitch;     /* bytes per row                                        */
+    unsigned rows;      /* rows in plane 0: luma, plus chroma for YUV           */
+    unsigned width;     /* valid pixels per row                                 */
+    unsigned height;    /* luma rows                                            */
+} xr_layout_t;
+
+/* Scan XRService's descriptors and the media graph. Needs root. */
+bool     xr_discover(xr_state_t *st, const char *process, char *err, size_t errn);
+unsigned xr_camera_stride(const xr_camera_t *c);
+void     xr_camera_layout(const xr_camera_t *c, xr_layout_t *l);
+const char *xr_fmt_name(xr_fmt_t f);
+void     xr_print(const xr_state_t *st, FILE *f);
+void     xr_slugify(const char *in, char *out, size_t n);

hands/frametop-camd.service

@@ -0,0 +1,22 @@
+# Template: the installer replaces @REPO@ with the repo path on the Frame.
+[Unit]
+Description=Frametop camera broker: the headset cameras' frames, for hand tracking
+Documentation=file://@REPO@/hands/README.md
+# It borrows XRService's camera buffers, so it comes and goes with SteamVR.
+After=steamvr.service
+PartOf=steamvr.service
+
+[Service]
+# On the host: the dev container can't reach XRService. Its file capabilities (set by
+# hands/run.sh install) let it borrow the buffers; it drops them once set up. It exits when
+# XRService restarts, and comes back to attach to the new one.
+# Mono cameras only: while the headset is worn the colour module writes only a half-size
+# image into the top-left quarter of its buffers (2026-09-30), which ft-camd can't use yet.
+# Add --with-color to try the colour cameras (ft-hands then picks them by the light).
+ExecStart=@REPO@/hands/build/ft-camd --status 60
+Restart=always
+RestartSec=5
+TimeoutStopSec=5
+
+[Install]
+WantedBy=steamvr.service

hands/frametop-hands.service

@@ -0,0 +1,20 @@
+# Template: the installer replaces @REPO@ with the repo path on the Frame.
+[Unit]
+Description=Frametop hand tracking: hands for the screens' hand cutouts, pinches for the pointer
+Documentation=file://@REPO@/hands/README.md
+After=steamvr.service frametop-camd.service
+Wants=frametop-camd.service
+PartOf=steamvr.service
+
+[Service]
+# In the dev container (it's built against Fedora's libraries). It reads ft-camd's ring and
+# writes /run/user/UID/frametop-hands/hands and gestures. Settings: HANDS_* in ~/.config/frametop.conf.
+ExecStartPre=-@REPO@/scripts/container-up.sh
+ExecStartPre=-/usr/bin/pkill -x ft-hands
+ExecStart=%h/.local/bin/distrobox enter dev -- @REPO@/hands/build/ft-hands --status 60
+Restart=always
+RestartSec=5
+TimeoutStopSec=5
+
+[Install]
+WantedBy=steamvr.service

hands/ft-cutouts

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# ft-cutouts: the hand cutouts without pinches and grips. Your hands show through Frametop's
+# screens, but ft-hands runs with --no-gestures, so nothing clicks or drags. ft-handsctl on is
+# the full version; each stops the other. Like it, this doesn't start with SteamVR, and it
+# stops when SteamVR does. On the Frame.
+#
+#   ft-cutouts on | off | status
+#
+# Runs this checkout's hands/build as transient user units, so it doesn't need hands/run.sh
+# install: only a build (hands/build.sh) and ft-camd's capabilities (hands/run.sh caps).
+set -euo pipefail
+here=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
+camd=frametop-cutouts-camd.service hands=frametop-cutouts-hands.service
+
+die() { echo "$*" >&2; exit 1; }
+
+run_unit() {  # unit, the ft-handsctl unit it replaces, description, then the command
+  local unit=$1 conflicts=$2 what=$3
+  shift 3
+  systemctl --user -q is-active "$unit" && return
+  systemd-run --user --quiet --collect --unit="$unit" --description="Frametop hand cutouts: $what" \
+    -p PartOf=steamvr.service -p After=steamvr.service -p Conflicts="$conflicts" \
+    -p Restart=always -p RestartSec=5 -p TimeoutStopSec=5 "$@"
+}
+
+case ${1:-status} in
+  on)
+    systemctl --user -q is-active steamvr.service || die "SteamVR isn't running"
+    [ -x "$here/build/ft-camd" ] && [ -x "$here/build/ft-hands" ] || die "not built: hands/build.sh"
+    grep -qa -- --no-gestures "$here/build/ft-hands" || die "ft-hands predates --no-gestures: hands/build.sh"
+    getcap "$here/build/ft-camd" | grep -q cap_sys_ptrace ||
+      die "ft-camd needs its capabilities: hands/run.sh caps (asks for sudo)"
+    if systemctl --user -q is-active frametop-camd.service frametop-hands.service; then
+      echo "stopping ft-handsctl's hand tracking (it has pinches and grips)"
+    fi
+    run_unit $camd frametop-camd.service "the camera broker" "$here/build/ft-camd" --status 60
+    if ! systemctl --user -q is-active $hands; then
+      "$here/../scripts/container-up.sh"  # so stopping the unit can't take the container down
+      pkill -x ft-hands || true
+    fi
+    run_unit $hands frametop-hands.service "hand tracking, no gestures" "$HOME/.local/bin/distrobox" enter dev -- \
+      "$here/build/ft-hands" --status 60 --no-gestures
+    "$here/ft-handsctl" cutouts on >/dev/null || true
+    sleep 4
+    "$0" status ;;
+  off)
+    systemctl --user stop $hands $camd 2>/dev/null || true
+    systemctl --user -q is-active frametop-hands.service || pkill -x ft-hands || true
+    echo "hand cutouts off" ;;
+  status)
+    for u in $camd $hands; do echo "$u: $(systemctl --user is-active $u || true)"; done
+    echo "ft-screens cutouts: $("$here/ft-handsctl" cutouts state 2>&1)"
+    inv=$(systemctl --user show -p InvocationID --value $hands)  # this run's lines only
+    [ -n "$inv" ] && journalctl --user _SYSTEMD_INVOCATION_ID="$inv" --no-pager -o cat |
+      grep -E '^ *[0-9.]+s |sets with a hand|cameras:' | tail -2 | cut -c1-160 || true ;;
+  *) sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
+esac

hands/ft-handsctl

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# ft-handsctl: turn hand tracking on and off by hand, on the Frame. With it on, your hands show
+# through Frametop's screens (the hand cutouts); pinches and grips only move the pointer with
+# POINTER_HANDS=1. It doesn't start with SteamVR (hands/run.sh install leaves it off), and it
+# stops when SteamVR does.
+#
+#   ft-handsctl on | off | status | log [lines]
+#   ft-handsctl cutouts on|off|state   ft-screens' hand cutouts, without stopping tracking
+#   ft-handsctl gestures               watch pinches and grips live (Ctrl+C to stop)
+#
+# Needs the services installed once: hands/run.sh install (it sets ft-camd's capabilities).
+set -euo pipefail
+here=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
+units="frametop-camd.service frametop-hands.service"
+
+ask_screens() {  # a command to ft-screens' control socket, and its reply
+  python3 - "$1" <<'EOF'
+import socket, sys
+s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+s.bind("")
+s.settimeout(2)
+try:
+    s.sendto(sys.argv[1].encode(), "\0ft_screens")
+    print(s.recv(512).decode())
+except OSError as e:
+    sys.exit("ft-screens didn't answer (is the Frametop desktop running?): %s" % e)
+EOF
+}
+
+case ${1:-status} in
+  on)
+    systemctl --user -q is-active steamvr.service || { echo "SteamVR isn't running" >&2; exit 1; }
+    systemctl --user start $units
+    sleep 4
+    "$0" status ;;
+  off)
+    systemctl --user stop $units
+    echo "hand tracking off" ;;
+  status)
+    for u in $units; do echo "$u: $(systemctl --user is-active $u || true)"; done
+    journalctl --user -u frametop-hands.service --no-pager -o cat -n 40 |
+      grep -E '^ *[0-9.]+s |sets with a hand|cameras:' | tail -2 | cut -c1-160 || true ;;
+  log) journalctl --user -u frametop-camd.service -u frametop-hands.service --no-pager -o short -n "${2:-30}" ;;
+  cutouts)
+    ask_screens "cutouts ${2:-state}" ;;
+  gestures) exec python3 "$here/tools/watch_gestures.py" --distance ;;
+  *) sed -n '2,11p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
+esac

hands/include/fh_gestures.h

@@ -0,0 +1,81 @@
+/*
+ * fh_gestures - hand gestures ft-hands publishes for input (the pointer helper): look at
+ * something and pinch to click it, or close the hand (a grip) to press and drag it (the
+ * Vision Pro model, with the eye tracker doing the looking).
+ * /run/user/UID/frametop-hands/gestures, next to the hands
+ * file, with the same sequence lock (read seq, copy, read seq again; use the copy only if
+ * both reads are the same even number) and the same frame: metres in the head frame at
+ * capture time, OpenVR's HMD frame (+x right, +y up, -z forward).
+ *
+ * One slot per side and gesture: pinch[0] and grip[0] are the left hand, [1] the right.
+ * A gesture follows the hand it began on until it ends.
+ *   Pinch: begins when the thumb and index tips close within begin_m and ends when they
+ * open past end_m (the gap between keeps it from flickering). point is the index and middle
+ * knuckles, which don't move as the fingers open and close (the tips' midpoint did).
+ *   Grip: a closed hand. It begins when all four fingers are curled in (each fingertip
+ * nearer the wrist than grip_begin times its knuckle is) and ends when they open past
+ * grip_end on average. distance is that average (about 2 open, under 1.2 closed), strength
+ * 0 open .. 1 closed, and point the palm's centre. A grip ends a pinch on the same hand
+ * (closing the hand can pass through a pinch on the way), as lost.
+ * Either ends, as lost (FH_PINCH_LOST), when its hand stays lost too long.
+ *
+ * Don't miss short gestures: a reader that polls slower than a quick tap still sees it,
+ * because begins and ends count every one. When begins changed, one began at begin_ns;
+ * when ends changed, one ended at end_ns. begins - ends is 1 while it's down.
+ *
+ * Drags: point is where the gesture is now, begin_point where it began. Turn each into the
+ * room with the HMD pose at its capture time (capture_ns, begin_ns) before subtracting,
+ * so turning your head doesn't drag.
+ *
+ * Version 1 had only the pinches (192 bytes); version 2 adds the grips after them.
+ */
+
+#pragma once
+
+#include <assert.h>
+#include <stdint.h>
+
+#define FH_GESTURES_MAGIC   "FHGEST01"
+#define FH_GESTURES_VERSION 2
+
+enum {
+    FH_PINCH_TRACKED = 1u << 0, /* the hand was tracked in this frame              */
+    FH_PINCH_DOWN    = 1u << 1, /* the gesture is held now                          */
+    FH_PINCH_LOST    = 1u << 2, /* the last one ended because the hand was lost     */
+                                /* (or, for a pinch, a grip took over)              */
+};
+
+typedef struct {
+    uint32_t flags;             /* FH_PINCH_*                                       */
+    uint32_t hand_id;           /* fh_hand_t.id of the hand, 0 if none              */
+    uint32_t begins;            /* begun so far                                     */
+    uint32_t ends;              /* ended so far                                     */
+    uint64_t begin_ns;          /* capture time (CLOCK_MONOTONIC) the current or    */
+                                /* last one began                                   */
+    uint64_t end_ns;            /* ... the last one ended                           */
+    float    distance;          /* pinch: thumb tip to index tip, m, at this user's */
+                                /* hand size. grip: the fingers' mean curl (above)  */
+    float    strength;          /* 0 open .. 1 closed                               */
+    float    point[3];          /* pinch: the index and middle knuckles; grip: the  */
+                                /* palm's centre                                    */
+    float    begin_point[3];    /* point when the current or last one began         */
+} fh_pinch_t;                   /* 64 bytes */
+
+typedef struct {
+    char              magic[8];
+    uint32_t          version;
+    uint32_t          size;
+    volatile uint64_t seq;
+    uint64_t          capture_ns;   /* CLOCK_MONOTONIC when the cameras took the frames */
+    uint64_t          publish_ns;   /* CLOCK_MONOTONIC when this was written            */
+    float             begin_m;      /* the pinch thresholds in use                      */
+    float             end_m;
+    float             grip_begin;   /* the grip thresholds in use (curl ratios)         */
+    float             grip_end;
+    uint8_t           reserved[8];
+    fh_pinch_t        pinch[2];     /* [0] left hand, [1] right hand                    */
+    fh_pinch_t        grip[2];      /* version 2                                        */
+} fh_gestures_t;
+
+static_assert(sizeof(fh_pinch_t) == 64, "fh_pinch_t layout");
+static_assert(sizeof(fh_gestures_t) == 64 + 4 * 64, "fh_gestures_t layout");

hands/include/fh_hands.h

@@ -0,0 +1,58 @@
+/*
+ * fh_hands - the tracked-hands file ft-hands publishes for ft-screens' hand cutouts
+ * (/run/user/UID/frametop-hands/hands, directory mode 0700), rewritten in place
+ * under a sequence lock: read seq, copy, read seq again; use the copy only if
+ * both reads are the same even number.
+ *
+ * Positions are metres in the head frame at capture time, which is OpenVR's HMD
+ * frame (+x right, +y up, -z forward). Turn them into the room with the HMD pose
+ * at capture_ns (CLOCK_MONOTONIC). Writer: ft-hands (hands/track/io.cpp).
+ */
+
+#pragma once
+
+#include <assert.h>
+#include <stdint.h>
+
+#define FH_HANDS_MAGIC        "FHHANDS1"
+#define FH_HANDS_VERSION      1
+#define FH_HANDS_MAX_HANDS    2
+#define FH_HANDS_MAX_CAPSULES 64
+
+enum {
+    FH_HAND_RIGHT  = 1u << 0,   /* else the left hand                          */
+    FH_HAND_STEREO = 1u << 1,   /* triangulated from two or more cameras       */
+};
+
+typedef struct {
+    uint32_t id;                /* stays the same while the hand is tracked     */
+    uint32_t flags;             /* FH_HAND_*                                    */
+    float    confidence;
+    float    reserved;
+    float    pts[21][3];        /* MediaPipe hand landmarks                     */
+    uint32_t ncapsules;         /* this hand's capsules, which follow the       */
+                                /* previous hands' in capsules[]                */
+} fh_hand_t;                    /* 272 bytes */
+
+typedef struct {
+    float a[3], b[3];           /* segment ends                                 */
+    float ra, rb;               /* radius at each end                           */
+} fh_capsule_t;                 /* 32 bytes: the hand's shape, to cut out       */
+
+typedef struct {
+    char              magic[8];
+    uint32_t          version;
+    uint32_t          size;
+    volatile uint64_t seq;
+    uint64_t          capture_ns;   /* CLOCK_MONOTONIC when the cameras took the frames */
+    uint64_t          publish_ns;   /* CLOCK_MONOTONIC when this was written            */
+    uint32_t          nhands;
+    uint32_t          ncapsules;
+    uint8_t           reserved[16];
+    fh_hand_t         hands[FH_HANDS_MAX_HANDS];
+    fh_capsule_t      capsules[FH_HANDS_MAX_CAPSULES];
+} fh_hands_t;
+
+static_assert(sizeof(fh_hand_t) == 272, "fh_hand_t layout");
+static_assert(sizeof(fh_capsule_t) == 32, "fh_capsule_t layout");
+static_assert(sizeof(fh_hands_t) == 64 + 2 * 272 + 64 * 32, "fh_hands_t layout");

hands/models/NOTICE

@@ -0,0 +1,12 @@
+The models in ncnn/ are converted from the OpenCV Zoo ONNX ports of Google's MediaPipe hand
+models, by tools/convert_models.py:
+
+- palm.ncnn.*: palm_detection_mediapipe_2023feb (https://huggingface.co/opencv/palm_detection_mediapipe)
+- hand.ncnn.*: handpose_estimation_mediapipe_2023feb (https://huggingface.co/opencv/handpose_estimation_mediapipe)
+
+MediaPipe is Copyright Google LLC. The models and the OpenCV Zoo ports are licensed under the
+Apache License, Version 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
+
+Changes made here: converted to ncnn with pnnx, with the palm detector's channel pads
+rewritten as ncnn Padding layers, and quantized to 8 bits (the *-int8.ncnn.* files) with
+ncnn's tools.

hands/models/ncnn/hand-int8.ncnn.param

@@ -0,0 +1,79 @@
+7767517
+77 90
+Input                    in0                      0 1 in0
+Convolution              convclip_0               1 1 in0 2 0=24 1=3 3=2 15=1 16=1 5=1 6=648 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_0             1 1 2 3 0=24 1=3 4=1 5=1 6=216 7=24 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_10                  1 1 3 4 0=16 1=1 5=1 6=384 8=2
+Split                    splitncnn_0              1 2 4 5 6
+Convolution              convclip_1               1 1 6 7 0=64 1=1 5=1 6=1024 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_1             1 1 7 8 0=64 1=3 3=2 15=1 16=1 5=1 6=576 7=64 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_12                  1 1 8 9 0=16 1=1 5=1 6=1024 8=2
+Pooling                  maxpool2d_1              1 1 5 10 1=2 2=2 5=1
+BinaryOp                 add_0                    2 1 9 10 11
+Split                    splitncnn_1              1 2 11 12 13
+Convolution              convclip_2               1 1 13 14 0=96 1=1 5=1 6=1536 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_2             1 1 14 15 0=96 1=3 4=1 5=1 6=864 7=96 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_14                  1 1 15 16 0=16 1=1 5=1 6=1536 8=2
+BinaryOp                 add_1                    2 1 16 12 17
+Convolution              convclip_3               1 1 17 18 0=96 1=1 5=1 6=1536 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_3             1 1 18 19 0=96 1=5 3=2 4=1 15=2 16=2 5=1 6=2400 7=96 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_16                  1 1 19 20 0=24 1=1 5=1 6=2304 8=2
+Split                    splitncnn_2              1 2 20 21 22
+Convolution              convclip_4               1 1 22 23 0=144 1=1 5=1 6=3456 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_4             1 1 23 24 0=144 1=5 4=2 5=1 6=3600 7=144 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_18                  1 1 24 25 0=24 1=1 5=1 6=3456 8=2
+BinaryOp                 add_2                    2 1 25 21 26
+Convolution              convclip_5               1 1 26 27 0=144 1=1 5=1 6=3456 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_5             1 1 27 28 0=144 1=3 3=2 15=1 16=1 5=1 6=1296 7=144 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_20                  1 1 28 29 0=48 1=1 5=1 6=6912 8=2
+Split                    splitncnn_3              1 2 29 30 31
+Convolution              convclip_6               1 1 31 32 0=288 1=1 5=1 6=13824 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_6             1 1 32 33 0=288 1=3 4=1 5=1 6=2592 7=288 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_22                  1 1 33 34 0=48 1=1 5=1 6=13824 8=2
+BinaryOp                 add_3                    2 1 34 30 35
+Split                    splitncnn_4              1 2 35 36 37
+Convolution              convclip_7               1 1 37 38 0=288 1=1 5=1 6=13824 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_7             1 1 38 39 0=288 1=3 4=1 5=1 6=2592 7=288 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_24                  1 1 39 40 0=48 1=1 5=1 6=13824 8=2
+BinaryOp                 add_4                    2 1 40 36 41
+Convolution              convclip_8               1 1 41 42 0=288 1=1 5=1 6=13824 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_8             1 1 42 43 0=288 1=5 4=2 5=1 6=7200 7=288 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_26                  1 1 43 44 0=64 1=1 5=1 6=18432 8=2
+Split                    splitncnn_5              1 2 44 45 46
+Convolution              convclip_9               1 1 46 47 0=384 1=1 5=1 6=24576 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_9             1 1 47 48 0=384 1=5 4=2 5=1 6=9600 7=384 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_28                  1 1 48 49 0=64 1=1 5=1 6=24576 8=2
+BinaryOp                 add_5                    2 1 49 45 50
+Split                    splitncnn_6              1 2 50 51 52
+Convolution              convclip_10              1 1 52 53 0=384 1=1 5=1 6=24576 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_10            1 1 53 54 0=384 1=5 4=2 5=1 6=9600 7=384 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_30                  1 1 54 55 0=64 1=1 5=1 6=24576 8=2
+BinaryOp                 add_6                    2 1 55 51 56
+Convolution              convclip_11              1 1 56 57 0=384 1=1 5=1 6=24576 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_11            1 1 57 58 0=384 1=5 3=2 4=1 15=2 16=2 5=1 6=9600 7=384 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_32                  1 1 58 59 0=112 1=1 5=1 6=43008 8=2
+Split                    splitncnn_7              1 2 59 60 61
+Convolution              convclip_12              1 1 61 62 0=672 1=1 5=1 6=75264 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_12            1 1 62 63 0=672 1=5 4=2 5=1 6=16800 7=672 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_34                  1 1 63 64 0=112 1=1 5=1 6=75264 8=2
+BinaryOp                 add_7                    2 1 64 60 65
+Split                    splitncnn_8              1 2 65 66 67
+Convolution              convclip_13              1 1 67 68 0=672 1=1 5=1 6=75264 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_13            1 1 68 69 0=672 1=5 4=2 5=1 6=16800 7=672 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_36                  1 1 69 70 0=112 1=1 5=1 6=75264 8=2
+BinaryOp                 add_8                    2 1 70 66 71
+Split                    splitncnn_9              1 2 71 72 73
+Convolution              convclip_14              1 1 73 74 0=672 1=1 5=1 6=75264 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_14            1 1 74 75 0=672 1=5 4=2 5=1 6=16800 7=672 8=101 9=3 -23310=2,0.000000e+00,6.000000e+00
+Convolution              conv_38                  1 1 75 76 0=112 1=1 5=1 6=75264 8=2
+BinaryOp                 add_9                    2 1 76 72 77
+Convolution              convclip_15              1 1 77 78 0=672 1=1 5=1 6=75264 8=102 9=3 -23310=2,0.000000e+00,6.000000e+00
+ConvolutionDepthWise     convdwclip_15            1 1 78 79 0=672 1=3 4=1 5=1 6=6048 7=672 8=1 9=3 -23310=2,0.000000e+00,6.000000e+00
+Pooling                  gap_0                    1 1 79 80 0=1 4=1
+Reshape                  reshape_45               1 1 80 81 0=1 1=1 2=-1
+Squeeze                  squeeze_78               1 1 81 82 -23303=2,1,2
+Split                    splitncnn_10             1 4 82 83 84 85 86
+InnerProduct             linear_42                1 1 84 out0 0=63 1=1 2=42336 8=2
+InnerProduct             linear_43                1 1 83 out3 0=63 1=1 2=42336 8=2
+InnerProduct             fcsigmoid_0              1 1 85 out2 0=1 1=1 2=672 8=2 9=4
+InnerProduct             fcsigmoid_1              1 1 86 out1 0=1 1=1 2=672 8=2 9=4

hands/models/ncnn/hand.ncnn.param

@@ -0,0 +1,79 @@
+7767517
+77 90
+Input                    in0                      0 1 in0
+Convolution convclip_0 1 1 in0 2 0=24 1=3 -23310=2,0.0,6.0 11=3 12=1 13=2 14=0 15=1 16=1 2=1 3=2 4=0 5=1 6=648 9=3
+ConvolutionDepthWise convdwclip_0 1 1 2 3 0=24 1=3 -23310=2,0.0,6.0 11=3 12=1 13=1 14=1 2=1 3=1 4=1 5=1 6=216 7=24 9=3
+Convolution conv_10 1 1 3 4 0=16 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=384
+Split splitncnn_0 1 2 4 5 6
+Convolution convclip_1 1 1 6 7 0=64 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1024 9=3
+ConvolutionDepthWise convdwclip_1 1 1 7 8 0=64 1=3 -23310=2,0.0,6.0 11=3 12=1 13=2 14=0 15=1 16=1 2=1 3=2 4=0 5=1 6=576 7=64 9=3
+Convolution conv_12 1 1 8 9 0=16 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1024
+Pooling maxpool2d_1 1 1 5 10 0=0 1=2 11=2 12=2 13=0 2=2 3=0 5=1
+BinaryOp add_0 2 1 9 10 11 0=0
+Split splitncnn_1 1 2 11 12 13
+Convolution convclip_2 1 1 13 14 0=96 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1536 9=3
+ConvolutionDepthWise convdwclip_2 1 1 14 15 0=96 1=3 -23310=2,0.0,6.0 11=3 12=1 13=1 14=1 2=1 3=1 4=1 5=1 6=864 7=96 9=3
+Convolution conv_14 1 1 15 16 0=16 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1536
+BinaryOp add_1 2 1 16 12 17 0=0
+Convolution convclip_3 1 1 17 18 0=96 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1536 9=3
+ConvolutionDepthWise convdwclip_3 1 1 18 19 0=96 1=5 -23310=2,0.0,6.0 11=5 12=1 13=2 14=1 15=2 16=2 2=1 3=2 4=1 5=1 6=2400 7=96 9=3
+Convolution conv_16 1 1 19 20 0=24 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=2304
+Split splitncnn_2 1 2 20 21 22
+Convolution convclip_4 1 1 22 23 0=144 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=3456 9=3
+ConvolutionDepthWise convdwclip_4 1 1 23 24 0=144 1=5 -23310=2,0.0,6.0 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=3600 7=144 9=3
+Convolution conv_18 1 1 24 25 0=24 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=3456
+BinaryOp add_2 2 1 25 21 26 0=0
+Convolution convclip_5 1 1 26 27 0=144 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=3456 9=3
+ConvolutionDepthWise convdwclip_5 1 1 27 28 0=144 1=3 -23310=2,0.0,6.0 11=3 12=1 13=2 14=0 15=1 16=1 2=1 3=2 4=0 5=1 6=1296 7=144 9=3
+Convolution conv_20 1 1 28 29 0=48 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=6912
+Split splitncnn_3 1 2 29 30 31
+Convolution convclip_6 1 1 31 32 0=288 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=13824 9=3
+ConvolutionDepthWise convdwclip_6 1 1 32 33 0=288 1=3 -23310=2,0.0,6.0 11=3 12=1 13=1 14=1 2=1 3=1 4=1 5=1 6=2592 7=288 9=3
+Convolution conv_22 1 1 33 34 0=48 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=13824
+BinaryOp add_3 2 1 34 30 35 0=0
+Split splitncnn_4 1 2 35 36 37
+Convolution convclip_7 1 1 37 38 0=288 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=13824 9=3
+ConvolutionDepthWise convdwclip_7 1 1 38 39 0=288 1=3 -23310=2,0.0,6.0 11=3 12=1 13=1 14=1 2=1 3=1 4=1 5=1 6=2592 7=288 9=3
+Convolution conv_24 1 1 39 40 0=48 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=13824
+BinaryOp add_4 2 1 40 36 41 0=0
+Convolution convclip_8 1 1 41 42 0=288 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=13824 9=3
+ConvolutionDepthWise convdwclip_8 1 1 42 43 0=288 1=5 -23310=2,0.0,6.0 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=7200 7=288 9=3
+Convolution conv_26 1 1 43 44 0=64 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=18432
+Split splitncnn_5 1 2 44 45 46
+Convolution convclip_9 1 1 46 47 0=384 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=24576 9=3
+ConvolutionDepthWise convdwclip_9 1 1 47 48 0=384 1=5 -23310=2,0.0,6.0 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=9600 7=384 9=3
+Convolution conv_28 1 1 48 49 0=64 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=24576
+BinaryOp add_5 2 1 49 45 50 0=0
+Split splitncnn_6 1 2 50 51 52
+Convolution convclip_10 1 1 52 53 0=384 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=24576 9=3
+ConvolutionDepthWise convdwclip_10 1 1 53 54 0=384 1=5 -23310=2,0.0,6.0 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=9600 7=384 9=3
+Convolution conv_30 1 1 54 55 0=64 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=24576
+BinaryOp add_6 2 1 55 51 56 0=0
+Convolution convclip_11 1 1 56 57 0=384 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=24576 9=3
+ConvolutionDepthWise convdwclip_11 1 1 57 58 0=384 1=5 -23310=2,0.0,6.0 11=5 12=1 13=2 14=1 15=2 16=2 2=1 3=2 4=1 5=1 6=9600 7=384 9=3
+Convolution conv_32 1 1 58 59 0=112 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=43008
+Split splitncnn_7 1 2 59 60 61
+Convolution convclip_12 1 1 61 62 0=672 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=75264 9=3
+ConvolutionDepthWise convdwclip_12 1 1 62 63 0=672 1=5 -23310=2,0.0,6.0 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=16800 7=672 9=3
+Convolution conv_34 1 1 63 64 0=112 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=75264
+BinaryOp add_7 2 1 64 60 65 0=0
+Split splitncnn_8 1 2 65 66 67
+Convolution convclip_13 1 1 67 68 0=672 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=75264 9=3
+ConvolutionDepthWise convdwclip_13 1 1 68 69 0=672 1=5 -23310=2,0.0,6.0 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=16800 7=672 9=3
+Convolution conv_36 1 1 69 70 0=112 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=75264
+BinaryOp add_8 2 1 70 66 71 0=0
+Split splitncnn_9 1 2 71 72 73
+Convolution convclip_14 1 1 73 74 0=672 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=75264 9=3
+ConvolutionDepthWise convdwclip_14 1 1 74 75 0=672 1=5 -23310=2,0.0,6.0 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=16800 7=672 9=3
+Convolution conv_38 1 1 75 76 0=112 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=75264
+BinaryOp add_9 2 1 76 72 77 0=0
+Convolution convclip_15 1 1 77 78 0=672 1=1 -23310=2,0.0,6.0 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=75264 9=3
+ConvolutionDepthWise convdwclip_15 1 1 78 79 0=672 1=3 -23310=2,0.0,6.0 11=3 12=1 13=1 14=1 2=1 3=1 4=1 5=1 6=6048 7=672 9=3
+Pooling gap_0 1 1 79 80 0=1 4=1
+Reshape reshape_45 1 1 80 81 0=1 1=1 2=-1
+Squeeze squeeze_78 1 1 81 82 -23303=2,1,2
+Split splitncnn_10 1 4 82 83 84 85 86
+InnerProduct linear_42 1 1 84 out0 0=63 1=1 2=42336
+InnerProduct linear_43 1 1 83 out3 0=63 1=1 2=42336
+InnerProduct fcsigmoid_0 1 1 85 out2 0=1 1=1 2=672 9=4
+InnerProduct fcsigmoid_1 1 1 86 out1 0=1 1=1 2=672 9=4

hands/models/ncnn/palm-int8.ncnn.param

@@ -0,0 +1,151 @@
+7767517
+149 177
+Input                    in0                      0 1 in0
+Convolution              padconv_0                1 1 in0 2 0=32 1=5 3=2 4=1 15=2 16=2 5=1 6=2400 8=2
+PReLU                    prelu_41                 1 1 2 3 0=32
+Split                    splitncnn_0              1 2 3 4 5
+ConvolutionDepthWise     convdw_76                1 1 5 6 0=32 1=5 4=2 5=1 6=800 7=32 8=101
+Convolution              conv_12                  1 1 6 7 0=32 1=1 5=1 6=1024 8=2
+BinaryOp                 add_0                    2 1 4 7 8
+PReLU                    prelu_42                 1 1 8 9 0=32
+Split                    splitncnn_1              1 2 9 10 11
+ConvolutionDepthWise     convdw_77                1 1 11 12 0=32 1=5 4=2 5=1 6=800 7=32 8=101
+Convolution              conv_13                  1 1 12 13 0=32 1=1 5=1 6=1024 8=2
+BinaryOp                 add_1                    2 1 10 13 14
+PReLU                    prelu_43                 1 1 14 15 0=32
+Split                    splitncnn_2              1 2 15 16 17
+ConvolutionDepthWise     convdw_78                1 1 17 18 0=32 1=5 4=2 5=1 6=800 7=32 8=101
+Convolution              conv_14                  1 1 18 19 0=32 1=1 5=1 6=1024 8=2
+BinaryOp                 add_2                    2 1 16 19 20
+PReLU                    prelu_44                 1 1 20 21 0=32
+Split                    splitncnn_3              1 2 21 22 23
+Pooling                  maxpool2d_2              1 1 22 24 1=2 2=2 5=1
+Padding                  Pad_16                   1 1 24 25 8=32
+ConvolutionDepthWise     padconvdw_0              1 1 23 26 0=32 1=5 3=2 4=1 15=2 16=2 5=1 6=800 7=32 8=101
+Convolution              conv_15                  1 1 26 27 0=64 1=1 5=1 6=2048 8=2
+BinaryOp                 add_3                    2 1 25 27 28
+PReLU                    prelu_45                 1 1 28 29 0=64
+Split                    splitncnn_4              1 2 29 30 31
+ConvolutionDepthWise     convdw_80                1 1 31 32 0=64 1=5 4=2 5=1 6=1600 7=64 8=101
+Convolution              conv_16                  1 1 32 33 0=64 1=1 5=1 6=4096 8=2
+BinaryOp                 add_4                    2 1 30 33 34
+PReLU                    prelu_46                 1 1 34 35 0=64
+Split                    splitncnn_5              1 2 35 36 37
+ConvolutionDepthWise     convdw_81                1 1 37 38 0=64 1=5 4=2 5=1 6=1600 7=64 8=101
+Convolution              conv_17                  1 1 38 39 0=64 1=1 5=1 6=4096 8=2
+BinaryOp                 add_5                    2 1 36 39 40
+PReLU                    prelu_47                 1 1 40 41 0=64
+Split                    splitncnn_6              1 2 41 42 43
+ConvolutionDepthWise     convdw_82                1 1 43 44 0=64 1=5 4=2 5=1 6=1600 7=64 8=101
+Convolution              conv_18                  1 1 44 45 0=64 1=1 5=1 6=4096 8=2
+BinaryOp                 add_6                    2 1 42 45 46
+PReLU                    prelu_48                 1 1 46 47 0=64
+Split                    splitncnn_7              1 2 47 48 49
+Pooling                  maxpool2d_3              1 1 48 50 1=2 2=2 5=1
+Padding                  Pad_34                   1 1 50 51 8=64
+ConvolutionDepthWise     padconvdw_1              1 1 49 52 0=64 1=5 3=2 4=1 15=2 16=2 5=1 6=1600 7=64 8=101
+Convolution              conv_19                  1 1 52 53 0=128 1=1 5=1 6=8192 8=2
+BinaryOp                 add_7                    2 1 51 53 54
+PReLU                    prelu_49                 1 1 54 55 0=128
+Split                    splitncnn_8              1 2 55 56 57
+ConvolutionDepthWise     convdw_84                1 1 57 58 0=128 1=5 4=2 5=1 6=3200 7=128 8=101
+Convolution              conv_20                  1 1 58 59 0=128 1=1 5=1 6=16384 8=2
+BinaryOp                 add_8                    2 1 56 59 60
+PReLU                    prelu_50                 1 1 60 61 0=128
+Split                    splitncnn_9              1 2 61 62 63
+ConvolutionDepthWise     convdw_85                1 1 63 64 0=128 1=5 4=2 5=1 6=3200 7=128 8=101
+Convolution              conv_21                  1 1 64 65 0=128 1=1 5=1 6=16384 8=2
+BinaryOp                 add_9                    2 1 62 65 66
+PReLU                    prelu_51                 1 1 66 67 0=128
+Split                    splitncnn_10             1 2 67 68 69
+ConvolutionDepthWise     convdw_86                1 1 69 70 0=128 1=5 4=2 5=1 6=3200 7=128 8=101
+Convolution              conv_22                  1 1 70 71 0=128 1=1 5=1 6=16384 8=2
+BinaryOp                 add_10                   2 1 68 71 72
+PReLU                    prelu_52                 1 1 72 73 0=128
+Split                    splitncnn_11             1 3 73 74 75 76
+Pooling                  maxpool2d_4              1 1 75 77 1=2 2=2 5=1
+Padding                  Pad_52                   1 1 77 78 8=128
+ConvolutionDepthWise     padconvdw_2              1 1 76 79 0=128 1=5 3=2 4=1 15=2 16=2 5=1 6=3200 7=128 8=101
+Convolution              conv_23                  1 1 79 80 0=256 1=1 5=1 6=32768 8=2
+BinaryOp                 add_11                   2 1 78 80 81
+PReLU                    prelu_53                 1 1 81 82 0=256
+Split                    splitncnn_12             1 2 82 83 84
+ConvolutionDepthWise     convdw_88                1 1 84 85 0=256 1=5 4=2 5=1 6=6400 7=256 8=101
+Convolution              conv_24                  1 1 85 86 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_12                   2 1 83 86 87
+PReLU                    prelu_54                 1 1 87 88 0=256
+Split                    splitncnn_13             1 2 88 89 90
+ConvolutionDepthWise     convdw_89                1 1 90 91 0=256 1=5 4=2 5=1 6=6400 7=256 8=101
+Convolution              conv_25                  1 1 91 92 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_13                   2 1 89 92 93
+PReLU                    prelu_55                 1 1 93 94 0=256
+Split                    splitncnn_14             1 2 94 95 96
+ConvolutionDepthWise     convdw_90                1 1 96 97 0=256 1=5 4=2 5=1 6=6400 7=256 8=101
+Convolution              conv_26                  1 1 97 98 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_14                   2 1 95 98 99
+PReLU                    prelu_56                 1 1 99 100 0=256
+Split                    splitncnn_15             1 3 100 101 102 103
+Pooling                  maxpool2d_5              1 1 102 104 1=2 2=2 5=1
+ConvolutionDepthWise     padconvdw_3              1 1 103 105 0=256 1=5 3=2 4=1 15=2 16=2 5=1 6=6400 7=256 8=101
+Convolution              conv_27                  1 1 105 106 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_15                   2 1 104 106 107
+PReLU                    prelu_57                 1 1 107 108 0=256
+Split                    splitncnn_16             1 2 108 109 110
+ConvolutionDepthWise     convdw_92                1 1 110 111 0=256 1=5 4=2 5=1 6=6400 7=256 8=101
+Convolution              conv_28                  1 1 111 112 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_16                   2 1 109 112 113
+PReLU                    prelu_58                 1 1 113 114 0=256
+Split                    splitncnn_17             1 2 114 115 116
+ConvolutionDepthWise     convdw_93                1 1 116 117 0=256 1=5 4=2 5=1 6=6400 7=256 8=101
+Convolution              conv_29                  1 1 117 118 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_17                   2 1 115 118 119
+PReLU                    prelu_59                 1 1 119 120 0=256
+Split                    splitncnn_18             1 2 120 121 122
+ConvolutionDepthWise     convdw_94                1 1 122 123 0=256 1=5 4=2 5=1 6=6400 7=256 8=101
+Convolution              conv_30                  1 1 123 124 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_18                   2 1 121 124 125
+PReLU                    prelu_60                 1 1 125 126 0=256
+Interp                   interpolate_0            1 1 126 127 0=2 3=12 4=12
+Convolution              conv_31                  1 1 127 128 0=256 1=1 5=1 6=65536 8=2
+PReLU                    prelu_61                 1 1 128 129 0=256
+BinaryOp                 add_19                   2 1 101 129 130
+Split                    splitncnn_19             1 2 130 131 132
+ConvolutionDepthWise     convdw_95                1 1 132 133 0=256 1=5 4=2 5=1 6=6400 7=256 8=101
+Convolution              conv_32                  1 1 133 134 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_20                   2 1 131 134 135
+PReLU                    prelu_62                 1 1 135 136 0=256
+Split                    splitncnn_20             1 2 136 137 138
+ConvolutionDepthWise     convdw_96                1 1 138 139 0=256 1=5 4=2 5=1 6=6400 7=256 8=101
+Convolution              conv_33                  1 1 139 140 0=256 1=1 5=1 6=65536 8=2
+BinaryOp                 add_21                   2 1 137 140 141
+PReLU                    prelu_63                 1 1 141 142 0=256
+Split                    splitncnn_21             1 3 142 143 144 145
+Convolution              conv_34                  1 1 145 146 0=108 1=1 5=1 6=27648 8=2
+Permute                  permute_68               1 1 146 147 0=3
+Reshape                  reshape_72               1 1 147 148 0=18 1=864
+Convolution              conv_35                  1 1 144 149 0=6 1=1 5=1 6=1536 8=2
+Permute                  permute_69               1 1 149 150 0=3
+Reshape                  reshape_73               1 1 150 151 0=1 1=864
+Interp                   interpolate_1            1 1 143 152 0=2 3=24 4=24
+Convolution              conv_36                  1 1 152 153 0=128 1=1 5=1 6=32768 8=2
+PReLU                    prelu_64                 1 1 153 154 0=128
+BinaryOp                 add_22                   2 1 74 154 155
+Split                    splitncnn_22             1 2 155 156 157
+ConvolutionDepthWise     convdw_97                1 1 157 158 0=128 1=5 4=2 5=1 6=3200 7=128 8=101
+Convolution              conv_37                  1 1 158 159 0=128 1=1 5=1 6=16384 8=2
+BinaryOp                 add_23                   2 1 156 159 160
+PReLU                    prelu_65                 1 1 160 161 0=128
+Split                    splitncnn_23             1 2 161 162 163
+ConvolutionDepthWise     convdw_98                1 1 163 164 0=128 1=5 4=2 5=1 6=3200 7=128 8=101
+Convolution              conv_38                  1 1 164 165 0=128 1=1 5=1 6=16384 8=2
+BinaryOp                 add_24                   2 1 162 165 166
+PReLU                    prelu_66                 1 1 166 167 0=128
+Split                    splitncnn_24             1 2 167 168 169
+Convolution              conv_39                  1 1 169 170 0=36 1=1 5=1 6=4608 8=2
+Permute                  permute_70               1 1 170 171 0=3
+Reshape                  reshape_74               1 1 171 172 0=18 1=1152
+Concat                   cat_0                    2 1 172 148 out0
+Convolution              conv_40                  1 1 168 174 0=2 1=1 5=1 6=256 8=2
+Permute                  permute_71               1 1 174 175 0=3
+Reshape                  reshape_75               1 1 175 176 0=1 1=1152
+Concat                   cat_1                    2 1 176 151 out1

hands/models/ncnn/palm.ncnn.param

@@ -0,0 +1,151 @@
+7767517
+149 177
+Input                    in0                      0 1 in0
+Convolution padconv_0 1 1 in0 2 0=32 1=5 11=5 12=1 13=2 14=1 15=2 16=2 2=1 3=2 4=1 5=1 6=2400
+PReLU prelu_41 1 1 2 3 0=32
+Split splitncnn_0 1 2 3 4 5
+ConvolutionDepthWise convdw_76 1 1 5 6 0=32 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=800 7=32
+Convolution conv_12 1 1 6 7 0=32 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1024
+BinaryOp add_0 2 1 4 7 8 0=0
+PReLU prelu_42 1 1 8 9 0=32
+Split splitncnn_1 1 2 9 10 11
+ConvolutionDepthWise convdw_77 1 1 11 12 0=32 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=800 7=32
+Convolution conv_13 1 1 12 13 0=32 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1024
+BinaryOp add_1 2 1 10 13 14 0=0
+PReLU prelu_43 1 1 14 15 0=32
+Split splitncnn_2 1 2 15 16 17
+ConvolutionDepthWise convdw_78 1 1 17 18 0=32 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=800 7=32
+Convolution conv_14 1 1 18 19 0=32 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1024
+BinaryOp add_2 2 1 16 19 20 0=0
+PReLU prelu_44 1 1 20 21 0=32
+Split splitncnn_3 1 2 21 22 23
+Pooling maxpool2d_2 1 1 22 24 0=0 1=2 11=2 12=2 13=0 2=2 3=0 5=1
+Padding Pad_16 1 1 24 25 0=0 1=0 2=0 3=0 4=0 5=0.000000e+00 7=0 8=32
+ConvolutionDepthWise padconvdw_0 1 1 23 26 0=32 1=5 11=5 12=1 13=2 14=1 15=2 16=2 2=1 3=2 4=1 5=1 6=800 7=32
+Convolution conv_15 1 1 26 27 0=64 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=2048
+BinaryOp add_3 2 1 25 27 28 0=0
+PReLU prelu_45 1 1 28 29 0=64
+Split splitncnn_4 1 2 29 30 31
+ConvolutionDepthWise convdw_80 1 1 31 32 0=64 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=1600 7=64
+Convolution conv_16 1 1 32 33 0=64 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=4096
+BinaryOp add_4 2 1 30 33 34 0=0
+PReLU prelu_46 1 1 34 35 0=64
+Split splitncnn_5 1 2 35 36 37
+ConvolutionDepthWise convdw_81 1 1 37 38 0=64 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=1600 7=64
+Convolution conv_17 1 1 38 39 0=64 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=4096
+BinaryOp add_5 2 1 36 39 40 0=0
+PReLU prelu_47 1 1 40 41 0=64
+Split splitncnn_6 1 2 41 42 43
+ConvolutionDepthWise convdw_82 1 1 43 44 0=64 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=1600 7=64
+Convolution conv_18 1 1 44 45 0=64 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=4096
+BinaryOp add_6 2 1 42 45 46 0=0
+PReLU prelu_48 1 1 46 47 0=64
+Split splitncnn_7 1 2 47 48 49
+Pooling maxpool2d_3 1 1 48 50 0=0 1=2 11=2 12=2 13=0 2=2 3=0 5=1
+Padding Pad_34 1 1 50 51 0=0 1=0 2=0 3=0 4=0 5=0.000000e+00 7=0 8=64
+ConvolutionDepthWise padconvdw_1 1 1 49 52 0=64 1=5 11=5 12=1 13=2 14=1 15=2 16=2 2=1 3=2 4=1 5=1 6=1600 7=64
+Convolution conv_19 1 1 52 53 0=128 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=8192
+BinaryOp add_7 2 1 51 53 54 0=0
+PReLU prelu_49 1 1 54 55 0=128
+Split splitncnn_8 1 2 55 56 57
+ConvolutionDepthWise convdw_84 1 1 57 58 0=128 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=3200 7=128
+Convolution conv_20 1 1 58 59 0=128 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=16384
+BinaryOp add_8 2 1 56 59 60 0=0
+PReLU prelu_50 1 1 60 61 0=128
+Split splitncnn_9 1 2 61 62 63
+ConvolutionDepthWise convdw_85 1 1 63 64 0=128 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=3200 7=128
+Convolution conv_21 1 1 64 65 0=128 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=16384
+BinaryOp add_9 2 1 62 65 66 0=0
+PReLU prelu_51 1 1 66 67 0=128
+Split splitncnn_10 1 2 67 68 69
+ConvolutionDepthWise convdw_86 1 1 69 70 0=128 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=3200 7=128
+Convolution conv_22 1 1 70 71 0=128 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=16384
+BinaryOp add_10 2 1 68 71 72 0=0
+PReLU prelu_52 1 1 72 73 0=128
+Split splitncnn_11 1 3 73 74 75 76
+Pooling maxpool2d_4 1 1 75 77 0=0 1=2 11=2 12=2 13=0 2=2 3=0 5=1
+Padding Pad_52 1 1 77 78 0=0 1=0 2=0 3=0 4=0 5=0.000000e+00 7=0 8=128
+ConvolutionDepthWise padconvdw_2 1 1 76 79 0=128 1=5 11=5 12=1 13=2 14=1 15=2 16=2 2=1 3=2 4=1 5=1 6=3200 7=128
+Convolution conv_23 1 1 79 80 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=32768
+BinaryOp add_11 2 1 78 80 81 0=0
+PReLU prelu_53 1 1 81 82 0=256
+Split splitncnn_12 1 2 82 83 84
+ConvolutionDepthWise convdw_88 1 1 84 85 0=256 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=6400 7=256
+Convolution conv_24 1 1 85 86 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_12 2 1 83 86 87 0=0
+PReLU prelu_54 1 1 87 88 0=256
+Split splitncnn_13 1 2 88 89 90
+ConvolutionDepthWise convdw_89 1 1 90 91 0=256 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=6400 7=256
+Convolution conv_25 1 1 91 92 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_13 2 1 89 92 93 0=0
+PReLU prelu_55 1 1 93 94 0=256
+Split splitncnn_14 1 2 94 95 96
+ConvolutionDepthWise convdw_90 1 1 96 97 0=256 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=6400 7=256
+Convolution conv_26 1 1 97 98 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_14 2 1 95 98 99 0=0
+PReLU prelu_56 1 1 99 100 0=256
+Split splitncnn_15 1 3 100 101 102 103
+Pooling maxpool2d_5 1 1 102 104 0=0 1=2 11=2 12=2 13=0 2=2 3=0 5=1
+ConvolutionDepthWise padconvdw_3 1 1 103 105 0=256 1=5 11=5 12=1 13=2 14=1 15=2 16=2 2=1 3=2 4=1 5=1 6=6400 7=256
+Convolution conv_27 1 1 105 106 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_15 2 1 104 106 107 0=0
+PReLU prelu_57 1 1 107 108 0=256
+Split splitncnn_16 1 2 108 109 110
+ConvolutionDepthWise convdw_92 1 1 110 111 0=256 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=6400 7=256
+Convolution conv_28 1 1 111 112 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_16 2 1 109 112 113 0=0
+PReLU prelu_58 1 1 113 114 0=256
+Split splitncnn_17 1 2 114 115 116
+ConvolutionDepthWise convdw_93 1 1 116 117 0=256 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=6400 7=256
+Convolution conv_29 1 1 117 118 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_17 2 1 115 118 119 0=0
+PReLU prelu_59 1 1 119 120 0=256
+Split splitncnn_18 1 2 120 121 122
+ConvolutionDepthWise convdw_94 1 1 122 123 0=256 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=6400 7=256
+Convolution conv_30 1 1 123 124 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_18 2 1 121 124 125 0=0
+PReLU prelu_60 1 1 125 126 0=256
+Interp interpolate_0 1 1 126 127 0=2 3=12 4=12 6=0
+Convolution conv_31 1 1 127 128 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+PReLU prelu_61 1 1 128 129 0=256
+BinaryOp add_19 2 1 101 129 130 0=0
+Split splitncnn_19 1 2 130 131 132
+ConvolutionDepthWise convdw_95 1 1 132 133 0=256 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=6400 7=256
+Convolution conv_32 1 1 133 134 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_20 2 1 131 134 135 0=0
+PReLU prelu_62 1 1 135 136 0=256
+Split splitncnn_20 1 2 136 137 138
+ConvolutionDepthWise convdw_96 1 1 138 139 0=256 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=6400 7=256
+Convolution conv_33 1 1 139 140 0=256 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=65536
+BinaryOp add_21 2 1 137 140 141 0=0
+PReLU prelu_63 1 1 141 142 0=256
+Split splitncnn_21 1 3 142 143 144 145
+Convolution conv_34 1 1 145 146 0=108 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=27648
+Permute permute_68 1 1 146 147 0=3
+Reshape reshape_72 1 1 147 148 0=18 1=864
+Convolution conv_35 1 1 144 149 0=6 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=1536
+Permute permute_69 1 1 149 150 0=3
+Reshape reshape_73 1 1 150 151 0=1 1=864
+Interp interpolate_1 1 1 143 152 0=2 3=24 4=24 6=0
+Convolution conv_36 1 1 152 153 0=128 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=32768
+PReLU prelu_64 1 1 153 154 0=128
+BinaryOp add_22 2 1 74 154 155 0=0
+Split splitncnn_22 1 2 155 156 157
+ConvolutionDepthWise convdw_97 1 1 157 158 0=128 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=3200 7=128
+Convolution conv_37 1 1 158 159 0=128 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=16384
+BinaryOp add_23 2 1 156 159 160 0=0
+PReLU prelu_65 1 1 160 161 0=128
+Split splitncnn_23 1 2 161 162 163
+ConvolutionDepthWise convdw_98 1 1 163 164 0=128 1=5 11=5 12=1 13=1 14=2 2=1 3=1 4=2 5=1 6=3200 7=128
+Convolution conv_38 1 1 164 165 0=128 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=16384
+BinaryOp add_24 2 1 162 165 166 0=0
+PReLU prelu_66 1 1 166 167 0=128
+Split splitncnn_24 1 2 167 168 169
+Convolution conv_39 1 1 169 170 0=36 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=4608
+Permute permute_70 1 1 170 171 0=3
+Reshape reshape_74 1 1 171 172 0=18 1=1152
+Concat cat_0 2 1 172 148 out0 0=0
+Convolution conv_40 1 1 168 174 0=2 1=1 11=1 12=1 13=1 14=0 2=1 3=1 4=0 5=1 6=256
+Permute permute_71 1 1 174 175 0=3
+Reshape reshape_75 1 1 175 176 0=1 1=1152
+Concat cat_1 2 1 176 151 out1 0=0

hands/run.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# Install, start, stop, or inspect hand tracking on the Frame: ft-camd (the camera broker) and
+# ft-hands (the tracker), user services that stop with SteamVR. They don't start with it:
+# ft-handsctl on|off (on the Frame) or hands/run.sh start|stop.
+# Usage: hands/run.sh install|uninstall
+#        hands/run.sh caps      # give ft-camd its capabilities again (a rebuild clears them)
+#        hands/run.sh start|stop|restart|status|log [lines]
+# install and caps need the password (sudo setcap, once per build of ft-camd): it's asked in
+# the terminal, on the Frame or from a PC (frame_sudo in scripts/_env.sh, which also takes it
+# from the repo's .env).
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+. "$root/scripts/_env.sh"
+frame="$root/scripts/frame.sh"
+units="frametop-camd.service frametop-hands.service"
+# pidfd_getfd on XRService (ptrace_scope=1), system-wide tracepoints, and their root-only
+# format files. ft-camd drops them all once it has set up.
+caps=cap_sys_ptrace,cap_perfmon,cap_dac_read_search+ep
+
+sudo_run() { frame_sudo "$1"; }
+
+set_caps() {  # only when missing: a rebuild clears them, a reinstall doesn't
+  local bin
+  bin=$(printf %q "$FRAME_REPO/hands/build/ft-camd")
+  if on_frame "getcap $bin | grep -q cap_sys_ptrace"; then
+    echo "ft-camd has its capabilities"
+    return
+  fi
+  sudo_run "setcap $caps $bin && getcap $bin"
+}
+
+states="for u in $units; do echo \"\$u: \$(systemctl --user is-active \$u)\"; done"
+
+case ${1:-status} in
+  install)
+    "$root/hands/build.sh"
+    set_caps
+    for u in $units; do
+      fill_template "$root/hands/$u" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$u"
+    done
+    # Installed but not started with SteamVR: ft-handsctl on|off (linked into ~/.local/bin).
+    "$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user disable $units 2>/dev/null || true
+mkdir -p ~/.local/bin && ln -sfn $(printf %q "$FRAME_REPO/hands/ft-handsctl") ~/.local/bin/ft-handsctl
+$states; echo 'start it with: ft-handsctl on'" ;;
+  caps) set_caps ;;
+  uninstall) "$frame" --host "systemctl --user disable --now $units 2>/dev/null
+for u in $units; do rm -f ~/.config/systemd/user/\$u; done; systemctl --user daemon-reload
+[ -L ~/.local/bin/ft-handsctl ] && rm -f ~/.local/bin/ft-handsctl; echo removed" ;;
+  start|stop|restart) "$frame" --host "systemctl --user $1 $units; $states" ;;
+  status) "$frame" --host "$states; journalctl --user -u frametop-hands.service --no-pager -o cat -n 4" || true ;;
+  log) "$frame" --host "journalctl --user -u frametop-camd.service -u frametop-hands.service --no-pager -o short -n ${2:-30}" ;;
+  *) echo "usage: $0 install|uninstall|caps|start|stop|restart|status|log [lines]" >&2; exit 2 ;;
+esac

hands/tools/calib.py

@@ -0,0 +1,197 @@
+"""Tracking-camera calibration from the headset's factory files.
+
+/persist/xrservice.json (written by Valve's calibration, loaded by XRService)
+holds, per camera, Kannala-Brandt fisheye intrinsics ("kb": fx fy cx cy k1-k4,
+pixel centres at integer coordinates, as in OpenCV's fisheye model) and a pose
+in the slam_right (Cam0) frame: plus_x/plus_z are the camera axes and position
+its origin, in mm. /persist/device_config.json gives Cam0's pose in the CAD
+frame (cv.cad_from_cal, metres) and the head's pose in CAD (head). The CAD frame
+is +X head-left, +Y up, +Z forward; the head frame is OpenVR's: +x right, +y up,
+-z forward. Camera frames: +z along the optical axis, +x right and +y down in
+the image.
+
+Everything here returns metres in the head frame.
+"""
+import json
+import os
+
+import numpy as np
+
+XRSERVICE_JSON = '/persist/xrservice.json'
+DEVICE_JSON = '/persist/device_config.json'
+# The Arcturus color module's EEPROM: some binary, then its calibration as JSON (world-readable)
+ARCTURUS_EEPROM = '/sys/devices/platform/soc@0/ac15000.cci/i2c-0/0-0050/eeprom'
+ARCTURUS_WIDTH = 1972   # valid pixels per row that XRService's buffers deliver (of 2464)
+
+
+def _pose(d, scale=1.0):
+    """4x4 transform from a {plus_x, plus_z, position} pose (child axes in the parent frame)."""
+    x = np.asarray(d['plus_x'], float)
+    z = np.asarray(d['plus_z'], float)
+    y = np.cross(z, x)
+    T = np.eye(4)
+    T[:3, 0], T[:3, 1], T[:3, 2] = x, y, z
+    T[:3, 3] = np.asarray(d['position'], float) * scale
+    return T
+
+
+class Camera:
+    def __init__(self, name, width, height, kb, head_from_cam):
+        self.name = name
+        self.width, self.height = width, height
+        self.fx, self.fy, self.cx, self.cy = kb['fx'], kb['fy'], kb['cx'], kb['cy']
+        self.k = np.array([kb['k1'], kb['k2'], kb['k3'], kb['k4']])
+        self.head_from_cam = head_from_cam
+        self.R = head_from_cam[:3, :3]               # camera axes in the head frame
+        self.origin = head_from_cam[:3, 3]           # camera centre in the head frame
+
+    def __repr__(self):
+        return 'Camera(%s %dx%d at %s mm)' % (self.name, self.width, self.height,
+                                              np.round(self.origin * 1000, 1))
+
+    def _theta_d(self, theta):
+        t2 = theta * theta
+        k1, k2, k3, k4 = self.k
+        return theta * (1 + t2 * (k1 + t2 * (k2 + t2 * (k3 + t2 * k4))))
+
+    def project_cam(self, p):
+        """Camera-frame points (N,3) -> pixels (N,2). Points behind the lens still map (the lens sees ~180 deg)."""
+        p = np.atleast_2d(p)
+        r = np.hypot(p[:, 0], p[:, 1])
+        theta = np.arctan2(r, p[:, 2])
+        scale = np.where(r > 1e-12, self._theta_d(theta) / np.maximum(r, 1e-12), 0.0)
+        return np.stack([self.fx * p[:, 0] * scale + self.cx, self.fy * p[:, 1] * scale + self.cy], axis=1)
+
+    def unproject(self, uv):
+        """Pixels (N,2) -> unit rays (N,3) in the camera frame."""
+        uv = np.atleast_2d(np.asarray(uv, float))
+        mx = (uv[:, 0] - self.cx) / self.fx
+        my = (uv[:, 1] - self.cy) / self.fy
+        td = np.hypot(mx, my)
+        theta = td.copy()
+        k1, k2, k3, k4 = self.k
+        for _ in range(8):                          # Newton on theta_d(theta) = td
+            t2 = theta * theta
+            f = self._theta_d(theta) - td
+            df = 1 + t2 * (3 * k1 + t2 * (5 * k2 + t2 * (7 * k3 + t2 * 9 * k4)))
+            theta = np.clip(theta - f / df, 0.0, np.pi)
+        s = np.where(td > 1e-12, np.sin(theta) / np.maximum(td, 1e-12), 1.0)
+        return np.stack([mx * s, my * s, np.cos(theta)], axis=1)
+
+    def rays(self, uv):
+        """Pixels -> unit rays in the head frame (all starting at self.origin)."""
+        return self.unproject(uv) @ self.R.T
+
+    def project(self, p_head):
+        """Head-frame points (N,3) -> pixels (N,2) and depth along the optical axis (N,)."""
+        p = (np.atleast_2d(p_head) - self.origin) @ self.R
+        return self.project_cam(p), p[:, 2]
+
+    def angle_from_axis(self, uv):
+        """Angle in degrees between each pixel's ray and the optical axis."""
+        return np.degrees(np.arccos(np.clip(self.unproject(uv)[:, 2], -1, 1)))
+
+
+def device_path(path):
+    """A headset file such as /persist/xrservice.json. In the dev container the host's / is
+    at /run/host (distrobox doesn't mount /persist); off the Frame, FRAME_JOB_DEVICE_ROOT can
+    point at a folder with copies of them."""
+    root = os.environ.get('FRAME_JOB_DEVICE_ROOT')
+    if root:
+        return root + path
+    if not os.access(path, os.R_OK) and os.access('/run/host' + path, os.R_OK):
+        return '/run/host' + path
+    return path
+
+
+def load(xrservice=XRSERVICE_JSON, device=DEVICE_JSON):
+    """{calibration name: Camera} for the tracking cameras, posed in the head frame."""
+    with open(device_path(xrservice)) as f:
+        rig = json.load(f)
+    with open(device_path(device)) as f:
+        dev = json.load(f)
+    cad_from_cam0 = _pose(dev['cv']['cad_from_cal'])
+    head_from_cad = np.linalg.inv(_pose(dev['head']))
+    cams = {}
+    for c in rig['cameras']:
+        kb = next(i for i in c['intrinsics'] if i['cameraModel'] == 'kb')
+        cam0_from_cam = _pose(c['extrinsics'], 1e-3)
+        cams[c['sourceCamera']] = Camera(c['sourceCamera'], c['width'], c['height'], kb,
+                                         head_from_cad @ cad_from_cam0 @ cam0_from_cam)
+    return cams
+
+
+def load_color(eeprom=ARCTURUS_EEPROM, device=DEVICE_JSON, scale=2, crop='subtract'):
+    """{"passthrough_left"/"passthrough_right": Camera} for the Arcturus color cameras, posed in
+    the head frame, for ft-camd --with-color's images (luma at 1/scale size).
+
+    Their calibration is in the CAD frame (mm) with pixel coordinates on the full 2464x2464
+    sensor; each camera also has a cropRegion. crop says how that maps to the delivered
+    image: 'subtract' (image x = sensor x - cropRegion.x) or 'none'. tools/check_color.py
+    tells which fits.
+    """
+    with open(device_path(eeprom), 'rb') as f:
+        raw = f.read()
+    i = raw.rfind(b'{', 0, raw.find(b'"alignment_method"'))
+    rig, _ = json.JSONDecoder().raw_decode(raw[i:].decode('latin1'))
+    with open(device_path(device)) as f:
+        dev = json.load(f)
+    head_from_cad = np.linalg.inv(_pose(dev['head']))
+    cams = {}
+    for c in rig['cameras']:
+        kb = dict(next(k for k in c['intrinsics'] if k['cameraModel'] == 'kb'))
+        region = c.get('cropRegion', {}) if crop == 'subtract' else {}
+        # integer pixel centres: sensor u -> image (u - crop + 0.5) / scale - 0.5
+        kb['cx'] = (kb['cx'] - region.get('x', 0) + 0.5) / scale - 0.5
+        kb['cy'] = (kb['cy'] - region.get('y', 0) + 0.5) / scale - 0.5
+        kb['fx'] /= scale
+        kb['fy'] /= scale
+        cams[c['sourceCamera']] = Camera(c['sourceCamera'], ARCTURUS_WIDTH // scale, c['height'] // scale, kb,
+                                         head_from_cad @ _pose(c['extrinsics'], 1e-3))
+    return cams
+
+
+def triangulate(origins, dirs, weights=None):
+    """Least-squares point closest to several rays. Returns (point, rms distance to the rays)."""
+    A = np.zeros((3, 3))
+    b = np.zeros(3)
+    w = np.ones(len(origins)) if weights is None else np.asarray(weights, float)
+    for o, d, wi in zip(origins, dirs, w):
+        P = np.eye(3) - np.outer(d, d)
+        A += wi * P
+        b += wi * P @ o
+    p = np.linalg.solve(A, b)
+    res = [np.linalg.norm((np.eye(3) - np.outer(d, d)) @ (p - o)) for o, d in zip(origins, dirs)]
+    return p, float(np.sqrt(np.mean(np.square(res))))
+
+
+def triangulate_many(origins, dirs, weights):
+    """Triangulate K points seen from V cameras at once.
+
+    origins (V,3), dirs (V,K,3) unit rays, weights (V,). Returns points (K,3) and
+    each point's rms distance to its rays (K,).
+    """
+    P = np.eye(3) - dirs[..., :, None] * dirs[..., None, :]           # (V,K,3,3)
+    w = weights[:, None, None, None]
+    A = (w * P).sum(0)
+    b = (w * (P @ origins[:, None, :, None])).sum(0)[..., 0]
+    pts = np.linalg.solve(A, b[..., None])[..., 0]
+    off = pts[None] - origins[:, None, :]                              # (V,K,3)
+    perp = off - (off * dirs).sum(-1, keepdims=True) * dirs
+    return pts, np.sqrt((perp ** 2).sum(-1).mean(0))
+
+
+if __name__ == '__main__':
+    cams = load()
+    for cam in cams.values():
+        fwd = [float(v) for v in cam.R[:, 2]]
+        print('%-12s at x %+6.1f y %+6.1f z %+6.1f mm, looks %s' % (
+            cam.name, *(cam.origin * 1000),
+            'right' * (fwd[0] > 0.3) + 'left' * (fwd[0] < -0.3) + ' up' * (fwd[1] > 0.3) +
+            ' down' * (fwd[1] < -0.3) + ' forward' * (fwd[2] < -0.3) + ' back' * (fwd[2] > 0.3)),
+            np.round(fwd, 2))
+        uv = np.array([[cam.cx + 200, cam.cy - 100], [cam.cx - 0.4 * cam.width, cam.cy + 0.3 * cam.height]])
+        err = np.abs(cam.project_cam(cam.unproject(uv)) - uv).max()
+        assert err < 1e-6, err
+    a, b = cams['slam_left'], cams['slam_right']
+    print('slam baseline %.2f mm' % (1000 * np.linalg.norm(a.origin - b.origin)))

hands/tools/check_color.py

@@ -0,0 +1,64 @@
+"""Which color camera is which, and how their calibration maps onto ft-camd's images.
+
+usage: python tools/check_color.py REC_DIR [--sets N]
+
+A recording made with ft-camd --with-color holds color_video<N> frames with each set.
+This matches features between the two color images and scores every reading of the
+calibration: which video node is passthrough_left, and whether the calibration's
+cropRegion is subtracted from x ('subtract') or not ('none'). Only the right reading
+makes true matches' rays meet in front of both cameras. Then it checks the winner against
+the side tracking cameras, which tests the CAD-to-head chain shared with them.
+"""
+import argparse
+import itertools
+import os
+import sys
+
+import numpy as np
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+from tools.check_sides import load_cams, matches, score  # noqa: E402
+from tools.show_set import index, read_set  # noqa: E402
+from tools import calib  # noqa: E402
+
+
+def load_color(crop):
+    return calib.load_color(crop=crop)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('rec')
+    ap.add_argument('--sets', type=int, default=8)
+    a = ap.parse_args()
+    path = os.path.join(a.rec, 'sets.bin')
+    offs = index(path)
+    sets = [read_set(path, offs[n]) for n in np.linspace(0, len(offs) - 1, a.sets).astype(int)]
+    nodes = sorted(k for k in sets[0] if k.startswith('color_video'))
+    if len(nodes) != 2:
+        sys.exit('need two color_video<N> cameras in the recording (ft-camd --with-color); found %s' % nodes)
+    pairs = [matches(s[nodes[0]][0], s[nodes[1]][0]) for s in sets]
+    print('%d sets, %d matches between %s and %s' % (len(sets), sum(len(p[0]) for p in pairs), *nodes))
+
+    best = None
+    for crop, left in itertools.product(['subtract', 'none'], nodes):
+        cams = load_color(crop)
+        right = nodes[1] if left == nodes[0] else nodes[0]
+        cam = {left: cams['passthrough_left'], right: cams['passthrough_right']}
+        s = np.mean([score(cam[nodes[0]], cam[nodes[1]], ua, ub) for ua, ub in pairs])
+        print('  %s = passthrough_left, crop %-8s: %3.0f%% of matches meet' % (left, crop, 100 * s))
+        if best is None or s > best[0]:
+            best = (s, crop, left, cam)
+    s, crop, left, cam = best
+    print('best: %s = passthrough_left, crop %s (%.0f%%)' % (left, crop, 100 * s))
+
+    mono = load_cams()
+    for node in nodes:
+        for side in ['slam_left', 'slam_right']:
+            ms = [matches(st[node][0], st[side][0]) for st in sets if side in st]
+            sc = np.mean([score(cam[node], mono[side], ua, ub) for ua, ub in ms]) if ms else 0
+            print('  %s vs %-10s: %4d matches, %3.0f%% meet' % (node, side, sum(len(m[0]) for m in ms), 100 * sc))
+
+
+if __name__ == '__main__':
+    main()

hands/tools/check_sides.py

@@ -0,0 +1,128 @@
+"""Check that the side cameras' images carry the right names (slam_left vs slam_right).
+
+usage: python tools/check_sides.py REC_DIR [--sets N]
+       python tools/check_sides.py --ring [--sets N]    (live, from ft-camd's ring)
+
+With --ring it exits 0 when the names are right, 3 when they're swapped (run ft-hands
+with --swap-sides), and 2 when it can't tell (too little texture in view, or the headset
+isn't worn).
+
+ft-camd tells the two side cameras' buffers apart by the order XRService allocated them,
+and after some XRService restarts that order puts each camera's images under the other's
+name. The tracker then sees every hand in one camera only, at the wrong depth. This
+matches features between the two images and measures how close each pair's rays pass
+with the factory calibration, once as named and once swapped: true matches meet in
+front of both cameras only under the right naming.
+"""
+import argparse
+import os
+import sys
+
+import cv2
+import numpy as np
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+from tools.show_set import index, read_set  # noqa: E402
+from tools import calib  # noqa: E402
+
+
+PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right'}   # as ft-hands maps them
+
+
+def load_cams():
+    return calib.load()
+
+
+def matches(a, b):
+    """Pixel pairs (N,2), (N,2) of ORB matches between two grey images."""
+    clahe = cv2.createCLAHE(2.0, (8, 8))
+    orb = cv2.ORB_create(3000)
+    ka, da = orb.detectAndCompute(clahe.apply(a), None)
+    kb, db = orb.detectAndCompute(clahe.apply(b), None)
+    if da is None or db is None:
+        return np.zeros((0, 2)), np.zeros((0, 2))
+    pairs = cv2.BFMatcher(cv2.NORM_HAMMING).knnMatch(da, db, k=2)
+    good = [p[0] for p in pairs if len(p) == 2 and p[0].distance < 0.75 * p[1].distance]
+    return (np.array([ka[m.queryIdx].pt for m in good]).reshape(-1, 2),
+            np.array([kb[m.trainIdx].pt for m in good]).reshape(-1, 2))
+
+
+def meet(cam_a, cam_b, ua, ub):
+    """Per match: closest distance between the two rays (m), and whether they meet in front of both."""
+    ra, rb = cam_a.rays(ua), cam_b.rays(ub)
+    w = cam_b.origin - cam_a.origin
+    n = np.cross(ra, rb)
+    nn = np.linalg.norm(n, axis=1)
+    dist = np.abs(w @ n.T) / np.maximum(nn, 1e-12)
+    # ray parameters at the closest points
+    ta = np.einsum('ij,ij->i', np.cross(np.broadcast_to(w, rb.shape), rb), n) / np.maximum(nn ** 2, 1e-12)
+    tb = np.einsum('ij,ij->i', np.cross(np.broadcast_to(w, ra.shape), ra), n) / np.maximum(nn ** 2, 1e-12)
+    return dist, (ta > 0.05) & (tb > 0.05)
+
+
+def score(cam_a, cam_b, ua, ub):
+    """Share of matches whose rays meet within 1 cm, in front of both cameras."""
+    if len(ua) == 0:
+        return 0.0
+    d, front = meet(cam_a, cam_b, ua, ub)
+    return float(np.mean((d < 0.01) & front))
+
+
+def recorded_pairs(rec, count):
+    """(label, slam_left image, slam_right image) from sets spread across a recording."""
+    path = os.path.join(rec, 'sets.bin')
+    offs = index(path)
+    for n in np.linspace(0, len(offs) - 1, count).astype(int):
+        images = read_set(path, offs[n])
+        if 'slam_left' in images and 'slam_right' in images:
+            yield 'set %5d' % n, images['slam_left'][0], images['slam_right'][0]
+
+
+def live_pairs(count):
+    """(label, slam_left image, slam_right image) from ft-camd's ring, half a second apart."""
+    import time
+    from tools.ring import Ring
+    ring = Ring()
+    if not ring.alive():
+        sys.exit('ft-camd isn\'t running (no heartbeat)')
+    cams = {}
+    for c in ring.cams:
+        name = PIPES.get(open('/sys/class/video4linux/video%d/name' % c.node).read().strip())
+        if name and not c.name.endswith('-dark'):
+            cams[name] = c
+    for k in range(count):
+        a, b = ring.read(cams['slam_left']), ring.read(cams['slam_right'])
+        if a is not None and b is not None:
+            yield 'frame %2d' % k, a.image, b.image
+        time.sleep(0.5)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('rec', nargs='?')
+    ap.add_argument('--ring', action='store_true', help='check the live cameras instead of a recording')
+    ap.add_argument('--sets', type=int, default=8, help='how many sets or live frames to check')
+    a = ap.parse_args()
+    if not a.ring and not a.rec:
+        ap.error('give a recording or --ring')
+    cams = load_cams()
+    left, right = cams['slam_left'], cams['slam_right']
+    named = swapped = 0.0
+    n = total_matches = 0
+    for label, img_l, img_r in (live_pairs(a.sets) if a.ring else recorded_pairs(a.rec, a.sets)):
+        ua, ub = matches(img_l, img_r)
+        s_named = score(left, right, ua, ub)     # slam_left's image seen by the left camera
+        s_swapped = score(right, left, ua, ub)   # ... by the right camera
+        named, swapped, n, total_matches = named + s_named, swapped + s_swapped, n + 1, total_matches + len(ua)
+        print('%s: %4d matches, meeting as named %3.0f%%, swapped %3.0f%%' %
+              (label, len(ua), 100 * s_named, 100 * s_swapped))
+    if n == 0 or total_matches < 100 or abs(named - swapped) / n < 0.2:
+        print('side cameras: can\'t tell (%d matches)' % total_matches)
+        sys.exit(2)
+    print('side cameras: %s (named %.2f, swapped %.2f)' %
+          ('as named' if named > swapped else 'SWAPPED', named / n, swapped / n))
+    sys.exit(0 if named > swapped else 3)
+
+
+if __name__ == '__main__':
+    main()

hands/tools/convert_models.py

@@ -0,0 +1,81 @@
+"""Convert the OpenCV Zoo ONNX ports of MediaPipe's hand models to ncnn.
+
+The ONNX files are Apache-2.0 ports of MediaPipe's palm detector and hand
+landmark models (huggingface.co/opencv/palm_detection_mediapipe and
+huggingface.co/opencv/handpose_estimation_mediapipe). pnnx does the
+conversion; two fix-ups follow:
+
+- The palm detector widens channels with ONNX Pad on the channel axis. pnnx
+  emits an ncnn layer called "Pad", which ncnn doesn't have, so rewrite those
+  as ncnn Padding with the channel-end amount (param 8 = behind).
+- Both models take NHWC input and start with a Permute to NCHW. Drop it, so we
+  can hand ncnn planar CHW Mats straight from the preprocessing step.
+
+usage: python convert_models.py   (writes models/ncnn/{palm,hand}.ncnn.{param,bin})
+"""
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+ROOT = os.path.join(HERE, '..')
+PNNX = os.path.join(sys.prefix, 'lib', 'python%d.%d' % sys.version_info[:2], 'site-packages', 'pnnx', 'pnnx')
+MODELS = [('palm', 'palm_detection_mediapipe_2023feb', 192),
+          ('hand', 'handpose_estimation_mediapipe_2023feb', 224)]
+
+
+def patch(param_text, pnnx_param_text):
+    lines = param_text.splitlines()
+    assert lines[0] == '7767517'
+    nlayers, nblobs = map(int, lines[1].split())
+    body = lines[2:]
+
+    # Channel pads: amounts come from the pnnx graph, which keeps the pads tuple.
+    pads = dict(re.findall(r'^Pad\s+(\S+)\s.*pads=\(0,0,0,0,0,(\d+),0,0\)', pnnx_param_text, re.M))
+    for i, line in enumerate(body):
+        f = line.split()
+        if f[0] == 'Pad':
+            amount = pads[f[1]]
+            body[i] = 'Padding %s %s %s %s %s 0=0 1=0 2=0 3=0 4=0 5=0.000000e+00 7=0 8=%s' % (
+                f[1], f[2], f[3], f[4], f[5], amount)
+
+    # Input permute: feed its consumers from in0 instead.
+    perm = next(i for i, line in enumerate(body) if line.split()[0] == 'Permute')
+    f = body[perm].split()
+    assert f[4] == 'in0' and f[6] == '0=4', body[perm]
+    blob = f[5]
+    del body[perm]
+    for i, line in enumerate(body):
+        f = line.split()
+        if f[0] == 'Input':
+            continue
+        nin, nout = int(f[2]), int(f[3])
+        ins = ['in0' if b == blob else b for b in f[4:4 + nin]]
+        body[i] = ' '.join(f[:4] + ins + f[4 + nin:])
+    return '\n'.join(['7767517', '%d %d' % (nlayers - 1, nblobs - 1)] + body) + '\n'
+
+
+def main():
+    out = os.path.join(ROOT, 'models', 'ncnn')
+    os.makedirs(out, exist_ok=True)
+    for short, name, size in MODELS:
+        src = os.path.join(ROOT, 'models', 'onnx', name + '.onnx')
+        with tempfile.TemporaryDirectory() as tmp:
+            shutil.copy(src, tmp)
+            subprocess.run([PNNX, name + '.onnx', 'inputshape=[1,%d,%d,3]' % (size, size), 'fp16=1'],
+                           cwd=tmp, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+            with open(os.path.join(tmp, name + '.ncnn.param')) as f:
+                param = f.read()
+            with open(os.path.join(tmp, name + '.pnnx.param')) as f:
+                pparam = f.read()
+            with open(os.path.join(out, short + '.ncnn.param'), 'w') as f:
+                f.write(patch(param, pparam))
+            shutil.copy(os.path.join(tmp, name + '.ncnn.bin'), os.path.join(out, short + '.ncnn.bin'))
+        print('wrote', short)
+
+
+if __name__ == '__main__':
+    main()

hands/tools/cut_sets.py

@@ -0,0 +1,51 @@
+"""Copy a few frame sets out of a recording (ft-hands --record) into a small one, to look at
+or check elsewhere without moving gigabytes. Plain Python, so it runs on the Frame's host.
+
+usage: python3 tools/cut_sets.py REC_DIR OUT_DIR [--sets N] (8, spread evenly) | [--at I,J,...]
+"""
+import argparse
+import os
+import struct
+
+HDR = struct.Struct('<8sII')
+
+
+def offsets(path):
+    offs, size = [], os.path.getsize(path)
+    with open(path, 'rb') as f:
+        off = 0
+        while off + HDR.size <= size:
+            f.seek(off)
+            magic, _, nbytes = HDR.unpack(f.read(HDR.size))
+            if magic[:7] != b'FHSET01' or off + nbytes > size:
+                break
+            offs.append((off, nbytes))
+            off += nbytes
+    return offs
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('rec')
+    ap.add_argument('out')
+    ap.add_argument('--sets', type=int, default=8)
+    ap.add_argument('--at', default='')
+    a = ap.parse_args()
+    src = os.path.join(a.rec, 'sets.bin')
+    offs = offsets(src)
+    if a.at:
+        pick = [int(i) for i in a.at.split(',')]
+    else:
+        n = max(1, min(a.sets, len(offs)))
+        pick = [round(i * (len(offs) - 1) / max(n - 1, 1)) for i in range(n)]
+    os.makedirs(a.out, exist_ok=True)
+    with open(src, 'rb') as f, open(os.path.join(a.out, 'sets.bin'), 'wb') as out:
+        for i in pick:
+            off, nbytes = offs[i]
+            f.seek(off)
+            out.write(f.read(nbytes))
+    print('%d of %d sets (%s) -> %s' % (len(pick), len(offs), ','.join(map(str, pick)), a.out))
+
+
+if __name__ == '__main__':
+    main()

hands/tools/depth_report.py

@@ -0,0 +1,256 @@
+"""How good the tracker's depth is, from a replay's depth dump, without ground truth.
+
+usage: python3 tools/depth_report.py DEPTH [DEPTH...] [--still M/S]
+
+DEPTH comes from `hands/build/ft-handreplay DIR --depth DEPTH`. Every measure is split by how the
+hand was seen: by the two lower cameras ("lower pair"), by a lower and an upper camera on
+one side ("lower+upper"), or by one camera. Distances are from the head (between the eyes).
+
+1. How the hands were seen: the share of hand updates in each way, by distance.
+2. Noise along the line of sight against across it. Each update's palm is compared with a
+   straight line through the two updates before it (ft-handreplay's jitter measure), and the
+   miss is split along the line from the hand's cameras to the palm and across it. Given
+   as a robust sigma per axis, measured (as triangulated) and published (after the One Euro
+   filter), on updates where the published palm moved slower than --still (default 0.15
+   m/s), so the miss is mostly noise and not the hand speeding up. For two cameras,
+   geometry predicts along/across = 2 Z / B: Z the distance, B the cameras' baseline
+   across the line of sight.
+3. One-camera distance: on two-camera updates, each camera's one-view guess (distance from
+   how big the palm looks, at the user's learned hand size) against the triangulated
+   distance from that camera.
+4. A camera lost: from two-camera updates, what the tracker would have had if one of the
+   two cameras dropped out there. It keeps the last distance and moves a share of the way
+   to the one-view guess each update (0.1 now, kMonoDepthGain in track/tracker.cpp);
+   also shown with other shares, 0 (keep the distance) and 1 (take each guess), and with
+   the guess first scaled by how far off it was while both cameras saw the hand. Compared with the
+   triangulated distance from that camera, 0.1-2 s after the loss.
+"""
+import argparse
+import collections
+
+import numpy as np
+
+BINS = [0.0, 0.35, 0.50, 0.65, 9.0]
+BIN_NAMES = ['<35 cm', '35-50', '50-65', '65+ cm']
+MODES = ['lower pair', 'lower+upper', 'one camera']
+HORIZONS = [0.1, 0.25, 0.5, 1.0, 2.0]
+# (share of the way toward the one-view guess per update, whether the guess is first scaled by
+# how far off it was while both cameras saw the hand)
+GAINS = [(0.0, False), (0.02, False), (0.05, False), (0.1, False), (1.0, False), (0.1, True), (1.0, True)]
+GAP = 0.1   # s: a longer gap between a hand's updates breaks its run
+
+
+def load(path):
+    cams, rows = {}, []
+    with open(path) as f:
+        for line in f:
+            w = line.split()
+            if not w:
+                continue
+            if w[0] == '#':
+                if w[1] == 'cam':
+                    cams[w[2]] = (np.array([float(x) for x in w[3:6]]), float(w[6]))
+                continue
+            r = {'t': float(w[0]), 'id': int(w[1]), 'side': w[2], 'n': int(w[3]),
+                 'cams': w[4], 'res': float(w[5]), 'scale': float(w[6]),
+                 'raw': np.array([float(x) for x in w[7:10]]), 'sm': np.array([float(x) for x in w[10:13]]),
+                 'views': {}}
+            for k in range(13, len(w), 5):
+                r['views'][w[k]] = np.array([float(x) for x in w[k + 2:k + 5]])
+            rows.append(r)
+    return cams, rows
+
+
+def mode(r):
+    names = r['cams'].split('+')
+    if r['n'] == 1:
+        return 'one camera'
+    if sorted(names) == ['slam_left', 'slam_right']:
+        return 'lower pair'
+    if len(names) == 2 and all(n.startswith(('slam_', 'upper_')) for n in names) and \
+            names[0].split('_')[1] == names[1].split('_')[1]:
+        return 'lower+upper'
+    return 'other'
+
+
+def dist_bin(p):
+    return min(np.searchsorted(BINS, np.linalg.norm(p), side='right') - 1, len(BIN_NAMES) - 1)
+
+
+def tracks(rows):
+    """A hand's updates, in order, split where they're more than GAP apart."""
+    by_id = collections.defaultdict(list)
+    for r in rows:
+        by_id[r['id']].append(r)
+    for rs in by_id.values():
+        run = [rs[0]]
+        for r in rs[1:]:
+            if r['t'] - run[-1]['t'] >= GAP:
+                yield run
+                run = []
+            run.append(r)
+        yield run
+
+
+def two_camera_runs(rows):
+    """Stretches of a hand's updates all seen by the same two cameras."""
+    for run in tracks(rows):
+        seg = []
+        for r in run:
+            ok = r['n'] == 2 and mode(r) != 'other'
+            if ok and seg and r['cams'] == seg[-1]['cams']:
+                seg.append(r)
+                continue
+            if len(seg) > 2:
+                yield seg
+            seg = [r] if ok else []
+        if len(seg) > 2:
+            yield seg
+
+
+def pct(v, q):
+    return np.percentile(v, q) if len(v) else float('nan')
+
+
+def seen_share(rows):
+    print('\n1. How the hands were seen (share of hand updates)')
+    count = collections.Counter((mode(r), dist_bin(r['raw'])) for r in rows)
+    total = collections.Counter(dist_bin(r['raw']) for r in rows)
+    print('%-12s' % '' + ''.join('%10s' % b for b in BIN_NAMES) + '%10s' % 'all')
+    for m in MODES + ['other']:
+        cells = [100 * count[m, b] / max(total[b], 1) for b in range(len(BIN_NAMES))]
+        allp = 100 * sum(count[m, b] for b in range(len(BIN_NAMES))) / max(len(rows), 1)
+        print('%-12s' % m + ''.join('%9.0f%%' % c for c in cells) + '%9.0f%%' % allp)
+    print('%-12s' % 'updates' + ''.join('%10d' % total[b] for b in range(len(BIN_NAMES))) + '%10d' % len(rows))
+    res = collections.defaultdict(list)
+    for r in rows:
+        if r['res'] >= 0:
+            res[mode(r)].append(r['res'] * 1000)
+    print('triangulation residual (rms ray miss, median): ' +
+          ', '.join('%s %.1f mm' % (m, np.median(v)) for m, v in res.items()))
+
+
+def noise(rows, cams, still):
+    print('\n2. Noise along the line of sight vs across it (sigma per axis, mm; palm slower than %.2f m/s)' % still)
+    acc = collections.defaultdict(lambda: collections.defaultdict(list))
+    for run in tracks(rows):
+        for a, b, c in zip(run, run[1:], run[2:]):
+            if not (mode(a) == mode(b) == mode(c)) or a['cams'] != b['cams'] or b['cams'] != c['cams']:
+                continue
+            dt0, dt1 = b['t'] - a['t'], c['t'] - b['t']
+            if dt0 < 1e-3 or np.linalg.norm(b['sm'] - a['sm']) / dt0 > still:
+                continue
+            names = b['cams'].split('+')
+            origins = [cams[n][0] for n in names]
+            o = np.mean(origins, axis=0)
+            u = b['raw'] - o
+            z = np.linalg.norm(u)
+            u /= z
+            key = (mode(b), dist_bin(b['raw']))
+            for kind in ('raw', 'sm'):
+                miss = c[kind] - b[kind] - (b[kind] - a[kind]) * (dt1 / dt0)
+                along = miss @ u
+                acc[key][kind + '_along'].append(abs(along))
+                acc[key][kind + '_across'].append(np.linalg.norm(miss - along * u))
+            if len(origins) == 2:
+                base = origins[0] - origins[1]
+                acc[key]['pred'].append(2 * z / np.linalg.norm(base - (base @ u) * u))
+    # |along| is half-normal: sigma = median / 0.674; |across| is Rayleigh (2 axes): sigma = median / 1.177
+    print('%-12s %-7s %6s | %-24s | %-24s | %s' % ('', '', 'n', 'measured along/across', 'published along/across',
+                                                  'ratio measured (geometry)'))
+    for m in MODES:
+        for bi, bn in enumerate(BIN_NAMES):
+            d = acc.get((m, bi))
+            if not d or len(d['raw_along']) < 20:
+                continue
+            s = {k: np.median(v) / (0.674 if k.endswith('along') else 1.177) * 1000
+                 for k, v in d.items() if k != 'pred'}
+            pred = '(%.1f)' % np.median(d['pred']) if d['pred'] else ''
+            print('%-12s %-7s %6d | %7.1f / %-5.1f  x%-6.1f | %7.1f / %-5.1f  x%-6.1f | x%.1f %s' % (
+                m, bn, len(d['raw_along']), s['raw_along'], s['raw_across'], s['raw_along'] / s['raw_across'],
+                s['sm_along'], s['sm_across'], s['sm_along'] / s['sm_across'],
+                s['raw_along'] / s['raw_across'], pred))
+
+
+def one_camera(rows, cams):
+    print('\n3. One-camera distance vs triangulated, on two-camera updates (error of the one-view guess)')
+    acc = collections.defaultdict(list)
+    for r in rows:
+        if r['n'] != 2 or mode(r) == 'other':
+            continue
+        for name, p in r['views'].items():
+            if np.isnan(p).any():
+                continue
+            o = cams[name][0]
+            truth = np.linalg.norm(r['raw'] - o)
+            acc[name.split('_')[0], dist_bin(r['raw'])].append((np.linalg.norm(p - o) - truth, truth))
+    print('%-8s %-7s %6s %12s %12s %14s %12s' % ('camera', '', 'n', 'median |err|', '90% |err|', 'median |err| %',
+                                                  'bias'))
+    for cam in ('slam', 'upper'):
+        for bi, bn in enumerate(BIN_NAMES):
+            v = acc.get((cam, bi))
+            if not v or len(v) < 20:
+                continue
+            e = np.array([x[0] for x in v])
+            rel = e / np.array([x[1] for x in v])
+            print('%-8s %-7s %6d %9.0f mm %9.0f mm %13.0f%% %+11.0f%%' % (
+                'lower' if cam == 'slam' else 'upper', bn, len(v), 1000 * np.median(abs(e)), 1000 * pct(abs(e), 90),
+                100 * np.median(abs(rel)), 100 * np.median(rel)))
+
+
+def lost_camera(rows, cams):
+    print('\n4. A camera lost: distance error after the loss (median |err| mm / 90% mm), by how the tracker '
+          'moves toward the one-view guess each update ("scaled": the guess times how far off it was, '
+          'triangulated / guess, median over the last 30 two-camera updates)')
+    errs = collections.defaultdict(list)
+    for run in two_camera_runs(rows):
+        for s in range(1, len(run) - 1, 3):
+            for name in run[s]['views']:
+                o = cams[name][0]
+                guess = lambda r: np.linalg.norm(r['views'][name] - o)
+                ratios = [np.linalg.norm(r['raw'] - o) / guess(r) for r in run[max(0, s - 30):s]
+                          if not np.isnan(r['views'][name]).any()]
+                ratio = np.median(ratios) if ratios else 1.0
+                for g, scaled in GAINS:
+                    d = np.linalg.norm(run[s - 1]['raw'] - o)
+                    h = 0
+                    for r in run[s:]:
+                        if np.isnan(r['views'][name]).any():
+                            break
+                        d += g * (guess(r) * (ratio if scaled else 1.0) - d)
+                        elapsed = r['t'] - run[s - 1]['t']
+                        while h < len(HORIZONS) and elapsed >= HORIZONS[h]:
+                            errs[g, scaled, HORIZONS[h], name.split('_')[0]].append(abs(d - np.linalg.norm(r['raw'] - o)))
+                            h += 1
+    print('%-8s %-18s' % ('camera', 'toward guess') + ''.join('%14s' % ('%.2g s' % t) for t in HORIZONS))
+    for cam in ('slam', 'upper'):
+        for g, scaled in GAINS:
+            label = {0.0: '0 (keep)', 0.1: '0.1 (now)', 1.0: '1 (guess)'}.get(g, '%g' % g)
+            if scaled:
+                label = '%g scaled' % g
+            cells = []
+            for t in HORIZONS:
+                v = errs.get((g, scaled, t, cam), [])
+                cells.append('%5.0f / %-4.0f' % (1000 * np.median(v), 1000 * pct(v, 90)) if len(v) >= 20 else '%14s' % '-')
+            print('%-8s %-18s' % ('lower' if cam == 'slam' else 'upper', label) + ''.join('%14s' % c for c in cells))
+    n = sum(len(errs.get((0.1, False, HORIZONS[0], c), [])) for c in ('slam', 'upper'))
+    print('(%d simulated losses)' % n)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('depth', nargs='+')
+    ap.add_argument('--still', type=float, default=0.15, help='m/s: palm speed limit for the noise measure')
+    a = ap.parse_args()
+    for path in a.depth:
+        cams, rows = load(path)
+        print('== %s: %d hand updates, %d hands' % (path, len(rows), len({r['id'] for r in rows})))
+        seen_share(rows)
+        noise(rows, cams, a.still)
+        one_camera(rows, cams)
+        lost_camera(rows, cams)
+        print()
+
+
+if __name__ == '__main__':
+    main()

hands/tools/ring.py

@@ -0,0 +1,80 @@
+"""Read frames from ft-camd's shared-memory ring (layout: camd/fhring.h)."""
+import mmap
+import os
+import struct
+import time
+
+import numpy as np
+
+RING_FILE = '/run/user/%d/frametop-hands/cam-ring' % os.getuid()
+MAGIC = b'FHRING01'
+HDR = struct.Struct('<8sIIIIQqQ16x')                     # 64 bytes
+CAM = struct.Struct('<32s32siIIIIIQQQQQ32x')             # 160 bytes
+SLOT = struct.Struct('<QQQQQIf16x')                       # 64 bytes
+MAX_CAMS = 8
+LATEST_OFF = 32 + 32 + 4 * 6 + 8 * 2                      # cam.latest within fh_ring_cam_t
+HEARTBEAT_OFF = 40
+
+
+class Frame:
+    __slots__ = ('cam', 'frame', 'capture_ns', 'dqbuf_ns', 'publish_ns', 'v4l2_seq', 'mean', 'image')
+
+    def __init__(self, cam, fields, image):
+        self.cam = cam
+        (_, self.frame, self.capture_ns, self.dqbuf_ns, self.publish_ns, self.v4l2_seq, self.mean) = fields
+        self.image = image
+
+
+class RingCamera:
+    def __init__(self, index, fields):
+        (sensor, name, self.node, self.format, self.width, self.height, self.stride, self.nslots,
+         self.slot_offset, self.slot_bytes, _latest, _pub, _drop) = fields
+        self.index = index
+        self.sensor = sensor.split(b'\0', 1)[0].decode()
+        self.name = name.split(b'\0', 1)[0].decode()
+        self.latest_off = HDR.size + index * CAM.size + LATEST_OFF
+
+    def __repr__(self):
+        return 'RingCamera(video%d %s %dx%d)' % (self.node, self.sensor, self.width, self.height)
+
+
+class Ring:
+    def __init__(self, path=RING_FILE):
+        fd = os.open(path, os.O_RDONLY)
+        try:
+            self.map = mmap.mmap(fd, 0, mmap.MAP_SHARED, mmap.PROT_READ)
+        finally:
+            os.close(fd)
+        magic, version, hdr_bytes, ncams, _, file_bytes, self.writer_pid, _ = HDR.unpack_from(self.map, 0)
+        if magic != MAGIC or version != 1:
+            raise RuntimeError('%s is not an ft-camd ring (magic %r version %d)' % (path, magic, version))
+        self.cams = [RingCamera(i, CAM.unpack_from(self.map, HDR.size + i * CAM.size)) for i in range(ncams)]
+
+    def heartbeat_ns(self):
+        return struct.unpack_from('<Q', self.map, HEARTBEAT_OFF)[0]
+
+    def alive(self, max_age=1.0):
+        hb = self.heartbeat_ns()
+        return hb != 0 and (time.clock_gettime_ns(time.CLOCK_MONOTONIC) - hb) / 1e9 < max_age
+
+    def latest(self, cam):
+        return struct.unpack_from('<Q', self.map, cam.latest_off)[0]
+
+    def read(self, cam, n=None):
+        """Copy frame n (default: the newest) of a camera, or None if it's gone or being written."""
+        for _ in range(3):
+            if n is None or n == 0:
+                n = self.latest(cam)
+            if n == 0:
+                return None
+            off = cam.slot_offset + (n % cam.nslots) * cam.slot_bytes
+            fields = SLOT.unpack_from(self.map, off)
+            if fields[0] != 2 * n + 2:
+                return None
+            start = off + SLOT.size
+            image = np.frombuffer(self.map, np.uint8, cam.stride * cam.height, start).reshape(cam.height, cam.stride)
+            image = image[:, :cam.width].copy()
+            if struct.unpack_from('<Q', self.map, off)[0] == fields[0]:
+                return Frame(cam, fields, image)
+            n = None                                      # overwritten while copying: take the newest
+        return None

hands/tools/show_set.py

@@ -0,0 +1,129 @@
+"""Draw frame sets from a recording (ft-hands --record) with what the tracker saw.
+
+usage: python tools/show_set.py REC_DIR SET [SET...] [--timeline TL] [--out DIR]
+
+SET is a set index (ft-handreplay's timeline gives them). With --timeline (ft-handreplay
+--timeline), each camera shows the tracker's views at that set: the crop for the next
+frame, labelled with the hand and presence. Recordings made with ft-camd --with-dark get
+a second row: each camera's latest dark frame (<name>_dk), stretched to be visible and
+labelled with its mean brightness. Recordings made with ft-camd --with-color get a row of
+the color cameras (color_video<N>). Writes OUT/set_<n>.jpg (default /tmp).
+"""
+import argparse
+import os
+import struct
+
+import cv2
+import numpy as np
+
+HDR = struct.Struct('<8sII')
+CAM = struct.Struct('<16sIIQQ')
+ORDER = ['slam_left', 'slam_right', 'upper_left', 'upper_right']
+
+
+def index(path):
+    """Byte offset of every set in sets.bin."""
+    offs, size = [], os.path.getsize(path)
+    with open(path, 'rb') as f:
+        off = 0
+        while off + HDR.size <= size:
+            f.seek(off)
+            magic, n, nbytes = HDR.unpack(f.read(HDR.size))
+            if magic[:7] != b'FHSET01' or off + nbytes > size:
+                break
+            offs.append(off)
+            off += nbytes
+    return offs
+
+
+def read_set(path, off):
+    with open(path, 'rb') as f:
+        f.seek(off)
+        _, n, _ = HDR.unpack(f.read(HDR.size))
+        cams = [CAM.unpack(f.read(CAM.size)) for _ in range(n)]
+        out = {}
+        for name, w, h, cap, dq in cams:
+            px = np.frombuffer(f.read(w * h), np.uint8).reshape(h, w)
+            out[name.rstrip(b'\0').decode()] = (px, cap)
+    return out
+
+
+def views_at(timeline, n):
+    out = []
+    for line in open(timeline):
+        f = line.split()
+        if len(f) > 1 and f[1] == 'view' and int(f[-1]) == n:
+            out.append({'hand': int(f[2]), 'cam': f[3], 'presence': float(f[5]),
+                        'c': (float(f[7]), float(f[8])), 'size': float(f[9]), 'rot': float(f[10])})
+    return out
+
+
+def dark_tile(frame, shape, name):
+    """A dark frame, stretched from its 1st to 99.5th percentile; black if there's none."""
+    h, w = shape
+    if frame is None:
+        return np.zeros((h, w, 3), np.uint8)
+    px = frame[0]
+    lo, hi = np.percentile(px, (1, 99.5))
+    gain = 255 / max(hi - lo, 1)
+    img = np.clip((px.astype(np.float32) - lo) * gain, 0, 255).astype(np.uint8)
+    img = cv2.cvtColor(cv2.resize(img, (w, h)), cv2.COLOR_GRAY2BGR)
+    cv2.putText(img, '%s_dk mean %.1f, x%.0f' % (name, px.mean(), gain), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1.0,
+                (255, 255, 0), 2)
+    return img
+
+
+def view_tile(px, name, views):
+    """A frame, CLAHE'd, with the tracker's views on it, 512 px high."""
+    img = cv2.cvtColor(cv2.createCLAHE(2.0, (8, 8)).apply(px), cv2.COLOR_GRAY2BGR)
+    for v in views:
+        if v['cam'] != name:
+            continue
+        c, s, r = v['c'], v['size'], v['rot']
+        box = cv2.boxPoints(((c[0], c[1]), (s, s), np.degrees(r)))
+        col = (0, 255, 0) if v['presence'] >= 0.5 else (0, 0, 255)
+        cv2.polylines(img, [box.astype(np.int32)], True, col, 2)
+        cv2.putText(img, 'h%d %.2f' % (v['hand'], v['presence']), (int(c[0] - s / 2), int(c[1] - s / 2) - 6),
+                    cv2.FONT_HERSHEY_SIMPLEX, 0.8, col, 2)
+    cv2.putText(img, name, (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1.0, (255, 255, 0), 2)
+    scale = 512 / img.shape[0]
+    return cv2.resize(img, (int(img.shape[1] * scale), 512))
+
+
+def draw(images, views):
+    """Rows: the mono cameras; their dark frames, if recorded; the color cameras, if recorded."""
+    tiles, dark = [], []
+    for name in ORDER:
+        if name not in images:
+            continue
+        tiles.append(view_tile(images[name][0], name, views))
+        dark.append(dark_tile(images.get(name + '_dk'), tiles[-1].shape[:2], name))
+    rows = [np.hstack(tiles)]
+    if any(k.endswith('_dk') for k in images):
+        rows.append(np.hstack(dark))
+    color = sorted(k for k in images if k.startswith('color_'))
+    if color:
+        rows.append(np.hstack([view_tile(images[k][0], k, views) for k in color]))
+    width = max(r.shape[1] for r in rows)
+    return np.vstack([np.pad(r, ((0, 0), (0, width - r.shape[1]), (0, 0))) for r in rows])
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('rec')
+    ap.add_argument('sets', type=int, nargs='+')
+    ap.add_argument('--timeline')
+    ap.add_argument('--out', default='/tmp')
+    a = ap.parse_args()
+    path = os.path.join(a.rec, 'sets.bin')
+    offs = index(path)
+    for n in a.sets:
+        images = read_set(path, offs[n])
+        views = views_at(a.timeline, n) if a.timeline else []
+        out = os.path.join(a.out, 'set_%05d.jpg' % n)
+        cv2.imwrite(out, draw(images, views), [cv2.IMWRITE_JPEG_QUALITY, 85])
+        print(out)
+
+
+if __name__ == '__main__':
+    main()

hands/tools/watch_gestures.py

@@ -0,0 +1,95 @@
+"""Watch the gestures ft-hands publishes, live: pinches and grips, begins, ends, and drags.
+
+usage: python3 tools/watch_gestures.py [--every S] [--distance]
+
+Prints a line when a pinch or a grip (a closed hand) begins or ends on either hand. It goes
+by the counters, so a quick tap between two reads still shows. While one is held, every
+--every seconds (default 0.1) it prints how far its point has moved since it began, in the
+head frame (turning your head moves it too; a real consumer turns both points into the room
+first, see include/fh_gestures.h). --distance also prints each hand's thumb-to-index
+distance and finger curl, to see how close a gesture comes to the thresholds.
+Version 1 files (pinches only) still work.
+"""
+import argparse
+import mmap
+import os
+import struct
+import time
+
+HDR = struct.Struct('<8sIIQQQffff8x')        # 64 bytes
+SLOT = struct.Struct('<IIIIQQff3f3f')        # 64 bytes
+TRACKED, DOWN, LOST = 1, 2, 4
+SIDES = ('left ', 'right')
+KINDS = ('pinch', 'grip ')
+
+
+def path():
+    return '/run/user/%d/frametop-hands/gestures' % os.getuid()
+
+
+def read(m):
+    """(header, [[pinch left, right], [grip left, right]]) under the sequence lock, or None
+    if it's being written. Version 1 has no grips: they read as all zero."""
+    for _ in range(10):
+        s1 = struct.unpack_from('<Q', m, 16)[0]
+        if s1 % 2 == 0:
+            h = HDR.unpack_from(m, 0)
+            kinds = 2 if h[1] >= 2 and len(m) >= HDR.size + 4 * SLOT.size else 1
+            g = [[SLOT.unpack_from(m, HDR.size + (k * 2 + s) * SLOT.size) for s in range(2)] for k in range(kinds)]
+            if kinds == 1:
+                g.append([(0,) * 14, (0,) * 14])
+            if struct.unpack_from('<Q', m, 16)[0] == s1:
+                return h, g
+        time.sleep(0.0005)
+    return None
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('--every', type=float, default=0.1, help='seconds between drag lines while held')
+    ap.add_argument('--distance', action='store_true', help="print each hand's pinch distance and finger curl")
+    a = ap.parse_args()
+    with open(path(), 'rb') as f:
+        m = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
+    first = read(m)
+    if first is None or first[0][0] != b'FHGEST01':
+        raise SystemExit('%s is not an ft-hands gestures file' % path())
+    h, g = first
+    print('version %d; thresholds: pinch begins under %.3f m, ends over %.3f m; grip begins with every finger '
+          'curled under %.2f, ends over %.2f' % (h[1], h[6], h[7], h[8], h[9]))
+    seen = [[(q[2], q[3]) for q in kind] for kind in g]   # begins, ends
+    last_drag = last_dist = 0.0
+    while True:
+        got = read(m)
+        if got:
+            h, g = got
+            now = time.monotonic()
+            for k, kind in enumerate(g):
+                for s, q in enumerate(kind):
+                    flags, hand, begins, ends, begin_ns, end_ns, dist, strength = q[:8]
+                    point, begin_point = q[8:11], q[11:14]
+                    if begins != seen[k][s][0]:
+                        print('%s %s BEGIN (#%d, hand %d) at %+.3f %+.3f %+.3f  %s %.3f' %
+                              (SIDES[s], KINDS[k], begins, hand, *begin_point, 'curl' if k else 'd', dist), flush=True)
+                    if ends != seen[k][s][1]:
+                        held = (end_ns - begin_ns) / 1e9 if end_ns >= begin_ns else 0
+                        print('%s %s %s after %.2f s' % (SIDES[s], KINDS[k], 'LOST' if flags & LOST else 'END', held),
+                              flush=True)
+                    seen[k][s] = (begins, ends)
+                    if flags & DOWN and now - last_drag >= a.every:
+                        d = [point[i] - begin_point[i] for i in range(3)]
+                        print('%s %s   drag %+6.1f %+6.1f %+6.1f mm  (%.0f mm)' %
+                              (SIDES[s], KINDS[k], *(1000 * x for x in d), 1000 * sum(x * x for x in d) ** 0.5),
+                              flush=True)
+            if any(q[0] & DOWN for kind in g for q in kind) and now - last_drag >= a.every:
+                last_drag = now
+            if a.distance and now - last_dist >= 0.2:
+                last_dist = now
+                print('    ' + '   '.join(
+                    '%s %s' % (SIDES[s].strip(), 'd %.3f curl %.2f' % (g[0][s][6], g[1][s][6]) if g[0][s][0] & TRACKED
+                               else '-') for s in range(2)), flush=True)
+        time.sleep(0.005)
+
+
+if __name__ == '__main__':
+    main()

hands/track/calib.cpp

@@ -0,0 +1,221 @@
+#include <cstdlib>
+#include "calib.h"
+
+#include <json/json.h>
+#include <unistd.h>
+
+#include <algorithm>
+
+#include <fstream>
+#include <iterator>
+#include <memory>
+
+namespace {
+
+double theta_d(const Camera &c, double t) {
+    const double t2 = t * t;
+    return t * (1 + t2 * (c.k[0] + t2 * (c.k[1] + t2 * (c.k[2] + t2 * c.k[3]))));
+}
+
+// 4x4 transform (row-major) from a {plus_x, plus_z, position} pose.
+void pose(const Json::Value &d, double scale, double T[4][4]) {
+    V3 x{d["plus_x"][0].asDouble(), d["plus_x"][1].asDouble(), d["plus_x"][2].asDouble()};
+    V3 z{d["plus_z"][0].asDouble(), d["plus_z"][1].asDouble(), d["plus_z"][2].asDouble()};
+    V3 y{z[1] * x[2] - z[2] * x[1], z[2] * x[0] - z[0] * x[2], z[0] * x[1] - z[1] * x[0]};
+    for (int i = 0; i < 3; ++i) {
+        T[i][0] = x[i], T[i][1] = y[i], T[i][2] = z[i];
+        T[i][3] = d["position"][i].asDouble() * scale;
+        T[3][i] = 0;
+    }
+    T[3][3] = 1;
+}
+
+void mul(const double A[4][4], const double B[4][4], double C[4][4]) {
+    for (int i = 0; i < 4; ++i)
+        for (int j = 0; j < 4; ++j) {
+            C[i][j] = 0;
+            for (int k = 0; k < 4; ++k) C[i][j] += A[i][k] * B[k][j];
+        }
+}
+
+void invert_rigid(const double A[4][4], double B[4][4]) {
+    for (int i = 0; i < 3; ++i)
+        for (int j = 0; j < 3; ++j) B[i][j] = A[j][i];
+    for (int i = 0; i < 3; ++i) B[i][3] = -(B[i][0] * A[0][3] + B[i][1] * A[1][3] + B[i][2] * A[2][3]);
+    B[3][0] = B[3][1] = B[3][2] = 0, B[3][3] = 1;
+}
+
+bool read_json(const char *path, Json::Value &v, std::string &err) {
+    std::ifstream f(path);
+    Json::CharReaderBuilder b;
+    std::string e;
+    if (!f || !Json::parseFromStream(b, f, &v, &e)) {
+        err = std::string(path) + ": " + (f ? e : "can't open");
+        return false;
+    }
+    return true;
+}
+
+}  // namespace
+
+V2 Camera::project_cam(V3 p) const {
+    const double r = std::hypot(p[0], p[1]);
+    const double s = r > 1e-12 ? theta_d(*this, std::atan2(r, p[2])) / r : 0;
+    return {fx * p[0] * s + cx, fy * p[1] * s + cy};
+}
+
+V3 Camera::unproject(V2 uv) const {
+    const double mx = (uv[0] - cx) / fx, my = (uv[1] - cy) / fy, td = std::hypot(mx, my);
+    double t = td;
+    for (int i = 0; i < 8; ++i) {  // Newton on theta_d(t) = td
+        const double t2 = t * t;
+        const double df = 1 + t2 * (3 * k[0] + t2 * (5 * k[1] + t2 * (7 * k[2] + t2 * 9 * k[3])));
+        t = std::clamp(t - (theta_d(*this, t) - td) / df, 0.0, M_PI);
+    }
+    const double s = td > 1e-12 ? std::sin(t) / td : 1;
+    return {mx * s, my * s, std::cos(t)};
+}
+
+V3 Camera::ray(V2 uv) const {
+    const V3 c = unproject(uv);
+    return {R[0][0] * c[0] + R[0][1] * c[1] + R[0][2] * c[2], R[1][0] * c[0] + R[1][1] * c[1] + R[1][2] * c[2],
+            R[2][0] * c[0] + R[2][1] * c[1] + R[2][2] * c[2]};
+}
+
+V2 Camera::project(V3 head, double *depth) const {
+    const V3 d = head - origin;
+    const V3 c{R[0][0] * d[0] + R[1][0] * d[1] + R[2][0] * d[2], R[0][1] * d[0] + R[1][1] * d[1] + R[2][1] * d[2],
+               R[0][2] * d[0] + R[1][2] * d[1] + R[2][2] * d[2]};
+    if (depth) *depth = c[2];
+    return project_cam(c);
+}
+
+double Camera::off_axis(V2 uv) const { return std::acos(std::clamp(unproject(uv)[2], -1.0, 1.0)) * 180 / M_PI; }
+
+// A headset file such as /persist/xrservice.json. In the dev container the host's / is at
+// /run/host (distrobox doesn't mount /persist); off the Frame, FRAME_JOB_DEVICE_ROOT can
+// point at a folder with copies of them.
+static std::string device_path(const char *path) {
+    if (const char *root = std::getenv("FRAME_JOB_DEVICE_ROOT")) return std::string(root) + path;
+    const std::string host = std::string("/run/host") + path;
+    return access(path, R_OK) != 0 && access(host.c_str(), R_OK) == 0 ? host : path;
+}
+
+bool load_calibration(std::map<std::string, Camera> &out, std::string &err) {
+    Json::Value rig, dev;
+    if (!read_json(device_path("/persist/xrservice.json").c_str(), rig, err) ||
+        !read_json(device_path("/persist/device_config.json").c_str(), dev, err))
+        return false;
+    double cad_from_cam0[4][4], cad_from_head[4][4], head_from_cad[4][4], head_from_cam0[4][4];
+    pose(dev["cv"]["cad_from_cal"], 1.0, cad_from_cam0);
+    pose(dev["head"], 1.0, cad_from_head);
+    invert_rigid(cad_from_head, head_from_cad);
+    mul(head_from_cad, cad_from_cam0, head_from_cam0);
+    for (const Json::Value &c : rig["cameras"]) {
+        Camera cam;
+        cam.name = c["sourceCamera"].asString();
+        cam.width = c["width"].asInt(), cam.height = c["height"].asInt();
+        for (const Json::Value &in : c["intrinsics"]) {
+            if (in["cameraModel"].asString() != "kb") continue;
+            cam.fx = in["fx"].asDouble(), cam.fy = in["fy"].asDouble();
+            cam.cx = in["cx"].asDouble(), cam.cy = in["cy"].asDouble();
+            cam.k[0] = in["k1"].asDouble(), cam.k[1] = in["k2"].asDouble();
+            cam.k[2] = in["k3"].asDouble(), cam.k[3] = in["k4"].asDouble();
+        }
+        double cam0_from_cam[4][4], head_from_cam[4][4];
+        pose(c["extrinsics"], 1e-3, cam0_from_cam);
+        mul(head_from_cam0, cam0_from_cam, head_from_cam);
+        for (int i = 0; i < 3; ++i) {
+            for (int j = 0; j < 3; ++j) cam.R[i][j] = head_from_cam[i][j];
+            cam.origin[i] = head_from_cam[i][3];
+        }
+        out[cam.name] = cam;
+    }
+    if (out.empty()) err = "no cameras in /persist/xrservice.json";
+    return !out.empty();
+}
+
+bool load_color_calibration(std::map<std::string, Camera> &out, const std::string &left_node,
+                            const std::string &right_node, bool crop_subtract, int scale, std::string &err) {
+    // The module's EEPROM: some binary, then the calibration as JSON (world-readable)
+    const std::string path = device_path("/sys/devices/platform/soc@0/ac15000.cci/i2c-0/0-0050/eeprom");
+    std::ifstream f(path, std::ios::binary);
+    const std::string raw((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
+    const size_t key = raw.find("\"alignment_method\"");
+    const size_t start = key == std::string::npos ? key : raw.rfind('{', key);
+    Json::Value rig, dev;
+    std::string e;
+    std::unique_ptr<Json::CharReader> reader(Json::CharReaderBuilder().newCharReader());
+    if (start == std::string::npos || !reader->parse(raw.data() + start, raw.data() + raw.size(), &rig, &e))
+        return err = path + ": no calibration JSON " + e, false;
+    if (!read_json(device_path("/persist/device_config.json").c_str(), dev, err)) return false;
+    double cad_from_head[4][4], head_from_cad[4][4];
+    pose(dev["head"], 1.0, cad_from_head);
+    invert_rigid(cad_from_head, head_from_cad);
+    constexpr int kValidWidth = 1972;   // pixels per row XRService's buffers deliver (of 2464)
+    int n = 0;
+    for (const Json::Value &c : rig["cameras"]) {
+        const std::string source = c["sourceCamera"].asString();
+        const std::string name = source == "passthrough_left" ? left_node : source == "passthrough_right" ? right_node : "";
+        if (name.empty()) continue;
+        Camera cam;
+        cam.name = name;
+        cam.width = kValidWidth / scale, cam.height = c["height"].asInt() / scale;
+        const double dx = crop_subtract ? c["cropRegion"]["x"].asDouble() : 0, dy = crop_subtract ? c["cropRegion"]["y"].asDouble() : 0;
+        for (const Json::Value &in : c["intrinsics"]) {
+            if (in["cameraModel"].asString() != "kb") continue;
+            // integer pixel centres: sensor u -> image (u - crop + 0.5) / scale - 0.5
+            cam.fx = in["fx"].asDouble() / scale, cam.fy = in["fy"].asDouble() / scale;
+            cam.cx = (in["cx"].asDouble() - dx + 0.5) / scale - 0.5, cam.cy = (in["cy"].asDouble() - dy + 0.5) / scale - 0.5;
+            cam.k[0] = in["k1"].asDouble(), cam.k[1] = in["k2"].asDouble();
+            cam.k[2] = in["k3"].asDouble(), cam.k[3] = in["k4"].asDouble();
+        }
+        double cad_from_cam[4][4], head_from_cam[4][4];
+        pose(c["extrinsics"], 1e-3, cad_from_cam);
+        mul(head_from_cad, cad_from_cam, head_from_cam);
+        for (int i = 0; i < 3; ++i) {
+            for (int j = 0; j < 3; ++j) cam.R[i][j] = head_from_cam[i][j];
+            cam.origin[i] = head_from_cam[i][3];
+        }
+        out[name] = cam;
+        ++n;
+    }
+    if (n != 2) err = path + ": expected passthrough_left and passthrough_right";
+    return n == 2;
+}
+
+V3 triangulate(const V3 *origins, const V3 *dirs, const double *weights, int n, double *rms) {
+    double A[3][3] = {}, b[3] = {};
+    for (int v = 0; v < n; ++v) {
+        const V3 &d = dirs[v], &o = origins[v];
+        for (int i = 0; i < 3; ++i)
+            for (int j = 0; j < 3; ++j) {
+                const double P = (i == j ? 1.0 : 0.0) - d[i] * d[j];
+                A[i][j] += weights[v] * P;
+                b[i] += weights[v] * P * o[j];
+            }
+    }
+    // Cramer's rule for the 3x3 system
+    auto det3 = [](const double m[3][3]) {
+        return m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1]) - m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0]) +
+               m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0]);
+    };
+    const double D = det3(A);
+    V3 p{};
+    for (int c = 0; c < 3; ++c) {
+        double M[3][3];
+        for (int i = 0; i < 3; ++i)
+            for (int j = 0; j < 3; ++j) M[i][j] = j == c ? b[i] : A[i][j];
+        p[c] = std::fabs(D) > 1e-18 ? det3(M) / D : 0;
+    }
+    if (rms) {
+        double s = 0;
+        for (int v = 0; v < n; ++v) {
+            const V3 off = p - origins[v];
+            const V3 perp = off - dirs[v] * dot(off, dirs[v]);
+            s += dot(perp, perp);
+        }
+        *rms = std::sqrt(s / n);
+    }
+    return p;
+}

hands/track/calib.h

@@ -0,0 +1,39 @@
+// Tracking-camera calibration from the headset's factory files (see tools/calib.py
+// for the conventions): Kannala-Brandt fisheye intrinsics, and each camera's pose in the
+// head frame (OpenVR's: +x right, +y up, -z forward), metres.
+#pragma once
+
+#include "geom.h"
+
+#include <map>
+#include <string>
+#include <vector>
+
+struct Camera {
+    std::string name;
+    int width = 0, height = 0;
+    double fx = 1, fy = 1, cx = 0, cy = 0, k[4] = {};
+    double R[3][3] = {};   // camera axes (columns) in the head frame
+    V3 origin{};           // camera centre in the head frame
+
+    V2 project_cam(V3 p) const;               // camera frame -> pixels
+    V3 unproject(V2 uv) const;                // pixels -> unit ray, camera frame
+    V3 ray(V2 uv) const;                      // pixels -> unit ray, head frame
+    V2 project(V3 head, double *depth) const; // head frame -> pixels; depth along the optical axis
+    double off_axis(V2 uv) const;             // degrees between the pixel's ray and the axis
+};
+
+// Loads /persist/xrservice.json and /persist/device_config.json. Keyed by calibration
+// name: slam_left, slam_right, upper_left, upper_right.
+bool load_calibration(std::map<std::string, Camera> &out, std::string &err);
+
+// The Arcturus color cameras (tools/calib.py load_color has the conventions), for
+// ft-camd --with-color's images: luma at 1/scale size, recorded as color_video<N>. They're
+// keyed by those recorded names: left_node is passthrough_left, right_node
+// passthrough_right. crop_subtract: image x = sensor x - the calibration's cropRegion.x.
+// tools/check_color.py tells which node is which and which crop reading fits.
+bool load_color_calibration(std::map<std::string, Camera> &out, const std::string &left_node,
+                            const std::string &right_node, bool crop_subtract, int scale, std::string &err);
+
+// The point closest to several rays (weighted), and its rms distance to them.
+V3 triangulate(const V3 *origins, const V3 *dirs, const double *weights, int n, double *rms);

hands/track/geom.h

@@ -0,0 +1,22 @@
+// Small vector helpers for the tracker.
+#pragma once
+
+#include <array>
+#include <cmath>
+
+using V2 = std::array<double, 2>;
+using V3 = std::array<double, 3>;
+
+inline V3 operator+(V3 a, V3 b) { return {a[0] + b[0], a[1] + b[1], a[2] + b[2]}; }
+inline V3 operator-(V3 a, V3 b) { return {a[0] - b[0], a[1] - b[1], a[2] - b[2]}; }
+inline V3 operator*(V3 a, double s) { return {a[0] * s, a[1] * s, a[2] * s}; }
+inline double dot(V3 a, V3 b) { return a[0] * b[0] + a[1] * b[1] + a[2] * b[2]; }
+inline double norm(V3 a) { return std::sqrt(dot(a, a)); }
+inline V3 unit(V3 a) { double n = norm(a); return n > 0 ? a * (1 / n) : a; }
+
+inline V2 operator+(V2 a, V2 b) { return {a[0] + b[0], a[1] + b[1]}; }
+inline V2 operator-(V2 a, V2 b) { return {a[0] - b[0], a[1] - b[1]}; }
+inline V2 operator*(V2 a, double s) { return {a[0] * s, a[1] * s}; }
+inline double norm(V2 a) { return std::hypot(a[0], a[1]); }
+
+inline double wrap_angle(double a) { return std::remainder(a, 2 * M_PI); }

hands/track/io.cpp

@@ -0,0 +1,171 @@
+#include "io.h"
+
+#include <fcntl.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <time.h>
+#include <unistd.h>
+
+#include <algorithm>
+#include <cstdlib>
+#include <cstring>
+
+uint64_t mono_ns() {
+    timespec ts;
+    clock_gettime(CLOCK_MONOTONIC, &ts);
+    return uint64_t(ts.tv_sec) * 1'000'000'000 + uint64_t(ts.tv_nsec);
+}
+
+int64_t raw_minus_mono_ns() {
+    timespec a, r, b;
+    clock_gettime(CLOCK_MONOTONIC, &a);
+    clock_gettime(CLOCK_MONOTONIC_RAW, &r);
+    clock_gettime(CLOCK_MONOTONIC, &b);
+    const int64_t ma = int64_t(a.tv_sec) * 1'000'000'000 + a.tv_nsec, mb = int64_t(b.tv_sec) * 1'000'000'000 + b.tv_nsec;
+    return int64_t(r.tv_sec) * 1'000'000'000 + r.tv_nsec - (ma + mb) / 2;
+}
+
+// ------------------------------------------------------------------------------ ring
+
+bool Ring::open(const char *path, std::string &err) {
+    const int fd = ::open(path, O_RDONLY | O_CLOEXEC);
+    if (fd < 0) return err = std::string(path) + ": " + std::strerror(errno), false;
+    struct stat st;
+    fstat(fd, &st);
+    len_ = size_t(st.st_size);
+    void *m = len_ >= sizeof(fh_ring_hdr_t) ? mmap(nullptr, len_, PROT_READ, MAP_SHARED, fd, 0) : MAP_FAILED;
+    close(fd);
+    if (m == MAP_FAILED) return err = std::string(path) + ": can't map it", false;
+    map_ = static_cast<const uint8_t *>(m);
+    hdr_ = reinterpret_cast<const fh_ring_hdr_t *>(map_);
+    if (std::memcmp(hdr_->magic, FH_RING_MAGIC, 8) || hdr_->version != FH_RING_VERSION || hdr_->file_bytes > len_)
+        return err = std::string(path) + " is not an ft-camd ring", false;
+    return true;
+}
+
+bool Ring::alive() const {
+    const uint64_t hb = __atomic_load_n(&hdr_->heartbeat_ns, __ATOMIC_ACQUIRE);
+    return hb && mono_ns() - hb < 1'000'000'000;
+}
+
+uint64_t Ring::latest(int i) const { return __atomic_load_n(&hdr_->cams[i].latest, __ATOMIC_ACQUIRE); }
+
+bool Ring::read(int i, uint64_t n, std::vector<uint8_t> &out, fh_ring_slot_t *meta) const {
+    const fh_ring_cam_t &c = hdr_->cams[i];
+    if (!n || c.slot_offset + c.nslots * c.slot_bytes > len_) return false;
+    const uint8_t *slot = map_ + c.slot_offset + (n % c.nslots) * c.slot_bytes;
+    const auto *s = reinterpret_cast<const fh_ring_slot_t *>(slot);
+    const uint64_t seq = __atomic_load_n(&s->seq, __ATOMIC_ACQUIRE);
+    if (seq != 2 * n + 2) return false;
+    std::memcpy(meta, slot, sizeof *meta);
+    out.resize(size_t(c.width) * c.height);
+    for (uint32_t y = 0; y < c.height; ++y)
+        std::memcpy(out.data() + size_t(y) * c.width, slot + sizeof(fh_ring_slot_t) + size_t(y) * c.stride, c.width);
+    __atomic_thread_fence(__ATOMIC_ACQUIRE);
+    return __atomic_load_n(&s->seq, __ATOMIC_RELAXED) == seq;
+}
+
+bool Ring::meta(int i, uint64_t n, fh_ring_slot_t *meta) const {
+    const fh_ring_cam_t &c = hdr_->cams[i];
+    if (!n || c.slot_offset + c.nslots * c.slot_bytes > len_) return false;
+    const uint8_t *slot = map_ + c.slot_offset + (n % c.nslots) * c.slot_bytes;
+    const auto *s = reinterpret_cast<const fh_ring_slot_t *>(slot);
+    const uint64_t seq = __atomic_load_n(&s->seq, __ATOMIC_ACQUIRE);
+    if (seq != 2 * n + 2) return false;
+    std::memcpy(meta, slot, sizeof *meta);
+    __atomic_thread_fence(__ATOMIC_ACQUIRE);
+    return __atomic_load_n(&s->seq, __ATOMIC_RELAXED) == seq;
+}
+
+// ------------------------------------------------------------------------- publisher
+
+namespace {
+
+// The hand's shape to cut out, as capsules.
+// Radii are a real hand's half-widths plus a small margin for tracking noise.
+const int kThumb[][2] = {{0, 1}, {1, 2}, {2, 3}, {3, 4}};
+const int kFingers[][2] = {{5, 6}, {6, 7}, {7, 8}, {9, 10}, {10, 11}, {11, 12}, {13, 14}, {14, 15}, {15, 16},
+                           {17, 18}, {18, 19}, {19, 20}};
+const int kPalm[][2] = {{0, 5}, {0, 9}, {0, 13}, {0, 17}, {5, 9}, {9, 13}, {13, 17}, {1, 5}};
+constexpr double kThumbR = 0.0095, kFinger = 0.0085, kPalmR = 0.015, kArm[2] = {0.028, 0.034}, kArmLen = 0.16,
+                 kMargin = 0.004;
+// Nothing is cut closer than this in front of the eyes (head frame, -z is forward). A
+// point near the eyes' plane lands far across a screen with a huge radius, so one bad
+// estimate there tears a hole through it; real hands that close aren't tracked anyway.
+constexpr double kNear = 0.12;
+
+// Adds the capsule, clipped to the part at least kNear in front of the eyes.
+void put(fh_capsule_t *caps, uint32_t &n, V3 a, V3 b, double ra, double rb) {
+    if (n >= FH_HANDS_MAX_CAPSULES) return;
+    const double za = -a[2] - kNear, zb = -b[2] - kNear;   // >= 0: far enough in front
+    if (za < 0 && zb < 0) return;
+    if (za < 0 || zb < 0) {
+        const double t = za / (za - zb);   // where the segment crosses the near plane
+        const V3 m = a + (b - a) * t;
+        const double rm = ra + (rb - ra) * t;
+        if (za < 0) a = m, ra = rm;
+        else b = m, rb = rm;
+    }
+    fh_capsule_t &c = caps[n++];
+    for (int k = 0; k < 3; ++k) c.a[k] = float(a[k]), c.b[k] = float(b[k]);
+    c.ra = float(ra + kMargin), c.rb = float(rb + kMargin);
+}
+
+}  // namespace
+
+std::string run_dir() {
+    const std::string dir = "/run/user/" + std::to_string(getuid()) + "/frametop-hands";
+    mkdir(dir.c_str(), 0700);
+    return dir;
+}
+
+bool Publisher::open(std::string &err) {
+    const std::string path = run_dir() + "/hands";
+    const int fd = ::open(path.c_str(), O_RDWR | O_CREAT | O_NOFOLLOW | O_CLOEXEC, 0600);
+    if (fd < 0 || ftruncate(fd, sizeof(fh_hands_t)) < 0) return err = path + ": " + std::strerror(errno), false;
+    void *m = mmap(nullptr, sizeof(fh_hands_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+    close(fd);
+    if (m == MAP_FAILED) return err = path + ": can't map it", false;
+    out_ = static_cast<fh_hands_t *>(m);
+    std::memset(out_, 0, sizeof *out_);
+    std::memcpy(out_->magic, FH_HANDS_MAGIC, 8);
+    out_->version = FH_HANDS_VERSION;
+    out_->size = sizeof(fh_hands_t);
+    return true;
+}
+
+void Publisher::write(const std::vector<const Hand *> &in, uint64_t capture_ns) {
+    std::vector<const Hand *> hands = in;
+    std::sort(hands.begin(), hands.end(), [](const Hand *a, const Hand *b) { return a->frames > b->frames; });
+    if (hands.size() > FH_HANDS_MAX_HANDS) hands.resize(FH_HANDS_MAX_HANDS);
+    __atomic_store_n(&out_->seq, 2 * ++seq_ - 1, __ATOMIC_RELAXED);
+    __atomic_thread_fence(__ATOMIC_RELEASE);
+    uint32_t nc = 0;
+    for (size_t k = 0; k < FH_HANDS_MAX_HANDS; ++k) {
+        fh_hand_t &o = out_->hands[k];
+        std::memset(&o, 0, sizeof o);
+        if (k >= hands.size()) continue;
+        const Hand &h = *hands[k];
+        o.id = uint32_t(h.id);
+        o.flags = (h.right() ? FH_HAND_RIGHT : 0) | (h.nviews >= 2 ? FH_HAND_STEREO : 0);
+        o.confidence = float(std::min(1.0, h.frames / 5.0));
+        for (int i = 0; i < 21; ++i)
+            for (int j = 0; j < 3; ++j) o.pts[i][j] = float(h.smooth[i][j]);
+        const uint32_t first = nc;
+        for (auto &b : kThumb) put(out_->capsules, nc, h.smooth[b[0]], h.smooth[b[1]], kThumbR, kThumbR);
+        for (auto &b : kFingers) put(out_->capsules, nc, h.smooth[b[0]], h.smooth[b[1]], kFinger, kFinger);
+        for (auto &b : kPalm) put(out_->capsules, nc, h.smooth[b[0]], h.smooth[b[1]], kPalmR, kPalmR);
+        // the forearm carries on from the hand's own axis (middle knuckle -> wrist); the
+        // wrist bends, but much less than a guess at where the elbow is gets wrong
+        const V3 wrist = h.smooth[0], d = wrist - h.smooth[9];
+        const double n = norm(d);
+        if (n > 0.02) put(out_->capsules, nc, wrist, wrist + d * (kArmLen / n), kArm[0], kArm[1]);
+        o.ncapsules = nc - first;
+    }
+    for (uint32_t k = nc; k < FH_HANDS_MAX_CAPSULES; ++k) std::memset(&out_->capsules[k], 0, sizeof(fh_capsule_t));
+    out_->capture_ns = capture_ns;
+    out_->publish_ns = mono_ns();
+    out_->nhands = uint32_t(hands.size());
+    out_->ncapsules = nc;
+    __atomic_store_n(&out_->seq, 2 * seq_, __ATOMIC_RELEASE);
+}

hands/track/io.h

@@ -0,0 +1,52 @@
+// Frames in from ft-camd's ring (camd/fhring.h), hands out to the hands file
+// (include/fh_hands.h, read by Frametop's ft-screens).
+#pragma once
+
+#include "tracker.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+extern "C" {
+#include "../camd/fhring.h"
+#include "../include/fh_hands.h"
+}
+
+class Ring {
+public:
+    bool open(const char *path, std::string &err);
+    bool alive() const;                      // the writer's heartbeat is fresh
+    int cameras() const { return int(hdr_->ncams); }
+    const fh_ring_cam_t &camera(int i) const { return hdr_->cams[i]; }
+    uint64_t latest(int i) const;
+    // Copy frame n of camera i into out (width x height, tightly packed). False if it's
+    // gone or was being written.
+    bool read(int i, uint64_t n, std::vector<uint8_t> &out, fh_ring_slot_t *meta) const;
+    // Just frame n's slot header (capture time etc.), without copying the image.
+    bool meta(int i, uint64_t n, fh_ring_slot_t *meta) const;
+
+private:
+    const uint8_t *map_ = nullptr;
+    const fh_ring_hdr_t *hdr_ = nullptr;
+    size_t len_ = 0;
+};
+
+class Publisher {
+public:
+    bool open(std::string &err);
+    void write(const std::vector<const Hand *> &hands, uint64_t capture_ns);
+
+private:
+    fh_hands_t *out_ = nullptr;
+    uint64_t seq_ = 0;
+};
+
+uint64_t mono_ns();
+int64_t raw_minus_mono_ns();   // camera timestamps are CLOCK_MONOTONIC_RAW
+
+// /run/user/UID/frametop-hands, created private to the user if it's missing: where ft-camd's ring
+// (FH_RING_NAME) and the hands and gestures files live. Not $XDG_RUNTIME_DIR: a terminal in
+// the Frametop desktop has a private one of its own. And not /run/user/UID/frametop: that is
+// the desktop session's private runtime folder, which it deletes whenever it starts.
+std::string run_dir();

hands/track/main.cpp

@@ -0,0 +1,638 @@
+// ft-hands: hands in 3D from ft-camd's ring, published for Frametop's ft-screens (the hand
+// cutouts), and pinches and grips for the pointer. It started as a port of frame-hands'
+// Python prototype: the same scheduling, with the models on a few threads.
+//
+//   ft-hands [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N]
+//            [--no-publish] [--no-gestures] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
+//            (--help lists them all)
+//
+// --no-gestures: hands for the cutouts only. No pinch or grip detection, so nothing reaches
+// the pointer and a closing hand doesn't raise the rate; the gestures file is removed.
+//
+// Which cameras (--cams, HANDS_CAMERAS): the four mono IR cameras light the hands with their
+// own IR and track well in dim rooms, but in bright light (a sunny room, a window behind the
+// hands) they expose for the room and the hands come out dark. The two Arcturus colour
+// cameras (the passthrough pair, forward-facing, 145 degrees) are the other way round: dark
+// and grainy in a dim room, clear in a lit one. auto (the default) picks by how bright the
+// colour cameras' frames are: at HANDS_BRIGHT_ON (mean luma) or over for 2 s, it tracks with
+// HANDS_BRIGHT (all: every camera, so hands low at the sides stay in the side cameras; or
+// color); under HANDS_BRIGHT_OFF for 2 s, with the mono cameras again. ft-camd runs the colour
+// cameras at 2 fps, enough to tell the light, until ft-hands asks for 30
+// (/run/user/UID/frametop-hands/color-fps). The colour frames' capture times are on their
+// own clock, so they're placed on the mono cameras' by when they were dequeued, less the
+// mono cameras' measured delay.
+//
+// Settings in ~/.config/frametop.conf (FT_<name> in the environment overrides them, and
+// options override both): HANDS_SWAP_SIDES (1: as --swap-sides), HANDS_CPUS (as --cpus),
+// HANDS_CAMERAS, HANDS_BRIGHT, HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT (which
+// colour camera is passthrough_left: color_video0 or color_video3), HANDS_COLOR_CROP
+// (subtract or none: tools/check_color.py tells both).
+#include "io.h"
+#include "pinch.h"
+#include "record.h"
+
+#include <sched.h>
+#include <sys/resource.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include <cmath>
+#include <ctime>
+#include <memory>
+
+#include <csignal>
+#include <cstdlib>
+#include <cstdio>
+#include <cstring>
+#include <fstream>
+#include <thread>
+#include <utility>
+
+namespace {
+
+volatile std::sig_atomic_t g_stop = 0, g_record = 0;
+
+// which calibrated camera each capture pipe carries (XRService's fixed routing)
+const char *camera_for_pipe(int node) {
+    char path[64], name[64] = "";
+    std::snprintf(path, sizeof path, "/sys/class/video4linux/video%d/name", node);
+    std::ifstream f(path);
+    f.getline(name, sizeof name);
+    if (!std::strcmp(name, "msm_vfe3_video0")) return "slam_left";
+    if (!std::strcmp(name, "msm_vfe4_video0")) return "slam_right";
+    if (!std::strcmp(name, "msm_vfe2_video0")) return "upper_left";
+    if (!std::strcmp(name, "msm_vfe2_video1")) return "upper_right";
+    return nullptr;
+}
+
+// A setting from ~/.config/frametop.conf, or FT_<key> from the environment; "" if unset.
+std::string setting(const std::string &key) {
+    if (const char *v = std::getenv(("FT_" + key).c_str())) return v;
+    const char *home = std::getenv("HOME");
+    std::ifstream in(std::string(home ? home : "") + "/.config/frametop.conf");
+    std::string line, value;
+    auto trim = [](std::string s) {
+        s.erase(0, s.find_first_not_of(" \t\"'"));
+        s.erase(s.find_last_not_of(" \t\"'") + 1);
+        return s;
+    };
+    while (std::getline(in, line)) {
+        line = line.substr(0, line.find('#'));
+        const auto eq = line.find('=');
+        if (eq != std::string::npos && trim(line.substr(0, eq)) == key) value = trim(line.substr(eq + 1));
+    }
+    return value;
+}
+
+std::vector<int> parse_cpus(const char *p) {
+    std::vector<int> out;
+    while (*p) {
+        char *end;
+        const long c = std::strtol(p, &end, 10);
+        if (end == p) break;
+        out.push_back(int(c));
+        p = *end == ',' ? end + 1 : end;
+    }
+    return out;
+}
+
+// Where SIGUSR1 puts recordings: $XDG_DATA_HOME/frametop/hands (~/.local/share/...).
+std::string recordings_dir() {
+    const char *data = std::getenv("XDG_DATA_HOME"), *home = std::getenv("HOME");
+    std::string dir = data && *data ? data : std::string(home ? home : "") + "/.local/share";
+    for (const char *part : {"/frametop", "/hands"}) mkdir((dir += part).c_str(), 0700);
+    return dir;
+}
+
+double cpu_seconds() {
+    rusage r;
+    getrusage(RUSAGE_SELF, &r);
+    return r.ru_utime.tv_sec + r.ru_stime.tv_sec + (r.ru_utime.tv_usec + r.ru_stime.tv_usec) / 1e6;
+}
+
+enum class Cams { Mono, Color, All };
+
+const char *cams_name(Cams c) { return c == Cams::Mono ? "mono" : c == Cams::Color ? "color" : "all"; }
+
+bool parse_cams(const std::string &s, Cams &out) {
+    if (s == "mono") out = Cams::Mono;
+    else if (s == "color") out = Cams::Color;
+    else if (s == "all") out = Cams::All;
+    else return false;
+    return true;
+}
+
+// How bright it is, for auto (see the top): the colour frames' mean luma, smoothed over about
+// a second, with hysteresis and a 2 s hold each way. No colour frames for 3 s (ft-camd paused
+// them, or has none) reads as dim.
+struct Lighting {
+    double on = 40, off = 25;
+    double level = -1;
+    bool bright = false;
+    uint64_t at_ns = 0, since_ns = 0;   // the last frame; since when it's wanted the other way
+
+    void add(double mean, uint64_t t_ns) {
+        const double dt = at_ns && t_ns > at_ns ? (t_ns - at_ns) / 1e9 : 1.0;
+        level = level < 0 ? mean : level + (mean - level) * std::min(1.0, dt / 1.0);
+        at_ns = t_ns;
+    }
+    // True when it switched.
+    bool update(uint64_t now_ns) {
+        if (level >= 0 && now_ns - at_ns > 3'000'000'000ull) level = -1;
+        const bool want = level >= 0 && (bright ? level > off : level >= on);
+        if (want == bright) return since_ns = 0, false;
+        if (!since_ns) since_ns = now_ns;
+        if (now_ns - since_ns < 2'000'000'000ull) return false;
+        bright = want, since_ns = 0;
+        return true;
+    }
+};
+
+}  // namespace
+
+int main(int argc, char **argv) {
+    double seconds = 0, status = 5;
+    int threads = 3, niceness = 5;
+    bool int8 = false, publish = true, track = true, swap_sides = false, gestures_on = true;
+    std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
+    std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME;
+    // SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for
+    // SteamVR's compositor, whose threads there run at real-time priority, so they always
+    // win. XRService pins its head tracking to 2-3. frame-hands' probes/core_ab.py
+    // (2026-09-29, headset on, 3 rounds): on 5-7 a step took 8.4 ms against 13.2 on 2-4,
+    // latency 9.6 against 14.1 ms, and the compositor's late frames and CPU/GPU time didn't change.
+    std::vector<int> cpus = {5, 6, 7};
+    if (const auto c = parse_cpus(setting("HANDS_CPUS").c_str()); !c.empty()) cpus = c;
+    swap_sides = setting("HANDS_SWAP_SIDES") == "1";
+    // Which cameras (see the top).
+    std::string cams_arg = setting("HANDS_CAMERAS"), bright_arg = setting("HANDS_BRIGHT");
+    std::string color_left = setting("HANDS_COLOR_LEFT"), color_crop = setting("HANDS_COLOR_CROP");
+    if (cams_arg.empty()) cams_arg = "auto";
+    if (bright_arg.empty()) bright_arg = "all";
+    if (color_left.empty()) color_left = "color_video0";
+    if (color_crop.empty()) color_crop = "subtract";
+    Lighting light;
+    if (const std::string v = setting("HANDS_BRIGHT_ON"); !v.empty()) light.on = std::atof(v.c_str());
+    if (const std::string v = setting("HANDS_BRIGHT_OFF"); !v.empty()) light.off = std::atof(v.c_str());
+    // How crops are equalized. CLAHE helps the palm search find hands (about 10% more in the
+    // dim recording), but makes the landmarks jitter, so they get plain crops.
+    Contrast palm_contrast, hand_contrast{Contrast::None};
+    double keep_presence = 0.5;   // landmark presence a tracked view needs to stay
+    PinchParams pinch_params;
+    GripParams grip_params;
+    bool gesture_log = false;   // what the pinch and grip detectors measure, 10 times a second
+    double record_for = 120;
+    for (int i = 1; i < argc; ++i) {
+        const std::string a = argv[i];
+        const bool more = i + 1 < argc;
+        if (a == "--seconds" && more) seconds = std::atof(argv[++i]);
+        else if (a == "--threads" && more) threads = std::max(1, std::atoi(argv[++i]));
+        else if (a == "--status" && more) status = std::atof(argv[++i]);
+        else if (a == "--models" && more) models = argv[++i];
+        else if (a == "--nice" && more) niceness = std::atoi(argv[++i]);
+        else if (a == "--int8") int8 = true;
+        else if (a == "--no-publish") publish = false;
+        else if (a == "--no-gestures") gestures_on = false;
+        else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]);
+        else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
+        else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
+        else if (a == "--pinch-palm-down" && more) pinch_params.palm_down_max = std::atof(argv[++i]);
+        else if (a == "--grip-begin" && more) grip_params.begin = std::atof(argv[++i]);
+        else if (a == "--grip-end" && more) grip_params.end = std::atof(argv[++i]);
+        else if (a == "--gesture-log") gesture_log = true;
+        else if (a == "--swap-sides") swap_sides = true;
+        else if (a == "--record-only") track = publish = false;
+        else if (a == "--ring" && more) ring_path = argv[++i];
+        else if (a == "--record" && more) record = argv[++i];
+        else if (a == "--record-for" && more) record_for = std::atof(argv[++i]);
+        else if (a == "--keep-presence" && more) keep_presence = std::atof(argv[++i]);
+        else if (a == "--cams" && more) cams_arg = argv[++i];
+        else if (a == "--bright" && more) bright_arg = argv[++i];
+        else if (a == "--bright-on" && more) light.on = std::atof(argv[++i]);
+        else if (a == "--bright-off" && more) light.off = std::atof(argv[++i]);
+        else if (a == "--color-left" && more) color_left = argv[++i];
+        else if (a == "--color-crop" && more) color_crop = argv[++i];
+        else if (a == "--contrast" && more) {
+            if (!Contrast::parse_pair(argv[++i], palm_contrast, hand_contrast))
+                return std::fprintf(stderr, "--contrast MODE or PALM/HAND, each clahe[:CLIP]|none|stretch\n"), 1;
+        } else if (a == "--cpus" && more) {
+            cpus = parse_cpus(argv[++i]);
+            if (cpus.empty()) cpus = {5, 6, 7};
+        }
+        else {
+            std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n"
+                        "          [--no-gestures] (hands for the cutouts only: no pinches or grips)\n"
+                        "          [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n"
+                        "          [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
+                        "          [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
+                        "          [--color-left color_video0|color_video3] [--color-crop subtract|none]\n"
+                        "          [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (1: off)\n"
+                        "          [--grip-begin R] (1.2) [--grip-end R] (1.45) [--gesture-log]\n"
+                        "          [--contrast MODE|PALM/HAND] (clahe[:CLIP], none, stretch; default clahe:2/none)\n"
+                        "Recording saves every frame set for S seconds (120) to DIR/sets.bin, for ft-handreplay; SIGUSR1\n"
+                        "starts one in ~/.local/share/frametop/hands/rec-<time>. --record-only records without tracking, so it\n"
+                        "can run beside a tracking ft-hands. With ft-camd --with-dark, recordings also get each\n"
+                        "camera's newest dark frame, as <name>_dk; with --with-color, the color cameras' as color_video<N>.\n"
+                        "auto picks the cameras by the light (see the top of track/main.cpp).\n"
+                        "Settings in ~/.config/frametop.conf: HANDS_SWAP_SIDES=1, HANDS_CPUS=5,6,7, HANDS_CAMERAS, HANDS_BRIGHT,\n"
+                        "HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT, HANDS_COLOR_CROP (FT_<name> overrides).\n",
+                        argv[0]);
+            return a == "--help" ? 0 : 1;
+        }
+    }
+    const bool automatic = cams_arg == "auto";
+    Cams fixed = Cams::Mono, bright_cams = Cams::All;
+    if ((!automatic && !parse_cams(cams_arg, fixed)) || !parse_cams(bright_arg, bright_cams) || bright_cams == Cams::Mono)
+        return std::fprintf(stderr, "--cams auto|mono|color|all, --bright all|color\n"), 1;
+    if (color_crop != "subtract" && color_crop != "none") return std::fprintf(stderr, "--color-crop subtract|none\n"), 1;
+    std::setvbuf(stdout, nullptr, _IOLBF, 0);   // whole lines to the journal as they come
+    if (nice(niceness) < 0) std::perror("nice");   // the VR stack wins contested CPUs
+    std::signal(SIGINT, [](int) { g_stop = 1; });
+    std::signal(SIGTERM, [](int) { g_stop = 1; });
+    std::signal(SIGUSR1, [](int) { g_record = 1; });
+
+    std::string err;
+    std::map<std::string, Camera> calib;
+    Ring ring;
+    Nets nets;
+    Publisher pub;
+    GesturePublisher gestures;
+    Pinch pinch(pinch_params);
+    Grip grip(grip_params);
+    std::unique_ptr<Recorder> rec;
+    uint64_t rec_start = 0;
+    auto start_recording = [&](const std::string &dir, std::string &e) {
+        rec = std::make_unique<Recorder>();
+        if (!rec->open(dir, e)) return rec.reset(), false;
+        rec_start = mono_ns();
+        std::printf("recording to %s for %.0f s\n", dir.c_str(), record_for);
+        std::fflush(stdout);
+        return true;
+    };
+    if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
+        (publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
+        (!record.empty() && !start_recording(record, err))) {
+        std::fprintf(stderr, "%s\n", err.c_str());
+        return 1;
+    }
+    if (!ring.alive()) return std::fprintf(stderr, "ft-camd isn't running (no heartbeat)\n"), 1;
+    // so a reader can't take an earlier run's file for this one's
+    if (publish && !gestures_on) unlink((run_dir() + "/gestures").c_str());
+
+    std::map<std::string, int> index;   // mono calibration name -> ring camera
+    std::map<std::string, int> color;   // colour calibration name (color_video<N>) -> ring camera
+    // Recorded as they are with each set: "<name>_dk" (ft-camd --with-dark) and "color_video<N>"
+    // (--with-color). Recorded names hold 15 characters, so "upper_right_dark" wouldn't fit.
+    std::map<std::string, int> dark;
+    std::map<std::string, Camera> used;
+    for (int i = 0; i < ring.cameras(); ++i) {
+        if (ring.camera(i).flags & FH_CAM_COLOR) {
+            const std::string name = "color_video" + std::to_string(ring.camera(i).node);
+            dark[name] = i, color[name] = i;
+            continue;
+        }
+        // ft-camd's cameras by capture pipe; ft-ringplay's (no device) by the name it gives
+        const char *name = camera_for_pipe(ring.camera(i).node);
+        if (!name && ring.camera(i).node < 0) name = ring.camera(i).name;
+        if (!name || !calib.count(name)) continue;
+        if (ring.camera(i).flags & FH_CAM_DARK) dark[std::string(name) + "_dk"] = i;
+        else index[name] = i, used[name] = calib[name];
+    }
+    // ft-camd tells the side cameras' buffers apart by XRService's allocation order, which
+    // some XRService restarts reverse; tools/check_sides.py --ring tells when.
+    if (swap_sides && index.count("slam_left") && index.count("slam_right")) {
+        std::swap(index["slam_left"], index["slam_right"]);
+        if (dark.count("slam_left_dk") && dark.count("slam_right_dk")) std::swap(dark["slam_left_dk"], dark["slam_right_dk"]);
+        std::printf("side cameras swapped (--swap-sides)\n");
+    }
+    // The colour pair, calibrated (see the top), unless only the mono cameras are wanted.
+    if (color.size() == 2 && (automatic || fixed != Cams::Mono)) {
+        const std::string left = color.count(color_left) ? color_left : color.begin()->first;
+        const std::string right = color.begin()->first == left ? std::next(color.begin())->first : color.begin()->first;
+        const int scale = int(std::lround(1972.0 / ring.camera(color[left]).width));
+        std::map<std::string, Camera> cc;
+        std::string e;
+        if (load_color_calibration(cc, left, right, color_crop == "subtract", scale, e)) {
+            for (auto &[name, cam] : cc) used[name] = cam;
+            std::printf("colour cameras: %s is passthrough_left, crop %s, 1/%d size\n", left.c_str(), color_crop.c_str(), scale);
+        } else {
+            std::fprintf(stderr, "colour cameras left out: %s\n", e.c_str());
+            color.clear();
+        }
+    } else {
+        color.clear();
+    }
+    if (color.empty() && (automatic || fixed != Cams::Mono)) {
+        if (!automatic) std::printf("no colour cameras (ft-camd --with-color): tracking with the mono ones\n");
+        fixed = Cams::Mono;
+    }
+    const bool switching = automatic && !color.empty();
+    Cams mode = switching || color.empty() ? Cams::Mono : fixed;
+    std::printf("cameras:");
+    for (auto &[name, i] : index) std::printf(" %s=video%d", name.c_str(), ring.camera(i).node);
+    for (auto &[name, i] : color) std::printf(" %s", name.c_str());
+    std::printf("  tracking with %s%s  models: %s%s, %d threads on CPUs", cams_name(mode),
+                switching ? " (auto: by the light)" : "", models.c_str(), int8 ? " (int8)" : "", threads);
+    for (int c : cpus) std::printf(" %d", c);
+    std::printf("\n");
+
+    cpu_set_t set;   // the main loop too
+    CPU_ZERO(&set);
+    for (int c : cpus) CPU_SET(c, &set);
+    if (sched_setaffinity(0, sizeof set, &set) < 0) std::perror("sched_setaffinity");
+    nets.set_contrast(palm_contrast, hand_contrast);
+    Pool pool(threads, cpus);
+    Tracker tracker(used, nets, pool);
+    tracker.set_keep_presence(keep_presence);
+    std::map<std::string, std::vector<uint8_t>> pixels;
+    std::map<std::string, uint64_t> last;       // per camera: the frame last used
+    std::map<std::string, uint64_t> lit_seen;   // per colour camera: the frame last counted for the light
+    const uint64_t start = mono_ns();
+    uint64_t t_status = start, next_ns = 0, t_want = 0, t_glog = 0;
+    double cpu0 = cpu_seconds();
+    std::vector<double> lat;
+    double hands_sum = 0, resid_sum = 0;
+    int resid_n = 0, left_sets = 0, right_sets = 0, both_sets = 0, color_steps = 0;
+    // The mono cameras' delay from capture to dequeue (their capture clock is CLOCK_MONOTONIC_RAW),
+    // to place the colour frames, whose capture clock is their own (see the top).
+    double mono_delay_ns = -1;
+    const std::string want_file = run_dir() + "/color-fps";
+
+    // The colour pair's newest frames if both are newer than last used and taken together
+    // (their own clock): their time, on the mono cameras' capture clock, else 0.
+    auto color_pair = [&](int64_t raw_off, bool copy, std::map<std::string, Image> &images) -> uint64_t {
+        fh_ring_slot_t meta[2];
+        std::string names[2];
+        uint64_t n[2];
+        int k = 0;
+        for (auto &[name, i] : color) {
+            names[k] = name, n[k] = ring.latest(i);
+            if (n[k] <= last[name] || !ring.meta(i, n[k], &meta[k])) return 0;
+            ++k;
+        }
+        if (k != 2 || mono_delay_ns < 0) return 0;
+        const int64_t apart = int64_t(meta[0].capture_ns) - int64_t(meta[1].capture_ns);
+        if (std::llabs(apart) > 3'000'000) return 0;   // one is a frame ahead: wait for the other
+        const uint64_t dq = std::min(meta[0].dqbuf_ns, meta[1].dqbuf_ns);
+        const uint64_t t = uint64_t(int64_t(dq) - int64_t(mono_delay_ns) + raw_off);
+        if (!copy) return t;
+        for (int j = 0; j < 2; ++j) {
+            const int i = color[names[j]];
+            if (!ring.read(i, n[j], pixels[names[j]], &meta[j])) return 0;
+            const auto &c = ring.camera(i);
+            images[names[j]] = {pixels[names[j]].data(), int(c.width), int(c.height), int(c.width)};
+        }
+        for (int j = 0; j < 2; ++j) last[names[j]] = n[j];
+        return t;
+    };
+    auto switch_to = [&](Cams to, const char *why) {
+        if (to == mode) return;
+        for (auto &[name, cam] : used) {
+            const bool is_color = color.count(name) > 0;
+            const bool keep = to == Cams::All || (to == Cams::Color) == is_color;
+            if (!keep) tracker.drop_camera(name);
+        }
+        std::printf("cameras: %s -> %s (%s)\n", cams_name(mode), cams_name(to), why);
+        mode = to;
+    };
+    auto ambient = [&] {   // the mono cameras' dark frames: the room's IR light
+        double sum = 0;
+        int n = 0;
+        for (auto &[name, i] : index)
+            if (ring.camera(i).dark_mean > 0) sum += ring.camera(i).dark_mean, ++n;
+        return n ? sum / n : -1;
+    };
+
+    while (!g_stop && (seconds <= 0 || (mono_ns() - start) / 1e9 < seconds)) {
+        if (!ring.alive()) return std::fprintf(stderr, "ft-camd stopped\n"), 2;
+        const uint64_t now0 = mono_ns();
+        const int64_t raw_off = raw_minus_mono_ns();
+
+        // The light, from the colour frames' brightness (their slot headers only).
+        for (auto &[name, i] : color) {
+            fh_ring_slot_t m;
+            const uint64_t n = ring.latest(i);
+            if (n && n != lit_seen[name] && ring.meta(i, n, &m)) light.add(m.mean, m.dqbuf_ns), lit_seen[name] = n;
+        }
+        if (switching && light.update(now0)) {
+            char why[96];
+            std::snprintf(why, sizeof why, "colour frames at %.0f, ambient IR %.1f", light.level, ambient());
+            switch_to(light.bright ? bright_cams : Cams::Mono, why);
+        }
+        // Ask ft-camd for the colour cameras' full rate while tracking or recording with them.
+        const bool want_color = !color.empty() && (mode != Cams::Mono || rec || g_record);
+        if (!color.empty() && now0 - t_want > 1'000'000'000) {
+            t_want = now0;
+            if (want_color) {
+                if (FILE *f = std::fopen(want_file.c_str(), "w")) std::fputs("30\n", f), std::fclose(f);
+            } else {
+                unlink(want_file.c_str());
+            }
+        }
+
+        const bool use_mono = mode != Cams::Color, use_color = mode != Cams::Mono;
+        const bool mono_driven = use_mono || rec || g_record || !track;
+        std::map<std::string, Image> images;
+        uint64_t tmin = UINT64_MAX, dq = 0;
+
+        if (mono_driven) {
+            // a new frame set: every mono camera has a newer frame, taken at the same moment
+            std::map<std::string, uint64_t> latest;
+            bool ready = true;
+            for (auto &[name, i] : index) {
+                latest[name] = ring.latest(i);
+                ready = ready && latest[name] > last[name];
+            }
+            if (!ready) {
+                std::this_thread::sleep_for(std::chrono::milliseconds(2));
+                continue;
+            }
+            // not needed at the current rate, and not recorded: skip it without copying images
+            if (track && !rec && !g_record) {
+                uint64_t t0 = UINT64_MAX, t1 = 0;
+                bool ok = true;
+                for (auto &[name, i] : index) {
+                    fh_ring_slot_t meta;
+                    ok = ok && ring.meta(i, latest[name], &meta);
+                    if (ok) t0 = std::min(t0, meta.capture_ns), t1 = std::max(t1, meta.capture_ns);
+                }
+                if (ok && t1 - t0 <= 3'000'000 && t0 < next_ns) {
+                    for (auto &[name, i] : index) last[name] = latest[name];
+                    continue;
+                }
+            }
+            std::vector<SetFrame> frames;
+            uint64_t tmax = 0;
+            bool ok = true;
+            for (auto &[name, i] : index) {
+                fh_ring_slot_t meta;
+                ok = ok && ring.read(i, latest[name], pixels[name], &meta);
+                if (!ok) break;
+                const auto &c = ring.camera(i);
+                images[name] = {pixels[name].data(), int(c.width), int(c.height), int(c.width)};
+                frames.push_back({name, pixels[name].data(), c.width, c.height, meta.capture_ns, meta.dqbuf_ns});
+                tmin = std::min(tmin, meta.capture_ns), tmax = std::max(tmax, meta.capture_ns), dq = std::max(dq, meta.dqbuf_ns);
+                const double delay = double(int64_t(meta.dqbuf_ns) - (int64_t(meta.capture_ns) - raw_off));
+                mono_delay_ns = mono_delay_ns < 0 ? delay : mono_delay_ns + 0.02 * (delay - mono_delay_ns);
+            }
+            if (!ok || tmax - tmin > 3'000'000) {   // torn, or a camera is a frame behind
+                std::this_thread::sleep_for(std::chrono::milliseconds(1));
+                continue;
+            }
+            for (auto &[name, i] : index) last[name] = latest[name];
+            if (g_record && !rec) {
+                g_record = 0;
+                char name[64];
+                const std::time_t now = std::time(nullptr);
+                std::strftime(name, sizeof name, "rec-%Y%m%d-%H%M%S", std::localtime(&now));
+                const std::string dir = recordings_dir();
+                std::string e;
+                if (!start_recording(dir + "/" + name, e)) std::fprintf(stderr, "%s\n", e.c_str());
+            }
+            if (rec) {   // about 80 MB/s; dark frames double that, color frames add 70 MB/s
+                if ((mono_ns() - rec_start) / 1e9 < record_for) {
+                    for (auto &[name, i] : dark) {   // the newest dark and color frames, as they are
+                        fh_ring_slot_t meta;
+                        const uint64_t n = ring.latest(i);
+                        const auto &c = ring.camera(i);
+                        if (n && ring.read(i, n, pixels[name + "#rec"], &meta))
+                            frames.push_back({name, pixels[name + "#rec"].data(), c.width, c.height, meta.capture_ns,
+                                              meta.dqbuf_ns});
+                    }
+                    rec->add(frames);
+                } else {
+                    const size_t n = rec->written(), d = rec->dropped();
+                    rec.reset();   // writes out what's queued
+                    std::printf("recording done: %zu sets, %zu dropped\n", n, d);
+                    std::fflush(stdout);
+                    if (!track) break;
+                }
+            }
+            if (!track && rec && status > 0 && (mono_ns() - t_status) / 1e9 >= status) {
+                std::printf("%5.1fs  recorded %zu sets, dropped %zu\n", (mono_ns() - start) / 1e9, rec->written(), rec->dropped());
+                std::fflush(stdout);
+                t_status = mono_ns();
+            }
+            if (!track || tmin < next_ns) continue;   // not needed yet at the current rate
+            if (!use_mono) images.clear();            // colour only, driven by mono while recording
+            if (use_color && color_pair(raw_off, true, images)) ++color_steps;
+            if (images.empty()) continue;
+        } else {
+            // colour only: a new pair of colour frames
+            for (auto &[name, i] : index) {   // keep the mono cameras' delay current
+                fh_ring_slot_t meta;
+                const uint64_t n = ring.latest(i);
+                if (n && ring.meta(i, n, &meta)) {
+                    const double delay = double(int64_t(meta.dqbuf_ns) - (int64_t(meta.capture_ns) - raw_off));
+                    mono_delay_ns = mono_delay_ns < 0 ? delay : mono_delay_ns + 0.02 * (delay - mono_delay_ns);
+                }
+                break;
+            }
+            const uint64_t t = color_pair(raw_off, false, images);
+            if (!t) {
+                std::this_thread::sleep_for(std::chrono::milliseconds(2));
+                continue;
+            }
+            if (t < next_ns) {   // not needed yet: pass it by without copying
+                for (auto &[name, i] : color) last[name] = ring.latest(i);
+                continue;
+            }
+            if (!color_pair(raw_off, true, images)) continue;
+            tmin = t, ++color_steps;
+            for (auto &[name, i] : color) {
+                fh_ring_slot_t meta;
+                if (ring.meta(i, last[name], &meta)) dq = std::max(dq, meta.dqbuf_ns);
+            }
+        }
+
+        const auto hands = tracker.step(images, int64_t(tmin));
+        const uint64_t capture = uint64_t(int64_t(tmin) - raw_off);   // CLOCK_MONOTONIC
+        const std::vector<Seen> views = tracker.views_now();
+        if (gestures_on) {
+            grip.update(hands, views, int64_t(capture));
+            pinch.update(hands, views, int64_t(capture), grip.gripping());
+        }
+        if (gestures_on && gesture_log && capture - t_glog >= 100'000'000) {
+            t_glog = capture;
+            for (int k = 0; k < 2; ++k)
+                if (pinch.world_d[k] >= 0)
+                    std::printf("gesture %s d world %.3f tri %.3f palm-down %.2f curl %.2f%s\n", k ? "right" : "left ",
+                                pinch.world_d[k], pinch.tri_d[k], pinch.palm_down[k], grip.curl[k],
+                                pinch.side(k).flags & FH_PINCH_DOWN ? " PINCH" : grip.side(k).flags & FH_PINCH_DOWN ? " GRIP" : "");
+        }
+        // a gesture down or closing gets the full rate, even while the palm holds still
+        next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) -
+                                   0.005) * 1e9);
+        if (publish) pub.write(hands, capture);
+        if (publish && gestures_on) gestures.write(pinch, grip, capture);
+        for (const Pinch::Event &e : grip.events)
+            std::printf("grip  %s %-5s curl %.2f at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
+                        e.point[0], e.point[1], e.point[2]);
+        for (const Pinch::Event &e : pinch.events)
+            std::printf("pinch %s %-5s d %.3f m at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
+                        e.point[0], e.point[1], e.point[2]);
+        lat.push_back((mono_ns() - dq) / 1e6);
+        hands_sum += double(hands.size());
+        bool on_left = false, on_right = false;   // by where the wrist is, not the model's label
+        for (const Hand *h : hands) {
+            if (h->residual >= 0) resid_sum += h->residual * 1000, ++resid_n;
+            (h->pts[0][0] < 0 ? on_left : on_right) = true;
+        }
+        left_sets += on_left, right_sets += on_right, both_sets += on_left && on_right;
+
+        const uint64_t now = mono_ns();
+        if (status > 0 && (now - t_status) / 1e9 >= status) {
+            const double dt = (now - t_status) / 1e9, cpu1 = cpu_seconds();
+            const Stats &s = tracker.stats;
+            std::sort(lat.begin(), lat.end());
+            std::printf("%5.1fs %4.1f sets/s  hands %.2f views %zu  palm %3d calls %4.1f ms/batch  hand %3d calls %4.1f ms/batch  "
+                        "step %4.1f ms  latency %4.1f ms  resid %.1f mm  CPU %3.0f%%\n",
+                        (now - start) / 1e9, s.sets / dt, s.sets ? hands_sum / s.sets : 0, tracker.views(), s.palm_calls,
+                        s.palm_batches ? s.palm_ms / s.palm_batches : 0, s.hand_calls,
+                        s.hand_batches ? s.hand_ms / s.hand_batches : 0, s.sets ? s.step_ms / s.sets : 0,
+                        lat.empty() ? 0 : lat[lat.size() / 2], resid_n ? resid_sum / resid_n : 0, 100 * (cpu1 - cpu0) / dt);
+            if (!color.empty())
+                std::printf("        cameras %s%s: colour frames at %.1f (bright at %.0f, dim under %.0f), ambient IR %.1f, "
+                            "%d steps with colour, colour placed %.1f ms after capture\n",
+                            cams_name(mode), switching ? " (auto)" : "", light.level, light.on, light.off, ambient(),
+                            color_steps, mono_delay_ns / 1e6);
+            if (s.sets)
+                std::printf("        sets with a hand: left %2.0f%% right %2.0f%% both %2.0f%%  views lost %d, handoff misses %d, "
+                            "dups %d, splits %d  hands new %d merged %d forgotten %d%s\n",
+                            100.0 * left_sets / s.sets, 100.0 * right_sets / s.sets, 100.0 * both_sets / s.sets, s.lost,
+                            s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten,
+                            !rec ? "" : ("  recorded " + std::to_string(rec->written()) + " dropped " +
+                                         std::to_string(rec->dropped())).c_str());
+            if (gestures_on) {
+                std::printf("        pinches: left %u right %u (held back, palm down: %d %d)  grips: left %u right %u",
+                            pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1],
+                            grip.side(0).begins, grip.side(1).begins);
+                for (int k = 0; k < 2; ++k)
+                    if (pinch.side(k).flags & FH_PINCH_TRACKED)
+                        std::printf("  %s %s d %.3f curl %.2f", k ? "right" : "left",
+                                    grip.side(k).flags & FH_PINCH_DOWN    ? "GRIP"
+                                    : pinch.side(k).flags & FH_PINCH_DOWN ? "PINCH"
+                                                                          : "open",
+                                    pinch.side(k).distance, grip.curl[k]);
+                std::printf("\n");
+            }
+            for (const Hand *h : hands)
+                std::printf("        hand %d %-5s views %d wrist %+.3f %+.3f %+.3f m  scale %.2f  speed %.2f m/s\n", h->id,
+                            h->right() ? "right" : "left", h->nviews, h->pts[0][0], h->pts[0][1], h->pts[0][2], h->scale,
+                            h->speed);
+            std::fflush(stdout);
+            tracker.stats = Stats{};
+            t_status = now, cpu0 = cpu1;
+            lat.clear(), hands_sum = 0, resid_sum = 0, resid_n = 0, left_sets = right_sets = both_sets = 0;
+            color_steps = 0;
+        }
+    }
+    if (!color.empty()) unlink(want_file.c_str());
+    if (publish) pub.write({}, mono_ns());
+    if (publish && gestures_on) {
+        pinch.release(int64_t(mono_ns()));   // a drag in progress ends, as lost
+        grip.release(int64_t(mono_ns()));
+        gestures.write(pinch, grip, mono_ns());
+    }
+    return 0;
+}

hands/track/nets.cpp

@@ -0,0 +1,255 @@
+#include "nets.h"
+
+#include <mat.h>
+
+#include <algorithm>
+#include <cstdlib>
+#include <numeric>
+
+namespace {
+
+constexpr int kPalmSize = 192, kHandSize = 224;
+const int kRoiLandmarks[] = {0, 1, 2, 3, 5, 6, 9, 10, 13, 14, 17, 18};
+
+// 2x3 affine taking crop pixels (0..out) to image pixels.
+void crop_matrix(V2 center, double size, double rotation, int out, float tm[6]) {
+    const double c = std::cos(rotation), s = std::sin(rotation), k = size / out;
+    tm[0] = float(c * k), tm[1] = float(-s * k), tm[3] = float(s * k), tm[4] = float(c * k);
+    tm[2] = float(center[0] - (tm[0] + tm[1]) * out / 2.0);
+    tm[5] = float(center[1] - (tm[3] + tm[4]) * out / 2.0);
+}
+
+V2 to_image(const float tm[6], double x, double y) {
+    return {tm[0] * x + tm[1] * y + tm[2], tm[3] * x + tm[4] * y + tm[5]};
+}
+
+// OpenCV's CLAHE (4x4 tiles) on a square crop, in place.
+void clahe(uint8_t *img, int n, double clip_limit) {
+    constexpr int kTiles = 4;
+    const int ts = n / kTiles, area = ts * ts;
+    const int clip = std::max(1, int(clip_limit * area / 256));
+    uint8_t lut[kTiles][kTiles][256];
+    for (int ty = 0; ty < kTiles; ++ty)
+        for (int tx = 0; tx < kTiles; ++tx) {
+            int hist[256] = {};
+            for (int y = ty * ts; y < (ty + 1) * ts; ++y)
+                for (int x = tx * ts; x < (tx + 1) * ts; ++x) ++hist[img[y * n + x]];
+            int excess = 0;
+            for (int &h : hist)
+                if (h > clip) excess += h - clip, h = clip;
+            const int add = excess / 256, residual = excess - add * 256;
+            for (int i = 0; i < 256; ++i) hist[i] += add + (i < residual ? 1 : 0);
+            int sum = 0;
+            const float scale = 255.f / area;
+            for (int i = 0; i < 256; ++i) {
+                sum += hist[i];
+                lut[ty][tx][i] = uint8_t(std::min(255, int(sum * scale + 0.5f)));
+            }
+        }
+    std::vector<uint8_t> out(size_t(n) * n);
+    for (int y = 0; y < n; ++y) {
+        const float fy = (y + 0.5f) / ts - 0.5f;
+        const int y0 = std::clamp(int(std::floor(fy)), 0, kTiles - 1), y1 = std::min(y0 + 1, kTiles - 1);
+        const float wy = std::clamp(fy - y0, 0.f, 1.f);
+        for (int x = 0; x < n; ++x) {
+            const float fx = (x + 0.5f) / ts - 0.5f;
+            const int x0 = std::clamp(int(std::floor(fx)), 0, kTiles - 1), x1 = std::min(x0 + 1, kTiles - 1);
+            const float wx = std::clamp(fx - x0, 0.f, 1.f);
+            const uint8_t v = img[y * n + x];
+            const float top = lut[y0][x0][v] * (1 - wx) + lut[y0][x1][v] * wx;
+            const float bot = lut[y1][x0][v] * (1 - wx) + lut[y1][x1][v] * wx;
+            out[size_t(y) * n + x] = uint8_t(top * (1 - wy) + bot * wy + 0.5f);
+        }
+    }
+    std::copy(out.begin(), out.end(), img);
+}
+
+// Linear stretch of the 1st..99th percentile to 0..255, in place.
+void stretch(uint8_t *img, int n) {
+    int hist[256] = {};
+    const int total = n * n;
+    for (int i = 0; i < total; ++i) ++hist[img[i]];
+    int lo = 0, hi = 255, acc = 0;
+    for (int v = 0; v < 256; ++v)
+        if ((acc += hist[v]) > total / 100) { lo = v; break; }
+    acc = 0;
+    for (int v = 255; v >= 0; --v)
+        if ((acc += hist[v]) > total / 100) { hi = v; break; }
+    if (hi <= lo) return;
+    for (int i = 0; i < total; ++i) img[i] = uint8_t(std::clamp((img[i] - lo) * 255 / (hi - lo), 0, 255));
+}
+
+// A crop as the models' input: RGB (the mono plane three times), 0..1.
+ncnn::Mat crop(const Image &img, const float tm[6], int n, const Contrast &contrast) {
+    std::vector<uint8_t> patch(size_t(n) * n);
+    ncnn::warpaffine_bilinear_c1(img.data, img.width, img.height, img.stride, patch.data(), n, n, n, tm, 0, 0);
+    if (contrast.mode == Contrast::Clahe) clahe(patch.data(), n, contrast.clip);
+    else if (contrast.mode == Contrast::Stretch) stretch(patch.data(), n);
+    ncnn::Mat m = ncnn::Mat::from_pixels(patch.data(), ncnn::Mat::PIXEL_GRAY2RGB, n, n);
+    const float norm[3] = {1 / 255.f, 1 / 255.f, 1 / 255.f};
+    m.substract_mean_normalize(nullptr, norm);
+    return m;
+}
+
+}  // namespace
+
+bool Contrast::parse(const std::string &s, Contrast &out) {
+    if (s == "none") return out.mode = None, true;
+    if (s == "stretch") return out.mode = Stretch, true;
+    if (s.rfind("clahe", 0) == 0) {
+        out.mode = Clahe;
+        out.clip = s.size() > 6 && s[5] == ':' ? std::atof(s.c_str() + 6) : 2.0;
+        return out.clip > 0;
+    }
+    return false;
+}
+
+bool Contrast::parse_pair(const std::string &s, Contrast &palm, Contrast &hand) {
+    const size_t slash = s.find('/');
+    if (slash == std::string::npos) return parse(s, palm) && parse(s, hand);
+    return parse(s.substr(0, slash), palm) && parse(s.substr(slash + 1), hand);
+}
+
+namespace {
+
+bool load_net(ncnn::Net &net, const std::string &base, std::string &err) {
+    net.opt.num_threads = 1;
+    net.opt.use_vulkan_compute = false;
+    net.opt.use_fp16_packed = net.opt.use_fp16_storage = net.opt.use_fp16_arithmetic = true;
+    if (net.load_param((base + ".param").c_str()) || net.load_model((base + ".bin").c_str())) {
+        err = "can't load " + base + ".param/.bin";
+        return false;
+    }
+    return true;
+}
+
+}  // namespace
+
+Roi Palm::roi() const {
+    const V2 a = kp[0], b = kp[2];
+    const double rot = wrap_angle(M_PI / 2 - std::atan2(-(b[1] - a[1]), b[0] - a[0]));
+    const double h = size[1];
+    const V2 shift{-h * -0.5 * std::sin(rot), h * -0.5 * std::cos(rot)};
+    return {center + shift, std::max(size[0], size[1]) * 2.6, rot};
+}
+
+Roi roi_from_points(const V2 *p) {
+    const V2 w = p[0];
+    V2 m = (p[5] + p[13]) * 0.5;
+    m = (m + p[9]) * 0.5;
+    const double rot = wrap_angle(M_PI / 2 - std::atan2(-(m[1] - w[1]), m[0] - w[0]));
+    V2 lo{1e9, 1e9}, hi{-1e9, -1e9};
+    for (int i : kRoiLandmarks)
+        for (int k = 0; k < 2; ++k) lo[k] = std::min(lo[k], p[i][k]), hi[k] = std::max(hi[k], p[i][k]);
+    V2 center = (lo + hi) * 0.5;
+    const double c = std::cos(-rot), s = std::sin(-rot);
+    V2 qlo{1e9, 1e9}, qhi{-1e9, -1e9};
+    for (int i : kRoiLandmarks) {
+        const V2 d = p[i] - center;
+        const V2 q{d[0] * c - d[1] * s, d[0] * s + d[1] * c};
+        for (int k = 0; k < 2; ++k) qlo[k] = std::min(qlo[k], q[k]), qhi[k] = std::max(qhi[k], q[k]);
+    }
+    const V2 mid = (qlo + qhi) * 0.5;
+    const double c2 = std::cos(rot), s2 = std::sin(rot);
+    center = center + V2{mid[0] * c2 - mid[1] * s2, mid[0] * s2 + mid[1] * c2};
+    const double w2 = qhi[0] - qlo[0], h2 = qhi[1] - qlo[1];
+    center = center + V2{-h2 * -0.1 * s2, h2 * -0.1 * c2};
+    return {center, std::max(w2, h2) * 2.0, rot};
+}
+
+Roi Landmarks::next_roi() const { return roi_from_points(pts); }
+
+bool Nets::load(const std::string &dir, bool int8, std::string &err) {
+    const std::string suffix = int8 ? "-int8.ncnn" : ".ncnn";
+    if (!load_net(palm_, dir + "/palm" + suffix, err) || !load_net(hand_, dir + "/hand" + suffix, err)) return false;
+    // SSD anchors of palm_detection_full: strides 8 (2 per cell) and 16 (6 per cell)
+    for (auto [stride, per] : {std::pair{8, 2}, std::pair{16, 6}}) {
+        const int n = kPalmSize / stride;
+        for (int y = 0; y < n; ++y)
+            for (int x = 0; x < n; ++x)
+                for (int k = 0; k < per; ++k) anchors_.push_back({(x + 0.5) / n * kPalmSize, (y + 0.5) / n * kPalmSize});
+    }
+    return true;
+}
+
+std::vector<Palm> Nets::palms(const Image &img, V2 center, double size, double rotation) const {
+    float tm[6];
+    crop_matrix(center, size, rotation, kPalmSize, tm);
+    ncnn::Extractor ex = palm_.create_extractor();
+    ex.input("in0", crop(img, tm, kPalmSize, palm_contrast_));
+    ncnn::Mat boxes, scores;
+    ex.extract("out0", boxes);
+    ex.extract("out1", scores);
+    const float *raw = boxes, *logit = scores;
+    const int n = int(anchors_.size());
+    const float min_logit = std::log(0.5f / 0.5f);  // score 0.5
+    struct Cand { V2 c, s; V2 kp[7]; double score; };
+    std::vector<Cand> cand;
+    for (int i = 0; i < n; ++i) {
+        if (logit[i] <= min_logit) continue;
+        const float *r = raw + i * 18;
+        Cand c;
+        c.c = {r[0] + anchors_[i][0], r[1] + anchors_[i][1]};
+        c.s = {r[2], r[3]};
+        for (int k = 0; k < 7; ++k) c.kp[k] = {r[4 + 2 * k] + anchors_[i][0], r[5 + 2 * k] + anchors_[i][1]};
+        c.score = 1 / (1 + std::exp(-std::clamp(double(logit[i]), -100.0, 100.0)));
+        cand.push_back(c);
+    }
+    // MediaPipe's weighted NMS: overlapping boxes are averaged, weighted by score
+    std::sort(cand.begin(), cand.end(), [](const Cand &a, const Cand &b) { return a.score > b.score; });
+    std::vector<bool> used(cand.size());
+    std::vector<Palm> out;
+    for (size_t i = 0; i < cand.size(); ++i) {
+        if (used[i]) continue;
+        double wsum = 0;
+        Cand acc{};
+        for (size_t j = i; j < cand.size(); ++j) {
+            if (used[j]) continue;
+            const double ix = std::max(0.0, std::min(cand[i].c[0] + cand[i].s[0] / 2, cand[j].c[0] + cand[j].s[0] / 2) -
+                                                std::max(cand[i].c[0] - cand[i].s[0] / 2, cand[j].c[0] - cand[j].s[0] / 2));
+            const double iy = std::max(0.0, std::min(cand[i].c[1] + cand[i].s[1] / 2, cand[j].c[1] + cand[j].s[1] / 2) -
+                                                std::max(cand[i].c[1] - cand[i].s[1] / 2, cand[j].c[1] - cand[j].s[1] / 2));
+            const double inter = ix * iy;
+            const double uni = cand[i].s[0] * cand[i].s[1] + cand[j].s[0] * cand[j].s[1] - inter;
+            if (j != i && inter / (uni + 1e-9) <= 0.3) continue;
+            used[j] = true;
+            const double w = cand[j].score;
+            wsum += w;
+            acc.c = acc.c + cand[j].c * w;
+            acc.s = acc.s + cand[j].s * w;
+            for (int k = 0; k < 7; ++k) acc.kp[k] = acc.kp[k] + cand[j].kp[k] * w;
+        }
+        Palm p;
+        const V2 c = acc.c * (1 / wsum);
+        p.center = to_image(tm, c[0], c[1]);
+        p.size = acc.s * (1 / wsum * size / kPalmSize);
+        for (int k = 0; k < 7; ++k) {
+            const V2 q = acc.kp[k] * (1 / wsum);
+            p.kp[k] = to_image(tm, q[0], q[1]);
+        }
+        p.score = cand[i].score;
+        out.push_back(p);
+    }
+    return out;
+}
+
+Landmarks Nets::landmarks(const Image &img, const Roi &roi) const {
+    float tm[6];
+    crop_matrix(roi.center, roi.size, roi.rotation, kHandSize, tm);
+    ncnn::Extractor ex = hand_.create_extractor();
+    ex.input("in0", crop(img, tm, kHandSize, hand_contrast_));
+    ncnn::Mat screen, presence, right, world;
+    ex.extract("out0", screen);
+    ex.extract("out1", presence);
+    ex.extract("out2", right);
+    ex.extract("out3", world);
+    Landmarks lm;
+    const float *s = screen, *w = world;
+    for (int i = 0; i < 21; ++i) {
+        lm.pts[i] = to_image(tm, s[3 * i], s[3 * i + 1]);
+        for (int k = 0; k < 3; ++k) lm.world[i][k] = w[3 * i + k];
+    }
+    lm.presence = presence[0];
+    lm.right = right[0];
+    return lm;
+}