Let the 3D mouse click all of SteamVR's Settings page

Most of SteamVR's Settings page took no clicks from the 3D mouse: they
went through to a desktop screen behind it, and the page covered the dot.
Steam's pages (Library and the rest) are drawn in
valve.steam.gamepadui.main, which ComputeOverlayIntersection hits exactly.
For SteamVR Settings that overlay is hidden and the page is drawn by the
dashboard's scene-graph panel, whose shape OpenVR doesn't give out. The
helper guessed a plane and a 0.5 m circle from the panel's transform, but
its origin is at the page's left edge and the page's surface is nearer
than that plane, so the laser, starting a few cm in front of the guess,
started behind the page.

On that page only (dashboard open, the main overlay hidden, the line of
sight crossing the page's measured area), the laser now starts 25 cm from
the eye so SteamVR's own hit test finds the page. The dot is drawn 0.6 m
out, in front of the page, and the laser-catching dot sits 8 m out,
invisible, with SteamVR's hit dot hidden on it. The beam and SteamVR's hit
dot show there, like a controller's; everywhere else nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -53,7 +53,7 @@ If you work in the desktop for long stretches, stop Steam from putting the heads
 | Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & wrist tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
 | Play a VR game | The screens hide and your controllers stay in the game. Open the SteamVR dashboard, or press Meta+Shift+H, to see and use them. To keep them visible over games, change During VR games on the Visibility & wrist tab; the controllers still stay in the game, and you use the screens with the mouse or the dashboard |
 
-You can map the mouse's extra buttons to actions such as Toggle SteamVR dashboard or Recenter pointer on the Buttons page of Frametop Input Settings. Pointer speed, dot size, and the rest are on its Pointer page and take effect immediately.
+You can map the mouse's extra buttons to actions such as Toggle SteamVR dashboard, Recenter pointer, or Head follow on/off on the Buttons page of Frametop Input Settings, and the Frame controllers' buttons on its Controllers page. Pointer speed, dot size, and the rest are on its Pointer page and take effect immediately. Head follow, which is experimental and off by default, makes the pointer come along when you turn your head: it stays put until your head turns past the leash angle, then glides back to its place in your view, and a leash of 0 keeps it fixed in your view. It's only lightly tested and not polished; tuning its settings, or improving how it feels, is open to anyone who wants to take it further.
 
 Restarting the desktop (Restart desktop in Frametop Display Settings) closes its windows, but background work you started in it, such as servers, tmux sessions, or builds, keeps running.
 
@@ -65,7 +65,9 @@ This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 2026092
 - The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes.
 - During a VR game you can't show the screens with a controller button, because the game owns the buttons. Open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead.
 - Flatscreen games aren't detected as games. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & wrist tab).
+- Typing follows your last click. A controller click on a panel other than the screens (the dashboard, a Steam app) doesn't move typing there; click it with the mouse, or click a screen to bring typing back.
 - The screens don't draw a mouse cursor of their own. The 3D mouse's dot or SteamVR's laser shows where you're pointing.
+- On SteamVR's Settings page, the 3D mouse shows a laser beam and a larger hit dot, like a controller. SteamVR doesn't tell other programs where that page is (unlike Steam's pages, such as Library), so the mouse used to miss most of it: clicks went through to a desktop screen behind, and the dot disappeared. As a workaround, on that page only, the laser starts near your eye and SteamVR finds the page itself. See docs/design.md.
 - Remote desktop over VNC (`./desktops.sh remote on`) needs Tailscale on the Frame.
 
 ## Reporting problems
@@ -98,7 +100,7 @@ setup/bluetooth/install.sh uninstall   # if you installed the Bluetooth fixes
 
 ## How it works
 
-A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland compositor. KWin opens one window per screen, ft-screens sets each window's size, and each frame goes to SteamVR as an overlay without being copied. An input relay (`input/`) keeps Bluetooth mice working in SteamVR and feeds the mouse to the 3D pointer, which drives a virtual SteamVR controller (`pointer/`). [docs/reference.md](docs/reference.md) covers each piece, and [docs/design.md](docs/design.md) explains the design and what we learned about SteamVR on the Frame.
+A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland compositor. KWin opens one window per screen, ft-screens sets each window's size, and each frame goes to SteamVR as an overlay without being copied. An input relay (`input/`) keeps Bluetooth mice working in SteamVR and feeds the mouse to the 3D pointer, which drives a virtual SteamVR controller (`pointer/`). [docs/reference.md](docs/reference.md) covers each piece, and [docs/design.md](docs/design.md) explains the design and what we learned about SteamVR on the Frame. [docs/hazards.md](docs/hazards.md) lists known ways the input handling can go wrong.
 
 | Folder | What it is |
 | --- | --- |

docs/design.md

@@ -77,9 +77,14 @@ The laser starts partway along your line of sight to the cursor rather than at y
 A few overlays need special handling:
 
 - The dashboard's dock and the floating windows' controls are scene-graph overlays with no texture (0 × 0) and a placeholder width, so `ComputeOverlayIntersection` never hits them. For those the helper tests the overlay's plane within `POINTER_SCENE_RADIUS` of its origin.
+- SteamVR's Settings page is the one page `ComputeOverlayIntersection` can't find. Steam's pages, Library and the rest, are drawn in `valve.steam.gamepadui.main`, which it hits exactly. For SteamVR Settings that overlay is hidden, and the page is drawn by the dashboard's scene-graph panel, whose shape OpenVR doesn't give out, and whose transform's plane isn't the page's surface. The laser started behind the page, so most of it took no clicks (they went to a desktop screen behind it), and the page covered the dot. On that page only, the laser now starts near the eye so SteamVR's own hit test finds the page, the dot is drawn close in front of it, and the laser-catching dot sits far behind everything, invisible, with SteamVR's hit dot hidden on it. There the beam and SteamVR's hit dot look like a controller's; everywhere else nothing changes.
 - Just off a panel, the cursor stays on that panel's plane within `POINTER_EDGE_REACH`, so resize margins and window controls just outside the panel are reachable.
 - While the left button is held, the cursor keeps the distance it had at the press and stops re-testing collisions, so dragging past a panel's edge doesn't make it jump.
 
+Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
+
+Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse or controller button mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz, and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze; the mouse is gaze mode's only pointer device for now. The dot shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. The gaze moving the pointer doesn't show it: you know where you're looking. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured.
+
 Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
 
 `dashboard.laserRayWidthScale` controls the beam's width, but SteamVR only applies a change from its own settings screen or at restart, so it can't be switched per device while running.
@@ -100,6 +105,12 @@ SteamVR opens every input device only when it starts. When a Bluetooth mouse sle
 
 Keyboards aren't grabbed by default, because a grabbed keyboard's keys went into a virtual keyboard nothing typed from; the relay forwards them to ft-screens instead.
 
+An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's `InputStealer`, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's `OverlayFocusChanged` (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. So with `SHARE_KEYS=1` the relay sends a grabbed keyboard's keys to `@frametop_keys` as datagrams (`key <code> <value> <device name>`). It's off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first gets every key typed into the desktop, passwords included. A listener should accept only its own user (`SO_PASSCRED`) and skip any keyboard of its own that the relay grabs too.
+
+Volume keys must never reach gamescope. With the openvr backend, gamescope sends volume up and down to Steam by moving keyboard focus to Steam for the key and then back to the previously focused surface. When nothing had focus, the one it moves back to is null, and wlroots aborts on a null focus surface (`wlr_seat_keyboard_notify_enter: Assertion 'surface' failed`), which ends the whole VR session. Keyboard focus is often empty while you work in VR, so one press of the headset's volume button could take everything down. gamescope reads the headset's buttons and every keyboard itself (`InputStealer`), as do SteamVR's processes, so the relay has to stop volume keys at the device. Grabbing `gpio-keys` would also take the headset's click button, so the relay remaps the volume entries in each device's keymap (`EVIOCSKEYCODE`) and handles the stand-in codes itself. That fix covers every device at once, including keyboards that aren't grabbed.
+
+The Frame controllers can be mapped like mouse buttons, but they aren't input devices on the host: they reach SteamVR over the headset's own radio, and no evdev or hidraw node exists for them. So only a SteamVR client can read them. Overlay apps normally get controller input only while they have input focus, which a background helper never has. SteamVR's experimental global action set priority (`steamvr/globalActionSetPriority`, "Enable global input from overlays") lets an overlay's action set receive input anyway, and takes the inputs it binds from the scene app. Binding every button would take them all from games, so the helper's action manifest puts each button in an action set of its own, and it activates only the sets of mapped buttons. The mapping itself stays in the relay, which does the action, so mice and controllers share one list of actions.
+
 ## The desktop session
 
 The session is modeled on SteamOS's `steamos-nested-desktop` and runs beside it. It has its own runtime directory, config (`~/.config/frametop`), and state, so it never disturbs the stock desktop's layout or panels. It runs on a private D-Bus from `dbus-run-session`, which has two consequences. KDE only launches apps in systemd scopes when systemd is on the session bus, so everything started in the desktop lands in its systemd unit, and stopping the unit would kill all of it; `session/keep-apps.sh` moves those programs out first. And tools that need the real user bus, like podman and `distrobox-host-exec`, have to be pointed at it explicitly.

docs/hazards.md

@@ -0,0 +1,27 @@
+# Potential hazards
+
+Known ways the input changes can go wrong, and what to check when something looks off. Each has been reasoned through but not all have been seen on a headset. [design.md](design.md) explains why the input relay works the way it does.
+
+## Volume keys
+
+The input relay takes the volume keys from every device that has them, so gamescope never sees one (a volume press with nothing focused aborts gamescope and ends the VR session). On devices with a keymap it remaps the volume entries to stand-in codes (`KEY_MACRO29`, `KEY_MACRO30`), and it grabs `pmic_resin`.
+
+- **Keymaps stay remapped if the relay dies.** The keymaps go back when the relay exits normally or on `systemctl stop` (SIGTERM). A crash or SIGKILL skips that, and until the relay starts again the volume keys do nothing, on the headset and on any keyboard it remapped. The headset's own buttons don't reconnect, so for them only the relay coming back (or a reboot) fixes it. The relay recognizes the stand-in codes on start and takes them over again.
+- **The headset's other buttons share the device.** `gpio-keys` carries the click button as well as volume. Only the volume entries are remapped, but if the click button stops working, check this first (`--no-grab` leaves the volume keys alone).
+- **The relay opens more devices than it used to.** It now opens any device with volume keys, whatever its bus, not only USB and Bluetooth mice and keyboards. A device it can't remap and that has more than volume keys is left alone, and its volume keys still reach gamescope (the log says "can't take over its volume keys").
+- **Volume goes to the default output.** `wpctl` steps `@DEFAULT_AUDIO_SINK@` by 5%, capped at 100%. If sound plays somewhere other than the default output, the keys change the wrong one. Steam never sees the keys, so anything it did on a volume press no longer happens.
+- **Repeat is the relay's own.** Holding a key repeats after 0.4 s, every 0.1 s, and kernel autorepeat from keyboards is ignored. If a device disconnects mid-hold, the repeat stops with it.
+
+## Key releases
+
+ft-screens drops keys while no screen has focus or the SteamVR dashboard is open, but always lets through the release of a key the desktop saw pressed, so a modifier held as the dashboard opens doesn't stay down.
+
+- **A release that never arrives leaves the key held in the desktop.** KWin repeats held keys itself, so a stuck letter repeats and a stuck modifier changes every later key (Ctrl+Alt held turns T into Konsole). Pressing and releasing the key again clears it.
+- **A keyboard that disconnects mid-press is one way to get there.** The relay forgets the held key without telling ft-screens. The same goes for the relay restarting while a key is down.
+- **Switching where typing goes waits for keys to come up.** The relay changes a keyboard's grab only while none of its keys are down, so a press and its release go to the same side. A key held for a long time delays the switch until it's let go.
+
+## Typing and grabbed keyboards
+
+- **Programs that watch every keyboard lose grabbed ones.** While typing goes to the desktop, the relay grabs pass-through keyboards, so a hotkey tool reading them directly stops seeing their keys. `SHARE_KEYS=1` in `~/.config/frametop.conf` sends their keys to the abstract socket `@frametop_keys` instead. It's off by default: abstract sockets have no permissions, and any local process that binds the name first receives every key typed into the desktop, passwords included.
+- **Typing starts out going to Steam.** After the desktop starts, keys go to Steam until you click a screen.
+- **Controller clicks don't move typing.** Overlay apps don't see controller clicks on other panels, so after one typing stays where it was. A mouse click, or a click on a screen, moves it.

docs/reference.md

@@ -51,7 +51,7 @@ In the last three modes the hotkey shows the screens anyway. Two more settings o
 - During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up.
 - Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.
 
-Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay, from any keyboard it doesn't grab and any key a pointer device passes through, and go to the screen you clicked last, except while the SteamVR dashboard is open.
+Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay, from pass-through keyboards and any key a pointer device passes through. Typing follows your last click: after a click on a screen it goes to the desktop, even with the SteamVR dashboard open, and after a mouse click on any other panel (the dashboard, Steam, an app like Spotify) it goes there instead. While it goes to the desktop, the relay grabs pass-through keyboards so gamescope, which reads every keyboard itself, doesn't type them into the Steam app too. A program that watches every keyboard for a hotkey loses a grabbed one; with `SHARE_KEYS=1` in `~/.config/frametop.conf`, their keys also go to `@frametop_keys` for it. That's off by default, since any local process that binds the name first would get everything typed into the desktop. Hidden screens don't take typing.
 
 ft-screens listens for datagrams on the abstract socket `@ft_screens` and replies to the sender:
 
@@ -69,6 +69,8 @@ SteamVR opens input devices only when it starts. A Bluetooth mouse that sleeps a
 
 It runs as the user service `frametop-input-relay.service`, ordered before `steamvr.service`.
 
+The relay also owns the volume keys, on every device that has them, the headset's buttons included. It changes the volume itself (`wpctl`, 5% a step, repeating while held), and nothing else sees a volume key, gamescope and SteamVR included: on devices with a keymap (the headset's `gpio-keys`, USB and Bluetooth keyboards) it remaps just the volume entries to unused codes (`KEY_MACRO29`, `KEY_MACRO30`), so the headset's click button and the other keys still work, and it grabs `pmic_resin`, which has only volume down. The keymaps go back when the relay stops. With `--no-grab` it leaves the volume keys alone.
+
 ```
 desktops.sh relay install     # enable it (starts with the next reboot or SteamVR start)
 desktops.sh relay status | log | uninstall
@@ -87,7 +89,7 @@ A mouse drives SteamVR the way a controller's laser does, but shows up as a smal
 
 Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again.
 
-To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button (or a Meta tap) wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant.
+To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button (or a Meta tap, with `META_DASHBOARD=1`) wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant.
 
 ```
 pointer/driver/build.sh && pointer/driver/install.sh install   # then restart SteamVR
@@ -96,15 +98,17 @@ pointer/helper/run.sh status | log | restart
 pointer/driver/install.sh probe     # devices, hand roles, who owns the dashboard pointer
 ```
 
-The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, and `POINTER_LASER_WIDTH`. The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper.
+The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, and `POINTER_GAZE_SHOW`. The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper.
 
 ## Frametop Input Settings
 
-A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has four pages:
+A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has six pages:
 
-- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (not grabbed; the default for keyboards, where a Meta tap still toggles the dashboard), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
+- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, where a Meta tap toggles the dashboard if `META_DASHBOARD=1` is in `~/.config/frametop.conf`), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
-- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, faster or slower, pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
+- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, faster or slower, pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
-- Pointer has sliders for the pointer settings, which apply immediately, and a Recenter button.
+- Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through. Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
+- Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
+- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), the gaze mode sliders, the gaze service's state (headset, samples per second, whether only one eye is tracked, the calibration, the nudges learned), and Calibrate (opens the gaze probe), Reload calibration, and Forget nudges.
 - Bluetooth lists paired devices and has Apply Bluetooth fixes, which runs `/etc/steamframe/bt-fixups.sh` through `pkexec`. Pair new devices in Steam.
 
 Device rules are saved in `~/.config/frametop-input.json`. `input-settings/install.sh` installs the menu entry. Its launcher hands podman the real `XDG_RUNTIME_DIR` and user bus and gives the app the session's Wayland socket, because the desktop session runs on a private D-Bus and podman fails on it.
@@ -113,6 +117,8 @@ Device rules are saved in `~/.config/frametop-input.json`. `input-settings/insta
 
 When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout.
 
+The desktop's own screen arrangement follows where the screens are around you, whatever their numbers: a screen you see to the left of another is to its left in Plasma too, so the pointer and dragged windows cross straight to it. Screens one above the other stack, and screens pinned to a wrist come last. It's updated at startup, after arranging or saving the layout, and half a second after you let go of a screen you moved. With the headset off there's no head pose to go by, and the arrangement stays as it was.
+
 Frametop Display Settings has three tabs:
 
 - Screens: add and remove screens, and set each one's resolution (presets from 1080p to 4K, ultrawide, super ultrawide, portrait, or custom), its width in VR (0.5 to 6 m), its scale, whether it's curved, and whether it has the taskbar. Resolution, width, and curve apply at once. Adding or removing a screen takes a desktop restart, which the app offers.
@@ -125,7 +131,7 @@ Frametop Display Settings has three tabs:
 layout/ft-layout apply      # arrange every screen
 layout/ft-layout capture    # save the current arrangement and sizes as the layout
 layout/ft-layout plan       # print the arrangement as JSON (no VR needed)
-layout/ft-layout scale      # per-screen scale, positions, and taskbar screen, to KWin
+layout/ft-layout scale      # per-screen scale, positions (as the screens are around you), and taskbar screen, to KWin
 layout/ft-layout toggle     # hide or show all screens
 display-settings/install.sh # menu entries and the Meta+Shift+R and Meta+Shift+H shortcuts
 ```

gaze/README.md

@@ -0,0 +1,58 @@
+# Gaze (experimental)
+
+The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (the pointer goes where you look, and the mouse does the last bit), and the tools to calibrate and measure it.
+
+- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on.
+- `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log.
+- `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground. It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
+
+```
+gaze/build.sh                 # build ft-gaze
+gaze/probe/install.sh         # build, and add Frametop Gaze Probe to the app menu
+gaze/probe/ft-gazeprobe --screen 1
+gaze/run.sh install           # the gaze service, with SteamVR
+gaze/ft-gazectl on            # the pointer follows your gaze (off: the mouse alone)
+```
+
+## Gaze pointer
+
+Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:
+
+- `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. It uses SteamVR's combined gaze (mmap set 1), which keeps going when the tracker loses one eye; set 2's combined direction is off by half of whatever the lost eye reads, and the tracker does lose an eye for minutes at a time. It drops blinks (both eyes closing), smooths with a fixation lock, and applies the calibration from the probe (`calibration.json`, reloaded when the probe changes it) plus what the pointer has learned since (`pointer-lessons.json`). It sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
+- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse or controller button mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on. The dot only shows while the mouse moves it, while a press is held, and as a pulse when you click.
+- **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
+- Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
+
+Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.
+
+## Gaze sources
+
+| Source | Where it comes from |
+| --- | --- |
+| SteamVR action | An `eyetracking` action bound to `/user/head/eyetracking` (`actions/`), read with `IVRInput::GetEyeTrackingDataRelativeToNow`. This is the supported way. |
+| mmap set 1, set 2 | `/dev/shm/eye-server.mmap`, which SteamVR's eyetracking process writes for the HMD driver (`driver_cv.so`). It has two sets of per-eye directions in head space. The layout is undocumented (offsets are in `ft-gaze.cpp`) and may change with a SteamVR update. ft-gaze maps it read-only; the file also carries calibration clicks to the tracker and must never be written. |
+
+The tracker stops when the headset is off your head. SteamVR also calibrates gaze on its own from laser-mouse clicks, treating each click as a spot you were looking at. That includes mouse clicks through the Frametop pointer, so a click where the pointer's dot isn't what you're looking at teaches SteamVR a wrong sample (it only takes clicks within 5 degrees of your gaze). In the probe, use Enter or Space as the trigger: keys don't go through SteamVR's laser. See `Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`.
+
+## Probe
+
+The trigger is Enter, Space, or a mouse button. Right-click anywhere in the window (or press the Menu key or Shift+F10) for a menu with Run calibration, Start accuracy test, Calibrate from last test, Reset calibration, the modes, the panel, fullscreen, and Quit. The buttons at the top right show and hide the panel, leave fullscreen, and quit. The keys do the same (Tab, F11, Esc), but only after you click the window once, since Frametop sends typing to the panel you clicked last. If ft-gaze stops, the probe starts it again after 3 s and shows why it stopped. Windowed mode stays on the screen it was on, and a small KWin script tells the probe where the window is, so the dot and targets are still in the right place.
+
+- **Run calibration (start here):** the initial calibration, modeled on Apple Vision Pro's eye setup. Face the centre and keep your head still. Look at one dot and press the trigger, then at each of six dots in a circle. That happens in three rounds, and the screen goes dark, then medium, then bright, because pupil size changes with brightness and the tracker's error with it. Each round turns the ring 20 degrees, and the middle round's ring is half the size, so the 21 dots cover the middle, halfway out, and the edge of your view. The ring's size is `Calibration ring` (degrees, 20 by default, less if the window is too small). Error grows toward the edge, and the calibration can only correct as far out as it has seen dots. The current dot is a bright pulsing dot with a point in the middle; finished dots fade to specks, so your eyes don't go back to them. Samples from blinks and from moments when the tracker lost an eye are dropped: openness under half of what it was during that look (not a fixed level, because your lids come down when you look down, and you squint in the bright round), or the angle between the eyes jumping more than 1.5 degrees from its median (that angle depends on how far away you're looking, so only a jump counts). Each dot is measured with medians, so one bad sample can't fail it. A look that lands where the gaze was for another dot of the round is refused as a look at the wrong dot. Mouse clicks don't count during a calibration run or a test: use Enter or Space. If a dot still fails, the message says why and the next try listens longer. After two failures, S (or the menu) skips the dot. Every attempt is logged to `calibration-attempts.jsonl`. At the end it fits every source's calibration from all the dots, replacing what it had learned (quadratic if the model was none). Esc cancels. The run is saved as `calibration-*.json`. With "Test after calibration" on (the default), the accuracy test starts right after, on new spots.
+- **Free look:** the gaze dot. The trigger calibrates wherever you're looking (see below).
+- **Accuracy test:** look at each target and press Enter or Space. "Test spots" picks where the targets go. **Calibrated area** (the default) puts 15 new spots inside the calibration ring (the centre, 7 halfway out, 7 near the ring), turned so none sits on a calibration dot. It checks the calibration where it was made, with your head facing the centre. The **window** grids reach past that area. On a wide screen that's far more than your eyes turn without your head, so they show how the calibration holds up beyond where it was made. For each source the test records the error before and after correction (degrees and pixels), the share of targets within 1 degree, jitter, and the corrected error by region of your view (centre, up, down-left, and so on), worst first. On screen, each target gets a faint line to the raw gaze and a solid line to where the corrected dot was, green under 1 degree, yellow under 2, red above. Tests since the last calibration are listed as a trend.
+- **Refine calibration:** refits from the latest calibration run's dots plus every calibrated-area test since. It tries offset, affine, quadratic, and quadratic+grid, scoring each on points it wasn't fitted on (leave-one-out), and uses the best. Then test again: each test adds its targets, so test, refine, test is the loop. The scores are in the panel and in `refinements.jsonl`.
+- **Snap practice:** a field of desktop-like elements (toolbar icons, list rows, buttons, tiles, small links), some close together, inside the calibrated area. The gaze snaps to the nearest element and highlights it, so the pointer lands on a whole element instead of a spot. Look at the orange one and tap Enter (or click) to click it. If the wrong one is highlighted, hold the press instead: the highlight locks and stops following your gaze. Glance toward the right one (look off to that side and back), and each glance steps the highlight to the next element that way. Or move the mouse, and the highlight follows it from where it was. Let go on the right one. A glance works however far off the tracker is, because only the eye movement counts, and the tracker gets that right: its error barely changes over a couple of degrees. Whichever element you let go on is taken as the one you were looking at when you pressed, and the gap from the gaze at the press is learned as the tracker's error there (not if it's over 6 degrees after the correction, which means a wrong element). That's what it would learn in real use, where nothing knows which element you meant. The probe does know (the orange one), so each click is also scored: right at the press, right in the end, and whether the snap would have been right with the calibration alone. Backspace takes back the last click's lesson. Clicks go to `snaps.jsonl`.
+- **Click practice:** the white dot is a gaze pointer, the way it would be in real use. Look at the target and press (click, or Enter), and keep looking at it. The dot stops following your gaze. If it isn't on the target, keep holding and move the mouse: the dot moves with it. Let go on the target. You were looking at where you let go when you pressed, so the drag is the tracker's error there, and the click corrections learn it (not if it's over 6 degrees after the correction). A click without a drag teaches nothing: it only says the dot was close enough. The target is only for scoring: would a plain gaze click have hit at the press, and did the drag end on it. The drag is drawn for a moment. Presses go to `practice.jsonl`. The Frametop pointer (the mouse's own white dot) stays where the mouse puts it. The probe only reads its movement. An earlier version steered the Frametop pointer onto the gaze with the pointer helper's `move` commands. It lost the user's pointer: the helper's pointer goes idle, or a controller takes the laser, and the moves piled up. Taking over the real pointer belongs in the pointer helper itself, which knows its own state and can aim straight at the gaze.
+
+The default trigger is **freeze and look**, the on-demand calibration. The press freezes the dot where the tracker says you're looking. Then look at the frozen dot: it's a target right where you're looking, and it stays put. After `settle` ms it averages the unsmoothed gaze for `capture` ms, or until you let go if you hold longer. The gap between the frozen dot and that average is the tracker's error at that spot, and the calibration learns it. The live dot is hidden while frozen so it can't pull your eye (the "Live dot while frozen" option shows it anyway). Freeze and look drops blink and dropout samples and uses medians, like the calibration run. A capture is thrown out if the gaze spread more than `max spread` (1 degree) or the error is over `max error` (12 degrees; the real error reaches 8-9 degrees looking well up or down). Each capture is logged to `captures.jsonl`.
+
+The older triggers, nudge with head and nudge with eyes, are still there. Hold, then move the frozen dot onto what you meant with your head or eyes (`nudge gain` scales the movement), and release. When nudging with your eyes, don't look at the dot: it follows your gaze, error included, so it runs away.
+
+Smoothing defaults to fixation lock. It holds the dot on the running mean of the current fixation and jumps when your gaze leaves the fixation radius. One Euro follows more smoothly, and its beta is per degree a second. Sitting still, raw gaze jitters by about 0.25-0.3 degrees, and mmap set 2 was the quietest source, so it's the default.
+
+**Click corrections** ("Learn from clicks", on by default) are learned on the fly from snap and practice clicks, on top of the calibration. Right-click, Clear click corrections forgets them and keeps the calibration. The first click shifts the whole correction. More clicks bend it (the same quadratic terms, held near zero except the offset), and what's left near each click is added within about 3 degrees of it: on your data, errors less than 3 degrees apart are alike, and ones further apart aren't. Recent clicks count more, so it follows SteamVR's gaze as that moves. A big element only weakly says where on it you looked, so a wide list row barely counts sideways. Replayed on logged points, a calibration from an earlier session was 4.95 degrees off; one click brought that to 2.3, five to 1.6, and twenty to 1.2. They're saved with the calibration and start over with a new calibration run.
+
+SteamVR's eye tracker also calibrates itself, from clicks (`Accept usercal` in `~/.local/share/Steam/logs/eyetracking.txt`). It takes a quick mouse-button down and up as "you were looking there", if the gaze was held within 5 degrees of the click. In the logs, the accepted clicks were all under 0.14 s, one of 0.38 s was "too slow", and a click that moved between down and up was refused. It keeps that inside the running `eyetracking` process and saves nothing, so when SteamVR starts again, its calibration starts over and the raw gaze moves: your 13:39 and 21:23 sessions had a restart between them, and the error's shape changed, not just its offset. The panel shows when the eye tracker started, whether that was after your calibration, and how many clicks it has learned from since. Snap and practice clicks are what keep up with it: a drag is too slow for SteamVR to take, and a quick click on the snapped element teaches both calibrations the same spot.
+
+Pick the **Correction model** in the panel or the right-click menu. Choosing one fits it right away from the latest calibration's dots and the calibrated-area tests since (`points.jsonl`), and the choice is saved with the calibration. The models (per source) are none, offset, affine (offset plus a straight-line change across your view), quadratic (the default), and affine+grid or quadratic+grid (plus a 10-degree grid for what's left). Quadratic is the second-order polynomial video eye trackers usually calibrate with. The Frame's error grows as your eyes turn away from the centre: it overstates vertical movement, more the further up or down you look, and looking up adds a sideways error. A straight line only follows part of that. A polynomial runs away outside the spots it was fitted on, so the model only follows it to 3 degrees past the range of view it has seen. They're keyed by where you're looking relative to your head, and saved in `~/.local/state/frametop/gaze/calibration.json`. Freeze captures go to `captures.jsonl`, nudges to `practice.jsonl`, and test results to `test-*.json` in the same folder. Every calibration dot and test target is also added to `points.jsonl`: where in your view it was, the raw error, the error with the calibration of the time, and the spread. That's the data for refining, and for finding where the calibration is off.

gaze/actions/bindings_frame_hmd.json

@@ -0,0 +1,13 @@
+{
+  "action_manifest_version": 0,
+  "controller_type": "frame_hmd",
+  "description": "Frametop gaze: the headset's eye tracker",
+  "name": "Frametop gaze",
+  "bindings": {
+    "/actions/gaze": {
+      "eyetracking": [
+        { "path": "/user/head/eyetracking", "output": "/actions/gaze/in/gaze" }
+      ]
+    }
+  }
+}

gaze/actions/ft_gaze_actions.json

@@ -0,0 +1,14 @@
+{
+  "default_bindings": [
+    { "controller_type": "frame_hmd", "binding_url": "bindings_frame_hmd.json" }
+  ],
+  "action_sets": [
+    { "name": "/actions/gaze", "usage": "single" }
+  ],
+  "actions": [
+    { "name": "/actions/gaze/in/gaze", "type": "eyetracking" }
+  ],
+  "localization": [
+    { "language_tag": "en_US", "/actions/gaze": "Frametop gaze", "/actions/gaze/in/gaze": "Gaze" }
+  ]
+}

gaze/build.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# Build ft-gaze in the dev container on the Frame (gaze/build/ft-gaze; it also runs there).
+# The eye tracking API (IVRInput::GetEyeTrackingDataRelativeToNow) is newer than the header
+# shipped with SteamVR's samples, so this uses the pinned public header ft-screens fetches.
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+"$root/scripts/sync.sh" >/dev/null
+exec "$root/scripts/frame.sh" -C gaze 'set -e; mkdir -p build/include
+openvr=v2.15.6
+[ -f build/include/openvr-$openvr ] || { curl -fsSL "https://raw.githubusercontent.com/ValveSoftware/openvr/$openvr/headers/openvr.h" -o build/include/openvr.h && touch build/include/openvr-$openvr; }
+g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include -I../pointer/common \
+  -o build/ft-gaze ft-gaze.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
+echo "built build/ft-gaze"'

gaze/frametop-gaze.service

@@ -0,0 +1,18 @@
+# Template: the installer replaces @REPO@ with the repo path on the Frame.
+[Unit]
+Description=Frametop gaze service: the eye tracking, corrected, for the pointer's gaze mode
+Documentation=file://@REPO@/gaze/README.md
+# Needs SteamVR's IPC (ft-gaze is an overlay client); it starts and stops with SteamVR.
+After=steamvr.service frametop-pointer.service
+PartOf=steamvr.service
+
+[Service]
+# Host Python; it runs ft-gaze in the dev container (distrobox enter), which quits when
+# the service's pipe to it closes.
+ExecStart=/usr/bin/python3 @REPO@/gaze/ft-gazed
+Restart=on-failure
+RestartSec=3
+TimeoutStopSec=5
+
+[Install]
+WantedBy=steamvr.service

gaze/ft-gaze.cpp

@@ -0,0 +1,474 @@
+// ft-gaze: the headset's eye tracking as rays and Frametop screen pixels (OpenVR overlay
+// client, runs in the dev container). An experiment for gaze input; ft-gazeprobe reads it.
+//
+// Every eye tracker sample (90 Hz) becomes one JSON line on stdout with each gaze source
+// hit-tested against the Frametop screens:
+//
+// Options: -v (log action errors), --watch-stdin (quit when stdin closes).
+//
+//   {"t":<sample time, CLOCK_MONOTONIC_RAW s>,"age":<ms old when read>,"n":<sample counter>,
+//    "head":{"yaw":..,"pitch":..,"hit":HIT},          head forward ray (for head nudging)
+//    "src":{"action":SRC,"mmap1":SRC,"mmap2":SRC}}
+//   SRC = {"hy":..,"hp":..,"hit":HIT} or {"ok":0}    hy/hp: gaze direction relative to the
+//                                                     head, degrees (yaw +left, pitch +up)
+//         mmap1 adds "open":[l,r] (probably eye openness, 0 in a blink) and "dist" (vergence
+//         distance, m); both mmap sets add "lr", the angle between the eyes (deg), which
+//         jumps when the tracker loses an eye, and "eyes":[[hy,hp],[hy,hp]], each eye's own
+//         direction (left, right), for calibrating the eyes separately.
+//   HIT = {"s":<screen>,"x":..,"y":..,"j":[dx/dhy,dy/dhy,dx/dhp,dy/dhp],"dpp":<deg per px>}
+//         or null. x, y are pixels on that screen; j is pixels per degree of head-relative
+//         yaw and pitch there, so a correction in degrees can be turned into pixels and back.
+//
+// Sources:
+//   action  SteamVR input: an "eyetracking" action bound to /user/head/eyetracking, read
+//           with IVRInput::GetEyeTrackingDataRelativeToNow. The supported way.
+//   mmap1/2 /dev/shm/eye-server.mmap, written by SteamVR's eyetracking process for the HMD
+//           driver. Undocumented; the layout below was worked out by reading it and can
+//           change with any SteamVR update. Two sets of per-eye directions in head space
+//           (-Z forward); which one has SteamVR's per-user calibration applied is what the
+//           probe is for. Opened read-only: the other half of the file carries calibration
+//           clicks to the eye tracker, and must never be written.
+//
+// The mmap samples are in head space, 17 ms or so old when they appear, so each is turned
+// into the room with the head pose at its own timestamp, from a short pose history.
+//
+// Screens come from ft-screens (@ft_screens: "screens", "get N"), refreshed 4 times a
+// second in the background. A curved screen is a cylinder toward its front (see OnSurface
+// in screens/vr.cpp).
+#include <openvr.h>
+
+#include "vrmath.h"
+
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <climits>
+#include <cmath>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <deque>
+#include <mutex>
+#include <string>
+#include <thread>
+#include <vector>
+
+#include <fcntl.h>
+#include <sys/mman.h>
+#include <sys/socket.h>
+#include <sys/stat.h>
+#include <sys/un.h>
+#include <time.h>
+#include <unistd.h>
+
+namespace {
+
+using namespace md;
+
+double NowRaw() {
+    timespec ts;
+    clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
+    return ts.tv_sec + ts.tv_nsec * 1e-9;
+}
+
+// --- eye-server.mmap (packed, unaligned: read with memcpy) ---
+constexpr size_t kCounter = 0x38;  // u32, one per sample
+constexpr size_t kTime = 0x157;    // f64, CLOCK_MONOTONIC_RAW seconds
+constexpr size_t kLeft1 = 0x15f, kRight1 = 0x16b;  // set 1: unit vectors, head space
+constexpr size_t kFix1 = 0x18f;    // set 1 fixation point: length is the vergence distance (m)
+constexpr size_t kLeft2 = 0x19b, kRight2 = 0x1a7;  // set 2
+constexpr size_t kOpen = 0x1cb;    // two floats, 0..1: probably eye openness or confidence
+constexpr size_t kNeed = 0x1d3;
+
+struct EyeFile {
+    const uint8_t *p = nullptr;
+    size_t size = 0;
+    bool Open() {
+        const int fd = open("/dev/shm/eye-server.mmap", O_RDONLY | O_CLOEXEC);
+        if (fd < 0) return false;
+        struct stat st {};
+        if (fstat(fd, &st) != 0 || size_t(st.st_size) < kNeed) {
+            close(fd);
+            return false;
+        }
+        void *m = mmap(nullptr, st.st_size, PROT_READ, MAP_SHARED, fd, 0);
+        close(fd);
+        if (m == MAP_FAILED) return false;
+        p = static_cast<const uint8_t *>(m);
+        size = st.st_size;
+        return true;
+    }
+    template <class T> T Get(size_t off) const {
+        T v;
+        std::memcpy(&v, p + off, sizeof v);
+        return v;
+    }
+    Vec3 V(size_t off) const {
+        float f[3];
+        std::memcpy(f, p + off, sizeof f);
+        return {f[0], f[1], f[2]};
+    }
+};
+
+struct EyeSample {
+    uint32_t n = 0;
+    double t = 0;
+    Vec3 left1, right1, fix1, left2, right2;
+    float open[2] = {0, 0};
+};
+
+// A consistent copy: the writer has no seqlock we can use, so read until the counter and
+// timestamp are the same before and after.
+bool ReadSample(const EyeFile &f, EyeSample &s) {
+    for (int attempt = 0; attempt < 4; ++attempt) {
+        const uint32_t n0 = f.Get<uint32_t>(kCounter);
+        const double t0 = f.Get<double>(kTime);
+        std::atomic_thread_fence(std::memory_order_acquire);
+        s.left1 = f.V(kLeft1), s.right1 = f.V(kRight1), s.fix1 = f.V(kFix1);
+        s.left2 = f.V(kLeft2), s.right2 = f.V(kRight2);
+        std::memcpy(s.open, f.p + kOpen, sizeof s.open);
+        std::atomic_thread_fence(std::memory_order_acquire);
+        if (f.Get<uint32_t>(kCounter) == n0 && f.Get<double>(kTime) == t0) {
+            s.n = n0, s.t = t0;
+            return true;
+        }
+    }
+    return false;
+}
+
+// --- Screens from ft-screens ---
+struct Screen {
+    int index = 0;
+    int wpx = 0, hpx = 0;
+    double metres = 0, height = 0, curve = 0;
+    Vec3 c;
+    Basis b;
+};
+
+class Screens {
+public:
+    void Start() {
+        thread_ = std::thread([this] {
+            const int fd = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0);
+            sockaddr_un me{};
+            me.sun_family = AF_UNIX;
+            const std::string name = "ft_gaze." + std::to_string(getpid());
+            std::memcpy(me.sun_path + 1, name.data(), name.size());
+            bind(fd, reinterpret_cast<sockaddr *>(&me), offsetof(sockaddr_un, sun_path) + 1 + name.size());
+            timeval tv{0, 200000};
+            setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv);
+            while (running_) {
+                std::vector<Screen> got;
+                Query(fd, got);
+                {
+                    std::lock_guard<std::mutex> guard(lock_);
+                    screens_ = std::move(got);
+                }
+                std::this_thread::sleep_for(std::chrono::milliseconds(250));
+            }
+            close(fd);
+        });
+    }
+    void Stop() {
+        running_ = false;
+        if (thread_.joinable()) thread_.join();
+    }
+    std::vector<Screen> Get() {
+        std::lock_guard<std::mutex> guard(lock_);
+        return screens_;
+    }
+
+private:
+    static std::string Ask(int fd, const std::string &cmd) {
+        sockaddr_un to{};
+        to.sun_family = AF_UNIX;
+        const char name[] = "ft_screens";
+        std::memcpy(to.sun_path + 1, name, sizeof name - 1);
+        sendto(fd, cmd.data(), cmd.size(), 0, reinterpret_cast<sockaddr *>(&to),
+               offsetof(sockaddr_un, sun_path) + 1 + sizeof name - 1);
+        char buf[1024];
+        const ssize_t n = recv(fd, buf, sizeof buf - 1, 0);
+        if (n <= 0) return "";
+        buf[n] = 0;
+        return buf;
+    }
+    static void Query(int fd, std::vector<Screen> &out) {
+        // "ok <count> <index>:<w>x<h>:<metres> ..."
+        const std::string list = Ask(fd, "screens");
+        if (list.rfind("ok ", 0) != 0) return;
+        const char *p = list.c_str() + 3;
+        int count = 0, used = 0;
+        if (std::sscanf(p, "%d%n", &count, &used) != 1) return;
+        p += used;
+        for (int k = 0; k < count; ++k) {
+            Screen s;
+            if (std::sscanf(p, " %d:%dx%d:%lf%n", &s.index, &s.wpx, &s.hpx, &s.metres, &used) != 4) break;
+            p += used;
+            // "ok x y z  xx xy xz  yx yy yz  zx zy zz  width height curve hand"
+            const std::string g = Ask(fd, "get " + std::to_string(s.index));
+            double v[15];
+            if (std::sscanf(g.c_str(), "ok %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf %lf", &v[0], &v[1],
+                            &v[2], &v[3], &v[4], &v[5], &v[6], &v[7], &v[8], &v[9], &v[10], &v[11], &v[12], &v[13],
+                            &v[14]) != 15)
+                continue;
+            s.c = {v[0], v[1], v[2]};
+            s.b = {{v[3], v[4], v[5]}, {v[6], v[7], v[8]}, {v[9], v[10], v[11]}};
+            s.metres = v[12], s.height = v[13], s.curve = v[14];
+            out.push_back(s);
+        }
+    }
+
+    std::thread thread_;
+    std::atomic<bool> running_{true};
+    std::mutex lock_;
+    std::vector<Screen> screens_;
+};
+
+// Where a ray meets a screen: distance along it, and the pixel. Rays that miss still count,
+// up to 40% of the screen past an edge (`inside` says whether it's on the screen itself):
+// the raw gaze can be 8 degrees or more off near the top and bottom of your view, and a
+// calibration dot near an edge must still get its samples.
+bool HitScreen(const Screen &s, Vec3 from, Vec3 d, double &along, double &px, double &py, bool *inside = nullptr) {
+    const Vec3 p = ToBasis(s.b, from - s.c), q = ToBasis(s.b, d);
+    double u, v;
+    if (s.curve <= 0) {
+        if (q.z >= -1e-6) return false;
+        along = -p.z / q.z;
+        u = p.x + q.x * along, v = p.y + q.y * along;
+    } else {
+        // Cylinder around the vertical line x = 0, z = r (in front of the screen).
+        const double r = s.curve, pz = p.z - r;
+        const double A = q.x * q.x + q.z * q.z, B = 2 * (p.x * q.x + pz * q.z), C = p.x * p.x + pz * pz - r * r;
+        const double disc = B * B - 4 * A * C;
+        if (A < 1e-12 || disc < 0) return false;
+        along = (-B + std::sqrt(disc)) / (2 * A);  // the far wall, seen from inside
+        const double x = p.x + q.x * along, z = p.z + q.z * along;
+        if (r - z <= 0) return false;  // the back half of the cylinder
+        u = std::atan2(x, r - z) * r;
+        v = p.y + q.y * along;
+    }
+    if (along <= 0.05) return false;
+    px = (u / s.metres + 0.5) * s.wpx;
+    py = (0.5 - v / s.height) * s.hpx;
+    if (inside) *inside = px >= 0 && px < s.wpx && py >= 0 && py < s.hpx;
+    return px > -0.4 * s.wpx && px < 1.4 * s.wpx && py > -0.4 * s.hpx && py < 1.4 * s.hpx;
+}
+
+// Head-relative angles of a head-space direction, in degrees (see md::Direction).
+void Angles(Vec3 dHead, double &yaw, double &pitch) {
+    yaw = std::atan2(-dHead.x, -dHead.z) * 180 / M_PI;
+    pitch = std::asin(std::clamp(dHead.y, -1.0, 1.0)) * 180 / M_PI;
+}
+
+// HIT for a head-relative direction (yaw, pitch), with the head at `head`.
+std::string HitJson(const std::vector<Screen> &screens, const vr::HmdMatrix34_t &head, double yaw, double pitch) {
+    const Vec3 o = Position(head);
+    const Screen *best = nullptr;
+    double bestAlong = 1e9, x = 0, y = 0;
+    bool bestInside = false;
+    const Vec3 d = Rotate(head, Direction(yaw, pitch));
+    for (const auto &s : screens) {
+        // A screen the ray is on beats one it only passes near; then the nearest.
+        double along, px, py;
+        bool inside = false;
+        if (!HitScreen(s, o, d, along, px, py, &inside)) continue;
+        if (!best || (inside && !bestInside) || (inside == bestInside && along < bestAlong))
+            best = &s, bestAlong = along, x = px, y = py, bestInside = inside;
+    }
+    if (!best) return "null";
+    // Pixels per degree, from rays a quarter degree off in each direction.
+    constexpr double kStep = 0.25;
+    double j[4] = {0, 0, 0, 0}, along, px, py;
+    if (HitScreen(*best, o, Rotate(head, Direction(yaw + kStep, pitch)), along, px, py))
+        j[0] = (px - x) / kStep, j[1] = (py - y) / kStep;
+    if (HitScreen(*best, o, Rotate(head, Direction(yaw, pitch + kStep)), along, px, py))
+        j[2] = (px - x) / kStep, j[3] = (py - y) / kStep;
+    const double pxPerDeg = std::sqrt(std::fabs(j[0] * j[3] - j[1] * j[2]));
+    char buf[256];
+    std::snprintf(buf, sizeof buf, "{\"s\":%d,\"x\":%.2f,\"y\":%.2f,\"j\":[%.3f,%.3f,%.3f,%.3f],\"dpp\":%.5f}",
+                  best->index, x, y, j[0], j[1], j[2], j[3], pxPerDeg > 1e-6 ? 1 / pxPerDeg : 0.0);
+    return buf;
+}
+
+std::string SrcJson(const std::vector<Screen> &screens, const vr::HmdMatrix34_t &head, Vec3 dHead,
+                    const std::string &extra = "") {
+    double yaw, pitch;
+    Angles(Normalize(dHead), yaw, pitch);
+    char buf[96];
+    std::snprintf(buf, sizeof buf, "{\"hy\":%.4f,\"hp\":%.4f,", yaw, pitch);
+    return buf + extra + "\"hit\":" + HitJson(screens, head, yaw, pitch) + "}";
+}
+
+// Head poses of the last half second, so a sample can use the pose at its own time.
+class PoseHistory {
+public:
+    void Add(double t, const vr::HmdMatrix34_t &m) {
+        poses_.push_back({t, m});
+        while (poses_.size() > 2 && t - poses_.front().t > 0.5) poses_.pop_front();
+    }
+    bool At(double t, vr::HmdMatrix34_t &out) const {
+        if (poses_.empty()) return false;
+        const Entry *best = &poses_.back();
+        for (const auto &e : poses_)
+            if (std::fabs(e.t - t) < std::fabs(best->t - t)) best = &e;
+        out = best->m;
+        return true;
+    }
+
+private:
+    struct Entry {
+        double t;
+        vr::HmdMatrix34_t m;
+    };
+    std::deque<Entry> poses_;
+};
+
+std::string ExeDir() {
+    char buf[PATH_MAX];
+    const ssize_t n = readlink("/proc/self/exe", buf, sizeof buf - 1);
+    if (n <= 0) return ".";
+    buf[n] = 0;
+    std::string p(buf);
+    return p.substr(0, p.rfind('/'));
+}
+
+}  // namespace
+
+int main(int argc, char **argv) {
+    bool verbose = false, watchStdin = false;
+    for (int i = 1; i < argc; ++i) {
+        if (std::strcmp(argv[i], "-v") == 0) verbose = true;
+        if (std::strcmp(argv[i], "--watch-stdin") == 0) watchStdin = true;
+    }
+    // --watch-stdin: quit when stdin closes. The probe runs us through distrobox, which
+    // passes neither its signals nor a closed stdout on to us, but does pass stdin's end.
+    std::atomic<bool> stdinClosed{false};
+    if (watchStdin)
+        std::thread([&stdinClosed] {
+            char c[256];
+            while (read(0, c, sizeof c) > 0) {
+            }
+            stdinClosed = true;
+        }).detach();
+    vr::EVRInitError err = vr::VRInitError_None;
+    vr::VR_Init(&err, vr::VRApplication_Overlay);
+    if (err != vr::VRInitError_None) {
+        std::fprintf(stderr, "ft-gaze: SteamVR: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
+        return 1;
+    }
+    auto *sys = vr::VRSystem();
+    auto *input = vr::VRInput();
+
+    // The build puts the binary in gaze/build; the manifest is in gaze/actions.
+    const std::string manifest = ExeDir() + "/../actions/ft_gaze_actions.json";
+    char real[PATH_MAX];
+    const vr::EVRInputError me = input->SetActionManifestPath(realpath(manifest.c_str(), real) ? real : manifest.c_str());
+    vr::VRActionHandle_t gaze = vr::k_ulInvalidActionHandle;
+    vr::VRActionSetHandle_t set = vr::k_ulInvalidActionSetHandle;
+    input->GetActionHandle("/actions/gaze/in/gaze", &gaze);
+    input->GetActionSetHandle("/actions/gaze", &set);
+    std::fprintf(stderr, "ft-gaze: action manifest %s: error %d\n", manifest.c_str(), int(me));
+
+    EyeFile eyes;
+    const bool haveMmap = eyes.Open();
+    std::fprintf(stderr, "ft-gaze: eye-server.mmap %s\n", haveMmap ? "open" : "not available");
+
+    Screens screens;
+    screens.Start();
+    PoseHistory history;
+    uint32_t lastN = 0;
+    double lastEmit = 0;
+    int actionErrors = 0;
+    vr::EVRInputError lastActionError = vr::VRInputError_None;
+
+    while (true) {
+        const double now = NowRaw();
+        vr::TrackedDevicePose_t hp;
+        sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &hp, 1);
+        if (hp.bPoseIsValid) history.Add(now, hp.mDeviceToAbsoluteTracking);
+
+        // One line per new eye sample, or at 90 Hz without the mmap.
+        EyeSample s;
+        bool fresh = false;
+        if (haveMmap && ReadSample(eyes, s) && s.n != lastN) fresh = true, lastN = s.n;
+        if (!haveMmap && now - lastEmit >= 1.0 / 90) fresh = true, s.t = now;
+
+        if (fresh && hp.bPoseIsValid) {
+            lastEmit = now;
+            const auto list = screens.Get();
+            const vr::HmdMatrix34_t &headNow = hp.mDeviceToAbsoluteTracking;
+            vr::HmdMatrix34_t headThen = headNow;
+            if (haveMmap) history.At(s.t, headThen);
+
+            // SteamVR's action: a room-space origin and fixation point, turned into the head
+            // frame so every source reports the same kind of angles.
+            std::string action = "{\"ok\":0}";
+            vr::VRActiveActionSet_t active{};
+            active.ulActionSet = set;
+            active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin;
+            input->UpdateActionState(&active, sizeof active, 1);
+            vr::VREyeTrackingData_t e{};
+            const vr::EVRInputError ae =
+                input->GetEyeTrackingDataRelativeToNow(gaze, vr::TrackingUniverseStanding, 0, &e, sizeof e);
+            if (ae == vr::VRInputError_None && e.bActive && e.bValid) {
+                const Vec3 o{e.vGazeOrigin.v[0], e.vGazeOrigin.v[1], e.vGazeOrigin.v[2]};
+                const Vec3 t{e.vGazeTarget.v[0], e.vGazeTarget.v[1], e.vGazeTarget.v[2]};
+                const Vec3 dHead = RotateInverse(headNow, Normalize(t - o));
+                char extra[96];
+                std::snprintf(extra, sizeof extra, "\"tracked\":%d,\"dist\":%.3f,", int(e.bTracked), Length(t - o));
+                action = SrcJson(list, headNow, dHead, extra);
+            } else if (ae != lastActionError || (verbose && ++actionErrors % 90 == 1)) {
+                std::fprintf(stderr, "ft-gaze: action: error %d active %d valid %d\n", int(ae), int(e.bActive),
+                             int(e.bValid));
+                lastActionError = ae;
+            }
+
+            std::string m1 = "{\"ok\":0}", m2 = m1;
+            if (haveMmap) {
+                // lr: the angle between the two eyes' directions. It's a fraction of a degree
+                // normally; when the tracker loses one eye (or during a blink) it jumps.
+                auto lr = [](Vec3 l, Vec3 r) {
+                    return std::acos(std::clamp(Dot(Normalize(l), Normalize(r)), -1.0, 1.0)) * 180 / M_PI;
+                };
+                auto eyes = [](Vec3 l, Vec3 r) {
+                    double ly, lp, ry, rp;
+                    Angles(Normalize(l), ly, lp);
+                    Angles(Normalize(r), ry, rp);
+                    char b[96];
+                    std::snprintf(b, sizeof b, "\"eyes\":[[%.4f,%.4f],[%.4f,%.4f]],", ly, lp, ry, rp);
+                    return std::string(b);
+                };
+                char extra[128];
+                std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1),
+                              s.open[0], s.open[1], lr(s.left1, s.right1));
+                m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1));
+                std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2));
+                m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2));
+            }
+
+            double yaw, pitch;
+            const Vec3 f = Rotate(headNow, {0, 0, -1});
+            yaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
+            pitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
+            std::printf("{\"t\":%.5f,\"age\":%.1f,\"n\":%u,\"head\":{\"yaw\":%.4f,\"pitch\":%.4f,\"hit\":%s},"
+                        "\"src\":{\"action\":%s,\"mmap1\":%s,\"mmap2\":%s}}\n",
+                        s.t, (now - s.t) * 1000, s.n, yaw, pitch, HitJson(list, headNow, 0, 0).c_str(), action.c_str(),
+                        m1.c_str(), m2.c_str());
+            if (std::fflush(stdout) != 0) break;  // the reader went away
+        }
+
+        vr::VREvent_t ev;
+        bool quit = false;
+        while (sys->PollNextEvent(&ev, sizeof ev))
+            if (ev.eventType == vr::VREvent_Quit) quit = true;
+        if (quit) {
+            sys->AcknowledgeQuit_Exiting();
+            break;
+        }
+        if (stdinClosed) break;
+        std::this_thread::sleep_for(std::chrono::milliseconds(2));
+    }
+    screens.Stop();
+    vr::VR_Shutdown();
+    return 0;
+}

gaze/ft-gazectl

@@ -0,0 +1,54 @@
+#!/usr/bin/python3
+"""ft-gazectl: turn the gaze pointer on or off, and ask the gaze service how it's doing.
+
+  ft-gazectl on|off|toggle   gaze mode in the pointer helper (until it restarts; the
+                             setting is POINTER_GAZE in ~/.config/frametop.conf)
+  ft-gazectl status          the gaze service (ft-gazed): samples, lessons, the correction
+  ft-gazectl forget          drop what the pointer's lessons taught (the calibration stays)
+  ft-gazectl reload          the service reads calibration.json again
+"""
+
+import json
+import os
+import socket
+import sys
+
+
+def ask(name, msg, timeout=1.0):
+    s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC)
+    s.bind(f"\0ft_gazectl.{os.getpid()}")
+    s.settimeout(timeout)
+    try:
+        s.sendto(msg.encode(), "\0" + name)
+        return s.recv(4096).decode()
+    except ConnectionRefusedError:
+        return None
+    except socket.timeout:
+        return ""
+    finally:
+        s.close()
+
+
+def main():
+    cmd = sys.argv[1] if len(sys.argv) > 1 else "status"
+    if cmd in ("on", "off", "toggle"):
+        r = ask("ft_pointer_helper", f"gaze {cmd}")
+        if r is None:
+            sys.exit("the pointer helper isn't running")
+        print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)")
+        if ask("ft_gazed", "status", 0.3) is None:
+            print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow")
+    elif cmd in ("status", "forget", "reload"):
+        r = ask("ft_gazed", cmd)
+        if r is None:
+            sys.exit("the gaze service (ft-gazed) isn't running")
+        try:
+            print(json.dumps(json.loads(r), indent=1))
+        except ValueError:
+            print(r)
+    else:
+        sys.exit(__doc__)
+
+
+if __name__ == "__main__":
+    main()

gaze/ft-gazed

@@ -0,0 +1,393 @@
+#!/usr/bin/python3
+"""ft-gazed: the gaze service. The headset's eye tracking, corrected, for the pointer.
+
+Runs ft-gaze (in the dev container), and for every eye tracker sample (90 Hz):
+
+  1. drops blinks: both eyes' openness under half its running median (each eye its own).
+     With the default source, mmap set 1, that's all: set 1 is SteamVR's combined gaze,
+     which keeps going when the tracker loses one eye (its two directions stay together).
+     Set 2's combined direction is the mean of the eyes' own, so with one eye lost it's
+     off by half of whatever that eye reads (9 to 14 degrees apart were seen): with set 2,
+     samples with an eye under its floor, or the angle between the eyes jumping more than
+     1.5 degrees from its median, are dropped too;
+  2. smooths it with a fixation lock (the running mean of the current fixation, 1 degree);
+  3. corrects it: the calibration from ft-gazeprobe (calibration.json, reloaded when the
+     probe changes it) plus what the pointer's corrections have taught since (LiveCorrection,
+     saved in pointer-lessons.json);
+  4. sends it to the pointer helper: "gz <yaw> <pitch> <raw yaw> <raw pitch>", head-relative
+     degrees (yaw +left, pitch +up). The helper uses it only in gaze mode.
+
+Lessons come back from the helper: when you nudge the gaze-placed pointer with the mouse and
+click, it sends "lesson <raw yaw> <raw pitch> <true yaw> <true pitch>": where the raw gaze
+was when the mouse took over, and where the pointer was when you clicked (you were looking
+there). The gap is the tracker's error there, and it's learned, unless it's more than
+LESSON_MAX degrees past the correction (then it wasn't a nudge onto what you looked at).
+
+SteamVR's eye tracking log is followed for the headset going on (its eye model starts over,
+and the error moves): lessons from before count less then, so the first few after it
+relearn the offset.
+
+Nothing here writes to SteamVR, its eye tracker, or its files: ft-gaze reads the eye
+tracker's shared memory read-only.
+
+Control socket: abstract unix datagram "@ft_gazed":
+  lesson <rhy> <rhp> <thy> <thp>   from the pointer helper (see above)
+  status                           reply: one JSON object
+  forget                           drop what the lessons taught (the calibration stays)
+  reload                           read calibration.json again
+
+Options: --source action|mmap1|mmap2 (default mmap1; set 2 was a little quieter in the probe,
+but loses the pointer whenever the tracker loses an eye), -v (a status line every
+5 s on stderr), --to NAME (send the gaze to the abstract socket @NAME instead of the pointer
+helper; for testing: a helper without gaze mode forwards what it doesn't know to its driver).
+"""
+
+import argparse
+import json
+import os
+import selectors
+import signal
+import socket
+import statistics
+import subprocess
+import sys
+import time
+from collections import deque
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from gazecal import DEFAULT_MODEL, MODELS, STATE, Correction, Fixation, LiveCorrection, SteamEyeLog  # noqa: E402
+
+REPO = Path(__file__).resolve().parents[1]
+HELPER = REPO / "gaze" / "build" / "ft-gaze"
+ME = "\0ft_gazed"
+POINTER = "\0ft_pointer_helper"
+CALIBRATION = STATE / "calibration.json"
+LESSONS = STATE / "pointer-lessons.json"
+LESSON_LOG = STATE / "pointer-lessons.jsonl"
+LESSON_MAX = 8.0  # degrees past the correction
+RETRY = 3.0       # seconds before starting ft-gaze again
+
+
+class PointerLessons(LiveCorrection):
+    """LiveCorrection, with the offset held back: one lesson moves the whole correction by a
+    third of what it measured, not all of it (two alike, half; three, three fifths). In the
+    probe, clicks came thick and fast on a stale calibration, where the error was mostly one
+    offset. Here lessons are few and the calibration is often fresh: in the first live test a
+    6 degree lesson shifted everything 6 degrees, and the next target, 10 degrees away and
+    1.4 off before, was 7.1 off (3.6 with this). Near the lesson, the kernel still takes up
+    most of it (5.1 of the 6 degrees)."""
+
+    RIDGE = [2.0] + LiveCorrection.RIDGE[1:]
+
+
+def log(msg):
+    print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
+
+
+class Service:
+    def __init__(self, source, verbose, to=POINTER):
+        self.source, self.verbose, self.to = source, verbose, to
+        STATE.mkdir(parents=True, exist_ok=True)
+        self.base = Correction()
+        self.mode = DEFAULT_MODEL
+        self.cal_mtime = None
+        self.live = PointerLessons()
+        self.dirty = False
+        self.load_calibration()
+        self.load_lessons()
+        self.steam = SteamEyeLog()
+        self.steam.poll()
+        self.live.wear_time = self.steam.worn()
+        self.live.refit(self.base, self.mode)
+        self.fix = Fixation(radius=1.0)
+        self.opens = (deque(maxlen=90), deque(maxlen=90))  # left, right
+        self.vergence = deque(maxlen=90)
+        self.counts = {"samples": 0, "sent": 0, "blinks": 0, "one_eye": 0, "dropped": 0, "lessons_taken": 0, "refused": 0}
+        self.last_sample = 0.0
+        self.last = None
+
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
+        self.sock.bind(ME)
+        self.out = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
+        self.sel = selectors.DefaultSelector()
+        self.sel.register(self.sock, selectors.EVENT_READ, "control")
+        self.proc = None
+        self.buf = b""
+        self.restart_at = 0.0
+        self.running = True
+
+    # --- Calibration and lessons ---
+
+    def load_calibration(self):
+        try:
+            mtime = CALIBRATION.stat().st_mtime
+            d = json.loads(CALIBRATION.read_text())
+        except (OSError, ValueError):
+            return
+        self.cal_mtime = mtime
+        if self.source in d:
+            self.base.from_json(d[self.source])
+        mode = d.get("_meta", {}).get("model")
+        self.mode = mode if mode in MODELS else DEFAULT_MODEL
+        log(f"calibration: {self.mode}, {self.base.samples} samples")
+
+    def load_lessons(self):
+        try:
+            d = json.loads(LESSONS.read_text())
+            if d.get("source") == self.source:
+                self.live.samples = d.get("samples", [])[-PointerLessons.KEEP:]
+        except (OSError, ValueError):
+            pass
+        log(f"{len(self.live.samples)} lessons")
+
+    def save_lessons(self):
+        tmp = LESSONS.with_suffix(".tmp")
+        tmp.write_text(json.dumps({"source": self.source, "samples": self.live.samples}))
+        tmp.replace(LESSONS)
+        self.dirty = False
+
+    def correction(self, hy, hp):
+        by, bp = self.base.get(hy, hp, self.mode)
+        ly, lp = self.live.get(hy, hp)
+        return by + ly, bp + lp
+
+    def lesson(self, rhy, rhp, thy, thp):
+        dy, dp = thy - rhy, thp - rhp  # the whole error there
+        cy, cp = self.correction(rhy, rhp)
+        left = ((dy - cy) ** 2 + (dp - cp) ** 2) ** 0.5
+        rec = {"time": time.time(), "source": self.source, "model": self.mode, "raw": [rhy, rhp],
+               "true": [thy, thp], "correction": [cy, cp], "lesson_deg": left, "wear": self.steam.worn()}
+        if left > LESSON_MAX:
+            rec["refused"] = f"more than {LESSON_MAX} deg past the correction"
+            self.counts["refused"] += 1
+        else:
+            self.live.add({"time": rec["time"], "hy": rhy, "hp": rhp, "dy": dy, "dp": dp, "wy": 1.0, "wp": 1.0,
+                           "how": "pointer"}, self.base, self.mode)
+            self.counts["lessons_taken"] += 1
+            self.dirty = True
+        try:
+            with open(LESSON_LOG, "a") as f:
+                f.write(json.dumps(rec) + "\n")
+        except OSError as e:
+            log(f"lesson log: {e}")
+        return rec
+
+    # --- ft-gaze ---
+
+    def start_helper(self):
+        if not HELPER.exists():
+            log(f"ft-gaze isn't built: run {REPO}/gaze/build.sh")
+            self.restart_at = time.monotonic() + 30
+            return
+        env = dict(os.environ)
+        env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"  # podman needs the real one
+        subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False)
+        distrobox = Path.home() / ".local" / "bin" / "distrobox"
+        # ft-gaze quits when its stdin closes: the one thing distrobox passes on.
+        self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin"], env=env,
+                                     stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+                                     start_new_session=True)
+        os.set_blocking(self.proc.stdout.fileno(), False)
+        os.set_blocking(self.proc.stderr.fileno(), False)
+        self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
+        self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr")
+        self.buf = b""
+        log("ft-gaze started")
+
+    def stop_helper(self):
+        if not self.proc:
+            return
+        for f in (self.proc.stdout, self.proc.stderr):
+            try:
+                self.sel.unregister(f)
+            except (KeyError, ValueError):
+                pass
+        if self.proc.stdin and not self.proc.stdin.closed:
+            self.proc.stdin.close()
+        try:
+            self.proc.wait(timeout=2)
+        except subprocess.TimeoutExpired:
+            try:
+                os.killpg(self.proc.pid, signal.SIGTERM)
+            except ProcessLookupError:
+                pass
+        self.proc = None
+
+    def read_stdout(self):
+        try:
+            data = os.read(self.proc.stdout.fileno(), 65536)
+        except BlockingIOError:
+            return
+        if not data:
+            log(f"ft-gaze stopped (exit {self.proc.poll()}); again in {RETRY:.0f} s")
+            self.stop_helper()
+            self.restart_at = time.monotonic() + RETRY
+            return
+        self.buf += data
+        *lines, self.buf = self.buf.split(b"\n")
+        for line in lines:
+            try:
+                self.on_sample(json.loads(line))
+            except (ValueError, KeyError, TypeError) as e:
+                log(f"bad sample: {e}")
+
+    def read_stderr(self):
+        try:
+            data = os.read(self.proc.stderr.fileno(), 65536)
+        except BlockingIOError:
+            return
+        for line in data.decode("utf-8", "replace").splitlines():
+            if line.strip():
+                log(line)
+
+    def on_sample(self, s):
+        src = s["src"].get(self.source) or {}
+        if "hy" not in src:
+            return
+        self.counts["samples"] += 1
+        self.last_sample = time.monotonic()
+        m1 = s["src"].get("mmap1") or {}
+        o = m1.get("open")
+        lr = src.get("lr", m1.get("lr"))
+        # Blinks and lost eyes, judged against the last second (see steady_samples: relative,
+        # because the lids come down looking down, and the vergence depends on distance).
+        # An eye's floor comes from its good readings, so a lost eye doesn't drag it to 0.
+        low = [False, False]
+        if o and len(o) == 2:
+            for k in (0, 1):
+                hist = self.opens[k]
+                good = [v for v in hist if v >= 0.12]
+                floor = max(0.12, 0.5 * statistics.median(good)) if len(good) >= 30 else 0.12
+                low[k] = o[k] < floor
+                hist.append(o[k])
+        if all(low):
+            self.counts["blinks"] += 1
+            return
+        if any(low):
+            self.counts["one_eye"] += 1
+        if self.source == "mmap2":
+            jump = (lr is not None and len(self.vergence) >= 30
+                    and abs(lr - statistics.median(self.vergence)) > 1.5)
+            if lr is not None and not any(low):
+                self.vergence.append(lr)
+            if any(low) or jump:
+                self.counts["dropped"] += 1
+                return
+        # The fixation lock works in degrees here (1 degree per "pixel").
+        fy, fp = self.fix(src["hy"], src["hp"], s["t"], 1.0)
+        cy, cp = self.correction(fy, fp)
+        self.last = (fy + cy, fp + cp, fy, fp)
+        try:
+            self.out.sendto(f"gz {fy + cy:.3f} {fp + cp:.3f} {fy:.3f} {fp:.3f}".encode(), self.to)
+            self.counts["sent"] += 1
+        except OSError:
+            pass  # the pointer helper isn't running
+
+    # --- Control ---
+
+    def on_control(self):
+        while True:
+            try:
+                data, addr = self.sock.recvfrom(512)
+            except BlockingIOError:
+                return
+            words = data.decode("utf-8", "replace").split()
+            reply = None
+            if words[:1] == ["lesson"] and len(words) == 5:
+                try:
+                    rec = self.lesson(*map(float, words[1:]))
+                    reply = "refused" if "refused" in rec else f"ok {rec['lesson_deg']:.2f}"
+                    log(f"lesson {rec['lesson_deg']:.2f} deg at {rec['raw'][0]:+.1f},{rec['raw'][1]:+.1f}"
+                        + (f": {rec['refused']}" if "refused" in rec else ""))
+                except ValueError:
+                    reply = "error bad lesson"
+            elif words[:1] == ["status"]:
+                reply = json.dumps(self.status())
+            elif words[:1] == ["forget"]:
+                self.live = PointerLessons()
+                self.live.wear_time = self.steam.worn()
+                self.save_lessons()
+                reply = "ok"
+            elif words[:1] == ["reload"]:
+                self.load_calibration()
+                self.live.refit(self.base, self.mode)
+                reply = "ok"
+            else:
+                reply = "error unknown command"
+            if reply and addr:
+                try:
+                    self.sock.sendto(reply.encode(), addr)
+                except OSError:
+                    pass
+
+    def status(self):
+        ly, lp = self.live.offset()
+        return {"source": self.source, "model": self.mode, "calibration_samples": self.base.samples,
+                "lessons": len(self.live.samples), "lesson_offset": [round(ly, 3), round(lp, 3)],
+                "ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
+                if self.last_sample else None, "headset_on": self.steam.wearing(), "headset_on_since": self.steam.worn(),
+                "last": [round(v, 2) for v in self.last] if self.last else None, **self.counts}
+
+    def periodic(self):
+        if self.steam.poll() or self.steam.worn() != self.live.wear_time:
+            if self.steam.worn() != self.live.wear_time:
+                log("headset on again: older lessons count less until new ones come in")
+            self.live.wear_time = self.steam.worn()
+            self.live.refit(self.base, self.mode)
+        try:
+            mtime = CALIBRATION.stat().st_mtime
+        except OSError:
+            mtime = None
+        if mtime != self.cal_mtime:
+            self.load_calibration()
+            self.live.refit(self.base, self.mode)
+        if self.dirty:
+            self.save_lessons()
+
+    def run(self):
+        next_periodic = time.monotonic()
+        next_verbose = time.monotonic() + 5
+        while self.running:
+            now = time.monotonic()
+            if not self.proc and now >= self.restart_at:
+                self.start_helper()
+            for key, _ in self.sel.select(timeout=0.5):
+                if key.data == "control":
+                    self.on_control()
+                elif key.data == "stdout" and self.proc:
+                    self.read_stdout()
+                elif key.data == "stderr" and self.proc:
+                    self.read_stderr()
+            if now >= next_periodic:
+                self.periodic()
+                next_periodic = now + 1.0
+            if self.verbose and now >= next_verbose:
+                log(json.dumps(self.status()))
+                next_verbose = now + 5
+        self.stop_helper()
+        if self.dirty:
+            self.save_lessons()
+
+
+def main():
+    ap = argparse.ArgumentParser(description="The gaze service: corrected eye tracking for the pointer")
+    ap.add_argument("--source", choices=["action", "mmap1", "mmap2"], default="mmap1")
+    ap.add_argument("-v", "--verbose", action="store_true")
+    ap.add_argument("--to", default="ft_pointer_helper", help="abstract socket to send the gaze to")
+    args = ap.parse_args()
+    try:
+        service = Service(args.source, args.verbose, "\0" + args.to)
+    except OSError as e:
+        log(f"can't bind @ft_gazed (already running?): {e}")
+        sys.exit(1)
+
+    def stop(*_):
+        service.running = False
+    signal.signal(signal.SIGTERM, stop)
+    signal.signal(signal.SIGINT, stop)
+    service.run()
+
+
+if __name__ == "__main__":
+    main()

gaze/gazecal.py

@@ -0,0 +1,520 @@
+"""gazecal: gaze calibration shared by ft-gazeprobe and ft-gazed.
+
+The correction models (Correction: the calibration fitted from calibration dots;
+LiveCorrection: what clicks teach on the fly, on top of it), the smoothing filters, the
+blink and dropout filter for one look at a spot, and SteamEyeLog, which follows SteamVR's
+eye tracking log. Angles are head-relative degrees (yaw +left, pitch +up), as ft-gaze
+reports them.
+"""
+
+import math
+import os
+import statistics
+import time
+from pathlib import Path
+
+STATE = Path.home() / ".local" / "state" / "frametop" / "gaze"
+
+# --- Small math ---------------------------------------------------------------------
+
+def px_from_deg(j, dy, dp):
+    """Pixels for a head-relative change of (yaw, pitch) degrees, from ft-gaze's Jacobian."""
+    return j[0] * dy + j[2] * dp, j[1] * dy + j[3] * dp
+
+
+def deg_from_px(j, dx, dy):
+    """Head-relative (yaw, pitch) degrees for a pixel offset: the Jacobian's inverse."""
+    det = j[0] * j[3] - j[2] * j[1]
+    if abs(det) < 1e-9:
+        return 0.0, 0.0
+    return (j[3] * dx - j[2] * dy) / det, (-j[1] * dx + j[0] * dy) / det
+
+
+class OneEuro:
+    """One Euro filter (Casiez et al. 2012): smooth when still, quick when moving.
+    `scale` turns the input's units into degrees, so beta is per degree a second."""
+
+    def __init__(self, min_cutoff=1.0, beta=0.01, d_cutoff=1.0):
+        self.min_cutoff, self.beta, self.d_cutoff = min_cutoff, beta, d_cutoff
+        self.x = self.dx = self.t = None
+
+    @staticmethod
+    def alpha(cutoff, dt):
+        tau = 1.0 / (2 * math.pi * cutoff)
+        return 1.0 / (1.0 + tau / dt)
+
+    def __call__(self, x, t, scale=1.0):
+        if self.t is None or t <= self.t or t - self.t > 0.5:
+            self.x, self.dx, self.t = x, 0.0, t
+            return x
+        dt = t - self.t
+        dx = (x - self.x) / dt
+        a_d = self.alpha(self.d_cutoff, dt)
+        self.dx = a_d * dx + (1 - a_d) * self.dx
+        cutoff = self.min_cutoff + self.beta * abs(self.dx) * scale
+        a = self.alpha(cutoff, dt)
+        self.x = a * x + (1 - a) * self.x
+        self.t = t
+        return self.x
+
+
+class Fixation:
+    """Dispersion-based fixations: while gaze stays within `radius` degrees of the current
+    fixation's mean, the output is that mean, so the dot sits still; two samples in a row
+    outside it start a new fixation there, so a glance elsewhere moves the dot at once."""
+
+    def __init__(self, radius=1.0):
+        self.radius = radius
+        self.reset()
+
+    def reset(self):
+        self.sum = [0.0, 0.0]
+        self.count = 0
+        self.outside = []
+        self.last_t = None
+
+    def __call__(self, x, y, t, dpp):
+        if self.last_t is not None and (t <= self.last_t or t - self.last_t > 0.5):
+            self.reset()
+        self.last_t = t
+        if self.count:
+            mx, my = self.sum[0] / self.count, self.sum[1] / self.count
+            if math.hypot(x - mx, y - my) * dpp > self.radius:
+                self.outside.append((x, y))
+                if len(self.outside) < 2:
+                    return mx, my  # one stray sample: probably noise
+                self.sum = [sum(p[0] for p in self.outside), sum(p[1] for p in self.outside)]
+                self.count = len(self.outside)
+                self.outside = []
+                return self.sum[0] / self.count, self.sum[1] / self.count
+        self.outside = []
+        if self.count >= 90:  # the last second or so: a slow drift still gets followed
+            self.sum = [self.sum[0] * 89 / 90, self.sum[1] * 89 / 90]
+            self.count = 89
+        self.sum[0] += x
+        self.sum[1] += y
+        self.count += 1
+        return self.sum[0] / self.count, self.sum[1] / self.count
+
+
+MODELS = ["none", "offset", "affine", "affine+grid", "quadratic", "quadratic+grid"]
+DEFAULT_MODEL = "quadratic"
+
+
+class Correction:
+    """Gaze correction in degrees, looked up by where in your view you're looking
+    (head-relative yaw and pitch, hy and hp):
+
+      offset          one (yaw, pitch) offset everywhere
+      affine          plus a straight-line change across the view: a gain and a tilt
+      quadratic       plus curvature (hy*hp, hy^2, hp^2): the second-order polynomial video
+                      eye trackers usually calibrate with. The tracker's error grows as
+                      the eye turns away from the centre (on the Frame it overstates
+                      vertical movement, more the further up or down you look, and looking
+                      up adds a sideways error), and a straight line can only follow part
+                      of that
+      ...+grid        plus a bilinear grid of what's left every 10 degrees
+
+    Coefficients: C @ f, f = [1, x, y, x*y, x^2, y^2] with x = hy/30, y = hp/30; the
+    terms a model doesn't use are 0. A polynomial runs away outside the spots it was fitted
+    on, so its input is clamped to the range of view it has seen, plus a margin.
+    """
+
+    YAWS = list(range(-40, 41, 10))
+    PITCHES = list(range(-30, 31, 10))
+    NF = 6
+    MARGIN = 3.0  # degrees past the fitted range that the polynomial still follows
+
+    def __init__(self):
+        self.reset()
+
+    def reset(self):
+        self.a = [[0.0] * self.NF, [0.0] * self.NF]
+        self.grid = [[[0.0, 0.0] for _ in self.PITCHES] for _ in self.YAWS]
+        self.samples = 0
+        self.range = None  # [hy min, hy max, hp min, hp max] of the samples so far
+
+    @staticmethod
+    def base(mode):
+        return mode.split("+")[0]
+
+    def clamp(self, hy, hp):
+        if not self.range:
+            return hy, hp
+        y0, y1, p0, p1 = self.range
+        m = self.MARGIN
+        return min(max(hy, y0 - m), y1 + m), min(max(hp, p0 - m), p1 + m)
+
+    def features(self, hy, hp, mode):
+        kind = self.base(mode)
+        if kind == "offset":
+            return [1.0, 0.0, 0.0, 0.0, 0.0, 0.0]
+        hy, hp = self.clamp(hy, hp)
+        x, y = hy / 30.0, hp / 30.0
+        if kind == "affine":
+            return [1.0, x, y, 0.0, 0.0, 0.0]
+        return [1.0, x, y, x * y, x * x, y * y]
+
+    def extend(self, hy, hp):
+        if self.range is None:
+            self.range = [hy, hy, hp, hp]
+        else:
+            r = self.range
+            self.range = [min(r[0], hy), max(r[1], hy), min(r[2], hp), max(r[3], hp)]
+
+    def weights(self, hy, hp):
+        def cell(v, axis):
+            v = min(max(v, axis[0]), axis[-1])
+            i = min(int((v - axis[0]) // 10), len(axis) - 2)
+            return i, (v - axis[i]) / 10.0
+        i, fy = cell(hy, self.YAWS)
+        k, fp = cell(hp, self.PITCHES)
+        return [((i, k), (1 - fy) * (1 - fp)), ((i + 1, k), fy * (1 - fp)),
+                ((i, k + 1), (1 - fy) * fp), ((i + 1, k + 1), fy * fp)]
+
+    def get(self, hy, hp, mode):
+        if mode == "none":
+            return 0.0, 0.0
+        f = self.features(hy, hp, mode)
+        cy = sum(a * b for a, b in zip(self.a[0], f))
+        cp = sum(a * b for a, b in zip(self.a[1], f))
+        if mode.endswith("+grid"):
+            for (i, k), w in self.weights(hy, hp):
+                cy += w * self.grid[i][k][0]
+                cp += w * self.grid[i][k][1]
+        return cy, cp
+
+    def learn(self, hy, hp, dy, dp, mode, rate):
+        """One sample: the correction here should have been (dy, dp) degrees more.
+        Normalized LMS for the polynomial; the grid takes half when it's on."""
+        if mode == "none":
+            return
+        self.samples += 1
+        self.extend(hy, hp)
+        grid = mode.endswith("+grid")
+        share = rate * 0.5 if grid else rate
+        f = self.features(hy, hp, mode)
+        norm = sum(v * v for v in f)
+        for row, d in ((self.a[0], dy), (self.a[1], dp)):
+            for n in range(self.NF):
+                row[n] += share * d * f[n] / norm
+        if grid:
+            rest = rate - share
+            for (i, k), w in self.weights(hy, hp):
+                self.grid[i][k][0] += rest * w * dy
+                self.grid[i][k][1] += rest * w * dp
+
+    def fit(self, points, mode, ridge=0.05, smooth=0.3):
+        """Batch fit from (hy, hp, dy, dp) points, each the whole error there (degrees)."""
+        self.reset()
+        if mode == "none" or not points:
+            return
+        self.samples = len(points)
+        for p in points:
+            self.extend(p[0], p[1])
+        kind = self.base(mode)
+        used = {"offset": 1, "affine": 3, "quadratic": 6}[kind]
+        if used > 1 and len(points) < used + 2:  # too few spots for this many terms
+            kind, used = ("affine", 3) if len(points) >= 5 else ("offset", 1)
+        if kind == "offset":
+            self.a[0][0] = statistics.fmean(p[2] for p in points)
+            self.a[1][0] = statistics.fmean(p[3] for p in points)
+        else:
+            # Least squares, with a little ridge on everything but the offset, so a
+            # lopsided set of spots can't bend it far.
+            X = [self.features(p[0], p[1], kind)[:used] for p in points]
+            M = [[sum(x[r] * x[c] for x in X) + (ridge * len(X) if r == c and r else 0.0) for c in range(used)]
+                 for r in range(used)]
+            for out, col in ((self.a[0], 2), (self.a[1], 3)):
+                b = [sum(x[r] * p[col] for x, p in zip(X, points)) for r in range(used)]
+                out[:used] = solve(M, b)
+        if not mode.endswith("+grid"):
+            return
+        # Each node: the weighted mean of what the polynomial left over near it, shrunk toward 0.
+        acc = [[[0.0, 0.0, 0.0] for _ in self.PITCHES] for _ in self.YAWS]
+        for hy, hp, dy, dp in points:
+            ly, lp = self.get(hy, hp, kind)
+            for (i, k), w in self.weights(hy, hp):
+                acc[i][k][0] += w * (dy - ly)
+                acc[i][k][1] += w * (dp - lp)
+                acc[i][k][2] += w
+        for i in range(len(self.YAWS)):
+            for k in range(len(self.PITCHES)):
+                sy, sp, sw = acc[i][k]
+                self.grid[i][k] = [sy / (sw + smooth), sp / (sw + smooth)]
+
+    def offset(self):
+        return self.a[0][0], self.a[1][0]
+
+    def to_json(self):
+        return {"coef": self.a, "range": self.range, "grid": self.grid, "samples": self.samples}
+
+    def from_json(self, d):
+        self.reset()
+        a = d.get("coef") or d.get("affine")  # "affine": the 3-term version of this file
+        if a and len(a) == 2 and all(len(r) in (3, self.NF) for r in a):
+            self.a = [list(map(float, r)) + [0.0] * (self.NF - len(r)) for r in a]
+        grid = d.get("grid")
+        if grid and len(grid) == len(self.YAWS) and all(len(r) == len(self.PITCHES) for r in grid):
+            self.grid = [[list(c) for c in row] for row in grid]
+        r = d.get("range")
+        self.range = list(map(float, r)) if r and len(r) == 4 else None
+        self.samples = d.get("samples", 0)
+
+
+def solve(M, b):
+    """Solve a small linear system (Gaussian elimination with pivoting); zeros if singular."""
+    n = len(b)
+    A = [row[:] + [b[i]] for i, row in enumerate(M)]
+    for c in range(n):
+        piv = max(range(c, n), key=lambda r: abs(A[r][c]))
+        if abs(A[piv][c]) < 1e-12:
+            return [0.0] * n
+        A[c], A[piv] = A[piv], A[c]
+        for r in range(n):
+            if r != c:
+                f = A[r][c] / A[c][c]
+                for k in range(c, n + 1):
+                    A[r][k] -= f * A[c][k]
+    return [A[i][n] / A[i][i] for i in range(n)]
+
+
+class LiveCorrection:
+    """Corrections learned on the fly from snapped clicks, on top of the calibration.
+
+    Each click on an element is a measurement: you were looking at that element when you
+    pressed, and the tracker put your gaze at the raw point, so the gap between them is the
+    whole error there. Whatever the calibration doesn't already explain (the residual) is
+    fitted with the same quadratic terms. Every term but the offset is held close to zero
+    (ridge), so one click shifts the whole correction and more clicks bend it. Whatever is
+    still left near a click is added within a few degrees of it: on the Frame, errors less
+    than 3 degrees apart are alike, and ones further apart are unrelated. Recent clicks count more (a
+    half-life counted in clicks), so it follows SteamVR's gaze as that drifts or relearns.
+    Replayed on logged points: a calibration from an earlier session was 4.95 degrees off;
+    one click brought that to 2.3, five to 1.6, twenty to 1.2.
+
+    A big element says little about where on it you looked, so each axis is weighted by the
+    element's size along it: a list row 12 degrees wide barely counts sideways.
+
+    Putting the headset back on moves the error (SteamVR starts its eye model over each
+    time, and the headset sits a little differently), so clicks from before the last time
+    it went on (`wear_time`, when set) count OLD_WEAR as much: the offset is relearned from
+    the first few clicks after, and the shape is kept meanwhile."""
+
+    RIDGE = [0.01, 0.5, 0.5, 0.5, 0.5, 0.5]
+    KERNEL = 2.5     # degrees: how far a click's leftover reaches
+    SHRINK = 0.5     # near one click, half its leftover; near several, nearly all
+    HALF_LIFE = 40   # clicks
+    KEEP = 150
+    MARGIN = 3.0
+    OLD_WEAR = 0.3
+
+    def __init__(self):
+        self.samples = []  # dicts: time, hy, hp, dy, dp (the whole error), wy, wp
+        self.wear_time = None
+        self.reset_fit()
+
+    def reset_fit(self):
+        self.cy = [0.0] * 6
+        self.cp = [0.0] * 6
+        self.left = []     # (hy, hp, leftover yaw, leftover pitch, wy, wp, decay)
+        self.range = None
+
+    def features(self, hy, hp):
+        if self.range:
+            y0, y1, p0, p1 = self.range
+            hy = min(max(hy, y0 - self.MARGIN), y1 + self.MARGIN)
+            hp = min(max(hp, p0 - self.MARGIN), p1 + self.MARGIN)
+        x, y = hy / 30.0, hp / 30.0
+        return [1.0, x, y, x * y, x * x, y * y]
+
+    def add(self, sample, base, mode):
+        self.samples = (self.samples + [sample])[-self.KEEP:]
+        self.refit(base, mode)
+
+    def undo(self, base, mode):
+        if self.samples:
+            self.samples.pop()
+            self.refit(base, mode)
+
+    def refit(self, base, mode):
+        """Refit from the samples against the calibration as it is now."""
+        self.reset_fit()
+        n = len(self.samples)
+        if not n:
+            return
+        self.range = [min(s["hy"] for s in self.samples), max(s["hy"] for s in self.samples),
+                      min(s["hp"] for s in self.samples), max(s["hp"] for s in self.samples)]
+        rows = []
+        for k, s in enumerate(self.samples):
+            decay = 0.5 ** ((n - 1 - k) / self.HALF_LIFE)
+            if self.wear_time and s.get("time", 0) < self.wear_time:
+                decay *= self.OLD_WEAR
+            by, bp = base.get(s["hy"], s["hp"], mode)
+            rows.append((s, self.features(s["hy"], s["hp"]), s["dy"] - by, s["dp"] - bp, decay))
+        for out, ri, wi in ((self.cy, 2, "wy"), (self.cp, 3, "wp")):
+            M = [[self.RIDGE[r] if r == c else 0.0 for c in range(6)] for r in range(6)]
+            b = [0.0] * 6
+            for row in rows:
+                w = row[4] * row[0][wi]
+                f = row[1]
+                for r in range(6):
+                    b[r] += w * f[r] * row[ri]
+                    for c in range(6):
+                        M[r][c] += w * f[r] * f[c]
+            out[:] = solve(M, b)
+        for s, f, ry, rp, decay in rows:
+            ly = ry - sum(a * v for a, v in zip(self.cy, f))
+            lp = rp - sum(a * v for a, v in zip(self.cp, f))
+            self.left.append((s["hy"], s["hp"], ly, lp, s["wy"] * decay, s["wp"] * decay))
+
+    def get(self, hy, hp):
+        if not self.samples:
+            return 0.0, 0.0
+        f = self.features(hy, hp)
+        cy = sum(a * v for a, v in zip(self.cy, f))
+        cp = sum(a * v for a, v in zip(self.cp, f))
+        k2 = 2 * self.KERNEL ** 2
+        sy = sp = wy = wp = 0.0
+        for y, p, ly, lp, ay, ap in self.left:
+            d2 = (y - hy) ** 2 + (p - hp) ** 2
+            if d2 > 9 * k2:
+                continue
+            g = math.exp(-d2 / k2)
+            sy += g * ay * ly
+            wy += g * ay
+            sp += g * ap * lp
+            wp += g * ap
+        return cy + sy / (wy + self.SHRINK), cp + sp / (wp + self.SHRINK)
+
+    def offset(self):
+        return self.cy[0], self.cp[0]
+
+
+class SteamEyeLog:
+    """Follows SteamVR's eye tracking log (read only) for what moves the raw gaze under a
+    calibration.
+
+    SteamVR's eye tracker calibrates itself from clicks: a quick mouse-button down and up
+    (the laser or the Frametop pointer), with the gaze within 5 degrees of the click and
+    held still, is taken as "you were looking there" ("Accept usercal"). Accepted clicks
+    were all under 0.14 s; 0.38 s was "too slow", and one that moved was refused. It keeps that in
+    the running `eyetracking` process and saves nothing, so when the process starts again
+    (SteamVR restarting), its calibration starts over. Both events are counted here, and
+    each time the headset goes on ("HMD on"): the eye model starts over then too."""
+
+    PATH = Path.home() / ".local" / "share" / "Steam" / "logs" / "eyetracking.txt"
+
+    def __init__(self):
+        self.pos = 0
+        self.inode = None
+        self.partial = ""
+        self.starts = []    # when the eyetracking process started
+        self.accepts = []   # when it learned from a click
+        self.rejects = []
+        self.wears = []     # when the headset went on
+        self.offs = []      # ... and off
+
+    @staticmethod
+    def stamp(line):
+        head = line.split(" [", 1)[0]
+        main, _, frac = head.partition(".")
+        try:
+            return time.mktime(time.strptime(main.strip(), "%a %b %d %Y %H:%M:%S")) + float("0." + (frac or "0"))
+        except ValueError:
+            return None
+
+    def poll(self):
+        """Read what's new. True if the eye tracker started again since the last poll."""
+        try:
+            st = os.stat(self.PATH)
+        except OSError:
+            return False
+        if st.st_ino != self.inode or st.st_size < self.pos:
+            self.inode, self.pos, self.partial = st.st_ino, 0, ""
+        if st.st_size == self.pos:
+            return False
+        first = self.pos == 0 and not self.starts
+        try:
+            with open(self.PATH, "rb") as f:
+                f.seek(self.pos)
+                data = f.read()
+        except OSError:
+            return False
+        self.pos += len(data)
+        lines = (self.partial + data.decode("utf-8", "replace")).split("\n")
+        self.partial = lines.pop()
+        restarted = False
+        for line in lines:
+            if "usercal" not in line and "startup with PID" not in line and "HMD o" not in line:
+                continue
+            t = self.stamp(line)
+            if t is None:
+                continue
+            if "startup with PID" in line:
+                self.starts.append(t)
+                restarted = not first
+            elif "HMD on" in line:
+                self.wears.append(t)
+            elif "HMD off" in line:
+                self.offs.append(t)
+            elif "Accept usercal" in line:
+                self.accepts.append(t)
+            elif "Reject usercal" in line:
+                self.rejects.append(t)
+        return restarted
+
+    def worn(self):
+        """When the headset last went on (None if not in this log)."""
+        return self.wears[-1] if self.wears else None
+
+    def wearing(self):
+        """Whether the headset is on, as far as the log says (None: it doesn't say)."""
+        if not self.wears and not self.offs:
+            return None
+        return bool(self.wears) and (not self.offs or self.wears[-1] > self.offs[-1])
+
+    def started(self):
+        return self.starts[-1] if self.starts else None
+
+    def accepted_since(self, t):
+        return sum(1 for a in self.accepts if a >= t)
+
+
+def cross_validate(points, mode):
+    """Leave-one-out: each point's error under a model fitted on all the others (degrees)."""
+    errs = []
+    for i in range(len(points)):
+        c = Correction()
+        c.fit(points[:i] + points[i + 1:], mode)
+        cy, cp = c.get(points[i][0], points[i][1], mode)
+        errs.append(math.hypot(points[i][2] - cy, points[i][3] - cp))
+    return errs
+
+
+def steady_samples(samples, vergence_jump=1.5):
+    """The samples of one look at one spot where the tracker had both eyes: none in a blink
+    (openness under half its median over the samples), and none where the angle between the eyes' directions (`lr`, the
+    vergence) is more than `vergence_jump` degrees from its median over the samples. The
+    vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
+    fraction of one far off), so only a jump away from what it was during this look means
+    the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept."""
+    # Openness: a blink is a sharp drop from what it was during this look. Not a fixed
+    # level: looking down, the upper lids come down with the eyes, and in bright light you
+    # squint, so the reading can stay under 0.5 for the whole look while the tracker follows
+    # the eyes fine (a calibration dot at the bottom of the bright round failed that way).
+    opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
+    floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12
+    opened = []
+    for smp in samples:
+        o = (smp["src"].get("mmap1") or {}).get("open")
+        if not o or min(o) >= floor:
+            opened.append(smp)
+
+    def vergence(smp):
+        return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
+    have = [v for v in map(vergence, opened) if v is not None]
+    if len(have) < 5:
+        return opened
+    med = statistics.median(have)
+    return [smp for smp in opened if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]

gaze/probe/ft-gazeprobe.desktop

@@ -0,0 +1,9 @@
+[Desktop Entry]
+Type=Application
+Name=Frametop Gaze Probe
+GenericName=Eye tracking playground
+Comment=How accurate the headset's eye tracking is on your screens, and gaze clicking with calibration
+Exec=@REPO@/gaze/probe/ft-gazeprobe
+Icon=view-visible
+Categories=Utility;Development;
+Keywords=eye;gaze;tracking;calibration;pointer;steamvr;frametop;

gaze/probe/install.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Install (or remove) Frametop Gaze Probe in the desktop's app menu, and build ft-gaze.
+# Usage: gaze/probe/install.sh [install|uninstall]
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+. "$root/scripts/_env.sh"
+"$root/scripts/sync.sh" >/dev/null
+apps=.local/share/applications
+case ${1:-install} in
+  install)
+    "$root/gaze/build.sh"
+    fill_template "$root/gaze/probe/ft-gazeprobe.desktop" | on_frame "mkdir -p ~/$apps && cat > ~/$apps/ft-gazeprobe.desktop"
+    on_frame "chmod +x gaze/probe/ft-gazeprobe"
+    echo "installed: Frametop Gaze Probe" ;;
+  uninstall)
+    on_frame "rm -f ~/$apps/ft-gazeprobe.desktop; echo removed" ;;
+  *) echo "usage: $0 [install|uninstall]" >&2; exit 2 ;;
+esac

gaze/run.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+# Install, start, stop, or inspect the gaze service (ft-gazed) on the Frame.
+# Usage: gaze/run.sh install|uninstall   # user service, starts with SteamVR
+#        gaze/run.sh start|stop|restart|status|log [lines]
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+. "$root/scripts/_env.sh"
+frame="$root/scripts/frame.sh"
+unit=frametop-gaze.service
+case ${1:-status} in
+  install)
+    "$root/gaze/build.sh"
+    fill_template "$root/gaze/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit"
+    on_frame "chmod +x gaze/ft-gazed gaze/ft-gazectl"
+    "$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable --now $unit
+sleep 2; echo \"$unit: \$(systemctl --user is-active $unit)\"; journalctl --user -u $unit --no-pager -o cat -n 5" ;;
+  uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;;
+  start|stop|restart) "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit" ;;
+  status) "$frame" --host "systemctl --user is-active $unit" || true; on_frame "gaze/ft-gazectl status" || true ;;
+  log) "$frame" --host "journalctl --user -u $unit --no-pager -o cat -n ${2:-30}" ;;
+  *) echo "usage: $0 install|uninstall|start|stop|restart|status|log" >&2; exit 2 ;;
+esac

input-settings/ft_input_settings.py

@@ -6,7 +6,12 @@ to the input relay over its control socket (@frametop_relay):
   - Devices: every USB/Bluetooth mouse and keyboard, a live activity light to
     identify them, and a role for each (3D pointer, pass through, ignore).
   - Buttons: press a button or key on a pointer device, then pick an action.
+  - Controllers: the same for the Frame controllers' buttons. They're read by the pointer
+    helper through SteamVR input (@ft_pointer_helper: vrstatus, vrglobal), and a mapped
+    button is taken from games.
   - Pointer: speed, dot size, distance and the rest, applied live.
+  - Gaze: the pointer's gaze mode (@ft_pointer_helper "gaze") and the gaze service
+    (gaze/ft-gazed, @ft_gazed: status, forget, reload).
   - Bluetooth: paired devices, and re-applying the Bluetooth LE fixes after pairing.
 Rules go to ~/.config/frametop-input.json and pointer settings to
 ~/.config/frametop.conf; then the relay (and through it the helper) reloads.
@@ -19,6 +24,7 @@ import shutil
 import socket
 import subprocess
 import sys
+import time
 
 from PySide6.QtCore import Property, QObject, QSocketNotifier, QTimer, QUrl, Signal, Slot
 from PySide6.QtGui import QGuiApplication, QIcon
@@ -29,6 +35,9 @@ RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
 CONF_PATH = os.path.expanduser("~/.config/frametop.conf")
 RELAY = "\0frametop_relay"
 HELPER = "\0ft_pointer_helper"
+GAZED = "\0ft_gazed"
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+GAZE_PROBE = os.path.join(REPO, "gaze", "probe", "ft-gazeprobe")
 BTN_MISC = 0x100
 # Header names that mark the start of a range, not a real key (BTN_MOUSE == BTN_LEFT).
 RANGE_ALIASES = {"BTN_MISC", "BTN_MOUSE", "BTN_JOYSTICK", "BTN_GAMEPAD", "BTN_DIGI", "BTN_WHEEL",
@@ -37,12 +46,32 @@ DEFAULT_BUTTONS = {0x110: "left", 0x111: "right", 0x112: "middle", 0x113: "back"
 ACTION_LABELS = {
     "left": "Left click", "right": "Right click", "middle": "Middle click", "back": "Back",
     "scroll_up": "Scroll up", "scroll_down": "Scroll down", "dashboard": "Toggle SteamVR dashboard",
-    "recenter": "Recenter pointer", "pointer_toggle": "Pointer on/off", "sens_up": "Faster pointer",
+    "recenter": "Recenter pointer", "pointer_toggle": "Pointer on/off",
+    "follow_toggle": "Head follow on/off (experimental)", "gaze_toggle": "Gaze pointer on/off (experimental)",
+    "sens_up": "Faster pointer",
     "sens_down": "Slower pointer", "layout_reset": "Reset desktop screen layout",
     "screens_toggle": "Hide/show desktop screens", "key": "Pass through as key",
     "none": "Do nothing",
 }
 ROLE_LABELS = {"pointer": "3D pointer", "passthrough": "Pass through", "ignore": "Ignore"}
+# Frame controller buttons the pointer helper can read (pointer/helper/vrbuttons.h). The
+# system button stays SteamVR's.
+CONTROLLER_BUTTONS = {
+    "left/view": "Left View", "left/dpad_up": "Left D-pad up", "left/dpad_down": "Left D-pad down",
+    "left/dpad_left": "Left D-pad left", "left/dpad_right": "Left D-pad right", "left/bumper": "Left bumper",
+    "left/trigger": "Left trigger", "left/grip": "Left grip", "left/thumbstick": "Left stick click",
+    "right/menu": "Right Menu", "right/a": "Right A", "right/b": "Right B", "right/x": "Right X", "right/y": "Right Y",
+    "right/bumper": "Right bumper", "right/trigger": "Right trigger", "right/grip": "Right grip",
+    "right/thumbstick": "Right stick click",
+}
+CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none")]
+# Gaze mode settings (pointer helper), like POINTER_SETTINGS.
+GAZE_SETTINGS = [
+    ("POINTER_GAZE_RETAKE", "Look away to hand back", 5, 1, 45, 0.5, "°"),
+    ("POINTER_GAZE_NUDGE_MAX", "Largest nudge to learn", 8, 1, 30, 0.5, "°"),
+    ("POINTER_GAZE_HOLD", "Hold still to drag", 0.5, 0.1, 2.0, 0.05, "s"),
+    ("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"),
+]
 # Pointer settings: key, label, default, min, max, step, unit.
 POINTER_SETTINGS = [
     ("POINTER_SENSITIVITY", "Speed", 0.03, 0.005, 0.12, 0.001, "°/count"),
@@ -52,6 +81,10 @@ POINTER_SETTINGS = [
     ("POINTER_ORIGIN_MARGIN", "Room for small controls", 0.15, 0.03, 0.5, 0.01, "m"),
     ("POINTER_SCENE_RADIUS", "Dock / window-control reach", 0.5, 0.1, 1.5, 0.05, "m"),
     ("POINTER_EDGE_REACH", "Panel edge reach", 0.3, 0.0, 1.0, 0.05, "m"),
+    ("POINTER_LEASH_DEG", "Head follow leash", 10, 0, 60, 1, "°"),
+    ("POINTER_LEASH_DELAY", "Head follow delay", 0.2, 0.0, 1.0, 0.05, "s"),
+    ("POINTER_LEASH_RETURN", "Head follow catch-up", 0.2, 0.05, 2.0, 0.05, "s"),
+    ("POINTER_FOLLOW_REACH", "Head follow reach", 70, 20, 85, 1, "°"),
     ("POINTER_WAKE_COUNTS", "Movement to wake", 40, 5, 200, 5, "counts"),
     ("POINTER_IDLE", "Release after idle", 30, 5, 120, 5, "s"),
 ]
@@ -128,8 +161,11 @@ class Backend(QObject):
     mappingsChanged = Signal()
     pointerChanged = Signal()
     bluetoothChanged = Signal()
+    controllersChanged = Signal()
+    gazeChanged = Signal()
     activity = Signal(str)  # device id
     captured = Signal(int, str)  # code, name
+    capturedController = Signal(str, str)  # button, label
     message = Signal(str, bool)  # text, is error
 
     def __init__(self):
@@ -140,6 +176,13 @@ class Backend(QObject):
         self._capture_id = ""
         self._bluetooth = []
         self._relay_ok = False
+        self._capture_vr = False
+        self._vr = {}  # the helper's vrstatus, {} when it doesn't answer
+        self._vr_at = 0.0
+        self._gaze = {}  # ft-gazed's status, {} when it isn't running
+        self._gaze_prev = None  # the status before, for rates
+        self._gaze_at = 0.0
+        self._gaze_mode = None  # the helper's gaze mode: True, False, None (no answer)
         self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
         self.sock.bind("")  # autobind an abstract address the relay can reply to
         self.sock.setblocking(False)
@@ -167,6 +210,19 @@ class Backend(QObject):
 
     def _refresh(self):
         self._send("devices")
+        self._send("vrstatus", HELPER)
+        self._send("gaze ?", HELPER)
+        if not self._send("status", GAZED) and self._gaze:
+            self._gaze, self._gaze_prev = {}, None
+            self.gazeChanged.emit()
+        now = time.monotonic()
+        # No answer for a while: that side isn't running (any more).
+        if self._vr and now - self._vr_at > 5:
+            self._vr = {}
+            self.controllersChanged.emit()
+        if self._gaze_mode is not None and now - self._gaze_at > 5:
+            self._gaze_mode = None
+            self.gazeChanged.emit()
 
     def _read(self):
         while True:
@@ -174,16 +230,37 @@ class Backend(QObject):
                 data = self.sock.recv(65536)
             except BlockingIOError:
                 return
+            text = data.decode(errors="replace")
+            if text in ("ok on", "ok off"):  # the helper's answer to "gaze ?" (from an unbound socket)
+                self._gaze_mode = text == "ok on"
+                self._gaze_at = time.monotonic()
+                self.gazeChanged.emit()
+                continue
             try:
                 msg = json.loads(data)
             except ValueError:
                 continue
+            if isinstance(msg, dict) and "samples" in msg and "t" not in msg:  # ft-gazed's status
+                self._gaze_status(msg)
+                continue
+            if not isinstance(msg, dict):
+                continue
             t = msg.get("t")
             if t == "devices":
                 self._nodes = msg.get("nodes", [])
                 self._pointer_mode = bool(msg.get("pointer_mode"))
                 self._relay_ok = True
                 self.devicesChanged.emit()
+            elif t == "vrstatus":
+                self._vr = msg
+                self._vr_at = time.monotonic()
+                self.controllersChanged.emit()
+            elif t == "event" and msg.get("type") == "vr":
+                self.activity.emit(msg["id"])
+                if self._capture_vr and msg["value"] == 1 and msg["code"] in CONTROLLER_BUTTONS:
+                    self._capture_vr = False
+                    self._send("vrcapture 0")
+                    self.capturedController.emit(msg["code"], CONTROLLER_BUTTONS[msg["code"]])
             elif t == "event":
                 self.activity.emit(msg["id"])
                 if (self._capture_id and msg["id"] == self._capture_id and msg["type"] == "key"
@@ -356,15 +433,186 @@ class Backend(QObject):
 
     @Slot(str, float)
     def setPointerSetting(self, key, value):
-        integer = key in ("POINTER_WAKE_COUNTS", "POINTER_IDLE")
+        integer = key in ("POINTER_WAKE_COUNTS", "POINTER_IDLE", "POINTER_LEASH_DEG", "POINTER_FOLLOW_REACH")
         write_conf_value(key, str(int(round(value))) if integer else f"{value:.3f}".rstrip("0").rstrip("."))
         self.reload_timer.start()  # debounce slider drags
         self.pointerChanged.emit()
 
+    @Property(bool, notify=pointerChanged)
+    def pointerFollow(self):
+        return read_conf().get("POINTER_FOLLOW", "0") not in ("", "0")
+
+    @Slot(bool)
+    def setPointerFollow(self, on):
+        write_conf_value("POINTER_FOLLOW", "1" if on else "0")
+        self.reload_timer.start()
+        self.pointerChanged.emit()
+
     @Slot(result=bool)
     def recenter(self):
         return self._send("recenter", HELPER)
 
+    # --- controllers ---
+    @Property("QVariantList", constant=True)
+    def controllerActions(self):
+        return [{"value": a, "text": ACTION_LABELS[a]} for a in CONTROLLER_ACTIONS]
+
+    @Property("QVariantList", constant=True)
+    def controllerButtons(self):
+        return [{"value": b, "text": label} for b, label in CONTROLLER_BUTTONS.items()]
+
+    @Property("QVariantList", notify=mappingsChanged)
+    def controllerMappings(self):
+        mapped = read_json(RULES_PATH).get("controller_buttons", {})
+        return [{"button": b, "label": label, "action": mapped[b],
+                 "actionLabel": ACTION_LABELS.get(mapped[b], mapped[b])}
+                for b, label in CONTROLLER_BUTTONS.items() if b in mapped]
+
+    @Property("QVariantMap", notify=controllersChanged)
+    def controllerStatus(self):
+        """helper: answering; manifest: its SteamVR input is set up; global: SteamVR's
+        "Enable global input from overlays"; active: mapped buttons SteamVR delivers now."""
+        vr = self._vr
+        return {"helper": bool(vr), "manifest": bool(vr.get("manifest")), "global": bool(vr.get("global")),
+                "inGame": bool(vr.get("in_game")), "bound": vr.get("bound", []), "active": vr.get("active", [])}
+
+    @Property(bool, notify=mappingsChanged)
+    def controllerInGames(self):
+        return bool(read_json(RULES_PATH).get("controller_in_games"))
+
+    @Slot(bool)
+    def setControllerInGames(self, on):
+        rules = read_json(RULES_PATH)
+        rules["controller_in_games"] = bool(on)
+        self._save_rules(rules)
+        self.message.emit("Mapped controller buttons: " + ("taken in games too" if on else "left to games"), False)
+
+    @Slot(str, str)
+    def setControllerMapping(self, button, action):
+        if button not in CONTROLLER_BUTTONS or action not in CONTROLLER_ACTIONS:
+            return
+        rules = read_json(RULES_PATH)
+        rules.setdefault("controller_buttons", {})[button] = action
+        self._save_rules(rules)
+        self.message.emit(f"{CONTROLLER_BUTTONS[button]} → {ACTION_LABELS[action]}", False)
+
+    @Slot(str)
+    def removeControllerMapping(self, button):
+        rules = read_json(RULES_PATH)
+        rules.get("controller_buttons", {}).pop(button, None)
+        self._save_rules(rules)
+        self.message.emit(f"{CONTROLLER_BUTTONS.get(button, button)}: back to games", False)
+
+    @Slot()
+    def clearControllerMappings(self):
+        rules = read_json(RULES_PATH)
+        removed = len(rules.pop("controller_buttons", {}) or {})
+        self._save_rules(rules)
+        self.message.emit(f"Removed {removed} controller binding{'s' if removed != 1 else ''}", False)
+
+    @Slot()
+    def startControllerCapture(self):
+        self._capture_vr = True
+        self._send("watch 60")
+        self._send("vrcapture 30")
+
+    @Slot()
+    def cancelControllerCapture(self):
+        if self._capture_vr:
+            self._capture_vr = False
+            self._send("vrcapture 0")
+
+    @Slot(bool)
+    def setGlobalInput(self, on):
+        """SteamVR's "Enable global input from overlays (Experimental)", which the helper needs
+        to get controller buttons while a game or the dashboard has focus."""
+        if self._send(f"vrglobal {'on' if on else 'off'}", HELPER):
+            self.message.emit(f"SteamVR global input from overlays {'on' if on else 'off'}", False)
+        else:
+            self.message.emit("The pointer helper isn't running (frametop-pointer.service)", True)
+
+    # --- gaze ---
+    def _gaze_status(self, status):
+        now = time.monotonic()
+        prev = self._gaze_prev
+        # Rates over the last poll: samples per second, and the share with only one eye.
+        if prev and now - prev[0] > 0.5 and status["samples"] >= prev[1]["samples"]:
+            n = status["samples"] - prev[1]["samples"]
+            status["rate"] = n / (now - prev[0])
+            status["one_eye_share"] = (status["one_eye"] - prev[1]["one_eye"]) / n if n else 0.0
+        elif self._gaze:
+            status.setdefault("rate", self._gaze.get("rate"))
+            status.setdefault("one_eye_share", self._gaze.get("one_eye_share"))
+        if not prev or now - prev[0] > 0.5:
+            self._gaze_prev = (now, status)
+        self._gaze = status
+        self.gazeChanged.emit()
+
+    @Property("QVariantMap", notify=gazeChanged)
+    def gazeStatus(self):
+        return self._gaze
+
+    @Property(bool, notify=gazeChanged)
+    def gazeServiceRunning(self):
+        return bool(self._gaze)
+
+    @Property(int, notify=gazeChanged)
+    def gazeMode(self):
+        """The helper's gaze mode now: 1 on, 0 off, -1 no answer (helper not running)."""
+        return -1 if self._gaze_mode is None else int(self._gaze_mode)
+
+    @Property(bool, notify=gazeChanged)
+    def gazeDefault(self):
+        return read_conf().get("POINTER_GAZE", "0") not in ("", "0")
+
+    @Slot(bool)
+    def setGazeMode(self, on):
+        """On or off now and from now on (POINTER_GAZE); a mapped button toggles it until restart."""
+        write_conf_value("POINTER_GAZE", "1" if on else "0")
+        self._send(f"gaze {'on' if on else 'off'}", HELPER)
+        self.reload_timer.start()
+        self.gazeChanged.emit()
+
+    @Property("QVariantList", notify=pointerChanged)
+    def gazeSettings(self):
+        conf = read_conf()
+        out = []
+        for key, label, default, lo, hi, step, unit in GAZE_SETTINGS:
+            try:
+                value = float(conf.get(key, default))
+            except ValueError:
+                value = default
+            out.append({"key": key, "label": label, "value": value, "min": lo, "max": hi, "step": step,
+                        "unit": unit, "default": default})
+        return out
+
+    @Slot()
+    def forgetGazeLessons(self):
+        if self._send("forget", GAZED):
+            self.message.emit("Forgot what the pointer's nudges taught; the calibration stays", False)
+        else:
+            self.message.emit("The gaze service isn't running (frametop-gaze.service)", True)
+
+    @Slot()
+    def reloadGazeCalibration(self):
+        if self._send("reload", GAZED):
+            self.message.emit("The gaze service read the calibration again", False)
+        else:
+            self.message.emit("The gaze service isn't running (frametop-gaze.service)", True)
+
+    @Slot()
+    def openGazeProbe(self):
+        """Calibrate in ft-gazeprobe (a GTK app on the host, fullscreen on a Frametop screen)."""
+        runner = ["distrobox-host-exec"] if shutil.which("distrobox-host-exec") else []
+        env = [f"{k}={os.environ[k]}" for k in ("WAYLAND_DISPLAY", "DISPLAY", "XAUTHORITY", "DBUS_SESSION_BUS_ADDRESS")
+               if os.environ.get(k)]
+        try:
+            subprocess.Popen(runner + ["env"] + env + [GAZE_PROBE], stdin=subprocess.DEVNULL,
+                             stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True)
+            self.message.emit("Opening the gaze probe: calibrate there, then close it", False)
+        except OSError as e:
+            self.message.emit(f"Couldn't open the gaze probe: {e}", True)
+
     # --- bluetooth ---
     @Property("QVariantList", notify=bluetoothChanged)
     def bluetooth(self):

input-settings/main.qml

@@ -18,7 +18,9 @@ Kirigami.ApplicationWindow {
         actions: [
             Kirigami.Action { text: "Devices"; icon.name: "input-mouse"; onTriggered: root.show(devicesPage) },
             Kirigami.Action { text: "Buttons"; icon.name: "input-keyboard"; onTriggered: root.show(buttonsPage) },
+            Kirigami.Action { text: "Controllers"; icon.name: "input-gamepad"; onTriggered: root.show(controllersPage) },
             Kirigami.Action { text: "Pointer"; icon.name: "transform-move"; onTriggered: root.show(pointerPage) },
+            Kirigami.Action { text: "Gaze"; icon.name: "view-visible"; onTriggered: root.show(gazePage) },
             Kirigami.Action { text: "Bluetooth"; icon.name: "preferences-system-bluetooth"; onTriggered: root.show(bluetoothPage) }
         ]
     }
@@ -28,8 +30,9 @@ Kirigami.ApplicationWindow {
         pageStack.push(page)
     }
 
-    // FT_INPUT_PAGE=buttons|pointer|bluetooth opens the app on that page.
+    // FT_INPUT_PAGE=buttons|controllers|pointer|gaze|bluetooth opens the app on that page.
-    pageStack.initialPage: ({ buttons: buttonsPage, pointer: pointerPage, bluetooth: bluetoothPage })[startPage] || devicesPage
+    pageStack.initialPage: ({ buttons: buttonsPage, controllers: controllersPage, pointer: pointerPage, gaze: gazePage,
+                              bluetooth: bluetoothPage })[startPage] || devicesPage
 
     Connections {
         target: backend
@@ -157,7 +160,7 @@ Kirigami.ApplicationWindow {
                 wrapMode: Text.Wrap
                 opacity: 0.7
                 text: "Move or press a device to see which row it is. 3D pointer: grabbed, drives the SteamVR pointer. "
-                      + "Pass through: left alone (a Meta tap still toggles the dashboard). Ignore: left alone."
+                      + "Pass through: left alone (a Meta tap toggles the dashboard if META_DASHBOARD=1). Ignore: left alone."
             }
         }
     }
@@ -295,6 +298,176 @@ Kirigami.ApplicationWindow {
                     opacity: 0.7
                     text: "Buttons without a mapping pass through (mouse buttons as clicks, keys as keys). "
                           + "The Z3's extra buttons show up as keys from its keyboard node. "
+                          + "The Frame controllers' buttons are on the Controllers page."
+                }
+            }
+        }
+    }
+
+    // ---------------------------------------------------------------- Controllers
+    Component {
+        id: controllersPage
+        Kirigami.ScrollablePage {
+            id: cpage
+            title: "Controllers"
+            actions: [
+                Kirigami.Action {
+                    text: "Remove all"
+                    icon.name: "edit-clear-all"
+                    tooltip: "Give every controller button back to games"
+                    enabled: backend.controllerMappings.length > 0
+                    onTriggered: backend.clearControllerMappings()
+                }
+            ]
+            property string capturedButton: ""
+            property string capturedLabel: ""
+            property bool capturing: false
+            property var status: backend.controllerStatus
+            Component.onDestruction: backend.cancelControllerCapture()
+
+            Connections {
+                target: backend
+                function onCapturedController(button, label) {
+                    cpage.capturedButton = button; cpage.capturedLabel = label; cpage.capturing = false
+                    buttonBox.currentIndex = buttonBox.indexOfValue(button)
+                }
+            }
+            Timer {
+                // The relay takes every button for 30 s while capturing.
+                running: cpage.capturing
+                interval: 30000
+                onTriggered: { backend.cancelControllerCapture(); cpage.capturing = false }
+            }
+
+            ColumnLayout {
+                spacing: Kirigami.Units.largeSpacing
+
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    visible: !cpage.status.helper
+                    type: Kirigami.MessageType.Error
+                    text: "The pointer helper isn't answering (frametop-pointer.service, needs SteamVR). "
+                          + "It reads the controller buttons."
+                }
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    visible: cpage.status.helper && !cpage.status.manifest
+                    type: Kirigami.MessageType.Error
+                    text: "The pointer helper couldn't set up SteamVR input (pointer/helper/actions). See its log."
+                }
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    visible: cpage.status.helper && !cpage.status.global && backend.controllerMappings.length > 0
+                    type: Kirigami.MessageType.Warning
+                    text: "Global input is off, so the mapped buttons only reach Frametop when nothing else has "
+                          + "focus, if at all."
+                }
+
+                Kirigami.FormLayout {
+                    Layout.fillWidth: true
+
+                    Controls.Switch {
+                        Kirigami.FormData.label: "Global input:"
+                        text: "SteamVR's \"Enable global input from overlays (Experimental)\", which mapped buttons need"
+                        checked: cpage.status.global
+                        enabled: cpage.status.helper
+                        onToggled: backend.setGlobalInput(checked)
+                    }
+                    Controls.Switch {
+                        Kirigami.FormData.label: "In games:"
+                        text: "Mapped buttons work while a game is running too (the game loses them)"
+                        checked: backend.controllerInGames
+                        onToggled: backend.setControllerInGames(checked)
+                    }
+                    RowLayout {
+                        Kirigami.FormData.label: "New mapping:"
+                        Controls.Button {
+                            text: cpage.capturing ? "Press a button on a controller…" : "Capture a button"
+                            icon.name: "input-gamepad"
+                            highlighted: cpage.capturing
+                            enabled: cpage.status.helper
+                            onClicked: {
+                                if (cpage.capturing) { backend.cancelControllerCapture(); cpage.capturing = false }
+                                else { cpage.capturedButton = ""; cpage.capturing = true; backend.startControllerCapture() }
+                            }
+                        }
+                        Controls.Label { text: "or" ; opacity: 0.7 }
+                        Controls.ComboBox {
+                            id: buttonBox
+                            model: backend.controllerButtons
+                            textRole: "text"
+                            valueRole: "value"
+                            Layout.preferredWidth: Kirigami.Units.gridUnit * 10
+                        }
+                        Controls.ComboBox {
+                            id: newControllerAction
+                            model: backend.controllerActions
+                            textRole: "text"
+                            valueRole: "value"
+                            Layout.preferredWidth: Kirigami.Units.gridUnit * 13
+                        }
+                        Controls.Button {
+                            text: "Map"
+                            icon.name: "dialog-ok-apply"
+                            onClicked: { backend.setControllerMapping(buttonBox.currentValue, newControllerAction.currentValue); cpage.capturedButton = "" }
+                        }
+                    }
+                }
+
+                Kirigami.Heading {
+                    visible: backend.controllerMappings.length > 0
+                    level: 3
+                    text: "Current mappings"
+                }
+
+                Repeater {
+                    model: backend.controllerMappings
+                    delegate: RowLayout {
+                        id: crow
+                        required property var modelData
+                        Layout.fillWidth: true
+                        spacing: Kirigami.Units.largeSpacing
+                        Controls.Label {
+                            text: crow.modelData.label
+                            Layout.preferredWidth: Kirigami.Units.gridUnit * 10
+                        }
+                        Controls.ComboBox {
+                            model: backend.controllerActions
+                            textRole: "text"
+                            valueRole: "value"
+                            Layout.preferredWidth: Kirigami.Units.gridUnit * 13
+                            Component.onCompleted: currentIndex = indexOfValue(crow.modelData.action)
+                            onActivated: backend.setControllerMapping(crow.modelData.button, currentValue)
+                        }
+                        Controls.Label {
+                            // Active: SteamVR delivers it to Frametop now (a controller is on).
+                            text: cpage.status.active.indexOf(crow.modelData.button) >= 0 ? "active"
+                                  : cpage.status.inGame && !backend.controllerInGames ? "game's" : "waiting"
+                            opacity: 0.6
+                            Layout.preferredWidth: Kirigami.Units.gridUnit * 4
+                            Controls.ToolTip.text: text === "active" ? "SteamVR gives this button to Frametop"
+                                                   : text === "game's" ? "A game is running: the button is the game's until it quits"
+                                                   : "No controller with this button is on, or SteamVR doesn't give it to Frametop"
+                            Controls.ToolTip.visible: hover.hovered
+                            HoverHandler { id: hover }
+                        }
+                        Controls.Button {
+                            text: "Remove"
+                            icon.name: "edit-delete-remove"
+                            onClicked: backend.removeControllerMapping(crow.modelData.button)
+                        }
+                        Item { Layout.fillWidth: true }
+                    }
+                }
+
+                Controls.Label {
+                    Layout.fillWidth: true
+                    wrapMode: Text.Wrap
+                    opacity: 0.7
+                    text: "Outside games, a mapped button is Frametop's; while a game runs it's the game's, unless "
+                          + "\"In games\" is on. Unmapped buttons are left alone, and the system button stays SteamVR's. "
+                          + "Clicks and scrolling go to the 3D pointer. The trigger and grip also move SteamVR's laser "
+                          + "to that controller, so they're better left unmapped."
                 }
             }
         }
@@ -314,6 +487,21 @@ Kirigami.ApplicationWindow {
             ]
 
             Kirigami.FormLayout {
+                Controls.Switch {
+                    Kirigami.FormData.label: "Head follow:"
+                    text: "Pointer follows your head (experimental; leash below, 0° locks it to your view)"
+                    checked: backend.pointerFollow
+                    onToggled: backend.setPointerFollow(checked)
+                }
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 30
+                    visible: true
+                    type: Kirigami.MessageType.Warning
+                    text: "Head follow is experimental. It's only lightly tested and not finished: the head "
+                          + "follow settings below are a starting point, and polishing how it feels is left open "
+                          + "for anyone who wants to take it further."
+                }
                 Repeater {
                     model: backend.pointerSettings
                     delegate: RowLayout {
@@ -355,6 +543,142 @@ Kirigami.ApplicationWindow {
         }
     }
 
+    // ---------------------------------------------------------------- Gaze
+    Component {
+        id: gazePage
+        Kirigami.ScrollablePage {
+            id: gpage
+            title: "Gaze"
+            property var status: backend.gazeStatus
+            actions: [
+                Kirigami.Action {
+                    text: "Calibrate…"
+                    icon.name: "crosshairs"
+                    tooltip: "Open the gaze probe to calibrate (fullscreen on a Frametop screen)"
+                    onTriggered: backend.openGazeProbe()
+                },
+                Kirigami.Action {
+                    text: "Reload calibration"
+                    icon.name: "view-refresh"
+                    enabled: backend.gazeServiceRunning
+                    onTriggered: backend.reloadGazeCalibration()
+                },
+                Kirigami.Action {
+                    text: "Forget nudges"
+                    icon.name: "edit-clear-history"
+                    enabled: backend.gazeServiceRunning
+                    tooltip: "Drop what your mouse nudges taught; the calibration stays"
+                    onTriggered: backend.forgetGazeLessons()
+                }
+            ]
+
+            Kirigami.FormLayout {
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 30
+                    visible: true
+                    type: Kirigami.MessageType.Warning
+                    text: "Gaze mode is experimental. The pointer goes where you look and the mouse does the last bit; "
+                          + "moving the mouse takes over, looking well away hands it back. A small nudge and a click "
+                          + "teach the gaze service where it was off."
+                }
+                Controls.Switch {
+                    Kirigami.FormData.label: "Gaze pointer:"
+                    text: backend.gazeMode < 0 ? "Pointer helper not running" : "Pointer goes where you look"
+                    enabled: backend.gazeMode >= 0
+                    checked: backend.gazeMode > 0
+                    onToggled: backend.setGazeMode(checked)
+                }
+                Controls.Label {
+                    visible: backend.gazeMode >= 0 && (backend.gazeMode > 0) !== backend.gazeDefault
+                    text: "Toggled by a button; it starts " + (backend.gazeDefault ? "on" : "off") + " after a restart."
+                    opacity: 0.7
+                    font: Kirigami.Theme.smallFont
+                }
+                Repeater {
+                    model: backend.gazeSettings
+                    delegate: RowLayout {
+                        required property var modelData
+                        Kirigami.FormData.label: modelData.label + ":"
+                        Controls.Slider {
+                            id: gslider
+                            from: modelData.min
+                            to: modelData.max
+                            stepSize: modelData.step
+                            value: modelData.value
+                            Layout.preferredWidth: Kirigami.Units.gridUnit * 14
+                            onMoved: backend.setPointerSetting(modelData.key, value)
+                        }
+                        Controls.Label {
+                            text: gslider.value.toFixed(modelData.step < 0.1 ? 2 : 1) + " " + modelData.unit
+                            Layout.preferredWidth: Kirigami.Units.gridUnit * 5
+                        }
+                        Controls.ToolButton {
+                            icon.name: "edit-undo"
+                            display: Controls.AbstractButton.IconOnly
+                            text: "Default (" + modelData.default + ")"
+                            Controls.ToolTip.text: text
+                            Controls.ToolTip.visible: hovered
+                            onClicked: { gslider.value = modelData.default; backend.setPointerSetting(modelData.key, modelData.default) }
+                        }
+                    }
+                }
+
+                Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Gaze service" }
+
+                Controls.Label {
+                    Kirigami.FormData.label: "Service:"
+                    text: backend.gazeServiceRunning
+                          ? (gpage.status.ft_gaze ? "running" : "running, eye tracker reader restarting")
+                          : "not running (frametop-gaze.service, starts with SteamVR)"
+                    color: backend.gazeServiceRunning ? Kirigami.Theme.textColor : Kirigami.Theme.negativeTextColor
+                }
+                Controls.Label {
+                    visible: backend.gazeServiceRunning
+                    Kirigami.FormData.label: "Headset:"
+                    text: gpage.status.headset_on ? "on" : "off"
+                }
+                Controls.Label {
+                    visible: backend.gazeServiceRunning
+                    Kirigami.FormData.label: "Tracking:"
+                    text: gpage.status.rate === undefined || gpage.status.rate === null ? "…"
+                          : Math.round(gpage.status.rate) + " samples/s"
+                            + (gpage.status.one_eye_share > 0.5 ? " · only one eye tracked" : "")
+                    color: gpage.status.one_eye_share > 0.5 ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.textColor
+                    Controls.ToolTip.text: "Only one eye tracked: reseat the headset or check the lenses. It still works, "
+                                           + "probably less precisely."
+                    Controls.ToolTip.visible: gpage.status.one_eye_share > 0.5 && ghover.hovered
+                    HoverHandler { id: ghover }
+                }
+                Controls.Label {
+                    visible: backend.gazeServiceRunning
+                    Kirigami.FormData.label: "Calibration:"
+                    text: gpage.status.calibration_samples > 0
+                          ? gpage.status.calibration_samples + " points (" + gpage.status.model + ")"
+                          : "none yet: use Calibrate…"
+                }
+                Controls.Label {
+                    visible: backend.gazeServiceRunning
+                    Kirigami.FormData.label: "Learned from nudges:"
+                    text: gpage.status.lessons + (gpage.status.lessons === 1 ? " nudge" : " nudges")
+                          + (gpage.status.refused > 0 ? " (" + gpage.status.refused + " too far off, ignored)" : "")
+                }
+            }
+
+            footer: Controls.Label {
+                padding: Kirigami.Units.largeSpacing
+                wrapMode: Text.Wrap
+                opacity: 0.7
+                text: "Map a mouse button (Buttons) or a controller button (Controllers) to \"Gaze pointer on/off\" "
+                      + "to switch it on the fly. A click waits for the release: if the gaze is off, drag onto the target "
+                      + "with the button held and let go there. Hold still to drag: hold a press this long without moving "
+                      + "to drag something instead. Look away to hand back: how far from the pointer you look before the "
+                      + "gaze takes it back from the mouse. Largest nudge to learn: bigger mouse moves before a click "
+                      + "are treated as using the mouse, not correcting the gaze."
+            }
+        }
+    }
+
     // ---------------------------------------------------------------- Bluetooth
     Component {
         id: bluetoothPage

input/input-relay.py

@@ -13,17 +13,47 @@ so all its event nodes share one role (the Swiftpoint Z3 has a mouse node and a
 keyboard node for its extra buttons). Roles, from ~/.config/frametop-input.json
 (written by the Frametop Input Settings app):
   pointer      grabbed; drives the universal 3D mouse (default for devices with a mouse node)
-  passthrough  not grabbed, only observed, e.g. for the Meta dashboard shortcut (default for keyboards)
+  passthrough  keys go to the desktop; grabbed only while typing goes there, otherwise only observed,
+               e.g. for the Meta dashboard shortcut (default for keyboards; the shortcut is off
+               unless META_DASHBOARD=1 is in ~/.config/frametop.conf)
   ignore       not grabbed, only observed for identification in the settings app
 Buttons and keys of pointer devices go through a per-device map to actions
 (left, right, middle, back, scroll_up, scroll_down, dashboard, recenter,
-pointer_toggle, sens_up, sens_down, layout_reset = put the desktop screens back in
+pointer_toggle, follow_toggle = head follow on or off, gaze_toggle = gaze mode on or off
-their saved layout, screens_toggle = hide or show the desktop screens, key = pass
+(the pointer goes where you look; see pointer/helper/ft-pointer.cpp), sens_up, sens_down,
+layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens, key = pass
 through as a key, none).
 
+Frame controller buttons can be mapped too ("controller_buttons": {"right/a": action} in the
+rules file; any action but key). The controllers aren't input devices here, only SteamVR sees
+them, so the pointer helper reads them with SteamVR input and sends "vrbtn <button> 1|0".
+It only takes the buttons the relay tells it to ("vrbind <button>..." to @ft_pointer_helper,
+sent on start, reload, and when the helper says "vrhello"), and only while no game runs,
+unless "controller_in_games" is true in the rules file (then a mapped button no longer
+reaches games; see pointer/helper/vrbuttons.h).
+
+In gaze mode outside games, the helper keeps the pointer ("gazeawake 1", repeated every 5
+seconds; "gazeawake 0" or silence ends it): the pointer isn't released when the mouse is idle.
+
 Keys also go to ft-screens (@ft_screens, the Frametop desktop's compositor), which
-types them into the desktop screen that has focus (not while the SteamVR dashboard is
+types them into the desktop screen that has focus: from pass-through keyboards, and
-open): from keyboards that aren't grabbed, and keys a pointer device passes through.
+keys a pointer device passes through. Typing goes to the panel clicked last, and
+ft-screens says which ("keyboard desktop|steam" on the control socket, every second).
+While it's the desktop, pass-through keyboards are grabbed, so gamescope, which reads
+every keyboard itself, doesn't type them into its focused app too. Without word from
+ft-screens for 3 seconds they're released. With SHARE_KEYS=1 in ~/.config/frametop.conf,
+a grabbed keyboard's keys also go out as "key <code> <value> <device name>" datagrams on
+@frametop_keys, for programs that watch every keyboard for a hotkey and lose it to the grab.
+It's off by default: any local process that binds that name first gets every key typed
+into the desktop.
+
+Volume keys, from every device that has them (the headset's own buttons included),
+are handled here: wpctl steps the default output. Nothing else may see a volume key,
+because gamescope aborts on one when no window has keyboard focus, which ends the
+whole VR session. Devices with a keymap (the headset's gpio-keys, USB and Bluetooth
+keyboards) get their volume entries remapped to unused stand-in codes, so their
+other keys keep working for SteamVR; a device without a keymap that has only volume
+keys (the headset's pmic_resin) is grabbed. The keymaps go back when the relay exits.
 
 Pointer mode (POINTER=1 in ~/.config/frametop.conf) sends pointer devices to
 the ft-pointer helper (pointer/helper), which drives the ft_pointer
@@ -34,6 +64,9 @@ Control socket (abstract datagram @frametop_relay, JSON replies to the sender):
   devices           list event nodes with id, name, kinds, role, grabbed
   watch <seconds>   stream input events from every candidate node (identification)
   reload            re-read both config files, re-apply roles, tell the helper
+  vrcapture <s>     take every controller button for s seconds (0: stop), so the settings
+                    app can capture one; watchers see them as events with id frame_controller
+  vrbtn, vrhello, gazeawake   from the pointer helper (above)
 
 Runs on the Frame host as a user service (frametop-input-relay.service). The
 virtual devices are parked in systemd's file descriptor store, so a relay
@@ -46,11 +79,13 @@ kernel's evdev and uinput interfaces.
   input-relay.py --no-grab  never grab, for testing next to a running SteamVR
 """
 import array
+import atexit
 import errno
 import fcntl
 import json
 import os
 import select
+import signal
 import socket
 import struct
 import subprocess
@@ -65,6 +100,12 @@ KEY_A = 30
 REL_X, REL_Y, REL_WHEEL, REL_MAX = 0x00, 0x01, 0x08, 0x0F
 BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_SIDE, BTN_EXTRA = 0x110, 0x111, 0x112, 0x113, 0x114
 KEY_LEFTMETA, KEY_RIGHTMETA = 125, 126
+KEY_VOLUMEDOWN, KEY_VOLUMEUP = 114, 115
+# Volume keys are remapped to KEY_MACRO29 and KEY_MACRO30: above 255, so X11 can't
+# carry them, and bound to nothing in the default keymap.
+VOLUME_STANDIN = {KEY_VOLUMEUP: 0x2AC, KEY_VOLUMEDOWN: 0x2AD}
+VOLUME_ORIGINAL = {v: k for k, v in VOLUME_STANDIN.items()}
+VOLUME_CODES = set(VOLUME_STANDIN) | set(VOLUME_ORIGINAL)
 BUS_USB, BUS_BLUETOOTH, BUS_VIRTUAL = 0x03, 0x05, 0x06
 
 # struct input_event on 64-bit: struct timeval (2 x long), u16 type, u16 code, s32 value.
@@ -90,7 +131,12 @@ UI_SET_EVBIT = _iow("U", 100, 4)
 UI_SET_KEYBIT = _iow("U", 101, 4)
 UI_SET_RELBIT = _iow("U", 102, 4)
 EVIOCGRAB = _iow("E", 0x90, 4)
+EVIOCGKEY = _ior("E", 0x18, (KEY_MAX + 8) // 8)
 EVIOCGID = _ior("E", 0x02, 8)
+KEYMAP_ENTRY = struct.Struct("BBHI32s")  # struct input_keymap_entry: flags, len, index, keycode, scancode
+INPUT_KEYMAP_BY_INDEX = 1
+EVIOCGKEYCODE_V2 = _ior("E", 0x04, KEYMAP_ENTRY.size)
+EVIOCSKEYCODE_V2 = _iow("E", 0x04, KEYMAP_ENTRY.size)
 EV_NAMES = {EV_KEY: "key", EV_REL: "rel"}
 
 
@@ -109,8 +155,16 @@ def eviocguniq(length):
 VIRTUAL_PREFIX = "frametop virtual"
 RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
 ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
-           "pointer_toggle", "sens_up", "sens_down", "layout_reset", "screens_toggle", "key", "none")
+           "pointer_toggle", "follow_toggle", "gaze_toggle", "sens_up", "sens_down", "layout_reset", "screens_toggle",
+           "key", "none")
+HELPER = "\0ft_pointer_helper"
+# Frame controller buttons the pointer helper can read (pointer/helper/vrbuttons.h).
+VR_BUTTONS = ("left/view", "left/dpad_up", "left/dpad_down", "left/dpad_left", "left/dpad_right", "left/bumper",
+              "left/trigger", "left/grip", "left/thumbstick", "right/menu", "right/a", "right/b", "right/x", "right/y",
+              "right/bumper", "right/trigger", "right/grip", "right/thumbstick")
+VR_DEVICE = "frame_controller"  # the id controller buttons have in watch events
 SCREENS = "\0ft_screens"
+KEYS = "\0frametop_keys"  # keys of keyboards grabbed for the desktop, for other readers
 FT_LAYOUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "layout", "ft-layout")
 DEFAULT_BUTTONS = {BTN_LEFT: "left", BTN_RIGHT: "right", BTN_MIDDLE: "middle",
                    BTN_SIDE: "back", BTN_EXTRA: "back"}
@@ -205,7 +259,8 @@ def read_config(path=os.path.expanduser("~/.config/frametop.conf")):
 
 
 def read_rules(path=RULES_PATH):
-    """{"devices": {id: {"role", "name"}}, "buttons": {id: {"<code>": action}}}."""
+    """{"devices": {id: {"role", "name"}}, "buttons": {id: {"<code>": action}},
+    "controller_buttons": {"<hand>/<button>": action}}."""
     try:
         with open(path) as f:
             rules = json.load(f)
@@ -213,9 +268,73 @@ def read_rules(path=RULES_PATH):
         rules = {}
     rules.setdefault("devices", {})
     rules.setdefault("buttons", {})
+    rules.setdefault("controller_buttons", {})
     return rules
 
 
+def remap_volume(fd, restore=False):
+    """Point a device's volume keys at their stand-ins in its keymap, or back with restore.
+
+    Returns how many keymap entries are volume keys or stand-ins, or None when the
+    device has no keymap to change (uinput devices, some platform buttons).
+    """
+    swap = VOLUME_ORIGINAL if restore else VOLUME_STANDIN
+    found = 0
+    for index in range(8192):
+        entry = bytearray(KEYMAP_ENTRY.pack(INPUT_KEYMAP_BY_INDEX, 0, index, 0, b""))
+        try:
+            fcntl.ioctl(fd, EVIOCGKEYCODE_V2, entry)
+        except OSError:
+            return found if index else None  # past the last entry
+        _, length, _, code, scancode = KEYMAP_ENTRY.unpack(entry)
+        if code in VOLUME_CODES:
+            found += 1
+        if code in swap:
+            fcntl.ioctl(fd, EVIOCSKEYCODE_V2,
+                        KEYMAP_ENTRY.pack(INPUT_KEYMAP_BY_INDEX, length, index, swap[code], scancode))
+    return found
+
+
+class Volume:
+    """Volume keys: wpctl steps the default output, repeating while a key is held.
+
+    The repeat is our own, since the headset's buttons have none; kernel autorepeat
+    from keyboards is ignored so every device repeats the same way.
+    """
+
+    STEP = 5  # percent
+    DELAY, RATE = 0.4, 0.1  # seconds before repeating, and between repeats
+
+    def __init__(self):
+        self.held = None  # (fd, code) of the key being held
+        self.next_at = None
+
+    def key(self, fd, code, value, now):
+        if value == 1:
+            self.held = (fd, code)
+            self.step(code)
+            self.next_at = now + self.DELAY
+        elif value == 0 and self.held == (fd, code):
+            self.release()
+
+    def release(self):
+        self.held = self.next_at = None
+
+    def step(self, code):
+        sign = "+" if VOLUME_ORIGINAL.get(code, code) == KEY_VOLUMEUP else "-"
+        subprocess.Popen(["wpctl", "set-volume", "--limit", "1.0", "@DEFAULT_AUDIO_SINK@",
+                          f"{self.STEP}%{sign}"],
+                         stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+
+    def tick(self, now):
+        if self.next_at is not None and now >= self.next_at:
+            self.step(self.held[1])
+            self.next_at = now + self.RATE
+
+    def timeout(self, now, default):
+        return default if self.next_at is None else max(0.0, min(default, self.next_at - now))
+
+
 class Pointer:
     """Drives the ft_pointer SteamVR driver from a mouse (pointer mode).
 
@@ -248,10 +367,11 @@ class Pointer:
         self.wake_counts = wake_counts
         self.pending = 0
         self.pending_since = 0.0
+        self.gaze_awake_until = 0.0  # the helper's gaze mode keeps the pointer until then
 
     def send(self, command):
         try:
-            self.sock.sendto(command.encode(), "\0ft_pointer_helper")
+            self.sock.sendto(command.encode(), HELPER)
         except OSError:
             pass  # helper not running (SteamVR not running)
 
@@ -311,6 +431,12 @@ class Pointer:
                 log("pointer off (toggle)")
             else:
                 self.wake(now)
+        elif name == "follow_toggle":
+            self.send("follow toggle")  # until the next restart; the setting is POINTER_FOLLOW
+            log("head follow toggled")
+        elif name == "gaze_toggle":
+            self.send("gaze toggle")  # until the next restart; the setting is POINTER_GAZE
+            log("gaze mode toggled")
         elif name == "screens_toggle":
             try:
                 self.sock.sendto(b"toggle", SCREENS)
@@ -361,7 +487,7 @@ class Pointer:
         if self.scroll_until is not None and now >= self.scroll_until:
             self.send("scroll 0 0")
             self.scroll_until = None
-        if self.active and now - self.last_used > self.idle:
+        if self.active and now - self.last_used > self.idle and now >= self.gaze_awake_until:
             self.send("hide")
             self.active = False
             log("pointer off (idle)")
@@ -372,17 +498,21 @@ class Pointer:
 
 
 class Node:
-    """One input event node of a candidate device (mouse or keyboard, USB or Bluetooth)."""
+    """One input event node: a candidate device (mouse or keyboard, USB or Bluetooth),
+    or any other device with volume keys (candidate False, role "volume")."""
 
-    def __init__(self, path, fd, name, bus, vendor, product, uniq, is_mouse, is_keyboard):
+    def __init__(self, path, fd, name, bus, vendor, product, uniq, is_mouse, is_keyboard,
+                 candidate=True, volume_keys=False, only_volume=False):
         self.path, self.fd, self.name = path, fd, name
         self.bus, self.vendor, self.product, self.uniq = bus, vendor, product, uniq
         self.is_mouse, self.is_keyboard = is_mouse, is_keyboard
+        self.candidate, self.volume_keys, self.only_volume = candidate, volume_keys, only_volume
         # One physical device, whatever its node: Bluetooth address, else USB ids plus name.
         base = self.name.split(" Mouse")[0].split(" Keyboard")[0]
         self.id = uniq.lower() if uniq else f"usb:{vendor:04x}:{product:04x}:{base}"
-        self.role = None
+        self.role = None if candidate else "volume"
         self.grabbed = False
+        self.remapped = False  # volume keys remapped to their stand-ins
         self.held = set()  # keys and buttons currently down, released if the device vanishes
         self.last_watch = 0.0
 
@@ -394,7 +524,8 @@ class Node:
 
 
 def probe(path):
-    """Open a node if it is a USB or Bluetooth mouse or keyboard, else return None."""
+    """Open a node if it is a USB or Bluetooth mouse or keyboard, or has volume keys,
+    else return None."""
     try:
         fd = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
     except OSError:
@@ -406,7 +537,7 @@ def probe(path):
         ident = bytearray(8)
         fcntl.ioctl(fd, EVIOCGID, ident)
         bus, vendor, product, _ = struct.unpack("HHHH", ident)
-        if name.startswith(VIRTUAL_PREFIX) or bus not in (BUS_USB, BUS_BLUETOOTH):
+        if name.startswith(VIRTUAL_PREFIX):
             raise ValueError
         uniq_buf = bytearray(64)
         try:
@@ -414,11 +545,15 @@ def probe(path):
             uniq = uniq_buf.split(b"\0", 1)[0].decode(errors="replace")
         except OSError:
             uniq = ""
+        keys = bits(fd, EV_KEY, KEY_MAX + 1)
         is_mouse = REL_X in bits(fd, EV_REL, REL_MAX + 1)
-        is_keyboard = KEY_A in bits(fd, EV_KEY, KEY_MAX + 1)
+        is_keyboard = KEY_A in keys
-        if not (is_mouse or is_keyboard):
+        candidate = bus in (BUS_USB, BUS_BLUETOOTH) and (is_mouse or is_keyboard)
+        volume_keys = bool(keys & VOLUME_CODES)  # stand-ins too: kept from before a relay restart
+        if not (candidate or volume_keys):
             raise ValueError
-        return Node(path, fd, name, bus, vendor, product, uniq, is_mouse, is_keyboard)
+        return Node(path, fd, name, bus, vendor, product, uniq, is_mouse, is_keyboard,
+                    candidate, volume_keys, keys <= VOLUME_CODES)
     except (OSError, ValueError):
         os.close(fd)
         return None
@@ -438,11 +573,17 @@ def main():
     control.setblocking(False)
     watchers = {}  # address -> watch end time
 
-    state = {"pointer": None, "rules": {}}
+    # desktop_until: typing goes to the Frametop desktop until then (ft-screens says so
+    # every second); typing_applied: the grabs match that as of the last apply_roles().
+    # vr_capture_until: every controller button is taken until then (the settings app capturing one).
+    state = {"pointer": None, "rules": {}, "meta_dashboard": False, "share_keys": False,
+             "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0}
 
     def load_config():
         conf = read_config()
         state["rules"] = read_rules()
+        state["meta_dashboard"] = conf.get("META_DASHBOARD", "0") == "1"
+        state["share_keys"] = conf.get("SHARE_KEYS", "0") == "1"
         if conf.get("POINTER", "0") == "1":
             p = state["pointer"] or Pointer(0.03, 30)
             p.sensitivity = float(conf.get("POINTER_SENSITIVITY", "0.03"))
@@ -458,6 +599,34 @@ def main():
     meta_down = False  # Meta pressed with no other key yet: a tap toggles the dashboard
     screens_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
 
+    def vr_bind(now):
+        """Tell the pointer helper which controller buttons to take from games."""
+        if now < state["vr_capture_until"]:
+            buttons = "*"
+        else:
+            state["vr_capture_until"] = 0.0
+            buttons = " ".join(b for b, a in state["rules"]["controller_buttons"].items()
+                               if b in VR_BUTTONS and a in ACTIONS and a not in ("key", "none")) or "-"
+            if state["rules"].get("controller_in_games"):
+                buttons = "+games " + buttons
+        try:
+            screens_sock.sendto(f"vrbind {buttons}".encode(), HELPER)
+        except OSError:
+            pass  # helper not running; it says vrhello when it starts
+
+    def vr_button(button, value, now):
+        """A Frame controller button from the pointer helper."""
+        for addr, until in list(watchers.items()):
+            if now > until:
+                del watchers[addr]
+            else:
+                reply(addr, {"t": "event", "id": VR_DEVICE, "path": "", "name": "Steam Frame controllers",
+                             "type": "vr", "code": button, "value": value})
+        action = state["rules"]["controller_buttons"].get(button)
+        if state["pointer"] and action in ACTIONS and action not in ("key", "none"):
+            state["pointer"].action(action, value, now)
+
+
     def to_screens(code, value):
         """A key for the desktop screens (ft-screens decides whether it types)."""
         if value in (0, 1) and code < BTN_MISC:
@@ -472,6 +641,63 @@ def main():
     seen = {}
     next_scan = 0.0
 
+    volume = Volume()
+    keys_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
+
+    def share_key(node, code, value):
+        """With SHARE_KEYS=1, a key from a keyboard grabbed for the desktop, for programs
+        that watch every keyboard for a hotkey and lose it to the grab. It can't go on
+        another input device: gamescope reads every keyboard itself and would type it
+        into its focused app."""
+        if not state["share_keys"]:
+            return
+        try:
+            keys_sock.sendto(f"key {code} {value} {node.name}".encode(), KEYS)
+        except OSError:
+            pass  # nobody listening
+
+    def restore_keymaps():
+        """Give remapped devices their volume keys back, so they work without the relay."""
+        for node in nodes.values():
+            if node.remapped:
+                try:
+                    remap_volume(node.fd, restore=True)
+                except OSError:
+                    pass  # device already gone
+
+    atexit.register(restore_keymaps)
+    signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))  # so atexit runs on systemctl stop
+
+    def take_volume(node):
+        """Keep the node's volume keys from gamescope and SteamVR, which read every device.
+
+        Returns False for a non-candidate node there's nothing to do with.
+        """
+        if not can_grab:
+            return node.candidate
+        try:
+            found = remap_volume(node.fd)
+        except OSError as e:
+            log(f"remapping volume keys failed for {node.name}: {e}")
+            # Some entries may have changed already: handle their stand-ins and restore them.
+            node.remapped = True
+            found = None
+        if found:
+            node.remapped = True
+            log(f"{node.name} ({node.path}): volume keys taken over (remapped)")
+            return True
+        if found is None and node.only_volume:
+            try:
+                fcntl.ioctl(node.fd, EVIOCGRAB, 1)
+                node.grabbed = True
+                log(f"{node.name} ({node.path}): volume keys taken over (grabbed)")
+                return True
+            except OSError as e:
+                log(f"grab failed for {node.name}: {e}")
+        # Grabbed pointer devices still have their volume keys handled here.
+        log(f"{node.name} ({node.path}): can't take over its volume keys")
+        return node.candidate
+
     def role_of(node):
         rule = state["rules"]["devices"].get(node.id, {})
         if rule.get("role") in ("pointer", "passthrough", "ignore"):
@@ -481,16 +707,39 @@ def main():
 
     def release_held(node):
         for code in node.held:
+            if node.role == "passthrough":
+                share_key(node, code, 0)
+            else:
                 (mouse if code >= BTN_MISC else keyboard).emit(EV_KEY, code, 0)
         node.held.clear()
         mouse.sync()
         keyboard.sync()
 
+    def keys_down(node):
+        buf = bytearray((KEY_MAX + 8) // 8)
+        try:
+            fcntl.ioctl(node.fd, EVIOCGKEY, buf)
+        except OSError:
+            return False
+        return any(buf)
+
     def apply_roles():
+        """Grab pointer devices, and pass-through keyboards while typing goes to the desktop.
+
+        A keyboard with a key down keeps its grab state until it's released, or the key
+        would stay held on one side. Returns True if one is still waiting.
+        """
+        desktop = time.monotonic() < state["desktop_until"]
+        waiting = False
         for node in nodes.values():
+            if not node.candidate:
+                continue  # volume keys only, taken over when found
             role = role_of(node)
-            want_grab = can_grab and role == "pointer"
+            want_grab = can_grab and (role == "pointer"
-            if want_grab != node.grabbed:
+                                      or (role == "passthrough" and node.is_keyboard and desktop))
+            if want_grab != node.grabbed and role == "passthrough" and keys_down(node):
+                waiting = True
+            elif want_grab != node.grabbed:
                 try:
                     fcntl.ioctl(node.fd, EVIOCGRAB, 1 if want_grab else 0)
                     node.grabbed = want_grab
@@ -501,9 +750,15 @@ def main():
             if role != node.role:
                 log(f"{node.name} ({node.path}, {node.id}): {role}{', grabbed' if node.grabbed else ''}")
                 node.role = role
+        if not waiting and desktop != state["typing_applied"]:
+            state["typing_applied"] = desktop
+            log(f"typing goes to {'the desktop (keyboards grabbed)' if desktop else 'Steam'}")
+        return waiting
 
     def drop(node, reason):
         release_held(node)
+        if volume.held and volume.held[0] == node.fd:
+            volume.release()
         os.close(node.fd)
         del nodes[node.fd]
         seen.pop(node.path, None)
@@ -521,13 +776,29 @@ def main():
                 data, addr = control.recvfrom(4096)
             except BlockingIOError:
                 return
-            if not addr:
-                continue  # unbound sender, nowhere to reply
             words = data.decode(errors="replace").split()
             cmd = words[0] if words else ""
+            if cmd == "keyboard":
+                # From ft-screens (unbound, no reply): where typing goes, repeated every second.
+                desktop = len(words) > 1 and words[1] == "desktop"
+                state["desktop_until"] = now + 3.0 if desktop else 0.0
+                continue
+            if cmd == "vrbtn" and len(words) == 3 and words[1] in VR_BUTTONS and words[2] in ("0", "1"):
+                vr_button(words[1], int(words[2]), now)
+                continue
+            if cmd == "vrhello":
+                vr_bind(now)
+                continue
+            if cmd == "gazeawake" and len(words) == 2:
+                if state["pointer"]:
+                    state["pointer"].gaze_awake_until = now + 12.0 if words[1] == "1" else 0.0
+                continue
+            if not addr:
+                continue  # unbound sender, nowhere to reply
             if cmd == "devices":
                 reply(addr, {"t": "devices", "pointer_mode": state["pointer"] is not None,
-                             "actions": ACTIONS, "nodes": [n.describe() for n in nodes.values()]})
+                             "actions": ACTIONS,
+                             "nodes": [n.describe() for n in nodes.values() if n.candidate]})
             elif cmd == "watch":
                 seconds = float(words[1]) if len(words) > 1 else 30
                 watchers[addr] = now + min(seconds, 600)
@@ -537,12 +808,18 @@ def main():
                 apply_roles()
                 if state["pointer"]:
                     state["pointer"].send("reload")
+                vr_bind(now)
                 reply(addr, {"t": "reloaded"})
+            elif cmd == "vrcapture":
+                seconds = float(words[1]) if len(words) > 1 else 30
+                state["vr_capture_until"] = now + min(seconds, 120) if seconds > 0 else 0.0
+                vr_bind(now)
+                reply(addr, {"t": "vrcapture", "seconds": seconds})
             else:
                 reply(addr, {"t": "error", "error": f"unknown command {cmd!r}"})
 
     def broadcast(node, etype, code, value, now):
-        if not watchers:
+        if not watchers or not node.candidate:
             return
         if etype == EV_REL and now - node.last_watch < 0.05:
             return  # motion: enough for an activity light
@@ -555,6 +832,8 @@ def main():
             else:
                 reply(addr, msg)
 
+    vr_bind(time.monotonic())  # a helper that's already running keeps its buttons in step
+    waiting = False  # a keyboard's grab waits for its keys to come up
     while True:
         now = time.monotonic()
         pointer = state["pointer"]
@@ -578,6 +857,9 @@ def main():
                     drop(old, "replaced by a new device node")
                 seen[path] = ino
                 node = probe(path)
+                if node and node.volume_keys and not take_volume(node):
+                    os.close(node.fd)
+                    node = None
                 if node:
                     nodes[node.fd] = node
                     added = True
@@ -585,10 +867,15 @@ def main():
                 apply_roles()
 
         ready, _, _ = select.select(list(nodes) + [control], [], [],
-                                    pointer.timeout() if pointer else 0.5)
+                                    volume.timeout(now, pointer.timeout() if pointer else 0.5))
         now = time.monotonic()
         if pointer:
             pointer.tick(now)
+        volume.tick(now)
+        if state["vr_capture_until"] and now >= state["vr_capture_until"]:
+            vr_bind(now)  # capture over: back to the mapped buttons
+        if (now < state["desktop_until"]) != state["typing_applied"] or waiting:
+            waiting = apply_roles()
         for fd in ready:
             if fd is control:
                 handle_control(now)
@@ -608,12 +895,29 @@ def main():
             for off in range(0, len(data) - EVENT.size + 1, EVENT.size):
                 _, _, etype, code, value = EVENT.unpack_from(data, off)
                 if etype in (EV_KEY, EV_REL):
-                    broadcast(node, etype, code, value, now)
+                    broadcast(node, etype, VOLUME_ORIGINAL.get(code, code) if node.remapped else code,
+                              value, now)
+                if etype == EV_KEY and ((node.remapped and code in VOLUME_ORIGINAL)
+                                        or (node.grabbed and code in VOLUME_STANDIN)):
+                    volume.key(fd, code, value, now)
+                    if value == 1:
+                        meta_down = False  # Meta used as a modifier, not a tap
+                    continue
+                if node.role == "volume":
+                    continue
                 if node.role != "pointer":
-                    # Observed only. A Meta tap on any keyboard toggles the dashboard.
+                    # Observed only, unless typing goes to the desktop. With META_DASHBOARD=1,
+                    # a Meta tap on any keyboard toggles the dashboard.
                     if node.role == "passthrough" and etype == EV_KEY:
                         to_screens(code, value)
-                    if pointer and node.role == "passthrough" and etype == EV_KEY:
+                        if node.grabbed and code < BTN_MISC and value in (0, 1):
+                            share_key(node, code, value)
+                            if value:
+                                node.held.add(code)
+                            else:
+                                node.held.discard(code)
+                    if (pointer and state["meta_dashboard"] and node.role == "passthrough"
+                            and etype == EV_KEY):
                         if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
                             if value == 1:
                                 meta_down = True

layout/ft_layout.py

@@ -39,7 +39,8 @@ Usage (on the Frame host; Frametop Display Settings calls it too):
                                      wait for the screens, skip if "auto" is off
   ft-layout capture                  save the current arrangement as the custom layout
   ft-layout plan                     print the arrangement as JSON (no VR needed)
-  ft-layout scale                    per-screen scale, positions, and primary to KWin
+  ft-layout scale                    per-screen scale, positions (as the screens are around
+                                     you), and primary to KWin
   ft-layout screen-args              ft-screens' --screen arguments for the session script
   ft-layout toggle                   hide or show all screens (ft-screens)
   ft-layout pin all|N left|right     pin screens to a wrist as they are; unpin all|N
@@ -611,6 +612,39 @@ def outputs(env):
 KSCREEN_ROTATION = {1: "normal", 2: "left", 4: "inverted", 8: "right"}  # kscreen-doctor -j "rotation"
 
 
+def arrangement(count):
+    """The screens as you see them from where you are: columns left to right, each top to
+    bottom (0-based screen indices), for KWin's output positions. Screens pinned to a
+    wrist come last. None when there's nothing to go by (gamescope, no ft-screens)."""
+    if backend() != "screens":
+        return None
+    try:
+        sock = screens_socket()
+        f = sock.ask("head").split()
+        eye, heading = tuple(map(float, f[1:4])), float(f[4])
+        gets = [parse_get(sock.ask(f"get {i + 1}")) for i in range(count)]
+    except (RuntimeError, ValueError, IndexError):
+        return None
+    seen, pinned = [], []
+    for i, g in enumerate(gets):
+        if g["hand"] != "none":
+            pinned.append(i)
+            continue
+        rel = turn_yaw(tuple(c - e for c, e in zip(g["center"], eye)), -heading)
+        yaw, pitch = yaw_pitch(rel)
+        half = math.degrees(math.atan2(g["metres"] / 2, max(0.1, math.sqrt(dot(rel, rel)))))
+        seen.append({"i": i, "x": -yaw, "pitch": pitch, "half": half})  # x grows to the right
+    seen.sort(key=lambda b: b["x"])
+    columns = []
+    for b in seen:
+        # One above the other: centres closer sideways than half the narrower screen.
+        if columns and abs(b["x"] - columns[-1][-1]["x"]) < min(b["half"], columns[-1][-1]["half"]):
+            columns[-1].append(b)
+        else:
+            columns.append([b])
+    return [[b["i"] for b in sorted(c, key=lambda b: -b["pitch"])] for c in columns] + [[i] for i in pinned]
+
+
 def apply_scales():
     """Per-screen scale and rotation, positions side by side, and the primary screen (the
     taskbar goes there) to KWin, which keeps them in the session's config."""
@@ -633,8 +667,8 @@ def apply_scales():
             args.append(f"output.{p['id']}.priority.1")
     if args:
         subprocess.run(["kscreen-doctor", *args], capture_output=True, env=env, timeout=20)
-    # Side by side in screen order, centred vertically, so the pointer and dragged windows
+    # Laid out as you see the screens around you (arrangement), centred on one line, so
-    # cross between neighbours.
+    # the pointer and dragged windows cross to the screen you see next to this one.
     outs = outputs(env)
     # kscreen's "size" is in pixels (already turned for a rotation); positions are in
     # logical units, the pixels divided by the scale (KWin rounds up).
@@ -642,18 +676,38 @@ def apply_scales():
               math.ceil(o["size"]["height"] / float(o.get("scale", 1)) - 1e-6))
              for o in outs if o.get("size")]
     if len(sizes) == len(outs) and outs:
-        tallest, x, moves = max(h for _, h in sizes), 0, []
+        columns = arrangement(len(outs))
-        for o, (w, h) in zip(outs, sizes):
+        if not columns or sorted(i for c in columns for i in c) != list(range(len(outs))):
-            want = (x, (tallest - h) // 2)
+            # Nothing to go by (no head pose with the headset off, say): keep KWin's order.
+            columns = [[i] for i in sorted(range(len(outs)), key=lambda i: (outs[i].get("pos", {}).get("x", 0),
+                                                                            outs[i].get("pos", {}).get("y", 0)))]
+        widths = [max(sizes[i][0] for i in c) for c in columns]
+        heights = [sum(sizes[i][1] for i in c) for c in columns]
+        tallest, x, moves = max(heights), 0, []
+        for c, cw, ch in zip(columns, widths, heights):
+            y = (tallest - ch) // 2
+            for i in c:
+                want = (x + (cw - sizes[i][0]) // 2, y)
+                y += sizes[i][1]
+                o = outs[i]
                 if (o.get("pos", {}).get("x"), o.get("pos", {}).get("y")) != want:
                     moves.append(f"output.{o['id']}.position.{want[0]},{want[1]}")
-            x += w
+            x += cw
         if moves:
             subprocess.run(["kscreen-doctor", *moves], capture_output=True, env=env, timeout=20)
             args += moves
     return args
 
 
+def kwin_follow():
+    """KWin's outputs after the screens moved, if the desktop is up."""
+    try:
+        changes = apply_scales()
+        log("kwin: " + (" ".join(changes) if changes else "unchanged"))
+    except RuntimeError as e:
+        log(f"kwin: {e}")
+
+
 def main(argv):
     if len(argv) < 2 or argv[1] in ("-h", "--help"):
         print(__doc__.split("Usage")[1].split("\n", 1)[1])
@@ -669,6 +723,7 @@ def main(argv):
             log(screens_socket().ask("toggle"))
         elif cmd in ("pin", "unpin") and len(argv) >= 3:
             log(screens_socket().ask(" ".join(argv[1:])))
+            kwin_follow()  # pinned screens go last
         elif cmd in ("apply", "capture", "scale"):
             with open(LOCK_PATH, "w") as lock:
                 try:
@@ -691,6 +746,8 @@ def main(argv):
                                 log(f"visibility: {e}")
                     else:
                         apply(wait)
+                    if not wait:
+                        kwin_follow()
                     if wait:
                         # KWin keeps these, but new screens or a changed layout need them once.
                         for _ in range(30):  # Plasma may still be starting
@@ -705,6 +762,7 @@ def main(argv):
                 elif cmd == "capture":
                     for i, s in enumerate(capture()):
                         log(f"screen {i + 1}: {s}")
+                    kwin_follow()
                 else:
                     changes = apply_scales()
                     log("kwin: " + (" ".join(changes) if changes else "unchanged"))

pointer/helper/actions/bindings_frame_controller.json

@@ -0,0 +1,242 @@
+{
+  "action_manifest_version": 0,
+  "controller_type": "frame_controller",
+  "description": "Frametop: each button in an action set of its own, so only mapped ones are taken",
+  "name": "Frametop pointer",
+  "bindings": {
+    "/actions/left_view": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/view",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_view/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/left_dpad_up": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/dpad_up",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_dpad_up/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/left_dpad_down": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/dpad_down",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_dpad_down/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/left_dpad_left": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/dpad_left",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_dpad_left/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/left_dpad_right": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/dpad_right",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_dpad_right/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/left_bumper": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/bumper",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_bumper/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/left_trigger": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/trigger",
+          "mode": "trigger",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_trigger/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/left_grip": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/grip",
+          "mode": "trigger",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_grip/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/left_thumbstick": {
+      "sources": [
+        {
+          "path": "/user/hand/left/input/thumbstick",
+          "mode": "joystick",
+          "inputs": {
+            "click": {
+              "output": "/actions/left_thumbstick/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_menu": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/menu",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_menu/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_a": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/a",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_a/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_b": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/b",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_b/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_x": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/x",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_x/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_y": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/y",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_y/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_bumper": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/bumper",
+          "mode": "button",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_bumper/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_trigger": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/trigger",
+          "mode": "trigger",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_trigger/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_grip": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/grip",
+          "mode": "trigger",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_grip/in/press"
+            }
+          }
+        }
+      ]
+    },
+    "/actions/right_thumbstick": {
+      "sources": [
+        {
+          "path": "/user/hand/right/input/thumbstick",
+          "mode": "joystick",
+          "inputs": {
+            "click": {
+              "output": "/actions/right_thumbstick/in/press"
+            }
+          }
+        }
+      ]
+    }
+  }
+}

pointer/helper/actions/ft_pointer_actions.json

@@ -0,0 +1,197 @@
+{
+  "default_bindings": [
+    {
+      "controller_type": "frame_controller",
+      "binding_url": "bindings_frame_controller.json"
+    }
+  ],
+  "action_sets": [
+    {
+      "name": "/actions/left_view",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/left_dpad_up",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/left_dpad_down",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/left_dpad_left",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/left_dpad_right",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/left_bumper",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/left_trigger",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/left_grip",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/left_thumbstick",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_menu",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_a",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_b",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_x",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_y",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_bumper",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_trigger",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_grip",
+      "usage": "hidden"
+    },
+    {
+      "name": "/actions/right_thumbstick",
+      "usage": "hidden"
+    }
+  ],
+  "actions": [
+    {
+      "name": "/actions/left_view/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/left_dpad_up/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/left_dpad_down/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/left_dpad_left/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/left_dpad_right/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/left_bumper/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/left_trigger/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/left_grip/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/left_thumbstick/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_menu/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_a/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_b/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_x/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_y/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_bumper/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_trigger/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_grip/in/press",
+      "type": "boolean"
+    },
+    {
+      "name": "/actions/right_thumbstick/in/press",
+      "type": "boolean"
+    }
+  ],
+  "localization": [
+    {
+      "language_tag": "en_US",
+      "/actions/left_view": "Frametop: left view",
+      "/actions/left_view/in/press": "Press",
+      "/actions/left_dpad_up": "Frametop: left dpad up",
+      "/actions/left_dpad_up/in/press": "Press",
+      "/actions/left_dpad_down": "Frametop: left dpad down",
+      "/actions/left_dpad_down/in/press": "Press",
+      "/actions/left_dpad_left": "Frametop: left dpad left",
+      "/actions/left_dpad_left/in/press": "Press",
+      "/actions/left_dpad_right": "Frametop: left dpad right",
+      "/actions/left_dpad_right/in/press": "Press",
+      "/actions/left_bumper": "Frametop: left bumper",
+      "/actions/left_bumper/in/press": "Press",
+      "/actions/left_trigger": "Frametop: left trigger",
+      "/actions/left_trigger/in/press": "Press",
+      "/actions/left_grip": "Frametop: left grip",
+      "/actions/left_grip/in/press": "Press",
+      "/actions/left_thumbstick": "Frametop: left thumbstick",
+      "/actions/left_thumbstick/in/press": "Press",
+      "/actions/right_menu": "Frametop: right menu",
+      "/actions/right_menu/in/press": "Press",
+      "/actions/right_a": "Frametop: right a",
+      "/actions/right_a/in/press": "Press",
+      "/actions/right_b": "Frametop: right b",
+      "/actions/right_b/in/press": "Press",
+      "/actions/right_x": "Frametop: right x",
+      "/actions/right_x/in/press": "Press",
+      "/actions/right_y": "Frametop: right y",
+      "/actions/right_y/in/press": "Press",
+      "/actions/right_bumper": "Frametop: right bumper",
+      "/actions/right_bumper/in/press": "Press",
+      "/actions/right_trigger": "Frametop: right trigger",
+      "/actions/right_trigger/in/press": "Press",
+      "/actions/right_grip": "Frametop: right grip",
+      "/actions/right_grip/in/press": "Press",
+      "/actions/right_thumbstick": "Frametop: right thumbstick",
+      "/actions/right_thumbstick/in/press": "Press"
+    }
+  ]
+}

pointer/helper/ft-pointer.cpp

@@ -74,6 +74,20 @@
 // desktops) also report 0x0, but they're placed as dashboard tabs, stay up when the
 // dashboard closes, and ComputeOverlayIntersection hits them normally.
 //
+// SteamVR Settings (a workaround for that page only): Steam's pages (Library and the rest)
+// are drawn in valve.steam.gamepadui.main, a dashboard overlay ComputeOverlayIntersection hits
+// exactly. SteamVR's Settings page isn't: the main overlay is hidden, and the page is drawn by
+// the scene-graph panel (valve.steam.gamepadui.frame.menu.N), whose shape OpenVR doesn't give
+// out. Its transform's plane isn't the page's surface, which is nearer, so the laser, starting
+// a few cm in front of where we thought the page was, started behind it: most of the page
+// took no clicks, which went through to a desktop screen behind, and the page covered our
+// dot. So while the cursor is on that page (OnSettingsPage), the laser starts near the eye
+// (SETTINGS_ORIGIN) and SteamVR's own hit test finds the page; our dot is drawn close in front
+// (SETTINGS_DOT), and the laser-catching dot sits far behind everything (SETTINGS_CATCHER),
+// invisible and with SteamVR's hit dot hidden, so it can't cover the page. The beam and
+// SteamVR's hit dot on the page then look like a controller's. Everywhere else nothing
+// changes.
+//
 // Panel edges: off a panel, the cursor stays on that panel's plane while it's within
 // POINTER_EDGE_REACH (0.3 m) of the last point it touched, instead of jumping to
 // POINTER_DISTANCE. A floating panel's resize margins and the window controls under it
@@ -86,6 +100,58 @@
 // doesn't jump the cursor to free space or swap in the laser-catching dot, which made
 // SteamVR's resize snap back.
 //
+// Head follow (experimental, off by default; POINTER_FOLLOW=1, or the relay's "follow toggle"): the cursor
+// is carried by a reference direction, where the head faced when it last settled, and turns
+// with it, keeping its offset (mouse movement changes the offset, up to POINTER_FOLLOW_REACH,
+// 70 deg, so the cursor can sit in a corner of the view). While the head stays within
+// POINTER_LEASH_DEG (10) of the reference, nothing moves on its own: the cursor stays put in
+// the room. Once the head has been past the leash for POINTER_LEASH_DELAY (0.2 s; a glance
+// out and back doesn't count), the reference follows: it eases toward the head's facing with a
+// time constant of POINTER_LEASH_RETURN (0.2 s), never falling further behind than the leash
+// (or than it already was), until it lands on the facing, and the cursor is back where it was
+// in the view. Then it waits for the leash again. Earlier tries: dragging the reference only at
+// the leash's end left it up to the leash off after turning back (getting it centred took an
+// overshoot), and easing it all the time moved the cursor on every small head movement. At 0
+// the reference is the head's facing, so the cursor is head-locked. Head roll is ignored (the
+// frames have no roll), so tilting the head doesn't swing the cursor. The ray origin (the
+// anchor) moves to the eye with the reference, so leaning inside the leash doesn't move the
+// cursor either. While the left
+// button is held (and the drop hold after it), the leash still moves the reference but the
+// cursor stays put in the room, so a click or a drag can't be nudged by the head; the offset
+// is taken up from where the cursor is when the hold ends, so it doesn't jump.
+//
+// Gaze mode (experimental, off by default; POINTER_GAZE=1, "gaze on|off|toggle", or the
+// relay's gaze_toggle): the pointer goes where you look, and the mouse does the last bit
+// (MAGIC pointing: Zhai, Morimoto and Ihde, CHI 1999). The gaze service (gaze/ft-gazed)
+// sends the corrected gaze 90 times a second, "gz <yaw> <pitch> <raw yaw> <raw pitch>"
+// (head-relative degrees), and while the gaze has the pointer, the cursor ray is simply
+// that gaze from the eye: nothing is steered, so nothing can pile up. Moving the mouse takes
+// the pointer from the gaze, and it moves from where the gaze left it, as usual. Looking
+// well away from it (more than POINTER_GAZE_RETAKE, 5 deg, for 120 ms, with the mouse still
+// for 300 ms) gives it back to the gaze; small eye movements around the pointer don't.
+//   A left press while the gaze has the pointer isn't sent yet: the pointer stops where the
+// gaze put it, and if the gaze is off, you drag it onto what you meant with the mouse
+// (still holding the button; panels only see it hover). The release clicks there, a press
+// and a release 40 ms apart. Held still for POINTER_GAZE_HOLD (0.5 s) instead, it becomes
+// a real press where the pointer is, so drags work: hold, then move. After a click that
+// didn't need correcting, and after a drag, the gaze has the pointer again.
+//   Outside games (no scene application), gaze mode keeps the pointer: the relay doesn't
+// release it when the mouse is idle ("gazeawake 1|0" tells it). A controller that moves
+// still releases it (the mouse is gaze mode's only pointer device), and in games the mouse
+// wakes it and idling releases it, as without gaze.
+//   The dot only shows while the mouse moves it (within POINTER_GAZE_SHOW, 1 s), while a
+// press is held, and briefly for each click (a pulse);
+// otherwise it's transparent (still there for the laser to land on). The gaze moving it
+// doesn't show it: you know where you're looking.
+//   When the mouse took the pointer and you then click, the nudge was probably onto what
+// you were looking at: from the raw gaze when the mouse took over to where you clicked is
+// the tracker's error there. The helper sends it to ft-gazed as a lesson ("lesson <raw yaw>
+// <raw pitch> <true yaw> <true pitch>", the true direction relative to the head as it was
+// when the mouse took over) if the mouse moved between 0.2 deg and POINTER_GAZE_NUDGE_MAX
+// (8 deg) and the click came within 10 s; more is using the mouse, not a nudge. A held
+// press dragged onto the target is the same: from the raw gaze at the press to the release. With no
+// fresh gaze (a blink, the service stopped, the headset off), the pointer stays put.
+//
 // Placement (for layout): SteamVR keeps a floating panel's position inside the
 // dashboard, where nothing outside can set it, so the helper carries panels like a user
 // would. It measures the panel (md::ScanPanel), aims the device at its grab bar
@@ -100,7 +166,10 @@
 // twice while it's more than 1.5 cm or 1 deg off.
 //
 // Commands (datagrams on @ft_pointer_helper): show, hide, recenter, move <dyaw> <dpitch>,
+// follow on|off|toggle (head follow, until the next restart or a change to POINTER_FOLLOW),
+// gaze on|off|toggle|? (gaze mode, likewise with POINTER_GAZE; ? only asks), gz ... (the gaze, from ft-gazed),
 // reload (re-read the settings below), debug (toggle a twice-a-second state log),
+// vrbind/vrglobal/vrstatus (Frame controller buttons, see vrbuttons.h),
 // and btn/scroll lines, which are forwarded to the driver unchanged. For layouts, with a
 // reply datagram to the sender's (abstract) address:
 //   place <overlay> <x> <y> <z> <yaw> <pitch> [roll [grab]]: centre in the standing
@@ -117,9 +186,14 @@
 // POINTER_ORIGIN_FRACTION (0.95): the laser starts this far along the eye-to-cursor line,
 // but never closer than POINTER_ORIGIN_MARGIN (0.15 m) to the cursor point: SteamVR's
 // small controls (undock, frame buttons) float a few centimetres in front of their
-// panel, and a laser that starts behind them can't hit them.
+// panel, and a laser that starts behind them can't hit them. POINTER_FOLLOW (0) and
+// POINTER_LEASH_DEG (10), POINTER_LEASH_DELAY (0.2 s), POINTER_LEASH_RETURN (0.2 s),
+// POINTER_FOLLOW_REACH (70 deg): head follow, above. POINTER_GAZE (0), POINTER_GAZE_RETAKE
+// (5 deg), POINTER_GAZE_NUDGE_MAX (8 deg), POINTER_GAZE_HOLD (0.5 s), POINTER_GAZE_SHOW (1 s):
+// gaze mode, above.
 #include <openvr.h>
 
+#include "vrbuttons.h"
 #include "vrmath.h"
 
 #include <algorithm>
@@ -137,6 +211,8 @@
 #include <tuple>
 #include <vector>
 
+#include <climits>
+
 #include <sys/socket.h>
 #include <sys/un.h>
 #include <unistd.h>
@@ -184,6 +260,15 @@ int AbstractSocket(const char *name, bool bindIt) {
     return fd;
 }
 
+std::string ExeDir() {
+    char buf[PATH_MAX];
+    const ssize_t n = readlink("/proc/self/exe", buf, sizeof buf - 1);
+    if (n <= 0) return ".";
+    buf[n] = 0;
+    std::string p(buf);
+    return p.substr(0, p.rfind('/'));
+}
+
 void SendTo(int fd, const char *name, const std::string &msg) {
     sockaddr_un addr{};
     addr.sun_family = AF_UNIX;
@@ -290,6 +375,42 @@ std::vector<uint8_t> DotTexture(int size) {
     return px;
 }
 
+// Unit vector v, turned toward unit vector `center` until it's at most maxRad from it.
+Vec3 PullWithin(Vec3 v, Vec3 center, double maxRad) {
+    if (std::acos(std::clamp(Dot(v, center), -1.0, 1.0)) <= maxRad) return v;
+    Vec3 axis = Cross(center, v);
+    if (Length(axis) < 1e-9) axis = Cross(center, {0, 1, 0});  // opposite: any perpendicular
+    return RotateAbout(center, Normalize(axis), maxRad);
+}
+
+// Unit direction with its pitch limited to +-maxDeg (AimBasis needs it off vertical).
+Vec3 LimitPitch(Vec3 d, double maxDeg) {
+    const double pitch = std::clamp(std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI, -maxDeg, maxDeg);
+    return Direction(std::atan2(-d.x, -d.z) * 180 / M_PI, pitch);
+}
+
+// SteamVR Settings page (see the top): the laser starts this far from the eye, the dot is
+// drawn this far out, and the laser-catching dot sits this far out (metres).
+constexpr double SETTINGS_ORIGIN = 0.25, SETTINGS_DOT = 0.6, SETTINGS_CATCHER = 8.0;
+
+// Whether the line of sight from `eye` along `d` crosses the SteamVR Settings page, drawn by
+// the dashboard's scene-graph panel (valve.steam.gamepadui.frame.menu.N, transform t); see
+// "SteamVR Settings" at the top. The page has no size in OpenVR, so this is its area as
+// measured on the Frame, generously: in metres from the panel's origin, which is near the
+// page's left edge, it ran from about -0.35 (the sidebar) to 1.15 across and +-0.4 up and
+// down, with the transform scaled 0.369. Kept in the transform's units so it scales with it.
+bool OnSettingsPage(const vr::HmdMatrix34_t &t, Vec3 eye, Vec3 d) {
+    const Vec3 c = Position(t), x{t.m[0][0], t.m[1][0], t.m[2][0]}, y{t.m[0][1], t.m[1][1], t.m[2][1]},
+               z{t.m[0][2], t.m[1][2], t.m[2][2]};
+    const double sx = Dot(x, x), sy = Dot(y, y), denom = Dot(d, z);
+    if (sx < 1e-9 || sy < 1e-9 || std::fabs(denom) < 1e-6) return false;
+    const double along = Dot(c - eye, z) / denom;
+    if (along <= 0) return false;
+    const Vec3 off = eye + d * along - c;
+    const double u = Dot(off, x) / sx, v = Dot(off, y) / sy;  // in the transform's units
+    return u >= -1.35 && u <= 3.4 && std::fabs(v) <= 1.25;
+}
+
 }  // namespace
 
 // Placement speeds (see "Placement" at the top).
@@ -298,7 +419,13 @@ constexpr double kPlaceDegPerSec = 60;      // tested: 40 deg/s is applied exact
 int main() {
     double freeDistance = 1.5, cursorDeg = 0.4, originFraction = 0.95, originMargin = 0.15, sceneRadius = 0.5,
            edgeReach = 0.3, grabOffset = 0.075,
-           slideSpeed = 0.5;
+           slideSpeed = 0.5, leashDeg = 10, leashReturn = 0.2, leashDelay = 0.2, followReach = 70;
+    // Head follow (see the top). followConf is POINTER_FOLLOW as last read: a reload only
+    // overrides a "follow" command when the setting itself changed.
+    bool follow = false, followConf = false, followReset = true;
+    // Gaze mode (see the top); gazeConf is POINTER_GAZE as last read, like followConf.
+    bool gazeOn = false, gazeConf = false;
+    double gazeRetake = 5, gazeNudgeMax = 8, gazeHold = 0.5, gazeShow = 1;
     auto loadConfig = [&] {
         const auto conf = ReadConfig();
         freeDistance = std::clamp(ConfDouble(conf, "POINTER_DISTANCE", 1.5), 0.3, 10.0);
@@ -309,6 +436,18 @@ int main() {
         edgeReach = std::clamp(ConfDouble(conf, "POINTER_EDGE_REACH", 0.3), 0.0, 2.0);
         grabOffset = std::clamp(ConfDouble(conf, "LAYOUT_GRAB_OFFSET", 0.075), 0.0, 1.0);
         slideSpeed = std::clamp(ConfDouble(conf, "LAYOUT_SLIDE_SPEED", 0.5), 0.02, 2.0);
+        leashDeg = std::clamp(ConfDouble(conf, "POINTER_LEASH_DEG", 10), 0.0, 90.0);
+        leashReturn = std::clamp(ConfDouble(conf, "POINTER_LEASH_RETURN", 0.2), 0.0, 5.0);
+        leashDelay = std::clamp(ConfDouble(conf, "POINTER_LEASH_DELAY", 0.2), 0.0, 5.0);
+        followReach = std::clamp(ConfDouble(conf, "POINTER_FOLLOW_REACH", 70), 10.0, 89.0);
+        const bool wantFollow = ConfDouble(conf, "POINTER_FOLLOW", 0) != 0;
+        if (wantFollow != followConf) follow = followConf = wantFollow, followReset = true;
+        gazeRetake = std::clamp(ConfDouble(conf, "POINTER_GAZE_RETAKE", 5), 1.0, 45.0);
+        gazeNudgeMax = std::clamp(ConfDouble(conf, "POINTER_GAZE_NUDGE_MAX", 8), 1.0, 30.0);
+        gazeHold = std::clamp(ConfDouble(conf, "POINTER_GAZE_HOLD", 0.5), 0.1, 5.0);
+        gazeShow = std::clamp(ConfDouble(conf, "POINTER_GAZE_SHOW", 1), 0.0, 30.0);
+        const bool wantGaze = ConfDouble(conf, "POINTER_GAZE", 0) != 0;
+        if (wantGaze != gazeConf) gazeOn = gazeConf = wantGaze;
     };
     loadConfig();
     const float laserWidth = float(ConfDouble(ReadConfig(), "POINTER_LASER_WIDTH", 0.8));
@@ -354,6 +493,14 @@ int main() {
 
     const int in = AbstractSocket("ft_pointer_helper", true);
     const int out = AbstractSocket(nullptr, false);
+    // Frame controller buttons (vrbuttons.h). The build puts the binary in pointer/helper/build.
+    ControllerButtons controllerButtons;
+    {
+        const std::string manifest = ExeDir() + "/../actions/ft_pointer_actions.json";
+        char real[PATH_MAX];
+        controllerButtons.Init(realpath(manifest.c_str(), real) ? real : manifest);
+    }
+    SendTo(out, "frametop_relay", "vrhello");  // the relay answers with the mapped buttons
     OverlayList overlays;
     overlays.Start();
     std::map<std::string, vr::VROverlayHandle_t> handles;
@@ -372,6 +519,8 @@ int main() {
     bool leftHeld = false, tilting = false, tiltStart = false, swallowedRight = false;
     double tiltYaw = 0, tiltPitch = 0;
     double dragDistance = 0, lastDistance = 1.5;  // drag lock: distance from the anchor at the press
+    bool onVrSettings = false;       // the cursor is on the SteamVR Settings page (kept while dragging)
+    bool catcherHidesHit = false;    // the laser-catching dot hides SteamVR's hit dot (Settings page)
     Clock::time_point dropHoldUntil{};  // after a left release: keep the drag pose this long
     bool debug = false;
     std::string lastHit;
@@ -392,7 +541,88 @@ int main() {
     };
     Vec3 anchor;
     double yaw = 0, pitch = 0;
+    Vec3 followRef{0, 0, -1};  // head follow's reference direction (see the top)
+    auto followAt = std::chrono::steady_clock::now();  // its last update, for the easing
+    bool following = false;                           // past the leash: easing toward the head
+    double followLag = 0;                             // radians the reference trails the head
+    std::chrono::steady_clock::time_point leashOutSince{};  // head past the leash since (delay)
+    // Gaze mode (see the top).
+    struct Gaze {
+        double hy = 0, hp = 0, rhy = 0, rhp = 0;  // corrected, and raw
+        Clock::time_point at{};
+    } gz;
+    bool gazeOwns = true;  // the pointer follows the gaze; false: the mouse has it
+    bool nudging = false;  // the mouse took it from the gaze: the next click may be a lesson
+    double nudgeRawHy = 0, nudgeRawHp = 0, nudgeMoved = 0;
+    vr::HmdMatrix34_t nudgeHead{}, lastHead{};
+    bool haveHead = false, havePoint = false;
+    Clock::time_point nudgeAt{}, retakeSince{};
+    // A held-back press (see the top): aimHeld while the button is down; then the click
+    // (clickPress at the end of the next frame, clickRelease 40 ms later). gazeBack: give
+    // the gaze the pointer again when the press or click is over.
     vr::TrackedDeviceIndex_t ours = vr::k_unTrackedDeviceIndexInvalid;
+    bool aimHeld = false, clickPress = false, clickRelease = false, gazeBack = false;
+    Clock::time_point aimSince{}, clickReleaseAt{};
+    // Gaze mode outside games, and its dot (see the top): lastMove/lastHeld/pulseAt.
+    bool inGame = false, gazeAwake = false;
+    Clock::time_point inGameAt{}, gazeAwakeAt{};
+    Clock::time_point lastMove{}, lastHeld{}, pulseAt{};
+    // The left button, as sent to the driver.
+    auto pressLeft = [&] {
+        // ft-screens sends the keyboard to the panel clicked last; it sees clicks on
+        // its own screens, but only we know when one lands on another panel.
+        SendTo(out, "ft_screens", "click " + (lastHit.empty() ? std::string("-") : lastHit));
+        // A click after nudging the gaze-placed pointer: the nudge is a lesson (see the top).
+        if (gazeOn && nudging && !gazeOwns && havePoint && Clock::now() - nudgeAt < std::chrono::seconds(10) &&
+            nudgeMoved >= 0.2 && nudgeMoved <= gazeNudgeMax) {
+            const Vec3 d = RotateInverse(nudgeHead, Normalize(lastPoint - Position(nudgeHead)));
+            char msg[160];
+            std::snprintf(msg, sizeof msg, "lesson %.3f %.3f %.3f %.3f", nudgeRawHy, nudgeRawHp,
+                          std::atan2(-d.x, -d.z) * 180 / M_PI, std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI);
+            SendTo(out, "ft_gazed", msg);
+            if (debug) std::printf("gaze %s (nudged %.2f deg)\n", msg, nudgeMoved);
+            if (debug) std::fflush(stdout);
+        }
+        nudging = false;
+        leftHeld = true;
+        dragDistance = lastDistance;
+        tiltYaw = tiltPitch = 0;  // a new drag starts untilted
+        dropHoldUntil = {};
+        pulseAt = Clock::now();
+        SendTo(out, "ft_pointer", "btn trigger 1");
+    };
+    auto releaseLeft = [&] {
+        leftHeld = false;
+        tilting = false;
+        // Hold the drag pose (tilt, frozen distance) while SteamVR finishes the drop.
+        dropHoldUntil = Clock::now() + std::chrono::milliseconds(500);
+        SendTo(out, "ft_pointer", "btn trigger 0");
+        if (gazeBack) gazeOwns = true, gazeBack = false;
+    };
+    // The left button, from the relay's "btn trigger", with gaze mode's held-back press (see
+    // the top).
+    auto leftButton = [&](bool down) {
+        if (down) {
+            if (gazeOn && gazeOwns && !aimHeld && !clickPress && !clickRelease) {
+                // Hold the press back: the pointer stops where the gaze put it.
+                gazeOwns = false;
+                nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
+                nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
+                nudgeAt = aimSince = Clock::now(), nudgeMoved = 0;
+                aimHeld = true;
+                return;
+            }
+            pressLeft();
+            return;
+        }
+        if (aimHeld) {
+            aimHeld = false;
+            clickPress = true;  // after this frame's pose, so it lands where the pointer was moved to
+            gazeBack = nudgeMoved < 0.2;
+            return;
+        }
+        if (leftHeld) releaseLeft();
+    };
     auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10);
 
     // --- Panel placement (see "Placement" at the top of the file) ---
@@ -570,6 +800,23 @@ int main() {
             std::fflush(stdout);
         }
 
+        // Outside games, gaze mode keeps the pointer (see the top); the relay needs to know.
+        {
+            const auto t = Clock::now();
+            if (t - inGameAt > std::chrono::milliseconds(500)) {
+                inGameAt = t;
+                inGame = vr::VRApplications()->GetCurrentSceneProcessId() != 0;
+            }
+            const bool awake = gazeOn && !inGame && !headsetOff;
+            if (awake != gazeAwake || t - gazeAwakeAt > std::chrono::seconds(5)) {
+                if (awake != gazeAwake) std::printf("gaze keeps the pointer: %s\n", awake ? "yes" : "no (off, in a game, or headset off)");
+                if (awake != gazeAwake) std::fflush(stdout);
+                gazeAwake = awake;
+                gazeAwakeAt = t;
+                SendTo(out, "frametop_relay", awake ? "gazeawake 1" : "gazeawake 0");
+            }
+        }
+
         // Commands from the relay.
         char buf[256];
         ssize_t n;
@@ -585,6 +832,42 @@ int main() {
                 if (senderLen > offsetof(sockaddr_un, sun_path))
                     sendto(out, msg.data(), msg.size(), 0, reinterpret_cast<const sockaddr *>(&sender), senderLen);
             };
+            // The gaze, from ft-gazed: not mouse input, it never wakes the pointer.
+            double g[4];
+            if (std::sscanf(buf, "gz %lf %lf %lf %lf", &g[0], &g[1], &g[2], &g[3]) == 4) {
+                gz = {g[0], g[1], g[2], g[3], Clock::now()};
+                continue;
+            }
+            if (std::strncmp(buf, "vrbind", 6) == 0) {
+                std::printf("controller buttons: %s\n", controllerButtons.Bind(buf + 6).c_str());
+                std::fflush(stdout);
+                continue;
+            }
+            if (std::strncmp(buf, "vrglobal", 8) == 0) {
+                const char *arg = buf + 8;
+                while (*arg == ' ') ++arg;
+                ControllerButtons::SetGlobal(std::strncmp(arg, "off", 3) != 0);
+                reply(controllerButtons.Status());
+                continue;
+            }
+            if (std::strncmp(buf, "vrstatus", 8) == 0) {
+                reply(controllerButtons.Status());
+                continue;
+            }
+            if (std::strncmp(buf, "gaze", 4) == 0) {
+                const char *arg = buf + 4;
+                while (*arg == ' ') ++arg;
+                if (*arg != '?') {  // "gaze ?" only asks
+                    gazeOn = std::strncmp(arg, "on", 2) == 0    ? true
+                             : std::strncmp(arg, "off", 3) == 0 ? false
+                                                                : !gazeOn;
+                    gazeOwns = true, nudging = false;
+                    std::printf("gaze mode %s\n", gazeOn ? "on" : "off");
+                    std::fflush(stdout);
+                }
+                reply(gazeOn ? "ok on" : "ok off");
+                continue;
+            }
             const bool mouseInput = std::strncmp(buf, "move", 4) == 0 || std::strncmp(buf, "btn", 3) == 0 ||
                                     std::strncmp(buf, "scroll", 6) == 0;
             if (mouseInput) lastMouse = Clock::now();
@@ -633,15 +916,11 @@ int main() {
                 continue;
             }
             if (std::strncmp(buf, "btn trigger 1", 13) == 0) {
-                leftHeld = true;
+                leftButton(true);
-                dragDistance = lastDistance;
+                continue;
-                tiltYaw = tiltPitch = 0;  // a new drag starts untilted
-                dropHoldUntil = {};
             } else if (std::strncmp(buf, "btn trigger 0", 13) == 0) {
-                leftHeld = false;
+                leftButton(false);
-                tilting = false;
+                continue;
-                // Hold the drag pose (tilt, frozen distance) while SteamVR finishes the drop.
-                dropHoldUntil = Clock::now() + std::chrono::milliseconds(500);
             } else if (std::strncmp(buf, "btn b 1", 7) == 0 && leftHeld) {
                 tilting = tiltStart = swallowedRight = true;  // right press while dragging: tilt, no right-click
                 continue;
@@ -655,6 +934,15 @@ int main() {
                 continue;
             }
             if (std::sscanf(buf, "move %lf %lf", &a, &b) == 2) {
+                lastMove = Clock::now();  // the dot shows while the mouse moves it (gaze mode)
+                if (gazeOn && gazeOwns) {
+                    // The mouse takes the pointer from the gaze, from where the gaze left it.
+                    gazeOwns = false;
+                    nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
+                    nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
+                    nudgeAt = Clock::now(), nudgeMoved = 0;
+                }
+                if (nudging || aimHeld) nudgeMoved += std::hypot(a, b);
                 if (!anchored) recenter = true;
                 yaw += a;
                 while (yaw > 180) yaw -= 360;
@@ -664,8 +952,16 @@ int main() {
                 recenter = true;
             } else if (std::strncmp(buf, "reload", 6) == 0) {
                 loadConfig();
-                std::printf("reloaded: free distance %.2f m, dot %.2f deg, origin %.2f\n", freeDistance, cursorDeg,
+                std::printf("reloaded: free distance %.2f m, dot %.2f deg, origin %.2f, head follow %s, leash %.0f deg\n",
-                            originFraction);
+                            freeDistance, cursorDeg, originFraction, follow ? "on" : "off", leashDeg);
+                std::fflush(stdout);
+            } else if (std::strncmp(buf, "follow", 6) == 0) {
+                const char *arg = buf + 6;
+                while (*arg == ' ') ++arg;
+                const bool was = follow;
+                follow = std::strncmp(arg, "on", 2) == 0 ? true : std::strncmp(arg, "off", 3) == 0 ? false : !follow;
+                if (follow && !was) followReset = true;
+                std::printf("head follow %s (leash %.0f deg)\n", follow ? "on" : "off", leashDeg);
                 std::fflush(stdout);
             } else if (std::strncmp(buf, "show", 4) == 0) {
                 if (!active) wake(Clock::now());
@@ -754,8 +1050,78 @@ int main() {
             pitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
             anchored = true;
             recenter = false;
+            followReset = true;
         }
 
+        // Gaze mode (see the top): the gaze has the pointer, or takes it back when you look
+        // well away from it. Not while a press holds the pointer, and only on fresh gaze.
+        if (hmd.bPoseIsValid) lastHead = hmd.mDeviceToAbsoluteTracking, haveHead = true;
+        if (gazeOn && active && hmd.bPoseIsValid && !tilting && !leftHeld && !aimHeld && !clickPress && !clickRelease &&
+            tnow >= dropHoldUntil &&
+            tnow - gz.at < std::chrono::milliseconds(150)) {
+            const Vec3 g = Rotate(hmd.mDeviceToAbsoluteTracking, Direction(gz.hy, gz.hp));
+            if (!gazeOwns && havePoint) {
+                const double off = std::acos(std::clamp(Dot(g, Normalize(lastPoint - eye)), -1.0, 1.0)) * 180 / M_PI;
+                if (off > gazeRetake && tnow - lastMouse > std::chrono::milliseconds(300)) {
+                    if (retakeSince == Clock::time_point{}) retakeSince = tnow;
+                    if (tnow - retakeSince >= std::chrono::milliseconds(120)) gazeOwns = true, nudging = false;
+                } else {
+                    retakeSince = {};
+                }
+            }
+            if (gazeOwns) {
+                retakeSince = {};
+                anchor = eye;
+                anchored = true;
+                yaw = std::atan2(-g.x, -g.z) * 180 / M_PI;
+                pitch = std::clamp(std::asin(std::clamp(g.y, -1.0, 1.0)) * 180 / M_PI, -85.0, 85.0);
+            }
+        }
+
+        if (aimHeld || leftHeld || clickPress || clickRelease) lastHeld = tnow;
+
+        // Head follow (see the top): past the leash (for the delay), ease the reference to the
+        // head's facing, and turn the cursor with it. Not in gaze mode: the gaze places it.
+        if (follow && !gazeOn && active && anchored && hmd.bPoseIsValid) {
+            const Vec3 head = LimitPitch(Vec3{-hm[0][2], -hm[1][2], -hm[2][2]}, 85);
+            if (followReset) {
+                followRef = head, followLag = 0, leashOutSince = {};
+                followReset = following = false;
+            }
+            const double dt = std::min(0.1, std::chrono::duration<double>(tnow - followAt).count());
+            const double leash = leashDeg * M_PI / 180;
+            const double lag = std::acos(std::clamp(Dot(followRef, head), -1.0, 1.0));
+            double keep = lag;  // how far the reference stays behind the head after this frame
+            if (leash <= 0) {
+                keep = 0;  // head-locked
+            } else if (following) {
+                keep = lag * (leashReturn > 0 ? std::exp(-dt / leashReturn) : 0.0);
+                // A fast turn drags it at the leash's end; past it already (after the delay), it
+                // can't fall further behind, and it closes in from there without a jump.
+                keep = std::min(keep, std::max(leash, followLag));
+                if (keep < 0.05 * M_PI / 180) keep = 0, following = false;  // landed on the facing
+            } else if (lag > leash) {
+                if (leashOutSince == decltype(leashOutSince){}) leashOutSince = tnow;
+                if (std::chrono::duration<double>(tnow - leashOutSince).count() >= leashDelay)
+                    following = true, leashOutSince = {};
+            } else {
+                leashOutSince = {};  // back inside before the delay: a glance
+            }
+            followLag = keep;
+            const Vec3 ref = LimitPitch(PullWithin(followRef, head, keep), 85);
+            if (!leftHeld && tnow >= dropHoldUntil) {
+                // The cursor keeps its offset from the reference (frames without roll).
+                Vec3 d = FromBasis(AimBasis(ref), ToBasis(AimBasis(followRef), Direction(yaw, pitch)));
+                d = PullWithin(Normalize(d), ref, followReach * M_PI / 180);
+                yaw = std::atan2(-d.x, -d.z) * 180 / M_PI;
+                pitch = std::clamp(std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI, -85.0, 85.0);
+                // The ray origin closes in on the eye as the reference does on the facing.
+                anchor = eye + (anchor - eye) * (leash <= 0 ? 0.0 : lag > 1e-6 ? keep / lag : following ? 0.0 : 1.0);
+            }
+            followRef = ref;
+        }
+        followAt = tnow;
+
         // Slow work, once a second: overlay handles, our device index, laser width.
         const auto now = std::chrono::steady_clock::now();
         if (now - lastSlow > std::chrono::seconds(1)) {
@@ -907,14 +1273,68 @@ int main() {
             }
             const bool onScene = !dragging && ((bestScene && best < 1e8) || onEdge);
             const bool onPanel = dragging || (best < 1e8 && !onScene);
+            const Vec3 sight = Normalize(point - eye);
+            if (!dragging) {
+                // SteamVR Settings (see the top): the dashboard's main panel is hidden and its
+                // scene-graph panel shows the page.
+                onVrSettings = false;
+                const auto mainIt = visible.find("valve.steam.gamepadui.main");
+                if (overlay->IsDashboardVisible() && !(mainIt != visible.end() && mainIt->second)) {
+                    for (const auto &[key, handle] : handles) {
+                        if (!visible[key] || key.rfind("valve.steam.gamepadui.frame.menu.", 0) != 0) continue;
+                        vr::ETrackingUniverseOrigin uo;
+                        vr::HmdMatrix34_t t{};
+                        if (overlay->GetOverlayTransformAbsolute(handle, &uo, &t) == vr::VROverlayError_None &&
+                            OnSettingsPage(t, eye, sight))
+                            onVrSettings = true;
+                    }
+                }
+            }
+            if (onVrSettings != catcherHidesHit) {
+                overlay->SetOverlayFlag(cursor, vr::VROverlayFlags_HideLaserIntersection, onVrSettings);
+                catcherHidesHit = onVrSettings;
+            }
 
-            {
+            if (onVrSettings) {
+                // The dot close in front of the page, which is nearer than any guess of ours;
+                // the laser-catching dot far behind everything, invisible, so it never covers
+                // the page, with SteamVR's hit dot hidden on it.
+                const Vec3 near = eye + sight * SETTINGS_DOT, far = eye + sight * SETTINGS_CATCHER;
+                double alpha = 1;
+                if (gazeOn) {
+                    auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
+                    alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
+                }
+                overlay->SetOverlayAlpha(marker, float(alpha));
+                overlay->SetOverlayWidthInMeters(marker, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360)));
+                auto mm = Billboard(near, eye);
+                overlay->SetOverlayTransformAbsolute(marker, vr::TrackingUniverseStanding, &mm);
+                overlay->SetOverlayAlpha(cursor, 0);
+                overlay->SetOverlayWidthInMeters(cursor, float(2 * SETTINGS_CATCHER * std::tan(cursorDeg * M_PI / 360)));
+                auto mc = Billboard(far, eye);
+                overlay->SetOverlayTransformAbsolute(cursor, vr::TrackingUniverseStanding, &mc);
+                overlay->ShowOverlay(marker);
+                overlay->ShowOverlay(cursor);
+            } else {
                 // On a panel: the non-interactive marker, pulled 5 mm toward the eye so it
                 // draws on top. In free space: the interactive dot the laser lands on.
                 const vr::VROverlayHandle_t show = onPanel ? marker : cursor, hide = onPanel ? cursor : marker;
                 const Vec3 at = onPanel ? point + Normalize(eye - point) * 0.005 : onScene ? point + dir * 0.05 : point;
                 const double dist = std::sqrt(Dot(at - eye, at - eye));
-                overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(cursorDeg * M_PI / 360)));
+                // Gaze mode: shown only while something moves it or a press holds it, and a pulse
+                // for each click (see the top); transparent otherwise, the laser still lands on it.
+                double scale = 1, alpha = 1;
+                if (gazeOn) {
+                    auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
+                    alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
+                    const double pulse = secs(pulseAt);
+                    if (pulse < 0.6) {
+                        scale = 1 + 1.5 * std::max(0.0, 1 - pulse / 0.3);
+                        alpha = std::max(alpha, std::clamp((0.6 - pulse) / 0.3, 0.0, 1.0));
+                    }
+                }
+                overlay->SetOverlayAlpha(show, float(alpha));
+                overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360)));
                 auto m = Billboard(at, eye);
                 overlay->SetOverlayTransformAbsolute(show, vr::TrackingUniverseStanding, &m);
                 overlay->ShowOverlay(show);
@@ -928,16 +1348,19 @@ int main() {
             const Vec3 aim = Rotate(R, RotateInverse(S, aimStanding));  // raw <- head <- standing
             // Origin partway along the line of sight to the cursor (smaller hit dot).
             const double toPoint = std::sqrt(Dot(point - eye, point - eye));
-            const double originDist = std::max(0.0, std::min(toPoint * originFraction, toPoint - originMargin));
+            double originDist = std::max(0.0, std::min(toPoint * originFraction, toPoint - originMargin));
+            if (onVrSettings) originDist = std::min(originDist, SETTINGS_ORIGIN);  // SteamVR finds the page
             const Vec3 originStanding = eye + Normalize(point - eye) * originDist;
             const Vec3 eyeRaw = Position(R) + Rotate(R, RotateInverse(S, originStanding - eye));
             lastPoint = point, lastOrigin = originStanding, lastAim = aimStanding;  // tilt starts from here
+            havePoint = true;
             if (debug && tnow - lastDebug > std::chrono::milliseconds(500)) {
                 lastDebug = tnow;
                 if (systemPointer == vr::k_ulOverlayHandleInvalid) overlay->FindOverlay("system.pointer", &systemPointer);
-                std::printf("dbg %s hit=%s dist=%.2f eye->point=%.2f origin=%.2f yaw=%.1f pitch=%.1f steamvr_dot=%d primary=%u\n",
+                std::printf("dbg %s hit=%s dist=%.2f eye->point=%.2f origin=%.2f vrsettings=%d yaw=%.1f pitch=%.1f gaze=%s steamvr_dot=%d primary=%u\n",
                             dragging ? "DRAG" : occluded ? "INFRONT" : onEdge ? "EDGE" : onScene ? "SCENE" : (best < 1e8 ? "PANEL" : "FREE"), lastHit.empty() ? "-" : lastHit.c_str(),
-                            distance, toPoint, originDist, yaw, pitch,
+                            distance, toPoint, originDist, onVrSettings, yaw, pitch,
+                            !gazeOn ? "off" : tnow - gz.at > std::chrono::milliseconds(150) ? "stale" : gazeOwns ? "owns" : "mouse",
                             systemPointer != vr::k_ulOverlayHandleInvalid && overlay->IsOverlayVisible(systemPointer),
                             overlay->GetPrimaryDashboardDevice());
                 std::fflush(stdout);
@@ -967,6 +1390,30 @@ int main() {
             }
         }
 
+        // A held-back press (see the top): held still long enough, it's a real press (a drag);
+        // released, it's a click where the pointer is now (this frame's pose has gone out).
+        if (!active) aimHeld = clickPress = false;  // released meanwhile: nothing to click
+        if (aimHeld && nudgeMoved < 0.2 && tnow - aimSince >= std::chrono::duration<double>(gazeHold)) {
+            aimHeld = false;
+            gazeBack = true;
+            pressLeft();
+        }
+        if (clickPress) {
+            clickPress = false;
+            pressLeft();
+            clickRelease = true;
+            clickReleaseAt = tnow + std::chrono::milliseconds(40);
+        } else if (clickRelease && tnow >= clickReleaseAt) {
+            clickRelease = false;
+            releaseLeft();
+        }
+
+        controllerButtons.Poll(
+            [&](const char *button, bool down) {
+                SendTo(out, "frametop_relay", std::string("vrbtn ") + button + (down ? " 1" : " 0"));
+            },
+            inGame);
+
         vr::VREvent_t ev;
         while (sys->PollNextEvent(&ev, sizeof ev)) {
             if (ev.eventType == vr::VREvent_Quit) {

pointer/helper/vrbuttons.h

@@ -0,0 +1,141 @@
+// Frame controller buttons for the relay's button mappings (SteamVR input, see ft-pointer.cpp).
+//
+// The Frame controllers aren't input devices on the host (no evdev node, no hidraw): only
+// SteamVR sees them. So the helper, an overlay client anyway, reads them with SteamVR input
+// and sends each press and release to the relay ("vrbtn <hand>/<button> 1|0" on
+// @frametop_relay), which does the mapped action, like for a mouse button.
+//
+// actions/ft_pointer_actions.json has one action set per button, each with one boolean action
+// bound to that button's click (actions/bindings_frame_controller.json). Only the sets of
+// mapped buttons are active: the relay says which ("vrbind <button>..."; "vrbind *" for all,
+// while the settings app captures a button; "vrbind -" for none), and asks again with
+// "vrhello" when the helper connects. While a game runs (a scene application), the mapped
+// buttons are left to it, unless the list starts with "+games"; capturing takes them anyway. The sets are active at an overlay-global priority, so
+// SteamVR delivers the buttons without the helper having input focus, and takes those
+// buttons (only those) from the game: a mapped button is Frametop's. That needs SteamVR's
+// "Enable global input from overlays (Experimental)" (steamvr/globalActionSetPriority);
+// "vrglobal on|off" sets it, "vrstatus" replies with the state as JSON.
+#pragma once
+
+#include <openvr.h>
+
+#include <cstdio>
+#include <cstring>
+#include <string>
+#include <vector>
+
+class ControllerButtons {
+  public:
+    static constexpr const char *kButtons[] = {
+        "left/view",   "left/dpad_up", "left/dpad_down", "left/dpad_left", "left/dpad_right", "left/bumper",
+        "left/trigger", "left/grip",   "left/thumbstick", "right/menu",    "right/a",         "right/b",
+        "right/x",     "right/y",      "right/bumper",   "right/trigger",  "right/grip",      "right/thumbstick"};
+    static constexpr int kCount = sizeof kButtons / sizeof kButtons[0];
+    // Above SteamVR's own overlay input, inside the overlay-global range.
+    static constexpr int32_t kPriority = vr::k_nActionSetOverlayGlobalPriorityMin + 0x100;
+
+    // After VR_Init. The manifest can only be set once per connection.
+    void Init(const std::string &manifest) {
+        input_ = vr::VRInput();
+        const vr::EVRInputError e = input_->SetActionManifestPath(manifest.c_str());
+        ok_ = e == vr::VRInputError_None;
+        std::printf("controller buttons: action manifest %s: %s\n", manifest.c_str(), ok_ ? "ok" : "error");
+        if (!ok_) std::printf("controller buttons: SetActionManifestPath error %d\n", int(e));
+        for (int i = 0; i < kCount; ++i) {
+            std::string name = kButtons[i];
+            name[name.find('/')] = '_';
+            input_->GetActionSetHandle(("/actions/" + name).c_str(), &set_[i]);
+            input_->GetActionHandle(("/actions/" + name + "/in/press").c_str(), &action_[i]);
+        }
+        std::fflush(stdout);
+    }
+
+    // "vrbind" arguments: [+games] then button names, "*" (all) or "-" (none). Returns what's
+    // bound now.
+    std::string Bind(const char *args) {
+        bool want[kCount] = {};
+        char word[64];
+        int used = 0;
+        games_ = capture_ = false;
+        while (std::sscanf(args, " %63s%n", word, &used) == 1) {
+            args += used;
+            if (std::strcmp(word, "+games") == 0) games_ = true;
+            if (std::strcmp(word, "*") == 0) capture_ = true;
+            for (int i = 0; i < kCount; ++i)
+                if (std::strcmp(word, "*") == 0 || std::strcmp(word, kButtons[i]) == 0) want[i] = true;
+        }
+        for (int i = 0; i < kCount; ++i) bound_[i] = want[i];
+        return Bound();
+    }
+
+    std::string Bound() const {
+        std::string s;
+        for (int i = 0; i < kCount; ++i)
+            if (bound_[i]) s += (s.empty() ? "" : " ") + std::string(kButtons[i]);
+        return (s.empty() ? "-" : s) + (games_ ? " (in games too)" : " (outside games)");
+    }
+
+    // Every frame. send(button, pressed) for each change; a button that's unbound (or can't
+    // be read any more) while down is released, so no action stays held. inGame: a scene
+    // application is running.
+    template <class Send> void Poll(Send send, bool inGame) {
+        if (!ok_) return;
+        inGame_ = inGame;
+        const bool take = !inGame || games_ || capture_;
+        std::vector<vr::VRActiveActionSet_t> sets;
+        for (int i = 0; i < kCount; ++i)
+            if (bound_[i] && take) {
+                vr::VRActiveActionSet_t s{};
+                s.ulActionSet = set_[i];
+                s.nPriority = kPriority;
+                sets.push_back(s);
+            }
+        const bool updated =
+            !sets.empty() && input_->UpdateActionState(sets.data(), sizeof sets[0], uint32_t(sets.size())) ==
+                                 vr::VRInputError_None;
+        for (int i = 0; i < kCount; ++i) {
+            bool down = false;
+            active_[i] = false;
+            if (bound_[i] && take && updated) {
+                vr::InputDigitalActionData_t d{};
+                if (input_->GetDigitalActionData(action_[i], &d, sizeof d, vr::k_ulInvalidInputValueHandle) ==
+                    vr::VRInputError_None) {
+                    active_[i] = d.bActive;
+                    down = d.bActive && d.bState;
+                }
+            }
+            if (down != down_[i]) {
+                down_[i] = down;
+                send(kButtons[i], down);
+            }
+        }
+    }
+
+    // {"manifest":true,"global":false,"bound":[...],"active":[...]}: active = bound and
+    // delivered (a controller that has the button is on, and SteamVR lets us have it).
+    std::string Status() const {
+        const bool global = vr::VRSettings()->GetBool("steamvr", "globalActionSetPriority", nullptr);
+        std::string bound, active;
+        for (int i = 0; i < kCount; ++i) {
+            if (bound_[i]) bound += std::string(bound.empty() ? "" : ",") + "\"" + kButtons[i] + "\"";
+            if (active_[i]) active += std::string(active.empty() ? "" : ",") + "\"" + kButtons[i] + "\"";
+        }
+        return std::string("{\"t\":\"vrstatus\",\"manifest\":") + (ok_ ? "true" : "false") +
+               ",\"global\":" + (global ? "true" : "false") + ",\"in_game\":" + (inGame_ ? "true" : "false") +
+               ",\"games\":" + (games_ ? "true" : "false") + ",\"bound\":[" + bound + "],\"active\":[" + active + "]}";
+    }
+
+    static void SetGlobal(bool on) {
+        vr::VRSettings()->SetBool("steamvr", "globalActionSetPriority", on);
+        std::printf("controller buttons: SteamVR global input from overlays %s\n", on ? "on" : "off");
+        std::fflush(stdout);
+    }
+
+  private:
+    vr::IVRInput *input_ = nullptr;
+    bool ok_ = false;
+    vr::VRActionSetHandle_t set_[kCount] = {};
+    vr::VRActionHandle_t action_[kCount] = {};
+    bool bound_[kCount] = {}, down_[kCount] = {}, active_[kCount] = {};
+    bool games_ = false, capture_ = false, inGame_ = false;
+};

screens/compositor.c

@@ -91,6 +91,13 @@ struct server {
     struct wl_event_source *tick;
     struct screen *pointer_focus;
     pid_t child;
+    // Where typing goes: the screens after a click on one, Steam after a click on another
+    // panel. The input relay grabs the keyboards while it's the screens (see keys_update).
+    bool keys_clicked;    // the last click was on a screen
+    bool keys_desktop;    // ...and the screens are showing: typing goes to the desktop
+    int relay_fd;         // unbound, so the relay can't reply into our control socket
+    uint32_t relay_sent;  // when the relay last heard from us (ms)
+    unsigned ticks;
 };
 
 static uint32_t now_ms(void) {
@@ -248,7 +255,10 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
                 wlr_seat_pointer_notify_button(s->seat, t, e->button,
                                                e->pressed ? WL_POINTER_BUTTON_STATE_PRESSED
                                                           : WL_POINTER_BUTTON_STATE_RELEASED);
-                if (e->pressed) wlr_seat_keyboard_notify_enter(s->seat, surface, NULL, 0, NULL);
+                if (e->pressed) {
+                    wlr_seat_keyboard_notify_enter(s->seat, surface, NULL, 0, NULL);
+                    s->keys_clicked = true;
+                }
             }
             break;
         case FT_SCROLL:
@@ -274,10 +284,30 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
     wlr_seat_pointer_notify_frame(s->seat);
 }
 
+// Tell the input relay where typing goes, on a change and every second: while it's the
+// desktop, the relay grabs pass-through keyboards so SteamVR (and the Steam app with
+// gamescope's focus) doesn't get the keys too. Without word from us for a few seconds,
+// the relay gives the keyboards back, so a closed desktop doesn't keep them.
+static void keys_update(struct server *s) {
+    const bool desktop = s->keys_clicked && ft_vr_screens_shown();
+    const uint32_t t = now_ms();
+    if (desktop == s->keys_desktop && t - s->relay_sent < 1000) return;
+    if (desktop != s->keys_desktop) wlr_log(WLR_INFO, "typing goes to %s", desktop ? "the desktop" : "Steam");
+    s->keys_desktop = desktop;
+    s->relay_sent = t;
+    struct sockaddr_un addr = {.sun_family = AF_UNIX};
+    const char name[] = "frametop_relay";
+    memcpy(addr.sun_path + 1, name, sizeof name - 1);
+    const char *msg = desktop ? "keyboard desktop" : "keyboard steam";
+    sendto(s->relay_fd, msg, strlen(msg), MSG_DONTWAIT, (struct sockaddr *)&addr,
+           offsetof(struct sockaddr_un, sun_path) + 1 + sizeof name - 1);
+}
+
 // Every ~11 ms (90 Hz): SteamVR events, and frame callbacks for screens that committed.
 static int tick(void *data) {
     struct server *s = data;
     ft_vr_poll(handle_vr_event, s);
+    if (++s->ticks % 9 == 0) keys_update(s);
     struct timespec now;
     clock_gettime(CLOCK_MONOTONIC, &now);
     for (int i = 0; i < MAX_SCREENS; ++i) {
@@ -303,13 +333,23 @@ static int child_exited(int sig, void *data) {
     return 0;
 }
 
+static bool key_held(const struct wlr_keyboard *kb, uint32_t code) {
+    for (size_t i = 0; i < kb->num_keycodes; i++)
+        if (kb->keycodes[i] == code) return true;
+    return false;
+}
+
 // Keys from the input relay (physical keyboards): "key <evdev code> <1 press|0 release>".
-// They go to the screen KWin has keyboard focus on (the last one clicked), but not while
+// They go to the screen KWin has keyboard focus on (the last one clicked), while typing
-// the SteamVR dashboard is open: typing belongs to Steam then.
+// goes to the desktop (keys_update). The release of a key the desktop got the press for
+// always goes through, or the key stays held there (a modifier held as typing moves to
+// Steam would otherwise modify every key typed after it).
 static void handle_key(struct server *s, uint32_t code, int value, char *reply, int size) {
     if (value == 2) return (void)snprintf(reply, size, "ok repeat ignored");  // KWin repeats itself
+    if (value || !key_held(&s->keyboard, code)) {
         if (!s->seat->keyboard_state.focused_surface) return (void)snprintf(reply, size, "ok no focus");
-    if (ft_vr_dashboard_visible()) return (void)snprintf(reply, size, "ok dashboard open");
+        if (!s->keys_desktop) return (void)snprintf(reply, size, "ok typing goes to Steam");
+    }
     struct wlr_keyboard_key_event ev = {
         .time_msec = now_ms(), .keycode = code, .update_state = true,
         .state = value ? WL_KEYBOARD_KEY_STATE_PRESSED : WL_KEYBOARD_KEY_STATE_RELEASED};
@@ -320,7 +360,9 @@ static void handle_key(struct server *s, uint32_t code, int value, char *reply,
 }
 
 // Control socket: abstract datagram @ft_screens. Here: "size <screen> <w> <h>" (a new
-// resolution, live) and "key <code> <value>"; the rest is in vr.cpp (ft_vr_command).
+// resolution, live), "key <code> <value>", and "click <overlay key>" (from the pointer
+// helper: a mouse click landed on that panel, "-" for none); the rest is in vr.cpp
+// (ft_vr_command).
 static int control_readable(int fd, uint32_t mask, void *data) {
     struct server *s = data;
     char buf[512], reply[2048];
@@ -345,6 +387,12 @@ static int control_readable(int fd, uint32_t mask, void *data) {
             handle_key(s, code, value, reply, sizeof reply);
             len = sizeof from;
             continue;  // no reply: keys are fire-and-forget
+        } else if (strncmp(buf, "click ", 6) == 0) {
+            // A click on another panel takes typing to Steam. Our own panels (the screens,
+            // their controls) leave it: a click on a screen arrives as a panel event.
+            if (strcmp(buf + 6, "-") != 0 && strncmp(buf + 6, "frametop.", 9) != 0) s->keys_clicked = false;
+            len = sizeof from;
+            continue;
         } else {
             ft_vr_command(buf, reply, sizeof reply);
         }
@@ -468,6 +516,7 @@ int main(int argc, char **argv) {
 
     const int control = open_control_socket();
     if (control < 0) return 1;
+    s.relay_fd = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0);
     wl_event_loop_add_fd(s.loop, control, WL_EVENT_READABLE, control_readable, &s);
 
     s.tick = wl_event_loop_add_timer(s.loop, tick, &s);

screens/vr.cpp

@@ -41,7 +41,12 @@
 
 #include <openvr.h>
 
+#include <fcntl.h>
 #include <linux/input-event-codes.h>
+#include <limits.h>
+#include <spawn.h>
+
+extern char **environ;  // for posix_spawn
 
 #include <algorithm>
 #include <chrono>
@@ -854,6 +859,33 @@ void EndDrag(Screen &s) {
     ApplyAlpha(s);
 }
 
+// KWin's outputs follow where the screens are, so the pointer and dragged windows cross
+// to the screen you see next to this one: `ft-layout scale` runs once a move has settled.
+long g_arrangeAt = -1;  // g_tick to run it at, -1 = not pending
+
+void ArrangeDesktopSoon() { g_arrangeAt = g_tick + 45; }  // about half a second
+
+void UpdateArrange() {
+    if (g_arrangeAt < 0 || g_tick < g_arrangeAt) return;
+    g_arrangeAt = -1;
+    char exe[PATH_MAX];
+    if (!realpath("/proc/self/exe", exe)) return;
+    std::string layout(exe);  // <repo>/screens/build/ft-screens -> <repo>/layout/ft-layout
+    for (int up = 0; up < 3 && layout.rfind('/') != std::string::npos; ++up) layout.resize(layout.rfind('/'));
+    layout += "/layout/ft-layout";
+    posix_spawn_file_actions_t io;
+    posix_spawn_file_actions_init(&io);
+    posix_spawn_file_actions_addopen(&io, 0, "/dev/null", O_RDONLY, 0);
+    posix_spawn_file_actions_addopen(&io, 1, "/tmp/frametop-layout.log", O_WRONLY | O_CREAT | O_APPEND, 0644);
+    posix_spawn_file_actions_adddup2(&io, 1, 2);
+    char scale[] = "scale";
+    char *argv[] = {layout.data(), scale, nullptr};
+    pid_t pid;  // reaped by the compositor's SIGCHLD handler
+    if (posix_spawn(&pid, layout.c_str(), &io, nullptr, argv, environ) != 0)
+        std::printf("can't run %s\n", layout.c_str());
+    posix_spawn_file_actions_destroy(&io);
+}
+
 // Let go: pin to the armed wrist, as the screen is now.
 void FinishDrag(Screen &s, int index) {
     const bool moved = s.drag == Drag::Move;
@@ -861,6 +893,7 @@ void FinishDrag(Screen &s, int index) {
     EndDrag(s);
     Mat c, p;
     if (!moved) return;
+    ArrangeDesktopSoon();
     if (target != kNone && DevicePose(target, &c) && ScreenPose(s, &p)) {
         Pin(s, target, Mul(Inverse(c), p));
         std::printf("screen %d: pinned to the %s controller\n", index + 1, HandName(target));
@@ -1032,7 +1065,7 @@ int ft_vr_modifiers(uint32_t format, uint64_t *out, int max) {
     return int(n < uint32_t(max) ? n : uint32_t(max));
 }
 
-bool ft_vr_dashboard_visible(void) { return vr::VROverlay()->IsDashboardVisible(); }
+bool ft_vr_screens_shown(void) { return ModeVisible(); }
 
 void ft_vr_screen_create(int index, double metres, int count) {
     Screen &s = g_screens[index];
@@ -1228,6 +1261,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
     }
     ++g_tick;
     UpdateGame();
+    UpdateArrange();
     UpdateVisibility();
     UpdateLasers();
     UpdateControls();

screens/vr.h

@@ -32,8 +32,8 @@ bool ft_vr_init(void);
 void ft_vr_shutdown(void);
 // Modifiers SteamVR can import for a DRM format. Returns the count (at most max).
 int ft_vr_modifiers(uint32_t format, uint64_t *out, int max);
-// The SteamVR dashboard is open (typing belongs to it then, not to the screens).
+// The screens are showing (by the visibility mode; not counting a wrist-pinned screen).
-bool ft_vr_dashboard_visible(void);
+bool ft_vr_screens_shown(void);
 // A panel for screen `index`, width in metres, placed in a row in front of the head.
 void ft_vr_screen_create(int index, double metres, int count);
 void ft_vr_screen_destroy(int index);

session/frametop.conf.example

@@ -18,5 +18,17 @@ POINTER_LASER_WIDTH=0.8    # controller beam width (dashboard.laserRayWidthScale
 POINTER_ORIGIN_MARGIN=0.15   # the laser starts at least this far (m) in front of its target, so small floating controls stay hittable
 POINTER_SCENE_RADIUS=0.5   # texture-less dashboard overlays (dock, window controls): hit radius (m) around their origin
 POINTER_EDGE_REACH=0.3     # just off a panel, the cursor stays on its plane this far (m), to reach its resize edges and window controls
+POINTER_FOLLOW=0           # 1 = the cursor follows your head on a leash (a mouse button mapped to Head follow on/off toggles it)
+POINTER_LEASH_DEG=10       # head follow: how far (degrees) your head can turn before the cursor comes along; 0 locks it to your view
+POINTER_LEASH_DELAY=0.2    # head follow: how long (s) your head has to stay past the leash, so a glance doesn't move the cursor
+POINTER_LEASH_RETURN=0.2   # head follow: once it comes along, how quickly (s) the cursor settles back to its place in your view
+POINTER_FOLLOW_REACH=70    # head follow: how far (degrees) from the middle of your view the mouse can move the cursor
+POINTER_GAZE=0             # 1 = the pointer goes where you look, the mouse does the last bit (needs the gaze service: gaze/run.sh install)
+POINTER_GAZE_RETAKE=5      # gaze mode: how far (degrees) you look away from the pointer before the gaze takes it back from the mouse
+POINTER_GAZE_NUDGE_MAX=8   # gaze mode: a mouse nudge up to this far (degrees) before a click is learned as the eye tracker's error
+POINTER_GAZE_HOLD=0.5      # gaze mode: a press held this long (s) without moving becomes a real press (to drag); moved or released sooner, it clicks where you let go
+POINTER_GAZE_SHOW=1        # gaze mode: the dot shows this long (s) after the mouse moves it; also while a press is held, and a pulse per click
+META_DASHBOARD=0   # 1 = a Meta tap on a pass-through keyboard toggles the SteamVR dashboard (pointer mode only)
+SHARE_KEYS=0       # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen)
 LAYOUT_GRAB_OFFSET=0.075   # arranging screens: where the floating window's grab bar is, in metres below the screen
 LAYOUT_SLIDE_SPEED=0.5     # arranging screens: how fast the invisible controller slides a grabbed screen (m/s)