Pointer: don't put a vanished panel back in the visibility map

The panel-edge test read visible[edgeKey], and when the last panel the cursor touched was
gone from the overlay list, that added it back as hidden. The map then had more entries than
there are handles, which made the 50 ms visibility poll run every frame, and since the last
commit it also counted as a visibility change each time, so unchanged frames were never
reused. The edge test now looks the key up without adding it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -13,8 +13,12 @@ Two settings apps come with it: Frametop Display Settings for the screens, profi
 
 Frametop is an independent project, not made by or affiliated with Valve.
 
+Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions, ideas, and help with your setup.
+
 ## Install on the headset
 
+> **Frametop doesn't work on the SteamOS beta right now.** On the beta (SteamOS 0.4.3), gaze mode can't read the eye tracker, and the desktop has started without its taskbar ([#15](https://github.com/DeeJanuz/frametop/issues/15)). Use the stable SteamOS release until this note is gone.
+
 You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space.
 
 1. In the launcher, choose Launch a program → Desktop.
@@ -27,7 +31,7 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
 
    It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/DeeJanuz/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
 
-   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you three things along the way: whether to install gaze mode (experimental, yes by default) and the Bluetooth fixes, then whether to restart SteamVR. The Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
+   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
 
 After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings.
 
@@ -54,8 +58,9 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
 | Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head |
 | Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) |
 | Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
+| Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own |
 | Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead |
-| Play a VR game | The screens hide and your controllers stay in the game. Open the SteamVR dashboard, or press Meta+Shift+H, to see and use them. To keep them visible over games, change During VR games on the Visibility & pins tab; the controllers still stay in the game, and you use the screens with the mouse or the dashboard |
+| Play a VR game | Frametop pauses so the game gets the headset to itself (see [Pause for VR games](#pause-for-vr-games)): the screens hide and your controllers stay in the game. Click both thumbsticks together twice to bring Frametop back. With the automatic pause off, the screens still hide, and the SteamVR dashboard or Meta+Shift+H shows them; to keep them visible over games, change During VR games on the Visibility & pins tab |
 
 Restarting the desktop (Restart desktop in Frametop Display Settings) closes its windows, but background work you started in it, such as servers, tmux sessions, or builds, keeps running.
 
@@ -90,9 +95,21 @@ A profile is a named setup: where the screens are, with their sizes and pins, wh
 | Pick a profile under Arrangement and press Open profile | Switches to it: the screens move, open windows of its apps go to their places, and the apps that aren't open start. Nothing closes |
 | Pick a profile under Start in profile, or run its entry (Frametop: NAME) from SteamVR's Launch a program list | The desktop starts in that profile, or switches to it if it's running. A profile can also go on a key combination, mouse button, or controller button in Frametop Input Settings |
 
+### Pause for VR games
+
+Frametop pauses while a VR game runs, so the game gets the headset's CPU and GPU, and comes back a few seconds after the game ends. Paused, the screens hide and the desktop nearly stops drawing, but its windows stay open. Gaze mode's eye tracking stops, and so do remote desktop and hand tracking if they run. The mouse works as a plain mouse in SteamVR.
+
+| Do this | To get this |
+| --- | --- |
+| Click both thumbsticks together, twice | Pauses Frametop, or brings it back, in a game or not. You hear a short sound. The game sees the clicks too |
+| Start a VR game | Frametop pauses, and comes back 5 seconds after the game ends. Bring it back during the game, and it stays on until that game ends |
+| Map Pause/resume Frametop to a mouse button, key combination, or controller button | The same, from that button (Frametop Input Settings) |
+
+The Game optimization page of Frametop Input Settings turns the automatic pause off, changes the gesture, closes the desktop instead of hiding it (more for the game, but its windows close), and turns the sound off. From a terminal: `input/ft-pause on`, `off`, or `status`.
+
 ### Gaze mode (experimental)
 
-In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later). Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
+In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later), and then our own eye tracker for it, which is more accurate than SteamVR's (or run `gaze/tracker/install.sh` later; it needs `sudo`). Gaze mode uses ours once it's installed, and SteamVR's until then. Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
 
 | Do this | To get this |
 | --- | --- |
@@ -119,8 +136,8 @@ This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 2026092
 
 - A SteamOS or SteamVR update can break parts of it until Frametop catches up. After an update, run `cd ~/frametop && scripts/doctor.sh` in a terminal. It checks what Frametop needs from SteamOS, and says what changed since the versions you last marked as working and what to try. Once everything works, `scripts/doctor.sh --mark-good` records the versions. If something stops working, please report it.
 - The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes.
-- During a VR game you can't show the screens with a controller button, because the game owns the buttons. Open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead.
-- Flatscreen games aren't detected as games. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab).
+- During a VR game, mapped controller buttons belong to the game, so they can't bring the screens up. The pause gesture still works: Frametop reads it without taking the thumbsticks from the game. With the automatic pause off, open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead.
+- Flatscreen games aren't detected as games, so they don't pause Frametop by themselves: click both thumbsticks twice to pause it. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab).
 - Typing follows your last click. A controller click on a panel other than the screens (the dashboard, a Steam app) doesn't move typing there; click it with the mouse, or click a screen to bring typing back.
 - The screens don't draw a mouse cursor of their own. The 3D mouse's dot or SteamVR's laser shows where you're pointing.
 - Profiles reopen apps, not what the apps had open. Tabs, files, and folders are left to each app's own restore.
@@ -138,7 +155,7 @@ In a terminal on the headset, run:
 cd ~/frametop && scripts/report.sh
 ```
 
-This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file.
+This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
 
 ## Update
 

display-settings/main.qml

@@ -802,7 +802,7 @@ Kirigami.ApplicationWindow {
                     Repeater {
                         model: [
                             { value: "hide", text: "Hide them unless the SteamVR dashboard is open", help: "The game has the view to itself; open the dashboard (or press Meta+Shift+H) to see the screens." },
-                            { value: "visible", text: "Keep them visible over the game", help: "They float over the game as they are outside it." }
+                            { value: "visible", text: "Keep them visible over the game", help: "They float over the game as they are outside it. Turn off Pause while a VR game runs in Frametop Input Settings (Game optimization), or Frametop pauses and hides them anyway." }
                         ]
                         delegate: ColumnLayout {
                             required property var modelData

display-settings/steam_settings.py

@@ -4,117 +4,26 @@ Steam, not systemd, puts the Frame to sleep: after "When Plugged In and Idle ->
 (an hour by default) without input, even while it charges. That's a Steam client setting,
 `system_idle_suspend_ac_sec` (0 = never), with no file or command line to change it. Steam
 on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context
-(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080. There
-`settingsStore.clientSettings` has the current values, and `SteamClient.Settings.SetSetting`
-takes a change as a serialized CMsgClientSettings protobuf, which is what Steam's own
-Settings -> Power page sends. Standard library only (a minimal WebSocket client).
+(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080
+(steam/steamui.py). There `settingsStore.clientSettings` has the current values, and
+`SteamClient.Settings.SetSetting` takes a change as a serialized CMsgClientSettings protobuf,
+which is what Steam's own Settings -> Power page sends.
 """
-import base64
-import json
 import os
-import socket
-import struct
-import urllib.request
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "steam"))
+from steamui import SteamUnreachable, evaluate  # noqa: E402,F401  (callers catch SteamUnreachable here)
 
-CDP_PORT = 8080
 # CMsgClientSettings field numbers (Steam's UI bundle maps the names to these).
 FIELDS = {"system_idle_suspend_ac_sec": 24004, "system_idle_suspend_battery_sec": 24003}
 
 
-class SteamUnreachable(Exception):
-    pass
-
-
-class _WebSocket:
-    def __init__(self, url, timeout=5):
-        host_port, path = url[len("ws://"):].split("/", 1)
-        host, port = host_port.rsplit(":", 1)
-        self.sock = socket.create_connection((host, int(port)), timeout=timeout)
-        key = base64.b64encode(os.urandom(16)).decode()
-        self.sock.sendall((f"GET /{path} HTTP/1.1\r\nHost: {host_port}\r\nUpgrade: websocket\r\n"
-                           f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode())
-        head = b""
-        while b"\r\n\r\n" not in head:
-            chunk = self.sock.recv(4096)
-            if not chunk:
-                raise SteamUnreachable("Steam closed the connection")
-            head += chunk
-        if b" 101 " not in head.split(b"\r\n", 1)[0]:
-            raise SteamUnreachable(head.split(b"\r\n", 1)[0].decode(errors="replace"))
-        self.buf = head.split(b"\r\n\r\n", 1)[1]
-
-    def send(self, text):
-        data, mask = text.encode(), os.urandom(4)
-        n = len(data)
-        if n < 126:
-            head = struct.pack(">BB", 0x81, 0x80 | n)
-        elif n < 65536:
-            head = struct.pack(">BBH", 0x81, 0x80 | 126, n)
-        else:
-            head = struct.pack(">BBQ", 0x81, 0x80 | 127, n)
-        self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(data)))
-
-    def _take(self, n):
-        while len(self.buf) < n:
-            chunk = self.sock.recv(65536)
-            if not chunk:
-                raise SteamUnreachable("Steam closed the connection")
-            self.buf += chunk
-        out, self.buf = self.buf[:n], self.buf[n:]
-        return out
-
-    def recv(self):
-        message = b""
-        while True:
-            b0, b1 = self._take(2)
-            n = b1 & 0x7F
-            if n == 126:
-                n = struct.unpack(">H", self._take(2))[0]
-            elif n == 127:
-                n = struct.unpack(">Q", self._take(8))[0]
-            message += self._take(n)
-            if b0 & 0x80:
-                return message.decode(errors="replace")
-
-    def close(self):
-        self.sock.close()
-
-
-def _evaluate(expression):
-    """Runs JavaScript in Steam's SharedJSContext and returns its (awaited) value."""
-    try:
-        with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}/json", timeout=3) as r:
-            targets = json.load(r)
-    except OSError as e:
-        raise SteamUnreachable(f"Steam isn't reachable on port {CDP_PORT} ({e})") from e
-    url = next((t["webSocketDebuggerUrl"] for t in targets if t.get("title") == "SharedJSContext"), None)
-    if not url:
-        raise SteamUnreachable("Steam's UI isn't running")
-    try:
-        ws = _WebSocket(url)
-        try:
-            ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
-                                "params": {"expression": expression, "awaitPromise": True, "returnByValue": True}}))
-            while True:
-                reply = json.loads(ws.recv())
-                if reply.get("id") == 1:
-                    break
-        finally:
-            ws.close()
-    except OSError as e:
-        raise SteamUnreachable(str(e)) from e
-    result = reply.get("result", {})
-    if "exceptionDetails" in result:
-        details = result["exceptionDetails"]
-        raise SteamUnreachable(details.get("exception", {}).get("description") or details.get("text", "error"))
-    return result.get("result", {}).get("value")
-
-
 def sleep_settings():
     """{"ac": seconds, "battery": seconds}: when Steam puts the Frame to sleep without input,
     plugged in and on battery (0 = never)."""
-    value = _evaluate("(() => { const c = settingsStore.clientSettings; "
-                      "return {ac: c.system_idle_suspend_ac_sec, battery: c.system_idle_suspend_battery_sec}; })()")
+    value = evaluate("(() => { const c = settingsStore.clientSettings; "
+                     "return {ac: c.system_idle_suspend_ac_sec, battery: c.system_idle_suspend_battery_sec}; })()")
     if not isinstance(value, dict) or not all(isinstance(value.get(k), int) for k in ("ac", "battery")):
         raise SteamUnreachable("Steam's settings don't have the sleep timeouts")
     return value
@@ -125,7 +34,7 @@ def set_sleep_setting(name, seconds):
     field, seconds = FIELDS[name], int(seconds)
     if seconds < 0:
         raise ValueError("seconds must be 0 (never) or more")
-    ok = _evaluate(f"""(async () => {{
+    ok = evaluate(f"""(async () => {{
         const bytes = [];
         const varint = n => {{ while (n > 127) {{ bytes.push((n & 127) | 128); n = Math.floor(n / 128); }} bytes.push(n); }};
         varint({field} * 8); varint({seconds});

docs/controller-click-stability.md

@@ -0,0 +1,34 @@
+# Controller desktop click stability
+
+Trigger presses reach KDE immediately, but controller motion within 8 logical
+pixels of the press stays at that position until release. Releasing without a motion outside this
+zone delivers the click at the original position, even if the hand moved during
+release. Moving outside the zone begins a normal drag immediately; returning to
+the zone does not turn it back into a click. There is no hold-duration timer.
+
+This filters overlay pointer content events on desktop monitors only, and only
+presses from hand controllers start it. The 3D mouse (whose laser comes from the
+`ft_pointer` virtual controller), SteamVR UI, separate screen grab bars and
+floating-app title-bar carrying are unaffected. Multi-button gestures keep their
+existing behavior. A motion onto another desktop monitor starts a drag;
+cross-monitor motion is not stabilized.
+
+CLI (runtime preferences, reset to 8 on desktop restart):
+
+```sh
+input/ft-clickctl status
+input/ft-clickctl threshold 8
+input/ft-clickctl threshold 0  # disable without a restart
+```
+
+Thresholds are 0–64 logical pixels, normalized to each panel's KDE scale.
+Status reports held state, suppressed motions, stabilized clicks and drags.
+Changing the threshold while a controller button is held is refused.
+
+This is a separate contribution from desktop mouse/controller ownership. Its
+hardware validation must check small controls, intentional text selection,
+long presses, cross-monitor dragging and simultaneous mouse use. The existing
+renderer laser remains tracked; this change stabilizes desktop input rather
+than smoothing the visual laser. Default threshold is a starting point to test.
+
+Run `scripts/test-controller-click.sh` for the isolated gesture-state tests.

docs/design.md

@@ -116,9 +116,9 @@ The driver starts disconnected, because holding the right-hand role while SteamV
 
 ### The cursor
 
-Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either.
+Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either. Both tests ask SteamVR about every visible overlay, so a frame where nothing moved (the mouse, the anchor, the eye by more than 5 mm, which overlays show) reuses the last result, for up to 100 ms, since overlays can also move on their own. The dots' overlay settings go to SteamVR only when they change, and the pose goes to the driver, which keeps the last one, only when the laser would land 0.1 mm or more elsewhere, and at least every 100 ms.
 
-OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately.
+OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately. Each run is a shell and a new SteamVR client, about 30 ms of CPU, and it ran every second while the pointer was awake, which in gaze mode is all the time. Now it runs every 20 seconds, and at once when the pointer wakes, when the dashboard opens or closes, when a game starts or ends, and when a left click hits nothing (a panel that came up since). An overlay already on the list showing or hiding needs no new list: the helper checks the visibility of the ones it knows every 50 ms.
 
 The laser starts partway along your line of sight to the cursor rather than at your eye. SteamVR sizes its hit dot by distance from the laser's origin, and a laser from the eye still shows a beam in each eye. Starting it close to the target makes the beam and the dot tiny, while `POINTER_ORIGIN_MARGIN` keeps the origin in front of the small window controls, which float a few centimetres in front of their panels. The helper's own white dot is the visible cursor. In empty space it's an interactive overlay that the laser lands on, so SteamVR never draws a laser into nothing.
 
@@ -141,7 +141,7 @@ Replacing a loaded driver's files, as re-running the installer used to do, leave
 
 The dashboard follows whichever device summoned it or last pressed its trigger. Frametop adds "last used wins": moving a real controller releases the pointer, and the next mouse movement takes the laser back. Moving means faster than 0.35 m/s or 2 rad/s (both times `POINTER_CONTROLLER_PICKUP`, 1 by default) for 100 ms in a row, while the controller is tracked normally. A single sample over the limit used to be enough, and controllers resting on a desk took the laser back on a knock or a tracking jump while the mouse was in use. Small movements don't count; waking needs `POINTER_WAKE_COUNTS` of mouse motion within a second, so desk jitter doesn't steal the laser. While the pointer is awake, a tiny transparent overlay with `MakeOverlaysInteractiveIfVisible` keeps SteamVR's laser mouse on, since otherwise the first click would only switch the laser on.
 
-When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off.
+When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off. With the pointer off it also stops running its loop every 8 ms, about 116 wakeups a second for nothing: it waits up to 250 ms for a command on its socket (20 ms while it reads mapped controller buttons, which SteamVR input only offers by polling), and leaves the overlay lookups until the pointer wakes.
 
 ### Moving floating windows
 
@@ -153,6 +153,8 @@ SteamVR opens every input device only when it starts. When a Bluetooth mouse sle
 
 Keyboards aren't grabbed by default, because a grabbed keyboard's keys went into a virtual keyboard nothing typed from; the relay forwards them to ft-screens instead.
 
+The relay never waits on the pointer helper. Its socket to the helper used to block, so when the helper stalled (a layout placement or `grabprobe` holds it for seconds, and the gaze service fills its socket 90 times a second meanwhile), the whole relay stopped with it: keyboards, the volume keys, and pausing. Now what the helper doesn't take waits in order and goes out on the next loops. Mouse moves add up into one while they wait, and a scroll notch is dropped, since scrolling seconds late is no use; presses and releases are kept, so no button stays down. Mouse motion goes to the helper at most every 4 ms, rather than once per report, which from a 1000 Hz mouse was 1000 datagrams a second to a helper that runs every 8 ms; a button sends the motion before it first, so the click lands where the pointer was.
+
 An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's `InputStealer`, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's `OverlayFocusChanged` (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. So with `SHARE_KEYS=1` the relay sends a grabbed keyboard's keys to `@frametop_keys` as datagrams (`key <code> <value> <device name>`). It's off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first gets every key typed into the desktop, passwords included. A listener should accept only its own user (`SO_PASSCRED`) and skip any keyboard of its own that the relay grabs too.
 
 Volume keys must never reach gamescope. With the openvr backend, gamescope sends volume up and down to Steam by moving keyboard focus to Steam for the key and then back to the previously focused surface. When nothing had focus, the one it moves back to is null, and wlroots aborts on a null focus surface (`wlr_seat_keyboard_notify_enter: Assertion 'surface' failed`), which ends the whole VR session. Keyboard focus is often empty while you work in VR, so one press of the headset's volume button could take everything down. gamescope reads the headset's buttons and every keyboard itself (`InputStealer`), as do SteamVR's processes, so the relay has to stop volume keys at the device. Grabbing `gpio-keys` would also take the headset's click button, so the relay remaps the volume entries in each device's keymap (`EVIOCSKEYCODE`) and handles the stand-in codes itself. That fix covers every device at once, including keyboards that aren't grabbed.
@@ -175,7 +177,7 @@ Flatpak apps need `XDG_DATA_DIRS` to include Flatpak's exports, or Plasma opens
 
 The private runtime directory also moves the session's document portal to `$XDG_RUNTIME_DIR/frametop/doc`, and that broke saving and uploading in Flatpak apps. The file picker (xdg-desktop-portal 1.18.4 on SteamOS) gives a sandboxed app the host path of the file it picked, `/run/user/1000/frametop/doc/ID/NAME`. Inside the sandbox the portal is at `/run/flatpak/doc`, and `/run/user/1000` is a private per-app folder (`.flatpak/APP/xdg-run` in the runtime directory). So Brave created the missing folder there, "finished" the download into it, and the file vanished when the session cleaned up. The session script now links that path to `/run/flatpak/doc` in each installed app's folder before Plasma starts. Upstream xdg-desktop-portal fixed this after 1.22.1 (commit `69ba5e1`) by handing Flatpak apps `/run/flatpak/doc` paths, after which the links go unused.
 
-A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it.
+A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. It then waits for distrobox-init to log `container_setup_done`, as `distrobox enter` does only for containers it starts itself. A new container's first start takes a minute or more (it installs distrobox's dependencies and sets up passwordless sudo), and an install that entered right away met a sudo password prompt with no terminal to answer it ([#9](https://github.com/DeeJanuz/frametop/issues/9)).
 
 Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`.
 
@@ -189,6 +191,15 @@ Movement is judged within 10-second windows. On the mount, the head pose jittere
 
 Staying awake while charging uses Steam's own setting rather than a logind sleep inhibitor. Steam suspends with `dbus-send ... login1.Manager.Suspend boolean:true`, and a block inhibitor does stop that (`CanSuspend` answers "challenge" while one is held), but it stops the power button too. `system_idle_suspend_ac_sec` is field 24004 of Steam's CMsgClientSettings. In Steam's SharedJSContext, reachable over CDP on port 8080 because Steam runs with `-cef-enable-debugging`, `SteamClient.Settings.SetSetting` takes a change as a base64 protobuf, the way Steam's Power page sends it (0 is never), and `settingsStore.clientSettings` has the current values.
 
+## Pausing for VR games
+
+Hiding the screens during a game kept them out of view, but Frametop kept using the headset. Measured on 2026-10-02 with gaze mode off and no game running, in shares of one core: our eye tracker (ft-eyes) about 60%, ft-eyegrab, ft-gaze and ft-gazed about 3 to 4% each; remote desktop (krdpserver, FreeRDP, Xvnc) about 2 cores while it ran; KWin about 13%, ft-screens about 4%. The gaze service ran at full rate whether gaze mode was on or not; now it idles while the gaze isn't used (gaze/README.md), and pausing stops it outright. Reading SteamVR's eye tracking 90 times a second also made it restart every 10 to 13 seconds during Beat Saber, and each restart took input focus from the game, which paused it (PR #13; since then ft-gaze skips SteamVR's gaze action during games, but our own tracker kept running). So pausing stops what costs the most and leaves windows where they are.
+
+- A hidden screen still cost as much as a visible one. ft-screens sent every committed screen its frame callback at 90 Hz whether its overlay showed or not, so KWin kept drawing, and its apps with it. Paused, ft-screens sends the callbacks once a second. A Wayland client draws again only after its last frame's callback, so KWin's output stalls, KWin's own clients stop getting theirs, and the whole desktop idles, without anything losing its connection. A second's pace, rather than none, keeps any client that waits on a callback from waiting forever. Stopping KWin or the apps with SIGSTOP would free the same, but a Wayland peer that stops reading overflows the other side's 4 KB socket buffer, which ends the connection: that's how the live desktop died once when its KWin stalled (`Data too big for buffer`). They also sit in different cgroups (KWin under steam.service when the VR launcher starts it, ft-screens in the dev container's), so no single freeze stops them together.
+- The relay does the pausing because it's the one part that always runs, and the pointer helper keeps running because stopping it leaves its virtual controller connected with its last pose (the driver has no staleness timeout), maybe holding a hand role, with the 3D mouse dead. Releasing it does the job. The helper already checks for a scene app twice a second, so it's what tells the relay a game started.
+- The gesture has to work during a game, but SteamVR input reaches only the app with input focus, and an overlay with global input (`steamvr/globalActionSetPriority`) takes the buttons it binds from the game. vrserver's web socket on 127.0.0.1:27062, which its controller binding page uses for the live view, reports every controller component whatever has focus, and reading it takes nothing. The game sees the clicks too, so the default is a gesture games hardly use: both thumbsticks, together, twice. "Together" means within 0.3 seconds of each other, so a stick held down to sprint while the other clicks doesn't count. The stream is about 160 messages a second, nearly all capacitive sensing, so the reader parses only the few that mention a gesture's button. A controller's root path changes while the 3D mouse holds its hand role (`/devices/cv/<serial>` instead of `/user/hand/right`), so the reader looks the controllers up again (an HTTP request to vrserver): when the relay's 3D mouse connects or lets go, when a message comes from a device it doesn't know, and every 30 seconds. It used to be every 3 seconds.
+- Resuming starts remote desktop through `systemd-run --scope`: started straight from the relay, it would join the relay's cgroup and end with the next relay restart.
+
 ## SteamOS updates
 
 On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-steamvr-rel` package), next to KWin, gamescope, and the kernel, so every SteamOS update can bring a new SteamVR too. Frametop survives updates: it lives in the home folder and the `dev` container, the Bluetooth fixes are in `/etc`, which SteamOS keeps across updates, and nothing goes into `/usr`. What an update can break is what Frametop uses from the image. The public OpenVR API is versioned and stays put. The rest is less certain: `IVRIPCResourceManagerClient`, which is newer than the header SteamVR ships; the text `vrcmd --overlays` prints; the eye tracker's shared memory layout; XRService's camera buffers; KWin's nested backend; and behavior Frametop works around, such as the SteamVR Settings page that `ComputeOverlayIntersection` can't find or the scale KWin's nested backend doesn't undo.

docs/profiles.md

@@ -46,8 +46,8 @@ So a "Work" profile can put three screens around you with a browser, two termina
 
 ## How it works
 
-- **Capture** (`ft-layout save NAME`, and Save as profile… in Display Settings). ft-layout captures the screens as before, then asks ft-floatd for the windows (`windows` on @frametop_float). ft-floatd has the KWin script report every window as it is now (`report-all`), then answers with every normal window: its desktop file name, the screen it's on, its rectangle there, and whether it's maximized. For floating windows it gives their panel's place, their size in pixels, and their scale. Windows with no desktop file name are kept by their process's command line (`/proc/<pid>/cmdline`) and window class. Windows of Plasma itself, the Frametop settings apps, and dialogs aren't recorded. If ft-floatd doesn't answer, the profile keeps the apps it had.
-- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name: single-instance and D-Bus-activated apps open their windows from a process that was already running.
+- **Capture** (`ft-layout save NAME`, and Save as profile… in Display Settings). ft-layout captures the screens as before, then asks ft-floatd for the windows (`windows` on @frametop_float). ft-floatd has the KWin script report every window as it is now (`report-all`), then answers with every normal window: its desktop file name, the screen it's on, its rectangle there, and whether it's maximized. For floating windows it gives their panel's place, their size in pixels, and their scale. A window whose app id has no desktop file (a Flatpak app's X11 window can give its own: RustDesk's says `com.carriez.flutter_hbb`, its desktop file is `com.rustdesk.RustDesk`) is kept by the desktop file whose `StartupWMClass` names its window class. Windows with no desktop file name are kept by their process's command line (`/proc/<pid>/cmdline`) and window class. Windows of Plasma itself, the Frametop settings apps, and dialogs aren't recorded. If ft-floatd doesn't answer, the profile keeps the apps it had.
+- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name (found the same way as at capture): single-instance and D-Bus-activated apps open their windows from a process that was already running.
 - **Default at start.** The session script runs `ft-layout start --wait 90`. That opens the profile in `FT_PROFILE` or `default_profile` (screens, then the apps once ft-floatd is up), or runs `apply --wait` if there's none. Start in profile on the Layout & profiles page sets `default_profile` (`ft-layout default NAME|none`). Plasma's session restore is turned off in the session (`ksmserverrc`: `loginMode=emptySession`).
 - **Launcher entries.** Each profile gets `~/.local/share/applications/frametop-profile-<name>.desktop` ("Frametop: Work"), written when it's saved and removed when it's deleted. They show in SteamVR's Launch a program list, the Application Launcher, and KRunner. Running one (`ft-layout open NAME`) switches to that profile if the desktop runs. Otherwise it starts the desktop with `FT_PROFILE` set (`systemd-run`, as `desktops.sh start` does), which overrides `default_profile` for that start. That needs SteamVR to be running.
 - **The action.** `profile:NAME` in the input relay (it runs `ft-layout use NAME`) for key combinations, mouse buttons, and controller buttons, with or without pointer mode. Input Settings lists one "Open profile NAME" action per profile.

docs/reference.md

@@ -66,12 +66,12 @@ place N x y z yaw pitch roll     width N metres          curve N radius|on|off
 pin N|all left|right|head [matrix]    unpin N|all        size N w h
 get N    screens    head    state    key code value    scale N s    vrkeyboard show|hide|toggle|close
 visibility always|dashboard|gesture|toggle    wrist degrees    gesture left|right degrees
-hide | show | toggle    controllers always|outside_games|dashboard    ingames hide|visible
+hide | show | toggle    controllers always|outside_games|dashboard    ingames hide|visible    pause on|off|state
 conceal N|all    reveal N|all    concealed    cutouts on|off|state    cutouts predict on|off    cutouts lead ms
 float N mpp x y w h title    unfloat N    pose N matrix    sub N k x y w h | sub N k off    minimized N 0|1    carry N
 ```
 
-`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar.
+`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `pause on` (from the input relay, when Frametop pauses for a VR game) hides every screen and floating window whatever else says, and slows the desktop down; `pause off` undoes it. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar.
 
 ## Input relay
 
@@ -85,6 +85,8 @@ The relay also owns the volume keys, on every device that has them, the headset'
 desktops.sh relay install     # enable it (starts with the next reboot or SteamVR start)
 desktops.sh relay status | log | uninstall
 input/input-relay.py --no-grab   # try it without taking devices from SteamVR
+input/test/keys-test.py          # key combinations and modifier taps, against fake devices (safe next to the live relay)
+steam/ft-steam menu              # what Open Steam menu does; ft-steam check: Steam's UI still has the calls
 ```
 
 The first time, the relay has to start before SteamVR, so reboot or restart SteamVR after installing it. After that it's safe to restart on its own: systemd keeps the virtual devices open in its file descriptor store (`FileDescriptorStorePreserve=yes`), so SteamVR keeps the same devices.
@@ -99,7 +101,7 @@ A mouse drives SteamVR the way a controller's laser does, but shows up as a smal
 
 Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again.
 
-To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button (or a Meta tap, with `META_DASHBOARD=1`) wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant.
+To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant. Open Steam menu / close dashboard (`steam_menu`) needs no pointer: `steam/ft-steam menu` asks Steam's UI, over its debugging port (`steam/steamui.py`), to show its dashboard overlay and focus the Steam frame's menu, or to hide the dashboard if it's up.
 
 ```
 pointer/driver/build.sh && pointer/driver/install.sh install   # then restart SteamVR
@@ -108,16 +110,17 @@ pointer/helper/run.sh status | log | restart
 pointer/driver/install.sh probe     # devices, hand roles, who owns the dashboard pointer
 ```
 
-The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
+The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
 
 ## Frametop Input Settings
 
-A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has eight pages:
+A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has nine pages:
 
-- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, where a Meta tap toggles the dashboard if `META_DASHBOARD=1` is in `~/.config/frametop.conf`), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
-- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
+- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
+- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, pause or resume Frametop ([Pausing for VR games](#pausing-for-vr-games)), Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
 - Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
-- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, on any keyboard, to any action but passing a key through or nothing. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. The combination's last key isn't typed, and the modifiers still reach the app. They're saved as `key_bindings` in the same file.
+- Game optimization has the pause for VR games: its state with Pause now or Resume, whether VR games pause Frametop by themselves, the controller gesture (one or two buttons, pressed once or twice; one button always takes two presses), what happens to the desktop, and the sound. They're saved as `pause_auto`, `pause_gesture`, `pause_desktop`, and `pause_sound` in `~/.config/frametop-input.json`. See [Pausing for VR games](#pausing-for-vr-games).
+- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
 - Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
 - Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
 - Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
@@ -197,14 +200,41 @@ power/run.sh off | on      # the displays off now, or back on
 power/run.sh log
 ```
 
+## Pausing for VR games
+
+Paused, Frametop leaves the headset's CPU and GPU to a VR game. The input relay does it (`input/game_pause.py`), since it's the one part that always runs:
+
+- The gaze service stops (`frametop-gaze`: ft-gazed, ft-gaze, our own eye tracker, the gaze panel), so nothing reads SteamVR's eye tracking. Our frame grabber, the root service `ft-eyegrab`, goes idle by itself 3 seconds after our eye tracker stops asking it for frames.
+- Hand tracking stops if it runs (`frametop-camd`, `frametop-hands`).
+- The desktop, as the Game optimization page of Frametop Input Settings says (`pause_desktop`): hidden (the default) or closed. Hidden, ft-screens hides every screen and floating window whatever the visibility mode, the hotkey, or the dashboard says, and gives KWin a frame callback once a second instead of 90 times. KWin draws a screen only after its frame callback, and its apps wait for theirs, so the desktop hardly draws, but its windows stay open. Remote desktop stops if it runs (`session/remote-ctl.sh`). Closed, `desktops.sh stop` closes the desktop and its windows, and resuming starts it again (about 12 seconds), in its start profile if it has one.
+- The relay lets go of the 3D mouse and feeds pointer devices to its virtual mouse and keyboard, as with `POINTER=0`. Typing goes to Steam. Mapped buttons and key combinations do nothing but pausing, the Steam menu, and commands; a key combination that does nothing is typed as usual.
+
+Resuming starts again only what pausing stopped, and plays a second sound. The pointer helper and ft-powerd keep running: they cost little, the helper is what says a game started, and stopping it would leave its virtual controller connected with its last pose.
+
+Ways to pause and resume:
+
+- The controller gesture, by default both thumbsticks clicked together twice: both go down within 0.3 seconds of each other, and the second time within 0.7 seconds of the first. The relay reads it from vrserver's web socket (`input/vrws.py`), which works whatever has input focus and takes nothing from the game, so the game sees the clicks too. The Game optimization page changes it: one or two of the buttons the Controllers page lists, pressed once or twice (one button always takes two), or none.
+- The Pause/resume Frametop action, on a mouse button, a key combination, or a controller button (outside games, like every mapped controller button).
+- VR games, with Pause while a VR game runs on (`pause_auto`, the default). The pointer helper tells the relay when a scene app starts and ends (`vrgame 1|0`, repeated every 5 seconds). A game starting pauses Frametop. A pause that starts while a game runs ends 5 seconds after the game does, unless another game starts first. Resumed during a game, Frametop stays on until that game ends. A pause that starts outside a game lasts until you resume. Flatscreen games aren't scene apps, so they don't pause it.
+- From a terminal or a script:
+
+```
+input/ft-pause on | off | toggle   # pause or resume
+input/ft-pause status              # the state as JSON (the relay's "pause ?")
+input/vrws.py 10                   # the controllers' buttons from vrserver's web socket, for 10 s
+input/test/pause-test.py           # the gesture and the automatic pause, offline
+```
+
+The state outlives a relay restart, in `/run/user/UID/frametop-pause.json`. A SteamVR restart while paused starts the gaze service with it, and the relay stops it again when the pointer helper comes back.
+
 ## Gaze pointer (experimental)
 
-In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off.
+In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. The service idles while the gaze isn't used: its eye tracker reader and our own eye tracker run only while gaze mode is on and someone wears the headset, while a check or the calibration runs, or while the Gaze page of Frametop Input Settings is open, and stop 30 seconds after ([gaze/README.md](../gaze/README.md)). Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off.
 
 - Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held.
 - The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual.
 - A correction before a click teaches the gaze service the tracker's error there. A correction bigger than `POINTER_GAZE_NUDGE_MAX` (55 degrees) isn't learned; it opens a quick check instead.
-- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode coming on without a calibration opens Calibrate by itself.
+- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it.
 
 [gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool.
 

float/ft_floatd.py

@@ -229,6 +229,24 @@ def cmdline(pid):
         return []
 
 
+def desktop_name(app, cls):
+    """The desktop file name for a window's app id and class. A Flatpak app's window can give an
+    id with no desktop file: RustDesk's (an X11 window) says com.carriez.flutter_hbb, and its
+    desktop file is com.rustdesk.RustDesk. That file's StartupWMClass names the window's class
+    then. The app id itself when nothing matches."""
+    try:
+        if not app or Gio.DesktopAppInfo.new(app + ".desktop"):
+            return app
+    except TypeError:  # PyGObject raises for the NULL a missing desktop file returns
+        pass
+    names = {n.lower() for n in (app, cls) if n}
+    for info in Gio.AppInfo.get_all():
+        wm = info.get_startup_wm_class() if isinstance(info, Gio.DesktopAppInfo) else None
+        if wm and wm.lower() in names:
+            return info.get_id().removesuffix(".desktop")
+    return app
+
+
 class Launch:
     """An app we started: its first window floats, or (for a profile) its windows go to the
     profile's entries for it, in order."""
@@ -745,7 +763,10 @@ class Daemon:
         Returns (reply, pid or None)."""
         if app:
             app = app.removesuffix(".desktop")
-            info = Gio.DesktopAppInfo.new(app + ".desktop")
+            try:
+                info = Gio.DesktopAppInfo.new(app + ".desktop")
+            except TypeError:  # PyGObject raises for the NULL a missing desktop file returns
+                info = None
             if info is None:
                 return f"error no app {app}", None
             pids = []
@@ -776,8 +797,10 @@ class Daemon:
             return None
         chain = pid_chain(int(ev.get("pid") or 0))
         app = ev.get("app", "")
+        # (An X11 window in a Flatpak gives its pid in the sandbox: only its app id can match.)
+        apps = {app, desktop_name(app, ev.get("cls", ""))} if app and any(la.app for la in self.launches) else {app}
         for la in self.launches:
-            if any(p in chain for p in la.pids) or (la.app and app and la.app == app):
+            if any(p in chain for p in la.pids) or (la.app and app and la.app in apps):
                 if not la.entries or len(la.entries) <= 1:
                     self.launches.remove(la)
                 return la
@@ -841,7 +864,7 @@ class Daemon:
         for wid, ev in self.windows.items():
             if not recordable(ev):
                 continue
-            entry = {"app": ev["app"]} if ev.get("app") else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")}
+            entry = {"app": desktop_name(ev["app"], ev.get("cls", ""))} if ev.get("app") else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")}
             if not entry.get("app") and not entry.get("cmd"):
                 continue
             f = self.floats.get(wid)
@@ -868,7 +891,7 @@ class Daemon:
         return out
 
     def window_key(self, ev):
-        return ev.get("app") or json.dumps(cmdline(ev.get("pid")))
+        return desktop_name(ev.get("app", ""), ev.get("cls", "")) or json.dumps(cmdline(ev.get("pid")))
 
     def open_profile(self, name):
         """Open a profile's apps (additive: nothing closes)."""
@@ -884,9 +907,9 @@ class Daemon:
             if key != "[]":
                 groups.setdefault(key, []).append(e)
         claimed = set()
+        keys = {wid: self.window_key(ev) for wid, ev in self.windows.items() if recordable(ev)}
         for key, entries in groups.items():
-            have = [ev for wid, ev in self.windows.items()
-                    if wid not in claimed and recordable(ev) and self.window_key(ev) == key]
+            have = [ev for wid, ev in self.windows.items() if wid not in claimed and keys.get(wid) == key]
             for e, ev in zip(entries, have):
                 claimed.add(ev["id"])
                 self.place_entry(ev, e)

gaze/README.md

@@ -7,7 +7,7 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (t
 - `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
 - `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
 
-Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`); the probe and our own tracker are installed by hand.
+Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`) and then our own tracker (`gaze/tracker/install.sh`), which gaze mode uses once it's installed; the probe is installed by hand.
 
 ```
 gaze/run.sh install           # the gaze service: builds ft-gaze and the panel, starts with SteamVR
@@ -23,7 +23,7 @@ gaze/probe/ft-gazeprobe --screen 1
 Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:
 
 - `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes:
-  - **Eye tracker:** SteamVR's (the default), or our own (Own tracker: see "Our own eye tracker" below). The gaze service runs ours while this is on. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
+  - **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
   - **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
 
   With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
@@ -33,8 +33,9 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
 - **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
 - **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
 - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
-- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
+- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
 - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
+- **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there.
 
 Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.
 

gaze/ft-gaze.cpp

@@ -479,9 +479,24 @@ int main(int argc, char **argv) {
     double lastEmit = 0;
     int actionErrors = 0;
     vr::EVRInputError lastActionError = vr::VRInputError_None;
+    // During a VR game, SteamVR's gaze action is left alone. With ft-gaze reading the eyes, SteamVR
+    // restarted its eye tracker every 10 s or so in a game, as if the headset came off, and each
+    // restart took input focus from the game: Beat Saber paused (PR #13). Of what ft-gaze reads,
+    // only the action reaches SteamVR (the mmap and our tracker are files), so gaze still works
+    // over the dashboard. Games are told apart the way ft-screens does it, by the scene app.
+    bool inGame = false;
+    double nextGameCheck = 0;
 
     while (true) {
         const double now = NowRaw();
+        if (now >= nextGameCheck) {
+            nextGameCheck = now + 0.5;
+            const bool game = vr::VRApplications()->GetCurrentSceneProcessId() != 0;
+            if (game != inGame)
+                std::fprintf(stderr, "ft-gaze: %s\n",
+                             game ? "a VR game is running: SteamVR's gaze action left alone" : "the VR game ended");
+            inGame = game;
+        }
         vr::TrackedDevicePose_t hp;
         sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &hp, 1);
         if (hp.bPoseIsValid) history.Add(now, hp.mDeviceToAbsoluteTracking);
@@ -502,24 +517,26 @@ int main(int argc, char **argv) {
             // SteamVR's action: a room-space origin and fixation point, turned into the head
             // frame so every source reports the same kind of angles.
             std::string action = "{\"ok\":0}";
-            vr::VRActiveActionSet_t active{};
-            active.ulActionSet = set;
-            active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin;
-            input->UpdateActionState(&active, sizeof active, 1);
-            vr::VREyeTrackingData_t e{};
-            const vr::EVRInputError ae =
-                input->GetEyeTrackingDataRelativeToNow(gaze, vr::TrackingUniverseStanding, 0, &e, sizeof e);
-            if (ae == vr::VRInputError_None && e.bActive && e.bValid) {
-                const Vec3 o{e.vGazeOrigin.v[0], e.vGazeOrigin.v[1], e.vGazeOrigin.v[2]};
-                const Vec3 t{e.vGazeTarget.v[0], e.vGazeTarget.v[1], e.vGazeTarget.v[2]};
-                const Vec3 dHead = RotateInverse(headNow, Normalize(t - o));
-                char extra[96];
-                std::snprintf(extra, sizeof extra, "\"tracked\":%d,\"dist\":%.3f,", int(e.bTracked), Length(t - o));
-                action = SrcJson(list, headNow, dHead, extra);
-            } else if (ae != lastActionError || (verbose && ++actionErrors % 90 == 1)) {
-                std::fprintf(stderr, "ft-gaze: action: error %d active %d valid %d\n", int(ae), int(e.bActive),
-                             int(e.bValid));
-                lastActionError = ae;
+            if (!inGame) {
+                vr::VRActiveActionSet_t active{};
+                active.ulActionSet = set;
+                active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin;
+                input->UpdateActionState(&active, sizeof active, 1);
+                vr::VREyeTrackingData_t e{};
+                const vr::EVRInputError ae =
+                    input->GetEyeTrackingDataRelativeToNow(gaze, vr::TrackingUniverseStanding, 0, &e, sizeof e);
+                if (ae == vr::VRInputError_None && e.bActive && e.bValid) {
+                    const Vec3 o{e.vGazeOrigin.v[0], e.vGazeOrigin.v[1], e.vGazeOrigin.v[2]};
+                    const Vec3 t{e.vGazeTarget.v[0], e.vGazeTarget.v[1], e.vGazeTarget.v[2]};
+                    const Vec3 dHead = RotateInverse(headNow, Normalize(t - o));
+                    char extra[96];
+                    std::snprintf(extra, sizeof extra, "\"tracked\":%d,\"dist\":%.3f,", int(e.bTracked), Length(t - o));
+                    action = SrcJson(list, headNow, dHead, extra);
+                } else if (ae != lastActionError || (verbose && ++actionErrors % 90 == 1)) {
+                    std::fprintf(stderr, "ft-gaze: action: error %d active %d valid %d\n", int(ae), int(e.bActive),
+                                 int(e.bValid));
+                    lastActionError = ae;
+                }
             }
 
             std::string m1 = "{\"ok\":0}", m2 = m1, left = m1, right = m1, eye = "null";

gaze/ft-gazectl

@@ -36,8 +36,17 @@ def main():
         if r is None:
             sys.exit("the pointer helper isn't running")
         print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)")
-        if ask("ft_gazed", "status", 0.3) is None:
+        st = ask("ft_gazed", "status", 0.3)
+        if st is None:
             print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow")
+        elif r == "ok on":
+            try:
+                calibrated = (json.loads(st).get("checks") or {}).get("calibrated")
+            except ValueError:
+                calibrated = None
+            if calibrated is False:
+                print("note: no calibration for this eye tracker yet; it opens in the headset "
+                      "(if it can't, ft-gazectl status says why: checks.problem)")
     elif cmd in ("status", "forget", "reload"):
         r = ask("ft_gazed", cmd)
         if r is None:

gaze/ft-gazed

@@ -3,8 +3,12 @@
 
 Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read
 again when the file changes:
-  GAZE_TRACKER=steam|own   SteamVR's eye tracker (default), or our own (gaze/tracker/ft-eyes,
-                           ft-gaze's source "own"; this service runs it, see below)
+  GAZE_TRACKER=auto|own|steam
+                           our own eye tracker (gaze/tracker/ft-eyes, ft-gaze's source "own";
+                           this service runs it, see below) or SteamVR's. auto (the default)
+                           is ours when it's installed (gaze/tracker/install.sh: the frame
+                           grabber, and ft-eyes' Python in this checkout), else SteamVR's; it
+                           switches when ours is installed or removed.
   GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far
                            off it was at your recent nudges; left or right counts that eye twice
                            as much as the other. Either eye alone carries the gaze when the
@@ -59,9 +63,18 @@ and the error moves): lessons from before count less then, so the first few afte
 relearn the offset.
 
 Our own tracker (gaze/tracker/ft-eyes) runs here too, in the dev container, while
-GAZE_TRACKER=own or the gaze probe asks for it ("eyes SECONDS", a lease the probe renews). It
-reads the eye-camera frames the root service frametop-eyegrab copies (gaze/tracker/install.sh),
-which copies them only while ft-eyes runs.
+GAZE_TRACKER=own and the gaze is in use (below), or the gaze probe asks for it ("eyes SECONDS",
+a lease the probe renews). It reads the eye-camera frames the root service frametop-eyegrab
+copies (gaze/tracker/install.sh), which copies them only while ft-eyes runs.
+
+Idle: ft-gaze and our own tracker run only while the gaze is in use: gaze mode on with someone
+wearing the headset (the pointer helper says both: "gaze ? headset" -> "ok on|off worn|away"), a
+check or calibration open or asked for, or a "wake" lease (the Gaze page of Frametop Input
+Settings renews one while it's open). IDLE_AFTER after the last use they stop, and the frame
+grabber goes idle with our tracker; with our tracker, that was over half a core with gaze mode
+off. A check asked for while idle waits for the tracker to start (at most WAKE_SETTLE). Our
+tracker's next click after it starts again re-seats it, as after the headset was off, so turning
+gaze mode on after a while opens the quick check.
 
 Checks and calibration (gaze/gazecheck.py): a one-dot quick check when the headset goes on or
 our tracker asks for a click, and the full calibration when gaze mode comes on without one,
@@ -79,6 +92,8 @@ Control socket: abstract unix datagram "@ft_gazed":
   reload                           read calibration.json and the settings again
   eyes <seconds>                   keep our own tracker running that much longer (at most 120),
                                    whatever GAZE_TRACKER says: the probe's lease. Reply: "ok"
+  wake <seconds>                   keep the gaze running (not idle) that much longer (at most 120),
+                                   whatever gaze mode says: Input Settings' Gaze page. Reply: "ok"
   quickcal                         the one-dot check now (the calibration if there's none)
   calibrate                        the full calibration in the panel
   fitcheck                         the headset fit check in the panel
@@ -119,6 +134,7 @@ EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes"  # our own tracker
 EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python"  # numpy, OpenCV (build.sh)
 EYES_SOCKET = "\0ft_eyes"                      # its control socket
 EYES_CAMS = Path("/dev/shm/frametop-eyes-cams")  # the frames it reads (frametop-eyegrab.service)
+EYEGRAB = (Path("/etc/frametop/ft-eyegrab"), Path("/etc/systemd/system/frametop-eyegrab.service"))  # tracker/install.sh
 CONF = Path.home() / ".config" / "frametop.conf"
 CALIBRATION = STATE / "calibration.json"
 LESSONS = STATE / "pointer-lessons.json"
@@ -126,6 +142,7 @@ LESSON_LOG = STATE / "pointer-lessons.jsonl"
 SOURCES = ("action", "mmap1", "mmap2", "left", "right")  # the ones with a calibration here
 SIDES = ("left", "right")  # ft-gaze's order, and the sources for each eye alone
 TRACKERS = ("steam", "own")
+TRACKER_SETTINGS = ("auto",) + TRACKERS
 BIASES = ("auto", "left", "right")
 NUDGE_MAX = 55.0  # degrees: POINTER_GAZE_NUDGE_MAX's default, the largest lesson taken
 HISTORY = 12.0    # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later)
@@ -134,6 +151,9 @@ RETRY = 3.0       # seconds before starting ft-gaze again
 EYES_RETRY = 10.0  # seconds before starting ft-eyes again after it stopped on its own
 EYES_LEASE_MAX = 120.0
 SETTLE = 0.3      # seconds after an eye is found again before the fallback learns from it
+IDLE_AFTER = 30.0  # seconds after the gaze was last in use before ft-gaze and our tracker stop
+WAKE_SETTLE = 15.0  # seconds after waking that the tracker may take to start sending
+WAKE_MAX = 120.0
 KEYBOARD_PITCH = -20.0  # degrees: gaze under this, on no screen, is a look at the keyboard
 
 
@@ -153,8 +173,14 @@ def log(msg):
     print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
 
 
+def own_installed():
+    """Is our own tracker installed: its frame grabber, and ft-eyes' Python here?"""
+    return all(p.exists() for p in EYEGRAB) and EYES_PYTHON.exists()
+
+
 def read_settings():
-    """(tracker, eye bias, nudge max) from frametop.conf, defaults for anything missing or unknown."""
+    """(tracker, GAZE_TRACKER, eye bias, nudge max) from frametop.conf, defaults for anything
+    missing or unknown. The tracker is "steam" or "own": auto picks ours when it's installed."""
     conf = {}
     try:
         for line in CONF.read_text().splitlines():
@@ -164,13 +190,15 @@ def read_settings():
                 conf[k.strip()] = v.strip().lower()
     except OSError:
         pass
-    tracker = conf.get("GAZE_TRACKER", "steam")
+    setting = conf.get("GAZE_TRACKER", "auto")
+    setting = setting if setting in TRACKER_SETTINGS else "auto"
+    tracker = setting if setting != "auto" else "own" if own_installed() else "steam"
     bias = conf.get("GAZE_EYE", "auto")
     try:
         nudge = min(max(float(conf.get("POINTER_GAZE_NUDGE_MAX", NUDGE_MAX)), 1.0), 110.0)  # the helper's range
     except ValueError:
         nudge = NUDGE_MAX
-    return tracker if tracker in TRACKERS else "steam", bias if bias in BIASES else "auto", nudge
+    return tracker, setting, bias if bias in BIASES else "auto", nudge
 
 
 def mtime(path):
@@ -185,7 +213,7 @@ class Service:
         self.override, self.verbose, self.to = source, verbose, to
         self.source = source or "mmap1"  # the older path's source
         STATE.mkdir(parents=True, exist_ok=True)
-        self.tracker, self.bias, self.nudge_max = read_settings()
+        self.tracker, self.tracker_setting, self.bias, self.nudge_max = read_settings()
         self.conf_mtime = mtime(CONF)
         self.models = {name: Correction() for name in SOURCES}
         self.mode = DEFAULT_MODEL
@@ -216,6 +244,10 @@ class Service:
         self.eyes_proc = None  # ft-eyes, while it runs
         self.eyes_until = 0.0  # the probe's lease (monotonic time)
         self.eyes_restart_at = 0.0
+        self.awake = False      # ft-gaze (and our tracker) run: the gaze is in use (see the top)
+        self.idle_at = 0.0      # idle from then, unless it's in use again first
+        self.woke_at = 0.0
+        self.wake_until = 0.0   # a "wake" lease
 
         self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK)
         self.sock.bind(ME)
@@ -246,9 +278,12 @@ class Service:
 
     def load_settings(self):
         self.conf_mtime = mtime(CONF)
-        tracker, bias, self.nudge_max = read_settings()
+        tracker, self.tracker_setting, bias, self.nudge_max = read_settings()
         if (tracker, bias) != (self.tracker, self.bias):
-            log(f"tracker {tracker}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
+            auto = ""
+            if self.tracker_setting == "auto":
+                auto = " (auto: ours is installed)" if tracker == "own" else " (auto: ours isn't installed)"
+            log(f"tracker {tracker}{auto}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
             self.tracker, self.bias = tracker, bias
             for w in self.weights.values():
                 w.bias = bias
@@ -382,6 +417,46 @@ class Service:
             log(f"lesson log: {e}")
         return rec
 
+    # --- Idle (see the top) ---
+
+    def use_reason(self):
+        """Why the gaze is in use now, or None."""
+        c = self.checks
+        if c.gaze_on and c.headset is not False:
+            return "gaze mode is on"
+        if c.active or c.pending:
+            return "a check"
+        if time.monotonic() < self.wake_until:
+            return "asked to stay awake"
+        return None
+
+    def idle_reason(self):
+        c = self.checks
+        if c.gaze_on is None:
+            return "the pointer helper isn't answering (SteamVR not running?)"
+        if c.gaze_on and c.headset is False:
+            return "nobody is wearing the headset"
+        return "gaze mode is off"
+
+    def waking(self):
+        """Idle, or awake too briefly for the tracker to be sending yet."""
+        return not self.awake or time.monotonic() - self.woke_at < WAKE_SETTLE
+
+    def update_awake(self):
+        now = time.monotonic()
+        why = self.use_reason()
+        if why:
+            self.idle_at = now + IDLE_AFTER
+        if why and not self.awake:
+            self.awake, self.woke_at, self.restart_at = True, now, 0.0
+            log(f"awake: {why}")
+        elif not why and self.awake and now >= self.idle_at:
+            self.awake = False
+            log(f"idle: {self.idle_reason()}; ft-gaze and our tracker stop until the gaze is used again")
+            self.stop_helper()
+            if self.eyes_proc and not self.eyes_wanted():
+                self.stop_eyes()
+
     # --- ft-gaze ---
 
     def start_helper(self):
@@ -424,7 +499,7 @@ class Service:
         self.proc = None
 
     def eyes_wanted(self):
-        return (self.tracker == "own" and not self.override) or time.monotonic() < self.eyes_until
+        return (self.tracker == "own" and not self.override and self.awake) or time.monotonic() < self.eyes_until
 
     def start_eyes(self):
         """Our own tracker, in the dev container, with build/venv's numpy and OpenCV. Like
@@ -683,6 +758,14 @@ class Service:
                     reply = "ok"
                 except ValueError:
                     reply = "error bad seconds"
+            elif words[:1] == ["wake"] and len(words) == 2:
+                try:
+                    secs = min(max(float(words[1]), 0.0), WAKE_MAX)
+                    self.wake_until = max(self.wake_until, time.monotonic() + secs)
+                    self.update_awake()
+                    reply = "ok"
+                except ValueError:
+                    reply = "error bad seconds"
             elif words[:1] == ["reload"]:
                 self.load_settings()
                 self.load_calibration()
@@ -733,7 +816,9 @@ class Service:
             st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
                       lessons=max(len(m) for m in w.misses), own_running=bool(own),
                       own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
-        st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists())
+        st.update(awake=self.awake, idle=None if self.awake else self.idle_reason())
+        st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists(),
+                  tracker_setting=self.tracker_setting, own_installed=own_installed())
         st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
                    if self.last_sample else None, "headset_on": self.steam.wearing(),
                    "headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None,
@@ -749,8 +834,10 @@ class Service:
         if mtime(CALIBRATION) != self.cal_mtime:
             self.load_calibration()
             self.refit()
-        if mtime(CONF) != self.conf_mtime:
+        if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
+                                              and own_installed() != (self.tracker == "own")):
             self.load_settings()
+        self.update_awake()
         want = self.eyes_wanted()
         if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:
             self.start_eyes()
@@ -771,7 +858,7 @@ class Service:
         next_verbose = time.monotonic() + 5
         while self.running:
             now = time.monotonic()
-            if not self.proc and now >= self.restart_at:
+            if self.awake and not self.proc and now >= self.restart_at:
                 self.start_helper()
             for key, _ in self.sel.select(timeout=0.05 if self.checks.active else 0.5):
                 if key.data == "control":

gaze/gazecal.py

@@ -640,31 +640,44 @@ def cross_validate(points, mode):
     return errs
 
 
-def steady_samples(samples, vergence_jump=1.5):
+def steady_samples(samples, vergence_jump=1.5, why=None):
     """The samples of one look at one spot where the tracker had both eyes: none in a blink
     (openness under half its median over the samples), none where it had lost an eye (its
     variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the
     vergence) is more than `vergence_jump` degrees from its median over the samples. The
     vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
     fraction of one far off), so only a jump away from what it was during this look means
-    the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept."""
+    the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.
+    `why`, a dict, gets how many were dropped for each reason: "lost_left", "lost_right",
+    "lost_both", "blink" and "vergence" (each sample once, for the first that applies)."""
+    if why is None:
+        why = {}
     # Openness: a blink is a sharp drop from what it was during this look. Not a fixed
     # level: looking down, the upper lids come down with the eyes, and in bright light you
     # squint, so the reading can stay under 0.5 for the whole look while the tracker follows
     # the eyes fine (a calibration dot at the bottom of the bright round failed that way).
     opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
     floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12
-    opened = []
+    seen = []
     for smp in samples:
-        o = (smp["src"].get("mmap1") or {}).get("open")
-        if not o or min(o) >= floor:
-            opened.append(smp)
+        m1 = smp["src"].get("mmap1") or {}
+        o = m1.get("open")
+        lost = [u > EYE_LOST for u in m1.get("unc") or [0, 0]]
+        # A lost eye's openness reads 0 too, so a lost eye is named before a blink.
+        key = ("lost_both" if all(lost) else "lost_left" if lost[0] else "lost_right") if any(lost) else \
+            "blink" if o and min(o) < floor else None
+        if key:
+            why[key] = why.get(key, 0) + 1
+        else:
+            seen.append(smp)
 
     def vergence(smp):
         return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
-    opened = [smp for smp in opened if max((smp["src"].get("mmap1") or {}).get("unc") or [0]) <= EYE_LOST]
-    have = [v for v in map(vergence, opened) if v is not None]
+    have = [v for v in map(vergence, seen) if v is not None]
     if len(have) < 5:
-        return opened
+        return seen
     med = statistics.median(have)
-    return [smp for smp in opened if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
+    kept = [smp for smp in seen if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
+    if len(kept) < len(seen):
+        why["vergence"] = why.get("vergence", 0) + len(seen) - len(kept)
+    return kept

gaze/gazecheck.py

@@ -10,16 +10,21 @@ directions: look at each one.
           once every QUICK_COOLDOWN, and on "quickcal" (Frametop Input Settings, or a mouse
           button or key combination mapped to Gaze quick check), and when a click's correction
           was past POINTER_GAZE_NUDGE_MAX (55 degrees; the helper's "recheck"): the tracker is
-          far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing.
+          far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing. The headset going
+          on is seen only while the gaze service is awake (gaze mode on, someone wearing it: see
+          ft-gazed), so it also opens when gaze mode comes on after the service idled.
   five    the middle and four around it, when the first FIVE_COUNT lessons after a quick check
           were all over FIVE_LIMIT degrees off: the quick check didn't fix it.
   full    the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil
           size, and the tracker's error with it, changes with brightness), each the middle and
           a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots)
           RING degrees out, half that in the middle round, turned 20 degrees a round. It opens
-          when gaze mode comes on without a calibration for the tracker in use, and on
-          "calibrate". Frametop's screens hide while it runs. Quitting it while there's still
-          no calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
+          whenever gaze mode is on without a calibration for the tracker in use and someone's
+          in the headset, and on "calibrate". One that closes unfinished (ignored, too few
+          dots) opens again only once the headset comes off and on, or gaze mode off and on.
+          Frametop's screens hide while it runs. Quitting it while there's still no
+          calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
+          Why gaze mode can't work yet goes in the status ("problem"), for Input Settings.
 
   fit     the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a
           card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was
@@ -27,6 +32,9 @@ directions: look at each one.
           adjust the headset. A left click or Meta+J runs its guided check (dots, then looks
           down, up, left and right); a right click or Meta+K closes it, as does FIT_TIMEOUT.
 
+A check asked for while the gaze service idles (quickcal, calibrate, fitcheck) wakes it and
+waits until the tracker sends, at most ft-gazed's WAKE_SETTLE; then it opens, or logs why not.
+
 The quick check's dot captures itself: from CHECK_SETTLE after it shows (the eyes getting
 there), once the gaze has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and
 capture time). It's the gaze holding still that counts, not where the tracker puts it, so it
@@ -34,7 +42,10 @@ works however far off the tracker is; a left click or Meta+J (the pointer helper
 "calaccept") takes it now. The full calibration's and five's dots wait for that click: you
 click when you're looking at the dot (the user asked for that: a steady gaze isn't always on
 the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as
-it takes, up to CLICK_IDLE. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
+it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason:
+gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a
+click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common
+reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
 renewed every second; the helper shows it again by itself when that stops).
 
 What a capture teaches:
@@ -90,6 +101,7 @@ ROUND_BG = (0.03, 0.33, 0.8)
 ROUND_NAMES = ("dark", "medium", "bright")
 RING_SCALE = (1.0, 0.5, 1.0)
 PANEL_RETRY = 10.0
+FULL_RETRY = 10.0      # seconds before an automatic calibration that failed to start tries again
 FIT_TIMEOUT = 300.0    # seconds the fit check stays up
 FIT_EVERY = 0.5        # seconds between its cards' updates (each is a new picture for the panel)
 FIT_HINT_WIDTH = 95    # characters a hint line holds in the panel
@@ -117,6 +129,42 @@ def check_dots(kind, own):
     return out
 
 
+# Why SteamVR's samples for a look were dropped (gazecal.steady_samples' counts) -> (short, long):
+# short for the small panel, long for the calibration's. FIT_HINT goes after the ones the
+# headset's fit causes, in the calibration's panel.
+STEAM_REASONS = {"lost_left": ("left eye lost", "SteamVR lost your left eye"),
+                 "lost_right": ("right eye lost", "SteamVR lost your right eye"),
+                 "lost_both": ("both eyes lost", "SteamVR lost both eyes"),
+                 "blink": ("blinked", "you blinked"),
+                 "vergence": ("eyes disagreed", "SteamVR's two eyes disagreed")}
+FIT_HINT = "check the headset fit"
+ACCEPT_WAIT = 1.5      # seconds after a click with no capture before the panel says what it waits for
+
+
+def reject_reason(reply, why):
+    """Why a dot wasn't taken -> (short, long, fit): our tracker's reply (`reply`), or SteamVR's
+    drop counts (`why`, when `reply` is None). `fit`: the headset's fit is the likely cause."""
+    if reply is None:
+        if not why:
+            return "no reading", "SteamVR sent no reading for that look", False
+        key = max(why, key=why.get)
+        return *STEAM_REASONS[key], key.startswith("lost")
+    words = reply.removeprefix("fail ").split()
+    # ft-eyes: "fail the left eye was seen in only 3 frames", "fail the left eye moved (4.2 px)"
+    if len(words) >= 3 and words[0] == "the" and words[2] == "eye":
+        eye = words[1]
+        if "seen" in words:
+            return f"{eye} eye not seen", f"our tracker saw your {eye} eye in only {words[-2]} frames", True
+        if "moved" in words:
+            return f"{eye} eye moved", f"your {eye} eye moved while you looked", False
+    if not reply:
+        return "no answer", "our tracker didn't answer", False
+    if reply.startswith("our tracker isn't running"):
+        return "tracker not running", "our tracker isn't running", False
+    text = reply.removeprefix("fail ")
+    return text[:24], f"our tracker said: {text}", False
+
+
 def spread(points):
     """The median point and the spread around it (1.4826 x the median distance: a standard
     deviation that one stray sample can't move far)."""
@@ -166,8 +214,13 @@ class Checks:
         sel.register(self.out, selectors.EVENT_READ, "checks")
         self.check = None
         self.gaze_on = None
+        self.headset = None      # someone wears it (the helper's "worn"), False "away", None unknown
         self.gaze_heard = 0.0
-        self.want_full_until = 0.0  # gaze mode came on before the tracker said whether it's calibrated
+        self.pending = None      # (command words, when): asked for while the gaze service idled
+        self.full_armed = True    # gaze mode on without a calibration opens the full one (need_full)
+        self.full_blocked = None  # why it can't open now
+        self.full_retry_at = 0.0
+        self.full_failed = None   # why the last calibration failed (too few dots), for problem()
         self.last_quick = 0.0
         self.sample_at = 0.0     # the tracker last sent anything
         self.seen_at = 0.0       # eyes last seen (SteamVR's tracker's variance for them, "unc")
@@ -248,17 +301,22 @@ class Checks:
             pass
 
     def on_readable(self):
-        """Replies on our socket: the helper's "ok on|off" to "gaze ?"; the panel's are dropped."""
+        """Replies on our socket: the helper's "ok on|off [worn|away]" to "gaze ? headset" (an
+        older helper leaves the headset out); the panel's are dropped."""
         while True:
             try:
-                data = self.out.recv(4096).decode("utf-8", "replace")
+                words = self.out.recv(4096).decode("utf-8", "replace").split()
             except (BlockingIOError, OSError):
                 return
-            if data in ("ok on", "ok off"):
-                on = data == "ok on"
-                was, self.gaze_on, self.gaze_heard = self.gaze_on, on, time.monotonic()
-                if on and was is False:
+            if words[:1] == ["ok"] and len(words) in (2, 3) and words[1] in ("on", "off"):
+                on = words[1] == "on"
+                headset = None if len(words) == 2 else words[2] == "worn"
+                was = (self.gaze_on, self.headset)
+                self.gaze_on, self.headset, self.gaze_heard = on, headset, time.monotonic()
+                if on and was[0] is False:
                     self.on_gaze_on()
+                if was != (on, headset):
+                    self.svc.update_awake()
 
     # --- State ---
 
@@ -282,11 +340,51 @@ class Checks:
         return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2
 
     def on_gaze_on(self):
+        self.full_armed = True
+        self.need_full("gaze mode came on without a calibration")
+
+    def need_full(self, reason):
+        """Gaze mode is on without a calibration: open the full one, once per arming (see the top),
+        or note why it can't open."""
         cal = self.calibrated()
-        if cal is False:
-            self.start("full", "gaze mode came on without a calibration")
-        elif cal is None:
-            self.want_full_until = time.monotonic() + 20
+        if cal:
+            self.full_armed = True  # missing one later (the other tracker picked) is news again
+        if not self.gaze_on or self.check or not self.full_armed or cal is not False:
+            self.full_blocked = None
+            return
+        now = time.monotonic()
+        if not self.can_run() and self.svc.waking():
+            return  # the tracker is still starting (the service idled)
+        if not self.can_run():
+            why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2
+                   else "no eyes seen (is the headset on?)")
+        elif now < self.full_retry_at:
+            return
+        else:
+            reply = self.start("full", reason)
+            why = None if reply == "ok" else reply.removeprefix("error ")
+            if why:
+                self.full_retry_at = now + FULL_RETRY
+        if why and why != self.full_blocked:
+            log(f"the calibration can't open: {why}")
+        self.full_blocked = why
+        if not why:
+            self.full_armed = False
+
+    def problem(self):
+        """Why gaze mode, on, can't follow your eyes yet, or None. Our tracker not having said
+        yet is None: Input Settings has its own line for our tracker."""
+        if not self.gaze_on or self.calibrated() is not False:
+            return None
+        if self.check and self.check["kind"] == "full":
+            return "Not calibrated yet: the calibration is open in the headset"
+        if self.full_blocked:
+            return f"Not calibrated, and the calibration can't open: {self.full_blocked}"
+        if not self.full_armed:
+            if self.full_failed:
+                return f"Not calibrated: the calibration failed (most dots: {self.full_failed}). Use Calibrate"
+            return "Not calibrated: the calibration closed unfinished. Use Calibrate"
+        return "Not calibrated: the calibration opens in the headset"
 
     def auto_quick(self, reason):
         now = time.monotonic()
@@ -318,7 +416,7 @@ class Checks:
         now = time.monotonic()
         self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
                       "started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
-                      "skipped": 0, "captured": 0, "points": {}}
+                      "skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""}
         log(f"{kind} check: {reason}")
         if kind == "full":
             st = ask(SCREENS, "state", 0.5).split()
@@ -402,10 +500,12 @@ class Checks:
         self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look")
         self.last_progress = None
         c["shown"] = time.monotonic()
-        c["run"], c["accept"], c["done_at"] = [], False, None
+        c["run"], c["accept"], c["done_at"], c["accept_at"] = [], False, None, None
 
     def on_sample(self, s):
         self.sample_at = time.monotonic()
+        if self.pending:
+            self.run_pending()
         unc = (s["src"].get("mmap1") or {}).get("unc")
         if unc and min(unc) <= EYE_LOST:
             self.seen_at = time.monotonic()
@@ -428,6 +528,7 @@ class Checks:
         if "hy" not in src:
             return
         now = time.monotonic()
+        c["gaze_at"] = now
         if now < c["shown"] + CHECK_SETTLE:
             return
         g = (src["hy"], src["hp"])
@@ -488,7 +589,9 @@ class Checks:
             if ok:
                 svc.weights["own"].add(miss)
         else:
-            steady = steady_samples(samples)
+            why = {}
+            steady = steady_samples(samples, why=why)
+            rec["dropped"] = why
             reads = {}
             for name in ("action", "mmap1", "mmap2", "left", "right"):
                 pts = [(smp["src"][name]["hy"], smp["src"][name]["hp"]) for smp in steady
@@ -522,23 +625,41 @@ class Checks:
                 if svc.kind == "eyes":
                     svc.weights["steam"].add(miss)
                 svc.dirty = True
+        if not ok:
+            short, long, fit = reject_reason(rec.get("reply", "") if c["own"] else None, rec.get("dropped"))
+            rec["reason"] = long
+            c["reasons"][long] = c["reasons"].get(long, 0) + 1
+            if fit:
+                c["fit_reasons"].add(long)
         rec["accepted"] = ok
         self.log_check(rec)
         if not ok:
             c["tries"] += 1
-            log(f"{c['kind']} check dot {c['i'] + 1}: not taken ({rec.get('reply', '')})")
-            if c["tries"] >= 2 or c["kind"] != "full":
+            log(f"{c['kind']} check dot {c['i'] + 1}: not taken: {rec['reason']} ({rec.get('reply', '')})")
+            full = c["kind"] == "full"
+            if c["tries"] >= 2 or not full:
+                self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if full else f"Skipped: {short}")
                 self.skip()
             else:
+                self.note(f"Not taken: {long}. Look at the dot and click again")
                 self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
-                c["run"], c["accept"] = [], False
+                c["run"], c["accept"], c["accept_at"] = [], False, None
                 c["shown"] = time.monotonic()  # settle again, then retry
             return
         c["captured"] += 1
         c["tries"] = 0
+        self.note("")
         self.to_panel(f"dot {yaw:.3f} {pitch:.3f} done")
         c["done_at"] = time.monotonic() + DONE_PAUSE
 
+    def note(self, text):
+        """The panel's warning line, over the instructions (empty: none). It stays until the
+        next dot is taken, so a skipped dot's reason is still there at the one after it."""
+        c = self.check
+        if c.get("note") != text:
+            c["note"] = text
+            self.to_panel(f"note {text}".rstrip())
+
     def skip(self):
         c = self.check
         yaw, pitch, _ = c["dots"][c["i"]]
@@ -567,12 +688,20 @@ class Checks:
             return
         n = len(c["dots"])
         if c["captured"] < n * 2 / 3:
-            log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays")
+            reasons = c["reasons"]
+            main = max(reasons, key=reasons.get) if reasons else None
+            self.full_failed = main
+            log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays"
+                + (f". Not taken: {', '.join(f'{r} ({k})' for r, k in reasons.items())}" if reasons else ""))
             self.to_panel(f"text Calibration failed: only {c['captured']} of {n} dots. Try again from Input Settings")
+            if main:
+                fit = main in c["fit_reasons"]
+                self.note(f"Most dots: {main}" + (". Check headset fit on the Gaze page" if fit else ""))
             self.to_panel("dot 0 0 off")
-            c["done_at"] = time.monotonic() + 3.0
+            c["done_at"] = time.monotonic() + (8.0 if main else 3.0)  # time to read why
             c["closing"] = True
             return
+        self.full_failed = None
         if c["own"]:
             reply = ask(EYES, "calib-fit", 10.0)
             log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
@@ -629,9 +758,28 @@ class Checks:
 
     # --- From the service ---
 
-    def command(self, words):
+    def run_pending(self):
+        """A check asked for while the service idled: once the tracker sends (and our tracker has
+        said whether it's calibrated), or WAKE_SETTLE after waking, when its error is the real one."""
+        svc = self.svc
+        words, _ = self.pending
+        ready = (time.monotonic() - svc.last_sample < 2 if words[0] == "fitcheck" else self.can_run()) \
+            and (svc.kind != "own" or self.calibrated() is not None)
+        if not ready and svc.waking():
+            return
+        self.pending = None
+        reply = self.command(words, queue=False)
+        if reply != "ok":
+            log(f"{words[0]}, asked for while idle: {reply.removeprefix('error ')}")
+
+    def command(self, words, queue=True):
         """quickcal, calibrate, calaccept, calquit -> a reply."""
         cmd = words[0]
+        if cmd in ("quickcal", "calibrate", "fitcheck") and queue and self.svc.waking() and not self.check:
+            # The tracker isn't running (or only just started): wake it, and do this once it sends.
+            self.pending = (words, time.monotonic())
+            self.svc.update_awake()
+            return "ok waking the eye tracker first"
         if cmd == "quickcal":
             return self.start("full" if self.calibrated() is False else "quick", "asked for")
         if cmd == "calibrate":
@@ -642,6 +790,8 @@ class Checks:
             if self.check and self.check["kind"] == "fit":
                 self.check["fit"].toggle_guide(time.monotonic())
             elif self.check:
+                if not self.check["accept"]:
+                    self.check["accept_at"] = time.monotonic()
                 self.check["accept"] = True
             return "ok"
         if cmd == "calquit":
@@ -688,6 +838,13 @@ class Checks:
             return
         if c["done_at"]:
             return
+        if c["accept"] and c["accept_at"] and now - c["accept_at"] > ACCEPT_WAIT:
+            # Clicked, but no capture yet (see on_sample): say what it's waiting for.
+            full = c["kind"] == "full"
+            if now - c.get("gaze_at", 0.0) > 0.5:
+                self.note("Waiting: the eye tracker isn't sending a gaze" if full else "Waiting: no gaze")
+            else:
+                self.note("Waiting for your gaze to hold still on the dot" if full else "Hold your look still")
         if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
             self.close("ignored")
         elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
@@ -698,23 +855,20 @@ class Checks:
         now = time.monotonic()
         if not self.panel_proc and now >= self.panel_restart_at:
             self.start_panel()
-        self.to_helper("gaze ?")
+        self.to_helper("gaze ? headset")
         if now - self.gaze_heard > 5:
-            self.gaze_on = None  # the helper isn't answering
+            self.gaze_on = self.headset = None  # the helper isn't answering
+        if self.pending:
+            self.run_pending()
         if self.check:
             self.to_helper("calpanel 1")
-        if self.want_full_until:
-            cal = self.calibrated()
-            if cal is not None or now > self.want_full_until:
-                self.want_full_until = 0.0
-                if cal is False and self.gaze_on:
-                    self.start("full", "gaze mode came on without a calibration")
         if not self.eyes_seen(AWAY_MIN):
             self.away, self.back_since = True, None
         elif self.back_since is not None and not self.eyes_seen():
             self.back_since = None  # gone again before DON_DELAY
         elif self.back_since is not None and now - self.back_since >= DON_DELAY:
             self.away, self.back_since = False, None
+            self.full_armed = True  # a calibration that closed unfinished opens again
             self.auto_quick("the headset went on")
         if svc.kind == "own" and now - svc.own_at < 5:
             reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values())
@@ -723,10 +877,13 @@ class Checks:
             elif not self.reseat_seen and self.can_run():
                 self.reseat_seen = True
                 self.auto_quick("our tracker asked for a click")
+        self.need_full("gaze mode is on without a calibration")
 
     def status(self):
         c = self.check
-        st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
+        st = {"check": None, "gaze_mode": self.gaze_on, "headset_worn": self.headset, "pending": self.pending[0][0]
+              if self.pending else None, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
+              "problem": self.problem(),
               "panel": self.panel_proc is not None,
               "last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None}
         if c:

gaze/panel/ft-gazepanel.cpp

@@ -22,6 +22,8 @@
 //                                look, capture (a ring filling to progress), done,
 //                                fail, off
 //   title <text> / text <text>   a line at the top / at the bottom (empty to clear)
+//   note <text>                  a warning line just above the bottom one, in orange (red on
+//                                the bright round): why a dot wasn't taken (empty to clear)
 //   eye <0|1> <r> <g> <b> <signal 0..1|-1> <seen 0..1|-1> <word>
 //                                fit: an eye's card (0 left): its state in that colour, the
 //                                tracker's signal, the share of the last 10 s it was seen
@@ -143,7 +145,7 @@ struct Panel {
     double wDeg = kQuickDeg;  // across
     std::vector<uint8_t> px;
     double bg = 0.05;
-    std::string title, text;
+    std::string title, text, note;
     bool dotOn = false;
     double dotYaw = 0, dotPitch = 0, progress = 0;
     std::string state = "off";
@@ -268,6 +270,9 @@ void Draw(Panel &p) {
     const int titleSize = int(pxPerDeg * (p.full ? 1.5 : 1.1)), textSize = int(pxPerDeg * (p.full ? 1.2 : 0.9));
     Text(p, p.title, titleSize, p.w / 2, int(titleSize * 1.2), faint);
     Text(p, p.text, textSize, p.w / 2, p.h - int(textSize * 1.3), faint);
+    // Under the full calibration's lowest dots (RING degrees down) and over the text.
+    if (light) Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 0.7, 0.12, 0.05);
+    else Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 1.0, 0.62, 0.3);
     if (p.fit) DrawFit(p, pxPerDeg, textSize, faint);
     if (!p.dotOn || p.state == "off") return;
     double x, y;
@@ -468,7 +473,7 @@ int main(int argc, char **argv) {
                 p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx;
                 p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx;
                 p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg;
-                p.title.clear(), p.text.clear(), p.dotOn = false, p.state = "off";
+                p.title.clear(), p.text.clear(), p.note.clear(), p.dotOn = false, p.state = "off";
                 p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
                 place();
                 visible = dirty = true;  // shown with its first picture
@@ -496,6 +501,8 @@ int main(int argc, char **argv) {
                 p.title = buf[5] == ' ' ? buf + 6 : "", dirty = true;
             } else if (!std::strncmp(buf, "text", 4)) {
                 p.text = buf[4] == ' ' ? buf + 5 : "", dirty = true;
+            } else if (!std::strncmp(buf, "note", 4)) {
+                p.note = buf[4] == ' ' ? buf + 5 : "", dirty = true;
             } else if (std::sscanf(buf, "%15s", word) == 1 && !std::strcmp(word, "ping")) {
                 reply = visible ? "ok shown" : "ok hidden";
             } else {

gaze/test/idle-test.py

@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+"""Offline test of the gaze service idling (gaze/ft-gazed, gaze/gazecheck.py): ft-gaze runs only
+while the gaze is in use, and a check asked for while it idles waits for the tracker.
+
+Runs ft-gazed's Service with its sockets renamed, a fake pointer helper (answers "gaze ?
+headset" as the test says), and a fake ft-gaze (prints samples, quits when its stdin closes).
+SteamVR's tracker is the one in use, so our own isn't started; the panel isn't either. Nothing
+reaches the live gaze service, the pointer helper, or SteamVR, so it's safe next to them.
+
+  gaze/test/idle-test.py
+"""
+import importlib.machinery
+import importlib.util
+import os
+import socket
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+GAZE = os.path.join(HERE, "..")
+sys.path.insert(0, GAZE)
+loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(GAZE, "ft-gazed"))
+gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
+loader.exec_module(gazed)
+import gazecheck  # noqa: E402  (the module ft-gazed imported)
+
+tag = f"ft_gaze_idle_test_{os.getpid()}"
+gazed.ME = f"\0{tag}_gazed"
+gazed.POINTER = gazecheck.POINTER = f"\0{tag}_helper"
+gazecheck.SCREENS = f"\0{tag}_screens"
+gazecheck.PANEL_PROG = gazed.REPO / "nonexistent-panel"  # "isn't built": no panel
+gazed.IDLE_AFTER, gazed.WAKE_SETTLE = 1.0, 3.0
+gazed.read_settings = lambda: ("steam", "steam", "auto", 55.0)
+logs = []
+gazed.log = gazecheck.log = lambda msg: logs.append(msg)
+
+# The fake ft-gaze: 90 samples a second, both eyes seen, until its stdin closes.
+tmp = tempfile.mkdtemp(prefix="ft-gaze-idle-test-")
+FAKE = os.path.join(tmp, "ft-gaze")
+with open(FAKE, "w") as f:
+    f.write('''import json, select, sys, time
+while True:
+    if select.select([sys.stdin], [], [], 1 / 90)[0] and not sys.stdin.read(1):
+        break
+    eye = {"hy": 1.0, "hp": 2.0}
+    print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001],
+          "open": [0.8, 0.8]}, "left": eye, "right": eye}}), flush=True)
+''')
+started = []
+
+
+def start_helper(self):
+    """ft-gaze, straight from here instead of the dev container."""
+    import selectors
+    self.proc = subprocess.Popen([sys.executable, FAKE], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+                                 stderr=subprocess.PIPE)
+    os.set_blocking(self.proc.stdout.fileno(), False)
+    os.set_blocking(self.proc.stderr.fileno(), False)
+    self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
+    self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr")
+    self.buf = b""
+    started.append(time.monotonic())
+
+
+gazed.Service.start_helper = start_helper
+
+# The fake pointer helper.
+helper_state = {"reply": "ok off worn"}
+helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+helper.bind(gazed.POINTER)
+helper.settimeout(0.2)
+
+
+def answer():
+    while True:
+        try:
+            data, addr = helper.recvfrom(512)
+        except socket.timeout:
+            continue
+        except OSError:
+            return
+        if data.startswith(b"gaze ?") and addr:
+            helper.sendto(helper_state["reply"].encode(), addr)
+
+
+threading.Thread(target=answer, daemon=True).start()
+svc = gazed.Service(None, False, gazed.POINTER)
+threading.Thread(target=svc.run, daemon=True).start()
+ctl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+ctl.bind("")
+ctl.settimeout(2)
+
+
+def ask(cmd):
+    ctl.sendto(cmd.encode(), gazed.ME)
+    return ctl.recv(4096).decode()
+
+
+failures = []
+
+
+def check(label, got, want):
+    ok = got == want
+    print(("ok    " if ok else "FAIL  ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(label)
+
+
+def wait(cond, seconds):
+    end = time.monotonic() + seconds
+    while time.monotonic() < end:
+        if cond():
+            return True
+        time.sleep(0.05)
+    return cond()
+
+
+time.sleep(2.5)
+check("gaze mode off: idle, no ft-gaze", (svc.awake, svc.proc is None), (False, True))
+check("status says why", ask("status").count('"idle": "gaze mode is off"'), 1)
+
+helper_state["reply"] = "ok on worn"
+check("gaze mode on, headset worn: awake within 2 s", wait(lambda: svc.awake and svc.proc is not None, 2.5), True)
+check("samples come in", wait(lambda: svc.counts["samples"] > 20, 2), True)
+
+helper_state["reply"] = "ok on away"
+check("headset off: still awake for IDLE_AFTER", wait(lambda: not svc.awake, 0.5), False)
+check("then idle, ft-gaze stopped", wait(lambda: not svc.awake and svc.proc is None, 3.5), True)
+check("why: nobody wears it", ask("status").count('"idle": "nobody is wearing the headset"'), 1)
+
+helper_state["reply"] = "ok on"
+check("an older helper (no headset word): awake with gaze mode on", wait(lambda: svc.awake, 2.5), True)
+helper_state["reply"] = "ok off worn"
+check("gaze mode off again: idle", wait(lambda: not svc.awake and svc.proc is None, 4.5), True)
+
+check("wake lease", ask("wake 2"), "ok")
+check("wake: awake at once", (svc.awake, wait(lambda: svc.proc is not None, 1)), (True, True))
+check("lease over (2 s + IDLE_AFTER): idle", wait(lambda: not svc.awake, 4.5), True)
+
+time.sleep(0.5)
+logs.clear()
+count = len(started)
+check("quick check while idle: queued, waking", ask("quickcal"), "ok waking the eye tracker first")
+check("it woke", wait(lambda: svc.awake and len(started) > count, 1.5), True)
+check("once the tracker sends, it ran (and says why it couldn't open)",
+      wait(lambda: any("quickcal, asked for while idle: the panel isn't running" in m for m in logs), 3), True)
+check("nothing left pending", svc.checks.pending, None)
+check("idle again after", wait(lambda: not svc.awake, 3), True)
+
+print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+svc.running = False
+time.sleep(0.7)
+os.remove(FAKE)
+os.rmdir(tmp)
+os._exit(1 if failures else 0)

gaze/tracker/install.sh

@@ -2,8 +2,9 @@
 # Install (or remove) the frame grabber our own eye tracker needs: ft-eyegrab, as the system
 # service frametop-eyegrab.service. It copies the eye-camera frames, read-only, out of
 # SteamVR's eyetracking process into /dev/shm/frametop-eyes-cams for ft-eyes, and only while
-# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when Eye tracker
-# is Own tracker or the gaze probe uses it.
+# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when ours is the
+# tracker in use (GAZE_TRACKER=auto, the default, picks it once this is installed) or the gaze
+# probe uses it. install.sh offers this after gaze mode.
 # Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit: it asks for
 # the password in the terminal, on the Frame or from a PC, or runs SUDO_ASKPASS when that's set
 # (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env).

hands/README.md

@@ -14,6 +14,8 @@ Two programs, each a user service that stops when SteamVR does:
 
 They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` doesn't install it.
 
+For the cutouts alone, `hands/ft-cutouts on` starts the same two programs with ft-hands' `--no-gestures`: your hands show through the screens, and no pinch or grip is detected, so nothing clicks or drags. It runs this checkout's build as transient user units, so it needs `hands/build.sh` and ft-camd's capabilities (`hands/run.sh caps`) but not `hands/run.sh install`. It and `ft-handsctl on` stop each other's services, and it stops with SteamVR too.
+
 ```
 ft-handsctl on | off            # on the Frame: start or stop hand tracking (SteamVR must be running)
 ft-handsctl status              # the services, and ft-hands' last status lines
@@ -21,6 +23,8 @@ ft-handsctl log [lines]
 ft-handsctl cutouts on|off|state  # ft-screens' hand cutouts, without stopping tracking
 ft-handsctl gestures            # pinches and grips, live (tools/watch_gestures.py --distance)
 
+hands/ft-cutouts on | off | status   # the cutouts only: no pinches or grips
+
 hands/run.sh install            # build, give ft-camd its capabilities (sudo, once per build), install disabled
 hands/run.sh start|stop         # start or stop the services
 hands/run.sh restart            # after changing a setting
@@ -104,6 +108,7 @@ Options:
 - `--models DIR`: where the models are.
 - `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
 - `--no-publish`: don't write the hands and gestures files.
+- `--no-gestures`: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. `ft-cutouts` runs it this way.
 - `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
 - `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
 - `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.

hands/ft-cutouts

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# ft-cutouts: the hand cutouts without pinches and grips. Your hands show through Frametop's
+# screens, but ft-hands runs with --no-gestures, so nothing clicks or drags. ft-handsctl on is
+# the full version; each stops the other. Like it, this doesn't start with SteamVR, and it
+# stops when SteamVR does. On the Frame.
+#
+#   ft-cutouts on | off | status
+#
+# Runs this checkout's hands/build as transient user units, so it doesn't need hands/run.sh
+# install: only a build (hands/build.sh) and ft-camd's capabilities (hands/run.sh caps).
+set -euo pipefail
+here=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
+camd=frametop-cutouts-camd.service hands=frametop-cutouts-hands.service
+
+die() { echo "$*" >&2; exit 1; }
+
+run_unit() {  # unit, the ft-handsctl unit it replaces, description, then the command
+  local unit=$1 conflicts=$2 what=$3
+  shift 3
+  systemctl --user -q is-active "$unit" && return
+  systemd-run --user --quiet --collect --unit="$unit" --description="Frametop hand cutouts: $what" \
+    -p PartOf=steamvr.service -p After=steamvr.service -p Conflicts="$conflicts" \
+    -p Restart=always -p RestartSec=5 -p TimeoutStopSec=5 "$@"
+}
+
+case ${1:-status} in
+  on)
+    systemctl --user -q is-active steamvr.service || die "SteamVR isn't running"
+    [ -x "$here/build/ft-camd" ] && [ -x "$here/build/ft-hands" ] || die "not built: hands/build.sh"
+    grep -qa -- --no-gestures "$here/build/ft-hands" || die "ft-hands predates --no-gestures: hands/build.sh"
+    getcap "$here/build/ft-camd" | grep -q cap_sys_ptrace ||
+      die "ft-camd needs its capabilities: hands/run.sh caps (asks for sudo)"
+    if systemctl --user -q is-active frametop-camd.service frametop-hands.service; then
+      echo "stopping ft-handsctl's hand tracking (it has pinches and grips)"
+    fi
+    run_unit $camd frametop-camd.service "the camera broker" "$here/build/ft-camd" --status 60
+    if ! systemctl --user -q is-active $hands; then
+      "$here/../scripts/container-up.sh"  # so stopping the unit can't take the container down
+      pkill -x ft-hands || true
+    fi
+    run_unit $hands frametop-hands.service "hand tracking, no gestures" "$HOME/.local/bin/distrobox" enter dev -- \
+      "$here/build/ft-hands" --status 60 --no-gestures
+    "$here/ft-handsctl" cutouts on >/dev/null || true
+    sleep 4
+    "$0" status ;;
+  off)
+    systemctl --user stop $hands $camd 2>/dev/null || true
+    systemctl --user -q is-active frametop-hands.service || pkill -x ft-hands || true
+    echo "hand cutouts off" ;;
+  status)
+    for u in $camd $hands; do echo "$u: $(systemctl --user is-active $u || true)"; done
+    echo "ft-screens cutouts: $("$here/ft-handsctl" cutouts state 2>&1)"
+    inv=$(systemctl --user show -p InvocationID --value $hands)  # this run's lines only
+    [ -n "$inv" ] && journalctl --user _SYSTEMD_INVOCATION_ID="$inv" --no-pager -o cat |
+      grep -E '^ *[0-9.]+s |sets with a hand|cameras:' | tail -2 | cut -c1-160 || true ;;
+  *) sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
+esac

hands/track/main.cpp

@@ -3,9 +3,12 @@
 // Python prototype: the same scheduling, with the models on a few threads.
 //
 //   ft-hands [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N]
-//            [--no-publish] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
+//            [--no-publish] [--no-gestures] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
 //            (--help lists them all)
 //
+// --no-gestures: hands for the cutouts only. No pinch or grip detection, so nothing reaches
+// the pointer and a closing hand doesn't raise the rate; the gestures file is removed.
+//
 // Which cameras (--cams, HANDS_CAMERAS): the four mono IR cameras light the hands with their
 // own IR and track well in dim rooms, but in bright light (a sunny room, a window behind the
 // hands) they expose for the room and the hands come out dark. The two Arcturus colour
@@ -150,7 +153,7 @@ struct Lighting {
 int main(int argc, char **argv) {
     double seconds = 0, status = 5;
     int threads = 3, niceness = 5;
-    bool int8 = false, publish = true, track = true, swap_sides = false;
+    bool int8 = false, publish = true, track = true, swap_sides = false, gestures_on = true;
     std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
     std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME;
     // SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for
@@ -189,6 +192,7 @@ int main(int argc, char **argv) {
         else if (a == "--nice" && more) niceness = std::atoi(argv[++i]);
         else if (a == "--int8") int8 = true;
         else if (a == "--no-publish") publish = false;
+        else if (a == "--no-gestures") gestures_on = false;
         else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]);
         else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
         else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
@@ -217,6 +221,7 @@ int main(int argc, char **argv) {
         }
         else {
             std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n"
+                        "          [--no-gestures] (hands for the cutouts only: no pinches or grips)\n"
                         "          [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n"
                         "          [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
                         "          [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
@@ -265,12 +270,14 @@ int main(int argc, char **argv) {
         return true;
     };
     if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
-        (publish && (!pub.open(err) || !gestures.open(pinch, grip, err))) ||
+        (publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
         (!record.empty() && !start_recording(record, err))) {
         std::fprintf(stderr, "%s\n", err.c_str());
         return 1;
     }
     if (!ring.alive()) return std::fprintf(stderr, "ft-camd isn't running (no heartbeat)\n"), 1;
+    // so a reader can't take an earlier run's file for this one's
+    if (publish && !gestures_on) unlink((run_dir() + "/gestures").c_str());
 
     std::map<std::string, int> index;   // mono calibration name -> ring camera
     std::map<std::string, int> color;   // colour calibration name (color_video<N>) -> ring camera
@@ -541,9 +548,11 @@ int main(int argc, char **argv) {
         const auto hands = tracker.step(images, int64_t(tmin));
         const uint64_t capture = uint64_t(int64_t(tmin) - raw_off);   // CLOCK_MONOTONIC
         const std::vector<Seen> views = tracker.views_now();
-        grip.update(hands, views, int64_t(capture));
-        pinch.update(hands, views, int64_t(capture), grip.gripping());
-        if (gesture_log && capture - t_glog >= 100'000'000) {
+        if (gestures_on) {
+            grip.update(hands, views, int64_t(capture));
+            pinch.update(hands, views, int64_t(capture), grip.gripping());
+        }
+        if (gestures_on && gesture_log && capture - t_glog >= 100'000'000) {
             t_glog = capture;
             for (int k = 0; k < 2; ++k)
                 if (pinch.world_d[k] >= 0)
@@ -554,7 +563,8 @@ int main(int argc, char **argv) {
         // a gesture down or closing gets the full rate, even while the palm holds still
         next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) -
                                    0.005) * 1e9);
-        if (publish) pub.write(hands, capture), gestures.write(pinch, grip, capture);
+        if (publish) pub.write(hands, capture);
+        if (publish && gestures_on) gestures.write(pinch, grip, capture);
         for (const Pinch::Event &e : grip.events)
             std::printf("grip  %s %-5s curl %.2f at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
                         e.point[0], e.point[1], e.point[2]);
@@ -593,17 +603,19 @@ int main(int argc, char **argv) {
                             s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten,
                             !rec ? "" : ("  recorded " + std::to_string(rec->written()) + " dropped " +
                                          std::to_string(rec->dropped())).c_str());
-            std::printf("        pinches: left %u right %u (held back, palm down: %d %d)  grips: left %u right %u",
-                        pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1],
-                        grip.side(0).begins, grip.side(1).begins);
-            for (int k = 0; k < 2; ++k)
-                if (pinch.side(k).flags & FH_PINCH_TRACKED)
-                    std::printf("  %s %s d %.3f curl %.2f", k ? "right" : "left",
-                                grip.side(k).flags & FH_PINCH_DOWN    ? "GRIP"
-                                : pinch.side(k).flags & FH_PINCH_DOWN ? "PINCH"
-                                                                      : "open",
-                                pinch.side(k).distance, grip.curl[k]);
-            std::printf("\n");
+            if (gestures_on) {
+                std::printf("        pinches: left %u right %u (held back, palm down: %d %d)  grips: left %u right %u",
+                            pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1],
+                            grip.side(0).begins, grip.side(1).begins);
+                for (int k = 0; k < 2; ++k)
+                    if (pinch.side(k).flags & FH_PINCH_TRACKED)
+                        std::printf("  %s %s d %.3f curl %.2f", k ? "right" : "left",
+                                    grip.side(k).flags & FH_PINCH_DOWN    ? "GRIP"
+                                    : pinch.side(k).flags & FH_PINCH_DOWN ? "PINCH"
+                                                                          : "open",
+                                    pinch.side(k).distance, grip.curl[k]);
+                std::printf("\n");
+            }
             for (const Hand *h : hands)
                 std::printf("        hand %d %-5s views %d wrist %+.3f %+.3f %+.3f m  scale %.2f  speed %.2f m/s\n", h->id,
                             h->right() ? "right" : "left", h->nviews, h->pts[0][0], h->pts[0][1], h->pts[0][2], h->scale,
@@ -616,8 +628,8 @@ int main(int argc, char **argv) {
         }
     }
     if (!color.empty()) unlink(want_file.c_str());
-    if (publish) {
-        pub.write({}, mono_ns());
+    if (publish) pub.write({}, mono_ns());
+    if (publish && gestures_on) {
         pinch.release(int64_t(mono_ns()));   // a drag in progress ends, as lost
         grip.release(int64_t(mono_ns()));
         gestures.write(pinch, grip, mono_ns());

input-settings/ft_input_settings.py

@@ -9,11 +9,15 @@ to the input relay over its control socket (@frametop_relay):
   - Controllers: the same for the Frame controllers' buttons, minus the gaze actions (gaze
     mode is a mouse feature). They're read by the pointer helper through SteamVR input
     (@ft_pointer_helper: vrstatus, vrglobal), and a mapped button is taken from games.
-  - Keyboard: when Frametop's keyboard opens, and key combinations for any action (Meta+Shift+F
-    floats a window unless the rules have their own list).
+  - Keyboard: when Frametop's keyboard opens, and key combinations for any action, a command
+    of your own too ("command:CMD"), or a modifier tapped alone (a Meta tap opens Steam's menu
+    and Meta+Shift+F floats a window, unless the rules have their own list).
   - Pointer: speed, dot size, distance and the rest, applied live.
   - Ignored panels: SteamVR overlays the pointer passes through (POINTER_IGNORE), by app or
     one by one. The helper lists them (@ft_pointer_helper "overlays").
+  - Game optimization: pausing Frametop while a VR game runs, so it leaves the CPU and GPU to the game
+    (input/game_pause.py, in the relay: "pause on|off|?"): pause now, pause by itself during VR
+    games, the controller gesture that pauses and resumes, what happens to the desktop, a sound.
   - Gaze: the pointer's gaze mode (@ft_pointer_helper "gaze") and the gaze service
     (gaze/ft-gazed, @ft_gazed: status, forget, reload), with its eye tracker (SteamVR's or
     our own) and eye bias (GAZE_TRACKER, GAZE_EYE in frametop.conf).
@@ -60,6 +64,7 @@ DEFAULT_BUTTONS = {0x110: "left", 0x111: "right", 0x112: "middle", 0x113: "back"
 ACTION_LABELS = {
     "left": "Left click", "right": "Right click", "middle": "Middle click", "back": "Back",
     "scroll_up": "Scroll up", "scroll_down": "Scroll down", "dashboard": "Toggle SteamVR dashboard",
+    "steam_menu": "Open Steam menu / close dashboard",
     "recenter": "Recenter pointer", "pointer_toggle": "Pointer on/off",
     "follow_toggle": "Head follow on/off (experimental)", "gaze_toggle": "Gaze pointer on/off (experimental)",
     "gaze_precision": "Gaze precision: hold to steer, release to click",
@@ -71,6 +76,7 @@ ACTION_LABELS = {
     "sens_down": "Slower pointer", "layout_reset": "Reset desktop screen layout",
     "screens_toggle": "Hide/show desktop screens", "keyboard_toggle": "Open/close keyboard",
     "float_toggle": "Float window in VR / put it back", "dock_all": "Put all floating windows back",
+    "pause_toggle": "Pause/resume Frametop (for VR games)",
     "key": "Pass through as key",
     "none": "Do nothing",
 }
@@ -93,6 +99,13 @@ CONTROLLER_BUTTONS = {
     "right/bumper": "Right bumper", "right/trigger": "Right trigger", "right/grip": "Right grip",
     "right/thumbstick": "Right stick click",
 }
+# Pausing for VR games (input/game_pause.py): the gesture a rules file without "pause_gesture" gets,
+# both thumbsticks clicked together twice, and what can happen to the desktop.
+DEFAULT_PAUSE_GESTURE = {"buttons": ["left/thumbstick", "right/thumbstick"], "presses": 2}
+PAUSE_DESKTOP_MODES = {"hide": "Hide it and slow it down (windows stay open)",
+                       "close": "Close it (windows close; it starts again when you resume)"}
+PAUSE_REASONS = {"game": "a VR game started", "gesture": "controller gesture", "command": "ft-pause",
+                 "settings": "from here"}
 # Gaze mode is a mouse and keyboard feature (docs/gaze-controllers.md; the relay's GAZE_ACTIONS).
 GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
 CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none") + GAZE_ACTIONS]
@@ -102,6 +115,8 @@ KEYBOARD_ONLY = ("gaze_left", "gaze_right")
 # Profiles (docs/profiles.md): "profile:NAME" opens one (the relay runs ft-layout use NAME).
 LAYOUT_PATH = os.path.expanduser("~/.config/frametop-layout.json")
 PROFILE = "profile:"
+# "command:CMD": the relay runs CMD with sh -c (key combinations only, here).
+COMMAND = "command:"
 
 
 def profile_actions():
@@ -113,6 +128,8 @@ def profile_actions():
 def action_label(a):
     if a.startswith(PROFILE):
         return f"Open profile {a[len(PROFILE):]}"
+    if a.startswith(COMMAND):
+        return f"Run: {a[len(COMMAND):]}"
     return ACTION_LABELS.get(a, a)
 
 
@@ -120,14 +137,18 @@ def is_profile(a):
     return a.startswith(PROFILE) and len(a) > len(PROFILE)
 
 
+def is_command(a):
+    return a.startswith(COMMAND) and bool(a[len(COMMAND):].strip())
+
+
 def mappable(a):
     """An action a controller button can have."""
     return a in CONTROLLER_ACTIONS or is_profile(a)
 
 
 def shortcut_mappable(a):
-    """An action a key combination can have: the gaze ones too."""
-    return a in SHORTCUT_ACTIONS or is_profile(a)
+    """An action a key combination can have: the gaze ones and commands too."""
+    return a in SHORTCUT_ACTIONS or is_profile(a) or is_command(a)
 # Gaze mode settings (pointer helper), like POINTER_SETTINGS.
 GAZE_SETTINGS = [
     ("POINTER_GAZE_RETAKE", "Look away to hand back", 5, 1, 45, 0.5, "°"),
@@ -141,9 +162,10 @@ GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, releas
 # Key combinations ("key_bindings" in the rules): modifiers, either side folded into the left code.
 MODIFIER_CODES = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
 MODIFIER_NAMES = {29: "Ctrl", 42: "Shift", 56: "Alt", 125: "Meta"}
-# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): Meta+J and
-# Meta+K click at the gaze, Meta+Shift+F floats a window.
-DEFAULT_KEY_BINDINGS = {"125+36": "gaze_left", "125+37": "gaze_right", "42+125+33": "float_toggle"}
+# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): a Meta tap
+# opens Steam's menu, Meta+J and Meta+K click at the gaze, Meta+Shift+F floats a window.
+DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
+                        "42+125+33": "float_toggle"}
 
 
 def key_bindings(rules):
@@ -151,8 +173,12 @@ def key_bindings(rules):
     counts as its own)."""
     bound = rules.get("key_bindings")
     return dict(bound) if isinstance(bound, dict) else dict(DEFAULT_KEY_BINDINGS)
-# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias.
+# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias. GAZE_TRACKER
+# can also be auto (the default): ours when it's installed, else SteamVR's.
 GAZE_TRACKERS = {"steam": "SteamVR's eye tracker", "own": "our own eye tracker"}
+# Our tracker's frame grabber (gaze/tracker/install.sh), on the host: this runs in the dev
+# container, which has the host's /etc under /run/host.
+EYEGRAB = ("/etc/frametop/ft-eyegrab", "/run/host/etc/frametop/ft-eyegrab")
 GAZE_EYES = {"auto": "auto", "left": "left eye", "right": "right eye"}
 # Pointer settings: key, label, default, min, max, step, unit.
 POINTER_SETTINGS = [
@@ -278,6 +304,7 @@ class Backend(QObject):
     gazeChanged = Signal()
     driverChanged = Signal()
     panelsChanged = Signal()
+    pauseChanged = Signal()
     activity = Signal(str)  # device id
     captured = Signal(int, str)  # code, name
     capturedController = Signal(str, str)  # button, label
@@ -295,6 +322,7 @@ class Backend(QObject):
         self._capture_vr = False
         self._capture_combo = ""  # the action a key combination is being captured for
         self._combo_mods = set()
+        self._combo_tap = None  # a modifier pressed alone, nothing since: released, it's a tap
         self._vr = {}  # the helper's vrstatus, {} when it doesn't answer
         self._vr_at = 0.0
         self._gaze = {}  # ft-gazed's status, {} when it isn't running
@@ -304,6 +332,8 @@ class Backend(QObject):
         self._check_asked = 0.0  # when quickcal or calibrate went to the gaze service (its errors)
         self._driver_block = ""  # set by _check_driver
         self._panels = None  # SteamVR's overlays, from the helper; None until it answers
+        self._pause = {}  # the relay's pause state ("pause ?"), {} when it doesn't answer
+        self._pause_at = 0.0
         self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
         self.sock.bind("")  # autobind an abstract address the relay can reply to
         self.sock.setblocking(False)
@@ -336,6 +366,7 @@ class Backend(QObject):
 
     def _refresh(self):
         self._send("devices")
+        self._send("pause ?")
         self._send("vrstatus", HELPER)
         self._send("gaze ?", HELPER)
         if not self._send("status", GAZED) and self._gaze:
@@ -349,6 +380,9 @@ class Backend(QObject):
         if self._gaze_mode is not None and now - self._gaze_at > 5:
             self._gaze_mode = None
             self.gazeChanged.emit()
+        if self._pause and now - self._pause_at > 5:
+            self._pause = {}
+            self.pauseChanged.emit()
 
     def _check_driver(self):
         block = driver_block()
@@ -387,13 +421,20 @@ class Backend(QObject):
             if t == "devices":
                 self._nodes = msg.get("nodes", [])
                 self._pointer_mode = bool(msg.get("pointer_mode"))
-                self._relay_ok = True
+                was_ok, self._relay_ok = self._relay_ok, True
+                if not was_ok:
+                    self.pauseChanged.emit()  # (pauseStatus' "old")
                 self.devicesChanged.emit()
             elif t == "overlays":
                 panels = [o for o in msg.get("list", []) if isinstance(o, dict) and isinstance(o.get("key"), str)]
                 if panels != self._panels:
                     self._panels = panels
                     self.panelsChanged.emit()
+            elif t == "pause":
+                self._pause_at = time.monotonic()
+                if msg != self._pause:
+                    self._pause = msg
+                    self.pauseChanged.emit()
             elif t == "vrstatus":
                 self._vr = msg
                 self._vr_at = time.monotonic()
@@ -408,13 +449,16 @@ class Backend(QObject):
                 self.activity.emit(msg["id"])
                 code, value = int(msg["code"]), msg["value"]
                 if code in MODIFIER_CODES:
-                    (self._combo_mods.add if value else self._combo_mods.discard)(MODIFIER_CODES[code])
+                    mod = MODIFIER_CODES[code]
+                    if value == 1:
+                        self._combo_tap = None if self._combo_mods else mod
+                    (self._combo_mods.add if value else self._combo_mods.discard)(mod)
+                    if value == 0 and self._combo_tap == mod:
+                        self._end_shortcut_capture(str(mod))  # the relay's modifier tap
                 elif value == 1 and code < BTN_MISC:
-                    self._save_shortcut("+".join(str(c) for c in sorted(self._combo_mods) + [code]),
-                                        self._capture_combo)
-                    self._capture_combo = ""
-                    self._combo_mods = set()
-                    self.shortcutCaptureChanged.emit()
+                    self._end_shortcut_capture("+".join(str(c) for c in sorted(self._combo_mods) + [code]))
+                elif value == 1:
+                    self._combo_tap = None  # a mouse button: no tap
             elif t == "event":
                 self.activity.emit(msg["id"])
                 if (self._capture_id and msg["id"] == self._capture_id and msg["type"] == "key"
@@ -798,6 +842,108 @@ class Backend(QObject):
         else:
             self.message.emit("The pointer helper isn't running (frametop-pointer.service)", True)
 
+    # --- pausing for VR games ---
+    @Property("QVariantMap", notify=pauseChanged)
+    def pauseStatus(self):
+        """The relay's pause state, and a line saying it ("relay": False when it doesn't answer)."""
+        st = self._pause
+        if not st:
+            # "old": the relay answers, but not about pausing (it predates it).
+            return {"relay": False, "old": self._relay_ok, "paused": False,
+                    "summary": "Unknown: the input relay " + ("can't pause" if self._relay_ok else "isn't answering")}
+        if st.get("paused"):
+            reason = PAUSE_REASONS.get(st.get("reason"), st.get("reason") or "")
+            summary = f"Paused since {time.strftime('%H:%M', time.localtime(st.get('since') or time.time()))}"
+            summary += f" ({reason})" if reason else ""
+            if st.get("ends_with_game"):
+                summary += ", until the game ends"
+        else:
+            summary = "Running" + (", with a VR game" if st.get("game") else "")
+        if st.get("busy"):
+            summary += "; working on it…"
+        return {"relay": True, "paused": bool(st.get("paused")), "summary": summary,
+                "gestureReader": bool(st.get("gesture_reader"))}
+
+    @Slot(bool)
+    def setPaused(self, on):
+        if self._send(f"pause {'on' if on else 'off'} settings"):
+            QTimer.singleShot(300, self._refresh)
+        else:
+            self.message.emit("The input relay isn't running (frametop-input-relay.service)", True)
+
+    def _set_rule(self, key, value, text):
+        rules = read_json(RULES_PATH)
+        rules[key] = value
+        self._save_rules(rules)
+        self.message.emit(text, False)
+
+    @Property(bool, notify=mappingsChanged)
+    def pauseAuto(self):
+        return read_json(RULES_PATH).get("pause_auto", True) is not False
+
+    @Slot(bool)
+    def setPauseAuto(self, on):
+        self._set_rule("pause_auto", bool(on), "VR games pause Frametop" if on else "VR games no longer pause Frametop")
+
+    @Property("QVariantList", constant=True)
+    def pauseDesktopModes(self):
+        return [{"value": k, "text": v} for k, v in PAUSE_DESKTOP_MODES.items()]
+
+    @Property(str, notify=mappingsChanged)
+    def pauseDesktop(self):
+        mode = read_json(RULES_PATH).get("pause_desktop")
+        return mode if mode in PAUSE_DESKTOP_MODES else "hide"
+
+    @Slot(str)
+    def setPauseDesktop(self, mode):
+        if mode in PAUSE_DESKTOP_MODES:
+            self._set_rule("pause_desktop", mode, "Paused, the desktop: " + PAUSE_DESKTOP_MODES[mode].split(" (")[0].lower())
+
+    @Property(bool, notify=mappingsChanged)
+    def pauseSound(self):
+        return read_json(RULES_PATH).get("pause_sound", True) is not False
+
+    @Slot(bool)
+    def setPauseSound(self, on):
+        self._set_rule("pause_sound", bool(on), "A sound on pause and resume" if on else "No sound on pause and resume")
+
+    @Property("QVariantList", constant=True)
+    def gestureButtons(self):
+        return [{"value": "", "text": "None"}] + [{"value": b, "text": label} for b, label in CONTROLLER_BUTTONS.items()]
+
+    @Property("QVariantMap", notify=mappingsChanged)
+    def pauseGesture(self):
+        """{"first", "second" ("" for none), "presses"} (the relay's game_pause.gesture_of)."""
+        spec = read_json(RULES_PATH).get("pause_gesture", DEFAULT_PAUSE_GESTURE)
+        buttons = [b for b in (spec or {}).get("buttons", []) if b in CONTROLLER_BUTTONS][:2] \
+            if isinstance(spec, dict) and isinstance(spec.get("buttons"), list) else []
+        presses = 1 if len(buttons) == 2 and spec.get("presses") == 1 else 2
+        return {"first": buttons[0] if buttons else "", "second": buttons[1] if len(buttons) > 1 else "",
+                "presses": presses}
+
+    @Slot(str, str, int)
+    def setPauseGesture(self, first, second, presses):
+        """No first button: no gesture. One button always takes two presses."""
+        buttons = list(dict.fromkeys(b for b in (first, second) if b in CONTROLLER_BUTTONS))
+        if not buttons:
+            self._set_rule("pause_gesture", None, "No controller gesture pauses Frametop")
+            return
+        presses = 1 if len(buttons) == 2 and presses == 1 else 2
+        self._set_rule("pause_gesture", {"buttons": buttons, "presses": presses},
+                       "Pause gesture: " + self.gestureText(buttons, presses))
+
+    @staticmethod
+    def gestureText(buttons, presses):
+        names = " and ".join(CONTROLLER_BUTTONS[b] for b in buttons)
+        together = " together" if len(buttons) == 2 else ""
+        return f"{names}{together}, {'twice' if presses == 2 else 'once'}"
+
+    @Property(str, notify=mappingsChanged)
+    def pauseGestureText(self):
+        g = self.pauseGesture
+        buttons = [b for b in (g["first"], g["second"]) if b]
+        return self.gestureText(buttons, g["presses"]) if buttons else "None"
+
     # --- gaze ---
     def _gaze_status(self, status):
         now = time.monotonic()
@@ -890,6 +1036,8 @@ class Backend(QObject):
     # --- key combinations ("key_bindings") ---
     def comboName(self, combo):
         parts = [int(c) for c in combo.split("+") if c.isdigit()]
+        if len(parts) == 1 and parts[0] in MODIFIER_NAMES:
+            return MODIFIER_NAMES[parts[0]] + " tap"
         return "+".join(MODIFIER_NAMES.get(c) or self.codeName(c).removeprefix("KEY_").title() for c in parts)
 
     @Property("QVariantList", notify=mappingsChanged)
@@ -900,7 +1048,8 @@ class Backend(QObject):
 
     @Property("QVariantList", constant=True)
     def shortcutActions(self):
-        return [{"value": a, "text": action_label(a)} for a in SHORTCUT_ACTIONS + profile_actions()]
+        return [{"value": a, "text": action_label(a)} for a in SHORTCUT_ACTIONS + profile_actions()] + \
+            [{"value": COMMAND, "text": "Run a command…"}]
 
     @Property(bool, notify=shortcutCaptureChanged)
     def capturingShortcut(self):
@@ -911,6 +1060,7 @@ class Backend(QObject):
         if shortcut_mappable(action):
             self._capture_combo = action
             self._combo_mods = set()
+            self._combo_tap = None
             self._send("watch 60")
             self.shortcutCaptureChanged.emit()
 
@@ -919,6 +1069,13 @@ class Backend(QObject):
         self._capture_combo = ""
         self.shortcutCaptureChanged.emit()
 
+    def _end_shortcut_capture(self, combo):
+        self._save_shortcut(combo, self._capture_combo)
+        self._capture_combo = ""
+        self._combo_mods = set()
+        self._combo_tap = None
+        self.shortcutCaptureChanged.emit()
+
     def _save_shortcut(self, combo, action):
         rules = read_json(RULES_PATH)
         rules["key_bindings"] = dict(key_bindings(rules), **{combo: action})
@@ -935,9 +1092,25 @@ class Backend(QObject):
 
     @Property(str, notify=gazeChanged)
     def gazeTracker(self):
-        """Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER)."""
-        v = read_conf().get("GAZE_TRACKER", "steam")
-        return v if v in GAZE_TRACKERS else "steam"
+        """Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER; auto is
+        ours when it's installed)."""
+        v = read_conf().get("GAZE_TRACKER", "auto")
+        if v in GAZE_TRACKERS:
+            return v
+        return "own" if self.gazeOwnInstalled else "steam"
+
+    @Property(bool, notify=gazeChanged)
+    def gazeTrackerAuto(self):
+        """GAZE_TRACKER is auto (or missing): ours once it's installed, without a choice here."""
+        return read_conf().get("GAZE_TRACKER", "auto") not in GAZE_TRACKERS
+
+    @Property(bool, notify=gazeChanged)
+    def gazeOwnInstalled(self):
+        """Is our own tracker installed? The gaze service says (it also checks ft-eyes' Python in
+        its checkout); without it, whether the frame grabber is."""
+        if "own_installed" in self._gaze:
+            return bool(self._gaze["own_installed"])
+        return any(os.path.exists(p) for p in EYEGRAB)
 
     @Property(str, notify=gazeChanged)
     def gazeEye(self):
@@ -990,6 +1163,12 @@ class Backend(QObject):
         else:
             self.message.emit("The gaze service isn't running (frametop-gaze.service)", True)
 
+    @Slot()
+    def keepGazeAwake(self):
+        """The Gaze page is open: the gaze service doesn't idle meanwhile (its status stays live,
+        and a check opens without waiting for the tracker). The page renews this every 10 s."""
+        self._send("wake 30", GAZED)
+
     @Slot()
     def gazeQuickCheck(self):
         """The gaze service's one-dot check, in the panel fixed to the headset."""

input-settings/main.qml

@@ -19,6 +19,7 @@ Kirigami.ApplicationWindow {
             Kirigami.Action { text: "Devices"; icon.name: "input-mouse"; onTriggered: root.show(devicesPage) },
             Kirigami.Action { text: "Buttons"; icon.name: "input-keyboard"; onTriggered: root.show(buttonsPage) },
             Kirigami.Action { text: "Controllers"; icon.name: "input-gamepad"; onTriggered: root.show(controllersPage) },
+            Kirigami.Action { text: "Game optimization"; icon.name: "applications-games"; onTriggered: root.show(gamesPage) },
             Kirigami.Action { text: "Keyboard"; icon.name: "input-keyboard-virtual"; onTriggered: root.show(keyboardPage) },
             Kirigami.Action { text: "Pointer"; icon.name: "transform-move"; onTriggered: root.show(pointerPage) },
             Kirigami.Action { text: "Ignored panels"; icon.name: "view-hidden"; onTriggered: root.show(ignorePage) },
@@ -56,8 +57,8 @@ Kirigami.ApplicationWindow {
         pageStack.push(page)
     }
 
-    // FT_INPUT_PAGE=buttons|controllers|keyboard|pointer|ignore|gaze|bluetooth opens the app on that page.
-    pageStack.initialPage: ({ buttons: buttonsPage, controllers: controllersPage, keyboard: keyboardPage,
+    // FT_INPUT_PAGE=buttons|controllers|games (Game optimization)|keyboard|pointer|ignore|gaze|bluetooth opens the app on that page.
+    pageStack.initialPage: ({ buttons: buttonsPage, controllers: controllersPage, games: gamesPage, keyboard: keyboardPage,
                               pointer: pointerPage, ignore: ignorePage, gaze: gazePage,
                               bluetooth: bluetoothPage })[startPage] || devicesPage
 
@@ -192,7 +193,7 @@ Kirigami.ApplicationWindow {
                 wrapMode: Text.Wrap
                 opacity: 0.7
                 text: "Move or press a device to see which row it is. 3D pointer: grabbed, drives the SteamVR pointer. "
-                      + "Pass through: left alone (a Meta tap toggles the dashboard if META_DASHBOARD=1). Ignore: left alone."
+                      + "Pass through: left alone, but its key combinations (Keyboard page) work. Ignore: left alone."
             }
         }
     }
@@ -545,7 +546,11 @@ Kirigami.ApplicationWindow {
                     delegate: RowLayout {
                         required property var modelData
                         Kirigami.FormData.label: modelData.label + ":"
-                        Controls.Label { text: modelData.actionLabel }
+                        Controls.Label {
+                            text: modelData.actionLabel
+                            elide: Text.ElideRight
+                            Layout.maximumWidth: Kirigami.Units.gridUnit * 24
+                        }
                         Controls.ToolButton {
                             icon.name: "edit-delete"
                             display: Controls.AbstractButton.IconOnly
@@ -568,16 +573,33 @@ Kirigami.ApplicationWindow {
                     }
                     Controls.Button {
                         text: backend.capturingShortcut ? "Press the keys… (Cancel)" : "Set keys…"
+                        enabled: backend.capturingShortcut || shortcutAction.currentValue !== "command:"
+                                 || shortcutCommand.text.trim() !== ""
                         onClicked: backend.capturingShortcut ? backend.cancelShortcutCapture()
-                                                             : backend.startShortcutCapture(shortcutAction.currentValue)
+                                                             : backend.startShortcutCapture(
+                                                                   shortcutAction.currentValue === "command:"
+                                                                   ? "command:" + shortcutCommand.text.trim()
+                                                                   : shortcutAction.currentValue)
                     }
                 }
+                Controls.TextField {
+                    id: shortcutCommand
+                    Kirigami.FormData.label: "Command:"
+                    visible: shortcutAction.currentValue === "command:"
+                    placeholderText: "e.g. ft-layout use Work"
+                    Layout.preferredWidth: Kirigami.Units.gridUnit * 24
+                }
                 Controls.Label {
                     Layout.maximumWidth: Kirigami.Units.gridUnit * 30
                     wrapMode: Text.WordWrap
-                    text: "Hold the modifiers (Ctrl, Alt, Shift, Meta), then press the key, on any keyboard. The "
-                          + "combination's last key isn't typed; the modifiers still reach the app. Meta+Shift+F floats "
-                          + "the desktop window under the pointer in VR, or puts it back, until you remove or change it."
+                    text: "Hold the modifiers (Ctrl, Alt, Shift, Meta), then press the key, on any keyboard. Or tap "
+                          + "one modifier on its own. The combination's last key isn't typed; the modifiers still reach "
+                          + "the app. While you're in Steam or a game, they see the keys too. Until you remove or change "
+                          + "them, a Meta tap opens the Steam menu (or closes the dashboard) instead of the desktop's "
+                          + "launcher, and Meta+Shift+F floats the desktop window under the pointer in VR, or puts it "
+                          + "back. A command runs with sh as the input relay's service, outside the desktop's session, "
+                          + "with Frametop's ft-layout, ft-float and ft-steam on its path; its output goes to the "
+                          + "relay's log (journalctl --user -u frametop-input-relay)."
                     opacity: 0.7
                     font: Kirigami.Theme.smallFont
                 }
@@ -805,6 +827,140 @@ Kirigami.ApplicationWindow {
         }
     }
 
+    // ---------------------------------------------------------------- Games
+    Component {
+        id: gamesPage
+        Kirigami.ScrollablePage {
+            id: gmpage
+            title: "Game optimization"
+            property var st: backend.pauseStatus
+
+            // The gesture's boxes follow the saved gesture (which may tidy what was picked).
+            function syncGesture() {
+                const g = backend.pauseGesture
+                firstBox.currentIndex = firstBox.indexOfValue(g.first)
+                secondBox.currentIndex = secondBox.indexOfValue(g.second)
+                pressBox.currentIndex = pressBox.indexOfValue(g.presses)
+            }
+            function saveGesture() {
+                backend.setPauseGesture(firstBox.currentValue, secondBox.currentValue, pressBox.currentValue)
+            }
+            Component.onCompleted: syncGesture()
+            Connections {
+                target: backend
+                function onMappingsChanged() { gmpage.syncGesture() }
+            }
+
+            ColumnLayout {
+                spacing: Kirigami.Units.largeSpacing
+
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    visible: !gmpage.st.relay
+                    type: Kirigami.MessageType.Error
+                    text: gmpage.st.old
+                          ? "The input relay that runs is older and can't pause. Restart it: "
+                            + "systemctl --user restart frametop-input-relay"
+                          : "The input relay isn't answering (frametop-input-relay.service). It does the pausing."
+                }
+
+                Kirigami.FormLayout {
+                    Layout.fillWidth: true
+
+                    RowLayout {
+                        Kirigami.FormData.label: "Frametop:"
+                        Controls.Label { text: gmpage.st.summary }
+                        Controls.Button {
+                            text: gmpage.st.paused ? "Resume" : "Pause now"
+                            icon.name: gmpage.st.paused ? "media-playback-start" : "media-playback-pause"
+                            enabled: gmpage.st.relay
+                            onClicked: backend.setPaused(!gmpage.st.paused)
+                        }
+                    }
+                    Controls.Switch {
+                        Kirigami.FormData.label: "VR games:"
+                        text: "Pause while a VR game runs, and resume when it ends"
+                        checked: backend.pauseAuto
+                        onToggled: backend.setPauseAuto(checked)
+                    }
+
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Controller gesture" }
+                    Controls.ComboBox {
+                        id: firstBox
+                        Kirigami.FormData.label: "Button:"
+                        model: backend.gestureButtons
+                        textRole: "text"
+                        valueRole: "value"
+                        Layout.preferredWidth: Kirigami.Units.gridUnit * 10
+                        onActivated: gmpage.saveGesture()
+                    }
+                    Controls.ComboBox {
+                        id: secondBox
+                        Kirigami.FormData.label: "Together with:"
+                        model: backend.gestureButtons
+                        textRole: "text"
+                        valueRole: "value"
+                        enabled: firstBox.currentValue !== ""
+                        Layout.preferredWidth: Kirigami.Units.gridUnit * 10
+                        onActivated: gmpage.saveGesture()
+                    }
+                    Controls.ComboBox {
+                        id: pressBox
+                        Kirigami.FormData.label: "Pressed:"
+                        model: [{ value: 2, text: "Twice" }, { value: 1, text: "Once" }]
+                        textRole: "text"
+                        valueRole: "value"
+                        // One button pressed once would go off far too easily in games.
+                        enabled: firstBox.currentValue !== "" && secondBox.currentValue !== ""
+                        Layout.preferredWidth: Kirigami.Units.gridUnit * 10
+                        onActivated: gmpage.saveGesture()
+                    }
+                    Controls.Label {
+                        Layout.fillWidth: true
+                        wrapMode: Text.Wrap
+                        opacity: 0.7
+                        text: (backend.pauseGesture.first === "" ? "No gesture: a mapped button, a key combination, or ft-pause still work."
+                               : "Pauses and resumes Frametop, in games too. The game still gets the presses, so pick buttons it "
+                                 + "doesn't use this way. Two buttons count as together when both go down within a third of a "
+                                 + "second, and twice means within 0.7 s.")
+                              + (gmpage.st.relay && backend.pauseGesture.first !== "" && !gmpage.st.gestureReader
+                                 ? " Not reading the controllers right now (SteamVR isn't running?)." : "")
+                    }
+
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "While paused" }
+                    Controls.ComboBox {
+                        Kirigami.FormData.label: "The desktop:"
+                        model: backend.pauseDesktopModes
+                        textRole: "text"
+                        valueRole: "value"
+                        Layout.preferredWidth: Kirigami.Units.gridUnit * 20
+                        Component.onCompleted: currentIndex = indexOfValue(backend.pauseDesktop)
+                        onActivated: backend.setPauseDesktop(currentValue)
+                    }
+                    Controls.Switch {
+                        Kirigami.FormData.label: "Sound:"
+                        text: "Play a sound on pause and resume"
+                        checked: backend.pauseSound
+                        onToggled: backend.setPauseSound(checked)
+                    }
+                }
+
+                Controls.Label {
+                    Layout.fillWidth: true
+                    wrapMode: Text.Wrap
+                    opacity: 0.7
+                    text: "Paused, Frametop leaves the headset's CPU and GPU to the game. The gaze service and our eye "
+                          + "tracker stop, and nothing reads SteamVR's eye tracking. Hand tracking and remote desktop stop "
+                          + "if they run. The desktop hides and slows down, or closes, as chosen above. The 3D mouse lets go, "
+                          + "so the mouse is a plain one for SteamVR. Mapped buttons and key combinations do nothing but "
+                          + "pausing, the Steam menu, and your own commands. Resuming brings back what pausing stopped. "
+                          + "Map \"Pause/resume Frametop\" to a button or key combination on the other pages, or run "
+                          + "input/ft-pause."
+                }
+            }
+        }
+    }
+
     // ---------------------------------------------------------------- Gaze
     Component {
         id: gazePage
@@ -813,6 +969,14 @@ Kirigami.ApplicationWindow {
             title: "Gaze"
             header: DriverWarning {}
             property var status: backend.gazeStatus
+            // The gaze service idles while the gaze isn't used; open, this page keeps it going.
+            Timer {
+                running: true
+                repeat: true
+                interval: 10000
+                triggeredOnStart: true
+                onTriggered: backend.keepGazeAwake()
+            }
             actions: [
                 Kirigami.Action {
                     text: "Quick check"
@@ -881,6 +1045,23 @@ Kirigami.ApplicationWindow {
                     opacity: 0.7
                     font: Kirigami.Theme.smallFont
                 }
+                Controls.Label {
+                    // Why the gaze pointer, on, can't follow your eyes yet, so it isn't left looking
+                    // like a plain mouse. The calibration part is the gaze service's (gaze/gazecheck.py).
+                    readonly property var checks: gpage.status.checks || {}
+                    readonly property string why: !backend.gazeServiceInstalled
+                        ? "The gaze service isn't installed: run gaze/run.sh install in the Frametop folder, in a terminal"
+                        : !backend.gazeServiceRunning ? "The gaze service isn't running (it starts with SteamVR)"
+                        : backend.gazeTracker === "own" && !gpage.status.eyegrab
+                        ? "Our eye tracker needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)"
+                        : checks.problem || ""
+                    visible: backend.gazeMode > 0 && why !== ""
+                    text: why
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 30
+                    wrapMode: Text.WordWrap
+                    color: checks.check ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor
+                    font: Kirigami.Theme.smallFont
+                }
                 ColumnLayout {
                     Kirigami.FormData.label: "Mouse left button:"
                     Repeater {
@@ -928,16 +1109,28 @@ Kirigami.ApplicationWindow {
                 }
                 RowLayout {
                     Kirigami.FormData.label: "Eye tracker:"
+                    Controls.RadioButton {
+                        text: "Own tracker (recommended)"
+                        checked: backend.gazeTracker === "own"
+                        onToggled: if (checked) backend.setGazeTracker("own")
+                    }
                     Controls.RadioButton {
                         text: "SteamVR"
                         checked: backend.gazeTracker === "steam"
                         onToggled: if (checked) backend.setGazeTracker("steam")
                     }
-                    Controls.RadioButton {
-                        text: "Own tracker"
-                        checked: backend.gazeTracker === "own"
-                        onToggled: if (checked) backend.setGazeTracker("own")
-                    }
+                }
+                Controls.Label {
+                    // Ours is the default once it's installed (GAZE_TRACKER=auto, gaze/ft-gazed).
+                    visible: backend.gazeTracker === "steam" && !backend.gazeOwnInstalled
+                    text: "Our own tracker is more accurate. Install it with gaze/tracker/install.sh in the "
+                          + "Frametop folder, in a terminal (asks for sudo)"
+                          + (backend.gazeTrackerAuto ? "; gaze then uses it, and you calibrate it once."
+                                                     : ", then pick Own tracker here and calibrate it once.")
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 30
+                    wrapMode: Text.WordWrap
+                    opacity: 0.7
+                    font: Kirigami.Theme.smallFont
                 }
                 Controls.Label {
                     // The gaze service runs ours (gaze/tracker/ft-eyes); it needs the frame grabber
@@ -1017,7 +1210,8 @@ Kirigami.ApplicationWindow {
                 Controls.Label {
                     Kirigami.FormData.label: "Service:"
                     text: backend.gazeServiceRunning
-                          ? (gpage.status.ft_gaze ? "running" : "running, eye tracker reader restarting")
+                          ? (gpage.status.awake === false ? "idle: " + gpage.status.idle + " (it starts when the gaze is used)"
+                             : gpage.status.ft_gaze ? "running" : "running, eye tracker reader starting")
                           : backend.gazeServiceInstalled ? "not running (frametop-gaze.service, starts with SteamVR)"
                           : "not installed: run gaze/run.sh install in the Frametop folder, in a terminal"
                     color: backend.gazeServiceRunning ? Kirigami.Theme.textColor : Kirigami.Theme.negativeTextColor

input/ft-clickctl

@@ -0,0 +1,29 @@
+#!/usr/bin/env python3
+"""Controller desktop click stability over local IPC; no SteamVR client."""
+import argparse
+import json
+import math
+import socket
+
+def request(command):
+    with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as client:
+        client.bind(''); client.settimeout(2)
+        client.sendto(command.encode(), '\0ft_screens')
+        response=client.recv(8192).decode()
+    if response.startswith('error'): raise RuntimeError(response)
+    return response
+
+def main():
+    parser=argparse.ArgumentParser(description=__doc__)
+    sub=parser.add_subparsers(dest='command', required=True)
+    sub.add_parser('status')
+    sub.add_parser('threshold').add_argument('pixels', type=float)
+    args=parser.parse_args()
+    try:
+        if args.command == 'threshold':
+            if not math.isfinite(args.pixels) or not 0 <= args.pixels <= 64:
+                raise ValueError('Threshold must be 0–64 logical pixels; 0 disables stabilization')
+            if request(f'controller-click {args.pixels:g}') != 'ok': raise RuntimeError('Threshold rejected')
+        print(json.dumps(json.loads(request('controller-click?')), indent=2))
+    except (OSError, ValueError, RuntimeError) as error: parser.exit(1, str(error)+'\n')
+if __name__ == '__main__': main()

input/ft-pause

@@ -0,0 +1,45 @@
+#!/usr/bin/python3
+"""ft-pause: pause Frametop for a VR game, so it leaves the CPU and GPU to the game, or resume it.
+
+  ft-pause on|off|toggle   pause or resume (input/game_pause.py has what pausing stops)
+  ft-pause status          the pause state, as JSON (the default)
+
+The input relay does it (frametop-input-relay.service). So do the pause_toggle action, the
+controller gesture (by default both thumbsticks clicked together twice), and, with auto pause on,
+a VR game starting; Frametop Input Settings has the settings, on its Game optimization page.
+"""
+
+import json
+import os
+import socket
+import sys
+
+
+def ask(msg, timeout=2.0):
+    s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC)
+    s.bind(f"\0ft_pause.{os.getpid()}")
+    s.settimeout(timeout)
+    try:
+        s.sendto(msg.encode(), "\0frametop_relay")
+        return json.loads(s.recv(65536))
+    except ConnectionRefusedError:
+        sys.exit("the input relay isn't running (frametop-input-relay.service)")
+    except (socket.timeout, ValueError):
+        sys.exit("no answer from the input relay (an older one, without pausing?)")
+    finally:
+        s.close()
+
+
+def main():
+    cmd = sys.argv[1] if len(sys.argv) > 1 else "status"
+    if cmd in ("on", "off", "toggle"):
+        r = ask(f"pause {cmd} command")
+        print("Frametop paused" if r.get("paused") else "Frametop running")
+    elif cmd == "status":
+        print(json.dumps(ask("pause ?"), indent=1))
+    else:
+        sys.exit(__doc__)
+
+
+if __name__ == "__main__":
+    main()

input/game_pause.py

@@ -0,0 +1,515 @@
+"""Pause Frametop for VR games, so it leaves the headset's CPU and GPU to the game. The input
+relay (input-relay.py) runs this: it's the one part of Frametop that always runs.
+
+Paused:
+  - The gaze service stops (frametop-gaze: ft-gazed, ft-gaze, our own eye tracker ft-eyes, the
+    gaze panel), and with it every read of SteamVR's eye tracking. Our frame grabber, the root
+    service ft-eyegrab, goes idle by itself 3 seconds after ft-eyes stops asking for frames.
+  - Hand tracking stops if it runs (frametop-camd, frametop-hands).
+  - The desktop, by "pause_desktop": "hide" (the default) hides every screen and floating window
+    and slows the desktop down ("pause on" to ft-screens: KWin gets one frame callback a second
+    instead of 90, so it and the apps on it hardly draw), and remote desktop stops if it runs
+    (session/remote-ctl.sh: krdp, Xvnc, FreeRDP). Windows stay open. "close" closes the desktop
+    (desktops.sh stop): its windows close, and resuming starts it again (about 12 seconds), in
+    its start profile if it has one.
+  - In the relay: the 3D mouse lets go and the mouse is a plain mouse for SteamVR, typing goes to
+    Steam, and mapped buttons and key combinations do nothing but PAUSED_ACTIONS and commands.
+Resuming starts again only what pausing stopped. The pointer helper and ft-powerd keep running:
+they cost little, and the helper is what says a VR game started.
+
+Toggled by:
+  - The pause_toggle action: a mouse button, a key combination, or a controller button outside
+    games (the pointer helper only reads those outside games).
+  - A controller gesture, by default both thumbsticks clicked together twice ("pause_gesture").
+    It's read from vrserver's web socket (vrws.py), which works in games and takes nothing from
+    them, so the game sees the clicks too.
+  - input/ft-pause on|off|toggle|status, which uses the relay's control socket
+    ("pause on|off|toggle|?").
+  - VR games, with "pause_auto" (on by default): the pointer helper says "vrgame 1|0" as a SteamVR
+    scene app starts and ends (and every 5 seconds). A game starting pauses; a pause that starts
+    while a game runs ends RESUME_DELAY after the game does, unless another game starts first.
+    Toggled back on during a game, Frametop stays on until that game ends. A pause that starts
+    outside a game lasts until it's toggled off.
+
+Settings in ~/.config/frametop-input.json (Frametop Input Settings, Game optimization page): "pause_auto"
+(bool), "pause_gesture" ({"buttons": one or two controller buttons, "presses": 1 or 2}, or null
+for none; one button always takes two presses), "pause_desktop" ("hide" or "close"), and
+"pause_sound" (a sound on pause and resume; bool). The state outlives a relay restart in
+STATE_PATH.
+"""
+import json
+import os
+import queue
+import socket
+import subprocess
+import threading
+import time
+
+import vrws
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+DESKTOPS = os.path.join(REPO, "desktops.sh")
+REMOTE_CTL = os.path.join(REPO, "session", "remote-ctl.sh")
+RELAY = "\0frametop_relay"
+SCREENS = "\0ft_screens"
+STATE_PATH = f"/run/user/{os.getuid()}/frametop-pause.json"
+# User services that pausing stops, if they run.
+UNITS = ("frametop-gaze.service", "frametop-hands.service", "frametop-camd.service")
+# What still works while paused, besides commands ("command:CMD"): the toggle itself, and Steam's menu.
+PAUSED_ACTIONS = ("pause_toggle", "steam_menu")
+DEFAULT_GESTURE = {"buttons": ["left/thumbstick", "right/thumbstick"], "presses": 2}
+DESKTOP_MODES = ("hide", "close")
+SOUNDS = {True: "/usr/share/sounds/ocean/stereo/device-removed.oga",
+          False: "/usr/share/sounds/ocean/stereo/device-added.oga"}
+RESUME_DELAY = 5.0  # after a VR game ends: loading the next one can end one scene app and start another
+GAME_STALE = 12.0  # the helper repeats "vrgame" every 5 s; silence this long means no game (or no SteamVR)
+CHORD_WINDOW = 0.3  # a two-button gesture's buttons go down within this of each other
+DOUBLE_WINDOW = 0.7  # a double press's second press comes within this of the first
+LOOKUP_EVERY = 30.0  # the controllers are looked up again this often (and sooner when they may have changed)
+LOOKUP_GAP = 3.0  # but not more often than this for a message from a device it doesn't know
+
+
+def device_of(text):
+    """A web socket message's "device", without parsing all of it (there are about 160 a second)."""
+    at = text.find('"device"')
+    if at < 0:
+        return None
+    start = text.find('"', text.find(":", at) + 1)
+    end = text.find('"', start + 1)
+    return text[start + 1:end] if 0 <= start < end else None
+
+
+def gesture_of(rules, buttons):
+    """The rules' pause gesture as (buttons, presses), or None for none. `buttons` are the
+    controller buttons there are (the relay's VR_BUTTONS)."""
+    spec = rules.get("pause_gesture", DEFAULT_GESTURE)
+    if not isinstance(spec, dict) or not isinstance(spec.get("buttons"), list):
+        return None
+    chosen = tuple(dict.fromkeys(b for b in spec["buttons"] if b in buttons))[:2]
+    if not chosen:
+        return None
+    return chosen, 1 if len(chosen) == 2 and spec.get("presses") == 1 else 2
+
+
+def settings_of(rules, buttons):
+    return {"auto": rules.get("pause_auto", True) is not False,
+            "gesture": gesture_of(rules, buttons),
+            "desktop": rules.get("pause_desktop") if rules.get("pause_desktop") in DESKTOP_MODES else "hide",
+            "sound": rules.get("pause_sound", True) is not False}
+
+
+class Gesture:
+    """The pause gesture, from button presses and releases. A press of a two-button gesture is
+    both buttons going down within CHORD_WINDOW of each other, so a button held for a while (to
+    sprint, say) and the other clicked doesn't count. Two presses within DOUBLE_WINDOW make a
+    double press."""
+
+    def __init__(self, buttons, presses):
+        self.buttons, self.presses = tuple(buttons), presses
+        self.down = {}  # button -> when it went down
+        self.first = None  # when the first press of a double press was
+        self.fired = None
+
+    def reset(self):
+        self.down.clear()
+        self.first = None
+
+    def feed(self, button, pressed, now):
+        """A button's state; True when it completes the gesture."""
+        if button not in self.buttons:
+            return False
+        if not pressed:
+            self.down.pop(button, None)
+            return False
+        if button in self.down:
+            return False  # still down
+        self.down[button] = now
+        if len(self.down) < len(self.buttons) or now - min(self.down.values()) > CHORD_WINDOW:
+            return False
+        if self.presses == 2 and (self.first is None or now - self.first > DOUBLE_WINDOW):
+            self.first = now
+            return False
+        self.first = None
+        if self.fired is not None and now - self.fired < 1.0:
+            return False  # a triple press is still one
+        self.fired = now
+        return True
+
+
+class ControllerWatch(threading.Thread):
+    """Follows the Frame controllers on vrserver's web socket for the pause gesture, and sends
+    the relay "pause toggle gesture" when it's made. Reconnects every 5 seconds while SteamVR
+    is away. With no gesture set, it doesn't connect."""
+
+    def __init__(self, log):
+        super().__init__(name="pause-gesture", daemon=True)
+        self.log = log
+        self.lock = threading.Lock()
+        self.gesture = None
+        self.changed = threading.Event()
+        self.connected = False
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+        self.lookup_at = None  # look the controllers up again then (see recheck)
+
+    def recheck(self, delay=1.5):
+        """The controllers' root paths may change soon: the 3D mouse's virtual controller is taking
+        or giving back its hand role. SteamVR takes a moment to hand the role over."""
+        self.lookup_at = time.monotonic() + delay
+
+    def set_gesture(self, spec):
+        """(buttons, presses), or None for no gesture."""
+        with self.lock:
+            g = self.gesture
+            if spec is None:
+                self.gesture = None
+            elif g is None or (g.buttons, g.presses) != spec:
+                self.gesture = Gesture(*spec)
+        self.changed.set()
+
+    def current(self):
+        with self.lock:
+            return self.gesture
+
+    def run(self):
+        while True:
+            if self.current() is None:
+                if self.connected:
+                    self.log("pause gesture: none set, stopped reading the controllers")
+                self.connected = False
+                self.changed.wait()
+                self.changed.clear()
+                continue
+            try:
+                self.session()
+            except (OSError, ValueError) as e:  # urllib's errors are OSErrors
+                if self.connected:
+                    self.log(f"pause gesture: SteamVR's web socket went away ({e}); retrying")
+                self.connected = False
+            self.changed.wait(5)
+            self.changed.clear()
+
+    def session(self):
+        ws = vrws.VrSocket()
+        try:
+            ws.open(f"frametop_pause_{os.getpid()}")
+            sides = {}  # root path -> side
+            next_poll = 0.0
+            last_poll = 0.0
+            while True:
+                g = self.current()
+                if g is None:
+                    return
+                now = time.monotonic()
+                lookup_at = self.lookup_at
+                if lookup_at is not None and now >= lookup_at:
+                    self.lookup_at = None
+                    next_poll = now
+                if now >= next_poll:
+                    # A controller connects later, or changes its root path when the 3D mouse's
+                    # virtual controller takes or gives back its hand role. Each lookup is an HTTP
+                    # request to vrserver, so it's at connect, every LOOKUP_EVERY, when a message
+                    # comes from a device not on the list, and when the relay says (recheck).
+                    next_poll, last_poll = now + LOOKUP_EVERY, now
+                    found = vrws.controllers()
+                    if found != sides:
+                        for path in sides.keys() - found.keys():
+                            ws.subscribe(path, on=False)
+                        for path in found.keys() - sides.keys():
+                            ws.subscribe(path)
+                        sides = found
+                        g.reset()
+                    if not self.connected:
+                        self.connected = True
+                        self.log(f"pause gesture: following {' and '.join(sorted(sides.values())) or 'no'} "
+                                 f"controller{'s' if len(sides) != 1 else ''} on SteamVR's web socket")
+                wanted = {b: (b.split("/", 1)[0], f"/input/{b.split('/', 1)[1]}/click") for b in g.buttons}
+                for text in ws.messages(timeout=1.0):
+                    device = device_of(text)
+                    if device and device not in sides:
+                        next_poll = min(next_poll, last_poll + LOOKUP_GAP)  # its root path changed, maybe
+                    # About 160 messages a second, nearly all capacitive sensing: parse only the
+                    # few that carry one of the gesture's buttons.
+                    if not any(c in text for _, c in wanted.values()):
+                        continue
+                    try:
+                        msg = json.loads(text)
+                    except ValueError:
+                        continue
+                    side = sides.get(msg.get("device"))
+                    components = msg.get("components") or {}
+                    for button, (s, component) in wanted.items():
+                        if s == side and component in components:
+                            if g.feed(button, bool(components[component]), time.monotonic()):
+                                self.log("pause gesture")
+                                try:
+                                    self.sock.sendto(b"pause toggle gesture", RELAY)
+                                except OSError:
+                                    pass
+        finally:
+            ws.close()
+
+
+def run(cmd, timeout=30):
+    """A command's exit status (None if it couldn't run), its output going nowhere."""
+    try:
+        return subprocess.run(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
+                              stderr=subprocess.DEVNULL, timeout=timeout).returncode
+    except (OSError, subprocess.TimeoutExpired):
+        return None
+
+
+def output(cmd, timeout=10):
+    try:
+        return subprocess.run(cmd, stdin=subprocess.DEVNULL, capture_output=True, text=True,
+                              timeout=timeout).stdout
+    except (OSError, subprocess.TimeoutExpired):
+        return ""
+
+
+def active(unit):
+    return run(["systemctl", "--user", "-q", "is-active", unit], timeout=10) == 0
+
+
+def desktop_running():
+    return run(["pgrep", "-x", "ft-screens"], timeout=5) == 0
+
+
+class Worker(threading.Thread):
+    """Does the slow part of pausing and resuming, in order, one job at a time: stopping and
+    starting services, remote desktop, and closing or starting the desktop. `stopped` is what
+    the last pause stopped, for the resume to start again."""
+
+    def __init__(self, log, save):
+        super().__init__(name="pause-worker", daemon=True)
+        self.log, self.save = log, save
+        self.jobs = queue.Queue()
+        self.lock = threading.Lock()
+        self.stopped = {"units": [], "remote": False, "desktop": None}
+        self.busy = False
+
+    def submit(self, job, *args):
+        self.busy = True
+        self.jobs.put((job, args))
+
+    def run(self):
+        while True:
+            job, args = self.jobs.get()
+            try:
+                job(*args)
+            except Exception as e:  # one failure mustn't stop the next pause or resume
+                self.log(f"pause: {job.__name__} failed: {e!r}")
+            self.busy = not self.jobs.empty()
+            self.save()
+
+    def snapshot(self):
+        with self.lock:
+            return {"units": list(self.stopped["units"]), "remote": self.stopped["remote"],
+                    "desktop": self.stopped["desktop"]}
+
+    def restore(self, stopped):
+        with self.lock:
+            self.stopped = {"units": [u for u in stopped.get("units", []) if u in UNITS],
+                            "remote": bool(stopped.get("remote")),
+                            "desktop": stopped.get("desktop") if stopped.get("desktop") in ("hidden", "closed") else None}
+
+    def stop_units(self):
+        """Stop the UNITS that run, adding them to what the resume starts again."""
+        units = [u for u in UNITS if active(u)]
+        if units:
+            run(["systemctl", "--user", "stop", *units], timeout=30)
+            with self.lock:
+                self.stopped["units"] += [u for u in units if u not in self.stopped["units"]]
+            self.log(f"pause: stopped {', '.join(units)}")
+
+    def pause(self, desktop):
+        if desktop == "close":
+            if desktop_running():
+                run([DESKTOPS, "stop"], timeout=40)  # remote desktop goes with it
+                with self.lock:
+                    self.stopped["desktop"] = "closed"
+                self.log("pause: desktop closed")
+        else:
+            if "running=1" in output([REMOTE_CTL, "status"]):
+                run([REMOTE_CTL, "stop"])
+                with self.lock:
+                    self.stopped["remote"] = True
+                self.log("pause: remote desktop stopped")
+            with self.lock:
+                self.stopped["desktop"] = "hidden"  # ft-screens was told already (or isn't running)
+        self.stop_units()
+
+    def enforce(self):
+        """While paused: SteamVR restarted, and started the gaze service with it."""
+        self.stop_units()
+
+    def resume(self):
+        stopped = self.snapshot()
+        units = stopped["units"]
+        if units:
+            # They need SteamVR (Requisite=steamvr.service): without it they start with it next time.
+            if run(["systemctl", "--user", "start", *units], timeout=30) == 0:
+                self.log(f"resume: started {', '.join(units)}")
+            else:
+                self.log(f"resume: couldn't start {', '.join(units)} (SteamVR not running?)")
+        if stopped["remote"]:
+            # In a scope of its own, not this service's, so a relay restart doesn't end it.
+            run(["systemd-run", "--user", "--scope", "--collect", "--quiet", REMOTE_CTL, "start"])
+            self.log("resume: remote desktop started")
+        if stopped["desktop"] == "closed":
+            if run([DESKTOPS, "start"], timeout=60) == 0:
+                self.log("resume: desktop started")
+            else:
+                self.log("resume: the desktop didn't start (see /tmp/frametop-session.log)")
+        with self.lock:
+            self.stopped = {"units": [], "remote": False, "desktop": None}
+
+
+class GamePause:
+    """The pause state, its automatic side, and the relay's way in. `on_change(paused)` is
+    called (from the relay's thread) when the state flips, for the relay's own part."""
+
+    def __init__(self, log, on_change, buttons):
+        self.log, self.on_change, self.buttons = log, on_change, buttons
+        self.lock = threading.Lock()  # the state file, written from both threads
+        self.paused = False
+        self.reason = ""  # what paused it: gesture, button, key, controller, command, game
+        self.since = 0.0  # wall clock
+        self.ends_with_game = False  # resume when the VR game ends
+        self.game = False
+        self.game_seen = 0.0  # last "vrgame" from the helper
+        self.resume_at = None
+        self.settings = settings_of({}, buttons)
+        self.screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
+        self.worker = Worker(log, self.save)
+        self.watch = ControllerWatch(log)
+        self.worker.start()
+        self.watch.start()
+        self.load()
+
+    # --- state file ---
+    def save(self):
+        with self.lock:
+            state = {"paused": self.paused, "reason": self.reason, "since": self.since,
+                     "ends_with_game": self.ends_with_game, "stopped": self.worker.snapshot()}
+            try:
+                tmp = STATE_PATH + ".tmp"
+                with open(tmp, "w") as f:
+                    json.dump(state, f)
+                os.replace(tmp, STATE_PATH)
+            except OSError as e:
+                self.log(f"pause: can't save {STATE_PATH}: {e}")
+
+    def load(self):
+        """A relay restarted while paused stays paused (the relay's part is applied by its first
+        configure)."""
+        try:
+            with open(STATE_PATH) as f:
+                state = json.load(f)
+        except (OSError, ValueError):
+            return
+        if not isinstance(state, dict) or not state.get("paused"):
+            return
+        self.paused = True
+        self.reason = str(state.get("reason", ""))
+        self.since = float(state.get("since", 0) or 0)
+        self.ends_with_game = bool(state.get("ends_with_game"))
+        self.worker.restore(state.get("stopped") or {})
+        self.log(f"paused since before this relay started ({self.reason})")
+        self.worker.submit(self.worker.enforce)
+
+    # --- settings ---
+    def configure(self, rules):
+        self.settings = settings_of(rules, self.buttons)
+        self.watch.set_gesture(self.settings["gesture"])
+        if not self.settings["auto"]:
+            self.ends_with_game, self.resume_at = False, None
+
+    # --- pausing and resuming ---
+    def sound(self, paused):
+        if self.settings["sound"] and os.path.exists(SOUNDS[paused]):
+            try:
+                subprocess.Popen(["pw-play", SOUNDS[paused]], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
+                                 stderr=subprocess.DEVNULL, start_new_session=True)
+            except OSError:
+                pass
+
+    def tell_screens(self, paused):
+        try:
+            self.screens.sendto(b"pause on" if paused else b"pause off", SCREENS)
+        except OSError:
+            pass  # the desktop isn't running
+
+    def pause(self, reason, now):
+        if self.paused:
+            return
+        self.paused, self.reason, self.since = True, reason, time.time()
+        self.ends_with_game = self.game and self.settings["auto"]
+        self.resume_at = None
+        self.log(f"Frametop paused ({reason}{', until the game ends' if self.ends_with_game else ''})")
+        self.sound(True)
+        self.tell_screens(True)  # at once, even when the desktop is closing: that takes a while
+        self.on_change(True)
+        self.worker.submit(self.worker.pause, self.settings["desktop"])
+        self.save()
+
+    def resume(self, reason, now):
+        if not self.paused:
+            return
+        self.paused, self.resume_at, self.ends_with_game = False, None, False
+        self.log(f"Frametop resumed ({reason})")
+        self.sound(False)
+        self.tell_screens(False)
+        self.on_change(False)
+        self.worker.submit(self.worker.resume)
+        self.save()
+
+    def toggle(self, reason, now):
+        """Pausing only happens by itself as a game starts, so a game resumed during stays on."""
+        (self.resume if self.paused else self.pause)(reason, now)
+
+    def set(self, on, reason, now):
+        if on != self.paused:
+            self.toggle(reason, now)
+
+    # --- VR games ---
+    def game_state(self, running, now):
+        """From the pointer helper: a VR game runs or not (on a change and every 5 s)."""
+        self.game_seen = now
+        if running == self.game:
+            return
+        self.game = running
+        self.log(f"a VR game {'started' if running else 'ended'}")
+        if running:
+            self.resume_at = None  # the next game came before the resume
+            if self.settings["auto"]:
+                self.pause("game", now)
+        elif self.paused and self.ends_with_game:
+            self.resume_at = now + RESUME_DELAY
+
+    def tick(self, now):
+        if self.game and now - self.game_seen > GAME_STALE:
+            self.game_state(False, now)  # the helper went quiet: SteamVR (or the helper) stopped
+        if self.resume_at is not None and now >= self.resume_at:
+            self.resume("the game ended", now)
+
+    def timeout(self, now):
+        """How long the relay may wait before tick() has something to do."""
+        if self.resume_at is not None:
+            return max(0.05, self.resume_at - now)
+        return 1.0 if self.game else 3600.0
+
+    def controllers_changed(self):
+        """The 3D mouse took or gave back its hand role, which changes a controller's root path."""
+        self.watch.recheck()
+
+    def helper_started(self):
+        """The pointer helper (re)started, so SteamVR did too, maybe with the gaze service."""
+        if self.paused:
+            self.worker.submit(self.worker.enforce)
+
+    def status(self):
+        return {"t": "pause", "paused": self.paused, "reason": self.reason, "since": self.since,
+                "ends_with_game": self.ends_with_game, "game": self.game,
+                "busy": self.worker.busy, "stopped": self.worker.snapshot(),
+                "gesture_reader": self.watch.connected, "auto": self.settings["auto"],
+                "desktop": self.settings["desktop"]}

input/input-relay.py

@@ -14,8 +14,7 @@ keyboard node for its extra buttons). Roles, from ~/.config/frametop-input.json
 (written by the Frametop Input Settings app):
   pointer      grabbed; drives the universal 3D mouse (default for devices with a mouse node)
   passthrough  keys go to the desktop; grabbed only while typing goes there, otherwise only observed,
-               e.g. for the Meta dashboard shortcut (default for keyboards; the shortcut is off
-               unless META_DASHBOARD=1 is in ~/.config/frametop.conf)
+               for the key combinations below (default for keyboards)
   ignore       not grabbed, only observed for identification in the settings app
 Buttons and keys of pointer devices go through a per-device map to actions
 (left, right, middle, back, scroll_up, scroll_down, dashboard, recenter,
@@ -32,19 +31,27 @@ layout_reset = put the desktop screens back in their saved layout, screens_toggl
 keyboard_toggle = open or close Frametop's keyboard, float_toggle = float the desktop window under the
 pointer (else the active one) in VR, or put it back if it floats, dock_all = put every floating
 window back (both to ft-floatd, @frametop_float), profile:NAME = switch to that profile (ft-layout
-use NAME: its screens and apps; docs/profiles.md), key = pass through as a key, none).
+use NAME: its screens and apps; docs/profiles.md), steam_menu = open the SteamVR dashboard on
+Steam's menu, or close the dashboard (steam/ft-steam menu, through Steam's UI), pause_toggle =
+pause Frametop for a VR game, or resume it (game_pause.py), command:CMD = run
+CMD with sh -c (on the host, as this service: its environment, output to its log, and
+COMMAND_PATH, so ft-layout, ft-float and ft-steam need no path), key = pass through as a key, none).
 
 Frame controller buttons can be mapped too ("controller_buttons": {"right/a": action} in the
 rules file; any action but key and the gaze ones, GAZE_ACTIONS: gaze mode is a mouse feature,
 docs/gaze-controllers.md). So can key combinations on any keyboard ("key_bindings":
 {"29+56+34": action}, evdev codes joined by "+", modifiers first and left-hand codes for
 either side, here Ctrl+Alt+G): the combination does the action, and its last key isn't typed.
-A rules file without "key_bindings" gets DEFAULT_KEY_BINDINGS (Meta+J: gaze_left, Meta+K:
-gaze_right, Meta+Shift+F: float_toggle); one with its own, even an empty one, doesn't. The float
-actions work without pointer mode too. A combination with Meta also sends the desktop an F24 press
-and Meta's release right away: so letting go of Meta doesn't open Plasma's launcher, and a gaze
-click isn't Meta+click (KWin's window move and resize). Another key while Meta is still held gives
-the desktop Meta back. The controllers aren't input devices here, only SteamVR sees
+A modifier on its own ("125": Meta) is a tap: pressed and released with no other key, mouse
+button, or scroll in between; the desktop gets an F24 press before its release, so Plasma's
+launcher doesn't open on a Meta tap that's bound. A rules file without "key_bindings" gets
+DEFAULT_KEY_BINDINGS (Meta tap: steam_menu, Meta+J: gaze_left, Meta+K: gaze_right, Meta+Shift+F:
+float_toggle); one with its own, even an empty one, doesn't. The float, profile, Steam menu,
+pause and command actions work without pointer mode too. A combination with Meta also sends the desktop an
+F24 press and Meta's release right away: so letting go of Meta doesn't open Plasma's launcher, and
+a gaze click isn't Meta+click (KWin's window move and resize). Another key while Meta is still
+held gives the desktop Meta back. While typing goes to Steam, keyboards aren't grabbed, so Steam
+or the game sees a combination's keys too. The controllers aren't input devices here, only SteamVR sees
 them, so the pointer helper reads them with SteamVR input and sends "vrbtn <button> 1|0".
 It only takes the buttons the relay tells it to ("vrbind <button>..." to @ft_pointer_helper,
 sent on start, reload, and when the helper says "vrhello"), and only while no game runs,
@@ -86,6 +93,12 @@ keyboards) get their volume entries remapped to unused stand-in codes, so their
 other keys keep working for SteamVR; a device without a keymap that has only volume
 keys (the headset's pmic_resin) is grabbed. The keymaps go back when the relay exits.
 
+Frametop can pause for VR games (game_pause.py: by hand, with a controller gesture, or by itself
+while a game runs). Paused, it stops the services that cost the game CPU and GPU, and the relay
+plays a plain one: the pointer devices feed the virtual mouse and keyboard as without pointer
+mode, typing goes to Steam, and mapped buttons and key combinations do only pause_toggle,
+steam_menu and command:CMD (game_pause.PAUSED_ACTIONS).
+
 Pointer mode (POINTER=1 in ~/.config/frametop.conf) sends pointer devices to
 the ft-pointer helper (pointer/helper), which drives the ft_pointer
 SteamVR driver. With POINTER=0, pointer devices go to the virtual mouse and
@@ -98,7 +111,10 @@ Control socket (abstract datagram @frametop_relay, JSON replies to the sender):
   vrcapture <s>     take every controller button for s seconds (0: stop), so the settings
                     app can capture one; watchers see them as events with id frame_controller
   vrbtn, vrhello, gazeawake   from the pointer helper (above)
+  vrgame 1|0        from the pointer helper: a VR game runs (on a change and every 5 s), for pausing
   textfield 1|0     from the desktop's input method (above)
+  pause on|off|toggle [reason]   pause Frametop or resume it (input/ft-pause; the gesture reader)
+  pause ?           the pause state, as {"t": "pause", ...}
 
 Runs on the Frame host as a user service (frametop-input-relay.service). The
 virtual devices are parked in systemd's file descriptor store, so a relay
@@ -112,6 +128,7 @@ kernel's evdev and uinput interfaces.
 """
 import array
 import atexit
+import collections
 import errno
 import fcntl
 import json
@@ -124,12 +141,15 @@ import subprocess
 import sys
 import time
 
+import game_pause
+
 # Linux input constants (include/uapi/linux/input-event-codes.h, input.h, uinput.h).
 EV_SYN, EV_KEY, EV_REL, EV_MSC = 0x00, 0x01, 0x02, 0x04
 SYN_REPORT = 0
 BTN_MISC, KEY_MAX = 0x100, 0x2FF
 KEY_A = 30
 REL_X, REL_Y, REL_WHEEL, REL_MAX = 0x00, 0x01, 0x08, 0x0F
+SCROLLS = {0x06, REL_WHEEL, 0x0B, 0x0C}  # REL_HWHEEL, REL_WHEEL and their _HI_RES
 BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_SIDE, BTN_EXTRA = 0x110, 0x111, 0x112, 0x113, 0x114
 KEY_LEFTMETA, KEY_RIGHTMETA = 125, 126
 KEY_VOLUMEDOWN, KEY_VOLUMEUP = 114, 115
@@ -189,12 +209,14 @@ RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
 ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
            "pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right",
            "gaze_quickcal", "sens_up", "sens_down", "layout_reset", "screens_toggle", "keyboard_toggle", "float_toggle",
-           "dock_all", "key", "none")
+           "dock_all", "steam_menu", "pause_toggle", "key", "none")
 # Gaze mode is a mouse feature: these never come from a controller button (docs/gaze-controllers.md).
 GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
-# Key combinations a rules file without "key_bindings" gets: Meta+J and Meta+K click at the gaze
-# (free on the Frametop desktop, and apps don't use Meta), Meta+Shift+F floats a window.
-DEFAULT_KEY_BINDINGS = {"125+36": "gaze_left", "125+37": "gaze_right", "42+125+33": "float_toggle"}
+# Key combinations a rules file without "key_bindings" gets: a Meta tap opens Steam's menu, Meta+J
+# and Meta+K click at the gaze (free on the Frametop desktop, and apps don't use Meta),
+# Meta+Shift+F floats a window.
+DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
+                        "42+125+33": "float_toggle"}
 KEY_F24 = 194  # sent to the desktop with a Meta combination (see the top)
 # Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
 MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
@@ -211,16 +233,33 @@ FLOAT = "\0frametop_float"  # ft-floatd, floating windows in the Frametop deskto
 # Actions for ft-floatd ("float_toggle", "dock_all"): they don't need pointer mode.
 FLOAT_ACTIONS = {"float_toggle": b"float pointer", "dock_all": b"dock all"}
 PROFILE = "profile:"  # "profile:NAME": switch to that profile (doesn't need pointer mode either)
+COMMAND = "command:"  # "command:CMD": run CMD (nor does this)
 
 
 def known_action(a):
-    return a in ACTIONS or (isinstance(a, str) and a.startswith(PROFILE) and len(a) > len(PROFILE))
+    return a in ACTIONS or (isinstance(a, str) and any(a.startswith(p) and a[len(p):].strip()
+                                                       for p in (PROFILE, COMMAND)))
 
 
 def needs_pointer(a):
-    return a not in FLOAT_ACTIONS and not a.startswith(PROFILE)
+    return a not in FLOAT_ACTIONS and a not in ("steam_menu", "pause_toggle") and not a.startswith((PROFILE, COMMAND))
+
+
+def works_paused(a):
+    """An action that still does something while Frametop is paused (game_pause.py)."""
+    return a in game_pause.PAUSED_ACTIONS or a.startswith(COMMAND)
+
+
+# What pressed pause_toggle, for the log (do_action's source).
+PAUSE_SOURCES = {"mouse": "mouse button", "keyboard": "key combination", "left": "controller button",
+                 "right": "controller button"}
 KEYS = "\0frametop_keys"  # keys of keyboards grabbed for the desktop, for other readers
-FT_LAYOUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "layout", "ft-layout")
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+FT_LAYOUT = os.path.join(REPO, "layout", "ft-layout")
+FT_STEAM = os.path.join(REPO, "steam", "ft-steam")
+# Commands ("command:CMD") find Frametop's own tools (ft-layout, ft-float, ft-steam) on their PATH.
+COMMAND_PATH = ":".join([os.path.join(REPO, d) for d in ("layout", "float", "steam")]
+                        + [os.environ.get("PATH", "/usr/local/bin:/usr/bin")])
 DEFAULT_BUTTONS = {BTN_LEFT: "left", BTN_RIGHT: "right", BTN_MIDDLE: "middle",
                    BTN_SIDE: "back", BTN_EXTRA: "back"}
 
@@ -421,14 +460,22 @@ class Pointer:
     CLAIM_PULSE = 0.06  # seconds the claim button (switchlaserhand, no click) is held
     RESUME_PAUSE = 1.5  # mouse idle this long, then moving again, re-claims the laser
     WAKE_WINDOW = 1.0  # seconds in which WAKE_COUNTS of motion must add up
+    QUEUE_MAX = 512  # commands kept while the helper is behind (see send)
+    MOVE_EVERY = 0.004  # mouse motion goes to the helper at most this often (see flush)
 
     def __init__(self, sensitivity, idle, wake_counts=40):
-        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+        # Never blocks (see send): a stalled helper must not stall the keyboard, volume keys and pausing.
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
+        # Commands the helper's full socket didn't take yet, in order: text, or [dyaw, dpitch] for
+        # mouse moves, which add up into one while they wait.
+        self.queue = collections.deque()
+        self.behind_logged = -60.0
         self.sensitivity = sensitivity  # degrees per mouse count
         self.idle = idle
         self.active = False
         self.last_used = 0.0
         self.dx = self.dy = 0
+        self.move_at = 0.0  # motion last went to the helper then
         self.scroll_until = None
         self.claim_at = None  # when to press the claim button
         self.claim_release = None
@@ -441,11 +488,59 @@ class Pointer:
         self.pending_since = 0.0
         self.gaze_awake_until = 0.0  # the helper's gaze mode keeps the pointer until then
 
-    def send(self, command):
-        try:
-            self.sock.sendto(command.encode(), HELPER)
-        except OSError:
-            pass  # helper not running (SteamVR not running)
+    def send(self, command, droppable=False):
+        """To the helper, in order, without blocking. While the helper doesn't keep up (place and
+        grabprobe hold it for seconds, and ft-gazed's 90 Hz gaze fills its socket meanwhile), commands
+        wait in the queue and go out from tick(). A droppable one (a scroll notch: scrolling seconds
+        late is no use) is dropped instead; presses, releases and the rest are kept, so no button
+        stays down. Moves add up (_move)."""
+        if not self.queue:
+            try:
+                self.sock.sendto(command.encode(), HELPER)
+                return
+            except BlockingIOError:
+                pass
+            except OSError:
+                return  # helper not running (SteamVR not running)
+        if not droppable:
+            self._queue(command)
+
+    def _move(self, dyaw, dpitch):
+        if self.queue and isinstance(self.queue[-1], list):
+            self.queue[-1][0] += dyaw
+            self.queue[-1][1] += dpitch
+            return
+        if not self.queue:
+            try:
+                self.sock.sendto(f"move {dyaw:.4f} {dpitch:.4f}".encode(), HELPER)
+                return
+            except BlockingIOError:
+                pass
+            except OSError:
+                return
+        self._queue([dyaw, dpitch])
+
+    def _queue(self, item):
+        if not self.queue and time.monotonic() - self.behind_logged > 60:
+            self.behind_logged = time.monotonic()
+            log("pointer helper is behind: holding its commands (logged once a minute)")
+        self.queue.append(item)
+        if len(self.queue) > self.QUEUE_MAX:
+            self.queue.popleft()  # stalled for long: the oldest goes
+
+    def drain(self):
+        """Send what waits, in order, as far as the helper takes it."""
+        while self.queue:
+            item = self.queue[0]
+            text = f"move {item[0]:.4f} {item[1]:.4f}" if isinstance(item, list) else item
+            try:
+                self.sock.sendto(text.encode(), HELPER)
+            except BlockingIOError:
+                return
+            except OSError:
+                self.queue.clear()  # the helper went away: nothing to deliver to
+                return
+            self.queue.popleft()
 
     def wake(self, now):
         if not self.active:
@@ -473,7 +568,7 @@ class Pointer:
         elif code == REL_Y:
             self.dy += value
         elif code == REL_WHEEL and value:
-            self.send(f"scroll 0 {1 if value > 0 else -1}")
+            self.send(f"scroll 0 {1 if value > 0 else -1}", droppable=True)
             self.scroll_until = now + self.SCROLL_PULSE
 
     def action(self, name, value, now, source="mouse"):
@@ -505,7 +600,7 @@ class Pointer:
             return  # the rest act on press
         elif name in ("scroll_up", "scroll_down"):
             self.wake(now)
-            self.send(f"scroll 0 {1 if name == 'scroll_up' else -1}")
+            self.send(f"scroll 0 {1 if name == 'scroll_up' else -1}", droppable=True)
             self.scroll_until = now + self.SCROLL_PULSE
         elif name == "dashboard":
             self.dashboard(now)
@@ -544,11 +639,16 @@ class Pointer:
             self.sensitivity *= 1.25 if name == "sens_up" else 0.8
             log(f"sensitivity {self.sensitivity:.4f} deg/count")
 
-    def flush(self):
-        if self.dx or self.dy:
-            # Mouse right turns the ray right (negative yaw); mouse down tilts it down.
-            self.send(f"move {-self.dx * self.sensitivity:.4f} {-self.dy * self.sensitivity:.4f}")
-            self.dx = self.dy = 0
+    def flush(self, now=None):
+        """Send the motion so far. With now (a mouse's SYN_REPORT), only once MOVE_EVERY has passed
+        since the last: a 1000 Hz mouse sent the helper, which runs every 8 ms, 1000 datagrams a
+        second. tick() sends the rest when it's due; a button sends it first, so it lands there."""
+        if not (self.dx or self.dy) or (now is not None and now - self.move_at < self.MOVE_EVERY):
+            return
+        self.move_at = time.monotonic() if now is None else now
+        # Mouse right turns the ray right (negative yaw); mouse down tilts it down.
+        self._move(-self.dx * self.sensitivity, -self.dy * self.sensitivity)
+        self.dx = self.dy = 0
 
     def dashboard(self, now=None):
         """Toggle the SteamVR dashboard with the virtual controller's system button.
@@ -563,6 +663,8 @@ class Pointer:
         self.system_at = now + (0.4 if woke else 0.0)
 
     def tick(self, now):
+        self.drain()
+        self.flush(now)
         if self.system_at is not None and now >= self.system_at:
             self.send("btn system 1")
             self.system_at = None
@@ -587,7 +689,26 @@ class Pointer:
 
     def timeout(self):
         pending = (self.scroll_until, self.claim_at, self.claim_release, self.system_at, self.system_release)
-        return 0.02 if any(t is not None for t in pending) else 0.5
+        wait = 0.02 if self.queue or any(t is not None for t in pending) else 0.5
+        if self.dx or self.dy:  # motion held back by flush
+            wait = max(0.0, min(wait, self.move_at + self.MOVE_EVERY - time.monotonic()))
+        return wait
+
+    def stand_down(self):
+        """Frametop is pausing: a pulse under way ends now, and the pointer lets go."""
+        if self.system_release is not None:
+            self.send("btn system 0")
+        if self.claim_release is not None:
+            self.send("btn a 0")
+        if self.scroll_until is not None:
+            self.send("scroll 0 0")
+        self.system_at = self.system_release = self.claim_at = self.claim_release = self.scroll_until = None
+        self.dx = self.dy = self.pending = 0
+        self.gaze_awake_until = 0.0
+        if self.active:
+            self.send("hide")
+            self.active = False
+            log("pointer off (paused)")
 
 
 class Node:
@@ -672,27 +793,46 @@ def main():
     # desktop_until: typing goes to the Frametop desktop until then (ft-screens says so
     # every second); typing_applied: the grabs match that as of the last apply_roles().
     # vr_capture_until: every controller button is taken until then (the settings app capturing one).
-    state = {"pointer": None, "rules": {}, "meta_dashboard": False, "share_keys": False,
-             "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0}
+    # pointer: the 3D mouse while it's in use, pointer_conf: the one the config asks for (they
+    # differ while Frametop is paused).
+    state = {"pointer": None, "pointer_conf": None, "rules": {}, "share_keys": False,
+             "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0, "vr_bind_retry": False,
+             "pointer_holding": False}
+
+    def pause_changed(paused):
+        """Frametop paused or resumed (game_pause.py): the relay's own part."""
+        p = state["pointer_conf"]
+        if paused:
+            if p:
+                p.stand_down()
+            state["pointer"] = None
+            state["desktop_until"] = 0.0  # typing goes to Steam
+        else:
+            state["pointer"] = p
+        apply_roles()
+        vr_bind(time.monotonic())
+
+    pause = game_pause.GamePause(log, pause_changed, VR_BUTTONS)
 
     def load_config():
         conf = read_config()
         state["rules"] = read_rules()
-        state["meta_dashboard"] = conf.get("META_DASHBOARD", "0") == "1"
         state["share_keys"] = conf.get("SHARE_KEYS", "0") == "1"
         if conf.get("POINTER", "0") == "1":
-            p = state["pointer"] or Pointer(0.03, 30)
+            p = state["pointer_conf"] or Pointer(0.03, 30)
             p.sensitivity = float(conf.get("POINTER_SENSITIVITY", "0.03"))
             p.idle = float(conf.get("POINTER_IDLE", "30"))
             p.wake_counts = int(conf.get("POINTER_WAKE_COUNTS", "40"))
-            state["pointer"] = p
+            state["pointer_conf"] = p
             log(f"pointer mode: {p.sensitivity} deg/count, idle {p.idle} s, wake {p.wake_counts} counts")
         else:
-            state["pointer"] = None
+            state["pointer_conf"] = None
             log("pointer mode off: pointer devices feed the virtual mouse and keyboard")
+        state["pointer"] = None if pause.paused else state["pointer_conf"]
+        pause.configure(state["rules"])
 
     load_config()
-    meta_down = False  # Meta pressed with no other key yet: a tap toggles the dashboard
+    tap = None  # the modifier (folded, MODIFIERS) pressed alone, with nothing since: its release is a tap
     screens_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
     last_typing = 0.0  # the helper was last told of a key then (see "typing" at the top)
 
@@ -704,11 +844,14 @@ def main():
             state["vr_capture_until"] = 0.0
             buttons = " ".join(b for b, a in state["rules"]["controller_buttons"].items()
                                if b in VR_BUTTONS and known_action(a) and a not in ("key", "none")
-                               and a not in GAZE_ACTIONS) or "-"
+                               and a not in GAZE_ACTIONS and (not pause.paused or works_paused(a))) or "-"
             if state["rules"].get("controller_in_games"):
                 buttons = "+games " + buttons
+        state["vr_bind_retry"] = False
         try:
             screens_sock.sendto(f"vrbind {buttons}".encode(), HELPER)
+        except BlockingIOError:
+            state["vr_bind_retry"] = True  # the helper is behind: again on the next loop
         except OSError:
             pass  # helper not running; it says vrhello when it starts
 
@@ -748,7 +891,13 @@ def main():
 
     def do_action(action, value, now, source="mouse"):
         """A mapped mouse or controller button, or key combination (pointer mode only, but
-        for FLOAT_ACTIONS and profiles)."""
+        for the actions needs_pointer() says don't)."""
+        if action == "pause_toggle":
+            if value == 1:
+                pause.toggle(PAUSE_SOURCES.get(source, source), now)
+            return
+        if pause.paused and not works_paused(action):
+            return
         if action == "keyboard_toggle":
             if value == 1 and vr_keyboard_mode() != "never":
                 vr_keyboard("toggle")
@@ -758,6 +907,17 @@ def main():
                 subprocess.Popen([FT_LAYOUT, "use", action[len(PROFILE):]], stdin=subprocess.DEVNULL,
                                  stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True)
                 log(action)
+        elif action == "steam_menu":
+            if value == 1:
+                # Talks to Steam's UI for a moment; its errors go to this service's log.
+                subprocess.Popen([FT_STEAM, "menu"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
+                                 start_new_session=True)
+                log(action)
+        elif action.startswith(COMMAND):
+            if value == 1:
+                subprocess.Popen(["sh", "-c", action[len(COMMAND):]], stdin=subprocess.DEVNULL,
+                                 env=dict(os.environ, PATH=COMMAND_PATH), start_new_session=True)
+                log(action)
         elif action in FLOAT_ACTIONS:
             if value == 1:
                 try:
@@ -776,14 +936,20 @@ def main():
     def key_binding(code, value, now):
         """A key from a keyboard: does it complete a key combination ("key_bindings")? True if
         it was taken for one (then it isn't typed)."""
-        nonlocal meta_down
+        nonlocal tap
         if code in MODIFIERS:
-            (held_modifiers.add if value else held_modifiers.discard)(MODIFIERS[code])
+            mod = MODIFIERS[code]
+            if value == 1:
+                tap = mod if not held_modifiers and not combos_down else None
+            (held_modifiers.add if value else held_modifiers.discard)(mod)
             if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
                 (held_meta.add if value else held_meta.discard)(code)
                 if value == 0 and code in meta_hidden:
                     meta_hidden.discard(code)
                     return True  # the desktop already had it come up (below)
+            if value == 0 and tap == mod:
+                tap = None
+                modifier_tap(mod, now)
             return False
         if value == 1 and code not in combos_down and meta_hidden:
             # Another key while Meta is still held after a combination: the desktop gets Meta
@@ -800,15 +966,14 @@ def main():
             return value == 2 and code in combos_down
         combo = "+".join(str(c) for c in sorted(held_modifiers) + [code])
         action = state["rules"]["key_bindings"].get(combo)
-        if not known_action(action) or action in ("key", "none"):
+        if not known_action(action) or action in ("key", "none") or (pause.paused and not works_paused(action)):
             return False
         combos_down[code] = action
         if held_meta - meta_hidden:
             # The desktop saw Meta go down. Another key in between keeps its release from
-            # opening Plasma's launcher (and Meta from toggling the dashboard here), and Meta
-            # comes up there now: KWin takes Meta with a mouse button for moving or resizing
-            # windows, which would swallow a gaze click. Its real release is dropped (above).
-            meta_down = False
+            # opening Plasma's launcher, and Meta comes up there now: KWin takes Meta with a
+            # mouse button for moving or resizing windows, which would swallow a gaze click.
+            # Its real release is dropped (above).
             to_screens(KEY_F24, 1)
             to_screens(KEY_F24, 0)
             for c in sorted(held_meta - meta_hidden):
@@ -819,6 +984,20 @@ def main():
         log(f"key combination {combo}: {action}")
         return True
 
+    def modifier_tap(mod, now):
+        """A modifier pressed and released alone: its binding, if it has one ("125": Meta tap)."""
+        action = state["rules"]["key_bindings"].get(str(mod))
+        if not known_action(action) or action in ("key", "none") or (pause.paused and not works_paused(action)):
+            return
+        # The desktop gets a key in between before the release goes there, so the tap isn't one
+        # there too: a Meta tap would open Plasma's launcher.
+        to_screens(KEY_F24, 1)
+        to_screens(KEY_F24, 0)
+        if state["pointer"] or not needs_pointer(action):
+            do_action(action, 1, now, "keyboard")
+            do_action(action, 0, now, "keyboard")
+        log(f"key combination {mod}: {action}")
+
 
     screens_down = set()  # keys the desktop was told went down and not yet up (see reconcile_desktop_keys)
 
@@ -1009,7 +1188,7 @@ def main():
             try:
                 if cmd == "keyboard":
                     # From ft-screens (unbound, no reply): where typing goes, repeated every second.
-                    desktop = len(words) > 1 and words[1] == "desktop"
+                    desktop = len(words) > 1 and words[1] == "desktop" and not pause.paused
                     state["desktop_until"] = now + 3.0 if desktop else 0.0
                     continue
                 if cmd == "vrbtn" and len(words) == 3 and words[1] in VR_BUTTONS and words[2] in ("0", "1"):
@@ -1017,6 +1196,20 @@ def main():
                     continue
                 if cmd == "vrhello":
                     vr_bind(now)
+                    pause.helper_started()
+                    continue
+                if cmd == "vrgame" and len(words) == 2:
+                    pause.game_state(words[1] == "1", now)
+                    continue
+                if cmd == "pause" and len(words) >= 2 and words[1] in ("on", "off", "toggle", "?"):
+                    # From input/ft-pause, the settings app, or the gesture reader (unbound, no reply).
+                    reason = words[2] if len(words) > 2 else "command"
+                    if words[1] == "toggle":
+                        pause.toggle(reason, now)
+                    elif words[1] != "?":
+                        pause.set(words[1] == "on", reason, now)
+                    if addr:
+                        reply(addr, pause.status())
                     continue
                 if cmd == "textfield" and len(words) == 2:
                     text_field(words[1] == "1")
@@ -1028,7 +1221,8 @@ def main():
                 if not addr:
                     continue  # unbound sender, nowhere to reply
                 if cmd == "devices":
-                    reply(addr, {"t": "devices", "pointer_mode": state["pointer"] is not None,
+                    reply(addr, {"t": "devices", "pointer_mode": state["pointer_conf"] is not None,
+                                 "paused": pause.paused,
                                  "actions": ACTIONS,
                                  "nodes": [n.describe() for n in nodes.values() if n.candidate]})
                 elif cmd == "watch":
@@ -1094,6 +1288,11 @@ def main():
                 # New here: a new device, or one that came back in the same place.
                 for old in [n for n in nodes.values() if n.path == path]:
                     drop(old, "replaced by a new device node")
+                # A new node is root's alone until udev gives it to the input group, a moment
+                # after it appears. Opened in that gap, it would fail and never be tried
+                # again: leave it for the next scan instead.
+                if not os.access(path, os.R_OK):
+                    continue
                 seen[path] = ino
                 node = probe(path)
                 if node and node.volume_keys and not take_volume(node):
@@ -1105,17 +1304,29 @@ def main():
             if added:
                 apply_roles()
 
+        # The 3D mouse connecting or letting go moves a hand role, so the pause gesture's reader
+        # looks the controllers up again (game_pause.py).
+        holding = bool(state["pointer_conf"] and state["pointer_conf"].active)
+        if holding != state["pointer_holding"]:
+            state["pointer_holding"] = holding
+            pause.controllers_changed()
         ready, _, _ = select.select(list(nodes) + [control], [], [],
-                                    volume.timeout(now, pointer.timeout() if pointer else 0.5))
+                                    min(volume.timeout(now, pointer.timeout() if pointer else 0.5), pause.timeout(now)))
         now = time.monotonic()
         if pointer:
             pointer.tick(now)
+        elif state["pointer_conf"]:
+            state["pointer_conf"].drain()  # paused: what waited still goes, in order
+        if state["vr_bind_retry"]:
+            vr_bind(now)
         volume.tick(now)
+        pause.tick(now)
         if state["vr_capture_until"] and now >= state["vr_capture_until"]:
             vr_bind(now)  # capture over: back to the mapped buttons
         if (now < state["desktop_until"]) != state["typing_applied"] or waiting:
             waiting = apply_roles()
         for fd in ready:
+            pointer = state["pointer"]  # (a pause or resume in this batch changes it)
             if fd is control:
                 handle_control(now)
                 continue
@@ -1136,44 +1347,41 @@ def main():
                 if etype in (EV_KEY, EV_REL):
                     broadcast(node, etype, VOLUME_ORIGINAL.get(code, code) if node.remapped else code,
                               value, now)
+                if (etype == EV_KEY and value == 1 and code not in MODIFIERS) or (etype == EV_REL and code in SCROLLS):
+                    tap = None  # a key, button or scroll in between: a modifier's release isn't a tap
                 if etype == EV_KEY and ((node.remapped and code in VOLUME_ORIGINAL)
                                         or (node.grabbed and code in VOLUME_STANDIN)):
                     volume.key(fd, code, value, now)
-                    if value == 1:
-                        meta_down = False  # Meta used as a modifier, not a tap
                     continue
                 if node.role == "volume":
                     continue
                 if node.role != "pointer":
-                    # Observed only, unless typing goes to the desktop. With META_DASHBOARD=1,
-                    # a Meta tap on any keyboard toggles the dashboard.
+                    # Observed only, unless typing goes to the desktop. Key combinations work on
+                    # any pass-through keyboard.
+                    if (node.role == "passthrough" and etype == EV_KEY and node.grabbed
+                            and code < BTN_MISC and value in (0, 1)):
+                        # Shared as pressed, key combinations included: the Meta release a
+                        # combination keeps from the desktop must still reach frame-voice, or
+                        # it waits for that release before typing anything.
+                        share_key(node, code, value)
+                        if value:
+                            node.held.add(code)
+                        else:
+                            node.held.discard(code)
                     if node.role == "passthrough" and etype == EV_KEY and code < BTN_MISC and key_binding(code, value, now):
                         continue
                     if node.role == "passthrough" and etype == EV_KEY:
                         if value == 1 and code < BTN_MISC and now - last_typing >= 0.25:
                             last_typing = now
-                            screens_sock.sendto(b"typing", HELPER)
+                            try:
+                                screens_sock.sendto(b"typing", HELPER)
+                            except OSError:
+                                pass  # helper not running (SteamVR not running)
                         to_screens(code, value)
-                        if node.grabbed and code < BTN_MISC and value in (0, 1):
-                            share_key(node, code, value)
-                            if value:
-                                node.held.add(code)
-                            else:
-                                node.held.discard(code)
-                    if (pointer and state["meta_dashboard"] and node.role == "passthrough"
-                            and etype == EV_KEY):
-                        if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
-                            if value == 1:
-                                meta_down = True
-                            elif value == 0 and meta_down:
-                                meta_down = False
-                                pointer.dashboard()
-                        elif value == 1:
-                            meta_down = False  # Meta used as a modifier, not a tap
                     continue
                 if etype == EV_KEY:
                     action = buttons.get(str(code), DEFAULT_BUTTONS.get(code, "key"))
-                    if pointer and action not in ("key", "none"):
+                    if (pointer or action == "pause_toggle") and action not in ("key", "none"):
                         do_action(action, value, now)
                         continue
                     if action == "none":
@@ -1192,7 +1400,7 @@ def main():
                         mouse.emit(etype, code, value)
                 elif etype == EV_SYN and code == SYN_REPORT:
                     if pointer:
-                        pointer.flush()
+                        pointer.flush(now)
                     mouse.sync()
                     keyboard.sync()
 

input/test/keys-test.py

@@ -0,0 +1,279 @@
+#!/usr/bin/env python3
+"""Offline test of the input relay's key combinations and modifier taps.
+
+Runs the relay's main() against a fake pass-through keyboard and a fake mouse (pipes), with
+every socket it sends to renamed, no uinput devices, no grabs, and ft-steam swapped for a
+logger. Pausing (game_pause.py) is a stub: no threads, state file, or services. Nothing reaches
+the live desktop, SteamVR, or the running relay, so it's safe next to them. Rules come from the
+test, not ~/.config/frametop-input.json.
+
+  input/test/keys-test.py [RELAY]   (default: input/input-relay.py next to this folder)
+"""
+import fcntl
+import importlib.util
+import os
+import shutil
+import socket
+import struct
+import sys
+import tempfile
+import threading
+import time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+relay_path = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "..", "input-relay.py")
+tag = f"ft_keys_test_{os.getpid()}"
+with open(relay_path) as f:
+    src = f.read().replace('"\\0frametop_relay"', f'"\\0{tag}_relay"')
+relay = importlib.util.module_from_spec(importlib.util.spec_from_loader("relay", loader=None))
+relay.__file__ = os.path.abspath(relay_path)
+sys.path.insert(0, os.path.dirname(os.path.abspath(relay_path)))  # its game_pause and vrws
+exec(compile(src, relay_path, "exec"), relay.__dict__)
+
+
+class StubPause:
+    """game_pause.GamePause without its threads, state file, or services."""
+    paused = False
+
+    def __init__(self, log, on_change, buttons):
+        self.on_change = on_change
+        stub["pause"] = self
+
+    def toggle(self, reason, now):
+        self.paused = not self.paused
+        self.on_change(self.paused)
+
+    def set(self, on, reason, now):
+        if on != self.paused:
+            self.toggle(reason, now)
+
+    def status(self):
+        return {"t": "pause", "paused": self.paused}
+
+    def timeout(self, now):
+        return 3600.0
+
+    def configure(self, *args):
+        pass
+
+    game_state = tick = helper_started = controllers_changed = configure
+
+
+stub = {}
+relay.game_pause.GamePause = StubPause
+
+for name in ("SCREENS", "HELPER", "FLOAT", "GAZED", "KEYS"):
+    setattr(relay, name, f"\0{tag}_{name.lower()}")
+OUT = tempfile.mkdtemp(prefix="ft-keys-test-")
+steam_log, cmd_log = f"{OUT}/steam.log", f"{OUT}/cmd.log"
+relay.FT_STEAM = f"{OUT}/ft-steam"
+with open(relay.FT_STEAM, "w") as f:
+    f.write(f'#!/bin/sh\necho "$@" >> {steam_log}\n')
+os.chmod(relay.FT_STEAM, 0o755)
+
+
+class NoDevice:
+    def __init__(self, *args, **kwargs):
+        pass
+
+    def emit(self, *args):
+        pass
+
+    def sync(self):
+        pass
+
+
+relay.Virtual = NoDevice
+relay.log = lambda *args: None  # the relay's own log
+bindings = {"now": None}  # the rules' key_bindings; None: the relay's defaults
+
+
+def read_rules(path=None):
+    rules = {"devices": {}, "buttons": {}, "controller_buttons": {}}
+    rules["key_bindings"] = dict(relay.DEFAULT_KEY_BINDINGS if bindings["now"] is None else bindings["now"])
+    return rules
+
+
+relay.read_rules = read_rules
+relay.read_config = lambda path=None: {"POINTER": "0"}
+
+# The fake devices: /dev/input/event900 (keyboard) and event901 (mouse), each a pipe.
+kb_r, kb_w = os.pipe()
+ms_r, ms_w = os.pipe()
+for fd in (kb_r, ms_r):
+    os.set_blocking(fd, False)
+FAKE = {"/dev/input/event900": kb_r, "/dev/input/event901": ms_r}
+HELD = {kb_r: set(), ms_r: set()}  # what EVIOCGKEY says each holds
+
+
+class Inode:
+    st_ino = 1
+
+
+fake_os = type(os)("os")
+fake_os.__dict__.update(os.__dict__)
+fake_os.listdir = lambda path: ["event900", "event901"] if path == "/dev/input" else os.listdir(path)
+fake_os.stat = lambda path, *a, **k: Inode() if path in FAKE else os.stat(path, *a, **k)
+fake_os.access = lambda path, mode, *a, **k: path in FAKE or os.access(path, mode, *a, **k)
+relay.os = fake_os
+
+
+def ioctl(fd, request, arg=0, *rest):
+    if fd in HELD and request == relay.EVIOCGKEY:
+        for c in HELD[fd]:
+            arg[c // 8] |= 1 << (c % 8)
+        return 0
+    return fcntl.ioctl(fd, request, arg, *rest)
+
+
+fake_fcntl = type(fcntl)("fcntl")
+fake_fcntl.__dict__.update(fcntl.__dict__)
+fake_fcntl.ioctl = ioctl
+relay.fcntl = fake_fcntl
+
+
+def probe(path):
+    if FAKE[path] == kb_r:
+        return relay.Node(path, kb_r, "test keyboard", relay.BUS_USB, 1, 2, "", False, True)
+    return relay.Node(path, ms_r, "test mouse", relay.BUS_USB, 3, 4, "", True, False)
+
+
+relay.probe = probe
+
+screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+screens.bind(relay.SCREENS)
+screens.settimeout(0.05)
+EVENT = struct.Struct("llHHi")
+
+
+def send(fd, etype, code, value):
+    os.write(fd, EVENT.pack(0, 0, etype, code, value) + EVENT.pack(0, 0, 0, 0, 0))
+    time.sleep(0.03)
+
+
+def key(code, value, fd=kb_w):
+    held = HELD[ms_r if fd == ms_w else kb_r]
+    (held.add if value else held.discard)(code)
+    send(fd, relay.EV_KEY, code, value)
+
+
+def typed():
+    """What the desktop was sent since the last call."""
+    got = []
+    while True:
+        try:
+            got.append(screens.recv(256).decode())
+        except socket.timeout:
+            return got
+
+
+def use(key_bindings):
+    bindings["now"] = key_bindings
+    c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    c.bind("")
+    c.settimeout(2)
+    c.sendto(b"reload", f"\0{tag}_relay")
+    c.recv(4096)
+    time.sleep(0.05)
+    typed()
+
+
+def pause(cmd):
+    c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    c.bind("")
+    c.settimeout(2)
+    c.sendto(f"pause {cmd} test".encode(), f"\0{tag}_relay")
+    c.recv(4096)
+    time.sleep(0.05)
+
+
+def lines(path, wait=0.3):
+    time.sleep(wait)
+    try:
+        with open(path) as f:
+            return f.read().split()
+    except OSError:
+        return []
+
+
+failures = []
+
+
+def check(label, got, want):
+    ok = got == want
+    print(("ok    " if ok else "FAIL  ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(label)
+
+
+META, RMETA, SHIFT, CTRL, RCTRL, A, F, J, P = 125, 126, 42, 29, 97, 30, 33, 36, 25
+F24 = ["key 194 1", "key 194 0"]
+
+
+def tests():
+    time.sleep(1.5)  # the relay's first device scan
+    typed()
+    key(META, 1); key(META, 0)
+    check("Meta tap (default): Steam menu", lines(steam_log), ["menu"])
+    check("Meta tap: the desktop gets F24 before Meta's release", typed(), ["key 125 1"] + F24 + ["key 125 0"])
+
+    key(RMETA, 1); key(RMETA, 2); key(RMETA, 2); key(RMETA, 0)
+    check("right Meta held (autorepeat), then released: a tap", lines(steam_log), ["menu", "menu"])
+    typed()
+
+    key(META, 1); key(A, 1); key(A, 0); key(META, 0)
+    check("Meta+A: no tap", lines(steam_log), ["menu", "menu"])
+    check("Meta+A (unbound) is typed as is", typed(), ["key 125 1", "key 30 1", "key 30 0", "key 125 0"])
+
+    key(META, 1); key(relay.BTN_LEFT, 1, ms_w); key(relay.BTN_LEFT, 0, ms_w); key(META, 0)
+    check("Meta+click: no tap", lines(steam_log), ["menu", "menu"])
+    key(META, 1); send(ms_w, relay.EV_REL, relay.REL_WHEEL, 1); key(META, 0)
+    check("Meta+scroll: no tap", lines(steam_log), ["menu", "menu"])
+    key(META, 1); key(SHIFT, 1); key(SHIFT, 0); key(META, 0)
+    check("Meta+Shift: no tap", lines(steam_log), ["menu", "menu"])
+    typed()
+
+    key(META, 1); key(J, 1); key(J, 0); key(META, 0)
+    check("Meta+J (default gaze click): no tap", lines(steam_log), ["menu", "menu"])
+    check("Meta+J: F24, Meta up early, its real release dropped", typed(), ["key 125 1"] + F24 + ["key 125 0"])
+
+    use({"29+42+33": f"command:echo combo >> {cmd_log}", "125": "none"})
+    key(CTRL, 1); key(SHIFT, 1); key(F, 1); key(F, 0); key(SHIFT, 0); key(CTRL, 0)
+    check("Ctrl+Shift+F runs its command", lines(cmd_log), ["combo"])
+    check("the combination's last key isn't typed", [k for k in typed() if k.startswith("key 33 ")], [])
+    key(META, 1); key(META, 0)
+    check("Meta tap bound to none: nothing", lines(steam_log), ["menu", "menu"])
+    check("Meta tap bound to none: no F24", typed(), ["key 125 1", "key 125 0"])
+
+    use({"29": f"command:command -v ft-steam ft-layout ft-float | wc -l >> {cmd_log}"})
+    key(RCTRL, 1); key(RCTRL, 0)
+    check("right Ctrl tap runs its command, which finds Frametop's tools", lines(cmd_log), ["combo", "3"])
+    check("Ctrl tap: F24 before Ctrl's release", typed(), ["key 97 1"] + F24 + ["key 97 0"])
+
+    use({"125+36": "gaze_left", "29": f"command:echo paused >> {cmd_log}", "42+125+25": "pause_toggle"})
+    pause("on")
+    check("pause on: paused", stub["pause"].paused, True)
+    key(META, 1); key(J, 1); key(J, 0); key(META, 0)
+    check("paused: Meta+J (gaze click) does nothing and is typed as is", typed(),
+          ["key 125 1", "key 36 1", "key 36 0", "key 125 0"])
+    key(RCTRL, 1); key(RCTRL, 0)
+    check("paused: a command still runs", lines(cmd_log), ["combo", "3", "paused"])
+    typed()
+    key(SHIFT, 1); key(META, 1); key(P, 1); key(P, 0); key(META, 0); key(SHIFT, 0)
+    check("paused: Meta+Shift+P (pause_toggle) resumes", stub["pause"].paused, False)
+    check("Meta+Shift+P isn't typed", [k for k in typed() if k.startswith("key 25 ")], [])
+    key(META, 1); key(J, 1); key(J, 0); key(META, 0)
+    check("resumed: Meta+J is a combination again", typed(), ["key 125 1"] + F24 + ["key 125 0"])
+
+    check("an empty command isn't an action", relay.known_action("command:  "), False)
+    check("steam_menu, pause_toggle and commands work without pointer mode",
+          (relay.needs_pointer("steam_menu"), relay.needs_pointer("pause_toggle"), relay.needs_pointer("command:ls")),
+          (False, False, False))
+    print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+    shutil.rmtree(OUT, ignore_errors=True)
+    os._exit(1 if failures else 0)
+
+
+threading.Thread(target=tests, daemon=True).start()
+sys.argv = [relay_path, "--no-grab"]
+relay.main()

input/test/pause-test.py

@@ -0,0 +1,214 @@
+#!/usr/bin/env python3
+"""Offline test of pausing for VR games (input/game_pause.py): the controller gesture, and when
+Frametop pauses and resumes by itself. The worker (services, remote desktop, the desktop) and
+the controller reader are stubs, the state file is a temporary one, and ft-screens' socket is
+renamed, so nothing reaches the live Frametop, SteamVR, or the running relay.
+
+  input/test/pause-test.py
+"""
+import os
+import sys
+import tempfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import game_pause as gp  # noqa: E402
+
+failures = []
+
+
+def check(label, got, want):
+    ok = got == want
+    print(("ok    " if ok else "FAIL  ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(label)
+
+
+# ---------------------------------------------------------------- the gesture
+L, R = "left/thumbstick", "right/thumbstick"
+
+
+def presses(g, events):
+    """events: (time, button, down); how many times the gesture went off."""
+    return sum(g.feed(b, d, t) for t, b, d in events)
+
+
+def both(t, hold=0.1, gap=0.02):
+    """Both sticks clicked at t (the right one gap later), held for hold."""
+    return [(t, L, True), (t + gap, R, True), (t + hold, L, False), (t + hold + gap, R, False)]
+
+
+g = gp.Gesture((L, R), 2)
+check("both sticks twice: goes off", presses(g, both(0) + both(0.3)), 1)
+g = gp.Gesture((L, R), 2)
+check("both sticks once: nothing", presses(g, both(0)), 0)
+g = gp.Gesture((L, R), 2)
+check("both sticks twice, 0.9 s apart: nothing", presses(g, both(0) + both(0.9)), 0)
+g = gp.Gesture((L, R), 2)
+check("left held (sprint), right clicked twice: nothing",
+      presses(g, [(0, L, True), (1.0, R, True), (1.1, R, False), (1.3, R, True), (1.4, R, False)]), 0)
+g = gp.Gesture((L, R), 2)
+check("sticks 0.5 s apart, twice: nothing", presses(g, both(0, gap=0.5, hold=0.6) + both(1.2, gap=0.5, hold=0.6)), 0)
+g = gp.Gesture((L, R), 2)
+check("three double presses in a row: once", presses(g, both(0) + both(0.3) + both(0.6)), 1)
+g = gp.Gesture((L, R), 2)
+check("again after a second: twice in all", presses(g, both(0) + both(0.3) + both(2.0) + both(2.3)), 2)
+g = gp.Gesture((L, R), 2)
+check("repeated down states count once",
+      presses(g, [(0, L, True), (0.01, L, True), (0.02, R, True), (0.03, R, True), (0.1, L, False), (0.1, R, False)]
+              + both(0.3)), 1)
+g = gp.Gesture((L, R), 1)
+check("both sticks once (presses 1): goes off", presses(g, both(0)), 1)
+g = gp.Gesture(("right/b",), 2)
+check("one button double-clicked: goes off",
+      presses(g, [(0, "right/b", True), (0.1, "right/b", False), (0.3, "right/b", True)]), 1)
+g = gp.Gesture(("right/b",), 2)
+check("other buttons don't count", presses(g, [(0, "right/a", True), (0.2, "right/a", True)]), 0)
+
+VR = ("left/thumbstick", "right/thumbstick", "right/b", "right/a")
+check("no pause_gesture: both sticks twice", gp.gesture_of({}, VR), ((L, R), 2))
+check("pause_gesture null: none", gp.gesture_of({"pause_gesture": None}, VR), None)
+check("one button, presses 1: still twice", gp.gesture_of({"pause_gesture": {"buttons": ["right/b"], "presses": 1}}, VR),
+      (("right/b",), 2))
+check("unknown buttons dropped", gp.gesture_of({"pause_gesture": {"buttons": ["right/zz", "right/a"]}}, VR),
+      (("right/a",), 2))
+check("settings: defaults", gp.settings_of({}, VR)["auto"], True)
+check("settings: unknown desktop mode is hide", gp.settings_of({"pause_desktop": "x"}, VR)["desktop"], "hide")
+
+
+# ---------------------------------------------------------------- pausing by itself
+class StubWorker:
+    def __init__(self, log, save):
+        self.jobs = []
+        self.busy = False
+
+    def start(self):
+        pass
+
+    def submit(self, job, *args):
+        self.jobs.append(job.__name__)
+
+    def snapshot(self):
+        return {"units": [], "remote": False, "desktop": None}
+
+    def restore(self, stopped):
+        self.restored = stopped
+
+    def pause(self, desktop):
+        pass
+
+    def resume(self):
+        pass
+
+    def enforce(self):
+        pass
+
+
+class StubWatch:
+    connected = False
+
+    def __init__(self, log):
+        pass
+
+    def start(self):
+        pass
+
+    def set_gesture(self, spec):
+        self.spec = spec
+
+
+tmp = tempfile.mkdtemp(prefix="ft-pause-test-")
+gp.STATE_PATH = os.path.join(tmp, "pause.json")
+gp.SCREENS = f"\0ft_pause_test_{os.getpid()}_screens"
+gp.Worker, gp.ControllerWatch = StubWorker, StubWatch
+changes = []
+
+
+def new(rules=None):
+    """A relay starting afresh: not paused before."""
+    if os.path.exists(gp.STATE_PATH):
+        os.remove(gp.STATE_PATH)
+    changes.clear()
+    p = gp.GamePause(lambda *a: None, changes.append, VR)
+    p.configure(dict({"pause_sound": False}, **(rules or {})))
+    return p
+
+
+p = new()
+p.game_state(True, 0)
+check("a game starts: paused", (p.paused, p.reason, p.ends_with_game), (True, "game", True))
+check("the relay heard", changes, [True])
+p.game_state(True, 5)
+p.tick(5)
+check("the helper repeats it: still paused, nothing new", (p.paused, changes), (True, [True]))
+p.game_state(False, 10)
+p.tick(12)
+check("the game ends: not yet resumed", p.paused, True)
+p.tick(15.1)
+check("5 s later: resumed", (p.paused, changes), (False, [True, False]))
+check("the worker paused and resumed", p.worker.jobs, ["pause", "resume"])
+
+p = new()
+p.game_state(True, 0)
+p.game_state(False, 10)
+p.game_state(True, 12)
+p.tick(16)
+check("the next game starts within 5 s: stays paused", p.paused, True)
+p.game_state(False, 20)
+p.tick(25.1)
+check("and resumes 5 s after that one ends", p.paused, False)
+
+p = new()
+p.game_state(True, 0)
+p.toggle("gesture", 1)
+check("resumed during a game: running", p.paused, False)
+p.game_state(True, 6)
+p.tick(30)
+check("and stays running while that game runs", p.paused, False)
+p.game_state(False, 40)
+p.game_state(True, 50)
+check("the next game pauses again", p.paused, True)
+
+p = new()
+p.toggle("command", 0)
+check("paused outside a game: not tied to one", (p.paused, p.ends_with_game), (True, False))
+p.game_state(True, 1)
+p.game_state(False, 10)
+p.tick(20)
+check("a game comes and goes: still paused", p.paused, True)
+p.toggle("command", 21)
+check("toggled: running", p.paused, False)
+
+p = new({"pause_auto": False})
+p.game_state(True, 0)
+check("auto off: a game doesn't pause", p.paused, False)
+p.toggle("gesture", 1)
+p.game_state(False, 10)
+p.tick(20)
+check("auto off: a pause during a game outlasts it", p.paused, True)
+
+p = new()
+p.game_state(True, 0)
+p.tick(0 + gp.GAME_STALE + 0.1)
+check("the helper goes quiet: the game counts as ended", p.game, False)
+p.tick(0 + gp.GAME_STALE + gp.RESUME_DELAY + 0.2)
+check("and the pause ends", p.paused, False)
+
+p = new()
+p.toggle("gesture", 0)
+q = gp.GamePause(lambda *a: None, changes.append, VR)
+check("a relay restarted while paused stays paused", (q.paused, q.reason), (True, "gesture"))
+check("and stops the services again", q.worker.jobs, ["enforce"])
+q.toggle("gesture", 1)
+r = gp.GamePause(lambda *a: None, changes.append, VR)
+check("resumed: the next relay starts running", r.paused, False)
+
+p = new({"pause_gesture": None})
+check("pause_gesture null: the reader gets none", p.watch.spec, None)
+check("status", sorted(p.status()), sorted(["t", "paused", "reason", "since", "ends_with_game", "game", "busy",
+                                            "stopped", "gesture_reader", "auto", "desktop"]))
+
+print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+for f in os.listdir(tmp):
+    os.remove(os.path.join(tmp, f))
+os.rmdir(tmp)
+sys.exit(1 if failures else 0)

input/vrws.py

@@ -0,0 +1,198 @@
+#!/usr/bin/python3
+"""vrws: vrserver's local web socket, with nothing but the standard library.
+
+vrserver (SteamVR) serves its controller binding page on 127.0.0.1:27062, and that page
+follows the controllers' raw input through a web socket there (mailbox input_server,
+request_input_state_updates). Reading it takes nothing from anyone: whatever the buttons are
+bound to still happens, and it works whatever app has input focus, VR games included. SteamVR
+input only reaches the focused app, or an overlay with global input, which then takes the button
+from the game (pointer/helper/vrbuttons.h). frame-voice reads its push-to-talk button the same
+way. It's undocumented, so a SteamVR update could change it (scripts/update-check.py checks it).
+The host's Python has no `websockets` module, so this is a small RFC 6455 client of its own.
+
+  controllers()   the Frame controllers SteamVR has now: {root path: "left" | "right"}. A
+                  controller's root path is /user/hand/<side>, or /devices/cv/<serial> while the
+                  3D mouse's virtual controller holds that hand role.
+  VrSocket()      a connection: open(mailbox), subscribe(path), messages(timeout) -> raw texts,
+                  each a JSON object like {"type": "update_component_states", "device": path,
+                  "components": {"/input/thumbstick/click": true, ...}} with the components
+                  that changed (mostly capacitive sensing, about 160 a second)
+
+Run as a program, it prints the controllers' button changes for a while (default 10 s):
+
+  input/vrws.py [seconds]
+"""
+import base64
+import json
+import os
+import select
+import socket
+import struct
+import sys
+import time
+import urllib.request
+
+HOST, PORT = "127.0.0.1", 27062
+ORIGIN = f"http://{HOST}:{PORT}"
+# vrserver answers the binding page's own requests; these headers make ours look like them.
+HEADERS = {"Referer": f"{ORIGIN}/dashboard/controllerbinding.html"}
+
+
+def getstate(timeout=3):
+    """vrserver's /input/getstate.json: the devices and their input components."""
+    req = urllib.request.Request(f"{ORIGIN}/input/getstate.json", headers=HEADERS)
+    with urllib.request.urlopen(req, timeout=timeout) as resp:
+        return json.load(resp)
+
+
+def controllers(timeout=3):
+    """{root path: side} for the Frame controllers SteamVR has now."""
+    return {d["root_path"]: d["side"] for d in getstate(timeout).get("devices", [])
+            if d.get("controller_type") == "frame_controller" and d.get("root_path")
+            and d.get("side") in ("left", "right")}
+
+
+class VrSocket:
+    def __init__(self, timeout=3):
+        self.sock = socket.create_connection((HOST, PORT), timeout=timeout)
+        try:
+            key = base64.b64encode(os.urandom(16)).decode()
+            self.sock.sendall((f"GET / HTTP/1.1\r\nHost: {HOST}:{PORT}\r\nUpgrade: websocket\r\n"
+                               f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n"
+                               f"Origin: {ORIGIN}\r\nReferer: {HEADERS['Referer']}\r\n\r\n").encode())
+            head = b""
+            while b"\r\n\r\n" not in head:
+                chunk = self.sock.recv(4096)
+                if not chunk:
+                    raise OSError("vrserver closed the connection during the handshake")
+                head += chunk
+            head, rest = head.split(b"\r\n\r\n", 1)
+            status = head.split(b"\r\n", 1)[0]
+            if b" 101 " not in status + b" ":
+                raise OSError(f"vrserver refused the web socket: {status.decode(errors='replace')}")
+        except BaseException:
+            self.sock.close()
+            raise
+        self.sock.settimeout(None)
+        self.buf = bytearray(rest)
+        self.parts = bytearray()  # a fragmented message so far
+        self.mailbox = ""
+
+    def fileno(self):
+        return self.sock.fileno()
+
+    def close(self):
+        try:
+            self._frame(0x8, b"")
+        except OSError:
+            pass
+        self.sock.close()
+
+    def _frame(self, opcode, payload):
+        """One frame to vrserver: final, masked (as a client's must be)."""
+        n = len(payload)
+        head = bytes([0x80 | opcode])
+        if n < 126:
+            head += bytes([0x80 | n])
+        elif n < 65536:
+            head += bytes([0x80 | 126]) + struct.pack(">H", n)
+        else:
+            head += bytes([0x80 | 127]) + struct.pack(">Q", n)
+        mask = os.urandom(4)
+        self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload)))
+
+    def send(self, text):
+        self._frame(0x1, text.encode())
+
+    def open(self, mailbox):
+        self.send(f"mailbox_open {mailbox}")
+        self.mailbox = mailbox
+
+    def subscribe(self, path, on=True):
+        kind = "request_input_state_updates" if on else "cancel_input_state_updates"
+        self.send("mailbox_send input_server " +
+                  json.dumps({"type": kind, "device_path": path, "returnAddress": self.mailbox}))
+
+    def messages(self, timeout=None):
+        """The text messages that came in, as strings: waits up to `timeout` seconds (None:
+        forever) for some, [] at the timeout. Raises OSError when the connection ends."""
+        out = self._parse()
+        if out:
+            return out
+        if not select.select([self.sock], [], [], timeout)[0]:
+            return []
+        chunk = self.sock.recv(65536)
+        if not chunk:
+            raise OSError("vrserver closed the web socket")
+        self.buf += chunk
+        return self._parse()
+
+    def _parse(self):
+        """Every whole frame in the buffer; a partial one waits for the rest."""
+        out, buf, pos = [], self.buf, 0
+        while len(buf) - pos >= 2:
+            b0, b1 = buf[pos], buf[pos + 1]
+            n, head = b1 & 0x7F, 2
+            if n == 126:
+                if len(buf) - pos < 4:
+                    break
+                n, head = struct.unpack_from(">H", buf, pos + 2)[0], 4
+            elif n == 127:
+                if len(buf) - pos < 10:
+                    break
+                n, head = struct.unpack_from(">Q", buf, pos + 2)[0], 10
+            masked = b1 & 0x80
+            if masked:
+                head += 4
+            if len(buf) - pos < head + n:
+                break
+            data = bytes(buf[pos + head:pos + head + n])
+            if masked:  # servers don't mask, but nothing says they can't
+                mask = buf[pos + head - 4:pos + head]
+                data = bytes(b ^ mask[i % 4] for i, b in enumerate(data))
+            pos += head + n
+            opcode = b0 & 0x0F
+            if opcode == 0x8:
+                raise OSError("vrserver closed the web socket")
+            if opcode == 0x9:
+                self._frame(0xA, data)  # ping: pong
+            elif opcode in (0x0, 0x1, 0x2):
+                self.parts += data
+                if b0 & 0x80:
+                    out.append(self.parts.decode(errors="replace"))
+                    self.parts = bytearray()
+        del buf[:pos]
+        return out
+
+
+def main():
+    seconds = float(sys.argv[1]) if len(sys.argv) > 1 else 10.0
+    ws = VrSocket()
+    ws.open(f"frametop_vrws_{os.getpid()}")
+    sides = controllers()
+    for path, side in sides.items():
+        print(f"{side} controller: {path}", flush=True)
+        ws.subscribe(path)
+    if not sides:
+        print("no Frame controllers", flush=True)
+    start = time.monotonic()
+    count = 0
+    try:
+        while time.monotonic() - start < seconds:
+            for text in ws.messages(timeout=0.5):
+                count += 1
+                if "/click" not in text:
+                    continue
+                msg = json.loads(text)
+                side = sides.get(msg.get("device"), msg.get("device"))
+                for name, value in sorted(msg.get("components", {}).items()):
+                    if name.endswith("/click"):
+                        print(f"{time.monotonic() - start:7.3f}  {side} {name[len('/input/'):-len('/click')]} "
+                              f"{'down' if value else 'up'}", flush=True)
+    finally:
+        ws.close()
+    print(f"{count} messages in {seconds:.0f} s")
+
+
+if __name__ == "__main__":
+    main()

install.sh

@@ -1,13 +1,13 @@
 #!/usr/bin/env bash
 # Install everything on the Steam Frame: the build container, Frametop (multi-screen
-# desktop, input relay, universal 3D mouse, settings app), and optionally gaze mode and the
-# Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
+# desktop, input relay, universal 3D mouse, settings app), and optionally gaze mode, our own
+# eye tracker for it, and the Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
 # for example after `git pull`. (Hand tracking, hands/, is deferred: it isn't offered here.)
 # (It also works from a PC over SSH; see "Developing from a PC" in the README.)
 #
 # Usage: ./install.sh [--yes] [--no-bluetooth]
-#   --yes           don't ask; installs gaze mode, skips the Bluetooth fixes and the SteamVR
-#                   restart
+#   --yes           don't ask; installs gaze mode, and our eye tracker if sudo can run without
+#                   a password prompt; skips the Bluetooth fixes and the SteamVR restart
 #   --no-bluetooth  don't offer the Bluetooth fixes
 set -euo pipefail
 
@@ -19,7 +19,7 @@ for arg in "$@"; do
   case $arg in
     --yes) assume_yes=1 ;;
     --no-bluetooth) bluetooth=0 ;;
-    -h|--help) sed -n '2,11p' "$0"; exit 0 ;;
+    -h|--help) sed -n '2,12p' "$0"; exit 0 ;;
     *) echo "unknown option: $arg" >&2; exit 2 ;;
   esac
 done
@@ -38,6 +38,13 @@ ask() {  # ask "question" default(y|n)
   answer=${answer:-$2}
   [[ $answer =~ ^[Yy] ]]
 }
+# Can sudo run here without a password prompt (SUDO_ASKPASS, the repo's .env, a recent sudo)?
+# See frame_sudo in scripts/_env.sh.
+sudo_quiet() {
+  { [ "$FRAME_LOCAL" = 1 ] && [ -n "${SUDO_ASKPASS:-}" ]; } && return 0
+  grep -q '^steamos_root_pwd=.' "$REPO_ROOT/.env" 2>/dev/null && return 0
+  on_frame 'sudo -n true' 2>/dev/null
+}
 
 if [ "$FRAME_LOCAL" = 1 ]; then
   echo "Installing on this Steam Frame from $FRAME_REPO"
@@ -50,7 +57,7 @@ if [ "$FRAME_LOCAL" = 0 ] || [ -n "${SSH_CONNECTION:-}" ]; then
   echo "that need SteamVR start with it if it isn't running now."
 fi
 
-step "1/9 distrobox (container tool, installed in your home folder)"
+step "1/10 distrobox (container tool, installed in your home folder)"
 if on_frame 'test -x ~/.local/bin/distrobox'; then
   echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')"
 else
@@ -60,25 +67,25 @@ else
 cd ~/dev/src/distrobox && ./install --prefix ~/.local'
 fi
 
-step "2/9 build container (Fedora 44 'dev', about 1-2 GB the first time)"
+step "2/10 build container (Fedora 44 'dev', about 1-2 GB the first time)"
 "$root/setup/dev-container.sh"
 
-step "3/9 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
+step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
 "$root/desktops.sh" relay install
 
-step "4/9 3D mouse: SteamVR driver"
+step "4/10 3D mouse: SteamVR driver"
 "$root/pointer/driver/build.sh"
 "$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open
 
-step "5/9 3D mouse: pointer helper service"
+step "5/10 3D mouse: pointer helper service"
 "$root/pointer/helper/build.sh"
 "$root/pointer/helper/run.sh" install
 
-step "6/9 power service (turns the displays off while the headset isn't used, even on a stand)"
+step "6/10 power service (turns the displays off while the headset isn't used, even on a stand)"
 "$root/power/build.sh"
 "$root/power/run.sh" install
 
-step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
+step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
 "$root/screens/build.sh"
 "$root/desktops.sh" install >/dev/null
 "$root/input-settings/install.sh"
@@ -87,14 +94,44 @@ step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Framet
 on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
 echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
 
-step "8/9 gaze mode (optional, experimental: the pointer goes where you look)"
+step "8/10 gaze mode (optional, experimental: the pointer goes where you look)"
+gaze=0
 if ask "Install gaze mode? You turn it on and calibrate it in Frametop Input Settings, on the Gaze page." y; then
   "$root/gaze/run.sh" install
+  gaze=1
 else
   echo "skipped. Install later with: gaze/run.sh install"
 fi
 
-step "9/9 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
+# Ours is gaze mode's default once it's installed (GAZE_TRACKER=auto). Without it, gaze mode
+# uses SteamVR's eye tracker.
+step "9/10 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
+if [ "$gaze" = 0 ]; then
+  echo "skipped: gaze mode isn't installed. Install it later with: gaze/tracker/install.sh"
+elif [ "$assume_yes" = 1 ] && ! sudo_quiet; then
+  echo "skipped: it needs your password (sudo), and --yes doesn't ask. Install it later with: gaze/tracker/install.sh"
+elif ask "Install our own eye tracker? Gaze mode then uses it instead of SteamVR's. Its frame grabber is a small system service, so it needs your password (sudo), and it downloads about 165 MB (numpy, OpenCV)." y; then
+  if "$root/gaze/tracker/install.sh" install; then
+    # Configs made before GAZE_TRACKER=auto say steam, which keeps SteamVR's.
+    tracker=$(on_frame "sed -n 's/^GAZE_TRACKER=\([a-z]*\).*/\1/p' ~/.config/frametop.conf | tail -1")
+    if [ "$tracker" = steam ] &&
+       ask "Your settings pick SteamVR's eye tracker (GAZE_TRACKER=steam, the old default). Use ours instead?" y; then
+      on_frame "sed -i 's~^GAZE_TRACKER=steam\b.*~GAZE_TRACKER=auto          # gaze service: auto = our own eye tracker when installed, else SteamVR | own | steam~' ~/.config/frametop.conf"
+      tracker=auto
+    fi
+    if [ "$tracker" = steam ]; then
+      echo "installed; your settings keep SteamVR's eye tracker (pick Own tracker on the Gaze page to use ours)"
+    else
+      echo "gaze mode uses our eye tracker: calibrate it once, with Calibrate on the Gaze page"
+    fi
+  else
+    echo "our eye tracker didn't install, so gaze mode uses SteamVR's. Try again with: gaze/tracker/install.sh"
+  fi
+else
+  echo "skipped: gaze mode uses SteamVR's eye tracker. Install ours later with: gaze/tracker/install.sh"
+fi
+
+step "10/10 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
 if [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; then
   "$root/setup/bluetooth/install.sh" install
 else

pointer/driver/driver_ft_pointer.cpp

@@ -12,7 +12,7 @@
 //   aim <yaw_deg> <pitch_deg>   absolute direction (yaw 0 = -Z, the SteamVR forward)
 //   gaze                         follow the head (origin and direction) again
 //   distance <metres>            cursor distance from the anchor (default 1.5)
-//   pose <x> <y> <z> <yaw> <pitch>  exact pose, sent every frame by the ft-pointer helper
+//   pose <x> <y> <z> <yaw> <pitch>  exact pose, from the ft-pointer helper when it changes (it holds)
 //   posq <x> <y> <z> <qw> <qx> <qy> <qz>  exact pose with a full rotation (tilting a panel while moving it)
 //   btn <name> <0|1>             name: trigger, b, x, system, joystick, a (a = claim the laser, no click)
 //   scroll <x> <y>               joystick deflection -1..1
@@ -224,12 +224,23 @@ public:
         pose.result = ok ? TrackingResult_Running_OK : TrackingResult_Uninitialized;
         pose.deviceIsConnected = snapshot.visible;
         pose_ = pose;
-        VRServerDriverHost()->TrackedDevicePoseUpdated(objectId_, pose_, sizeof(DriverPose_t));
+        // Connected, the pose goes out every frame, as SteamVR expects of a tracked device.
+        // Disconnected, once: it says the same thing every frame after.
+        if (snapshot.visible || !reportedOff_) {
+            VRServerDriverHost()->TrackedDevicePoseUpdated(objectId_, pose_, sizeof(DriverPose_t));
+            reportedOff_ = !snapshot.visible;
+        }
 
+        // Inputs only when they change (and all of them the first time).
         auto input = VRDriverInput();
-        for (int i = 0; i < kButtons; ++i) input->UpdateBooleanComponent(buttons_[i], snapshot.buttons[i], 0);
-        input->UpdateScalarComponent(scrollX_, snapshot.scrollX, 0);
-        input->UpdateScalarComponent(scrollY_, snapshot.scrollY, 0);
+        for (int i = 0; i < kButtons; ++i)
+            if (!inputsSent_ || snapshot.buttons[i] != sentButtons_[i])
+                input->UpdateBooleanComponent(buttons_[i], sentButtons_[i] = snapshot.buttons[i], 0);
+        if (!inputsSent_ || snapshot.scrollX != sentScrollX_)
+            input->UpdateScalarComponent(scrollX_, sentScrollX_ = snapshot.scrollX, 0);
+        if (!inputsSent_ || snapshot.scrollY != sentScrollY_)
+            input->UpdateScalarComponent(scrollY_, sentScrollY_ = snapshot.scrollY, 0);
+        inputsSent_ = true;
     }
 
 private:
@@ -239,6 +250,10 @@ private:
     int32_t role_ = TrackedControllerRole_RightHand;
     bool hinted_ = false;  // whether the role hint currently claims role_
     DriverPose_t pose_{};
+    bool reportedOff_ = false;  // the disconnected pose has gone out (it isn't repeated)
+    // What the input components were last set to.
+    bool inputsSent_ = false, sentButtons_[kButtons] = {};
+    float sentScrollX_ = 0.f, sentScrollY_ = 0.f;
     VRInputComponentHandle_t buttons_[kButtons] = {};
     VRInputComponentHandle_t scrollX_ = 0, scrollY_ = 0;
 };
@@ -293,65 +308,102 @@ private:
         }
     }
 
+    // Parsed first, then applied under the lock, which RunFrame (vrserver's frame) takes too.
     void Handle(const char *cmd) {
-        std::lock_guard<std::mutex> guard(state_.lock);
+        enum { kNone, kPosq, kPose, kMove, kRecenter, kAim, kGaze, kHide, kShow, kRole, kBtn, kDistance, kScroll };
+        int kind = kNone;
         char name[32];
-        float a, b;
-        int v;
-        float x, y, z, qw, qx, qy, qz;
-        if (std::sscanf(cmd, "posq %f %f %f %f %f %f %f", &x, &y, &z, &qw, &qx, &qy, &qz) == 7) {
+        float f[7];
+        int v = 0, role = 0, button = -1;
+        if (std::sscanf(cmd, "posq %f %f %f %f %f %f %f", &f[0], &f[1], &f[2], &f[3], &f[4], &f[5], &f[6]) == 7) {
+            kind = kPosq;
+        } else if (std::sscanf(cmd, "pose %f %f %f %f %f", &f[0], &f[1], &f[2], &f[3], &f[4]) == 5) {
+            kind = kPose;
+        } else if (std::sscanf(cmd, "move %f %f", &f[0], &f[1]) == 2) {
+            kind = kMove;
+        } else if (std::strncmp(cmd, "recenter", 8) == 0) {
+            kind = kRecenter;
+        } else if (std::sscanf(cmd, "aim %f %f", &f[0], &f[1]) == 2) {
+            kind = kAim;
+        } else if (std::strncmp(cmd, "gaze", 4) == 0) {
+            kind = kGaze;
+        } else if (std::strncmp(cmd, "hide", 4) == 0) {
+            kind = kHide;
+        } else if (std::strncmp(cmd, "show", 4) == 0) {
+            kind = kShow;
+        } else if (std::sscanf(cmd, "role %31s", name) == 1) {
+            role = !std::strcmp(name, "left")     ? TrackedControllerRole_LeftHand
+                   : !std::strcmp(name, "right")  ? TrackedControllerRole_RightHand
+                   : !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus
+                                                  : 0;
+            if (role) kind = kRole;
+        } else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) {
+            for (int i = 0; i < kButtons; ++i)
+                if (std::strcmp(name, kButtonNames[i]) == 0) button = i;
+            if (button >= 0) kind = kBtn;
+        } else if (std::sscanf(cmd, "distance %f", &f[0]) == 1) {
+            kind = kDistance;
+        } else if (std::sscanf(cmd, "scroll %f %f", &f[0], &f[1]) == 2) {
+            kind = kScroll;
+        }
+        if (kind == kNone) return;
+
+        std::lock_guard<std::mutex> guard(state_.lock);
+        switch (kind) {
+        case kPosq:
             state_.explicitPose = true;
             state_.hasQuat = true;
             state_.gaze = false;
-            state_.pos[0] = x;
-            state_.pos[1] = y;
-            state_.pos[2] = z;
-            state_.quat[0] = qw;
-            state_.quat[1] = qx;
-            state_.quat[2] = qy;
-            state_.quat[3] = qz;
-        } else if (std::sscanf(cmd, "pose %f %f %f %f %f", &x, &y, &z, &a, &b) == 5) {
+            std::memcpy(state_.pos, f, sizeof state_.pos);
+            std::memcpy(state_.quat, f + 3, sizeof state_.quat);
+            break;
+        case kPose:
             state_.explicitPose = true;
             state_.hasQuat = false;
             state_.gaze = false;
-            state_.pos[0] = x;
-            state_.pos[1] = y;
-            state_.pos[2] = z;
-            state_.yaw = a;
-            state_.pitch = b;
-        } else if (std::sscanf(cmd, "move %f %f", &a, &b) == 2) {
+            std::memcpy(state_.pos, f, sizeof state_.pos);
+            state_.yaw = f[3];
+            state_.pitch = f[4];
+            break;
+        case kMove:
             state_.explicitPose = false;
             if (state_.gaze) state_.recenter = true;  // first move starts from the gaze
-            state_.yaw += a;  // wrap by hand: libm remainder() is GLIBC_2.43 in the build container
+            state_.yaw += f[0];  // wrap by hand: libm remainder() is GLIBC_2.43 in the build container
             while (state_.yaw > 180.f) state_.yaw -= 360.f;
             while (state_.yaw < -180.f) state_.yaw += 360.f;
-            state_.pitch = std::fmax(-85.f, std::fmin(85.f, state_.pitch + b));
-        } else if (std::strncmp(cmd, "recenter", 8) == 0) {
+            state_.pitch = std::fmax(-85.f, std::fmin(85.f, state_.pitch + f[1]));
+            break;
+        case kRecenter:
             state_.explicitPose = false;
             state_.recenter = true;
-        } else if (std::sscanf(cmd, "aim %f %f", &a, &b) == 2) {
+            break;
+        case kAim:
             state_.gaze = false;
-            state_.yaw = a;
-            state_.pitch = b;
-        } else if (std::strncmp(cmd, "gaze", 4) == 0) {
+            state_.yaw = f[0];
+            state_.pitch = f[1];
+            break;
+        case kGaze:
             state_.gaze = true;
-        } else if (std::strncmp(cmd, "hide", 4) == 0) {
+            break;
+        case kHide:
             state_.visible = false;
-        } else if (std::strncmp(cmd, "show", 4) == 0) {
+            break;
+        case kShow:
             state_.visible = true;
-        } else if (std::sscanf(cmd, "role %31s", name) == 1) {
-            state_.role = !std::strcmp(name, "left")     ? TrackedControllerRole_LeftHand
-                          : !std::strcmp(name, "right")  ? TrackedControllerRole_RightHand
-                          : !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus
-                                                         : state_.role;
-        } else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) {
-            for (int i = 0; i < kButtons; ++i)
-                if (std::strcmp(name, kButtonNames[i]) == 0) state_.buttons[i] = v != 0;
-        } else if (std::sscanf(cmd, "distance %f", &a) == 1) {
-            state_.distance = std::fmax(0.3f, std::fmin(10.f, a));
-        } else if (std::sscanf(cmd, "scroll %f %f", &a, &b) == 2) {
-            state_.scrollX = a;
-            state_.scrollY = b;
+            break;
+        case kRole:
+            state_.role = role;
+            break;
+        case kBtn:
+            state_.buttons[button] = v != 0;
+            break;
+        case kDistance:
+            state_.distance = std::fmax(0.3f, std::fmin(10.f, f[0]));
+            break;
+        case kScroll:
+            state_.scrollX = f[0];
+            state_.scrollY = f[1];
+            break;
         }
     }
 

pointer/helper/ft-pointer.cpp

@@ -36,6 +36,22 @@
 // pointer is off the helper also stops listing overlays with vrcmd, whose connection every
 // second kept SteamVR from going to standby.
 //
+// Idle: while the pointer is off (and hand gestures are off), the main loop waits for a command
+// on its socket for up to 250 ms instead of running every 8 ms, or 20 ms while it reads mapped
+// Frame controller buttons (vrbuttons.h), which have no event to wait for. What has to go on
+// meanwhile still does: the headset's activity level, the game check and the relay's "vrgame"
+// and "gazeawake" repeats, and a mouse command wakes it at once. The overlay lookups (the 50 ms
+// visibility poll, the once-a-second handles) wait for the pointer to wake, and run right then.
+//
+// Unchanged frames: while the pointer is awake, a frame where nothing moved reuses the last one's
+// result. The collision passes (ComputeOverlayIntersection on every visible overlay, twice) run
+// again when the mouse or the anchor moved, the eye moved 5 mm, an overlay showed or hid, or
+// 100 ms have passed (overlays also move by themselves), and always while dragging. The dots'
+// setters go to SteamVR only when their value changes, and a dot's placement only when it moved
+// 0.2 mm or the eye 5 mm (its facing and size then change by well under 1%). The pose goes to the
+// driver, which keeps the last one, when the laser's origin moved 0.2 mm or its direction 0.04
+// deg (0.1 mm where it lands, 15 cm on), and at least every 100 ms.
+//
 // Last used wins: when a real controller moves (picked up), the pointer is released
 // (driver "hide", which also drops its hand role hint), so the controller gets
 // its role and laser back. The next mouse input reconnects and claims the laser again.
@@ -267,7 +283,8 @@
 //
 // Commands (datagrams on @ft_pointer_helper): show, hide, recenter, move <dyaw> <dpitch>,
 // follow on|off|toggle (head follow, until the next restart or a change to POINTER_FOLLOW),
-// gaze on|off|toggle|? (gaze mode, likewise with POINTER_GAZE; ? only asks), gz ... (the gaze, from ft-gazed),
+// gaze on|off|toggle|? (gaze mode, likewise with POINTER_GAZE; ? only asks, and "? headset" adds
+// worn|away to the reply), gz ... (the gaze, from ft-gazed),
 // reload (re-read the settings below), debug (toggle a twice-a-second state log),
 // overlays (replies with the overlay list as JSON, see OverlayList),
 // vrbind/vrglobal/vrstatus (Frame controller buttons, see vrbuttons.h),
@@ -310,6 +327,7 @@ extern "C" {
 #include <atomic>
 #include <chrono>
 #include <cmath>
+#include <condition_variable>
 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
@@ -327,6 +345,7 @@ extern "C" {
 #include <fcntl.h>
 #include <fnmatch.h>
 
+#include <poll.h>
 #include <sys/mman.h>
 #include <sys/socket.h>
 #include <sys/stat.h>
@@ -499,29 +518,62 @@ std::string JsonQuote(const std::string &s) {
 // public call to enumerate other apps' overlays). Hidden ones are listed too: the
 // window controls under a floating panel only appear while something hovers the
 // panel, and the cursor has to find them the moment they do, not a second later.
+// Each vrcmd run is a shell and a new SteamVR client, about 30 ms of CPU, so the list is read
+// every 20 seconds (kEvery) while the pointer is awake, and at once when something says it may
+// have changed (Kick: the pointer waking, the dashboard opening or closing, a click that hit
+// nothing), at most once a second. Showing and hiding the overlays it knows doesn't need a new
+// list: the main loop polls their visibility (IsOverlayVisible) every 50 ms.
 // Paused while the pointer is off: each vrcmd run connects to SteamVR as a new app, and a new
 // app every second kept SteamVR (and the headset's displays) from going to standby.
 // "overlays" requests (Frametop Input Settings' Ignored panels page) refresh the list even
 // while paused, and are answered from this thread once it's fresh.
 class OverlayList {
 public:
+    static constexpr auto kEvery = std::chrono::seconds(20), kGap = std::chrono::seconds(1);
     void Start() {
         out_ = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0);
         thread_ = std::thread([this] {
+            std::unique_lock<std::mutex> lk(lock_);
             while (running_) {
-                if (!paused_ || requested_) Refresh();
-                // Wait a second, or less when the pointer wakes (refresh right away then).
-                for (int i = 0; i < 10 && running_; ++i) {
-                    const bool wasPaused = paused_;
-                    std::this_thread::sleep_for(std::chrono::milliseconds(100));
-                    if ((wasPaused && !paused_) || requested_) break;
+                // Sleeps while paused (until a request or the wake); otherwise until it's time, or
+                // a kick once kGap has passed since the last read.
+                const auto due = std::max(last_ + kGap, kicked_ ? last_ : last_ + kEvery);
+                if (!waiting_.empty()) {
+                } else if (paused_) {
+                    cv_.wait(lk);
+                    continue;
+                } else if (std::chrono::steady_clock::now() < due) {
+                    cv_.wait_until(lk, due);
+                    continue;
                 }
+                kicked_ = false;
+                lk.unlock();
+                Refresh();
+                lk.lock();
+                last_ = std::chrono::steady_clock::now();
             }
         });
     }
-    void SetPaused(bool paused) { paused_ = paused; }
+    // Unpausing (the pointer woke) reads the list again at once.
+    void SetPaused(bool paused) {
+        std::lock_guard<std::mutex> guard(lock_);
+        if (paused == paused_) return;
+        if (!paused) kicked_ = true, last_ = {};
+        paused_ = paused;
+        cv_.notify_one();
+    }
+    // The list may have changed: read it again soon (not while paused).
+    void Kick() {
+        std::lock_guard<std::mutex> guard(lock_);
+        kicked_ = true;
+        cv_.notify_one();
+    }
     void Stop() {
-        running_ = false;
+        {
+            std::lock_guard<std::mutex> guard(lock_);
+            running_ = false;
+            cv_.notify_one();
+        }
         if (thread_.joinable()) thread_.join();
     }
     std::vector<std::string> Keys() {
@@ -530,12 +582,14 @@ public:
         for (const auto &e : entries_) keys.push_back(e.key);
         return keys;
     }
+    // Goes up by one each time the list is read, so the main loop knows to look the keys up again.
+    unsigned Generation() const { return generation_; }
     // Answer `to` with {"t":"overlays","list":[{"key","name","visible"}...]} after the next refresh.
     void Request(const sockaddr_un &to, socklen_t len) {
         if (len <= offsetof(sockaddr_un, sun_path)) return;
         std::lock_guard<std::mutex> guard(lock_);
         if (waiting_.size() < 8) waiting_.push_back({to, len});
-        requested_ = true;
+        cv_.notify_one();
     }
 
 private:
@@ -544,7 +598,6 @@ private:
         bool visible;
     };
     void Refresh() {
-        requested_ = false;
         FILE *p = popen("LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrcmd --overlays 2>/dev/null", "r");
         if (!p) return;
         std::vector<Entry> entries;
@@ -576,6 +629,7 @@ private:
             entries_ = std::move(entries);
             waiting.swap(waiting_);
         }
+        ++generation_;
         if (waiting.empty()) return;
         std::string msg = "{\"t\":\"overlays\",\"list\":[";
         for (size_t i = 0; i < entries_.size(); ++i)
@@ -588,10 +642,11 @@ private:
 
     std::thread thread_;
     int out_ = -1;
-    std::atomic<bool> paused_{false};
-    std::atomic<bool> running_{true};
-    std::atomic<bool> requested_{false};
-    std::mutex lock_;
+    std::mutex lock_;  // guards everything below
+    std::condition_variable cv_;
+    bool paused_ = false, running_ = true, kicked_ = false;
+    std::chrono::steady_clock::time_point last_{};  // the last read
+    std::atomic<unsigned> generation_{0};
     std::vector<Entry> entries_;  // written only by the thread; the lock guards readers
     std::vector<std::pair<sockaddr_un, socklen_t>> waiting_;
 };
@@ -619,6 +674,36 @@ bool Ignored(const std::vector<std::string> &patterns, const std::string &key) {
     return false;
 }
 
+vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye);
+
+// One of our dots, with what was last set on it: each setter goes to SteamVR only when its value
+// changes (see "Unchanged frames" at the top).
+struct DotOverlay {
+    vr::VROverlayHandle_t h = vr::k_ulOverlayHandleInvalid;
+    int shown = -1;  // unknown at first
+    float alpha = -1, width = -1;
+    bool placed = false;
+    Vec3 at, eye;
+    void Show(bool on) {
+        if (shown == int(on)) return;
+        shown = on;
+        if (on) vr::VROverlay()->ShowOverlay(h);
+        else vr::VROverlay()->HideOverlay(h);
+    }
+    void Alpha(float a) {
+        if (a != alpha) vr::VROverlay()->SetOverlayAlpha(h, alpha = a);
+    }
+    // Width w, at `to`, facing `from`: left as it is for a change under 0.5% in width, and while
+    // it moved less than 0.2 mm and the eye less than 5 mm.
+    void Place(Vec3 to, Vec3 from, float w) {
+        if (std::fabs(w - width) > 0.005f * width) vr::VROverlay()->SetOverlayWidthInMeters(h, width = w);
+        if (placed && Length(to - at) < 0.0002 && Length(from - eye) < 0.005) return;
+        placed = true, at = to, eye = from;
+        const auto m = Billboard(to, from);
+        vr::VROverlay()->SetOverlayTransformAbsolute(h, vr::TrackingUniverseStanding, &m);
+    }
+};
+
 vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye) {
     // Overlay faces +Z; point +Z at the eye, keep +Y roughly up.
     const Vec3 z = Normalize(eye - at);
@@ -802,6 +887,8 @@ int main() {
     below.m[1][3] = -50;
     overlay->SetOverlayTransformTrackedDeviceRelative(laserMode, vr::k_unTrackedDeviceIndex_Hmd, &below);
     bool laserModeShown = false;
+    DotOverlay cursorDot, markerDot;  // the setters for `cursor` and `marker` (see "Unchanged frames" at the top)
+    cursorDot.h = cursor, markerDot.h = marker;
     // Controller beams keep the user's width; nothing here changes it any more.
     vr::VRSettings()->SetFloat("dashboard", "laserRayWidthScale", laserWidth);
 
@@ -821,12 +908,34 @@ int main() {
     std::map<std::string, bool> sceneGraph;  // no texture: plane test instead of ComputeOverlayIntersection
     std::map<std::string, bool> visible;     // refreshed every 50 ms
     auto lastVisible = std::chrono::steady_clock::now();
+    auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10);
+    unsigned listGeneration = 0;  // the overlay list the handles were looked up from
     // The plane of the last panel the cursor was on, and the last point on it (panel edges).
     Vec3 edgePoint, edgeNormal, edgeLast;
     std::string edgeKey;
 
     bool active = false, recenter = false, anchored = false;
     using Clock = std::chrono::steady_clock;
+    // Unchanged frames (see the top): visibleGen goes up when an overlay shows or hides or the
+    // handles change; pass is the last collision pass, posed the last plain pose sent.
+    unsigned visibleGen = 0;
+    struct Pass {
+        bool valid = false;
+        Clock::time_point at{};
+        unsigned gen = 0;
+        double yaw = 0, pitch = 0;
+        Vec3 anchor, eye;
+        double best = 1e9, distance = 0;
+        std::string key;
+        bool scene = false, onEdge = false, occluded = false;
+        Vec3 point;
+    } pass;
+    struct Posed {
+        bool valid = false;
+        Clock::time_point at{};
+        Vec3 origin;
+        double yaw = 0, pitch = 0;
+    } posed;
     Clock::time_point lastMouse{}, claimAt{}, claimRelease{}, wokeAt{}, noWakeUntil{};
     bool claimPending = false, claimHeld = false;
     // Last used wins: since when each controller has been moving (zero: it isn't).
@@ -863,6 +972,9 @@ int main() {
         wokeAt = t;
         active = true;
         recenter = true;
+        // Overlay handles and visibility weren't looked at while it was off (see "Idle" at the top).
+        lastSlow = lastVisible = t - std::chrono::seconds(10);
+        pass.valid = posed.valid = false;
         SendTo(out, "ft_pointer", "role " + role);  // POINTER_ROLE, before it takes it
         SendTo(out, "ft_pointer", "show");
         claimPending = true;  // take the laser without clicking, once SteamVR has bound the device
@@ -931,8 +1043,8 @@ int main() {
     Clock::time_point handUsed{};  // a gesture began then (keeps the pointer, like gaze mode)
     Clock::time_point lastTyping{};  // the relay's last "typing": a key on a keyboard
     // Gaze mode outside games, and its dot (see the top): lastMove/lastHeld/pulseAt.
-    bool inGame = false, gazeAwake = false;
-    Clock::time_point inGameAt{}, gazeAwakeAt{};
+    bool inGame = false, gazeAwake = false, gameSent = false;
+    Clock::time_point inGameAt{}, gazeAwakeAt{}, gameSentAt{};
     Clock::time_point lastMove{}, lastHeld{}, pulseAt{};
     // The left button, as sent to the driver; pressRight: the next press is the right button
     // instead (gaze_right), heldButton: the one pressed.
@@ -964,6 +1076,8 @@ int main() {
             if (debug) std::fflush(stdout);
         }
         nudging = confirmLesson = false;
+        // A click on nothing: maybe on an overlay that came up since the list was read.
+        if (lastHit.empty()) overlays.Kick();
         leftHeld = true;
         dragDistance = lastDistance;
         pressKey.clear();
@@ -1087,7 +1201,6 @@ int main() {
         return gazeOn && gazeMouseHeld && !inGame && !headsetOff && Clock::now() - gz.at < std::chrono::seconds(1) &&
                !aimHeld && !leftHeld && !tilting && hold.src == Src::None && !clickPress && !clickRelease;
     };
-    auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10);
 
     // --- Panel placement (see "Placement" at the top of the file) ---
     // Device pose, given in the standing universe, sent to the driver in raw space.
@@ -1103,6 +1216,7 @@ int main() {
         char msg[200];
         std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", o.x, o.y, o.z, q[0], q[1], q[2], q[3]);
         SendTo(out, "ft_pointer", msg);
+        posed.valid = false;
     };
     auto sleepMs = [](int ms) { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); };
     auto headPos = [&] {
@@ -1114,8 +1228,8 @@ int main() {
     auto borrow = [&] {
         SendTo(out, "ft_pointer", "show");
         overlay->ShowOverlay(laserMode);
-        overlay->HideOverlay(cursor);
-        overlay->HideOverlay(marker);
+        cursorDot.Show(false);
+        markerDot.Show(false);
         sleepMs(active ? 50 : 400);  // a fresh connect needs SteamVR to bind the device
         SendTo(out, "ft_pointer", "btn a 1");
         sleepMs(60);
@@ -1242,6 +1356,36 @@ int main() {
         return msg;
     };
 
+    // "move <dyaw> <dpitch>" from the relay.
+    auto mouseMove = [&](double a, double b) {
+        // A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using
+        // the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back.
+        const bool heldBack = mouseMoveHeld();
+        if (!heldBack) lastMouse = Clock::now();
+        // Any mouse input wakes the pointer (after a controller took over, or a helper restart).
+        if (!active) wake(Clock::now());
+        if (heldBack) return;
+        if (tilting) {
+            tiltYaw += a;
+            tiltPitch = std::clamp(tiltPitch + b, -80.0, 80.0);
+            return;
+        }
+        lastMove = Clock::now();  // the dot shows while the mouse moves it (gaze mode)
+        if (gazeOn && gazeOwns) {
+            // The mouse takes the pointer from the gaze, from where the gaze left it.
+            gazeOwns = false;
+            nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
+            nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
+            nudgeAt = Clock::now(), nudgeMoved = 0;
+        }
+        if (nudging || aimHeld) nudgeMoved += std::hypot(a, b);
+        if (!anchored) recenter = true;
+        yaw += a;
+        while (yaw > 180) yaw -= 360;
+        while (yaw < -180) yaw += 360;
+        pitch = std::clamp(pitch + b, -85.0, 85.0);
+    };
+
     std::printf("ft-pointer running: free distance %.2f m, dot %.2f deg\n", freeDistance, cursorDeg);
     std::fflush(stdout);
 
@@ -1256,8 +1400,8 @@ int main() {
         if (headsetOff && active) {
             active = false;
             claimPending = claimHeld = false;
-            overlay->HideOverlay(cursor);
-            overlay->HideOverlay(marker);
+            cursorDot.Show(false);
+            markerDot.Show(false);
             SendTo(out, "ft_pointer", "btn a 0");
             SendTo(out, "ft_pointer", "hide");
             std::printf("headset off: pointer released\n");
@@ -1283,6 +1427,13 @@ int main() {
                 gazeAwakeAt = t;
                 SendTo(out, "frametop_relay", awake ? "gazeawake 1" : "gazeawake 0");
             }
+            // The relay pauses Frametop for VR games (input/game_pause.py). Repeated, so a relay
+            // that restarts learns it, and silence (SteamVR gone) ends the game there.
+            if (inGame != gameSent || t - gameSentAt > std::chrono::seconds(5)) {
+                gameSent = inGame;
+                gameSentAt = t;
+                SendTo(out, "frametop_relay", inGame ? "vrgame 1" : "vrgame 0");
+            }
         }
 
         // Commands from the relay.
@@ -1300,6 +1451,12 @@ int main() {
                 if (senderLen > offsetof(sockaddr_un, sun_path))
                     sendto(out, msg.data(), msg.size(), 0, reinterpret_cast<const sockaddr *>(&sender), senderLen);
             };
+            // Mouse moves first, the most frequent command.
+            double mx, my;
+            if (std::strncmp(buf, "move ", 5) == 0 && std::sscanf(buf + 5, "%lf %lf", &mx, &my) == 2) {
+                mouseMove(mx, my);
+                continue;
+            }
             // The gaze, from ft-gazed: not mouse input, it never wakes the pointer.
             double g[4];
             if (std::sscanf(buf, "gz %lf %lf %lf %lf", &g[0], &g[1], &g[2], &g[3]) == 4) {
@@ -1380,19 +1537,19 @@ int main() {
                     std::printf("gaze mode %s\n", gazeOn ? "on" : "off");
                     std::fflush(stdout);
                 }
-                reply(gazeOn ? "ok on" : "ok off");
+                // "gaze ? headset" (the gaze service, which idles while nobody wears the headset)
+                // also says whether someone does.
+                if (std::strstr(arg, "headset"))
+                    reply(gazeOn ? (headsetOff ? "ok on away" : "ok on worn") : (headsetOff ? "ok off away" : "ok off worn"));
+                else
+                    reply(gazeOn ? "ok on" : "ok off");
                 continue;
             }
-            const bool mouseInput = std::strncmp(buf, "move", 4) == 0 || std::strncmp(buf, "btn", 3) == 0 ||
-                                    std::strncmp(buf, "scroll", 6) == 0;
-            // A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using
-            // the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back.
-            const bool moveHeldBack = std::strncmp(buf, "move", 4) == 0 && mouseMoveHeld();
-            if (mouseInput && !moveHeldBack) lastMouse = Clock::now();
+            // (Moves were taken first, above.)
+            const bool mouseInput = std::strncmp(buf, "btn", 3) == 0 || std::strncmp(buf, "scroll", 6) == 0;
+            if (mouseInput) lastMouse = Clock::now();
             // Any mouse input wakes the pointer (after a controller took over, or a helper restart).
             if (!active && mouseInput) wake(Clock::now());
-            if (moveHeldBack) continue;
-            double a, b;
             char key[128];
             double px, py, pz, pyaw, ppitch, proll = 0, pgrab = -1;
             if (std::sscanf(buf, "grabprobe %127s", key) == 1) {
@@ -1451,31 +1608,12 @@ int main() {
                 tilting = swallowedRight = false;
                 continue;
             }
-            if (tilting && std::sscanf(buf, "move %lf %lf", &a, &b) == 2) {
-                tiltYaw += a;
-                tiltPitch = std::clamp(tiltPitch + b, -80.0, 80.0);
-                continue;
-            }
-            if (std::sscanf(buf, "move %lf %lf", &a, &b) == 2) {
-                lastMove = Clock::now();  // the dot shows while the mouse moves it (gaze mode)
-                if (gazeOn && gazeOwns) {
-                    // The mouse takes the pointer from the gaze, from where the gaze left it.
-                    gazeOwns = false;
-                    nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
-                    nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
-                    nudgeAt = Clock::now(), nudgeMoved = 0;
-                }
-                if (nudging || aimHeld) nudgeMoved += std::hypot(a, b);
-                if (!anchored) recenter = true;
-                yaw += a;
-                while (yaw > 180) yaw -= 360;
-                while (yaw < -180) yaw += 360;
-                pitch = std::clamp(pitch + b, -85.0, 85.0);
-            } else if (std::strncmp(buf, "recenter", 8) == 0) {
+            if (std::strncmp(buf, "recenter", 8) == 0) {
                 recenter = true;
             } else if (std::strncmp(buf, "reload", 6) == 0) {
                 loadConfig();
                 lastSlow = Clock::now() - std::chrono::seconds(10);  // apply POINTER_IGNORE now
+                pass.valid = false;
                 std::printf("reloaded: free distance %.2f m, dot %.2f deg, origin %.2f, head follow %s, leash %.0f deg, "
                             "controller pickup %.1fx, %zu ignored\n",
                             freeDistance, cursorDeg, originFraction, follow ? "on" : "off", leashDeg, pickupScale,
@@ -1493,8 +1631,8 @@ int main() {
                 if (!active) wake(Clock::now());
             } else if (std::strncmp(buf, "hide", 4) == 0) {
                 active = false;
-                overlay->HideOverlay(cursor);
-                overlay->HideOverlay(marker);
+                cursorDot.Show(false);
+                markerDot.Show(false);
                 SendTo(out, "ft_pointer", "hide");
             } else {
                 SendTo(out, "ft_pointer", buf);  // btn, scroll
@@ -1536,8 +1674,8 @@ int main() {
             sys->GetControllerRoleForTrackedDeviceIndex(ours) == vr::TrackedControllerRole_Invalid) {
             active = false;
             claimPending = claimHeld = false;
-            overlay->HideOverlay(cursor);
-            overlay->HideOverlay(marker);
+            cursorDot.Show(false);
+            markerDot.Show(false);
             SendTo(out, "ft_pointer", "btn a 0");
             SendTo(out, "ft_pointer", "hide");
             noWakeUntil = tnow + std::chrono::seconds(2);
@@ -1565,8 +1703,8 @@ int main() {
                 if (tnow - movingSince[i] >= std::chrono::milliseconds(100)) {
                     active = false;
                     claimPending = claimHeld = false;
-                    overlay->HideOverlay(cursor);
-                    overlay->HideOverlay(marker);
+                    cursorDot.Show(false);
+                    markerDot.Show(false);
                     SendTo(out, "ft_pointer", "btn a 0");
                     SendTo(out, "ft_pointer", "hide");
                     std::printf("controller %u moved (%.2f m/s, %.1f rad/s): pointer released\n", i, speed, spin);
@@ -1616,7 +1754,7 @@ int main() {
         }
 
         // Hands (see the top): pinches and grips from ft-hands.
-        {
+        if (handsOn) {
             vr::TrackedDevicePose_t h0;
             sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &h0, 1);
             if (h0.bPoseIsValid) poses.Add(tnow, h0.mDeviceToAbsoluteTracking);
@@ -1915,10 +2053,13 @@ int main() {
         }
         followAt = tnow;
 
-        // Slow work, once a second: overlay handles, our device index, laser width.
+        // Slow work, once a second and when the overlay list is new: overlay handles, our device
+        // index. Only while the pointer is awake (see "Idle" at the top).
         const auto now = std::chrono::steady_clock::now();
-        if (now - lastSlow > std::chrono::seconds(1)) {
+        if (active && (now - lastSlow > std::chrono::seconds(1) || overlays.Generation() != listGeneration)) {
             lastSlow = now;
+            listGeneration = overlays.Generation();
+            const auto before = handles;
             handles.clear();
             for (const auto &key : overlays.Keys()) {
                 if (Ignored(ignore, key)) continue;
@@ -1938,19 +2079,21 @@ int main() {
                 sceneGraph[key] = (tw == 0 || th == 0) && tt == vr::VROverlayTransform_Absolute &&
                                   key.rfind("frametop.", 0) != 0;
             }
-            ours = vr::k_unTrackedDeviceIndexInvalid;
-            for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i) {
+            if (handles != before) ++visibleGen;
+            // Our device keeps its index; it's looked for again when a device is activated or
+            // deactivated (the events below).
+            for (vr::TrackedDeviceIndex_t i = 0; ours == vr::k_unTrackedDeviceIndexInvalid && i < vr::k_unMaxTrackedDeviceCount; ++i) {
                 char type[64] = "";
                 sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type);
                 if (std::strcmp(type, "ft_pointer") == 0) ours = i;
             }
-
         }
 
-        if (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size()) {
+        if (active && (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size())) {
             lastVisible = now;
-            visible.clear();
-            for (const auto &[key, h] : handles) visible[key] = overlay->IsOverlayVisible(h);
+            std::map<std::string, bool> seen;
+            for (const auto &[key, h] : handles) seen[key] = overlay->IsOverlayVisible(h);
+            if (seen != visible) visible.swap(seen), ++visibleGen;
         }
 
         if (active && anchored && hmd.bPoseIsValid && tilting) {
@@ -1960,8 +2103,8 @@ int main() {
                 tiltOrigin = lastOrigin;
                 tiltBasis = AimBasis(lastAim);
                 tiltStart = false;  // angles carry on from any earlier tilt in this drag
-                overlay->HideOverlay(cursor);
-                overlay->HideOverlay(marker);
+                cursorDot.Show(false);
+                markerDot.Show(false);
             }
             const double yr = tiltYaw * M_PI / 180, pr = tiltPitch * M_PI / 180;
             const Vec3 up{0, 1, 0}, side = tiltBasis.x;
@@ -1977,6 +2120,7 @@ int main() {
             std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", originRaw.x, originRaw.y,
                           originRaw.z, q[0], q[1], q[2], q[3]);
             SendTo(out, "ft_pointer", msg);
+            posed.valid = false;
         } else if (active && anchored && hmd.bPoseIsValid) {
             const bool dragging = leftHeld || tnow < dropHoldUntil;
             if (!dragging) tiltYaw = tiltPitch = 0;  // drop finished: back to plain pointing
@@ -2020,85 +2164,98 @@ int main() {
                 }
                 return h;
             };
-            Hit first;
-            if (!dragging) first = nearest(anchor, dir);
-            double best = first.along;
-            std::string bestKey = first.key;
-            bool bestScene = first.scene;
-            Vec3 bestPoint = first.point, bestNormal = first.normal;
-            bool onEdge = false;
-            if (!dragging && best < 1e8 && !bestScene) {
-                edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint;
-            } else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible[edgeKey]) {
-                // Just off a panel: stay on its plane (see "Panel edges" at the top).
-                const double denom = Dot(dir, edgeNormal);
-                if (std::fabs(denom) > 1e-4) {
-                    const double along = Dot(edgePoint - anchor, edgeNormal) / denom;
-                    const Vec3 at = anchor + dir * along;
-                    if (along > 0.05 && std::sqrt(Dot(at - edgeLast, at - edgeLast)) <= edgeReach) {
-                        best = along;
-                        bestKey = edgeKey;
-                        onEdge = true;
+            // Unchanged frames (see the top): the last pass still holds.
+            const bool reuse = !dragging && pass.valid && tnow - pass.at < std::chrono::milliseconds(100) &&
+                               pass.gen == visibleGen && std::fabs(yaw - pass.yaw) < 1e-6 &&
+                               std::fabs(pitch - pass.pitch) < 1e-6 && Length(anchor - pass.anchor) < 0.001 &&
+                               Length(eye - pass.eye) < 0.005;
+            double best = pass.best, distance = pass.distance;
+            std::string bestKey = pass.key;
+            bool bestScene = pass.scene, onEdge = pass.onEdge, occluded = pass.occluded;
+            Vec3 point = pass.point;
+            if (!reuse) {
+                Hit first;
+                if (!dragging) first = nearest(anchor, dir);
+                best = first.along;
+                bestKey = first.key;
+                bestScene = first.scene;
+                Vec3 bestPoint = first.point, bestNormal = first.normal;
+                onEdge = false;
+                if (!dragging && best < 1e8 && !bestScene) {
+                    edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint;
+                } else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible.count(edgeKey) &&
+                           visible.at(edgeKey)) {
+                    // Just off a panel: stay on its plane (see "Panel edges" at the top).
+                    const double denom = Dot(dir, edgeNormal);
+                    if (std::fabs(denom) > 1e-4) {
+                        const double along = Dot(edgePoint - anchor, edgeNormal) / denom;
+                        const Vec3 at = anchor + dir * along;
+                        if (along > 0.05 && std::sqrt(Dot(at - edgeLast, at - edgeLast)) <= edgeReach) {
+                            best = along;
+                            bestKey = edgeKey;
+                            onEdge = true;
+                        }
                     }
                 }
-            }
-            // Held on one of ft-screens' panels that stays where it is: onto whichever of them
-            // the ray meets (see the top).
-            if (leftHeld && !pressKey.empty()) {
-                vr::ETrackingUniverseOrigin uo;
-                vr::HmdMatrix34_t now{};
-                auto it = handles.find(pressKey);
-                bool still = it != handles.end() &&
-                             overlay->GetOverlayTransformAbsolute(it->second, &uo, &now) == vr::VROverlayError_None;
-                for (int i = 0; still && i < 3; ++i)
-                    for (int j = 0; j < 4; ++j)
-                        if (std::fabs(now.m[i][j] - pressPose.m[i][j]) > 0.001f) still = false;
-                if (still) {
-                    Hit h;
-                    for (const auto &[key, handle] : handles) {
-                        if (!visible[key] || !FramePanel(key)) continue;
-                        vr::VROverlayIntersectionParams_t params{};
-                        params.vSource = {float(anchor.x), float(anchor.y), float(anchor.z)};
-                        params.vDirection = {float(dir.x), float(dir.y), float(dir.z)};
-                        params.eOrigin = vr::TrackingUniverseStanding;
-                        vr::VROverlayIntersectionResults_t r{};
-                        if (overlay->ComputeOverlayIntersection(handle, &params, &r) && r.fDistance > 0.05f &&
-                            r.fDistance < h.along)
-                            h.along = r.fDistance, h.key = key;
+                // Held on one of ft-screens' panels that stays where it is: onto whichever of them
+                // the ray meets (see the top).
+                if (leftHeld && !pressKey.empty()) {
+                    vr::ETrackingUniverseOrigin uo;
+                    vr::HmdMatrix34_t now{};
+                    auto it = handles.find(pressKey);
+                    bool still = it != handles.end() &&
+                                 overlay->GetOverlayTransformAbsolute(it->second, &uo, &now) == vr::VROverlayError_None;
+                    for (int i = 0; still && i < 3; ++i)
+                        for (int j = 0; j < 4; ++j)
+                            if (std::fabs(now.m[i][j] - pressPose.m[i][j]) > 0.001f) still = false;
+                    if (still) {
+                        Hit h;
+                        for (const auto &[key, handle] : handles) {
+                            if (!visible[key] || !FramePanel(key)) continue;
+                            vr::VROverlayIntersectionParams_t params{};
+                            params.vSource = {float(anchor.x), float(anchor.y), float(anchor.z)};
+                            params.vDirection = {float(dir.x), float(dir.y), float(dir.z)};
+                            params.eOrigin = vr::TrackingUniverseStanding;
+                            vr::VROverlayIntersectionResults_t r{};
+                            if (overlay->ComputeOverlayIntersection(handle, &params, &r) && r.fDistance > 0.05f &&
+                                r.fDistance < h.along)
+                                h.along = r.fDistance, h.key = key;
+                        }
+                        if (h.along < 1e8) dragDistance = h.along, lastHit = h.key;
+                    } else {
+                        pressKey.clear();  // carried: the lock holds for the rest of this press
                     }
-                    if (h.along < 1e8) dragDistance = h.along, lastHit = h.key;
-                } else {
-                    pressKey.clear();  // carried: the lock holds for the rest of this press
                 }
-            }
-            // While dragging: keep the press-time distance and show the non-interactive marker.
-            // On a scene-graph plane or a panel's edge: the laser-catching dot goes 5 cm behind it.
-            double distance = dragging ? dragDistance : (best < 1e8 ? best : freeDistance);
-            Vec3 point = anchor + dir * distance;
-            bool occluded = false;
-            if (!dragging) {
-                // The cursor lands on what you see under it: the ray above starts at the anchor,
-                // not the eye, so after leaning it can pick a panel that something nearer
-                // covers from where you are now (panels close together in view, at different
-                // depths). Anything in front of the point on the eye's line of sight wins.
-                const double toPoint = std::sqrt(Dot(point - eye, point - eye));
-                const Hit front = nearest(eye, Normalize(point - eye));
-                if (front.along < toPoint - 0.02) {
-                    occluded = true;
-                    bestKey = front.key, bestScene = front.scene;
-                    point = front.point;
-                    if (!front.scene) edgeKey = front.key, edgePoint = front.point, edgeNormal = Normalize(front.normal),
-                                      edgeLast = front.point;
-                    distance = std::sqrt(Dot(point - anchor, point - anchor));
-                    best = distance;
-                    onEdge = false;
+                // While dragging: keep the press-time distance and show the non-interactive marker.
+                // On a scene-graph plane or a panel's edge: the laser-catching dot goes 5 cm behind it.
+                distance = dragging ? dragDistance : (best < 1e8 ? best : freeDistance);
+                point = anchor + dir * distance;
+                occluded = false;
+                if (!dragging) {
+                    // The cursor lands on what you see under it: the ray above starts at the anchor,
+                    // not the eye, so after leaning it can pick a panel that something nearer
+                    // covers from where you are now (panels close together in view, at different
+                    // depths). Anything in front of the point on the eye's line of sight wins.
+                    const double toPoint = std::sqrt(Dot(point - eye, point - eye));
+                    const Hit front = nearest(eye, Normalize(point - eye));
+                    if (front.along < toPoint - 0.02) {
+                        occluded = true;
+                        bestKey = front.key, bestScene = front.scene;
+                        point = front.point;
+                        if (!front.scene)
+                            edgeKey = front.key, edgePoint = front.point, edgeNormal = Normalize(front.normal),
+                            edgeLast = front.point;
+                        distance = std::sqrt(Dot(point - anchor, point - anchor));
+                        best = distance;
+                        onEdge = false;
+                    }
+                    lastDistance = distance, lastHit = bestKey;
                 }
-                lastDistance = distance, lastHit = bestKey;
             }
             const bool onScene = !dragging && ((bestScene && best < 1e8) || onEdge);
             const bool onPanel = dragging || (best < 1e8 && !onScene);
             const Vec3 sight = Normalize(point - eye);
-            if (!dragging) {
+            if (!dragging && !reuse) {
                 // SteamVR Settings (see the top): the dashboard's main panel is hidden and its
                 // scene-graph panel shows the page.
                 onVrSettings = false;
@@ -2114,6 +2271,10 @@ int main() {
                     }
                 }
             }
+            if (!reuse) {
+                pass = {!dragging, tnow, visibleGen, yaw, pitch, anchor, eye,
+                        best, distance, bestKey, bestScene, onEdge, occluded, point};
+            }
             if (onVrSettings != catcherHidesHit) {
                 overlay->SetOverlayFlag(cursor, vr::VROverlayFlags_HideLaserIntersection, onVrSettings);
                 catcherHidesHit = onVrSettings;
@@ -2130,20 +2291,16 @@ int main() {
                     alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
                 }
                 if (tnow < calPanelUntil) alpha = 0;
-                overlay->SetOverlayAlpha(marker, float(alpha));
-                overlay->SetOverlayWidthInMeters(marker, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360)));
-                auto mm = Billboard(near, eye);
-                overlay->SetOverlayTransformAbsolute(marker, vr::TrackingUniverseStanding, &mm);
-                overlay->SetOverlayAlpha(cursor, 0);
-                overlay->SetOverlayWidthInMeters(cursor, float(2 * SETTINGS_CATCHER * std::tan(cursorDeg * M_PI / 360)));
-                auto mc = Billboard(far, eye);
-                overlay->SetOverlayTransformAbsolute(cursor, vr::TrackingUniverseStanding, &mc);
-                overlay->ShowOverlay(marker);
-                overlay->ShowOverlay(cursor);
+                markerDot.Alpha(float(alpha));
+                markerDot.Place(near, eye, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360)));
+                cursorDot.Alpha(0);
+                cursorDot.Place(far, eye, float(2 * SETTINGS_CATCHER * std::tan(cursorDeg * M_PI / 360)));
+                markerDot.Show(true);
+                cursorDot.Show(true);
             } else {
                 // On a panel: the non-interactive marker, pulled 5 mm toward the eye so it
                 // draws on top. In free space: the interactive dot the laser lands on.
-                const vr::VROverlayHandle_t show = onPanel ? marker : cursor, hide = onPanel ? cursor : marker;
+                DotOverlay &show = onPanel ? markerDot : cursorDot, &hide = onPanel ? cursorDot : markerDot;
                 const Vec3 at = onPanel ? point + Normalize(eye - point) * 0.005 : onScene ? point + dir * 0.05 : point;
                 const double dist = std::sqrt(Dot(at - eye, at - eye));
                 // Gaze mode: a pulse for each click, and with POINTER_GAZE_DOT=moving, shown only
@@ -2160,12 +2317,10 @@ int main() {
                     }
                 }
                 if (tnow < calPanelUntil) alpha = 0;  // the calibration panel is up (see the top)
-                overlay->SetOverlayAlpha(show, float(alpha));
-                overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360)));
-                auto m = Billboard(at, eye);
-                overlay->SetOverlayTransformAbsolute(show, vr::TrackingUniverseStanding, &m);
-                overlay->ShowOverlay(show);
-                overlay->HideOverlay(hide);
+                show.Alpha(float(alpha));
+                show.Place(at, eye, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360)));
+                show.Show(true);
+                hide.Show(false);
             }
 
             // Controller ray: from the eye, aimed at the cursor point, converted from the
@@ -2210,10 +2365,15 @@ int main() {
                 std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", oRaw.x, oRaw.y, oRaw.z, q[0],
                               q[1], q[2], q[3]);
                 SendTo(out, "ft_pointer", msg);
-            } else {
+                posed.valid = false;
+            } else if (!posed.valid || tnow - posed.at >= std::chrono::milliseconds(100) ||
+                       Length(eyeRaw - posed.origin) >= 0.0002 || std::fabs(std::remainder(ayaw - posed.yaw, 360.0)) >= 0.04 ||
+                       std::fabs(apitch - posed.pitch) >= 0.04) {
+                // The driver keeps the last pose: only a change goes out (see "Unchanged frames" at the top).
                 char msg[160];
                 std::snprintf(msg, sizeof msg, "pose %.5f %.5f %.5f %.4f %.4f", eyeRaw.x, eyeRaw.y, eyeRaw.z, ayaw, apitch);
                 SendTo(out, "ft_pointer", msg);
+                posed = {true, tnow, eyeRaw, ayaw, apitch};
             }
         }
 
@@ -2262,6 +2422,12 @@ int main() {
 
         vr::VREvent_t ev;
         while (sys->PollNextEvent(&ev, sizeof ev)) {
+            // The dashboard's overlays come and go with it, and a game brings its own.
+            if (ev.eventType == vr::VREvent_DashboardActivated || ev.eventType == vr::VREvent_DashboardDeactivated ||
+                ev.eventType == vr::VREvent_DashboardOverlayCreated || ev.eventType == vr::VREvent_SceneApplicationChanged)
+                overlays.Kick();
+            if (ev.eventType == vr::VREvent_TrackedDeviceActivated || ev.eventType == vr::VREvent_TrackedDeviceDeactivated)
+                ours = vr::k_unTrackedDeviceIndexInvalid;
             if (ev.eventType == vr::VREvent_Quit) {
                 sys->AcknowledgeQuit_Exiting();
 
@@ -2270,6 +2436,15 @@ int main() {
                 return 0;
             }
         }
-        std::this_thread::sleep_for(std::chrono::milliseconds(8));
+        // Idle (see the top): wait for a command instead. Not with anything still to finish (the
+        // claim pulse, a click's release, a press).
+        const bool idle = !active && !handsOn && !claimPending && !claimHeld && !clickRelease && !leftHeld &&
+                          hold.src == Src::None;
+        if (idle) {
+            pollfd p{in, POLLIN, 0};
+            poll(&p, 1, controllerButtons.Watching(inGame) ? 20 : 250);
+        } else {
+            std::this_thread::sleep_for(std::chrono::milliseconds(8));
+        }
     }
 }

pointer/helper/vrbuttons.h

@@ -111,6 +111,14 @@ class ControllerButtons {
         }
     }
 
+    // Whether Poll reads any button now (the main loop can't sleep long while it does).
+    bool Watching(bool inGame) const {
+        if (!ok_ || (inGame && !games_ && !capture_)) return false;
+        for (int i = 0; i < kCount; ++i)
+            if (bound_[i]) return true;
+        return false;
+    }
+
     // {"manifest":true,"global":false,"bound":[...],"active":[...]}: active = bound and
     // delivered (a controller that has the button is on, and SteamVR lets us have it).
     std::string Status() const {

screens/compositor.c

@@ -58,6 +58,7 @@
 #include <wlr/util/log.h>
 
 #include "vr.h"
+#include "controller-click.h"
 
 #define MAX_SCREENS 24  // screens and spare outputs
 
@@ -104,6 +105,7 @@ struct server {
     struct wl_list buffers;  // tracked_buffer
     struct wl_event_source *tick;
     struct screen *pointer_focus;
+    struct ft_controller_click controller_click;
     pid_t child;
     // Where typing goes: the screens after a click on one, Steam after a click on another
     // panel. The input relay grabs the keyboards while it's the screens (see keys_update).
@@ -292,6 +294,11 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
         return;
     }
     if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
+    struct ft_event filtered = *e;
+    if (e->screen < s->n_config &&
+        !ft_controller_click_filter(&s->controller_click, &filtered, s->scale[e->screen])) return;
+    e = &filtered;
+    if (e->screen < 0 || e->screen >= MAX_SCREENS || !s->screens[e->screen]) return;
     struct screen *sc = s->screens[e->screen];
     struct wlr_surface *surface = sc->toplevel->base->surface;
     const uint32_t t = now_ms();
@@ -374,7 +381,10 @@ static void keys_update(struct server *s) {
            offsetof(struct sockaddr_un, sun_path) + 1 + sizeof name - 1);
 }
 
-// Every ~11 ms (90 Hz): SteamVR events, and frame callbacks for screens that committed.
+// Every ~11 ms (90 Hz): SteamVR events, and frame callbacks for screens that committed. While
+// Frametop is paused for a VR game (everything hidden), every 100 ms, and the frame callbacks
+// once a second: KWin draws a screen only after its callback, and its apps wait for theirs, so
+// the desktop hardly draws until it's resumed.
 static int tick(void *data) {
     struct server *s = data;
     ft_vr_poll(handle_vr_event, s);
@@ -388,14 +398,15 @@ static int tick(void *data) {
     }
     struct timespec now;
     clock_gettime(CLOCK_MONOTONIC, &now);
-    for (int i = 0; i < MAX_SCREENS; ++i) {
+    const bool paused = ft_vr_paused();
+    for (int i = 0; i < MAX_SCREENS && (!paused || s->ticks % 10 == 0); ++i) {
         struct screen *sc = s->screens[i];
         if (sc && sc->frame_pending) {
             sc->frame_pending = false;
             wlr_surface_send_frame_done(sc->toplevel->base->surface, &now);
         }
     }
-    wl_event_source_timer_update(s->tick, 11);
+    wl_event_source_timer_update(s->tick, paused ? 100 : 11);
     return 0;
 }
 
@@ -514,7 +525,18 @@ static int control_readable(int fd, uint32_t mask, void *data) {
         unsigned code;
         int value, index, w, h;
         double scale;
-        if (sscanf(buf, "size %d %d %d", &index, &w, &h) == 3) {
+        char tail;
+        if (strcmp(buf, "controller-click?") == 0) {
+            snprintf(reply, sizeof reply,
+                "{\"supported\":true,\"threshold\":%.3f,\"held\":%s,\"dragging\":%s,\"suppressedMotions\":%lu,\"clicks\":%lu,\"drags\":%lu}",
+                s->controller_click.threshold, s->controller_click.held ? "true" : "false",
+                s->controller_click.dragging ? "true" : "false", s->controller_click.suppressed,
+                s->controller_click.clicks, s->controller_click.drags);
+        } else if (sscanf(buf, "controller-click %lf %c", &scale, &tail) == 1) {
+            if (!isfinite(scale) || scale < 0 || scale > 64 || s->controller_click.buttons)
+                snprintf(reply, sizeof reply, "error threshold or held controller button");
+            else { s->controller_click.threshold = scale; snprintf(reply, sizeof reply, "ok"); }
+        } else if (sscanf(buf, "size %d %d %d", &index, &w, &h) == 3) {
             // A new resolution for a screen, live: KWin resizes the screen to match. (KWin makes
             // it this size times its scale; ft-floatd sends spares' sizes divided by theirs.)
             const int min_w = index - 1 < s->n_config ? 320 : 64, min_h = index - 1 < s->n_config ? 200 : 64;
@@ -641,6 +663,7 @@ static bool setup_dmabuf(struct server *s) {
 
 int main(int argc, char **argv) {
     struct server s = {0};
+    s.controller_click.threshold = 8;
     for (int i = 0; i < MAX_SCREENS; ++i) s.scale[i] = 1;
     s.kb_screen = -1;
     const char *socket_name = "ft-screens-0", *control_name = "ft_screens";

screens/controller-click.h

@@ -0,0 +1,46 @@
+#ifndef FT_CONTROLLER_CLICK_H
+#define FT_CONTROLLER_CLICK_H
+#include <math.h>
+#include <linux/input-event-codes.h>
+#include "vr.h"
+struct ft_controller_click {
+    bool held, dragging;
+    int screen;
+    uint32_t buttons;
+    double x, y, threshold, radius;
+    unsigned long suppressed, clicks, drags;
+};
+// Hold the desktop position at press time until movement exceeds a logical-pixel
+// radius. No timer or delayed button-down. Only a hand controller's press starts
+// it: the 3D mouse drives a laser too, and its short drags must stay drags.
+static inline bool ft_controller_click_filter(struct ft_controller_click *c,
+                                              struct ft_event *e, double scale) {
+    if (e->type == FT_BUTTON && e->button >= BTN_LEFT && e->button < BTN_LEFT+8) {
+        uint32_t bit = 1u << (e->button-BTN_LEFT);
+        if (e->pressed) c->buttons |= bit; else c->buttons &= ~bit;
+    }
+    if (e->type == FT_BUTTON && e->pressed) {
+        if (e->button == BTN_LEFT && e->controller && !c->held && c->threshold > 0) {
+            c->held = true; c->dragging = false; c->screen = e->screen;
+            c->x = e->x; c->y = e->y; c->radius = c->threshold * scale;
+        } else if (e->button != BTN_LEFT) {
+            c->held = false; // Multi-button gestures retain their usual semantics.
+        }
+    } else if (e->type == FT_MOTION && c->held && !c->dragging) {
+        if (e->screen == c->screen && hypot(e->x-c->x, e->y-c->y) <= c->radius) {
+            ++c->suppressed;
+            return false;
+        }
+        c->dragging = true; ++c->drags;
+    } else if (e->type == FT_BUTTON && e->button == BTN_LEFT && !e->pressed && c->held) {
+        if (!c->dragging) {
+            e->screen = c->screen; e->x = c->x; e->y = c->y;
+            ++c->clicks;
+        }
+        c->held = false; c->dragging = false;
+    } else if (e->type == FT_LEAVE && c->held) {
+        return false; // Preserve the implicit grab through tiny edge excursions.
+    }
+    return true;
+}
+#endif

screens/tests/controller-click-test.c

@@ -0,0 +1,42 @@
+#include <assert.h>
+#include "../controller-click.h"
+static struct ft_event event(enum ft_event_type t, bool down, double x, double y) {
+    return (struct ft_event){.type=t,.screen=0,.button=BTN_LEFT,.pressed=down,.controller=true,.x=x,.y=y};
+}
+int main(void) {
+    struct ft_controller_click c={.threshold=8};
+    struct ft_event e=event(FT_BUTTON,true,100,100);
+    assert(ft_controller_click_filter(&c,&e,2));
+    e=event(FT_MOTION,false,112,108);
+    assert(!ft_controller_click_filter(&c,&e,2));
+    assert(c.held && !c.dragging && c.suppressed==1);
+    e=event(FT_BUTTON,false,116,111);
+    assert(ft_controller_click_filter(&c,&e,2));
+    assert(e.x==100 && e.y==100 && c.clicks==1 && !c.held);
+    e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
+    e=event(FT_MOTION,false,109,100);assert(ft_controller_click_filter(&c,&e,1));
+    assert(c.dragging && c.drags==1);
+    e=event(FT_MOTION,false,101,100);assert(ft_controller_click_filter(&c,&e,1));
+    e=event(FT_BUTTON,false,103,100);assert(ft_controller_click_filter(&c,&e,1));
+    assert(e.x==103 && c.clicks==1 && !c.held && !c.dragging);
+    e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
+    e=event(FT_LEAVE,false,0,0);assert(!ft_controller_click_filter(&c,&e,1));
+    e=event(FT_BUTTON,false,800,900);e.screen=1;ft_controller_click_filter(&c,&e,1);
+    assert(e.screen==0 && e.x==100 && c.clicks==2);
+    e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
+    e=event(FT_MOTION,false,101,100);e.screen=1;
+    assert(ft_controller_click_filter(&c,&e,1) && c.dragging);
+    e=event(FT_BUTTON,false,101,100);ft_controller_click_filter(&c,&e,1);
+    c.threshold=0;
+    e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
+    e=event(FT_MOTION,false,101,100);assert(ft_controller_click_filter(&c,&e,1) && !c.held);
+    c.threshold=8;
+    e=event(FT_BUTTON,true,100,100);ft_controller_click_filter(&c,&e,1);
+    e=event(FT_BUTTON,true,100,100);e.button=BTN_RIGHT;ft_controller_click_filter(&c,&e,1);
+    assert(!c.held);
+    // The 3D mouse's laser: a short drag stays a drag, and the release stays where it was.
+    e=event(FT_BUTTON,true,100,100);e.controller=false;assert(ft_controller_click_filter(&c,&e,1) && !c.held);
+    e=event(FT_MOTION,false,104,100);assert(ft_controller_click_filter(&c,&e,1));
+    e=event(FT_BUTTON,false,106,100);e.controller=false;
+    assert(ft_controller_click_filter(&c,&e,1) && e.x==106 && c.clicks==2 && c.suppressed==1);
+}

screens/vr.cpp

@@ -43,6 +43,10 @@
 //     own; also for flatscreen games, which aren't scene apps).
 //   - during a VR game the screens hide unless the dashboard is open (g_inGames, default),
 //     or stay visible over it; the hotkey still shows them.
+//   - paused ("pause on", from the input relay when Frametop pauses for a VR game,
+//     input/game_pause.py): every screen and floating window hides whatever the mode, the
+//     hotkey, or the dashboard says, and compositor.c gives KWin a frame callback once a
+//     second instead of 90 times, so KWin and its apps hardly draw. "pause off" undoes it.
 //   - hand cutouts (handcut.cpp): where ft-hands (hands/) tracks a hand between an eye and a
 //     screen, that eye sees through the screen (to Room View). Only then is the screen
 //     drawn by us, into a side-by-side buffer (one half per eye); otherwise its client
@@ -320,6 +324,7 @@ double g_gestureAngle = 20;  // gesture: look within this of the controller
 std::string g_gestureHand = "left";
 Lasers g_lasers = Lasers::OutsideGames;  // when controllers' lasers work the screens (see the top)
 bool g_gameRunning = false;              // a scene app (VR game) is running
+bool g_paused = false;                   // Frametop paused for a VR game: everything hidden (see the top)
 InGames g_inGames = InGames::Hide;       // during a VR game, the always mode acts like the dashboard mode
 
 // ---------------------------------------------------------------- chrome (bar, button, handle)
@@ -734,6 +739,7 @@ void UpdateGame() {
 }
 
 bool ModeVisible() {
+    if (g_paused) return false;
     switch (EffectiveMode()) {
         case Mode::Always: return !g_manual;
         case Mode::Toggle: return g_manual;
@@ -792,7 +798,7 @@ void UpdateVisibility() {
     Mat head;
     const bool haveHead = DevicePose(vr::k_unTrackedDeviceIndex_Hmd, &head);
     for (auto &[i, s] : g_screens) {
-        bool visible = s.shown && (shared || s.drag != Drag::None) && !s.alone;
+        bool visible = !g_paused && s.shown && (shared || s.drag != Drag::None) && !s.alone;
         // A floating window's panel: while a window floats on it, its output is on, and the
         // window isn't minimized (and once it has a crop).
         if (s.floating) visible = visible && s.floatOn && s.outputOn && !s.minimized && s.cropW > 0;
@@ -1330,7 +1336,8 @@ void UpdateCatcher() {
     for (int k = 0; k < 3; ++k) params.vSource.v[k] = l.m[k][3], params.vDirection.v[k] = -l.m[k][2];
     for (auto &[i, s] : g_screens) {
         if (!s.visible) continue;
-        std::vector<vr::VROverlayHandle_t> parts(s.All().begin(), s.All().end());
+        const auto all = s.All();  // one copy: two calls give two temporaries, not one range
+        std::vector<vr::VROverlayHandle_t> parts(all.begin(), all.end());
         for (const auto &[k, sub] : s.subs) parts.push_back(sub.overlay);
         for (auto o : parts) {
             vr::VROverlayIntersectionResults_t hit;
@@ -1568,6 +1575,7 @@ int ft_vr_modifiers(uint32_t format, uint64_t *out, int max) {
 }
 
 bool ft_vr_screens_shown(void) { return g_vr && ModeVisible(); }
+bool ft_vr_paused(void) { return g_paused; }
 
 }  // extern "C"
 
@@ -1741,6 +1749,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
                     e.type = FT_BUTTON;
                     e.button = LinuxButton(ev.data.mouse.button);
                     e.pressed = ev.eventType == vr::VREvent_MouseButtonDown;
+                    e.controller = IsHandController(ev.trackedDeviceIndex);
                     at();
                     if (!e.pressed && s.titleCarry) e.x = s.carryX, e.y = s.carryY, s.titleCarry = false;
                     if (e.pressed) {
@@ -2102,6 +2111,17 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
         g_manual = always ? !want : want;
         UpdateVisibility();
         std::snprintf(reply, size, "ok %s", want ? "shown" : "hidden");
+    } else if (std::sscanf(cmd, "pause %15s", word) == 1) {
+        if (!std::strcmp(word, "on") || !std::strcmp(word, "off")) {
+            const bool on = !std::strcmp(word, "on");
+            if (on != g_paused)
+                std::printf("%s\n", on ? "paused for a VR game: everything hidden, KWin slowed down" : "resumed");
+            g_paused = on;
+            UpdateVisibility();
+        } else if (std::strcmp(word, "state") != 0) {
+            return (void)std::snprintf(reply, size, "error pause on|off|state");
+        }
+        std::snprintf(reply, size, "ok %s", g_paused ? "paused" : "running");
     } else if (std::sscanf(cmd, "ingames %15s", word) == 1) {
         if (!std::strcmp(word, "hide")) g_inGames = InGames::Hide;
         else if (!std::strcmp(word, "visible")) g_inGames = InGames::Visible;

screens/vr.h

@@ -25,6 +25,7 @@ struct ft_event {
     double x, y;      // FT_MOTION: buffer pixels from the top left
     uint32_t button;  // FT_BUTTON: linux BTN_*
     bool pressed;
+    bool controller;  // FT_BUTTON: from a hand controller's laser (not the 3D mouse's)
     double dx, dy;    // FT_SCROLL: notches (positive dy: scroll down)
     uint32_t key;     // FT_KEY: linux KEY_* from our keyboard (pressed: down or up)
 };
@@ -35,6 +36,8 @@ void ft_vr_shutdown(void);
 int ft_vr_modifiers(uint32_t format, uint64_t *out, int max);
 // The screens are showing (by the visibility mode; not counting a wrist-pinned screen).
 bool ft_vr_screens_shown(void);
+// Frametop is paused for a VR game ("pause on"): everything is hidden, and KWin slows down.
+bool ft_vr_paused(void);
 // A panel for screen `index`, width in metres, placed in a row in front of the head.
 void ft_vr_screen_create(int index, double metres, int count);
 void ft_vr_screen_destroy(int index);

scripts/container-up.sh

@@ -6,7 +6,22 @@
 # `distrobox enter`.
 box=${FRAME_BOX:-dev}
 export XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
-[ "$(podman container inspect -f '{{.State.Running}}' "$box" 2>/dev/null)" = true ] && exit 0
+running() { [ "$(podman container inspect -f '{{.State.Running}}' "$box" 2>/dev/null)" = true ]; }
+running && exit 0
 podman container exists "$box" 2>/dev/null || exit 0  # not created yet: setup/dev-container.sh does that
-exec systemd-run --user --scope --quiet --collect --description="$box container (started for Frametop)" \
-  podman start "$box" >/dev/null
+since=$(date -u +%FT%T)
+systemd-run --user --scope --quiet --collect --description="$box container (started for Frametop)" \
+  podman start "$box" >/dev/null || exit
+
+# Every start runs distrobox-init in the container, and `distrobox enter` only waits for it
+# when it starts the container itself. The first start installs what distrobox needs and sets
+# up passwordless sudo, which takes a minute or more; until then sudo in the container asks
+# for a password (issue #9). Later starts take a few seconds.
+for i in $(seq 600); do
+  podman logs --since "$since" "$box" 2>&1 | grep -q '^container_setup_done' && exit 0
+  running || break
+  [ "$i" = 10 ] && echo "setting up the $box container (the first start takes a few minutes)" >&2
+  sleep 1
+done
+echo "the $box container didn't finish starting; see: podman logs $box" >&2
+exit 1

scripts/keys-report.py

@@ -112,7 +112,7 @@ def main():
     try:
         with open(os.path.join(HOME, ".config/frametop.conf")) as f:
             conf = [ln.split("#")[0].strip() for ln in f]
-        out("Settings: " + " ".join(c for c in conf if re.match(r"(POINTER|META_DASHBOARD|SHARE_KEYS|BACKEND)=", c)))
+        out("Settings: " + " ".join(c for c in conf if re.match(r"(POINTER|SHARE_KEYS|BACKEND)=", c)))
     except OSError:
         out("Settings: no ~/.config/frametop.conf")
     out("ft-screens state (visibility, manual, wrist, gesture, lasers, game running, in games): "

scripts/report.sh

@@ -25,6 +25,7 @@ for u in frametop-input-relay frametop-pointer frametop-power; do
   echo "$u: $(systemctl --user is-enabled $u 2>/dev/null) / $(systemctl --user is-active $u 2>/dev/null)"
 done
 echo "desktop: $(pgrep -x ft-screens >/dev/null && echo running || echo 'not running'), plasmashell: $(pgrep -c plasmashell || true)"
+echo "paused for VR games: $(cat /run/user/$(id -u)/frametop-pause.json 2>/dev/null || echo 'no (never paused since boot)')"
 LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrpathreg show 2>/dev/null | sed -n '/xternal/,$p'
 
 section "What Frametop needs from SteamOS (scripts/update-check.py)"

scripts/test-controller-click.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+"$root/scripts/frame.sh" -C screens 'mkdir -p build; gcc -std=c11 -Wall -Wextra -Werror tests/controller-click-test.c -lm -o build/controller-click-test && build/controller-click-test'

scripts/update-check.py

@@ -12,15 +12,18 @@ This checks each one, and compares the versions with the last ones recorded as w
 From a PC: ssh frame python3 - [--mark-good] < scripts/update-check.py
 
 It only reads. It connects to SteamVR as a background app (which never starts SteamVR),
-runs `vrcmd --overlays` as the pointer helper does, and maps the eye tracker's shared
-memory read-only. It never starts, stops, or restarts anything.
+runs `vrcmd --overlays` as the pointer helper does, maps the eye tracker's shared memory
+read-only, and follows a controller on vrserver's web socket for a moment. It never starts,
+stops, or restarts anything.
 """
+import base64
 import json
 import math
 import mmap
 import os
 import platform
 import re
+import socket
 import struct
 import subprocess
 import sys
@@ -34,6 +37,8 @@ LAUNCHER = os.path.join(HOME, ".local/share/applications/deckard-nested-desktop.
 VRPATHS = os.path.join(HOME, ".config/openvr/openvrpaths.vrpath")
 BACKLIGHT = "/sys/class/backlight/ae94000.dsi.0/brightness"  # as in power/ft-powerd.cpp
 EYE_MMAP = "/dev/shm/eye-server.mmap"
+VRSERVER = "http://127.0.0.1:27062"  # its web socket: input/vrws.py
+PAUSE_STATE = f"/run/user/{os.getuid()}/frametop-pause.json"  # input/game_pause.py
 HOST_GLIBC = (2, 39)  # the newest the pointer driver may need (pointer/driver/build.sh)
 
 # Packages in the OS image that Frametop depends on, and what to try by hand when one changes.
@@ -297,11 +302,19 @@ def check_overlays(desktop_up):
 
 
 def check_services(sockets, desktop_up):
+    try:
+        with open(PAUSE_STATE) as f:
+            pause = json.load(f)
+        paused = pause["stopped"]["units"] if pause.get("paused") else []
+    except (OSError, ValueError, KeyError, TypeError):
+        paused = []
     for unit, socket in UNITS.items():
         if systemctl("is-enabled", unit) != "enabled":
             continue
         state = systemctl("is-active", unit)
-        if state != "active":
+        if state != "active" and unit + ".service" in paused:
+            report("skip", unit, "stopped while Frametop is paused for a VR game (input/ft-pause off resumes it)")
+        elif state != "active":
             report("FAIL", unit, f"{state}; see journalctl --user -u {unit}")
         elif socket and socket not in sockets:
             report("FAIL", unit, f"runs, but hasn't opened {socket}")
@@ -321,6 +334,93 @@ def check_services(sockets, desktop_up):
         report("FAIL", "XRService", "not running, so hand tracking has no cameras")
 
 
+def check_steam_ui():
+    """Steam's UI calls the Steam menu shortcut uses (steam/ft-steam). They're Steam client
+    internals, so a Steam update can change them too, not just a SteamOS one."""
+    # Run from stdin (over SSH) there's no __file__: try the usual checkouts.
+    repos = [os.path.dirname(os.path.dirname(os.path.abspath(__file__)))] if "__file__" in globals() else []
+    repos += [os.path.join(HOME, "frametop"), os.path.join(HOME, "dev/frametop")]
+    tool = next((t for t in (os.path.join(r, "steam/ft-steam") for r in repos) if os.path.exists(t)), None)
+    if not tool:
+        report("skip", "Steam menu shortcut", "steam/ft-steam isn't in a checkout here")
+        return
+    try:
+        p = subprocess.run([tool, "check"], capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.TimeoutExpired) as e:
+        report("warn", "Steam menu shortcut", f"ft-steam check didn't run ({e})")
+        return
+    detail = (p.stdout + p.stderr).strip()
+    if p.returncode == 0:
+        report("ok", "Steam menu shortcut", "Steam's UI still has the calls steam/ft-steam makes")
+    elif detail.startswith("ft-steam:"):
+        report("skip", "Steam menu shortcut", detail[len("ft-steam:"):].strip())
+    else:
+        report("warn", "Steam menu shortcut", f"{detail}; a Meta tap won't open the Steam menu (steam/ft-steam)")
+
+
+def check_vr_socket():
+    """vrserver's web socket, which the pause gesture reads the controllers from (input/vrws.py,
+    input/game_pause.py). It's undocumented: SteamVR's controller binding page uses it."""
+    label, headers = "vrserver web socket", {"Referer": f"{VRSERVER}/dashboard/controllerbinding.html"}
+    try:
+        import urllib.request
+        with urllib.request.urlopen(urllib.request.Request(f"{VRSERVER}/input/getstate.json", headers=headers),
+                                    timeout=3) as r:
+            devices = json.load(r).get("devices", [])
+    except (OSError, ValueError) as e:
+        report("FAIL", label, f"/input/getstate.json didn't answer ({e}); the pause gesture can't read the controllers")
+        return
+    found = [d for d in devices if d.get("controller_type") == "frame_controller" and d.get("root_path")
+             and d.get("side") in ("left", "right")]
+    if not found:
+        report("skip", label, "no Frame controller is on, so it wasn't checked")
+        return
+    if not all(any(c.get("path") == "/input/thumbstick/click" for c in d.get("components", [])) for d in found):
+        report("warn", label, "a controller lists no /input/thumbstick/click; the default pause gesture needs it")
+    data = b""
+    try:
+        with socket.create_connection(("127.0.0.1", 27062), timeout=3) as s:
+            key = base64.b64encode(os.urandom(16)).decode()
+            s.sendall((f"GET / HTTP/1.1\r\nHost: 127.0.0.1:27062\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n"
+                       f"Sec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\nOrigin: {VRSERVER}\r\n"
+                       f"Referer: {headers['Referer']}\r\n\r\n").encode())
+            while b"\r\n\r\n" not in data:
+                chunk = s.recv(4096)
+                if not chunk:
+                    break
+                data += chunk
+            if b" 101 " not in data.split(b"\r\n", 1)[0] + b" ":
+                report("FAIL", label, "vrserver refused the web socket; the pause gesture can't read the controllers")
+                return
+            mailbox = f"frametop_check_{os.getpid()}"
+            # Every controller: one lying still (or asleep) may send nothing at all.
+            for text in [f"mailbox_open {mailbox}"] + ["mailbox_send input_server " + json.dumps(
+                    {"type": "request_input_state_updates", "device_path": d["root_path"], "returnAddress": mailbox})
+                    for d in found]:
+                payload, mask = text.encode(), os.urandom(4)
+                size = bytes([0x80 | len(payload)]) if len(payload) < 126 else bytes([0x80 | 126]) + struct.pack(">H", len(payload))
+                s.sendall(b"\x81" + size + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload)))
+            deadline = time.monotonic() + 3
+            try:
+                while b"update_component_states" not in data and time.monotonic() < deadline:
+                    chunk = s.recv(65536)
+                    if not chunk:
+                        break
+                    data += chunk
+            except socket.timeout:
+                report("skip", label, "it took the subscription, but no controller sent anything in 3 s "
+                       "(lying still or asleep?): pick one up and check again")
+                return
+    except OSError as e:
+        report("FAIL", label, f"{e}; the pause gesture can't read the controllers")
+        return
+    if b"update_component_states" in data and b"/click" in data:
+        report("ok", label, "it still streams the controllers' buttons, which the pause gesture reads")
+    else:
+        report("FAIL", label, "no button states came, so the pause gesture can't read the controllers "
+               "(input/vrws.py: the message format changed?)")
+
+
 def check_eye_tracker():
     gaze = systemctl("is-enabled", "frametop-gaze") == "enabled"
     try:
@@ -369,6 +469,8 @@ def main():
         check_openvr()
         check_overlays(desktop_up)
         check_services(sockets, desktop_up)
+        check_steam_ui()
+        check_vr_socket()
         check_eye_tracker()
 
     if args == ["--mark-good"]:

session/frametop.conf.example

@@ -38,7 +38,7 @@ POINTER_GAZE_SHOW=1        # gaze mode, with POINTER_GAZE_DOT=moving: the dot sh
 POINTER_GAZE_MOUSE=precision # gaze mode: the left button holds back its press, the mouse steers, the release clicks (precision) | clicks right away (direct)
 POINTER_HEAD_DEADZONE=0.5  # keyboard clicks at the gaze (Meta+J, Meta+K): degrees the head turns before the held dot moves with it
 POINTER_KEY_TAP=0.25       # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
-GAZE_TRACKER=steam         # gaze service: steam = SteamVR's eye tracker | own = our own (gaze/tracker: its frame grabber needs gaze/tracker/install.sh; calibrated with Calibrate on Frametop Input Settings' Gaze page)
+GAZE_TRACKER=auto          # gaze service: auto = our own eye tracker when it's installed (gaze/tracker/install.sh, which install.sh offers), else SteamVR's | own = ours | steam = SteamVR's. Each keeps its own calibration (Calibrate on Frametop Input Settings' Gaze page)
 GAZE_EYE=auto              # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
 HANDS_SWAP_SIDES=0         # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
 HANDS_CPUS=5,6,7           # hand tracking: the CPUs its model threads run on
@@ -54,7 +54,6 @@ POINTER_PINCH_DEADZONE=1.5 # hands: degrees the pinching hand moves before the p
 POINTER_GRIP_GAIN=1        # hands: the pointer moves this times the gripping hand's angle
 POINTER_GRIP_BELOW=0.35    # hands: grips beginning more than this (m) below the eyes are ignored (hands on a desk)
 POINTER_PINCH_TYPING=1     # hands: no pinch within this many seconds of a key (typing touches thumb to index)
-META_DASHBOARD=0   # 1 = a Meta tap on a pass-through keyboard toggles the SteamVR dashboard (pointer mode only)
 SHARE_KEYS=0       # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen)
 DISPLAY_OFF_MIN=0  # minutes without use before frametop-power turns the displays off, even if the headset seems worn (0 = never)
 DISPLAY_MOVE_MM=5  # displays off: the headset or a controller moving this far (mm) within 10 s counts as use

setup/dev-container.sh

@@ -45,9 +45,9 @@ fi
 "$distrobox" enter dev -- bash -c '
 set -euo pipefail
 echo "installing ${#@} packages (already-installed ones are skipped)"
-sudo dnf install -y -q "$@" 2>&1 | { grep -vE "is already installed|^Nothing to do|^$" || true; }
+sudo -n dnf install -y -q "$@" 2>&1 | { grep -vE "is already installed|^Nothing to do|^$" || true; }
 # OpenVR programs built here (the pointer helper and probe) look for the runtime at /opt/steamvr.
-[ -e /opt/steamvr ] || sudo ln -s /run/host/opt/steamvr /opt/steamvr
+[ -e /opt/steamvr ] || sudo -n ln -s /run/host/opt/steamvr /opt/steamvr
 echo "dev container ready: $(. /etc/os-release; echo $PRETTY_NAME), glibc $(ldd --version | head -1 | grep -oE "[0-9.]+$")"
 ' dev "$@"
 EOF

steam/ft-steam

@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+"""ft-steam: Steam's menu and checks, through Steam's own UI (steamui.py).
+
+  ft-steam menu    open the SteamVR dashboard on Steam's menu, or close the dashboard if it's
+                   open (the Steam button's job, as a shortcut: the input relay's steam_menu)
+  ft-steam check   do Steam's UI calls that the above needs still exist (update-check.py)
+
+No pointer mode needed: it asks Steam to show its dashboard overlay (window.ShowVROverlay, what
+Steam itself calls) and to focus the Steam frame's left menu (MenuStore.OpenMainMenu, which in
+VR invokes the frame's focus_left_frame_menu_action). SteamClient.OpenVR.VROverlay says
+whether the dashboard is up and hides it. These are Steam's UI internals, not an API, so a
+Steam client update can rename them: `ft-steam check` says so.
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
+from steamui import SteamUnreachable, evaluate  # noqa: E402
+
+WINDOW = "SteamUIStore.WindowStore.VRGamepadUIMainWindowInstance"
+
+MENU = f"""(async () => {{
+    const vr = SteamClient.OpenVR.VROverlay;
+    if (await vr.IsDashboardVisible()) {{ vr.HideDashboard(); return "closed"; }}
+    const w = {WINDOW};
+    if (!w) throw new Error("Steam's VR dashboard window isn't open");
+    ShowVROverlay(w.GetVROverlayKey());
+    // The frame's menu takes focus only once the dashboard is up.
+    for (let i = 0; i < 40 && !(await vr.IsDashboardVisible()); i++) await new Promise(r => setTimeout(r, 25));
+    w.MenuStore.OpenMainMenu();
+    return "opened";
+}})()"""
+
+CHECK = f"""(() => {{
+    const missing = [];
+    const vr = SteamClient.OpenVR && SteamClient.OpenVR.VROverlay;
+    for (const f of ["IsDashboardVisible", "HideDashboard"]) if (!vr || typeof vr[f] !== "function") missing.push("SteamClient.OpenVR.VROverlay." + f);
+    if (typeof window.ShowVROverlay !== "function") missing.push("ShowVROverlay");
+    const w = {WINDOW};
+    if (!w) missing.push("{WINDOW}");
+    else {{
+        if (typeof w.GetVROverlayKey !== "function") missing.push("GetVROverlayKey");
+        if (!w.MenuStore || typeof w.MenuStore.OpenMainMenu !== "function") missing.push("MenuStore.OpenMainMenu");
+    }}
+    return missing;
+}})()"""
+
+
+def main():
+    cmd = sys.argv[1] if len(sys.argv) == 2 else ""
+    if cmd not in ("menu", "check"):
+        sys.exit(__doc__)
+    try:
+        result = evaluate(MENU if cmd == "menu" else CHECK)
+    except SteamUnreachable as e:
+        sys.exit(f"ft-steam: {e}")
+    if cmd == "check":
+        if result:
+            sys.exit("Steam's UI lacks " + ", ".join(result))
+        print("ok")
+    else:
+        print(result)
+
+
+if __name__ == "__main__":
+    main()

steam/steamui.py

@@ -0,0 +1,108 @@
+"""Run JavaScript in Steam's UI, standard library only.
+
+Steam on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context
+(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080. Steam's
+own calls live there: SteamClient.* (settings, SteamVR's dashboard) and the UI's stores
+(SteamUIStore). Used by Frametop Display Settings (display-settings/steam_settings.py) and
+ft-steam (the Steam menu shortcut).
+
+  evaluate(expression)   the expression's value (awaited, JSON-able); SteamUnreachable when
+                         Steam or its UI isn't there, or the script threw
+"""
+import base64
+import json
+import os
+import socket
+import struct
+import urllib.request
+
+CDP_PORT = 8080
+
+
+class SteamUnreachable(Exception):
+    pass
+
+
+class _WebSocket:
+    def __init__(self, url, timeout=5):
+        host_port, path = url[len("ws://"):].split("/", 1)
+        host, port = host_port.rsplit(":", 1)
+        self.sock = socket.create_connection((host, int(port)), timeout=timeout)
+        key = base64.b64encode(os.urandom(16)).decode()
+        self.sock.sendall((f"GET /{path} HTTP/1.1\r\nHost: {host_port}\r\nUpgrade: websocket\r\n"
+                           f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode())
+        head = b""
+        while b"\r\n\r\n" not in head:
+            chunk = self.sock.recv(4096)
+            if not chunk:
+                raise SteamUnreachable("Steam closed the connection")
+            head += chunk
+        if b" 101 " not in head.split(b"\r\n", 1)[0]:
+            raise SteamUnreachable(head.split(b"\r\n", 1)[0].decode(errors="replace"))
+        self.buf = head.split(b"\r\n\r\n", 1)[1]
+
+    def send(self, text):
+        data, mask = text.encode(), os.urandom(4)
+        n = len(data)
+        if n < 126:
+            head = struct.pack(">BB", 0x81, 0x80 | n)
+        elif n < 65536:
+            head = struct.pack(">BBH", 0x81, 0x80 | 126, n)
+        else:
+            head = struct.pack(">BBQ", 0x81, 0x80 | 127, n)
+        self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(data)))
+
+    def _take(self, n):
+        while len(self.buf) < n:
+            chunk = self.sock.recv(65536)
+            if not chunk:
+                raise SteamUnreachable("Steam closed the connection")
+            self.buf += chunk
+        out, self.buf = self.buf[:n], self.buf[n:]
+        return out
+
+    def recv(self):
+        message = b""
+        while True:
+            b0, b1 = self._take(2)
+            n = b1 & 0x7F
+            if n == 126:
+                n = struct.unpack(">H", self._take(2))[0]
+            elif n == 127:
+                n = struct.unpack(">Q", self._take(8))[0]
+            message += self._take(n)
+            if b0 & 0x80:
+                return message.decode(errors="replace")
+
+    def close(self):
+        self.sock.close()
+
+
+def evaluate(expression):
+    """Runs JavaScript in Steam's SharedJSContext and returns its (awaited) value."""
+    try:
+        with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}/json", timeout=3) as r:
+            targets = json.load(r)
+    except OSError as e:
+        raise SteamUnreachable(f"Steam isn't reachable on port {CDP_PORT} ({e})") from e
+    url = next((t["webSocketDebuggerUrl"] for t in targets if t.get("title") == "SharedJSContext"), None)
+    if not url:
+        raise SteamUnreachable("Steam's UI isn't running")
+    try:
+        ws = _WebSocket(url)
+        try:
+            ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
+                                "params": {"expression": expression, "awaitPromise": True, "returnByValue": True}}))
+            while True:
+                reply = json.loads(ws.recv())
+                if reply.get("id") == 1:
+                    break
+        finally:
+            ws.close()
+    except OSError as e:
+        raise SteamUnreachable(str(e)) from e
+    result = reply.get("result", {})
+    if "exceptionDetails" in result:
+        details = result["exceptionDetails"]
+        raise SteamUnreachable(details.get("exception", {}).get("description") or details.get("text", "error"))
+    return result.get("result", {}).get("value")