Gaze: install our eye tracker, prefer it, and say why a calibration dot wasn't taken

A user on a fresh install got "Calibration failed: only 0 of 21 dots" with
no reason. The installer never installed our tracker, so gaze used
SteamVR's, and the only way SteamVR's tracker rejects a dot is losing an
eye for most of the look. The panel just showed a red ring.

- install.sh: step 9/10 installs our tracker (gaze/tracker/install.sh)
  after gaze mode, yes by default; it needs sudo, so --yes runs it only
  when sudo won't prompt. If it fails, gaze keeps SteamVR's tracker.
  Configs that still say GAZE_TRACKER=steam (the old template) are asked
  whether to switch.
- GAZE_TRACKER=auto, the new default: ours when it's installed (the
  frame grabber, its unit, and ft-eyes' Python), else SteamVR's.
  ft-gazed rechecks every second, so installing it switches over. Input
  Settings lists Own tracker first as recommended, and says how to
  install it when it's missing (checking the host's /etc through
  /run/host from the dev container).
- The calibration panel has a note line, orange over the instructions:
  why a dot wasn't taken (an eye lost, a blink, the eyes disagreeing for
  SteamVR's tracker, from steady_samples' new drop counts; ft-eyes' reply
  for ours), what a click is still waiting for after 1.5 s, and a failed
  calibration's most common reason, which the Gaze page shows too.
  steady_samples keeps the same samples as before (checked on 2037
  windows of recordings); a lost eye is named before a blink, since its
  openness reads 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -13,8 +13,6 @@ Two settings apps come with it: Frametop Display Settings for the screens, profi
 
 Frametop is an independent project, not made by or affiliated with Valve.
 
-Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions, ideas, and help with your setup.
-
 ## Install on the headset
 
 You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or the on-screen one), and about 3 GB of free space.
@@ -29,7 +27,7 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
 
    It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/DeeJanuz/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
 
-   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you three things along the way: whether to install gaze mode (experimental, yes by default) and the Bluetooth fixes, then whether to restart SteamVR. The Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
+   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
 
 After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings.
 
@@ -56,6 +54,7 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
 | Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head |
 | Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) |
 | Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
+| Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own |
 | Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead |
 | Play a VR game | The screens hide and your controllers stay in the game. Open the SteamVR dashboard, or press Meta+Shift+H, to see and use them. To keep them visible over games, change During VR games on the Visibility & pins tab; the controllers still stay in the game, and you use the screens with the mouse or the dashboard |
 
@@ -94,7 +93,7 @@ A profile is a named setup: where the screens are, with their sizes and pins, wh
 
 ### Gaze mode (experimental)
 
-In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later). Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
+In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later), and then our own eye tracker for it, which is more accurate than SteamVR's (or run `gaze/tracker/install.sh` later; it needs `sudo`). Gaze mode uses ours once it's installed, and SteamVR's until then. Turn it on and calibrate it on the Gaze page of Frametop Input Settings.
 
 | Do this | To get this |
 | --- | --- |
@@ -140,7 +139,7 @@ In a terminal on the headset, run:
 cd ~/frametop && scripts/report.sh
 ```
 
-This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file. Quick questions can go to [Discord](https://discord.gg/W3X9f7z3Bc) instead.
+This writes `frametop-report-<date>.txt` with version numbers, service states, settings, and recent logs. Bluetooth addresses and the headset's serial number are masked. Then [open an issue](https://github.com/DeeJanuz/frametop/issues), describe what you did, what you expected, and what happened, and attach the file.
 
 ## Update
 

display-settings/steam_settings.py

@@ -4,116 +4,25 @@ Steam, not systemd, puts the Frame to sleep: after "When Plugged In and Idle ->
 (an hour by default) without input, even while it charges. That's a Steam client setting,
 `system_idle_suspend_ac_sec` (0 = never), with no file or command line to change it. Steam
 on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context
-(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080. There
+(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080
-`settingsStore.clientSettings` has the current values, and `SteamClient.Settings.SetSetting`
+(steam/steamui.py). There `settingsStore.clientSettings` has the current values, and
-takes a change as a serialized CMsgClientSettings protobuf, which is what Steam's own
+`SteamClient.Settings.SetSetting` takes a change as a serialized CMsgClientSettings protobuf,
-Settings -> Power page sends. Standard library only (a minimal WebSocket client).
+which is what Steam's own Settings -> Power page sends.
 """
-import base64
-import json
 import os
-import socket
+import sys
-import struct
+
-import urllib.request
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "steam"))
+from steamui import SteamUnreachable, evaluate  # noqa: E402,F401  (callers catch SteamUnreachable here)
 
-CDP_PORT = 8080
 # CMsgClientSettings field numbers (Steam's UI bundle maps the names to these).
 FIELDS = {"system_idle_suspend_ac_sec": 24004, "system_idle_suspend_battery_sec": 24003}
 
 
-class SteamUnreachable(Exception):
-    pass
-
-
-class _WebSocket:
-    def __init__(self, url, timeout=5):
-        host_port, path = url[len("ws://"):].split("/", 1)
-        host, port = host_port.rsplit(":", 1)
-        self.sock = socket.create_connection((host, int(port)), timeout=timeout)
-        key = base64.b64encode(os.urandom(16)).decode()
-        self.sock.sendall((f"GET /{path} HTTP/1.1\r\nHost: {host_port}\r\nUpgrade: websocket\r\n"
-                           f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode())
-        head = b""
-        while b"\r\n\r\n" not in head:
-            chunk = self.sock.recv(4096)
-            if not chunk:
-                raise SteamUnreachable("Steam closed the connection")
-            head += chunk
-        if b" 101 " not in head.split(b"\r\n", 1)[0]:
-            raise SteamUnreachable(head.split(b"\r\n", 1)[0].decode(errors="replace"))
-        self.buf = head.split(b"\r\n\r\n", 1)[1]
-
-    def send(self, text):
-        data, mask = text.encode(), os.urandom(4)
-        n = len(data)
-        if n < 126:
-            head = struct.pack(">BB", 0x81, 0x80 | n)
-        elif n < 65536:
-            head = struct.pack(">BBH", 0x81, 0x80 | 126, n)
-        else:
-            head = struct.pack(">BBQ", 0x81, 0x80 | 127, n)
-        self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(data)))
-
-    def _take(self, n):
-        while len(self.buf) < n:
-            chunk = self.sock.recv(65536)
-            if not chunk:
-                raise SteamUnreachable("Steam closed the connection")
-            self.buf += chunk
-        out, self.buf = self.buf[:n], self.buf[n:]
-        return out
-
-    def recv(self):
-        message = b""
-        while True:
-            b0, b1 = self._take(2)
-            n = b1 & 0x7F
-            if n == 126:
-                n = struct.unpack(">H", self._take(2))[0]
-            elif n == 127:
-                n = struct.unpack(">Q", self._take(8))[0]
-            message += self._take(n)
-            if b0 & 0x80:
-                return message.decode(errors="replace")
-
-    def close(self):
-        self.sock.close()
-
-
-def _evaluate(expression):
-    """Runs JavaScript in Steam's SharedJSContext and returns its (awaited) value."""
-    try:
-        with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}/json", timeout=3) as r:
-            targets = json.load(r)
-    except OSError as e:
-        raise SteamUnreachable(f"Steam isn't reachable on port {CDP_PORT} ({e})") from e
-    url = next((t["webSocketDebuggerUrl"] for t in targets if t.get("title") == "SharedJSContext"), None)
-    if not url:
-        raise SteamUnreachable("Steam's UI isn't running")
-    try:
-        ws = _WebSocket(url)
-        try:
-            ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
-                                "params": {"expression": expression, "awaitPromise": True, "returnByValue": True}}))
-            while True:
-                reply = json.loads(ws.recv())
-                if reply.get("id") == 1:
-                    break
-        finally:
-            ws.close()
-    except OSError as e:
-        raise SteamUnreachable(str(e)) from e
-    result = reply.get("result", {})
-    if "exceptionDetails" in result:
-        details = result["exceptionDetails"]
-        raise SteamUnreachable(details.get("exception", {}).get("description") or details.get("text", "error"))
-    return result.get("result", {}).get("value")
-
-
 def sleep_settings():
     """{"ac": seconds, "battery": seconds}: when Steam puts the Frame to sleep without input,
     plugged in and on battery (0 = never)."""
-    value = _evaluate("(() => { const c = settingsStore.clientSettings; "
+    value = evaluate("(() => { const c = settingsStore.clientSettings; "
                      "return {ac: c.system_idle_suspend_ac_sec, battery: c.system_idle_suspend_battery_sec}; })()")
     if not isinstance(value, dict) or not all(isinstance(value.get(k), int) for k in ("ac", "battery")):
         raise SteamUnreachable("Steam's settings don't have the sleep timeouts")
@@ -125,7 +34,7 @@ def set_sleep_setting(name, seconds):
     field, seconds = FIELDS[name], int(seconds)
     if seconds < 0:
         raise ValueError("seconds must be 0 (never) or more")
-    ok = _evaluate(f"""(async () => {{
+    ok = evaluate(f"""(async () => {{
         const bytes = [];
         const varint = n => {{ while (n > 127) {{ bytes.push((n & 127) | 128); n = Math.floor(n / 128); }} bytes.push(n); }};
         varint({field} * 8); varint({seconds});

docs/desktop-mouse.md

@@ -1,121 +0,0 @@
-# Independent desktop mouse
-
-`input/ft-mousectl desktop --save` selects normal desktop input: the physical
-mouse moves across the nested KDE monitors, clicks and scrolls through KDE's
-Wayland seat, and displays KDE's own cursor. No mouse event becomes a virtual
-controller event. Physical controllers remain available to SteamVR and games.
-While a routed mouse is connected, controller events on desktop app panels are ignored;
-floating app panels, the separate grab bars and placement controls still work under the existing
-controller policy. SteamVR dashboard controls are unaffected.
-
-Native CLI commands (no additional project tooling required):
-
-```sh
-input/ft-mousectl status
-input/ft-mousectl desktop --save
-input/ft-mousectl center 2
-input/ft-mousectl position 2 960 540
-input/ft-mousectl layout
-input/ft-mousectl spatial --save
-```
-
-Numbers follow KDE's WL-0, WL-1 output order, starting at 1. Positions are local
-logical pixels; `layout` reads actual output geometry and scale from the running
-private KDE session. Geometry follows FrameTop layout changes too. A drag keeps
-the Wayland implicit grab on its initial output until all buttons are released,
-while the cursor can cross onto another monitor.
-
-The relay gathers motion and wheels until the mouse's SYN_REPORT. High resolution
-wheel reports use 1/120-notch units and replace accompanying ordinary reports;
-they are delivered immediately without the spatial pointer's pulse timer. The
-main and thumb wheels are independent. Standard mouse button codes are passed
-through, including back/forward; spatial mouse button mappings do not apply.
-`DESKTOP_MOUSE_SPEED` in `~/.config/frametop.conf` sets logical pixels per motion
-count, default 1, range .05–10. This initial implementation uses constant speed,
-not KDE's libinput acceleration settings, because FrameTop's relay owns the
-physical device outside the nested session.
-
-The cursor overlay only displays KDE's cursor; it has no input method, laser
-claim, or interactive overlay flag. Cursor shape, hotspot and buffer scale come
-from KDE. GPU cursor buffers use the same zero-copy DMA-BUF import as the monitor
-textures. Common 32-bit shared-memory cursor formats are converted to RGBA.
-
-Controller fallback: `DESKTOP_CONTROLLER_FALLBACK=1` (default) allows controller
-laser input into desktop apps when no grabbed physical mouse is routed by the
-relay. Idle mice do not trigger fallback. The relay reports presence every
-second; absence must persist for one second, so disconnect detection normally
-takes about one to two seconds. Keyboard routing still follows desktop clicks.
-The virtual spatial mouse/gaze pointer stays off; controller policy stays
-`dashboard`, preserving SteamVR's existing pointer-mode behavior.
-
-Reconnect or native mouse motion returns ownership to the native seat and
-releases controller-held buttons before clearing its focus. A held native mouse
-button prevents fallback until released. An unknown presence or a heartbeat
-older than three seconds blocks automatic controller desktop input. Set
-`DESKTOP_CONTROLLER_FALLBACK=0` to retain strict separation even without a mouse;
-reload the relay at an appropriate pause after changing this preference.
-`input/ft-mousectl status` includes `controllerFallback` with presence, freshness,
-configured enablement and actual desktop controller ownership. Older running
-compositors report this field unavailable until explicitly reloaded.
-Both paths support images up to 512×512 and buffer scale 8. FrameTop retains the
-image from the first surface commit, because KDE can commit it before selecting
-the active cursor and wlroots releases current.buffer after commit callbacks.
-An unsupported buffer is reported by cursorReady=false with cursorError
-and hides the old cursor image. Rendering in VR still needs a wearer check for
-visibility, stereo depth, monitor edges and cursor shape changes.
-
-The private desktop and apps launched from it use KDE's `kcminputrc` cursor
-preferences (24px Breeze by default). Steam's VR environment exports
-`XCURSOR_SIZE=256` and `XCURSOR_THEME=steam`; inheriting these makes app cursors
-huge and can make their appearance change across monitors. A live KDE
-CursorChanged notification updates KDE, but already running apps may retain
-the old environment until reopened. Do not resize the VR overlay to mask an
-app's incorrectly configured cursor size.
-
-The updated compositor, pointer helper and relay must all be loaded before
-activation. `status` is read-only and shows readiness, actual/desired mode,
-held buttons and delivery counters. Saving happens only after an acknowledged
-mode change. Switching refuses while buttons/keys on a pointer device are held.
-An absent compositor does not prevent explicit spatial recovery. Restarting the
-desktop requires the user's approval; use `desktops.sh restart` so background
-work is preserved, and restore the current app windows afterwards.
-
-If the compositor is lost, native mouse messages are not redirected into VR.
-The input helpers need reloading when upgrading this feature. A standalone
-helper/compositor restart may require `input/ft-mousectl desktop` again; inspect
-all three components in `status` rather than inferring success from motion alone.
-
-Offline verification (or run `scripts/test-desktop-input.sh`):
-
-```sh
-python3 -m unittest discover -s input -p 'test_*.py'
-scripts/frame.sh -C screens 'gcc -std=c11 -Wall -Wextra -Werror tests/desktop-mouse-test.c -lm -o build/desktop-mouse-test && build/desktop-mouse-test'
-scripts/frame.sh -C screens 'gcc -std=c11 -Wall -Wextra -Werror $(pkg-config --cflags libdrm) tests/desktop-cursor-test.c -o build/desktop-cursor-test && build/desktop-cursor-test'
-scripts/frame.sh -C screens 'gcc -std=c11 -Wall -Wextra -Werror tests/cursor-cache-test.c $(pkg-config --cflags --libs wlroots-0.20 wayland-server wayland-client libdrm pixman-1) -o build/cursor-cache-test && build/cursor-cache-test'
-```
-
-The direct seat and cursor path follow the [Wayland pointer protocol](https://wayland.freedesktop.org/docs/html/apa.html#protocol-spec-wl_pointer).
-
-
-The default remains `MOUSE_MODE=spatial`. Native mode is explicitly selected
-with `input/ft-mousectl desktop --save`; no driver bindings or SteamVR defaults
-are changed by installation. The new native mode suppresses virtual gaze, hand
-gesture and gaze-click input while selected. Switch back to spatial mode to
-use those features. Ordinary VR keyboard events remain available.
-
-## Experimental desktop handoff
-
-`input/ft-mousectl policy last-active` enables shared desktop input. Controller
-clicks and nonzero scrolls inside desktop monitors take ownership; aiming alone
-does not. Native mouse motion, clicks and scrolling reclaim ownership. Same
-millisecond claims favor the mouse. Held buttons keep ownership until released;
-other-device events are discarded rather than replayed after the drag. SteamVR
-controls and separate monitor grab/placement bars do not participate.
-
-`input/ft-mousectl policy mouse` restores the original mouse-first behavior
-(including configured mouse-absent fallback). `policy pointer` reserves desktop
-input for the controller. Changes refuse while either device holds a desktop
-button. `status` reports policy, owner, held buttons and ignored mouse events.
-These choices are runtime-only and reset to `mouse` when the desktop restarts.
-The original PR checkpoint is retained on `checkpoint/desktop-input-before-handoff`;
-this experiment is developed separately on `feat/desktop-input-handoff`.

docs/reference.md

@@ -2,8 +2,6 @@
 
 How each part of Frametop works, where its settings live, and the commands for running parts of it by hand. For why it's built this way, see [design.md](design.md).
 
-For independent mouse input, KDE cursor rendering and controller fallback, see [desktop-mouse.md](desktop-mouse.md).
-
 ## The desktop
 
 From the headset, open Launch a program → Desktop. The installer replaces that launcher entry with Frametop's (`~/.local/share/applications/deckard-nested-desktop.desktop`), and `desktops.sh uninstall` gives the stock single-screen desktop back.
@@ -87,6 +85,8 @@ The relay also owns the volume keys, on every device that has them, the headset'
 desktops.sh relay install     # enable it (starts with the next reboot or SteamVR start)
 desktops.sh relay status | log | uninstall
 input/input-relay.py --no-grab   # try it without taking devices from SteamVR
+input/test/keys-test.py          # key combinations and modifier taps, against fake devices (safe next to the live relay)
+steam/ft-steam menu              # what Open Steam menu does; ft-steam check: Steam's UI still has the calls
 ```
 
 The first time, the relay has to start before SteamVR, so reboot or restart SteamVR after installing it. After that it's safe to restart on its own: systemd keeps the virtual devices open in its file descriptor store (`FileDescriptorStorePreserve=yes`), so SteamVR keeps the same devices.
@@ -101,7 +101,7 @@ A mouse drives SteamVR the way a controller's laser does, but shows up as a smal
 
 Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again.
 
-To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button (or a Meta tap, with `META_DASHBOARD=1`) wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant.
+To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant. Open Steam menu / close dashboard (`steam_menu`) needs no pointer: `steam/ft-steam menu` asks Steam's UI, over its debugging port (`steam/steamui.py`), to show its dashboard overlay and focus the Steam frame's menu, or to hide the dashboard if it's up.
 
 ```
 pointer/driver/build.sh && pointer/driver/install.sh install   # then restart SteamVR
@@ -110,16 +110,16 @@ pointer/helper/run.sh status | log | restart
 pointer/driver/install.sh probe     # devices, hand roles, who owns the dashboard pointer
 ```
 
-The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
+The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
 
 ## Frametop Input Settings
 
 A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has eight pages:
 
-- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, where a Meta tap toggles the dashboard if `META_DASHBOARD=1` is in `~/.config/frametop.conf`), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
+- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
 - Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
 - Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
-- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, on any keyboard, to any action but passing a key through or nothing. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. The combination's last key isn't typed, and the modifiers still reach the app. They're saved as `key_bindings` in the same file.
+- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
 - Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
 - Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
 - Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
@@ -206,7 +206,7 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke
 - Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held.
 - The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual.
 - A correction before a click teaches the gaze service the tracker's error there. A correction bigger than `POINTER_GAZE_NUDGE_MAX` (55 degrees) isn't learned; it opens a quick check instead.
-- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode coming on without a calibration opens Calibrate by itself.
+- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it.
 
 [gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool.
 

gaze/README.md

@@ -7,7 +7,7 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (t
 - `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below.
 - `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments.
 
-Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`); the probe and our own tracker are installed by hand.
+Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`) and then our own tracker (`gaze/tracker/install.sh`), which gaze mode uses once it's installed; the probe is installed by hand.
 
 ```
 gaze/run.sh install           # the gaze service: builds ft-gaze and the panel, starts with SteamVR
@@ -23,7 +23,7 @@ gaze/probe/ft-gazeprobe --screen 1
 Gaze as an input method for the whole desktop, without replacing anything of SteamVR's:
 
 - `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes:
-  - **Eye tracker:** SteamVR's (the default), or our own (Own tracker: see "Our own eye tracker" below). The gaze service runs ours while this is on. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
+  - **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
   - **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
 
   With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
@@ -33,7 +33,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
 - **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
 - **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
 - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
-- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
+- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
 - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
 
 Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.

gaze/ft-gazectl

@@ -36,8 +36,17 @@ def main():
         if r is None:
             sys.exit("the pointer helper isn't running")
         print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)")
-        if ask("ft_gazed", "status", 0.3) is None:
+        st = ask("ft_gazed", "status", 0.3)
+        if st is None:
             print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow")
+        elif r == "ok on":
+            try:
+                calibrated = (json.loads(st).get("checks") or {}).get("calibrated")
+            except ValueError:
+                calibrated = None
+            if calibrated is False:
+                print("note: no calibration for this eye tracker yet; it opens in the headset "
+                      "(if it can't, ft-gazectl status says why: checks.problem)")
     elif cmd in ("status", "forget", "reload"):
         r = ask("ft_gazed", cmd)
         if r is None:

gaze/ft-gazed

@@ -3,8 +3,12 @@
 
 Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read
 again when the file changes:
-  GAZE_TRACKER=steam|own   SteamVR's eye tracker (default), or our own (gaze/tracker/ft-eyes,
+  GAZE_TRACKER=auto|own|steam
-                           ft-gaze's source "own"; this service runs it, see below)
+                           our own eye tracker (gaze/tracker/ft-eyes, ft-gaze's source "own";
+                           this service runs it, see below) or SteamVR's. auto (the default)
+                           is ours when it's installed (gaze/tracker/install.sh: the frame
+                           grabber, and ft-eyes' Python in this checkout), else SteamVR's; it
+                           switches when ours is installed or removed.
   GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far
                            off it was at your recent nudges; left or right counts that eye twice
                            as much as the other. Either eye alone carries the gaze when the
@@ -119,6 +123,7 @@ EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes"  # our own tracker
 EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python"  # numpy, OpenCV (build.sh)
 EYES_SOCKET = "\0ft_eyes"                      # its control socket
 EYES_CAMS = Path("/dev/shm/frametop-eyes-cams")  # the frames it reads (frametop-eyegrab.service)
+EYEGRAB = (Path("/etc/frametop/ft-eyegrab"), Path("/etc/systemd/system/frametop-eyegrab.service"))  # tracker/install.sh
 CONF = Path.home() / ".config" / "frametop.conf"
 CALIBRATION = STATE / "calibration.json"
 LESSONS = STATE / "pointer-lessons.json"
@@ -126,6 +131,7 @@ LESSON_LOG = STATE / "pointer-lessons.jsonl"
 SOURCES = ("action", "mmap1", "mmap2", "left", "right")  # the ones with a calibration here
 SIDES = ("left", "right")  # ft-gaze's order, and the sources for each eye alone
 TRACKERS = ("steam", "own")
+TRACKER_SETTINGS = ("auto",) + TRACKERS
 BIASES = ("auto", "left", "right")
 NUDGE_MAX = 55.0  # degrees: POINTER_GAZE_NUDGE_MAX's default, the largest lesson taken
 HISTORY = 12.0    # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later)
@@ -153,8 +159,14 @@ def log(msg):
     print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
 
 
+def own_installed():
+    """Is our own tracker installed: its frame grabber, and ft-eyes' Python here?"""
+    return all(p.exists() for p in EYEGRAB) and EYES_PYTHON.exists()
+
+
 def read_settings():
-    """(tracker, eye bias, nudge max) from frametop.conf, defaults for anything missing or unknown."""
+    """(tracker, GAZE_TRACKER, eye bias, nudge max) from frametop.conf, defaults for anything
+    missing or unknown. The tracker is "steam" or "own": auto picks ours when it's installed."""
     conf = {}
     try:
         for line in CONF.read_text().splitlines():
@@ -164,13 +176,15 @@ def read_settings():
                 conf[k.strip()] = v.strip().lower()
     except OSError:
         pass
-    tracker = conf.get("GAZE_TRACKER", "steam")
+    setting = conf.get("GAZE_TRACKER", "auto")
+    setting = setting if setting in TRACKER_SETTINGS else "auto"
+    tracker = setting if setting != "auto" else "own" if own_installed() else "steam"
     bias = conf.get("GAZE_EYE", "auto")
     try:
         nudge = min(max(float(conf.get("POINTER_GAZE_NUDGE_MAX", NUDGE_MAX)), 1.0), 110.0)  # the helper's range
     except ValueError:
         nudge = NUDGE_MAX
-    return tracker if tracker in TRACKERS else "steam", bias if bias in BIASES else "auto", nudge
+    return tracker, setting, bias if bias in BIASES else "auto", nudge
 
 
 def mtime(path):
@@ -185,7 +199,7 @@ class Service:
         self.override, self.verbose, self.to = source, verbose, to
         self.source = source or "mmap1"  # the older path's source
         STATE.mkdir(parents=True, exist_ok=True)
-        self.tracker, self.bias, self.nudge_max = read_settings()
+        self.tracker, self.tracker_setting, self.bias, self.nudge_max = read_settings()
         self.conf_mtime = mtime(CONF)
         self.models = {name: Correction() for name in SOURCES}
         self.mode = DEFAULT_MODEL
@@ -246,9 +260,12 @@ class Service:
 
     def load_settings(self):
         self.conf_mtime = mtime(CONF)
-        tracker, bias, self.nudge_max = read_settings()
+        tracker, self.tracker_setting, bias, self.nudge_max = read_settings()
         if (tracker, bias) != (self.tracker, self.bias):
-            log(f"tracker {tracker}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
+            auto = ""
+            if self.tracker_setting == "auto":
+                auto = " (auto: ours is installed)" if tracker == "own" else " (auto: ours isn't installed)"
+            log(f"tracker {tracker}{auto}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else ""))
             self.tracker, self.bias = tracker, bias
             for w in self.weights.values():
                 w.bias = bias
@@ -733,7 +750,8 @@ class Service:
             st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
                       lessons=max(len(m) for m in w.misses), own_running=bool(own),
                       own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
-        st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists())
+        st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists(),
+                  tracker_setting=self.tracker_setting, own_installed=own_installed())
         st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2)
                    if self.last_sample else None, "headset_on": self.steam.wearing(),
                    "headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None,
@@ -749,7 +767,8 @@ class Service:
         if mtime(CALIBRATION) != self.cal_mtime:
             self.load_calibration()
             self.refit()
-        if mtime(CONF) != self.conf_mtime:
+        if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
+                                              and own_installed() != (self.tracker == "own")):
             self.load_settings()
         want = self.eyes_wanted()
         if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:

gaze/gazecal.py

@@ -640,31 +640,44 @@ def cross_validate(points, mode):
     return errs
 
 
-def steady_samples(samples, vergence_jump=1.5):
+def steady_samples(samples, vergence_jump=1.5, why=None):
     """The samples of one look at one spot where the tracker had both eyes: none in a blink
     (openness under half its median over the samples), none where it had lost an eye (its
     variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the
     vergence) is more than `vergence_jump` degrees from its median over the samples. The
     vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a
     fraction of one far off), so only a jump away from what it was during this look means
-    the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept."""
+    the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.
+    `why`, a dict, gets how many were dropped for each reason: "lost_left", "lost_right",
+    "lost_both", "blink" and "vergence" (each sample once, for the first that applies)."""
+    if why is None:
+        why = {}
     # Openness: a blink is a sharp drop from what it was during this look. Not a fixed
     # level: looking down, the upper lids come down with the eyes, and in bright light you
     # squint, so the reading can stay under 0.5 for the whole look while the tracker follows
     # the eyes fine (a calibration dot at the bottom of the bright round failed that way).
     opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o]
     floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12
-    opened = []
+    seen = []
     for smp in samples:
-        o = (smp["src"].get("mmap1") or {}).get("open")
+        m1 = smp["src"].get("mmap1") or {}
-        if not o or min(o) >= floor:
+        o = m1.get("open")
-            opened.append(smp)
+        lost = [u > EYE_LOST for u in m1.get("unc") or [0, 0]]
+        # A lost eye's openness reads 0 too, so a lost eye is named before a blink.
+        key = ("lost_both" if all(lost) else "lost_left" if lost[0] else "lost_right") if any(lost) else \
+            "blink" if o and min(o) < floor else None
+        if key:
+            why[key] = why.get(key, 0) + 1
+        else:
+            seen.append(smp)
 
     def vergence(smp):
         return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr"))
-    opened = [smp for smp in opened if max((smp["src"].get("mmap1") or {}).get("unc") or [0]) <= EYE_LOST]
+    have = [v for v in map(vergence, seen) if v is not None]
-    have = [v for v in map(vergence, opened) if v is not None]
     if len(have) < 5:
-        return opened
+        return seen
     med = statistics.median(have)
-    return [smp for smp in opened if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
+    kept = [smp for smp in seen if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump]
+    if len(kept) < len(seen):
+        why["vergence"] = why.get("vergence", 0) + len(seen) - len(kept)
+    return kept

gaze/gazecheck.py

@@ -17,9 +17,12 @@ directions: look at each one.
           size, and the tracker's error with it, changes with brightness), each the middle and
           a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots)
           RING degrees out, half that in the middle round, turned 20 degrees a round. It opens
-          when gaze mode comes on without a calibration for the tracker in use, and on
+          whenever gaze mode is on without a calibration for the tracker in use and someone's
-          "calibrate". Frametop's screens hide while it runs. Quitting it while there's still
+          in the headset, and on "calibrate". One that closes unfinished (ignored, too few
-          no calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
+          dots) opens again only once the headset comes off and on, or gaze mode off and on.
+          Frametop's screens hide while it runs. Quitting it while there's still no
+          calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it.
+          Why gaze mode can't work yet goes in the status ("problem"), for Input Settings.
 
   fit     the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a
           card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was
@@ -34,7 +37,10 @@ works however far off the tracker is; a left click or Meta+J (the pointer helper
 "calaccept") takes it now. The full calibration's and five's dots wait for that click: you
 click when you're looking at the dot (the user asked for that: a steady gaze isn't always on
 the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as
-it takes, up to CLICK_IDLE. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
+it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason:
+gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a
+click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common
+reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
 renewed every second; the helper shows it again by itself when that stops).
 
 What a capture teaches:
@@ -90,6 +96,7 @@ ROUND_BG = (0.03, 0.33, 0.8)
 ROUND_NAMES = ("dark", "medium", "bright")
 RING_SCALE = (1.0, 0.5, 1.0)
 PANEL_RETRY = 10.0
+FULL_RETRY = 10.0      # seconds before an automatic calibration that failed to start tries again
 FIT_TIMEOUT = 300.0    # seconds the fit check stays up
 FIT_EVERY = 0.5        # seconds between its cards' updates (each is a new picture for the panel)
 FIT_HINT_WIDTH = 95    # characters a hint line holds in the panel
@@ -117,6 +124,42 @@ def check_dots(kind, own):
     return out
 
 
+# Why SteamVR's samples for a look were dropped (gazecal.steady_samples' counts) -> (short, long):
+# short for the small panel, long for the calibration's. FIT_HINT goes after the ones the
+# headset's fit causes, in the calibration's panel.
+STEAM_REASONS = {"lost_left": ("left eye lost", "SteamVR lost your left eye"),
+                 "lost_right": ("right eye lost", "SteamVR lost your right eye"),
+                 "lost_both": ("both eyes lost", "SteamVR lost both eyes"),
+                 "blink": ("blinked", "you blinked"),
+                 "vergence": ("eyes disagreed", "SteamVR's two eyes disagreed")}
+FIT_HINT = "check the headset fit"
+ACCEPT_WAIT = 1.5      # seconds after a click with no capture before the panel says what it waits for
+
+
+def reject_reason(reply, why):
+    """Why a dot wasn't taken -> (short, long, fit): our tracker's reply (`reply`), or SteamVR's
+    drop counts (`why`, when `reply` is None). `fit`: the headset's fit is the likely cause."""
+    if reply is None:
+        if not why:
+            return "no reading", "SteamVR sent no reading for that look", False
+        key = max(why, key=why.get)
+        return *STEAM_REASONS[key], key.startswith("lost")
+    words = reply.removeprefix("fail ").split()
+    # ft-eyes: "fail the left eye was seen in only 3 frames", "fail the left eye moved (4.2 px)"
+    if len(words) >= 3 and words[0] == "the" and words[2] == "eye":
+        eye = words[1]
+        if "seen" in words:
+            return f"{eye} eye not seen", f"our tracker saw your {eye} eye in only {words[-2]} frames", True
+        if "moved" in words:
+            return f"{eye} eye moved", f"your {eye} eye moved while you looked", False
+    if not reply:
+        return "no answer", "our tracker didn't answer", False
+    if reply.startswith("our tracker isn't running"):
+        return "tracker not running", "our tracker isn't running", False
+    text = reply.removeprefix("fail ")
+    return text[:24], f"our tracker said: {text}", False
+
+
 def spread(points):
     """The median point and the spread around it (1.4826 x the median distance: a standard
     deviation that one stray sample can't move far)."""
@@ -167,7 +210,10 @@ class Checks:
         self.check = None
         self.gaze_on = None
         self.gaze_heard = 0.0
-        self.want_full_until = 0.0  # gaze mode came on before the tracker said whether it's calibrated
+        self.full_armed = True    # gaze mode on without a calibration opens the full one (need_full)
+        self.full_blocked = None  # why it can't open now
+        self.full_retry_at = 0.0
+        self.full_failed = None   # why the last calibration failed (too few dots), for problem()
         self.last_quick = 0.0
         self.sample_at = 0.0     # the tracker last sent anything
         self.seen_at = 0.0       # eyes last seen (SteamVR's tracker's variance for them, "unc")
@@ -282,11 +328,49 @@ class Checks:
         return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2
 
     def on_gaze_on(self):
+        self.full_armed = True
+        self.need_full("gaze mode came on without a calibration")
+
+    def need_full(self, reason):
+        """Gaze mode is on without a calibration: open the full one, once per arming (see the top),
+        or note why it can't open."""
         cal = self.calibrated()
-        if cal is False:
+        if cal:
-            self.start("full", "gaze mode came on without a calibration")
+            self.full_armed = True  # missing one later (the other tracker picked) is news again
-        elif cal is None:
+        if not self.gaze_on or self.check or not self.full_armed or cal is not False:
-            self.want_full_until = time.monotonic() + 20
+            self.full_blocked = None
+            return
+        now = time.monotonic()
+        if not self.can_run():
+            why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2
+                   else "no eyes seen (is the headset on?)")
+        elif now < self.full_retry_at:
+            return
+        else:
+            reply = self.start("full", reason)
+            why = None if reply == "ok" else reply.removeprefix("error ")
+            if why:
+                self.full_retry_at = now + FULL_RETRY
+        if why and why != self.full_blocked:
+            log(f"the calibration can't open: {why}")
+        self.full_blocked = why
+        if not why:
+            self.full_armed = False
+
+    def problem(self):
+        """Why gaze mode, on, can't follow your eyes yet, or None. Our tracker not having said
+        yet is None: Input Settings has its own line for our tracker."""
+        if not self.gaze_on or self.calibrated() is not False:
+            return None
+        if self.check and self.check["kind"] == "full":
+            return "Not calibrated yet: the calibration is open in the headset"
+        if self.full_blocked:
+            return f"Not calibrated, and the calibration can't open: {self.full_blocked}"
+        if not self.full_armed:
+            if self.full_failed:
+                return f"Not calibrated: the calibration failed (most dots: {self.full_failed}). Use Calibrate"
+            return "Not calibrated: the calibration closed unfinished. Use Calibrate"
+        return "Not calibrated: the calibration opens in the headset"
 
     def auto_quick(self, reason):
         now = time.monotonic()
@@ -318,7 +402,7 @@ class Checks:
         now = time.monotonic()
         self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
                       "started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
-                      "skipped": 0, "captured": 0, "points": {}}
+                      "skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""}
         log(f"{kind} check: {reason}")
         if kind == "full":
             st = ask(SCREENS, "state", 0.5).split()
@@ -402,7 +486,7 @@ class Checks:
         self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look")
         self.last_progress = None
         c["shown"] = time.monotonic()
-        c["run"], c["accept"], c["done_at"] = [], False, None
+        c["run"], c["accept"], c["done_at"], c["accept_at"] = [], False, None, None
 
     def on_sample(self, s):
         self.sample_at = time.monotonic()
@@ -428,6 +512,7 @@ class Checks:
         if "hy" not in src:
             return
         now = time.monotonic()
+        c["gaze_at"] = now
         if now < c["shown"] + CHECK_SETTLE:
             return
         g = (src["hy"], src["hp"])
@@ -488,7 +573,9 @@ class Checks:
             if ok:
                 svc.weights["own"].add(miss)
         else:
-            steady = steady_samples(samples)
+            why = {}
+            steady = steady_samples(samples, why=why)
+            rec["dropped"] = why
             reads = {}
             for name in ("action", "mmap1", "mmap2", "left", "right"):
                 pts = [(smp["src"][name]["hy"], smp["src"][name]["hp"]) for smp in steady
@@ -522,23 +609,41 @@ class Checks:
                 if svc.kind == "eyes":
                     svc.weights["steam"].add(miss)
                 svc.dirty = True
+        if not ok:
+            short, long, fit = reject_reason(rec.get("reply", "") if c["own"] else None, rec.get("dropped"))
+            rec["reason"] = long
+            c["reasons"][long] = c["reasons"].get(long, 0) + 1
+            if fit:
+                c["fit_reasons"].add(long)
         rec["accepted"] = ok
         self.log_check(rec)
         if not ok:
             c["tries"] += 1
-            log(f"{c['kind']} check dot {c['i'] + 1}: not taken ({rec.get('reply', '')})")
+            log(f"{c['kind']} check dot {c['i'] + 1}: not taken: {rec['reason']} ({rec.get('reply', '')})")
-            if c["tries"] >= 2 or c["kind"] != "full":
+            full = c["kind"] == "full"
+            if c["tries"] >= 2 or not full:
+                self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if full else f"Skipped: {short}")
                 self.skip()
             else:
+                self.note(f"Not taken: {long}. Look at the dot and click again")
                 self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail")
-                c["run"], c["accept"] = [], False
+                c["run"], c["accept"], c["accept_at"] = [], False, None
                 c["shown"] = time.monotonic()  # settle again, then retry
             return
         c["captured"] += 1
         c["tries"] = 0
+        self.note("")
         self.to_panel(f"dot {yaw:.3f} {pitch:.3f} done")
         c["done_at"] = time.monotonic() + DONE_PAUSE
 
+    def note(self, text):
+        """The panel's warning line, over the instructions (empty: none). It stays until the
+        next dot is taken, so a skipped dot's reason is still there at the one after it."""
+        c = self.check
+        if c.get("note") != text:
+            c["note"] = text
+            self.to_panel(f"note {text}".rstrip())
+
     def skip(self):
         c = self.check
         yaw, pitch, _ = c["dots"][c["i"]]
@@ -567,12 +672,20 @@ class Checks:
             return
         n = len(c["dots"])
         if c["captured"] < n * 2 / 3:
-            log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays")
+            reasons = c["reasons"]
+            main = max(reasons, key=reasons.get) if reasons else None
+            self.full_failed = main
+            log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays"
+                + (f". Not taken: {', '.join(f'{r} ({k})' for r, k in reasons.items())}" if reasons else ""))
             self.to_panel(f"text Calibration failed: only {c['captured']} of {n} dots. Try again from Input Settings")
+            if main:
+                fit = main in c["fit_reasons"]
+                self.note(f"Most dots: {main}" + (". Check headset fit on the Gaze page" if fit else ""))
             self.to_panel("dot 0 0 off")
-            c["done_at"] = time.monotonic() + 3.0
+            c["done_at"] = time.monotonic() + (8.0 if main else 3.0)  # time to read why
             c["closing"] = True
             return
+        self.full_failed = None
         if c["own"]:
             reply = ask(EYES, "calib-fit", 10.0)
             log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
@@ -642,6 +755,8 @@ class Checks:
             if self.check and self.check["kind"] == "fit":
                 self.check["fit"].toggle_guide(time.monotonic())
             elif self.check:
+                if not self.check["accept"]:
+                    self.check["accept_at"] = time.monotonic()
                 self.check["accept"] = True
             return "ok"
         if cmd == "calquit":
@@ -688,6 +803,13 @@ class Checks:
             return
         if c["done_at"]:
             return
+        if c["accept"] and c["accept_at"] and now - c["accept_at"] > ACCEPT_WAIT:
+            # Clicked, but no capture yet (see on_sample): say what it's waiting for.
+            full = c["kind"] == "full"
+            if now - c.get("gaze_at", 0.0) > 0.5:
+                self.note("Waiting: the eye tracker isn't sending a gaze" if full else "Waiting: no gaze")
+            else:
+                self.note("Waiting for your gaze to hold still on the dot" if full else "Hold your look still")
         if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
             self.close("ignored")
         elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
@@ -703,18 +825,13 @@ class Checks:
             self.gaze_on = None  # the helper isn't answering
         if self.check:
             self.to_helper("calpanel 1")
-        if self.want_full_until:
-            cal = self.calibrated()
-            if cal is not None or now > self.want_full_until:
-                self.want_full_until = 0.0
-                if cal is False and self.gaze_on:
-                    self.start("full", "gaze mode came on without a calibration")
         if not self.eyes_seen(AWAY_MIN):
             self.away, self.back_since = True, None
         elif self.back_since is not None and not self.eyes_seen():
             self.back_since = None  # gone again before DON_DELAY
         elif self.back_since is not None and now - self.back_since >= DON_DELAY:
             self.away, self.back_since = False, None
+            self.full_armed = True  # a calibration that closed unfinished opens again
             self.auto_quick("the headset went on")
         if svc.kind == "own" and now - svc.own_at < 5:
             reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values())
@@ -723,10 +840,12 @@ class Checks:
             elif not self.reseat_seen and self.can_run():
                 self.reseat_seen = True
                 self.auto_quick("our tracker asked for a click")
+        self.need_full("gaze mode is on without a calibration")
 
     def status(self):
         c = self.check
         st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(),
+              "problem": self.problem(),
               "panel": self.panel_proc is not None,
               "last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None}
         if c:

gaze/panel/ft-gazepanel.cpp

@@ -22,6 +22,8 @@
 //                                look, capture (a ring filling to progress), done,
 //                                fail, off
 //   title <text> / text <text>   a line at the top / at the bottom (empty to clear)
+//   note <text>                  a warning line just above the bottom one, in orange (red on
+//                                the bright round): why a dot wasn't taken (empty to clear)
 //   eye <0|1> <r> <g> <b> <signal 0..1|-1> <seen 0..1|-1> <word>
 //                                fit: an eye's card (0 left): its state in that colour, the
 //                                tracker's signal, the share of the last 10 s it was seen
@@ -143,7 +145,7 @@ struct Panel {
     double wDeg = kQuickDeg;  // across
     std::vector<uint8_t> px;
     double bg = 0.05;
-    std::string title, text;
+    std::string title, text, note;
     bool dotOn = false;
     double dotYaw = 0, dotPitch = 0, progress = 0;
     std::string state = "off";
@@ -268,6 +270,9 @@ void Draw(Panel &p) {
     const int titleSize = int(pxPerDeg * (p.full ? 1.5 : 1.1)), textSize = int(pxPerDeg * (p.full ? 1.2 : 0.9));
     Text(p, p.title, titleSize, p.w / 2, int(titleSize * 1.2), faint);
     Text(p, p.text, textSize, p.w / 2, p.h - int(textSize * 1.3), faint);
+    // Under the full calibration's lowest dots (RING degrees down) and over the text.
+    if (light) Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 0.7, 0.12, 0.05);
+    else Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 1.0, 0.62, 0.3);
     if (p.fit) DrawFit(p, pxPerDeg, textSize, faint);
     if (!p.dotOn || p.state == "off") return;
     double x, y;
@@ -468,7 +473,7 @@ int main(int argc, char **argv) {
                 p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx;
                 p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx;
                 p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg;
-                p.title.clear(), p.text.clear(), p.dotOn = false, p.state = "off";
+                p.title.clear(), p.text.clear(), p.note.clear(), p.dotOn = false, p.state = "off";
                 p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
                 place();
                 visible = dirty = true;  // shown with its first picture
@@ -496,6 +501,8 @@ int main(int argc, char **argv) {
                 p.title = buf[5] == ' ' ? buf + 6 : "", dirty = true;
             } else if (!std::strncmp(buf, "text", 4)) {
                 p.text = buf[4] == ' ' ? buf + 5 : "", dirty = true;
+            } else if (!std::strncmp(buf, "note", 4)) {
+                p.note = buf[4] == ' ' ? buf + 5 : "", dirty = true;
             } else if (std::sscanf(buf, "%15s", word) == 1 && !std::strcmp(word, "ping")) {
                 reply = visible ? "ok shown" : "ok hidden";
             } else {

gaze/tracker/install.sh

@@ -2,8 +2,9 @@
 # Install (or remove) the frame grabber our own eye tracker needs: ft-eyegrab, as the system
 # service frametop-eyegrab.service. It copies the eye-camera frames, read-only, out of
 # SteamVR's eyetracking process into /dev/shm/frametop-eyes-cams for ft-eyes, and only while
-# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when Eye tracker
+# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when ours is the
-# is Own tracker or the gaze probe uses it.
+# tracker in use (GAZE_TRACKER=auto, the default, picks it once this is installed) or the gaze
+# probe uses it. install.sh offers this after gaze mode.
 # Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit: it asks for
 # the password in the terminal, on the Frame or from a PC, or runs SUDO_ASKPASS when that's set
 # (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env).

hands/README.md

@@ -14,6 +14,8 @@ Two programs, each a user service that stops when SteamVR does:
 
 They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` doesn't install it.
 
+For the cutouts alone, `hands/ft-cutouts on` starts the same two programs with ft-hands' `--no-gestures`: your hands show through the screens, and no pinch or grip is detected, so nothing clicks or drags. It runs this checkout's build as transient user units, so it needs `hands/build.sh` and ft-camd's capabilities (`hands/run.sh caps`) but not `hands/run.sh install`. It and `ft-handsctl on` stop each other's services, and it stops with SteamVR too.
+
 ```
 ft-handsctl on | off            # on the Frame: start or stop hand tracking (SteamVR must be running)
 ft-handsctl status              # the services, and ft-hands' last status lines
@@ -21,6 +23,8 @@ ft-handsctl log [lines]
 ft-handsctl cutouts on|off|state  # ft-screens' hand cutouts, without stopping tracking
 ft-handsctl gestures            # pinches and grips, live (tools/watch_gestures.py --distance)
 
+hands/ft-cutouts on | off | status   # the cutouts only: no pinches or grips
+
 hands/run.sh install            # build, give ft-camd its capabilities (sudo, once per build), install disabled
 hands/run.sh start|stop         # start or stop the services
 hands/run.sh restart            # after changing a setting
@@ -104,6 +108,7 @@ Options:
 - `--models DIR`: where the models are.
 - `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
 - `--no-publish`: don't write the hands and gestures files.
+- `--no-gestures`: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. `ft-cutouts` runs it this way.
 - `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
 - `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
 - `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.

hands/ft-cutouts

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# ft-cutouts: the hand cutouts without pinches and grips. Your hands show through Frametop's
+# screens, but ft-hands runs with --no-gestures, so nothing clicks or drags. ft-handsctl on is
+# the full version; each stops the other. Like it, this doesn't start with SteamVR, and it
+# stops when SteamVR does. On the Frame.
+#
+#   ft-cutouts on | off | status
+#
+# Runs this checkout's hands/build as transient user units, so it doesn't need hands/run.sh
+# install: only a build (hands/build.sh) and ft-camd's capabilities (hands/run.sh caps).
+set -euo pipefail
+here=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
+camd=frametop-cutouts-camd.service hands=frametop-cutouts-hands.service
+
+die() { echo "$*" >&2; exit 1; }
+
+run_unit() {  # unit, the ft-handsctl unit it replaces, description, then the command
+  local unit=$1 conflicts=$2 what=$3
+  shift 3
+  systemctl --user -q is-active "$unit" && return
+  systemd-run --user --quiet --collect --unit="$unit" --description="Frametop hand cutouts: $what" \
+    -p PartOf=steamvr.service -p After=steamvr.service -p Conflicts="$conflicts" \
+    -p Restart=always -p RestartSec=5 -p TimeoutStopSec=5 "$@"
+}
+
+case ${1:-status} in
+  on)
+    systemctl --user -q is-active steamvr.service || die "SteamVR isn't running"
+    [ -x "$here/build/ft-camd" ] && [ -x "$here/build/ft-hands" ] || die "not built: hands/build.sh"
+    grep -qa -- --no-gestures "$here/build/ft-hands" || die "ft-hands predates --no-gestures: hands/build.sh"
+    getcap "$here/build/ft-camd" | grep -q cap_sys_ptrace ||
+      die "ft-camd needs its capabilities: hands/run.sh caps (asks for sudo)"
+    if systemctl --user -q is-active frametop-camd.service frametop-hands.service; then
+      echo "stopping ft-handsctl's hand tracking (it has pinches and grips)"
+    fi
+    run_unit $camd frametop-camd.service "the camera broker" "$here/build/ft-camd" --status 60
+    if ! systemctl --user -q is-active $hands; then
+      "$here/../scripts/container-up.sh"  # so stopping the unit can't take the container down
+      pkill -x ft-hands || true
+    fi
+    run_unit $hands frametop-hands.service "hand tracking, no gestures" "$HOME/.local/bin/distrobox" enter dev -- \
+      "$here/build/ft-hands" --status 60 --no-gestures
+    "$here/ft-handsctl" cutouts on >/dev/null || true
+    sleep 4
+    "$0" status ;;
+  off)
+    systemctl --user stop $hands $camd 2>/dev/null || true
+    systemctl --user -q is-active frametop-hands.service || pkill -x ft-hands || true
+    echo "hand cutouts off" ;;
+  status)
+    for u in $camd $hands; do echo "$u: $(systemctl --user is-active $u || true)"; done
+    echo "ft-screens cutouts: $("$here/ft-handsctl" cutouts state 2>&1)"
+    inv=$(systemctl --user show -p InvocationID --value $hands)  # this run's lines only
+    [ -n "$inv" ] && journalctl --user _SYSTEMD_INVOCATION_ID="$inv" --no-pager -o cat |
+      grep -E '^ *[0-9.]+s |sets with a hand|cameras:' | tail -2 | cut -c1-160 || true ;;
+  *) sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
+esac

hands/track/main.cpp

@@ -3,9 +3,12 @@
 // Python prototype: the same scheduling, with the models on a few threads.
 //
 //   ft-hands [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N]
-//            [--no-publish] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
+//            [--no-publish] [--no-gestures] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
 //            (--help lists them all)
 //
+// --no-gestures: hands for the cutouts only. No pinch or grip detection, so nothing reaches
+// the pointer and a closing hand doesn't raise the rate; the gestures file is removed.
+//
 // Which cameras (--cams, HANDS_CAMERAS): the four mono IR cameras light the hands with their
 // own IR and track well in dim rooms, but in bright light (a sunny room, a window behind the
 // hands) they expose for the room and the hands come out dark. The two Arcturus colour
@@ -150,7 +153,7 @@ struct Lighting {
 int main(int argc, char **argv) {
     double seconds = 0, status = 5;
     int threads = 3, niceness = 5;
-    bool int8 = false, publish = true, track = true, swap_sides = false;
+    bool int8 = false, publish = true, track = true, swap_sides = false, gestures_on = true;
     std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
     std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME;
     // SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for
@@ -189,6 +192,7 @@ int main(int argc, char **argv) {
         else if (a == "--nice" && more) niceness = std::atoi(argv[++i]);
         else if (a == "--int8") int8 = true;
         else if (a == "--no-publish") publish = false;
+        else if (a == "--no-gestures") gestures_on = false;
         else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]);
         else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
         else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
@@ -217,6 +221,7 @@ int main(int argc, char **argv) {
         }
         else {
             std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n"
+                        "          [--no-gestures] (hands for the cutouts only: no pinches or grips)\n"
                         "          [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n"
                         "          [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
                         "          [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
@@ -265,12 +270,14 @@ int main(int argc, char **argv) {
         return true;
     };
     if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
-        (publish && (!pub.open(err) || !gestures.open(pinch, grip, err))) ||
+        (publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
         (!record.empty() && !start_recording(record, err))) {
         std::fprintf(stderr, "%s\n", err.c_str());
         return 1;
     }
     if (!ring.alive()) return std::fprintf(stderr, "ft-camd isn't running (no heartbeat)\n"), 1;
+    // so a reader can't take an earlier run's file for this one's
+    if (publish && !gestures_on) unlink((run_dir() + "/gestures").c_str());
 
     std::map<std::string, int> index;   // mono calibration name -> ring camera
     std::map<std::string, int> color;   // colour calibration name (color_video<N>) -> ring camera
@@ -541,9 +548,11 @@ int main(int argc, char **argv) {
         const auto hands = tracker.step(images, int64_t(tmin));
         const uint64_t capture = uint64_t(int64_t(tmin) - raw_off);   // CLOCK_MONOTONIC
         const std::vector<Seen> views = tracker.views_now();
+        if (gestures_on) {
             grip.update(hands, views, int64_t(capture));
             pinch.update(hands, views, int64_t(capture), grip.gripping());
-        if (gesture_log && capture - t_glog >= 100'000'000) {
+        }
+        if (gestures_on && gesture_log && capture - t_glog >= 100'000'000) {
             t_glog = capture;
             for (int k = 0; k < 2; ++k)
                 if (pinch.world_d[k] >= 0)
@@ -554,7 +563,8 @@ int main(int argc, char **argv) {
         // a gesture down or closing gets the full rate, even while the palm holds still
         next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) -
                                    0.005) * 1e9);
-        if (publish) pub.write(hands, capture), gestures.write(pinch, grip, capture);
+        if (publish) pub.write(hands, capture);
+        if (publish && gestures_on) gestures.write(pinch, grip, capture);
         for (const Pinch::Event &e : grip.events)
             std::printf("grip  %s %-5s curl %.2f at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
                         e.point[0], e.point[1], e.point[2]);
@@ -593,6 +603,7 @@ int main(int argc, char **argv) {
                             s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten,
                             !rec ? "" : ("  recorded " + std::to_string(rec->written()) + " dropped " +
                                          std::to_string(rec->dropped())).c_str());
+            if (gestures_on) {
                 std::printf("        pinches: left %u right %u (held back, palm down: %d %d)  grips: left %u right %u",
                             pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1],
                             grip.side(0).begins, grip.side(1).begins);
@@ -604,6 +615,7 @@ int main(int argc, char **argv) {
                                                                           : "open",
                                     pinch.side(k).distance, grip.curl[k]);
                 std::printf("\n");
+            }
             for (const Hand *h : hands)
                 std::printf("        hand %d %-5s views %d wrist %+.3f %+.3f %+.3f m  scale %.2f  speed %.2f m/s\n", h->id,
                             h->right() ? "right" : "left", h->nviews, h->pts[0][0], h->pts[0][1], h->pts[0][2], h->scale,
@@ -616,8 +628,8 @@ int main(int argc, char **argv) {
         }
     }
     if (!color.empty()) unlink(want_file.c_str());
-    if (publish) {
+    if (publish) pub.write({}, mono_ns());
-        pub.write({}, mono_ns());
+    if (publish && gestures_on) {
         pinch.release(int64_t(mono_ns()));   // a drag in progress ends, as lost
         grip.release(int64_t(mono_ns()));
         gestures.write(pinch, grip, mono_ns());

input-settings/ft_input_settings.py

@@ -9,8 +9,9 @@ to the input relay over its control socket (@frametop_relay):
   - Controllers: the same for the Frame controllers' buttons, minus the gaze actions (gaze
     mode is a mouse feature). They're read by the pointer helper through SteamVR input
     (@ft_pointer_helper: vrstatus, vrglobal), and a mapped button is taken from games.
-  - Keyboard: when Frametop's keyboard opens, and key combinations for any action (Meta+Shift+F
+  - Keyboard: when Frametop's keyboard opens, and key combinations for any action, a command
-    floats a window unless the rules have their own list).
+    of your own too ("command:CMD"), or a modifier tapped alone (a Meta tap opens Steam's menu
+    and Meta+Shift+F floats a window, unless the rules have their own list).
   - Pointer: speed, dot size, distance and the rest, applied live.
   - Ignored panels: SteamVR overlays the pointer passes through (POINTER_IGNORE), by app or
     one by one. The helper lists them (@ft_pointer_helper "overlays").
@@ -60,6 +61,7 @@ DEFAULT_BUTTONS = {0x110: "left", 0x111: "right", 0x112: "middle", 0x113: "back"
 ACTION_LABELS = {
     "left": "Left click", "right": "Right click", "middle": "Middle click", "back": "Back",
     "scroll_up": "Scroll up", "scroll_down": "Scroll down", "dashboard": "Toggle SteamVR dashboard",
+    "steam_menu": "Open Steam menu / close dashboard",
     "recenter": "Recenter pointer", "pointer_toggle": "Pointer on/off",
     "follow_toggle": "Head follow on/off (experimental)", "gaze_toggle": "Gaze pointer on/off (experimental)",
     "gaze_precision": "Gaze precision: hold to steer, release to click",
@@ -102,6 +104,8 @@ KEYBOARD_ONLY = ("gaze_left", "gaze_right")
 # Profiles (docs/profiles.md): "profile:NAME" opens one (the relay runs ft-layout use NAME).
 LAYOUT_PATH = os.path.expanduser("~/.config/frametop-layout.json")
 PROFILE = "profile:"
+# "command:CMD": the relay runs CMD with sh -c (key combinations only, here).
+COMMAND = "command:"
 
 
 def profile_actions():
@@ -113,6 +117,8 @@ def profile_actions():
 def action_label(a):
     if a.startswith(PROFILE):
         return f"Open profile {a[len(PROFILE):]}"
+    if a.startswith(COMMAND):
+        return f"Run: {a[len(COMMAND):]}"
     return ACTION_LABELS.get(a, a)
 
 
@@ -120,14 +126,18 @@ def is_profile(a):
     return a.startswith(PROFILE) and len(a) > len(PROFILE)
 
 
+def is_command(a):
+    return a.startswith(COMMAND) and bool(a[len(COMMAND):].strip())
+
+
 def mappable(a):
     """An action a controller button can have."""
     return a in CONTROLLER_ACTIONS or is_profile(a)
 
 
 def shortcut_mappable(a):
-    """An action a key combination can have: the gaze ones too."""
+    """An action a key combination can have: the gaze ones and commands too."""
-    return a in SHORTCUT_ACTIONS or is_profile(a)
+    return a in SHORTCUT_ACTIONS or is_profile(a) or is_command(a)
 # Gaze mode settings (pointer helper), like POINTER_SETTINGS.
 GAZE_SETTINGS = [
     ("POINTER_GAZE_RETAKE", "Look away to hand back", 5, 1, 45, 0.5, "°"),
@@ -141,9 +151,10 @@ GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, releas
 # Key combinations ("key_bindings" in the rules): modifiers, either side folded into the left code.
 MODIFIER_CODES = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
 MODIFIER_NAMES = {29: "Ctrl", 42: "Shift", 56: "Alt", 125: "Meta"}
-# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): Meta+J and
+# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): a Meta tap
-# Meta+K click at the gaze, Meta+Shift+F floats a window.
+# opens Steam's menu, Meta+J and Meta+K click at the gaze, Meta+Shift+F floats a window.
-DEFAULT_KEY_BINDINGS = {"125+36": "gaze_left", "125+37": "gaze_right", "42+125+33": "float_toggle"}
+DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
+                        "42+125+33": "float_toggle"}
 
 
 def key_bindings(rules):
@@ -151,8 +162,12 @@ def key_bindings(rules):
     counts as its own)."""
     bound = rules.get("key_bindings")
     return dict(bound) if isinstance(bound, dict) else dict(DEFAULT_KEY_BINDINGS)
-# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias.
+# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias. GAZE_TRACKER
+# can also be auto (the default): ours when it's installed, else SteamVR's.
 GAZE_TRACKERS = {"steam": "SteamVR's eye tracker", "own": "our own eye tracker"}
+# Our tracker's frame grabber (gaze/tracker/install.sh), on the host: this runs in the dev
+# container, which has the host's /etc under /run/host.
+EYEGRAB = ("/etc/frametop/ft-eyegrab", "/run/host/etc/frametop/ft-eyegrab")
 GAZE_EYES = {"auto": "auto", "left": "left eye", "right": "right eye"}
 # Pointer settings: key, label, default, min, max, step, unit.
 POINTER_SETTINGS = [
@@ -295,6 +310,7 @@ class Backend(QObject):
         self._capture_vr = False
         self._capture_combo = ""  # the action a key combination is being captured for
         self._combo_mods = set()
+        self._combo_tap = None  # a modifier pressed alone, nothing since: released, it's a tap
         self._vr = {}  # the helper's vrstatus, {} when it doesn't answer
         self._vr_at = 0.0
         self._gaze = {}  # ft-gazed's status, {} when it isn't running
@@ -408,13 +424,16 @@ class Backend(QObject):
                 self.activity.emit(msg["id"])
                 code, value = int(msg["code"]), msg["value"]
                 if code in MODIFIER_CODES:
-                    (self._combo_mods.add if value else self._combo_mods.discard)(MODIFIER_CODES[code])
+                    mod = MODIFIER_CODES[code]
+                    if value == 1:
+                        self._combo_tap = None if self._combo_mods else mod
+                    (self._combo_mods.add if value else self._combo_mods.discard)(mod)
+                    if value == 0 and self._combo_tap == mod:
+                        self._end_shortcut_capture(str(mod))  # the relay's modifier tap
                 elif value == 1 and code < BTN_MISC:
-                    self._save_shortcut("+".join(str(c) for c in sorted(self._combo_mods) + [code]),
+                    self._end_shortcut_capture("+".join(str(c) for c in sorted(self._combo_mods) + [code]))
-                                        self._capture_combo)
+                elif value == 1:
-                    self._capture_combo = ""
+                    self._combo_tap = None  # a mouse button: no tap
-                    self._combo_mods = set()
-                    self.shortcutCaptureChanged.emit()
             elif t == "event":
                 self.activity.emit(msg["id"])
                 if (self._capture_id and msg["id"] == self._capture_id and msg["type"] == "key"
@@ -890,6 +909,8 @@ class Backend(QObject):
     # --- key combinations ("key_bindings") ---
     def comboName(self, combo):
         parts = [int(c) for c in combo.split("+") if c.isdigit()]
+        if len(parts) == 1 and parts[0] in MODIFIER_NAMES:
+            return MODIFIER_NAMES[parts[0]] + " tap"
         return "+".join(MODIFIER_NAMES.get(c) or self.codeName(c).removeprefix("KEY_").title() for c in parts)
 
     @Property("QVariantList", notify=mappingsChanged)
@@ -900,7 +921,8 @@ class Backend(QObject):
 
     @Property("QVariantList", constant=True)
     def shortcutActions(self):
-        return [{"value": a, "text": action_label(a)} for a in SHORTCUT_ACTIONS + profile_actions()]
+        return [{"value": a, "text": action_label(a)} for a in SHORTCUT_ACTIONS + profile_actions()] + \
+            [{"value": COMMAND, "text": "Run a command…"}]
 
     @Property(bool, notify=shortcutCaptureChanged)
     def capturingShortcut(self):
@@ -911,6 +933,7 @@ class Backend(QObject):
         if shortcut_mappable(action):
             self._capture_combo = action
             self._combo_mods = set()
+            self._combo_tap = None
             self._send("watch 60")
             self.shortcutCaptureChanged.emit()
 
@@ -919,6 +942,13 @@ class Backend(QObject):
         self._capture_combo = ""
         self.shortcutCaptureChanged.emit()
 
+    def _end_shortcut_capture(self, combo):
+        self._save_shortcut(combo, self._capture_combo)
+        self._capture_combo = ""
+        self._combo_mods = set()
+        self._combo_tap = None
+        self.shortcutCaptureChanged.emit()
+
     def _save_shortcut(self, combo, action):
         rules = read_json(RULES_PATH)
         rules["key_bindings"] = dict(key_bindings(rules), **{combo: action})
@@ -935,9 +965,25 @@ class Backend(QObject):
 
     @Property(str, notify=gazeChanged)
     def gazeTracker(self):
-        """Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER)."""
+        """Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER; auto is
-        v = read_conf().get("GAZE_TRACKER", "steam")
+        ours when it's installed)."""
-        return v if v in GAZE_TRACKERS else "steam"
+        v = read_conf().get("GAZE_TRACKER", "auto")
+        if v in GAZE_TRACKERS:
+            return v
+        return "own" if self.gazeOwnInstalled else "steam"
+
+    @Property(bool, notify=gazeChanged)
+    def gazeTrackerAuto(self):
+        """GAZE_TRACKER is auto (or missing): ours once it's installed, without a choice here."""
+        return read_conf().get("GAZE_TRACKER", "auto") not in GAZE_TRACKERS
+
+    @Property(bool, notify=gazeChanged)
+    def gazeOwnInstalled(self):
+        """Is our own tracker installed? The gaze service says (it also checks ft-eyes' Python in
+        its checkout); without it, whether the frame grabber is."""
+        if "own_installed" in self._gaze:
+            return bool(self._gaze["own_installed"])
+        return any(os.path.exists(p) for p in EYEGRAB)
 
     @Property(str, notify=gazeChanged)
     def gazeEye(self):

input-settings/main.qml

@@ -192,7 +192,7 @@ Kirigami.ApplicationWindow {
                 wrapMode: Text.Wrap
                 opacity: 0.7
                 text: "Move or press a device to see which row it is. 3D pointer: grabbed, drives the SteamVR pointer. "
-                      + "Pass through: left alone (a Meta tap toggles the dashboard if META_DASHBOARD=1). Ignore: left alone."
+                      + "Pass through: left alone, but its key combinations (Keyboard page) work. Ignore: left alone."
             }
         }
     }
@@ -545,7 +545,11 @@ Kirigami.ApplicationWindow {
                     delegate: RowLayout {
                         required property var modelData
                         Kirigami.FormData.label: modelData.label + ":"
-                        Controls.Label { text: modelData.actionLabel }
+                        Controls.Label {
+                            text: modelData.actionLabel
+                            elide: Text.ElideRight
+                            Layout.maximumWidth: Kirigami.Units.gridUnit * 24
+                        }
                         Controls.ToolButton {
                             icon.name: "edit-delete"
                             display: Controls.AbstractButton.IconOnly
@@ -568,16 +572,33 @@ Kirigami.ApplicationWindow {
                     }
                     Controls.Button {
                         text: backend.capturingShortcut ? "Press the keys… (Cancel)" : "Set keys…"
+                        enabled: backend.capturingShortcut || shortcutAction.currentValue !== "command:"
+                                 || shortcutCommand.text.trim() !== ""
                         onClicked: backend.capturingShortcut ? backend.cancelShortcutCapture()
-                                                             : backend.startShortcutCapture(shortcutAction.currentValue)
+                                                             : backend.startShortcutCapture(
+                                                                   shortcutAction.currentValue === "command:"
+                                                                   ? "command:" + shortcutCommand.text.trim()
+                                                                   : shortcutAction.currentValue)
                     }
                 }
+                Controls.TextField {
+                    id: shortcutCommand
+                    Kirigami.FormData.label: "Command:"
+                    visible: shortcutAction.currentValue === "command:"
+                    placeholderText: "e.g. ft-layout use Work"
+                    Layout.preferredWidth: Kirigami.Units.gridUnit * 24
+                }
                 Controls.Label {
                     Layout.maximumWidth: Kirigami.Units.gridUnit * 30
                     wrapMode: Text.WordWrap
-                    text: "Hold the modifiers (Ctrl, Alt, Shift, Meta), then press the key, on any keyboard. The "
+                    text: "Hold the modifiers (Ctrl, Alt, Shift, Meta), then press the key, on any keyboard. Or tap "
-                          + "combination's last key isn't typed; the modifiers still reach the app. Meta+Shift+F floats "
+                          + "one modifier on its own. The combination's last key isn't typed; the modifiers still reach "
-                          + "the desktop window under the pointer in VR, or puts it back, until you remove or change it."
+                          + "the app. While you're in Steam or a game, they see the keys too. Until you remove or change "
+                          + "them, a Meta tap opens the Steam menu (or closes the dashboard) instead of the desktop's "
+                          + "launcher, and Meta+Shift+F floats the desktop window under the pointer in VR, or puts it "
+                          + "back. A command runs with sh as the input relay's service, outside the desktop's session, "
+                          + "with Frametop's ft-layout, ft-float and ft-steam on its path; its output goes to the "
+                          + "relay's log (journalctl --user -u frametop-input-relay)."
                     opacity: 0.7
                     font: Kirigami.Theme.smallFont
                 }
@@ -881,6 +902,23 @@ Kirigami.ApplicationWindow {
                     opacity: 0.7
                     font: Kirigami.Theme.smallFont
                 }
+                Controls.Label {
+                    // Why the gaze pointer, on, can't follow your eyes yet, so it isn't left looking
+                    // like a plain mouse. The calibration part is the gaze service's (gaze/gazecheck.py).
+                    readonly property var checks: gpage.status.checks || {}
+                    readonly property string why: !backend.gazeServiceInstalled
+                        ? "The gaze service isn't installed: run gaze/run.sh install in the Frametop folder, in a terminal"
+                        : !backend.gazeServiceRunning ? "The gaze service isn't running (it starts with SteamVR)"
+                        : backend.gazeTracker === "own" && !gpage.status.eyegrab
+                        ? "Our eye tracker needs its frame grabber: run gaze/tracker/install.sh (asks for sudo)"
+                        : checks.problem || ""
+                    visible: backend.gazeMode > 0 && why !== ""
+                    text: why
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 30
+                    wrapMode: Text.WordWrap
+                    color: checks.check ? Kirigami.Theme.neutralTextColor : Kirigami.Theme.negativeTextColor
+                    font: Kirigami.Theme.smallFont
+                }
                 ColumnLayout {
                     Kirigami.FormData.label: "Mouse left button:"
                     Repeater {
@@ -928,16 +966,28 @@ Kirigami.ApplicationWindow {
                 }
                 RowLayout {
                     Kirigami.FormData.label: "Eye tracker:"
+                    Controls.RadioButton {
+                        text: "Own tracker (recommended)"
+                        checked: backend.gazeTracker === "own"
+                        onToggled: if (checked) backend.setGazeTracker("own")
+                    }
                     Controls.RadioButton {
                         text: "SteamVR"
                         checked: backend.gazeTracker === "steam"
                         onToggled: if (checked) backend.setGazeTracker("steam")
                     }
-                    Controls.RadioButton {
-                        text: "Own tracker"
-                        checked: backend.gazeTracker === "own"
-                        onToggled: if (checked) backend.setGazeTracker("own")
                 }
+                Controls.Label {
+                    // Ours is the default once it's installed (GAZE_TRACKER=auto, gaze/ft-gazed).
+                    visible: backend.gazeTracker === "steam" && !backend.gazeOwnInstalled
+                    text: "Our own tracker is more accurate. Install it with gaze/tracker/install.sh in the "
+                          + "Frametop folder, in a terminal (asks for sudo)"
+                          + (backend.gazeTrackerAuto ? "; gaze then uses it, and you calibrate it once."
+                                                     : ", then pick Own tracker here and calibrate it once.")
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 30
+                    wrapMode: Text.WordWrap
+                    opacity: 0.7
+                    font: Kirigami.Theme.smallFont
                 }
                 Controls.Label {
                     // The gaze service runs ours (gaze/tracker/ft-eyes); it needs the frame grabber

input/ft-mousectl

@@ -1,168 +0,0 @@
-#!/usr/bin/env python3
-"""Native KDE mouse control, entirely over local IPC; no SteamVR client needed."""
-import argparse
-import importlib.util
-import json
-import math
-import os
-from pathlib import Path
-import re
-import socket
-import tempfile
-
-SCREENS = '\0ft_screens'
-RELAY = '\0frametop_relay'
-HELPER = '\0ft_pointer_helper'
-CONF = Path.home() / '.config/frametop.conf'
-
-
-def request(address, command):
-    with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as sock:
-        sock.bind('')
-        sock.settimeout(1)
-        sock.sendto(command.encode(), address)
-        result = sock.recv(8192).decode()
-    if result.startswith('error'):
-        raise RuntimeError(result)
-    return result
-
-
-def query(address, command):
-    try:
-        result = json.loads(request(address, command))
-        if not isinstance(result, dict): raise ValueError('Expected an object')
-        return result
-    except (OSError, ValueError, RuntimeError) as error:
-        return {'available': False, 'reason': str(error)}
-
-
-def status():
-    result = {'relay': query(RELAY, 'mouse-mode?'), 'desktop': query(SCREENS, 'mouse?'),
-              'pointer': query(HELPER, 'pointerstatus'),
-              'controllerFallback': query(SCREENS, 'mouse-presence?'),
-              'desktopPolicy': query(SCREENS, 'desktop-policy?')}
-    result['ready'] = bool(result['relay'].get('supported') and result['desktop'].get('supported')
-                           and 'desktopInput' in result['pointer'])
-    result['active'] = bool(result['ready'] and result['relay'].get('mode') == 'desktop'
-                            and result['desktop'].get('mode') == 'desktop'
-                            and result['pointer'].get('desktopInput'))
-    return result
-
-
-def require_ready():
-    result = status()
-    if not result['ready']:
-        raise RuntimeError('Native mouse support is not loaded in all three components. '
-                           'Inspect status; the updated desktop and input helpers must be running.')
-    return result
-
-
-def layout_outputs():
-    spec = importlib.util.spec_from_file_location('mouse_layout', Path(__file__).resolve().parents[1] / 'layout/ft_layout.py')
-    ft = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(ft)
-    env = ft.nested_env()
-    if env is None: raise RuntimeError('No running FrameTop KDE desktop')
-    outs = ft.outputs(env)
-    if not outs or not any(o.get('enabled') for o in outs):
-        raise RuntimeError('KDE output geometry is unavailable')
-    return outs
-
-
-def configure(outs):
-    commands = []
-    for index, output in enumerate(outs, 1):
-        if not output.get('enabled'):
-            commands.append(f'mouse-output-off {index}')
-            continue
-        scale = float(output['scale'])
-        if not math.isfinite(scale) or not .25 <= scale <= 8:
-            raise ValueError('Invalid KDE output scale')
-        w, h = (math.ceil(output['size'][k]/scale) for k in ('width', 'height'))
-        x, y = (float(output['pos'][k]) for k in ('x', 'y'))
-        if not all(math.isfinite(v) for v in (x, y)) or not (1 <= w <= 16384 and 1 <= h <= 16384):
-            raise ValueError('Invalid KDE output geometry')
-        commands.append(f'mouse-layout {index} {x:g} {y:g} {w} {h} {scale:g}')
-    # Validate the entire geometry before sending any updates.
-    for command in commands:
-        if request(SCREENS, command) != 'ok': raise RuntimeError('Mouse layout rejected')
-
-
-def save_mode(mode, path=None):
-    path = Path(path or CONF)
-    text = path.read_text() if path.exists() else ''
-    pattern = re.compile(r'^\s*MOUSE_MODE\s*=.*$', re.M)
-    text = pattern.sub('MOUSE_MODE='+mode, text) if pattern.search(text) else text.rstrip('\n')+'\nMOUSE_MODE='+mode+'\n'
-    path.parent.mkdir(parents=True, exist_ok=True)
-    fd, temporary = tempfile.mkstemp(prefix=path.name+'.', dir=path.parent)
-    try:
-        os.fchmod(fd, (path.stat().st_mode & 0o777) if path.exists() else 0o600)
-        with os.fdopen(fd, 'w') as file:
-            file.write(text)
-            file.flush()
-            os.fsync(file.fileno())
-        os.replace(temporary, path)
-    finally:
-        if os.path.exists(temporary): os.unlink(temporary)
-
-
-def switch(mode, save=False):
-    if mode == 'desktop':
-        require_ready()
-        configure(layout_outputs())
-    # Spatial recovery intentionally works with an absent desktop.
-    result = query(RELAY, 'mouse-mode '+mode)
-    if result.get('ok') is not True: raise RuntimeError(result.get('error', result.get('reason', 'Relay rejected mode')))
-    if save:
-        try: save_mode(mode)
-        except OSError as error:
-            raise RuntimeError(f'Live mode is {mode}, but saving the preference failed: {error}') from error
-    return dict(mode=mode, saved=save)
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    sub = parser.add_subparsers(dest='command', required=True)
-    sub.add_parser('status')
-    sub.add_parser('layout')
-    policy = sub.add_parser('policy', help='Choose desktop mouse/pointer ownership (runtime only)')
-    policy.add_argument('mode', choices=('mouse', 'pointer', 'last-active'))
-    for name in ('desktop', 'spatial'):
-        sub.add_parser(name).add_argument('--save', action='store_true')
-    pos = sub.add_parser('position')
-    pos.add_argument('screen', type=int)
-    pos.add_argument('x', type=float)
-    pos.add_argument('y', type=float)
-    center = sub.add_parser('center')
-    center.add_argument('screen', type=int)
-    args = parser.parse_args()
-    try:
-        if args.command == 'status': result = status()
-        elif args.command == 'policy':
-            require_ready()
-            if request(SCREENS, 'desktop-policy '+args.mode) != 'ok':
-                raise RuntimeError('Desktop policy rejected')
-            result = query(SCREENS, 'desktop-policy?')
-        elif args.command in ('desktop', 'spatial'): result = switch(args.command, args.save)
-        else:
-            require_ready()
-            outs = layout_outputs()
-            if args.command == 'layout':
-                configure(outs)
-                result = {'outputs': outs}
-            else:
-                if not 1 <= args.screen <= len(outs): raise ValueError('Invalid screen number')
-                if args.command == 'center':
-                    output = outs[args.screen-1]
-                    x, y = (math.ceil(output['size'][k]/output['scale'])/2 for k in ('width', 'height'))
-                else: x, y = args.x, args.y
-                if not all(math.isfinite(v) for v in (x, y)): raise ValueError('Position must be finite')
-                if request(SCREENS, f'mouse-position {args.screen} {x:g} {y:g}') != 'ok':
-                    raise RuntimeError('Mouse position rejected')
-                result = dict(screen=args.screen, x=x, y=y)
-        print(json.dumps(result, indent=2))
-    except (OSError, RuntimeError, ValueError, KeyError) as error:
-        parser.exit(1, str(error)+'\n')
-
-
-if __name__ == '__main__': main()

input/input-relay.py

@@ -14,8 +14,7 @@ keyboard node for its extra buttons). Roles, from ~/.config/frametop-input.json
 (written by the Frametop Input Settings app):
   pointer      grabbed; drives the universal 3D mouse (default for devices with a mouse node)
   passthrough  keys go to the desktop; grabbed only while typing goes there, otherwise only observed,
-               e.g. for the Meta dashboard shortcut (default for keyboards; the shortcut is off
+               for the key combinations below (default for keyboards)
-               unless META_DASHBOARD=1 is in ~/.config/frametop.conf)
   ignore       not grabbed, only observed for identification in the settings app
 Buttons and keys of pointer devices go through a per-device map to actions
 (left, right, middle, back, scroll_up, scroll_down, dashboard, recenter,
@@ -32,19 +31,26 @@ layout_reset = put the desktop screens back in their saved layout, screens_toggl
 keyboard_toggle = open or close Frametop's keyboard, float_toggle = float the desktop window under the
 pointer (else the active one) in VR, or put it back if it floats, dock_all = put every floating
 window back (both to ft-floatd, @frametop_float), profile:NAME = switch to that profile (ft-layout
-use NAME: its screens and apps; docs/profiles.md), key = pass through as a key, none).
+use NAME: its screens and apps; docs/profiles.md), steam_menu = open the SteamVR dashboard on
+Steam's menu, or close the dashboard (steam/ft-steam menu, through Steam's UI), command:CMD = run
+CMD with sh -c (on the host, as this service: its environment, output to its log, and
+COMMAND_PATH, so ft-layout, ft-float and ft-steam need no path), key = pass through as a key, none).
 
 Frame controller buttons can be mapped too ("controller_buttons": {"right/a": action} in the
 rules file; any action but key and the gaze ones, GAZE_ACTIONS: gaze mode is a mouse feature,
 docs/gaze-controllers.md). So can key combinations on any keyboard ("key_bindings":
 {"29+56+34": action}, evdev codes joined by "+", modifiers first and left-hand codes for
 either side, here Ctrl+Alt+G): the combination does the action, and its last key isn't typed.
-A rules file without "key_bindings" gets DEFAULT_KEY_BINDINGS (Meta+J: gaze_left, Meta+K:
+A modifier on its own ("125": Meta) is a tap: pressed and released with no other key, mouse
-gaze_right, Meta+Shift+F: float_toggle); one with its own, even an empty one, doesn't. The float
+button, or scroll in between; the desktop gets an F24 press before its release, so Plasma's
-actions work without pointer mode too. A combination with Meta also sends the desktop an F24 press
+launcher doesn't open on a Meta tap that's bound. A rules file without "key_bindings" gets
-and Meta's release right away: so letting go of Meta doesn't open Plasma's launcher, and a gaze
+DEFAULT_KEY_BINDINGS (Meta tap: steam_menu, Meta+J: gaze_left, Meta+K: gaze_right, Meta+Shift+F:
-click isn't Meta+click (KWin's window move and resize). Another key while Meta is still held gives
+float_toggle); one with its own, even an empty one, doesn't. The float, profile, Steam menu and
-the desktop Meta back. The controllers aren't input devices here, only SteamVR sees
+command actions work without pointer mode too. A combination with Meta also sends the desktop an
+F24 press and Meta's release right away: so letting go of Meta doesn't open Plasma's launcher, and
+a gaze click isn't Meta+click (KWin's window move and resize). Another key while Meta is still
+held gives the desktop Meta back. While typing goes to Steam, keyboards aren't grabbed, so Steam
+or the game sees a combination's keys too. The controllers aren't input devices here, only SteamVR sees
 them, so the pointer helper reads them with SteamVR input and sends "vrbtn <button> 1|0".
 It only takes the buttons the relay tells it to ("vrbind <button>..." to @ft_pointer_helper,
 sent on start, reload, and when the helper says "vrhello"), and only while no game runs,
@@ -115,7 +121,6 @@ import atexit
 import errno
 import fcntl
 import json
-import math
 import os
 import select
 import signal
@@ -130,7 +135,8 @@ EV_SYN, EV_KEY, EV_REL, EV_MSC = 0x00, 0x01, 0x02, 0x04
 SYN_REPORT = 0
 BTN_MISC, KEY_MAX = 0x100, 0x2FF
 KEY_A = 30
-REL_X, REL_Y, REL_HWHEEL, REL_WHEEL, REL_MAX = 0x00, 0x01, 0x06, 0x08, 0x0F
+REL_X, REL_Y, REL_WHEEL, REL_MAX = 0x00, 0x01, 0x08, 0x0F
+SCROLLS = {0x06, REL_WHEEL, 0x0B, 0x0C}  # REL_HWHEEL, REL_WHEEL and their _HI_RES
 BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_SIDE, BTN_EXTRA = 0x110, 0x111, 0x112, 0x113, 0x114
 KEY_LEFTMETA, KEY_RIGHTMETA = 125, 126
 KEY_VOLUMEDOWN, KEY_VOLUMEUP = 114, 115
@@ -190,12 +196,14 @@ RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
 ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
            "pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right",
            "gaze_quickcal", "sens_up", "sens_down", "layout_reset", "screens_toggle", "keyboard_toggle", "float_toggle",
-           "dock_all", "key", "none")
+           "dock_all", "steam_menu", "key", "none")
 # Gaze mode is a mouse feature: these never come from a controller button (docs/gaze-controllers.md).
 GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
-# Key combinations a rules file without "key_bindings" gets: Meta+J and Meta+K click at the gaze
+# Key combinations a rules file without "key_bindings" gets: a Meta tap opens Steam's menu, Meta+J
-# (free on the Frametop desktop, and apps don't use Meta), Meta+Shift+F floats a window.
+# and Meta+K click at the gaze (free on the Frametop desktop, and apps don't use Meta),
-DEFAULT_KEY_BINDINGS = {"125+36": "gaze_left", "125+37": "gaze_right", "42+125+33": "float_toggle"}
+# Meta+Shift+F floats a window.
+DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
+                        "42+125+33": "float_toggle"}
 KEY_F24 = 194  # sent to the desktop with a Meta combination (see the top)
 # Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
 MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
@@ -212,16 +220,23 @@ FLOAT = "\0frametop_float"  # ft-floatd, floating windows in the Frametop deskto
 # Actions for ft-floatd ("float_toggle", "dock_all"): they don't need pointer mode.
 FLOAT_ACTIONS = {"float_toggle": b"float pointer", "dock_all": b"dock all"}
 PROFILE = "profile:"  # "profile:NAME": switch to that profile (doesn't need pointer mode either)
+COMMAND = "command:"  # "command:CMD": run CMD (nor does this)
 
 
 def known_action(a):
-    return a in ACTIONS or (isinstance(a, str) and a.startswith(PROFILE) and len(a) > len(PROFILE))
+    return a in ACTIONS or (isinstance(a, str) and any(a.startswith(p) and a[len(p):].strip()
+                                                       for p in (PROFILE, COMMAND)))
 
 
 def needs_pointer(a):
-    return a not in FLOAT_ACTIONS and not a.startswith(PROFILE)
+    return a not in FLOAT_ACTIONS and a != "steam_menu" and not a.startswith((PROFILE, COMMAND))
 KEYS = "\0frametop_keys"  # keys of keyboards grabbed for the desktop, for other readers
-FT_LAYOUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "layout", "ft-layout")
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+FT_LAYOUT = os.path.join(REPO, "layout", "ft-layout")
+FT_STEAM = os.path.join(REPO, "steam", "ft-steam")
+# Commands ("command:CMD") find Frametop's own tools (ft-layout, ft-float, ft-steam) on their PATH.
+COMMAND_PATH = ":".join([os.path.join(REPO, d) for d in ("layout", "float", "steam")]
+                        + [os.environ.get("PATH", "/usr/local/bin:/usr/bin")])
 DEFAULT_BUTTONS = {BTN_LEFT: "left", BTN_RIGHT: "right", BTN_MIDDLE: "middle",
                    BTN_SIDE: "back", BTN_EXTRA: "back"}
 
@@ -591,75 +606,6 @@ class Pointer:
         return 0.02 if any(t is not None for t in pending) else 0.5
 
 
-def desktop_mouse_present(nodes):
-    """Only a physical mouse actually routed by this relay owns the desktop."""
-    return any(n.is_mouse and n.role == "pointer" and n.grabbed for n in nodes)
-
-
-def ipc_request(address, command, timeout=.2):
-    """Bounded local request/reply, with no OpenVR client or desktop wake."""
-    with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as sock:
-        sock.bind("")
-        sock.settimeout(timeout)
-        sock.sendto(command.encode(), address)
-        reply = sock.recv(8192).decode()
-    if reply.startswith("error"):
-        raise RuntimeError(reply)
-    return reply
-
-
-class DesktopMouse:
-    """Physical mouse -> nested KDE seat. Never sends to the VR pointer helper."""
-    def __init__(self, speed=1):
-        if not math.isfinite(speed) or not .05 <= speed <= 10:
-            raise ValueError("Desktop mouse speed must be between .05 and 10")
-        self.speed = speed
-        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
-        self.dx = self.dy = 0
-        self.wheel = [0, 0]
-        self.hires = [0, 0]
-        self.has_hires = [False, False]
-        self.held = set()
-        self.sent = self.failed = 0
-
-    def send(self, command):
-        try:
-            self.sock.sendto(command.encode(), SCREENS)
-            self.sent += 1
-        except OSError:
-            self.failed += 1
-
-    def motion(self, code, value):
-        if code == REL_X: self.dx += value
-        elif code == REL_Y: self.dy += value
-        elif code in (REL_HWHEEL, REL_WHEEL): self.wheel[code == REL_WHEEL] += value
-        elif code in (12, 11):
-            axis = int(code == 11)
-            self.hires[axis] += value
-            self.has_hires[axis] = True
-
-    def flush(self):
-        if self.dx or self.dy:
-            self.send(f"mouse-move {self.dx*self.speed:.6f} {self.dy*self.speed:.6f}")
-            self.dx = self.dy = 0
-        wheel = [self.hires[i]/120 if self.has_hires[i] else self.wheel[i] for i in (0, 1)]
-        if any(wheel): self.send(f"mouse-wheel {wheel[0]:.6f} {wheel[1]:.6f}")
-        self.wheel = [0, 0]; self.hires = [0, 0]; self.has_hires = [False, False]
-
-    def button(self, code, value):
-        if not BTN_LEFT <= code <= 279 or value not in (0, 1): return
-        self.flush()
-        self.send(f"mouse-button {code} {value}")
-        if value: self.held.add(code)
-        else: self.held.discard(code)
-
-    def release_all(self):
-        self.dx = self.dy = 0
-        self.wheel = [0, 0]; self.hires = [0, 0]; self.has_hires = [False, False]
-        for code in sorted(self.held): self.send(f"mouse-button {code} 0")
-        self.held.clear()
-
-
 class Node:
     """One input event node: a candidate device (mouse or keyboard, USB or Bluetooth),
     or any other device with volume keys (candidate False, role "volume")."""
@@ -742,23 +688,13 @@ def main():
     # desktop_until: typing goes to the Frametop desktop until then (ft-screens says so
     # every second); typing_applied: the grabs match that as of the last apply_roles().
     # vr_capture_until: every controller button is taken until then (the settings app capturing one).
-    state = {"pointer": None, "rules": {}, "meta_dashboard": False, "share_keys": False,
+    state = {"pointer": None, "rules": {}, "share_keys": False,
-             "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0,
+             "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0}
-             "desktop_mouse": None, "mouse_mode": "spatial", "desktop_speed": 1.0,
-             "controller_fallback": True}
 
     def load_config():
         conf = read_config()
         state["rules"] = read_rules()
-        state["meta_dashboard"] = conf.get("META_DASHBOARD", "0") == "1"
         state["share_keys"] = conf.get("SHARE_KEYS", "0") == "1"
-        state["mouse_mode"] = "desktop" if conf.get("MOUSE_MODE") == "desktop" else "spatial"
-        state["controller_fallback"] = conf.get("DESKTOP_CONTROLLER_FALLBACK", "1") == "1"
-        try:
-            speed = float(conf.get("DESKTOP_MOUSE_SPEED", "1"))
-            state["desktop_speed"] = speed if math.isfinite(speed) and .05 <= speed <= 10 else 1
-        except ValueError:
-            state["desktop_speed"] = 1
         if conf.get("POINTER", "0") == "1":
             p = state["pointer"] or Pointer(0.03, 30)
             p.sensitivity = float(conf.get("POINTER_SENSITIVITY", "0.03"))
@@ -771,15 +707,13 @@ def main():
             log("pointer mode off: pointer devices feed the virtual mouse and keyboard")
 
     load_config()
-    meta_down = False  # Meta pressed with no other key yet: a tap toggles the dashboard
+    tap = None  # the modifier (folded, MODIFIERS) pressed alone, with nothing since: its release is a tap
     screens_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
     last_typing = 0.0  # the helper was last told of a key then (see "typing" at the top)
 
     def vr_bind(now):
         """Tell the pointer helper which controller buttons to take from games."""
-        if state["desktop_mouse"]:
+        if now < state["vr_capture_until"]:
-            buttons = "-"  # leave every physical controller button to SteamVR/games
-        elif now < state["vr_capture_until"]:
             buttons = "*"
         else:
             state["vr_capture_until"] = 0.0
@@ -802,7 +736,7 @@ def main():
                 reply(addr, {"t": "event", "id": VR_DEVICE, "path": "", "name": "Steam Frame controllers",
                              "type": "vr", "code": button, "value": value})
         action = state["rules"]["controller_buttons"].get(button)
-        if not state["desktop_mouse"] and (state["pointer"] or (action and not needs_pointer(action))) and known_action(action) \
+        if (state["pointer"] or (action and not needs_pointer(action))) and known_action(action) \
                 and action not in ("key", "none") and action not in GAZE_ACTIONS:
             do_action(action, value, now, button.split("/")[0])
 
@@ -829,7 +763,7 @@ def main():
 
     def do_action(action, value, now, source="mouse"):
         """A mapped mouse or controller button, or key combination (pointer mode only, but
-        for FLOAT_ACTIONS and profiles)."""
+        for the actions needs_pointer() says don't)."""
         if action == "keyboard_toggle":
             if value == 1 and vr_keyboard_mode() != "never":
                 vr_keyboard("toggle")
@@ -839,6 +773,17 @@ def main():
                 subprocess.Popen([FT_LAYOUT, "use", action[len(PROFILE):]], stdin=subprocess.DEVNULL,
                                  stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True)
                 log(action)
+        elif action == "steam_menu":
+            if value == 1:
+                # Talks to Steam's UI for a moment; its errors go to this service's log.
+                subprocess.Popen([FT_STEAM, "menu"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
+                                 start_new_session=True)
+                log(action)
+        elif action.startswith(COMMAND):
+            if value == 1:
+                subprocess.Popen(["sh", "-c", action[len(COMMAND):]], stdin=subprocess.DEVNULL,
+                                 env=dict(os.environ, PATH=COMMAND_PATH), start_new_session=True)
+                log(action)
         elif action in FLOAT_ACTIONS:
             if value == 1:
                 try:
@@ -857,14 +802,20 @@ def main():
     def key_binding(code, value, now):
         """A key from a keyboard: does it complete a key combination ("key_bindings")? True if
         it was taken for one (then it isn't typed)."""
-        nonlocal meta_down
+        nonlocal tap
         if code in MODIFIERS:
-            (held_modifiers.add if value else held_modifiers.discard)(MODIFIERS[code])
+            mod = MODIFIERS[code]
+            if value == 1:
+                tap = mod if not held_modifiers and not combos_down else None
+            (held_modifiers.add if value else held_modifiers.discard)(mod)
             if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
                 (held_meta.add if value else held_meta.discard)(code)
                 if value == 0 and code in meta_hidden:
                     meta_hidden.discard(code)
                     return True  # the desktop already had it come up (below)
+            if value == 0 and tap == mod:
+                tap = None
+                modifier_tap(mod, now)
             return False
         if value == 1 and code not in combos_down and meta_hidden:
             # Another key while Meta is still held after a combination: the desktop gets Meta
@@ -886,10 +837,9 @@ def main():
         combos_down[code] = action
         if held_meta - meta_hidden:
             # The desktop saw Meta go down. Another key in between keeps its release from
-            # opening Plasma's launcher (and Meta from toggling the dashboard here), and Meta
+            # opening Plasma's launcher, and Meta comes up there now: KWin takes Meta with a
-            # comes up there now: KWin takes Meta with a mouse button for moving or resizing
+            # mouse button for moving or resizing windows, which would swallow a gaze click.
-            # windows, which would swallow a gaze click. Its real release is dropped (above).
+            # Its real release is dropped (above).
-            meta_down = False
             to_screens(KEY_F24, 1)
             to_screens(KEY_F24, 0)
             for c in sorted(held_meta - meta_hidden):
@@ -900,6 +850,20 @@ def main():
         log(f"key combination {combo}: {action}")
         return True
 
+    def modifier_tap(mod, now):
+        """A modifier pressed and released alone: its binding, if it has one ("125": Meta tap)."""
+        action = state["rules"]["key_bindings"].get(str(mod))
+        if not known_action(action) or action in ("key", "none"):
+            return
+        # The desktop gets a key in between before the release goes there, so the tap isn't one
+        # there too: a Meta tap would open Plasma's launcher.
+        to_screens(KEY_F24, 1)
+        to_screens(KEY_F24, 0)
+        if state["pointer"] or not needs_pointer(action):
+            do_action(action, 1, now, "keyboard")
+            do_action(action, 0, now, "keyboard")
+        log(f"key combination {mod}: {action}")
+
 
     screens_down = set()  # keys the desktop was told went down and not yet up (see reconcile_desktop_keys)
 
@@ -983,12 +947,6 @@ def main():
         return "pointer" if has_mouse else "passthrough"
 
     def release_held(node):
-        if node.role == "pointer" and state["desktop_mouse"]:
-            state["desktop_mouse"].release_all()
-            for code in node.held:
-                if code < BTN_MISC: to_screens(code, 0)
-            node.held.clear()
-            return
         for code in node.held:
             if node.role == "passthrough":
                 share_key(node, code, 0)
@@ -1082,40 +1040,6 @@ def main():
         except OSError:
             watchers.pop(addr, None)
 
-    def set_mouse_mode(mode, now):
-        if any(keys_down(n) for n in nodes.values() if n.role == "pointer"):
-            raise RuntimeError("Release mouse buttons before switching modes")
-        native = state["desktop_mouse"]
-        if native and native.held:
-            raise RuntimeError("Release mouse buttons before switching modes")
-        if mode == "desktop":
-            ipc_request(SCREENS, "mouse-mode desktop")
-            try:
-                ipc_request(HELPER, "desktopmouse on")
-            except (OSError, RuntimeError):
-                ipc_request(SCREENS, "mouse-mode spatial")
-                raise
-            if native:
-                state["mouse_mode"] = mode
-                return
-            state["desktop_mouse"] = DesktopMouse(state["desktop_speed"])
-            if state["pointer"]:
-                state["pointer"].active = False
-                state["pointer"].dx = state["pointer"].dy = 0
-                state["pointer"].claim_at = state["pointer"].claim_release = None
-                state["pointer"].scroll_until = state["pointer"].system_at = state["pointer"].system_release = None
-        else:
-            # An absent desktop must not prevent explicit spatial recovery.
-            try: ipc_request(SCREENS, "mouse-mode spatial")
-            except OSError: pass
-            ipc_request(HELPER, "desktopmouse off")
-            if native:
-                native.release_all(); native.sock.close()
-            state["desktop_mouse"] = None
-        state["mouse_mode"] = mode
-        vr_bind(now)
-        log(f"mouse mode: {mode}")
-
     def handle_control(now):
         while True:
             try:
@@ -1152,18 +1076,6 @@ def main():
                     reply(addr, {"t": "devices", "pointer_mode": state["pointer"] is not None,
                                  "actions": ACTIONS,
                                  "nodes": [n.describe() for n in nodes.values() if n.candidate]})
-                elif cmd == "mouse-mode?":
-                    native = state["desktop_mouse"]
-                    reply(addr, {"supported": True, "mode": "desktop" if native else "spatial",
-                                 "requestedMode": state["mouse_mode"], "speed": state["desktop_speed"],
-                                 "heldButtons": sorted(native.held) if native else [],
-                                 "sent": native.sent if native else 0, "sendFailures": native.failed if native else 0})
-                elif cmd == "mouse-mode" and len(words) == 2 and words[1] in ("desktop", "spatial"):
-                    try:
-                        set_mouse_mode(words[1], now)
-                        reply(addr, {"mode": words[1], "ok": True})
-                    except (OSError, RuntimeError, ValueError) as error:
-                        reply(addr, {"error": str(error), "ok": False})
                 elif cmd == "watch":
                     seconds = float(words[1]) if len(words) > 1 else 30
                     watchers[addr] = now + min(seconds, 600)
@@ -1211,9 +1123,6 @@ def main():
         if now >= next_scan:
             next_scan = now + 1.0
             reconcile_desktop_keys()
-            if (state["mouse_mode"] == "desktop") != bool(state["desktop_mouse"]):
-                try: set_mouse_mode(state["mouse_mode"], now)
-                except (OSError, RuntimeError, ValueError): pass  # desktop may be starting; retry on the next scan
             current = {}
             for name in os.listdir("/dev/input"):
                 if name.startswith("event"):
@@ -1240,18 +1149,11 @@ def main():
                     added = True
             if added:
                 apply_roles()
-            # Presence, not activity: an idle connected mouse keeps its own seat.
-            present = desktop_mouse_present(nodes.values())
-            try:
-                ipc_request(SCREENS, "mouse-presence %d %d" %
-                            (present, state["controller_fallback"]))
-            except (OSError, RuntimeError, ValueError):
-                pass  # Older/missing compositor: preserve existing native mode.
 
         ready, _, _ = select.select(list(nodes) + [control], [], [],
                                     volume.timeout(now, pointer.timeout() if pointer else 0.5))
         now = time.monotonic()
-        if pointer and not state["desktop_mouse"]:
+        if pointer:
             pointer.tick(now)
         volume.tick(now)
         if state["vr_capture_until"] and now >= state["vr_capture_until"]:
@@ -1279,50 +1181,39 @@ def main():
                 if etype in (EV_KEY, EV_REL):
                     broadcast(node, etype, VOLUME_ORIGINAL.get(code, code) if node.remapped else code,
                               value, now)
+                if (etype == EV_KEY and value == 1 and code not in MODIFIERS) or (etype == EV_REL and code in SCROLLS):
+                    tap = None  # a key, button or scroll in between: a modifier's release isn't a tap
                 if etype == EV_KEY and ((node.remapped and code in VOLUME_ORIGINAL)
                                         or (node.grabbed and code in VOLUME_STANDIN)):
                     volume.key(fd, code, value, now)
-                    if value == 1:
-                        meta_down = False  # Meta used as a modifier, not a tap
                     continue
                 if node.role == "volume":
                     continue
                 if node.role != "pointer":
-                    # Observed only, unless typing goes to the desktop. With META_DASHBOARD=1,
+                    # Observed only, unless typing goes to the desktop. Key combinations work on
-                    # a Meta tap on any keyboard toggles the dashboard.
+                    # any pass-through keyboard.
-                    if node.role == "passthrough" and etype == EV_KEY and code < BTN_MISC and key_binding(code, value, now):
+                    if (node.role == "passthrough" and etype == EV_KEY and node.grabbed
-                        continue
+                            and code < BTN_MISC and value in (0, 1)):
-                    if node.role == "passthrough" and etype == EV_KEY:
+                        # Shared as pressed, key combinations included: the Meta release a
-                        if value == 1 and code < BTN_MISC and now - last_typing >= 0.25:
+                        # combination keeps from the desktop must still reach frame-voice, or
-                            last_typing = now
+                        # it waits for that release before typing anything.
-                            screens_sock.sendto(b"typing", HELPER)
-                        to_screens(code, value)
-                        if node.grabbed and code < BTN_MISC and value in (0, 1):
                         share_key(node, code, value)
                         if value:
                             node.held.add(code)
                         else:
                             node.held.discard(code)
-                    if (pointer and not state["desktop_mouse"] and state["meta_dashboard"] and node.role == "passthrough"
+                    if node.role == "passthrough" and etype == EV_KEY and code < BTN_MISC and key_binding(code, value, now):
-                            and etype == EV_KEY):
+                        continue
-                        if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
+                    if node.role == "passthrough" and etype == EV_KEY:
-                            if value == 1:
+                        if value == 1 and code < BTN_MISC and now - last_typing >= 0.25:
-                                meta_down = True
+                            last_typing = now
-                            elif value == 0 and meta_down:
+                            try:
-                                meta_down = False
+                                screens_sock.sendto(b"typing", HELPER)
-                                pointer.dashboard()
+                            except OSError:
-                        elif value == 1:
+                                pass  # helper not running (SteamVR not running)
-                            meta_down = False  # Meta used as a modifier, not a tap
+                        to_screens(code, value)
                     continue
                 if etype == EV_KEY:
-                    if state["desktop_mouse"]:
-                        if code >= BTN_LEFT:
-                            state["desktop_mouse"].button(code, value)
-                        else:
-                            to_screens(code, value)
-                        if value: node.held.add(code)
-                        else: node.held.discard(code)
-                        continue
                     action = buttons.get(str(code), DEFAULT_BUTTONS.get(code, "key"))
                     if pointer and action not in ("key", "none"):
                         do_action(action, value, now)
@@ -1337,16 +1228,12 @@ def main():
                     else:
                         node.held.discard(code)
                 elif etype == EV_REL:
-                    if state["desktop_mouse"]:
+                    if pointer:
-                        state["desktop_mouse"].motion(code, value)
-                    elif pointer:
                         pointer.motion(code, value, now)
                     else:
                         mouse.emit(etype, code, value)
                 elif etype == EV_SYN and code == SYN_REPORT:
-                    if state["desktop_mouse"]:
+                    if pointer:
-                        state["desktop_mouse"].flush()
-                    elif pointer:
                         pointer.flush()
                     mouse.sync()
                     keyboard.sync()

input/test/keys-test.py

@@ -0,0 +1,219 @@
+#!/usr/bin/env python3
+"""Offline test of the input relay's key combinations and modifier taps.
+
+Runs the relay's main() against a fake pass-through keyboard and a fake mouse (pipes), with
+every socket it sends to renamed, no uinput devices, no grabs, and ft-steam swapped for a
+logger. Nothing reaches the live desktop, SteamVR, or the running relay, so it's safe next
+to them. Rules come from the test, not ~/.config/frametop-input.json.
+
+  input/test/keys-test.py [RELAY]   (default: input/input-relay.py next to this folder)
+"""
+import fcntl
+import importlib.util
+import os
+import shutil
+import socket
+import struct
+import sys
+import tempfile
+import threading
+import time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+relay_path = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "..", "input-relay.py")
+tag = f"ft_keys_test_{os.getpid()}"
+with open(relay_path) as f:
+    src = f.read().replace('"\\0frametop_relay"', f'"\\0{tag}_relay"')
+relay = importlib.util.module_from_spec(importlib.util.spec_from_loader("relay", loader=None))
+relay.__file__ = os.path.abspath(relay_path)
+exec(compile(src, relay_path, "exec"), relay.__dict__)
+
+for name in ("SCREENS", "HELPER", "FLOAT", "GAZED", "KEYS"):
+    setattr(relay, name, f"\0{tag}_{name.lower()}")
+OUT = tempfile.mkdtemp(prefix="ft-keys-test-")
+steam_log, cmd_log = f"{OUT}/steam.log", f"{OUT}/cmd.log"
+relay.FT_STEAM = f"{OUT}/ft-steam"
+with open(relay.FT_STEAM, "w") as f:
+    f.write(f'#!/bin/sh\necho "$@" >> {steam_log}\n')
+os.chmod(relay.FT_STEAM, 0o755)
+
+
+class NoDevice:
+    def __init__(self, *args, **kwargs):
+        pass
+
+    def emit(self, *args):
+        pass
+
+    def sync(self):
+        pass
+
+
+relay.Virtual = NoDevice
+relay.log = lambda *args: None  # the relay's own log
+bindings = {"now": None}  # the rules' key_bindings; None: the relay's defaults
+
+
+def read_rules(path=None):
+    rules = {"devices": {}, "buttons": {}, "controller_buttons": {}}
+    rules["key_bindings"] = dict(relay.DEFAULT_KEY_BINDINGS if bindings["now"] is None else bindings["now"])
+    return rules
+
+
+relay.read_rules = read_rules
+relay.read_config = lambda path=None: {"POINTER": "0"}
+
+# The fake devices: /dev/input/event900 (keyboard) and event901 (mouse), each a pipe.
+kb_r, kb_w = os.pipe()
+ms_r, ms_w = os.pipe()
+for fd in (kb_r, ms_r):
+    os.set_blocking(fd, False)
+FAKE = {"/dev/input/event900": kb_r, "/dev/input/event901": ms_r}
+HELD = {kb_r: set(), ms_r: set()}  # what EVIOCGKEY says each holds
+
+
+class Inode:
+    st_ino = 1
+
+
+fake_os = type(os)("os")
+fake_os.__dict__.update(os.__dict__)
+fake_os.listdir = lambda path: ["event900", "event901"] if path == "/dev/input" else os.listdir(path)
+fake_os.stat = lambda path, *a, **k: Inode() if path in FAKE else os.stat(path, *a, **k)
+relay.os = fake_os
+
+
+def ioctl(fd, request, arg=0, *rest):
+    if fd in HELD and request == relay.EVIOCGKEY:
+        for c in HELD[fd]:
+            arg[c // 8] |= 1 << (c % 8)
+        return 0
+    return fcntl.ioctl(fd, request, arg, *rest)
+
+
+fake_fcntl = type(fcntl)("fcntl")
+fake_fcntl.__dict__.update(fcntl.__dict__)
+fake_fcntl.ioctl = ioctl
+relay.fcntl = fake_fcntl
+
+
+def probe(path):
+    if FAKE[path] == kb_r:
+        return relay.Node(path, kb_r, "test keyboard", relay.BUS_USB, 1, 2, "", False, True)
+    return relay.Node(path, ms_r, "test mouse", relay.BUS_USB, 3, 4, "", True, False)
+
+
+relay.probe = probe
+
+screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+screens.bind(relay.SCREENS)
+screens.settimeout(0.05)
+EVENT = struct.Struct("llHHi")
+
+
+def send(fd, etype, code, value):
+    os.write(fd, EVENT.pack(0, 0, etype, code, value) + EVENT.pack(0, 0, 0, 0, 0))
+    time.sleep(0.03)
+
+
+def key(code, value, fd=kb_w):
+    held = HELD[ms_r if fd == ms_w else kb_r]
+    (held.add if value else held.discard)(code)
+    send(fd, relay.EV_KEY, code, value)
+
+
+def typed():
+    """What the desktop was sent since the last call."""
+    got = []
+    while True:
+        try:
+            got.append(screens.recv(256).decode())
+        except socket.timeout:
+            return got
+
+
+def use(key_bindings):
+    bindings["now"] = key_bindings
+    c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    c.bind("")
+    c.settimeout(2)
+    c.sendto(b"reload", f"\0{tag}_relay")
+    c.recv(4096)
+    time.sleep(0.05)
+    typed()
+
+
+def lines(path, wait=0.3):
+    time.sleep(wait)
+    try:
+        with open(path) as f:
+            return f.read().split()
+    except OSError:
+        return []
+
+
+failures = []
+
+
+def check(label, got, want):
+    ok = got == want
+    print(("ok    " if ok else "FAIL  ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(label)
+
+
+META, RMETA, SHIFT, CTRL, RCTRL, A, F, J = 125, 126, 42, 29, 97, 30, 33, 36
+F24 = ["key 194 1", "key 194 0"]
+
+
+def tests():
+    time.sleep(1.5)  # the relay's first device scan
+    typed()
+    key(META, 1); key(META, 0)
+    check("Meta tap (default): Steam menu", lines(steam_log), ["menu"])
+    check("Meta tap: the desktop gets F24 before Meta's release", typed(), ["key 125 1"] + F24 + ["key 125 0"])
+
+    key(RMETA, 1); key(RMETA, 2); key(RMETA, 2); key(RMETA, 0)
+    check("right Meta held (autorepeat), then released: a tap", lines(steam_log), ["menu", "menu"])
+    typed()
+
+    key(META, 1); key(A, 1); key(A, 0); key(META, 0)
+    check("Meta+A: no tap", lines(steam_log), ["menu", "menu"])
+    check("Meta+A (unbound) is typed as is", typed(), ["key 125 1", "key 30 1", "key 30 0", "key 125 0"])
+
+    key(META, 1); key(relay.BTN_LEFT, 1, ms_w); key(relay.BTN_LEFT, 0, ms_w); key(META, 0)
+    check("Meta+click: no tap", lines(steam_log), ["menu", "menu"])
+    key(META, 1); send(ms_w, relay.EV_REL, relay.REL_WHEEL, 1); key(META, 0)
+    check("Meta+scroll: no tap", lines(steam_log), ["menu", "menu"])
+    key(META, 1); key(SHIFT, 1); key(SHIFT, 0); key(META, 0)
+    check("Meta+Shift: no tap", lines(steam_log), ["menu", "menu"])
+    typed()
+
+    key(META, 1); key(J, 1); key(J, 0); key(META, 0)
+    check("Meta+J (default gaze click): no tap", lines(steam_log), ["menu", "menu"])
+    check("Meta+J: F24, Meta up early, its real release dropped", typed(), ["key 125 1"] + F24 + ["key 125 0"])
+
+    use({"29+42+33": f"command:echo combo >> {cmd_log}", "125": "none"})
+    key(CTRL, 1); key(SHIFT, 1); key(F, 1); key(F, 0); key(SHIFT, 0); key(CTRL, 0)
+    check("Ctrl+Shift+F runs its command", lines(cmd_log), ["combo"])
+    check("the combination's last key isn't typed", [k for k in typed() if k.startswith("key 33 ")], [])
+    key(META, 1); key(META, 0)
+    check("Meta tap bound to none: nothing", lines(steam_log), ["menu", "menu"])
+    check("Meta tap bound to none: no F24", typed(), ["key 125 1", "key 125 0"])
+
+    use({"29": f"command:command -v ft-steam ft-layout ft-float | wc -l >> {cmd_log}"})
+    key(RCTRL, 1); key(RCTRL, 0)
+    check("right Ctrl tap runs its command, which finds Frametop's tools", lines(cmd_log), ["combo", "3"])
+    check("Ctrl tap: F24 before Ctrl's release", typed(), ["key 97 1"] + F24 + ["key 97 0"])
+
+    check("an empty command isn't an action", relay.known_action("command:  "), False)
+    check("steam_menu and commands work without pointer mode",
+          (relay.needs_pointer("steam_menu"), relay.needs_pointer("command:ls")), (False, False))
+    print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+    shutil.rmtree(OUT, ignore_errors=True)
+    os._exit(1 if failures else 0)
+
+
+threading.Thread(target=tests, daemon=True).start()
+sys.argv = [relay_path, "--no-grab"]
+relay.main()

input/test_desktop_mouse.py

@@ -1,147 +0,0 @@
-"""Offline physical mouse tests: no input devices, desktop or SteamVR accessed."""
-import importlib.machinery
-import importlib.util
-import math
-from pathlib import Path
-import tempfile
-import unittest
-from types import SimpleNamespace
-from unittest.mock import patch
-
-
-def module(name, file):
-    loader = importlib.machinery.SourceFileLoader(name, str(Path(__file__).with_name(file)))
-    spec = importlib.util.spec_from_loader(name, loader)
-    result = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(result)
-    return result
-
-
-relay = module('desktop_relay', 'input-relay.py')
-ctl = module('mouse_control', 'ft-mousectl')
-
-
-class DesktopMouseTests(unittest.TestCase):
-    def test_presence_requires_a_routed_grabbed_physical_mouse(self):
-        def node(mouse=True, role='pointer', grabbed=True):
-            return SimpleNamespace(is_mouse=mouse, role=role, grabbed=grabbed)
-        self.assertFalse(relay.desktop_mouse_present([]))
-        for n in (node(mouse=False), node(role='ignore'), node(role='passthrough'), node(grabbed=False)):
-            self.assertFalse(relay.desktop_mouse_present([n]))
-        self.assertTrue(relay.desktop_mouse_present([node(mouse=False), node()]))
-    def mouse(self, speed=1):
-        with patch.object(relay.socket, 'socket'):
-            m = relay.DesktopMouse(speed)
-        m.messages = []
-        m.send = m.messages.append
-        return m
-
-    def test_motion_precedes_click_at_new_location(self):
-        m = self.mouse(.5)
-        m.motion(relay.REL_X, 12)
-        m.motion(relay.REL_Y, -8)
-        m.button(272, 1)
-        m.button(272, 0)
-        m.flush()
-        self.assertEqual(m.messages, ['mouse-move 6.000000 -4.000000', 'mouse-button 272 1', 'mouse-button 272 0'])
-
-    def test_hires_replaces_companion_legacy_in_both_orders(self):
-        for events in ([(8, 1), (11, 120)], [(11, 120), (8, 1)]):
-            m = self.mouse()
-            for code, value in events: m.motion(code, value)
-            m.flush()
-            self.assertEqual(m.messages, ['mouse-wheel 0.000000 1.000000'])
-
-    def test_fractional_scroll_has_no_pulse_timer(self):
-        m = self.mouse()
-        for value in (15, 15, -30):
-            m.motion(11, value); m.flush()
-        self.assertEqual(m.messages, ['mouse-wheel 0.000000 0.125000', 'mouse-wheel 0.000000 0.125000', 'mouse-wheel 0.000000 -0.250000'])
-
-    def test_thumb_and_main_wheels_are_independent(self):
-        m = self.mouse()
-        m.motion(6, -2); m.motion(8, 3); m.motion(11, 60); m.flush()
-        self.assertEqual(m.messages, ['mouse-wheel -2.000000 0.500000'])
-        m.motion(8, -1); m.flush()
-        self.assertEqual(m.messages[-1], 'mouse-wheel 0.000000 -1.000000')
-
-    def test_hotplug_releases_all_buttons_without_stale_motion(self):
-        m = self.mouse()
-        m.button(272, 1); m.button(275, 1)
-        m.motion(0, 20); m.motion(8, 1)
-        m.release_all(); m.flush()
-        self.assertEqual(m.messages, ['mouse-button 272 1', 'mouse-button 275 1', 'mouse-button 272 0', 'mouse-button 275 0'])
-        self.assertFalse(m.held)
-
-    def test_button_repeat_is_ignored(self):
-        m = self.mouse()
-        m.button(272, 2); m.button(280, 1)
-        self.assertEqual(m.messages, [])
-
-    def test_native_messages_never_target_virtual_controller(self):
-        with patch.object(relay.socket, 'socket') as sock:
-            m = relay.DesktopMouse()
-            m.motion(0, 3); m.motion(8, 1); m.button(272, 1); m.release_all()
-            self.assertTrue(sock.return_value.sendto.call_args_list)
-            for call in sock.return_value.sendto.call_args_list:
-                self.assertEqual(call.args[1], relay.SCREENS)
-
-    def test_bad_speed_rejected_before_socket_created(self):
-        with patch.object(relay.socket, 'socket') as sock:
-            for speed in (math.nan, math.inf, 0, 11):
-                with self.assertRaises(ValueError): relay.DesktopMouse(speed)
-            sock.assert_not_called()
-
-
-class ControlTests(unittest.TestCase):
-    def test_old_components_cannot_trigger_partial_activation(self):
-        with patch.object(ctl, 'status', return_value={'ready': False}), patch.object(ctl, 'configure') as configure, patch.object(ctl, 'save_mode') as save:
-            with self.assertRaisesRegex(RuntimeError, 'not loaded'): ctl.switch('desktop', True)
-            configure.assert_not_called(); save.assert_not_called()
-
-    def test_rejected_switch_does_not_persist(self):
-        with patch.object(ctl, 'require_ready'), patch.object(ctl, 'layout_outputs'), patch.object(ctl, 'configure'), patch.object(ctl, 'query', return_value={'ok': False, 'error': 'held buttons'}), patch.object(ctl, 'save_mode') as save:
-            with self.assertRaisesRegex(RuntimeError, 'held buttons'): ctl.switch('desktop', True)
-            save.assert_not_called()
-
-    def test_spatial_recovery_does_not_need_kde(self):
-        with patch.object(ctl, 'layout_outputs') as layout, patch.object(ctl, 'query', return_value={'ok': True}) as query:
-            self.assertEqual(ctl.switch('spatial'), {'mode': 'spatial', 'saved': False})
-            layout.assert_not_called(); query.assert_called_once_with(ctl.RELAY, 'mouse-mode spatial')
-
-    def test_failed_persistence_reports_actual_live_mode(self):
-        with patch.object(ctl, 'query', return_value={'ok': True}), patch.object(ctl, 'save_mode', side_effect=OSError('read only')):
-            with self.assertRaisesRegex(RuntimeError, 'Live mode is spatial'): ctl.switch('spatial', True)
-
-    def test_scaled_outputs_use_actual_logical_geometry(self):
-        outs = [{'enabled': True, 'pos': {'x': -1280, 'y': 20}, 'size': {'width': 1920, 'height': 1080}, 'scale': 1.5}, {'enabled': False}]
-        with patch.object(ctl, 'request', return_value='ok') as request:
-            ctl.configure(outs)
-            self.assertEqual([c.args for c in request.call_args_list], [(ctl.SCREENS, 'mouse-layout 1 -1280 20 1280 720 1.5'), (ctl.SCREENS, 'mouse-output-off 2')])
-
-    def test_invalid_later_output_does_not_partially_change_geometry(self):
-        outs = [{'enabled': True, 'pos': {'x': 0, 'y': 0}, 'size': {'width': 1920, 'height': 1080}, 'scale': 1}, {'enabled': True, 'scale': 0}]
-        with patch.object(ctl, 'request') as request:
-            with self.assertRaises(ValueError): ctl.configure(outs)
-            request.assert_not_called()
-
-    def test_save_preserves_other_preferences_and_permissions(self):
-        with tempfile.TemporaryDirectory() as root:
-            path = Path(root) / 'frametop.conf'
-            path.write_text('# user settings\nPOINTER=1\nMOUSE_MODE=spatial\nMETA_DASHBOARD=0\n')
-            path.chmod(0o600)
-            ctl.save_mode('desktop', path)
-            self.assertEqual(path.read_text(), '# user settings\nPOINTER=1\nMOUSE_MODE=desktop\nMETA_DASHBOARD=0\n')
-            self.assertEqual(path.stat().st_mode & 0o777, 0o600)
-
-    def test_failed_atomic_save_preserves_old_file(self):
-        with tempfile.TemporaryDirectory() as root:
-            path = Path(root) / 'frametop.conf'
-            path.write_text('MOUSE_MODE=spatial\n')
-            with patch.object(ctl.os, 'replace', side_effect=OSError('failed')):
-                with self.assertRaises(OSError): ctl.save_mode('desktop', path)
-            self.assertEqual(path.read_text(), 'MOUSE_MODE=spatial\n')
-            self.assertEqual(list(Path(root).iterdir()), [path])
-
-
-if __name__ == '__main__': unittest.main()

install.sh

@@ -1,13 +1,13 @@
 #!/usr/bin/env bash
 # Install everything on the Steam Frame: the build container, Frametop (multi-screen
-# desktop, input relay, universal 3D mouse, settings app), and optionally gaze mode and the
+# desktop, input relay, universal 3D mouse, settings app), and optionally gaze mode, our own
-# Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
+# eye tracker for it, and the Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
 # for example after `git pull`. (Hand tracking, hands/, is deferred: it isn't offered here.)
 # (It also works from a PC over SSH; see "Developing from a PC" in the README.)
 #
 # Usage: ./install.sh [--yes] [--no-bluetooth]
-#   --yes           don't ask; installs gaze mode, skips the Bluetooth fixes and the SteamVR
+#   --yes           don't ask; installs gaze mode, and our eye tracker if sudo can run without
-#                   restart
+#                   a password prompt; skips the Bluetooth fixes and the SteamVR restart
 #   --no-bluetooth  don't offer the Bluetooth fixes
 set -euo pipefail
 
@@ -19,7 +19,7 @@ for arg in "$@"; do
   case $arg in
     --yes) assume_yes=1 ;;
     --no-bluetooth) bluetooth=0 ;;
-    -h|--help) sed -n '2,11p' "$0"; exit 0 ;;
+    -h|--help) sed -n '2,12p' "$0"; exit 0 ;;
     *) echo "unknown option: $arg" >&2; exit 2 ;;
   esac
 done
@@ -38,6 +38,13 @@ ask() {  # ask "question" default(y|n)
   answer=${answer:-$2}
   [[ $answer =~ ^[Yy] ]]
 }
+# Can sudo run here without a password prompt (SUDO_ASKPASS, the repo's .env, a recent sudo)?
+# See frame_sudo in scripts/_env.sh.
+sudo_quiet() {
+  { [ "$FRAME_LOCAL" = 1 ] && [ -n "${SUDO_ASKPASS:-}" ]; } && return 0
+  grep -q '^steamos_root_pwd=.' "$REPO_ROOT/.env" 2>/dev/null && return 0
+  on_frame 'sudo -n true' 2>/dev/null
+}
 
 if [ "$FRAME_LOCAL" = 1 ]; then
   echo "Installing on this Steam Frame from $FRAME_REPO"
@@ -50,7 +57,7 @@ if [ "$FRAME_LOCAL" = 0 ] || [ -n "${SSH_CONNECTION:-}" ]; then
   echo "that need SteamVR start with it if it isn't running now."
 fi
 
-step "1/9 distrobox (container tool, installed in your home folder)"
+step "1/10 distrobox (container tool, installed in your home folder)"
 if on_frame 'test -x ~/.local/bin/distrobox'; then
   echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')"
 else
@@ -60,25 +67,25 @@ else
 cd ~/dev/src/distrobox && ./install --prefix ~/.local'
 fi
 
-step "2/9 build container (Fedora 44 'dev', about 1-2 GB the first time)"
+step "2/10 build container (Fedora 44 'dev', about 1-2 GB the first time)"
 "$root/setup/dev-container.sh"
 
-step "3/9 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
+step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
 "$root/desktops.sh" relay install
 
-step "4/9 3D mouse: SteamVR driver"
+step "4/10 3D mouse: SteamVR driver"
 "$root/pointer/driver/build.sh"
 "$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open
 
-step "5/9 3D mouse: pointer helper service"
+step "5/10 3D mouse: pointer helper service"
 "$root/pointer/helper/build.sh"
 "$root/pointer/helper/run.sh" install
 
-step "6/9 power service (turns the displays off while the headset isn't used, even on a stand)"
+step "6/10 power service (turns the displays off while the headset isn't used, even on a stand)"
 "$root/power/build.sh"
 "$root/power/run.sh" install
 
-step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
+step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
 "$root/screens/build.sh"
 "$root/desktops.sh" install >/dev/null
 "$root/input-settings/install.sh"
@@ -87,14 +94,44 @@ step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Framet
 on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
 echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
 
-step "8/9 gaze mode (optional, experimental: the pointer goes where you look)"
+step "8/10 gaze mode (optional, experimental: the pointer goes where you look)"
+gaze=0
 if ask "Install gaze mode? You turn it on and calibrate it in Frametop Input Settings, on the Gaze page." y; then
   "$root/gaze/run.sh" install
+  gaze=1
 else
   echo "skipped. Install later with: gaze/run.sh install"
 fi
 
-step "9/9 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
+# Ours is gaze mode's default once it's installed (GAZE_TRACKER=auto). Without it, gaze mode
+# uses SteamVR's eye tracker.
+step "9/10 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
+if [ "$gaze" = 0 ]; then
+  echo "skipped: gaze mode isn't installed. Install it later with: gaze/tracker/install.sh"
+elif [ "$assume_yes" = 1 ] && ! sudo_quiet; then
+  echo "skipped: it needs your password (sudo), and --yes doesn't ask. Install it later with: gaze/tracker/install.sh"
+elif ask "Install our own eye tracker? Gaze mode then uses it instead of SteamVR's. Its frame grabber is a small system service, so it needs your password (sudo), and it downloads about 165 MB (numpy, OpenCV)." y; then
+  if "$root/gaze/tracker/install.sh" install; then
+    # Configs made before GAZE_TRACKER=auto say steam, which keeps SteamVR's.
+    tracker=$(on_frame "sed -n 's/^GAZE_TRACKER=\([a-z]*\).*/\1/p' ~/.config/frametop.conf | tail -1")
+    if [ "$tracker" = steam ] &&
+       ask "Your settings pick SteamVR's eye tracker (GAZE_TRACKER=steam, the old default). Use ours instead?" y; then
+      on_frame "sed -i 's~^GAZE_TRACKER=steam\b.*~GAZE_TRACKER=auto          # gaze service: auto = our own eye tracker when installed, else SteamVR | own | steam~' ~/.config/frametop.conf"
+      tracker=auto
+    fi
+    if [ "$tracker" = steam ]; then
+      echo "installed; your settings keep SteamVR's eye tracker (pick Own tracker on the Gaze page to use ours)"
+    else
+      echo "gaze mode uses our eye tracker: calibrate it once, with Calibrate on the Gaze page"
+    fi
+  else
+    echo "our eye tracker didn't install, so gaze mode uses SteamVR's. Try again with: gaze/tracker/install.sh"
+  fi
+else
+  echo "skipped: gaze mode uses SteamVR's eye tracker. Install ours later with: gaze/tracker/install.sh"
+fi
+
+step "10/10 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
 if [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; then
   "$root/setup/bluetooth/install.sh" install
 else

layout/ft_layout.py

@@ -941,29 +941,6 @@ def send_scales(outs):
                 log(f"screen {i + 1}: scale: {reply}")
     except RuntimeError as e:
         log(f"scale: {e}")
-    finally:
-        if 'sock' in locals(): sock.sock.close()
-    # Native mouse coordinates follow the actual logical KWin output layout.
-    # Capability probing keeps this compatible with older ft-screens builds.
-    try:
-        with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as probe:
-            probe.bind(''); probe.settimeout(1)
-            probe.sendto(b'mouse?', SCREENS)
-            if not json.loads(probe.recv(8192)).get('supported'): return
-        sock = screens_socket()
-        try:
-            for i, o in enumerate(outs):
-                if not o.get('enabled', True):
-                    sock.ask(f'mouse-output-off {i+1}')
-                    continue
-                scale = float(o.get('scale', 1))
-                size, pos = o.get('size', {}), o.get('pos', {})
-                w = math.ceil(size['width']/scale - 1e-6)
-                h = math.ceil(size['height']/scale - 1e-6)
-                sock.ask(f"mouse-layout {i+1} {pos.get('x',0)} {pos.get('y',0)} {w} {h} {scale:g}")
-        finally: sock.sock.close()
-    except (OSError, ValueError, RuntimeError, KeyError):
-        pass  # old compositor, closing desktop, or a drag in progress; next layout refresh retries
 
 
 def logical_size(o):
@@ -1021,7 +998,6 @@ def apply_scales():
         if moves:
             subprocess.run(["kscreen-doctor", *moves], capture_output=True, env=env, timeout=20)
             args += moves
-            send_scales(outputs(env))
     return args
 
 

pointer/helper/ft-pointer.cpp

@@ -857,10 +857,9 @@ int main() {
     overlay->FindOverlay("system.pointer", &systemPointer);
     Vec3 pivot, tiltOrigin, lastPoint, lastOrigin, lastAim{0, 0, -1};
     Basis tiltBasis{};
-    bool desktopInput = false;  // native desktop input never claims a SteamVR hand
     bool headsetOff = false;  // nobody is wearing the headset (see the main loop)
     auto wake = [&](Clock::time_point t) {
-        if (t < noWakeUntil || headsetOff || desktopInput) return;
+        if (t < noWakeUntil || headsetOff) return;
         wokeAt = t;
         active = true;
         recenter = true;
@@ -1274,7 +1273,7 @@ int main() {
             }
             // Hand gestures in the last two minutes keep it the same way.
             const bool handsRecent = handsOn && handUsed != Clock::time_point{} && t - handUsed < std::chrono::minutes(2);
-            const bool awake = (gazeOn || handsRecent) && !inGame && !headsetOff && !desktopInput;
+            const bool awake = (gazeOn || handsRecent) && !inGame && !headsetOff;
             if (awake != gazeAwake || t - gazeAwakeAt > std::chrono::seconds(5)) {
                 if (awake != gazeAwake)
                     std::printf("%s keeps the pointer: %s\n", gazeOn ? "gaze" : "hand use",
@@ -1366,34 +1365,6 @@ int main() {
                 reply(controllerButtons.Status());
                 continue;
             }
-            if (std::strcmp(buf, "pointerstatus") == 0) {
-                char msg[256];
-                const int role = ours == vr::k_unTrackedDeviceIndexInvalid ? 0 :
-                    int(vr::VRSystem()->GetControllerRoleForTrackedDeviceIndex(ours));
-                std::snprintf(msg, sizeof msg,
-                    "{\"active\":%s,\"headsetOff\":%s,\"gazeMode\":%s,\"handRole\":%d,\"desktopInput\":%s}",
-                    active ? "true" : "false", headsetOff ? "true" : "false",
-                    gazeOn ? "true" : "false", role, desktopInput ? "true" : "false");
-                reply(msg);
-                continue;
-            }
-            if (std::strncmp(buf, "desktopmouse ", 13) == 0) {
-                if (std::strcmp(buf+13, "on") && std::strcmp(buf+13, "off")) {
-                    reply("error desktopmouse on|off");
-                    continue;
-                }
-                desktopInput = std::strcmp(buf+13, "on") == 0;
-                if (desktopInput) {
-                    if (leftHeld) releaseLeft();
-                    active = claimPending = claimHeld = false;
-                    overlay->HideOverlay(cursor); overlay->HideOverlay(marker);
-                    SendTo(out, "ft_pointer", "scroll 0 0");
-                    SendTo(out, "ft_pointer", "btn a 0");
-                    SendTo(out, "ft_pointer", "hide");
-                }
-                reply("ok");
-                continue;
-            }
             if (std::strncmp(buf, "overlays", 8) == 0) {
                 overlays.Request(sender, senderLen);  // answered from the list's thread
                 continue;
@@ -1781,7 +1752,6 @@ int main() {
             hp = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
             return true;
         };
-        if (desktopInput) keyPresses.clear();
         for (const KeyPress &k : keyPresses) {
             (k.right ? keyRightDown : keyLeftDown) = k.down;
             if (!k.down) {
@@ -1871,7 +1841,7 @@ int main() {
                 handBaseline = true, handOpens = handFile.opens;
             }
             // Not over a VR game (unless the dashboard is up), and not with the headset off.
-            const bool allowed = !desktopInput && !headsetOff && (!inGame || overlay->IsDashboardVisible());
+            const bool allowed = !headsetOff && (!inGame || overlay->IsDashboardVisible());
             for (int k = 1; k >= 0; --k)  // grips first: one takes over a pinch
                 for (int s = 0; s < 2; ++s) {
                     const fh_pinch_t &g = slots[k][s];

screens/compositor.c

@@ -25,11 +25,8 @@
 //   COMMAND     run with WAYLAND_DISPLAY set to our socket (e.g. the Frametop session)
 // Runs in the dev container (wlroots 0.20); KWin connects from the host.
 #define _GNU_SOURCE
-#include "controller-fallback.h"
-#include "desktop-handoff.h"
 #include <drm_fourcc.h>
 #include <linux/input-event-codes.h>
-#include <math.h>
 #include <signal.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -61,11 +58,8 @@
 #include <wlr/util/log.h>
 
 #include "vr.h"
-#include "desktop-mouse.h"
-#include "desktop-cursor.h"
-#include "cursor-cache.h"
 
-#define MAX_SCREENS FT_MOUSE_OUTPUTS  // screens and spare outputs
+#define MAX_SCREENS 24  // screens and spare outputs
 
 struct config {
     int width, height;
@@ -110,23 +104,6 @@ struct server {
     struct wl_list buffers;  // tracked_buffer
     struct wl_event_source *tick;
     struct screen *pointer_focus;
-    bool desktop_mouse, cursor_ready, cursor_frame_pending;
-    struct ft_controller_fallback controller_fallback;
-    bool controller_desktop;
-    struct ft_handoff handoff;
-    struct ft_event controller_position[MAX_SCREENS];
-    bool controller_position_known[MAX_SCREENS];
-    unsigned long mouse_events_ignored;
-    uint32_t controller_buttons;
-    struct ft_cursor_cache cursor_cache;
-    const char *cursor_error;
-    const char *cursor_buffer;
-    struct wlr_buffer *cursor_held;
-    struct ft_mouse mouse;
-    struct wlr_surface *cursor_surface;
-    int cursor_hx, cursor_hy;
-    struct wl_listener request_cursor, cursor_commit, cursor_destroy;
-    unsigned long mouse_motions, mouse_clicks, mouse_wheels, vr_mouse_ignored;
     pid_t child;
     // Where typing goes: the screens after a click on one, Steam after a click on another
     // panel. The input relay grabs the keyboards while it's the screens (see keys_update).
@@ -224,8 +201,6 @@ static void screen_destroy(struct wl_listener *l, void *data) {
     if (sc->held) wlr_buffer_unlock(sc->held);
     ft_vr_screen_destroy(sc->index);
     if (sc->server->pointer_focus == sc) sc->server->pointer_focus = NULL;
-    sc->server->mouse.outputs[sc->index].enabled = false;
-    if (sc->server->mouse.grab == sc->index) sc->server->mouse.buttons = 0, sc->server->mouse.grab = -1;
     if (sc->decoration) wl_list_remove(&sc->decoration_destroy.link);
     sc->server->screens[sc->index] = NULL;
     wl_list_remove(&sc->commit.link);
@@ -300,7 +275,7 @@ static void new_decoration(struct wl_listener *l, void *data) {
 
 static void panel_key(struct server *s, uint32_t code, bool pressed);
 
-static void deliver_pointer_event(const struct ft_event *e, void *data) {
+static void handle_vr_event(const struct ft_event *e, void *data) {
     struct server *s = data;
     if (e->type == FT_QUIT) {
         wl_display_terminate(s->display);
@@ -360,11 +335,11 @@ static void deliver_pointer_event(const struct ft_event *e, void *data) {
             }
             if (e->dy != 0)
                 wlr_seat_pointer_notify_axis(s->seat, t, WL_POINTER_AXIS_VERTICAL_SCROLL, e->dy * 15,
-                                             (int32_t)llround(e->dy * 120), WL_POINTER_AXIS_SOURCE_WHEEL,
+                                             (int32_t)(e->dy * 120), WL_POINTER_AXIS_SOURCE_WHEEL,
                                              WL_POINTER_AXIS_RELATIVE_DIRECTION_IDENTICAL);
             if (e->dx != 0)
                 wlr_seat_pointer_notify_axis(s->seat, t, WL_POINTER_AXIS_HORIZONTAL_SCROLL, e->dx * 15,
-                                             (int32_t)llround(e->dx * 120), WL_POINTER_AXIS_SOURCE_WHEEL,
+                                             (int32_t)(e->dx * 120), WL_POINTER_AXIS_SOURCE_WHEEL,
                                              WL_POINTER_AXIS_RELATIVE_DIRECTION_IDENTICAL);
             break;
         case FT_LEAVE:
@@ -379,185 +354,11 @@ static void deliver_pointer_event(const struct ft_event *e, void *data) {
     wlr_seat_pointer_notify_frame(s->seat);
 }
 
-static void set_desktop_owner(struct server *s, bool pointer) {
-    if (pointer == s->controller_desktop) return;
-    // Claims are refused during a held drag. Legacy disconnect recovery may
-    // still revoke a stale controller, releasing its buttons safely.
-    for (unsigned bit = 0; bit < 8; ++bit)
-        if (s->controller_buttons & (1u << bit))
-            wlr_seat_pointer_notify_button(s->seat, now_ms(), BTN_LEFT+bit,
-                                           WL_POINTER_BUTTON_STATE_RELEASED);
-    s->controller_buttons = 0;
-    wlr_seat_pointer_notify_clear_focus(s->seat);
-    wlr_seat_pointer_notify_frame(s->seat);
-    s->pointer_focus = NULL;
-    s->controller_desktop = pointer;
-    wlr_log(WLR_INFO, "desktop owner %s (policy %s)", pointer ? "pointer" : "mouse",
-            ft_handoff_name(s->handoff.policy));
-    ft_vr_cursor_move(-1, 0, 0, 1, false);
-}
-
-static void update_controller_fallback(struct server *s) {
-    bool active = s->desktop_mouse && !s->mouse.buttons &&
-        ft_fallback_active(&s->controller_fallback, now_ms());
-    if (s->desktop_mouse && s->handoff.policy != FT_PREFER_MOUSE) {
-        active = s->controller_desktop;
-        if (!s->mouse.buttons && !s->controller_buttons &&
-            (s->handoff.policy == FT_PREFER_POINTER ||
-             ft_fallback_active(&s->controller_fallback, now_ms()))) active = true;
-    }
-    set_desktop_owner(s, active);
-}
-
-static bool claim_desktop_mouse(struct server *s) {
-    if (!ft_handoff_claim(&s->handoff, false, now_ms(), s->mouse.buttons, s->controller_buttons)) {
-        ++s->mouse_events_ignored;
-        return false;
-    }
-    ft_fallback_presence(&s->controller_fallback, true, s->controller_fallback.enabled, now_ms());
-    set_desktop_owner(s, false);
-    return true;
-}
-
-static void handle_vr_event(const struct ft_event *e, void *data) {
-    struct server *s = data;
-    // Events here are panel content, not SteamVR or separate screen grab bars.
-    bool desktop = e->screen >= 0 && e->screen < s->n_config;
-    if (desktop && e->type == FT_MOTION) {
-        s->controller_position[e->screen] = *e;
-        s->controller_position_known[e->screen] = true;
-    }
-    update_controller_fallback(s);
-    if (s->desktop_mouse && desktop) {
-        bool pointer_event = e->type == FT_MOTION || e->type == FT_BUTTON ||
-                             e->type == FT_SCROLL || e->type == FT_LEAVE;
-        uint32_t bit = e->type == FT_BUTTON && e->button >= BTN_LEFT && e->button < BTN_LEFT+8
-            ? 1u << (e->button-BTN_LEFT) : 0;
-        bool action = (e->type == FT_BUTTON && e->pressed && bit) ||
-                      (e->type == FT_SCROLL && (e->dx != 0 || e->dy != 0));
-        if (action && s->handoff.policy != FT_PREFER_MOUSE &&
-            ft_handoff_claim(&s->handoff, true, now_ms(), s->mouse.buttons, s->controller_buttons)) {
-            set_desktop_owner(s, true);
-            if (e->type == FT_SCROLL && s->controller_position_known[e->screen])
-                deliver_pointer_event(&s->controller_position[e->screen], s);
-        }
-        if (pointer_event && (!s->controller_desktop || s->mouse.buttons ||
-            (e->type == FT_BUTTON && !e->pressed && !(s->controller_buttons & bit)) ||
-            (e->type == FT_LEAVE && s->controller_buttons))) {
-            ++s->vr_mouse_ignored; return;
-        }
-        if (bit) {
-            if (e->pressed) s->controller_buttons |= bit;
-            else s->controller_buttons &= ~bit;
-        }
-    }
-    deliver_pointer_event(e, data);
-}
-
-static void desktop_motion(struct server *s) {
-    int index = ft_mouse_target(&s->mouse);
-    if (index < 0 || !s->screens[index]) return;
-    const struct ft_mouse_output *o = &s->mouse.outputs[index];
-    struct ft_event event = {.type = FT_MOTION, .screen = index,
-        .x = (s->mouse.x-o->x)*o->scale, .y = (s->mouse.y-o->y)*o->scale};
-    deliver_pointer_event(&event, s);
-    const struct ft_mouse_output *display = &s->mouse.outputs[s->mouse.screen];
-    ft_vr_cursor_move(s->mouse.screen, (s->mouse.x-display->x)*display->scale,
-                     (s->mouse.y-display->y)*display->scale, display->scale, s->desktop_mouse);
-}
-
-static void cursor_hide(struct server *s) {
-    ft_vr_cursor_image(NULL, 0, 0, 0, 0, 1);
-    if (s->cursor_held) wlr_buffer_unlock(s->cursor_held);
-    s->cursor_held = NULL;
-}
-static void cursor_update(struct server *s) {
-    struct wlr_surface *surface = s->cursor_surface;
-    struct wlr_buffer *buffer = surface ? ft_cursor_cached(&s->cursor_cache, surface) : NULL;
-    s->cursor_ready = false;
-    s->cursor_error = "no cached KDE cursor buffer";
-    if (surface) s->cursor_frame_pending = true;
-    s->cursor_buffer = "none";
-    if (!buffer) { cursor_hide(s); return; }
-    if (buffer->width < 1 || buffer->height < 1 || buffer->width > 512 || buffer->height > 512) {
-        cursor_hide(s); return;
-    }
-    int scale = surface->current.scale > 0 ? surface->current.scale : 1;
-    if (scale > 8 || s->cursor_hx < -512 || s->cursor_hy < -512 || s->cursor_hx > 512 || s->cursor_hy > 512) {
-        s->cursor_error = "invalid cursor scale or hotspot";
-        cursor_hide(s); return;
-    }
-    struct wlr_dmabuf_attributes dma;
-    if (wlr_buffer_get_dmabuf(buffer, &dma)) {
-        s->cursor_buffer = "dmabuf";
-        struct ft_dmabuf b = {.width = dma.width, .height = dma.height, .format = dma.format,
-                             .modifier = dma.modifier, .n_planes = dma.n_planes};
-        for (int i = 0; i < dma.n_planes && i < 4; ++i) {
-            b.offset[i] = dma.offset[i]; b.stride[i] = dma.stride[i]; b.fd[i] = dma.fd[i];
-        }
-        track_buffer(s, buffer);
-        s->cursor_ready = ft_vr_cursor_dmabuf(buffer, &b, s->cursor_hx*scale, s->cursor_hy*scale, scale);
-        s->cursor_error = s->cursor_ready ? "" : "SteamVR rejected GPU cursor";
-        if (s->cursor_ready) {
-            wlr_buffer_lock(buffer);
-            if (s->cursor_held) wlr_buffer_unlock(s->cursor_held);
-            s->cursor_held = buffer;
-        } else cursor_hide(s);
-        return;
-    }
-    s->cursor_buffer = "shm";
-    void *data; uint32_t format; size_t stride;
-    if (!wlr_buffer_begin_data_ptr_access(buffer, WLR_BUFFER_DATA_PTR_ACCESS_READ, &data, &format, &stride)) {
-        s->cursor_error = "KDE cursor buffer cannot be read";
-        cursor_hide(s); return;
-    }
-    uint8_t *rgba = malloc((size_t)buffer->width*buffer->height*4);
-    s->cursor_error = "unsupported cursor format or allocation failed";
-    if (rgba && ft_cursor_rgba(rgba, data, buffer->width, buffer->height, stride, format)) {
-        s->cursor_error = "SteamVR rejected cursor texture";
-        s->cursor_ready = ft_vr_cursor_image(rgba, buffer->width, buffer->height,
-                                            s->cursor_hx*scale, s->cursor_hy*scale, scale);
-        if (s->cursor_ready) s->cursor_error = "";
-    }
-    free(rgba);
-    wlr_buffer_end_data_ptr_access(buffer);
-    if (!s->cursor_ready) cursor_hide(s);
-    else if (s->cursor_held) { wlr_buffer_unlock(s->cursor_held); s->cursor_held = NULL; }
-}
-
-static void cursor_committed(struct wl_listener *l, void *data) {
-    struct server *s = wl_container_of(l, s, cursor_commit);
-    cursor_update(s);
-}
-static void cursor_destroyed(struct wl_listener *l, void *data) {
-    struct server *s = wl_container_of(l, s, cursor_destroy);
-    wl_list_remove(&s->cursor_commit.link); wl_list_remove(&s->cursor_destroy.link);
-    s->cursor_surface = NULL; s->cursor_ready = false;
-    cursor_hide(s);
-}
-static void cursor_requested(struct wl_listener *l, void *data) {
-    struct server *s = wl_container_of(l, s, request_cursor);
-    struct wlr_seat_pointer_request_set_cursor_event *e = data;
-    if (e->seat_client != s->seat->pointer_state.focused_client) return;
-    if (s->cursor_surface) {
-        wl_list_remove(&s->cursor_commit.link); wl_list_remove(&s->cursor_destroy.link);
-    }
-    s->cursor_surface = e->surface; s->cursor_hx = e->hotspot_x; s->cursor_hy = e->hotspot_y;
-    if (e->surface) {
-        s->cursor_commit.notify = cursor_committed;
-        s->cursor_destroy.notify = cursor_destroyed;
-        wl_signal_add(&e->surface->events.commit, &s->cursor_commit);
-        wl_signal_add(&e->surface->events.destroy, &s->cursor_destroy);
-    }
-    cursor_update(s);
-}
-
 // Tell the input relay where typing goes, on a change and every second: while it's the
 // desktop, the relay grabs pass-through keyboards so SteamVR (and the Steam app with
 // gamescope's focus) doesn't get the keys too. Without word from us for a few seconds,
 // the relay gives the keyboards back, so a closed desktop doesn't keep them.
 static void keys_update(struct server *s) {
-    if (!s->vr) return;  // the isolated test desktop must never route live keyboards
     if (!s->vr) return;  // a test instance leaves the running desktop's keyboards alone
     const bool desktop = s->keys_clicked && ft_vr_screens_shown();
     const uint32_t t = now_ms();
@@ -576,7 +377,6 @@ static void keys_update(struct server *s) {
 // Every ~11 ms (90 Hz): SteamVR events, and frame callbacks for screens that committed.
 static int tick(void *data) {
     struct server *s = data;
-    update_controller_fallback(s);
     ft_vr_poll(handle_vr_event, s);
     if (++s->ticks % 9 == 0) keys_update(s);
     if (s->kb_close_at && s->ticks >= s->kb_close_at) {
@@ -588,10 +388,6 @@ static int tick(void *data) {
     }
     struct timespec now;
     clock_gettime(CLOCK_MONOTONIC, &now);
-    if (s->cursor_surface && s->cursor_frame_pending) {
-        s->cursor_frame_pending = false;
-        wlr_surface_send_frame_done(s->cursor_surface, &now);
-    }
     for (int i = 0; i < MAX_SCREENS; ++i) {
         struct screen *sc = s->screens[i];
         if (sc && sc->frame_pending) {
@@ -717,121 +513,8 @@ static int control_readable(int fd, uint32_t mask, void *data) {
         buf[n] = 0;
         unsigned code;
         int value, index, w, h;
-        double scale, wheel_x, wheel_y;
+        double scale;
-        double mx, my, mw, mh;
+        if (sscanf(buf, "size %d %d %d", &index, &w, &h) == 3) {
-        char mode[16], tail;
-        if (strcmp(buf, "desktop-policy?") == 0) {
-            snprintf(reply, sizeof reply,
-                "{\"supported\":true,\"policy\":\"%s\",\"owner\":\"%s\",\"mouseButtons\":%u,\"pointerButtons\":%u,\"mouseEventsIgnored\":%lu}",
-                ft_handoff_name(s->handoff.policy), s->controller_desktop ? "pointer" : "mouse",
-                s->mouse.buttons, s->controller_buttons, s->mouse_events_ignored);
-        } else if (sscanf(buf, "desktop-policy %15s %c", mode, &tail) == 1) {
-            if (s->mouse.buttons || s->controller_buttons ||
-                (strcmp(mode, "mouse") && strcmp(mode, "pointer") && strcmp(mode, "last-active")))
-                snprintf(reply, sizeof reply, "error policy or held button");
-            else {
-                s->handoff = (struct ft_handoff){.policy = !strcmp(mode, "last-active") ? FT_LAST_ACTIVE :
-                    !strcmp(mode, "pointer") ? FT_PREFER_POINTER : FT_PREFER_MOUSE};
-                set_desktop_owner(s, s->handoff.policy == FT_PREFER_POINTER);
-                update_controller_fallback(s);
-                snprintf(reply, sizeof reply, "ok");
-            }
-        } else if (strcmp(buf, "mouse-presence?") == 0) {
-            snprintf(reply, sizeof reply,
-                     "{\"supported\":true,\"known\":%s,\"present\":%s,\"enabled\":%s,\"controllerDesktop\":%s,\"fresh\":%s}",
-                     s->controller_fallback.known ? "true" : "false",
-                     s->controller_fallback.present ? "true" : "false",
-                     s->controller_fallback.enabled ? "true" : "false",
-                     s->controller_desktop ? "true" : "false",
-                     s->controller_fallback.known && (uint32_t)(now_ms()-s->controller_fallback.received)<3000 ? "true" : "false");
-        } else if (sscanf(buf, "mouse-presence %d %d %c", &index, &value, &tail) == 2) {
-            if ((index != 0 && index != 1) || (value != 0 && value != 1))
-                snprintf(reply, sizeof reply, "error invalid presence");
-            else {
-                ft_fallback_presence(&s->controller_fallback, index, value, now_ms());
-                update_controller_fallback(s);
-                snprintf(reply, sizeof reply, "ok");
-            }
-        } else if (strcmp(buf, "mouse?") == 0) {
-            snprintf(reply, sizeof reply,
-                "{\"supported\":true,\"mode\":\"%s\",\"screen\":%d,\"x\":%.3f,\"y\":%.3f,"
-                "\"buttons\":%u,\"cursorReady\":%s,\"cursorSource\":\"kde\",\"cursorCache\":true,\"cursorBuffer\":\"%s\",\"cursorError\":\"%s\","
-                "\"motionEvents\":%lu,\"buttonEvents\":%lu,\"wheelEvents\":%lu,\"controllerEventsIgnored\":%lu}",
-                s->desktop_mouse ? "desktop" : "spatial", s->mouse.screen+1, s->mouse.x, s->mouse.y,
-                s->mouse.buttons, s->cursor_ready ? "true" : "false", s->cursor_buffer ? s->cursor_buffer : "none", s->cursor_error ? s->cursor_error : "no cursor request", s->mouse_motions,
-                s->mouse_clicks, s->mouse_wheels, s->vr_mouse_ignored);
-        } else if (sscanf(buf, "mouse-output-off %d %c", &index, &tail) == 1) {
-            if (index < 1 || index > MAX_SCREENS || s->mouse.buttons)
-                snprintf(reply, sizeof reply, "error invalid mouse output or held button");
-            else { s->mouse.outputs[index-1].enabled = false; snprintf(reply, sizeof reply, "ok"); }
-        } else if (sscanf(buf, "mouse-layout %d %lf %lf %lf %lf %lf %c", &index, &mx, &my, &mw, &mh, &scale, &tail) == 6) {
-            if (index < 1 || index > MAX_SCREENS || !s->screens[index-1] ||
-                !ft_mouse_output_set(&s->mouse, index-1, mx, my, mw, mh, scale))
-                snprintf(reply, sizeof reply, "error invalid mouse output or held button");
-            else { s->scale[index-1] = scale; snprintf(reply, sizeof reply, "ok"); }
-        } else if (sscanf(buf, "mouse-mode %15s %c", mode, &tail) == 1) {
-            if (s->mouse.buttons || s->controller_buttons || (strcmp(mode, "desktop") && strcmp(mode, "spatial")))
-                snprintf(reply, sizeof reply, "error mouse mode or held button");
-            else if (!strcmp(mode, "desktop") && !ft_mouse_move(&s->mouse, 0, 0))
-                snprintf(reply, sizeof reply, "error configure mouse layout first");
-            else {
-                s->desktop_mouse = !strcmp(mode, "desktop");
-                if (s->desktop_mouse) desktop_motion(s);
-                else {
-                    wlr_seat_pointer_notify_clear_focus(s->seat); s->pointer_focus = NULL;
-                    wlr_seat_pointer_notify_frame(s->seat);
-                    ft_vr_cursor_move(-1, 0, 0, 1, false);
-                }
-                snprintf(reply, sizeof reply, "ok");
-            }
-        } else if (sscanf(buf, "mouse-move %lf %lf %c", &mx, &my, &tail) == 2) {
-            if (!s->desktop_mouse || (s->vr && !ft_vr_screens_shown()) || !isfinite(mx) || !isfinite(my) || fabs(mx)>16384 || fabs(my)>16384)
-                snprintf(reply, sizeof reply, "error desktop mouse unavailable or invalid motion");
-            else { if ((mx != 0 || my != 0) && claim_desktop_mouse(s) && ft_mouse_move(&s->mouse, mx, my)) { desktop_motion(s); ++s->mouse_motions; } snprintf(reply, sizeof reply, "ok"); }
-        } else if (sscanf(buf, "mouse-position %d %lf %lf %c", &index, &mx, &my, &tail) == 3) {
-            if (!s->desktop_mouse || s->mouse.buttons || s->controller_buttons || index < 1 || index > MAX_SCREENS ||
-                !s->mouse.outputs[index-1].enabled || !isfinite(mx) || !isfinite(my) ||
-                mx < 0 || my < 0 || mx >= s->mouse.outputs[index-1].width || my >= s->mouse.outputs[index-1].height)
-                snprintf(reply, sizeof reply, "error mouse position or held button");
-            else {
-                ft_mouse_position(&s->mouse, s->mouse.outputs[index-1].x+mx, s->mouse.outputs[index-1].y+my);
-                if (claim_desktop_mouse(s)) desktop_motion(s);
-                snprintf(reply, sizeof reply, "ok");
-            }
-        } else if (sscanf(buf, "mouse-button %u %d %c", &code, &value, &tail) == 2) {
-            if (!s->desktop_mouse || (value != 0 && value != 1) ||
-                (s->vr && !ft_vr_screens_shown() && value) || code < BTN_LEFT || code > BTN_TASK)
-                snprintf(reply, sizeof reply, "error mouse button or hidden desktop");
-            else if ((!value && !(s->mouse.buttons & (1u << (code-BTN_LEFT)))) || !claim_desktop_mouse(s))
-                snprintf(reply, sizeof reply, "ok");
-            else {
-                desktop_motion(s);
-                int target = ft_mouse_target(&s->mouse);
-                const struct ft_mouse_output *o = target >= 0 ? &s->mouse.outputs[target] : NULL;
-                if (!o || !s->screens[target]) snprintf(reply, sizeof reply, "error no mouse surface");
-                else {
-                    struct ft_event event = {.type = FT_BUTTON, .screen = target, .button = code, .pressed = value,
-                        .x = (s->mouse.x-o->x)*o->scale, .y = (s->mouse.y-o->y)*o->scale};
-                    deliver_pointer_event(&event, s);
-                    ft_mouse_button(&s->mouse, code, value); ++s->mouse_clicks;
-                    if (!s->mouse.buttons) desktop_motion(s);
-                    snprintf(reply, sizeof reply, "ok");
-                }
-            }
-        } else if (sscanf(buf, "mouse-wheel %lf %lf %c", &wheel_x, &wheel_y, &tail) == 2) {
-            int target = ft_mouse_target(&s->mouse);
-            if (!s->desktop_mouse || (s->vr && !ft_vr_screens_shown()) || target < 0 || !s->screens[target] ||
-                !s->mouse.outputs[target].enabled ||
-                !isfinite(wheel_x) || !isfinite(wheel_y) || fabs(wheel_x)>120 || fabs(wheel_y)>120)
-                snprintf(reply, sizeof reply, "error mouse wheel or hidden desktop");
-            else if ((wheel_x == 0 && wheel_y == 0) || !claim_desktop_mouse(s))
-                snprintf(reply, sizeof reply, "ok");
-            else {
-                desktop_motion(s);
-                struct ft_event event = {.type = FT_SCROLL, .screen = target, .dx = wheel_x, .dy = -wheel_y};
-                deliver_pointer_event(&event, s); ++s->mouse_wheels; snprintf(reply, sizeof reply, "ok");
-            }
-        } else if (sscanf(buf, "size %d %d %d", &index, &w, &h) == 3) {
             // A new resolution for a screen, live: KWin resizes the screen to match. (KWin makes
             // it this size times its scale; ft-floatd sends spares' sizes divided by theirs.)
             const int min_w = index - 1 < s->n_config ? 320 : 64, min_h = index - 1 < s->n_config ? 200 : 64;
@@ -870,13 +553,11 @@ static int control_readable(int fd, uint32_t mask, void *data) {
             else if (strcmp(button, "middle") == 0) e.button = BTN_MIDDLE;
             if (got >= 2 && strcmp(what, "leave") == 0) e.type = FT_LEAVE;
             else if (got >= 4 && strcmp(what, "move") == 0) e.type = FT_MOTION;
-            else if (got >= 4 && strcmp(what, "scroll") == 0 && isfinite(x) && isfinite(y) && fabs(x)<=120 && fabs(y)<=120)
-                e.type = FT_SCROLL, e.dx = x, e.dy = y;
             else if (got >= 4 && (strcmp(what, "down") == 0 || strcmp(what, "up") == 0))
                 e.type = FT_BUTTON, e.pressed = what[0] == 'd';
             else index = 0;
             if (index < 1 || index > MAX_SCREENS || !s->screens[index - 1]) {
-                snprintf(reply, sizeof reply, "error input <screen> move|down|up|scroll|leave [x y [button]]");
+                snprintf(reply, sizeof reply, "error input <screen> move|down|up|leave [x y [button]]");
             } else {
                 handle_vr_event(&e, s);
                 snprintf(reply, sizeof reply, "ok");
@@ -960,7 +641,6 @@ static bool setup_dmabuf(struct server *s) {
 
 int main(int argc, char **argv) {
     struct server s = {0};
-    ft_mouse_init(&s.mouse);
     for (int i = 0; i < MAX_SCREENS; ++i) s.scale[i] = 1;
     s.kb_screen = -1;
     const char *socket_name = "ft-screens-0", *control_name = "ft_screens";
@@ -1005,8 +685,7 @@ int main(int argc, char **argv) {
     s.display = wl_display_create();
     s.loop = wl_display_get_event_loop(s.display);
     wl_list_init(&s.buffers);
-    struct wlr_compositor *compositor = wlr_compositor_create(s.display, 6, NULL);
+    wlr_compositor_create(s.display, 6, NULL);
-    ft_cursor_cache_init(&s.cursor_cache, compositor);
     wlr_subcompositor_create(s.display);
     const uint32_t shm_formats[] = {DRM_FORMAT_ARGB8888, DRM_FORMAT_XRGB8888};  // wlroots wants DRM codes
     wlr_shm_create(s.display, 1, shm_formats, 2);
@@ -1021,8 +700,6 @@ int main(int argc, char **argv) {
     wl_signal_add(&s.decoration->events.new_toplevel_decoration, &s.new_decoration);
 
     s.seat = wlr_seat_create(s.display, "seat0");
-    s.request_cursor.notify = cursor_requested;
-    wl_signal_add(&s.seat->events.request_set_cursor, &s.request_cursor);
     wlr_keyboard_init(&s.keyboard, &keyboard_impl, "ft-screens-keyboard");
     struct xkb_context *xkb = xkb_context_new(XKB_CONTEXT_NO_FLAGS);
     struct xkb_keymap *keymap = xkb_keymap_new_from_names(xkb, NULL, XKB_KEYMAP_COMPILE_NO_FLAGS);
@@ -1069,7 +746,6 @@ int main(int argc, char **argv) {
     // wlroots asserts that nothing still listens to its globals when they go.
     wl_list_remove(&s.new_toplevel.link);
     wl_list_remove(&s.new_decoration.link);
-    ft_cursor_cache_finish(&s.cursor_cache);
     ft_vr_shutdown();
     wl_display_destroy(s.display);
     return 0;

screens/controller-fallback.h

@@ -1,19 +0,0 @@
-#ifndef FT_CONTROLLER_FALLBACK_H
-#define FT_CONTROLLER_FALLBACK_H
-#include <stdbool.h>
-#include <stdint.h>
-struct ft_controller_fallback {
-    bool known, present, enabled;
-    uint32_t received, absent_since;
-};
-static inline void ft_fallback_presence(struct ft_controller_fallback *f,
-                                       bool present, bool enabled, uint32_t now) {
-    if (!f->known || (uint32_t)(now-f->received) >= 3000 || f->present || present)
-        f->absent_since = now;
-    f->known = true; f->present = present; f->enabled = enabled; f->received = now;
-}
-static inline bool ft_fallback_active(const struct ft_controller_fallback *f, uint32_t now) {
-    return f->known && f->enabled && !f->present &&
-        (uint32_t)(now-f->received) < 3000 && (uint32_t)(now-f->absent_since) >= 1000;
-}
-#endif

screens/cursor-cache.h

@@ -1,67 +0,0 @@
-#pragma once
-#include <stdlib.h>
-#include <wayland-server-core.h>
-#include <wlr/types/wlr_buffer.h>
-#include <wlr/types/wlr_compositor.h>
-
-// A rendererless compositor has no persistent surface texture. wlroots releases
-// current.buffer after each commit callback. KDE commits a cursor before issuing
-// set_cursor, so retain small buffers from the first commit, not from set_cursor.
-// This also keeps animation/callback-only commits from erasing the last image.
-struct ft_cursor_cache;
-struct ft_cursor_frame {
-    struct ft_cursor_cache *cache;
-    struct wlr_surface *surface;
-    struct wlr_buffer *buffer;
-    struct wl_listener commit, destroy;
-    struct wl_list link;
-};
-struct ft_cursor_cache {
-    struct wl_listener new_surface;
-    struct wl_list frames;
-};
-static inline struct wlr_buffer *ft_cursor_cached(struct ft_cursor_cache *cache, struct wlr_surface *surface) {
-    struct ft_cursor_frame *f;
-    wl_list_for_each(f, &cache->frames, link) if (f->surface == surface) return f->buffer;
-    return NULL;
-}
-static inline void ft_cursor_frame_committed(struct wl_listener *listener, void *data) {
-    (void)data;
-    struct ft_cursor_frame *f = wl_container_of(listener, f, commit);
-    if (!(f->surface->current.committed & WLR_SURFACE_STATE_BUFFER)) return;
-    struct wlr_buffer *buffer = f->surface->current.buffer;
-    // Desktop DMA-BUFs go straight to SteamVR; do not retain them here.
-    struct wlr_buffer *next = buffer && buffer->width > 0 && buffer->height > 0 &&
-        buffer->width <= 512 && buffer->height <= 512 ? wlr_buffer_lock(buffer) : NULL;
-    if (f->buffer) wlr_buffer_unlock(f->buffer);
-    f->buffer = next;
-}
-static inline void ft_cursor_frame_destroyed(struct wl_listener *listener, void *data) {
-    (void)data;
-    struct ft_cursor_frame *f = wl_container_of(listener, f, destroy);
-    wl_list_remove(&f->commit.link); wl_list_remove(&f->destroy.link); wl_list_remove(&f->link);
-    if (f->buffer) wlr_buffer_unlock(f->buffer);
-    free(f);
-}
-static inline void ft_cursor_surface_created(struct wl_listener *listener, void *data) {
-    struct ft_cursor_cache *cache = wl_container_of(listener, cache, new_surface);
-    struct wlr_surface *surface = data;
-    struct ft_cursor_frame *f = calloc(1, sizeof *f);
-    if (!f) { wl_client_post_no_memory(wl_resource_get_client(surface->resource)); return; }
-    f->cache = cache; f->surface = surface;
-    f->commit.notify = ft_cursor_frame_committed;
-    f->destroy.notify = ft_cursor_frame_destroyed;
-    wl_signal_add(&surface->events.commit, &f->commit);
-    wl_signal_add(&surface->events.destroy, &f->destroy);
-    wl_list_insert(&cache->frames, &f->link);
-}
-static inline void ft_cursor_cache_init(struct ft_cursor_cache *cache, struct wlr_compositor *compositor) {
-    wl_list_init(&cache->frames);
-    cache->new_surface.notify = ft_cursor_surface_created;
-    wl_signal_add(&compositor->events.new_surface, &cache->new_surface);
-}
-static inline void ft_cursor_cache_finish(struct ft_cursor_cache *cache) {
-    wl_list_remove(&cache->new_surface.link);
-    struct ft_cursor_frame *f, *tmp;
-    wl_list_for_each_safe(f, tmp, &cache->frames, link) ft_cursor_frame_destroyed(&f->destroy, NULL);
-}

screens/desktop-cursor.h

@@ -1,26 +0,0 @@
-#pragma once
-#include <drm_fourcc.h>
-#include <stdbool.h>
-#include <stddef.h>
-#include <stdint.h>
-#include <string.h>
-// Shared-memory Wayland cursor images use ARGB with premultiplied alpha.
-// SteamVR's raw overlay texture is RGBA. Respect padding and recover RGB at edges.
-static inline bool ft_cursor_rgba(uint8_t *out, const void *data, int w, int h,
-                                  size_t stride, uint32_t format) {
-    if (!data || w < 1 || h < 1 || w > 512 || h > 512 || stride < (size_t)w*4 ||
-        (format != DRM_FORMAT_ARGB8888 && format != DRM_FORMAT_ABGR8888 &&
-         format != DRM_FORMAT_XRGB8888 && format != DRM_FORMAT_XBGR8888)) return false;
-    const bool swap = format == DRM_FORMAT_ABGR8888 || format == DRM_FORMAT_XBGR8888;
-    const bool opaque = format == DRM_FORMAT_XRGB8888 || format == DRM_FORMAT_XBGR8888;
-    for (int y = 0; y < h; ++y) for (int x = 0; x < w; ++x) {
-        uint32_t v; memcpy(&v, (const uint8_t *)data + (size_t)y*stride + x*4, 4);
-        unsigned a = opaque ? 255 : v >> 24;
-        unsigned rgb[3] = {swap ? v & 255 : (v >> 16) & 255, (v >> 8) & 255,
-                           swap ? (v >> 16) & 255 : v & 255};
-        uint8_t *p = out + ((size_t)y*w+x)*4;
-        for (int c = 0; c < 3; ++c) { unsigned n = a ? (rgb[c]*255 + a/2)/a : 0; p[c] = n > 255 ? 255 : n; }
-        p[3] = a;
-    }
-    return true;
-}

screens/desktop-handoff.h

@@ -1,24 +0,0 @@
-#ifndef FT_DESKTOP_HANDOFF_H
-#define FT_DESKTOP_HANDOFF_H
-#include <stdbool.h>
-#include <stdint.h>
-enum ft_handoff_policy { FT_PREFER_MOUSE, FT_LAST_ACTIVE, FT_PREFER_POINTER };
-struct ft_handoff { enum ft_handoff_policy policy; bool mouse_seen; uint32_t mouse_at; };
-// Only deliberate desktop actions call this. Tracking motion never claims a seat.
-static inline bool ft_handoff_claim(struct ft_handoff *h, bool pointer,
-                                    uint32_t now, uint32_t mouse_buttons,
-                                    uint32_t pointer_buttons) {
-    if (pointer) {
-        if (mouse_buttons || h->policy == FT_PREFER_MOUSE) return false;
-        if (h->policy == FT_LAST_ACTIVE && !pointer_buttons && h->mouse_seen && now == h->mouse_at)
-            return false; // Same millisecond: native mouse wins.
-    } else {
-        if (pointer_buttons || h->policy == FT_PREFER_POINTER) return false;
-        h->mouse_seen = true; h->mouse_at = now;
-    }
-    return true;
-}
-static inline const char *ft_handoff_name(enum ft_handoff_policy p) {
-    return p == FT_LAST_ACTIVE ? "last-active" : p == FT_PREFER_POINTER ? "pointer" : "mouse";
-}
-#endif

screens/desktop-mouse.h

@@ -1,61 +0,0 @@
-#pragma once
-#include <math.h>
-#include <stdbool.h>
-#include <stdint.h>
-
-#define FT_MOUSE_OUTPUTS 24
-struct ft_mouse_output { bool enabled; double x, y, width, height, scale; };
-struct ft_mouse {
-    struct ft_mouse_output outputs[FT_MOUSE_OUTPUTS];
-    double x, y;
-    uint32_t buttons;
-    int screen, grab;
-    bool positioned;
-};
-static inline void ft_mouse_init(struct ft_mouse *m) { *m = (struct ft_mouse){.screen = -1, .grab = -1}; }
-static inline bool ft_mouse_output_set(struct ft_mouse *m, int i, double x, double y,
-                                      double w, double h, double scale) {
-    if (i < 0 || i >= FT_MOUSE_OUTPUTS || m->buttons || !isfinite(x) || !isfinite(y) ||
-        !isfinite(w) || !isfinite(h) || !isfinite(scale) || fabs(x) > 65536 || fabs(y) > 65536 ||
-        w < 1 || h < 1 || w > 16384 || h > 16384 || scale < .25 || scale > 8) return false;
-    m->outputs[i] = (struct ft_mouse_output){true, x, y, w, h, scale};
-    return true;
-}
-static inline bool ft_mouse_position(struct ft_mouse *m, double x, double y) {
-    if (!isfinite(x) || !isfinite(y)) return false;
-    double best = INFINITY, bx = 0, by = 0;
-    int screen = -1;
-    for (int i = 0; i < FT_MOUSE_OUTPUTS; ++i) {
-        const struct ft_mouse_output *o = &m->outputs[i];
-        if (!o->enabled) continue;
-        double px = fmin(fmax(x, o->x), o->x + o->width - 1);
-        double py = fmin(fmax(y, o->y), o->y + o->height - 1);
-        double d = (px-x)*(px-x) + (py-y)*(py-y);
-        if (d < best) best = d, bx = px, by = py, screen = i;
-    }
-    if (screen < 0) return false;
-    m->x = bx; m->y = by; m->screen = screen; m->positioned = true;
-    return true;
-}
-static inline bool ft_mouse_move(struct ft_mouse *m, double dx, double dy) {
-    if (!isfinite(dx) || !isfinite(dy) || fabs(dx) > 16384 || fabs(dy) > 16384) return false;
-    if (!m->positioned) {
-        for (int i = 0; i < FT_MOUSE_OUTPUTS; ++i) if (m->outputs[i].enabled) {
-            const struct ft_mouse_output *o = &m->outputs[i];
-            if (!ft_mouse_position(m, o->x + o->width/2, o->y + o->height/2)) return false;
-            break;
-        }
-    }
-    return ft_mouse_position(m, m->x + dx, m->y + dy);
-}
-// The host seat's implicit grab stays on one KWin output during a drag. Motion
-// remains in that output's coordinates, including positions beyond its edges;
-// nested KWin adds the output origin and can drag windows across its monitors.
-static inline int ft_mouse_target(const struct ft_mouse *m) { return m->buttons ? m->grab : m->screen; }
-static inline bool ft_mouse_button(struct ft_mouse *m, unsigned button, bool down) {
-    if (!m->positioned || button < 272 || button > 279) return false;
-    uint32_t bit = 1u << (button - 272);
-    if (down) { if (!m->buttons) m->grab = m->screen; m->buttons |= bit; }
-    else { m->buttons &= ~bit; if (!m->buttons) m->grab = -1; }
-    return true;
-}

screens/tests/controller-fallback-test.c

@@ -1,29 +0,0 @@
-#include <assert.h>
-#include <stdio.h>
-#include "../controller-fallback.h"
-int main(void) {
-    struct ft_controller_fallback f = {0};
-    assert(!ft_fallback_active(&f, 100));
-    ft_fallback_presence(&f, false, true, 100);
-    assert(!ft_fallback_active(&f, 1099));
-    assert(ft_fallback_active(&f, 1100));
-    ft_fallback_presence(&f, false, true, 1100);
-    assert(ft_fallback_active(&f, 1100)); // heartbeat doesn't reset debounce
-    assert(!ft_fallback_active(&f, 4100)); // lost relay fails closed
-    ft_fallback_presence(&f, false, true, 4100);
-    assert(!ft_fallback_active(&f, 4100)); // old absence cannot bypass new debounce
-    assert(ft_fallback_active(&f, 5100));
-    ft_fallback_presence(&f, true, true, 4200);
-    assert(!ft_fallback_active(&f, 9000)); // connected but idle is still a mouse
-    ft_fallback_presence(&f, false, false, 9100);
-    assert(!ft_fallback_active(&f, 10100)); // disabled preference
-    ft_fallback_presence(&f, true, true, 10200);
-    ft_fallback_presence(&f, false, true, 10300);
-    assert(!ft_fallback_active(&f, 11299));
-    assert(ft_fallback_active(&f, 11300));
-    ft_fallback_presence(&f, true, true, 11301);
-    assert(!ft_fallback_active(&f, 11301)); // reconnect wins immediately
-    ft_fallback_presence(&f, false, true, UINT32_MAX-500);
-    assert(ft_fallback_active(&f, 500)); // monotonic counter wrap
-    puts("disconnect debounce, heartbeat expiry, reconnect and disabled fallback passed");
-}

screens/tests/cursor-cache-test.c

@@ -1,106 +0,0 @@
-// Actual rendererless Wayland/wlroots lifecycle. No OpenVR or real desktop.
-#define _GNU_SOURCE
-#define WLR_USE_UNSTABLE
-#include <assert.h>
-#include <drm_fourcc.h>
-#include <fcntl.h>
-#include <signal.h>
-#include <stdio.h>
-#include <string.h>
-#include <sys/mman.h>
-#include <sys/socket.h>
-#include <sys/wait.h>
-#include <unistd.h>
-#include <wayland-client.h>
-#include <wlr/types/wlr_shm.h>
-#include "../cursor-cache.h"
-
-static struct wl_compositor *client_compositor;
-static struct wl_shm *client_shm;
-static void global(void *data, struct wl_registry *registry, uint32_t name, const char *interface, uint32_t version) {
-    (void)data; (void)version;
-    if (!strcmp(interface, "wl_compositor")) client_compositor = wl_registry_bind(registry, name, &wl_compositor_interface, 4);
-    if (!strcmp(interface, "wl_shm")) client_shm = wl_registry_bind(registry, name, &wl_shm_interface, 1);
-}
-static void removed(void *data, struct wl_registry *registry, uint32_t name) { (void)data; (void)registry; (void)name; }
-static const struct wl_registry_listener registry_listener = {global, removed};
-static void stage(struct wl_display *display, int report, int ack, char value) {
-    assert(wl_display_roundtrip(display) >= 0);
-    assert(write(report, &value, 1) == 1);
-    char answer;
-    assert(read(ack, &answer, 1) == 1 && answer == value);
-}
-static void client(int fd, int report, int ack) {
-    alarm(10);
-    struct wl_display *d = wl_display_connect_to_fd(fd);
-    assert(d);
-    struct wl_registry *registry = wl_display_get_registry(d);
-    wl_registry_add_listener(registry, &registry_listener, NULL);
-    assert(wl_display_roundtrip(d) >= 0 && client_compositor && client_shm);
-    int memory = memfd_create("offline-cursor", MFD_CLOEXEC);
-    assert(memory >= 0 && ftruncate(memory, 600*4) == 0);
-    uint32_t *pixels = mmap(NULL, 600*4, PROT_READ|PROT_WRITE, MAP_SHARED, memory, 0);
-    assert(pixels != MAP_FAILED); pixels[0] = 0xff123456;
-    struct wl_shm_pool *pool = wl_shm_create_pool(client_shm, memory, 600*4);
-    struct wl_buffer *small = wl_shm_pool_create_buffer(pool, 0, 8, 1, 32, WL_SHM_FORMAT_ARGB8888);
-    struct wl_buffer *large = wl_shm_pool_create_buffer(pool, 0, 600, 1, 2400, WL_SHM_FORMAT_ARGB8888);
-    struct wl_surface *surface = wl_compositor_create_surface(client_compositor);
-    wl_surface_attach(surface, small, 0, 0); wl_surface_commit(surface);
-    stage(d, report, ack, 1); // image committed before any set_cursor/consumer
-    wl_surface_commit(surface); stage(d, report, ack, 2); // callback-only commit preserves image
-    wl_surface_attach(surface, NULL, 0, 0); wl_surface_commit(surface); stage(d, report, ack, 3);
-    wl_surface_attach(surface, large, 0, 0); wl_surface_commit(surface); stage(d, report, ack, 4);
-    wl_surface_attach(surface, small, 0, 0); wl_surface_commit(surface); stage(d, report, ack, 5);
-    wl_surface_destroy(surface); stage(d, report, ack, 6);
-    wl_buffer_destroy(small); wl_buffer_destroy(large); wl_shm_pool_destroy(pool);
-    wl_registry_destroy(registry); wl_display_disconnect(d);
-    munmap(pixels, 600*4); close(memory);
-    _exit(0);
-}
-int main(void) {
-    alarm(10);
-    struct wl_display *display = wl_display_create();
-    struct wlr_compositor *compositor = wlr_compositor_create(display, 6, NULL);
-    uint32_t formats[] = {DRM_FORMAT_ARGB8888, DRM_FORMAT_XRGB8888};
-    assert(wlr_shm_create(display, 1, formats, 2));
-    struct ft_cursor_cache cache;
-    ft_cursor_cache_init(&cache, compositor);
-    int connection[2], report[2], ack[2];
-    assert(socketpair(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0, connection) == 0);
-    assert(pipe(report) == 0 && pipe(ack) == 0);
-    pid_t pid = fork(); assert(pid >= 0);
-    if (!pid) { close(connection[0]); close(report[0]); close(ack[1]); client(connection[1], report[1], ack[0]); }
-    close(connection[1]); close(report[1]); close(ack[0]);
-    fcntl(report[0], F_SETFL, O_NONBLOCK);
-    assert(wl_client_create(display, connection[0]));
-    struct wl_event_loop *loop = wl_display_get_event_loop(display);
-    struct wlr_buffer *first = NULL;
-    for (char expected = 1; expected <= 6;) {
-        assert(wl_event_loop_dispatch(loop, 10) >= 0); wl_display_flush_clients(display);
-        char observed;
-        if (read(report[0], &observed, 1) != 1) continue;
-        assert(observed == expected);
-        if (expected == 6) assert(wl_list_empty(&cache.frames));
-        else {
-            assert(!wl_list_empty(&cache.frames));
-            struct ft_cursor_frame *f = wl_container_of(cache.frames.next, f, link);
-            assert(f->surface->current.buffer == NULL); // wlroots already released it
-            struct wlr_buffer *buffer = ft_cursor_cached(&cache, f->surface);
-            if (expected == 3 || expected == 4) assert(buffer == NULL);
-            else {
-                assert(buffer && buffer->width == 8 && buffer->height == 1);
-                if (expected == 1) first = buffer;
-                if (expected == 2) assert(buffer == first);
-                void *data; uint32_t format; size_t stride;
-                assert(wlr_buffer_begin_data_ptr_access(buffer, WLR_BUFFER_DATA_PTR_ACCESS_READ, &data, &format, &stride));
-                assert(*(uint32_t *)data == 0xff123456 && format == DRM_FORMAT_ARGB8888 && stride == 32);
-                wlr_buffer_end_data_ptr_access(buffer);
-            }
-        }
-        assert(write(ack[1], &observed, 1) == 1); ++expected;
-    }
-    int status; assert(waitpid(pid, &status, 0) == pid && WIFEXITED(status) && WEXITSTATUS(status) == 0);
-    close(report[0]); close(ack[1]); wl_display_destroy_clients(display);
-    ft_cursor_cache_finish(&cache); wl_display_destroy(display);
-    puts("cursor before selection, buffer-only lifetime, null/oversize commits and destruction passed");
-}

screens/tests/desktop-cursor-test.c

@@ -1,23 +0,0 @@
-#include "../desktop-cursor.h"
-#include <assert.h>
-#include <stdio.h>
-int main(void) {
-    uint32_t input[] = {0x80402010, 0, 0xdeadbeef, 0xff102030, 0x00ffffff, 0};
-    uint8_t out[16];
-    assert(ft_cursor_rgba(out, input, 2, 2, 12, DRM_FORMAT_ARGB8888));
-    assert(out[0] == 128 && out[1] == 64 && out[2] == 32 && out[3] == 128);
-    assert(out[4] == 0 && out[7] == 0);
-    assert(out[8] == 16 && out[9] == 32 && out[10] == 48 && out[11] == 255);
-    assert(out[12] == 0 && out[13] == 0 && out[14] == 0 && out[15] == 0);
-    assert(ft_cursor_rgba(out, input, 1, 1, 4, DRM_FORMAT_ABGR8888));
-    assert(out[0] == 32 && out[1] == 64 && out[2] == 128);
-    assert(ft_cursor_rgba(out, input, 1, 1, 4, DRM_FORMAT_XRGB8888));
-    assert(out[0] == 64 && out[1] == 32 && out[2] == 16 && out[3] == 255);
-    assert(ft_cursor_rgba(out, input, 1, 1, 4, DRM_FORMAT_XBGR8888));
-    assert(out[0] == 16 && out[2] == 64 && out[3] == 255);
-    assert(!ft_cursor_rgba(out, input, 1, 1, 3, DRM_FORMAT_ARGB8888));
-    assert(!ft_cursor_rgba(out, input, 513, 1, 2052, DRM_FORMAT_ARGB8888));
-    assert(!ft_cursor_rgba(out, NULL, 1, 1, 4, DRM_FORMAT_ARGB8888));
-    assert(!ft_cursor_rgba(out, input, 1, 1, 4, DRM_FORMAT_RGB565));
-    puts("KDE cursor channels, alpha, padding and bounds passed");
-}

screens/tests/desktop-handoff-test.c

@@ -1,22 +0,0 @@
-#include <assert.h>
-#include "../desktop-handoff.h"
-int main(void) {
-    struct ft_handoff h = {.policy=FT_LAST_ACTIVE};
-    assert(ft_handoff_claim(&h, false, 100, 0, 0));
-    assert(!ft_handoff_claim(&h, true, 100, 0, 0));
-    assert(ft_handoff_claim(&h, true, 101, 0, 0));
-    assert(!ft_handoff_claim(&h, true, 102, 1, 0));
-    assert(!ft_handoff_claim(&h, false, 102, 0, 1));
-    assert(ft_handoff_claim(&h, true, 102, 0, 1));
-    assert(ft_handoff_claim(&h, false, 102, 1, 0));
-    assert(ft_handoff_claim(&h, false, 103, 0, 0));
-    h.policy=FT_PREFER_MOUSE;
-    assert(!ft_handoff_claim(&h, true, 104, 0, 0));
-    assert(ft_handoff_claim(&h, false, 104, 0, 0));
-    h.policy=FT_PREFER_POINTER;
-    assert(!ft_handoff_claim(&h, false, 105, 0, 0));
-    assert(ft_handoff_claim(&h, true, 105, 0, 0));
-    h=(struct ft_handoff){.policy=FT_LAST_ACTIVE};
-    assert(ft_handoff_claim(&h, false, UINT32_MAX, 0, 0));
-    assert(ft_handoff_claim(&h, true, 0, 0, 0));
-}

screens/tests/desktop-input-integration.py

@@ -1,89 +0,0 @@
-"""Only addresses upstream's isolated test compositor, never live input sockets."""
-import json
-import socket
-import time
-
-with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as client:
-    client.bind('')
-    client.settimeout(2)
-    def ask(command):
-        client.sendto(command.encode(), '\0ft_screens_test')
-        return client.recv(8192).decode()
-    for command in ('mouse-layout 1 0 0 960 540 2',
-                    'mouse-layout 2 960 0 540 960 2', 'mouse-output-off 24',
-                    'mouse-mode desktop', 'mouse-position 1 100 100', 'mouse-move 10 5',
-                    'mouse-button 272 1', 'mouse-move 900 0', 'mouse-button 272 0',
-                    'mouse-wheel 0 0.125'):
-        assert ask(command) == 'ok', command
-    assert ask('mouse-output-off 25').startswith('error')
-    assert ask('mouse-layout 25 0 0 960 540 1').startswith('error')
-    ask('mouse-presence 1 1')
-    before = json.loads(ask('mouse?'))
-    ask('input 1 down 200 100 left'); ask('input 1 up 200 100 left')
-    after = json.loads(ask('mouse?'))
-    assert after['controllerEventsIgnored'] >= before['controllerEventsIgnored'] + 2
-    ask('mouse-presence 0 1')
-    time.sleep(1.1)
-    assert json.loads(ask('mouse-presence?'))['controllerDesktop']
-    before = json.loads(ask('mouse?'))
-    ask('input 1 down 200 100 left')
-    after = json.loads(ask('mouse?'))
-    assert after['controllerEventsIgnored'] == before['controllerEventsIgnored']
-    ask('mouse-presence 1 1')
-    assert not json.loads(ask('mouse-presence?'))['controllerDesktop']
-    ask('mouse-button 272 1'); ask('mouse-button 272 0')
-print('Native motion/click/wheel, cross-output drag, output bounds and controller fallback passed')
-
-with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as client:
-    client.bind(''); client.settimeout(2)
-    def ask(command):
-        client.sendto(command.encode(), '\0ft_screens_test')
-        return client.recv(8192).decode()
-    def policy(): return json.loads(ask('desktop-policy?'))
-    assert ask('desktop-policy last-active') == 'ok'
-    assert ask('mouse-presence 1 1') == 'ok'
-    assert ask('mouse-move 3 0') == 'ok'
-    assert policy()['owner'] == 'mouse'
-    for _ in range(3): ask('input 1 move 200 100')
-    assert policy()['owner'] == 'mouse', 'aiming must not claim ownership'
-    time.sleep(.01)
-    ask('input 1 down 200 100 left')
-    assert policy()['owner'] == 'pointer'
-    assert policy()['pointerButtons'] == 1
-    before = json.loads(ask('mouse?'))
-    ask('mouse-move 40 0'); ask('mouse-button 272 1'); ask('mouse-button 272 0'); ask('mouse-wheel 0 1')
-    after = json.loads(ask('mouse?'))
-    assert before['x'] == after['x'], 'mouse motion must not accumulate during pointer drag'
-    assert before['buttonEvents'] == after['buttonEvents']
-    assert before['wheelEvents'] == after['wheelEvents']
-    assert policy()['owner'] == 'pointer'
-    assert ask('desktop-policy mouse').startswith('error'), 'cannot change policy during drag'
-    ask('mouse-presence 1 1')
-    assert policy()['owner'] == 'pointer', 'presence heartbeat must not steal ownership'
-    ask('input 1 up 200 100 left')
-    assert policy()['pointerButtons'] == 0
-    ask('mouse-wheel 0 1')
-    assert policy()['owner'] == 'mouse'
-    ask('mouse-button 272 1'); time.sleep(.01)
-    ask('input 1 down 200 100 left'); ask('input 1 up 200 100 left')
-    assert policy()['owner'] == 'mouse'
-    assert policy()['mouseButtons'] == 1
-    ask('mouse-button 272 0')
-    time.sleep(.01); ask('input 1 down 200 100 left'); ask('input 1 up 200 100 left')
-    assert policy()['owner'] == 'pointer'
-    ask('mouse-move 0 0')
-    assert policy()['owner'] == 'pointer', 'zero movement must not claim'
-    ask('mouse-move 1 0')
-    assert policy()['owner'] == 'mouse'
-    time.sleep(.01)
-    ask('input 1 move 200 100'); ask('input 1 scroll 0 1')
-    assert policy()['owner'] == 'pointer', 'controller scroll must claim desktop'
-    ask('mouse-wheel 0 1')
-    assert policy()['owner'] == 'mouse'
-    assert ask('desktop-policy pointer') == 'ok'
-    ask('mouse-move 1 0')
-    assert policy()['owner'] == 'pointer'
-    assert ask('desktop-policy mouse') == 'ok'
-    ask('input 1 down 200 100 left'); ask('input 1 up 200 100 left')
-    assert policy()['owner'] == 'mouse'
-print('Desktop handoff, aiming exclusion, drag locks, ignored releases, presence stability and policy controls passed')

screens/tests/desktop-mouse-test.c

@@ -1,34 +0,0 @@
-#include "../desktop-mouse.h"
-#include <assert.h>
-#include <stdio.h>
-int main(void) {
-    struct ft_mouse m;
-    ft_mouse_init(&m);
-    assert(!ft_mouse_move(&m, 0, 0));
-    assert(ft_mouse_output_set(&m, 0, -1280, 0, 1280, 720, 1.5));
-    assert(ft_mouse_output_set(&m, 1, 0, 0, 1920, 1080, 1));
-    assert(ft_mouse_move(&m, 0, 0));
-    assert(m.screen == 0 && m.x == -640 && m.y == 360);
-    assert(ft_mouse_position(&m, -1, 100));
-    assert(ft_mouse_button(&m, 272, true));
-    assert(!ft_mouse_output_set(&m, 0, 0, 0, 100, 100, 1));
-    assert(ft_mouse_move(&m, 20, 0));
-    assert(m.screen == 1 && m.x == 19 && ft_mouse_target(&m) == 0);
-    // During the grab, host coordinates exceed the old output's width.
-    assert((m.x-m.outputs[0].x)*m.outputs[0].scale == 1948.5);
-    assert(ft_mouse_button(&m, 275, true));
-    assert(ft_mouse_button(&m, 272, false));
-    assert(ft_mouse_target(&m) == 0);
-    assert(ft_mouse_button(&m, 275, false));
-    assert(ft_mouse_target(&m) == 1 && m.grab == -1);
-    assert(ft_mouse_move(&m, 10000, 10000));
-    assert(m.x == 1919 && m.y == 1079);
-    assert(!ft_mouse_move(&m, NAN, 0));
-    assert(!ft_mouse_position(&m, INFINITY, 1));
-    assert(!ft_mouse_move(&m, 16385, 0));
-    assert(!ft_mouse_output_set(&m, FT_MOUSE_OUTPUTS, 0, 0, 100, 100, 1));
-    assert(!ft_mouse_output_set(&m, 0, 0, 0, 0, 100, 1));
-    assert(!ft_mouse_output_set(&m, 0, 0, 0, 100, 100, NAN));
-    assert(!ft_mouse_button(&m, 280, true));
-    puts("desktop mouse geometry, drag and validation passed");
-}

screens/vr.cpp

@@ -302,12 +302,6 @@ struct Screen {
     }
 };
 std::map<int, Screen> g_screens;
-struct DesktopCursor {
-    vr::VROverlayHandle_t overlay = vr::k_ulOverlayHandleInvalid;
-    int width = 0, height = 0, hx = 0, hy = 0, scale = 1, screen = -1;
-    double x = 0, y = 0, outputScale = 1;
-    bool enabled = false, image = false;
-} g_cursor;
 std::map<const void *, vr::SharedTextureHandle_t> g_imports;
 
 // Hand cutouts (see the top and handcut.h).
@@ -590,32 +584,6 @@ Mat OnSurface(const Screen &s, double u, double v, double dz) {
     return m;
 }
 
-void UpdateDesktopCursor() {
-    if (g_cursor.overlay == vr::k_ulOverlayHandleInvalid) return;
-    const auto it = g_screens.find(g_cursor.screen);
-    const auto activity = vr::VRSystem()->GetTrackedDeviceActivityLevel(vr::k_unTrackedDeviceIndex_Hmd);
-    const bool off = activity == vr::k_EDeviceActivityLevel_Idle || activity == vr::k_EDeviceActivityLevel_Standby ||
-                     activity == vr::k_EDeviceActivityLevel_Idle_Timeout;
-    Mat pose;
-    if (!g_cursor.enabled || !g_cursor.image || off || it == g_screens.end() ||
-        !it->second.visible || it->second.width < 1 || !ScreenPose(it->second, &pose)) {
-        vr::VROverlay()->HideOverlay(g_cursor.overlay);
-        return;
-    }
-    const Screen &s = it->second;
-    const double perPixel = s.metres / s.width;
-    const double cursorPixel = perPixel * g_cursor.outputScale / g_cursor.scale;
-    const double u = (g_cursor.x - s.width/2.0)*perPixel + (g_cursor.width/2.0-g_cursor.hx)*cursorPixel;
-    const double v = (s.height/2.0-g_cursor.y)*perPixel + (g_cursor.hy-g_cursor.height/2.0)*cursorPixel;
-    const double width = g_cursor.width*cursorPixel;
-    const Mat m = Mul(pose, OnSurface(s, u, v, 0.004));
-    vr::VROverlay()->SetOverlayTransformAbsolute(g_cursor.overlay, vr::TrackingUniverseStanding, &m);
-    vr::VROverlay()->SetOverlayWidthInMeters(g_cursor.overlay, float(width));
-    vr::VROverlay()->SetOverlayCurvature(g_cursor.overlay, s.curve > 0 ? float(std::min(1.0, width/(2*M_PI*s.curve))) : 0);
-    vr::VROverlay()->SetOverlayAlpha(g_cursor.overlay, s.alpha);
-    vr::VROverlay()->ShowOverlay(g_cursor.overlay);
-}
-
 double BarY(const Screen &s) { return -(s.heightMetres() / 2 + s.chrome * 0.06 + s.chrome * 12 / 256); }
 Mat BarOffset(const Screen &s) { return OnSurface(s, 0, BarY(s), 0.003); }
 
@@ -1572,7 +1540,6 @@ bool ft_vr_init(void) {
 
 void ft_vr_shutdown(void) {
     if (!g_vr) return;
-    if (g_cursor.overlay != vr::k_ulOverlayHandleInvalid) vr::VROverlay()->DestroyOverlay(g_cursor.overlay);
     if (g_cutterState == 1)
         for (auto &[i, s] : g_screens) g_cutter.DropPanel(i);  // drops their imports while SteamVR is up
     if (g_catcher != vr::k_ulOverlayHandleInvalid) vr::VROverlay()->DestroyOverlay(g_catcher);
@@ -1602,71 +1569,6 @@ int ft_vr_modifiers(uint32_t format, uint64_t *out, int max) {
 
 bool ft_vr_screens_shown(void) { return g_vr && ModeVisible(); }
 
-static bool EnsureDesktopCursor() {
-    if (!g_vr) return false;
-    if (g_cursor.overlay == vr::k_ulOverlayHandleInvalid) {
-        if (vr::VROverlay()->CreateOverlay("frametop.desktop.cursor", "KDE desktop cursor", &g_cursor.overlay) != vr::VROverlayError_None) return false;
-        vr::VROverlay()->SetOverlayInputMethod(g_cursor.overlay, vr::VROverlayInputMethod_None);
-        vr::VROverlay()->SetOverlaySortOrder(g_cursor.overlay, 200);
-    }
-    return true;
-}
-
-bool ft_vr_cursor_image(const uint8_t *rgba, int width, int height, int hx, int hy, int scale) {
-    if (!rgba) { g_cursor.image = false; UpdateDesktopCursor(); return true; }
-    if (width < 1 || height < 1 || width > 512 || height > 512 || scale < 1 || scale > 8 || !EnsureDesktopCursor()) return false;
-    if (vr::VROverlay()->SetOverlayRaw(g_cursor.overlay, const_cast<uint8_t *>(rgba), width, height, 4) != vr::VROverlayError_None) return false;
-    g_cursor.width = width; g_cursor.height = height; g_cursor.hx = hx; g_cursor.hy = hy; g_cursor.scale = scale;
-    g_cursor.image = true;
-    UpdateDesktopCursor();
-    return true;
-}
-
-void ft_vr_cursor_move(int screen, double x, double y, double scale, bool enabled) {
-    g_cursor.screen = screen; g_cursor.x = x; g_cursor.y = y; g_cursor.outputScale = scale; g_cursor.enabled = enabled;
-    UpdateDesktopCursor();
-}
-
-static vr::SharedTextureHandle_t ImportBuffer(const void *key, const struct ft_dmabuf *b) {
-    auto it = g_imports.find(key);
-    if (it == g_imports.end()) {
-        vr::DmabufAttributes_t a{};
-        a.unWidth = uint32_t(b->width);
-        a.unHeight = uint32_t(b->height);
-        a.unDepth = a.unMipLevels = a.unArrayLayers = a.unSampleCount = 1;
-        a.unFormat = b->format;
-        a.ulModifier = b->modifier;
-        a.unPlaneCount = uint32_t(b->n_planes);
-        for (int i = 0; i < b->n_planes && i < int(vr::MaxDmabufPlaneCount); ++i) {
-            a.plane[i].unOffset = b->offset[i];
-            a.plane[i].unStride = b->stride[i];
-            a.plane[i].nFd = b->fd[i];
-        }
-        vr::SharedTextureHandle_t h = 0;
-        if (!vr::VRIPCResourceManager()->ImportDmabuf(vr::VRApplication_Overlay, &a, &h)) {
-            std::fprintf(stderr, "openvr: ImportDmabuf failed: %dx%d format 0x%x modifier 0x%llx\n", b->width,
-                         b->height, b->format, (unsigned long long)b->modifier);
-            return 0;
-        }
-        it = g_imports.emplace(key, h).first;
-    }
-    return it->second;
-}
-
-bool ft_vr_cursor_dmabuf(const void *key, const struct ft_dmabuf *b, int hx, int hy, int scale) {
-    if (!b || b->width < 1 || b->height < 1 || b->width > 512 || b->height > 512 ||
-        scale < 1 || scale > 8 || !EnsureDesktopCursor()) return false;
-    vr::SharedTextureHandle_t handle = ImportBuffer(key, b);
-    if (!handle) return false;
-    vr::Texture_t tex = {&handle, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
-    if (vr::VROverlay()->SetOverlayTexture(g_cursor.overlay, &tex) != vr::VROverlayError_None) return false;
-    g_cursor.width = b->width; g_cursor.height = b->height; g_cursor.hx = hx; g_cursor.hy = hy; g_cursor.scale = scale;
-    g_cursor.image = true;
-    UpdateDesktopCursor();
-    return true;
-}
-
-
 }  // extern "C"
 
 namespace {
@@ -1991,7 +1893,6 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
     UpdateLasers();
     UpdateControls();
     UpdateGuides();
-    UpdateDesktopCursor();
     UpdateCutouts();
     UpdateCatcher();
 }

screens/vr.h

@@ -35,11 +35,6 @@ void ft_vr_shutdown(void);
 int ft_vr_modifiers(uint32_t format, uint64_t *out, int max);
 // The screens are showing (by the visibility mode; not counting a wrist-pinned screen).
 bool ft_vr_screens_shown(void);
-// Desktop cursor: KDE supplies the image; this is a non-interactive display
-// overlay only. Mouse input goes straight to the Wayland seat, never SteamVR.
-bool ft_vr_cursor_image(const uint8_t *rgba, int width, int height, int hotspot_x, int hotspot_y, int buffer_scale);
-bool ft_vr_cursor_dmabuf(const void *key, const struct ft_dmabuf *buf, int hotspot_x, int hotspot_y, int buffer_scale);
-void ft_vr_cursor_move(int screen, double pixel_x, double pixel_y, double output_scale, bool enabled);
 // A panel for screen `index`, width in metres, placed in a row in front of the head.
 void ft_vr_screen_create(int index, double metres, int count);
 void ft_vr_screen_destroy(int index);

scripts/keys-report.py

@@ -112,7 +112,7 @@ def main():
     try:
         with open(os.path.join(HOME, ".config/frametop.conf")) as f:
             conf = [ln.split("#")[0].strip() for ln in f]
-        out("Settings: " + " ".join(c for c in conf if re.match(r"(POINTER|META_DASHBOARD|SHARE_KEYS|BACKEND)=", c)))
+        out("Settings: " + " ".join(c for c in conf if re.match(r"(POINTER|SHARE_KEYS|BACKEND)=", c)))
     except OSError:
         out("Settings: no ~/.config/frametop.conf")
     out("ft-screens state (visibility, manual, wrist, gesture, lasers, game running, in games): "

scripts/test-desktop-input.sh

@@ -1,29 +0,0 @@
-#!/usr/bin/env bash
-# Offline native input tests; --headless also starts an isolated test desktop.
-set -euo pipefail
-root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
-cd "$root"
-[[ $# == 0 || ($# == 1 && $1 == --headless) ]] || { echo 'Usage: scripts/test-desktop-input.sh [--headless]' >&2; exit 2; }
-python3 -m unittest discover -s input -p 'test_desktop_mouse.py'
-"$root/scripts/frame.sh" -C screens 'set -e; mkdir -p build
-gcc -std=c11 -Wall -Wextra -Werror tests/desktop-mouse-test.c -lm -o build/desktop-mouse-test
-build/desktop-mouse-test
-gcc -std=c11 -Wall -Wextra -Werror tests/controller-fallback-test.c -o build/controller-fallback-test
-build/controller-fallback-test
-gcc -std=c11 -Wall -Wextra -Werror tests/desktop-handoff-test.c -o build/desktop-handoff-test
-build/desktop-handoff-test
-gcc -std=c11 -Wall -Wextra -Werror $(pkg-config --cflags libdrm) tests/desktop-cursor-test.c -o build/desktop-cursor-test
-build/desktop-cursor-test
-gcc -std=c11 -Wall -Wextra -Werror tests/cursor-cache-test.c $(pkg-config --cflags --libs wlroots-0.20 wayland-server wayland-client libdrm pixman-1) -o build/cursor-cache-test
-build/cursor-cache-test'
-if [[ ${1:-} == --headless ]]; then
-  # Reuses upstream's throwaway @ft_screens_test fixture, never the live desktop.
-  if pgrep -f '[f]t-screens.*--control ft_screens_test' >/dev/null; then
-    echo 'Test desktop already running; stop it yourself before this test.' >&2
-    exit 1
-  fi
-  "$root/screens/build.sh"
-  trap '"$root/screens/test/headless.sh" stop' EXIT
-  "$root/screens/test/headless.sh" start 2 3
-  python3 "$root/screens/tests/desktop-input-integration.py"
-fi

scripts/update-check.py

@@ -321,6 +321,30 @@ def check_services(sockets, desktop_up):
         report("FAIL", "XRService", "not running, so hand tracking has no cameras")
 
 
+def check_steam_ui():
+    """Steam's UI calls the Steam menu shortcut uses (steam/ft-steam). They're Steam client
+    internals, so a Steam update can change them too, not just a SteamOS one."""
+    # Run from stdin (over SSH) there's no __file__: try the usual checkouts.
+    repos = [os.path.dirname(os.path.dirname(os.path.abspath(__file__)))] if "__file__" in globals() else []
+    repos += [os.path.join(HOME, "frametop"), os.path.join(HOME, "dev/frametop")]
+    tool = next((t for t in (os.path.join(r, "steam/ft-steam") for r in repos) if os.path.exists(t)), None)
+    if not tool:
+        report("skip", "Steam menu shortcut", "steam/ft-steam isn't in a checkout here")
+        return
+    try:
+        p = subprocess.run([tool, "check"], capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.TimeoutExpired) as e:
+        report("warn", "Steam menu shortcut", f"ft-steam check didn't run ({e})")
+        return
+    detail = (p.stdout + p.stderr).strip()
+    if p.returncode == 0:
+        report("ok", "Steam menu shortcut", "Steam's UI still has the calls steam/ft-steam makes")
+    elif detail.startswith("ft-steam:"):
+        report("skip", "Steam menu shortcut", detail[len("ft-steam:"):].strip())
+    else:
+        report("warn", "Steam menu shortcut", f"{detail}; a Meta tap won't open the Steam menu (steam/ft-steam)")
+
+
 def check_eye_tracker():
     gaze = systemctl("is-enabled", "frametop-gaze") == "enabled"
     try:
@@ -369,6 +393,7 @@ def main():
         check_openvr()
         check_overlays(desktop_up)
         check_services(sockets, desktop_up)
+        check_steam_ui()
         check_eye_tracker()
 
     if args == ["--mark-good"]:

session/frametop-session.sh

@@ -177,11 +177,6 @@ export XDG_CONFIG_HOME=$HOME/.config/frametop
 export XDG_STATE_HOME=$HOME/.local/state/frametop
 mkdir -p "$XDG_CONFIG_HOME" "$XDG_STATE_HOME"
 
-# Steam's VR session exports a 256px cursor. Use this desktop's own KDE
-# preferences instead of inheriting the dashboard cursor size and theme.
-export XCURSOR_SIZE=$(kreadconfig6 --file kcminputrc --group Mouse --key cursorSize --default 24)
-export XCURSOR_THEME=$(kreadconfig6 --file kcminputrc --group Mouse --key cursorTheme --default breeze_cursors)
-
 # Remote desktop over VNC: session/remote-desktop.sh captures the desktop with
 # krdp on 127.0.0.1, and session/vnc-bridge.sh re-serves its primary screen over VNC. krdpserver runs from the container, so KWin can't
 # match it to an installed app. KWin's permission check for screencast and fake

session/frametop.conf.example

@@ -10,9 +10,6 @@ REMOTE=0           # 1 = serve the desktop's primary screen over VNC on the tail
 FLOAT_SLOTS=8      # screens backend: how many windows can float in VR at once (spare outputs; 0 turns it off; restart the desktop after a change)
 FLOAT_MARGIN=300   # floating windows: pixels around each window on its output, so menus have room past its edges
 POINTER=0          # 1 = the mouse drives the universal 3D pointer (ft_pointer driver) instead of a plain mouse
-MOUSE_MODE=spatial # desktop: physical mouse controls KDE directly; controllers remain separate. CLI: input/ft-mousectl desktop --save
-DESKTOP_MOUSE_SPEED=1 # logical desktop pixels per physical mouse count (0.05 to 10)
-DESKTOP_CONTROLLER_FALLBACK=1 # desktop controller pointers when no routed physical mouse is connected
 POINTER_SENSITIVITY=0.03   # degrees per mouse count
 POINTER_IDLE=30    # seconds without mouse use before the controllers get their laser back
 POINTER_WAKE_COUNTS=40     # mouse counts within 1 s needed to wake or re-claim the laser (ignores desk jitter)
@@ -41,7 +38,7 @@ POINTER_GAZE_SHOW=1        # gaze mode, with POINTER_GAZE_DOT=moving: the dot sh
 POINTER_GAZE_MOUSE=precision # gaze mode: the left button holds back its press, the mouse steers, the release clicks (precision) | clicks right away (direct)
 POINTER_HEAD_DEADZONE=0.5  # keyboard clicks at the gaze (Meta+J, Meta+K): degrees the head turns before the held dot moves with it
 POINTER_KEY_TAP=0.25       # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
-GAZE_TRACKER=steam         # gaze service: steam = SteamVR's eye tracker | own = our own (gaze/tracker: its frame grabber needs gaze/tracker/install.sh; calibrated with Calibrate on Frametop Input Settings' Gaze page)
+GAZE_TRACKER=auto          # gaze service: auto = our own eye tracker when it's installed (gaze/tracker/install.sh, which install.sh offers), else SteamVR's | own = ours | steam = SteamVR's. Each keeps its own calibration (Calibrate on Frametop Input Settings' Gaze page)
 GAZE_EYE=auto              # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
 HANDS_SWAP_SIDES=0         # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
 HANDS_CPUS=5,6,7           # hand tracking: the CPUs its model threads run on
@@ -57,7 +54,6 @@ POINTER_PINCH_DEADZONE=1.5 # hands: degrees the pinching hand moves before the p
 POINTER_GRIP_GAIN=1        # hands: the pointer moves this times the gripping hand's angle
 POINTER_GRIP_BELOW=0.35    # hands: grips beginning more than this (m) below the eyes are ignored (hands on a desk)
 POINTER_PINCH_TYPING=1     # hands: no pinch within this many seconds of a key (typing touches thumb to index)
-META_DASHBOARD=0   # 1 = a Meta tap on a pass-through keyboard toggles the SteamVR dashboard (pointer mode only)
 SHARE_KEYS=0       # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen)
 DISPLAY_OFF_MIN=0  # minutes without use before frametop-power turns the displays off, even if the headset seems worn (0 = never)
 DISPLAY_MOVE_MM=5  # displays off: the headset or a controller moving this far (mm) within 10 s counts as use

steam/ft-steam

@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+"""ft-steam: Steam's menu and checks, through Steam's own UI (steamui.py).
+
+  ft-steam menu    open the SteamVR dashboard on Steam's menu, or close the dashboard if it's
+                   open (the Steam button's job, as a shortcut: the input relay's steam_menu)
+  ft-steam check   do Steam's UI calls that the above needs still exist (update-check.py)
+
+No pointer mode needed: it asks Steam to show its dashboard overlay (window.ShowVROverlay, what
+Steam itself calls) and to focus the Steam frame's left menu (MenuStore.OpenMainMenu, which in
+VR invokes the frame's focus_left_frame_menu_action). SteamClient.OpenVR.VROverlay says
+whether the dashboard is up and hides it. These are Steam's UI internals, not an API, so a
+Steam client update can rename them: `ft-steam check` says so.
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
+from steamui import SteamUnreachable, evaluate  # noqa: E402
+
+WINDOW = "SteamUIStore.WindowStore.VRGamepadUIMainWindowInstance"
+
+MENU = f"""(async () => {{
+    const vr = SteamClient.OpenVR.VROverlay;
+    if (await vr.IsDashboardVisible()) {{ vr.HideDashboard(); return "closed"; }}
+    const w = {WINDOW};
+    if (!w) throw new Error("Steam's VR dashboard window isn't open");
+    ShowVROverlay(w.GetVROverlayKey());
+    // The frame's menu takes focus only once the dashboard is up.
+    for (let i = 0; i < 40 && !(await vr.IsDashboardVisible()); i++) await new Promise(r => setTimeout(r, 25));
+    w.MenuStore.OpenMainMenu();
+    return "opened";
+}})()"""
+
+CHECK = f"""(() => {{
+    const missing = [];
+    const vr = SteamClient.OpenVR && SteamClient.OpenVR.VROverlay;
+    for (const f of ["IsDashboardVisible", "HideDashboard"]) if (!vr || typeof vr[f] !== "function") missing.push("SteamClient.OpenVR.VROverlay." + f);
+    if (typeof window.ShowVROverlay !== "function") missing.push("ShowVROverlay");
+    const w = {WINDOW};
+    if (!w) missing.push("{WINDOW}");
+    else {{
+        if (typeof w.GetVROverlayKey !== "function") missing.push("GetVROverlayKey");
+        if (!w.MenuStore || typeof w.MenuStore.OpenMainMenu !== "function") missing.push("MenuStore.OpenMainMenu");
+    }}
+    return missing;
+}})()"""
+
+
+def main():
+    cmd = sys.argv[1] if len(sys.argv) == 2 else ""
+    if cmd not in ("menu", "check"):
+        sys.exit(__doc__)
+    try:
+        result = evaluate(MENU if cmd == "menu" else CHECK)
+    except SteamUnreachable as e:
+        sys.exit(f"ft-steam: {e}")
+    if cmd == "check":
+        if result:
+            sys.exit("Steam's UI lacks " + ", ".join(result))
+        print("ok")
+    else:
+        print(result)
+
+
+if __name__ == "__main__":
+    main()

steam/steamui.py

@@ -0,0 +1,108 @@
+"""Run JavaScript in Steam's UI, standard library only.
+
+Steam on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context
+(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080. Steam's
+own calls live there: SteamClient.* (settings, SteamVR's dashboard) and the UI's stores
+(SteamUIStore). Used by Frametop Display Settings (display-settings/steam_settings.py) and
+ft-steam (the Steam menu shortcut).
+
+  evaluate(expression)   the expression's value (awaited, JSON-able); SteamUnreachable when
+                         Steam or its UI isn't there, or the script threw
+"""
+import base64
+import json
+import os
+import socket
+import struct
+import urllib.request
+
+CDP_PORT = 8080
+
+
+class SteamUnreachable(Exception):
+    pass
+
+
+class _WebSocket:
+    def __init__(self, url, timeout=5):
+        host_port, path = url[len("ws://"):].split("/", 1)
+        host, port = host_port.rsplit(":", 1)
+        self.sock = socket.create_connection((host, int(port)), timeout=timeout)
+        key = base64.b64encode(os.urandom(16)).decode()
+        self.sock.sendall((f"GET /{path} HTTP/1.1\r\nHost: {host_port}\r\nUpgrade: websocket\r\n"
+                           f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode())
+        head = b""
+        while b"\r\n\r\n" not in head:
+            chunk = self.sock.recv(4096)
+            if not chunk:
+                raise SteamUnreachable("Steam closed the connection")
+            head += chunk
+        if b" 101 " not in head.split(b"\r\n", 1)[0]:
+            raise SteamUnreachable(head.split(b"\r\n", 1)[0].decode(errors="replace"))
+        self.buf = head.split(b"\r\n\r\n", 1)[1]
+
+    def send(self, text):
+        data, mask = text.encode(), os.urandom(4)
+        n = len(data)
+        if n < 126:
+            head = struct.pack(">BB", 0x81, 0x80 | n)
+        elif n < 65536:
+            head = struct.pack(">BBH", 0x81, 0x80 | 126, n)
+        else:
+            head = struct.pack(">BBQ", 0x81, 0x80 | 127, n)
+        self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(data)))
+
+    def _take(self, n):
+        while len(self.buf) < n:
+            chunk = self.sock.recv(65536)
+            if not chunk:
+                raise SteamUnreachable("Steam closed the connection")
+            self.buf += chunk
+        out, self.buf = self.buf[:n], self.buf[n:]
+        return out
+
+    def recv(self):
+        message = b""
+        while True:
+            b0, b1 = self._take(2)
+            n = b1 & 0x7F
+            if n == 126:
+                n = struct.unpack(">H", self._take(2))[0]
+            elif n == 127:
+                n = struct.unpack(">Q", self._take(8))[0]
+            message += self._take(n)
+            if b0 & 0x80:
+                return message.decode(errors="replace")
+
+    def close(self):
+        self.sock.close()
+
+
+def evaluate(expression):
+    """Runs JavaScript in Steam's SharedJSContext and returns its (awaited) value."""
+    try:
+        with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}/json", timeout=3) as r:
+            targets = json.load(r)
+    except OSError as e:
+        raise SteamUnreachable(f"Steam isn't reachable on port {CDP_PORT} ({e})") from e
+    url = next((t["webSocketDebuggerUrl"] for t in targets if t.get("title") == "SharedJSContext"), None)
+    if not url:
+        raise SteamUnreachable("Steam's UI isn't running")
+    try:
+        ws = _WebSocket(url)
+        try:
+            ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
+                                "params": {"expression": expression, "awaitPromise": True, "returnByValue": True}}))
+            while True:
+                reply = json.loads(ws.recv())
+                if reply.get("id") == 1:
+                    break
+        finally:
+            ws.close()
+    except OSError as e:
+        raise SteamUnreachable(str(e)) from e
+    result = reply.get("result", {})
+    if "exceptionDetails" in result:
+        details = result["exceptionDetails"]
+        raise SteamUnreachable(details.get("exception", {}).get("description") or details.get("text", "error"))
+    return result.get("result", {}).get("value")