diff --git a/README.md b/README.md index e094ed5..1623436 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/DeeJanuz/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental). - The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you three things along the way: whether to install gaze mode (experimental, yes by default) and the Bluetooth fixes, then whether to restart SteamVR. The Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too. + The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too. After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings. @@ -57,9 +57,11 @@ If you work in the desktop for long stretches, or leave the headset on a stand, | While carrying a screen, sweep its laser across your other controller's ring, then let go | Pins it to that wrist, at its size and distance, as you hold it when you let go; it shows while you see its front. Grab its bar to adjust it (it stays pinned); sweep across the ring again to take it off | | Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head | | Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) | +| Meta+Alt+Tab, or Meta+Alt+Shift+Tab | Spins every screen and floating window around you together, like a lazy susan, so the next one on your right (or left) glides to straight ahead, with the pointer and typing going to it. Their arrangement stays the same: the room turns instead of you. Tap again to keep going; Meta+Shift+R puts the screens back. Pinned screens stay where they are | | Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist | +| Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own | | Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead | -| Play a VR game | The screens hide and your controllers stay in the game. Open the SteamVR dashboard, or press Meta+Shift+H, to see and use them. To keep them visible over games, change During VR games on the Visibility & pins tab; the controllers still stay in the game, and you use the screens with the mouse or the dashboard | +| Play a VR game | Frametop pauses so the game gets the headset to itself (see [Pause for VR games](#pause-for-vr-games)): the screens hide and your controllers stay in the game. Click both thumbsticks together twice to bring Frametop back. With the automatic pause off, the screens still hide, and the SteamVR dashboard or Meta+Shift+H shows them; to keep them visible over games, change During VR games on the Visibility & pins tab | Restarting the desktop (Restart desktop in Frametop Display Settings) closes its windows, but background work you started in it, such as servers, tmux sessions, or builds, keeps running. @@ -94,9 +96,21 @@ A profile is a named setup: where the screens are, with their sizes and pins, wh | Pick a profile under Arrangement and press Open profile | Switches to it: the screens move, open windows of its apps go to their places, and the apps that aren't open start. Nothing closes | | Pick a profile under Start in profile, or run its entry (Frametop: NAME) from SteamVR's Launch a program list | The desktop starts in that profile, or switches to it if it's running. A profile can also go on a key combination, mouse button, or controller button in Frametop Input Settings | +### Pause for VR games + +Frametop pauses while a VR game runs, so the game gets the headset's CPU and GPU, and comes back a few seconds after the game ends. Paused, the screens hide and the desktop nearly stops drawing, but its windows stay open. Gaze mode's eye tracking stops, and so do remote desktop and hand tracking if they run. The mouse works as a plain mouse in SteamVR. + +| Do this | To get this | +| --- | --- | +| Click both thumbsticks together, twice | Pauses Frametop, or brings it back, in a game or not. You hear a short sound. The game sees the clicks too | +| Start a VR game | Frametop pauses, and comes back 5 seconds after the game ends. Bring it back during the game, and it stays on until that game ends | +| Map Pause/resume Frametop to a mouse button, key combination, or controller button | The same, from that button (Frametop Input Settings) | + +The Game optimization page of Frametop Input Settings turns the automatic pause off, changes the gesture, closes the desktop instead of hiding it (more for the game, but its windows close), and turns the sound off. From a terminal: `input/ft-pause on`, `off`, or `status`. + ### Gaze mode (experimental) -In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later). Turn it on and calibrate it on the Gaze page of Frametop Input Settings. +In gaze mode the pointer goes where you look, and the mouse or the keyboard does the last bit. The installer offers it (or run `gaze/run.sh install` later), and then our own eye tracker for it, which is more accurate than SteamVR's (or run `gaze/tracker/install.sh` later; it needs `sudo`). Gaze mode uses ours once it's installed, and SteamVR's until then. Turn it on and calibrate it on the Gaze page of Frametop Input Settings. | Do this | To get this | | --- | --- | @@ -123,15 +137,16 @@ This is an early release, tested on one Steam Frame (SteamOS 0.3.0 build 2026092 - A SteamOS or SteamVR update can break parts of it until Frametop catches up. After an update, run `cd ~/frametop && scripts/doctor.sh` in a terminal. It checks what Frametop needs from SteamOS, and says what changed since the versions you last marked as working and what to try. Once everything works, `scripts/doctor.sh --mark-good` records the versions. If something stops working, please report it. - The first install downloads 1–2 GB for the build container and compiles everything on the headset, which takes several minutes. -- During a VR game you can't show the screens with a controller button, because the game owns the buttons. Open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead. -- Flatscreen games aren't detected as games. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab). +- During a VR game, mapped controller buttons belong to the game, so they can't bring the screens up. The pause gesture still works: Frametop reads it without taking the thumbsticks from the game. With the automatic pause off, open the SteamVR dashboard, press Meta+Shift+H, or use a mapped mouse button instead. +- Flatscreen games aren't detected as games, so they don't pause Frametop by themselves: click both thumbsticks twice to pause it. If your controllers end up working the screens instead of the game, set Controllers on the screens to "Only with the SteamVR dashboard open" (Frametop Display Settings, Visibility & pins tab). - Typing follows your last click. A controller click on a panel other than the screens (the dashboard, a Steam app) doesn't move typing there; click it with the mouse, or click a screen to bring typing back. - The screens don't draw a mouse cursor of their own. The 3D mouse's dot or SteamVR's laser shows where you're pointing. - Profiles reopen apps, not what the apps had open. Tabs, files, and folders are left to each app's own restore. - Dragging something from one panel to another (a screen and a floating window) works, but the dragged item's icon doesn't show while the pointer is between panels. - Gaze mode is only as good as its calibration, and that depends on how the headset sits on your face. If the pointer lands off after you adjust the headset, run Quick check or Calibrate on the Gaze page of Frametop Input Settings. - On SteamVR's Settings page, the 3D mouse shows a laser beam and a larger hit dot, like a controller. SteamVR doesn't tell other programs where that page is (unlike Steam's pages, such as Library), so the mouse used to miss most of it: clicks went through to a desktop screen behind, and the dot disappeared. As a workaround, on that page only, the laser starts near your eye and SteamVR finds the page itself. See docs/design.md. -- Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart. +- Remote desktop over VNC (Frametop Remote Access in the app menu, or `./desktops.sh remote on`) needs Tailscale on the Frame. It shows the primary screen only. The app turns it on and off, shows the address, and shows, copies, or changes the VNC password. The password is made at random on the Frame and kept in `~/.config/frametop-remote` (only you can read it); VNC limits it to 8 characters, and the tailnet encrypts the connection. Turning it on in a desktop that started with it off takes a desktop restart. It costs almost nothing until a viewer connects; the picture then takes a few seconds to appear. +- The desktop has no blur behind panels and menus, and no window animations, so it leaves the headset's GPU to SteamVR. Turn them back on in the Frametop desktop's System Settings (Desktop Effects, and Animation speed under General Behavior); Frametop won't turn them off again. - Turning the displays off on a stand only turns their backlight off. SteamVR has no way for other programs to put the headset in standby, so tracking and rendering keep running, and the headset draws nearly its full power. ## Reporting problems diff --git a/display-settings/ft-layout-reset.desktop b/display-settings/ft-layout-reset.desktop index 29f7eab..741b2b2 100644 --- a/display-settings/ft-layout-reset.desktop +++ b/display-settings/ft-layout-reset.desktop @@ -3,7 +3,7 @@ Type=Application Name=Reset Screen Layout GenericName=Put the VR desktop's screens back in their layout Comment=Float the screens and arrange them in the layout from Frametop Display Settings -Exec=@REPO@/layout/ft-layout apply +Exec=@REPO@/layout/ft-layout-reset Icon=view-restore Categories=Settings; Keywords=display;screen;layout;arrange;reset;steamvr;frametop; diff --git a/display-settings/ft-screens-toggle.desktop b/display-settings/ft-screens-toggle.desktop index 39e880e..17869c6 100644 --- a/display-settings/ft-screens-toggle.desktop +++ b/display-settings/ft-screens-toggle.desktop @@ -3,7 +3,7 @@ Type=Application Name=Hide/Show Screens GenericName=Hide or show the VR desktop's screens Comment=Hide the screens (and SteamVR's laser) for a VR game; press again to bring them back -Exec=@REPO@/layout/ft-layout toggle +Exec=@REPO@/layout/ft-hide-show Icon=view-visible Categories=Settings; Keywords=display;screen;hide;show;steamvr;frametop; diff --git a/display-settings/main.qml b/display-settings/main.qml index 1aa820d..f49dac4 100644 --- a/display-settings/main.qml +++ b/display-settings/main.qml @@ -802,7 +802,7 @@ Kirigami.ApplicationWindow { Repeater { model: [ { value: "hide", text: "Hide them unless the SteamVR dashboard is open", help: "The game has the view to itself; open the dashboard (or press Meta+Shift+H) to see the screens." }, - { value: "visible", text: "Keep them visible over the game", help: "They float over the game as they are outside it." } + { value: "visible", text: "Keep them visible over the game", help: "They float over the game as they are outside it. Turn off Pause while a VR game runs in Frametop Input Settings (Game optimization), or Frametop pauses and hides them anyway." } ] delegate: ColumnLayout { required property var modelData diff --git a/display-settings/steam_settings.py b/display-settings/steam_settings.py index 1a115fb..172af94 100644 --- a/display-settings/steam_settings.py +++ b/display-settings/steam_settings.py @@ -4,117 +4,26 @@ Steam, not systemd, puts the Frame to sleep: after "When Plugged In and Idle -> (an hour by default) without input, even while it charges. That's a Steam client setting, `system_idle_suspend_ac_sec` (0 = never), with no file or command line to change it. Steam on the Frame runs with -cef-enable-debugging, so its UI's JavaScript context -(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080. There -`settingsStore.clientSettings` has the current values, and `SteamClient.Settings.SetSetting` -takes a change as a serialized CMsgClientSettings protobuf, which is what Steam's own -Settings -> Power page sends. Standard library only (a minimal WebSocket client). +(SharedJSContext) is reachable over the Chrome DevTools Protocol on 127.0.0.1:8080 +(steam/steamui.py). There `settingsStore.clientSettings` has the current values, and +`SteamClient.Settings.SetSetting` takes a change as a serialized CMsgClientSettings protobuf, +which is what Steam's own Settings -> Power page sends. """ -import base64 -import json import os -import socket -import struct -import urllib.request +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "steam")) +from steamui import SteamUnreachable, evaluate # noqa: E402,F401 (callers catch SteamUnreachable here) -CDP_PORT = 8080 # CMsgClientSettings field numbers (Steam's UI bundle maps the names to these). FIELDS = {"system_idle_suspend_ac_sec": 24004, "system_idle_suspend_battery_sec": 24003} -class SteamUnreachable(Exception): - pass - - -class _WebSocket: - def __init__(self, url, timeout=5): - host_port, path = url[len("ws://"):].split("/", 1) - host, port = host_port.rsplit(":", 1) - self.sock = socket.create_connection((host, int(port)), timeout=timeout) - key = base64.b64encode(os.urandom(16)).decode() - self.sock.sendall((f"GET /{path} HTTP/1.1\r\nHost: {host_port}\r\nUpgrade: websocket\r\n" - f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode()) - head = b"" - while b"\r\n\r\n" not in head: - chunk = self.sock.recv(4096) - if not chunk: - raise SteamUnreachable("Steam closed the connection") - head += chunk - if b" 101 " not in head.split(b"\r\n", 1)[0]: - raise SteamUnreachable(head.split(b"\r\n", 1)[0].decode(errors="replace")) - self.buf = head.split(b"\r\n\r\n", 1)[1] - - def send(self, text): - data, mask = text.encode(), os.urandom(4) - n = len(data) - if n < 126: - head = struct.pack(">BB", 0x81, 0x80 | n) - elif n < 65536: - head = struct.pack(">BBH", 0x81, 0x80 | 126, n) - else: - head = struct.pack(">BBQ", 0x81, 0x80 | 127, n) - self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(data))) - - def _take(self, n): - while len(self.buf) < n: - chunk = self.sock.recv(65536) - if not chunk: - raise SteamUnreachable("Steam closed the connection") - self.buf += chunk - out, self.buf = self.buf[:n], self.buf[n:] - return out - - def recv(self): - message = b"" - while True: - b0, b1 = self._take(2) - n = b1 & 0x7F - if n == 126: - n = struct.unpack(">H", self._take(2))[0] - elif n == 127: - n = struct.unpack(">Q", self._take(8))[0] - message += self._take(n) - if b0 & 0x80: - return message.decode(errors="replace") - - def close(self): - self.sock.close() - - -def _evaluate(expression): - """Runs JavaScript in Steam's SharedJSContext and returns its (awaited) value.""" - try: - with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}/json", timeout=3) as r: - targets = json.load(r) - except OSError as e: - raise SteamUnreachable(f"Steam isn't reachable on port {CDP_PORT} ({e})") from e - url = next((t["webSocketDebuggerUrl"] for t in targets if t.get("title") == "SharedJSContext"), None) - if not url: - raise SteamUnreachable("Steam's UI isn't running") - try: - ws = _WebSocket(url) - try: - ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate", - "params": {"expression": expression, "awaitPromise": True, "returnByValue": True}})) - while True: - reply = json.loads(ws.recv()) - if reply.get("id") == 1: - break - finally: - ws.close() - except OSError as e: - raise SteamUnreachable(str(e)) from e - result = reply.get("result", {}) - if "exceptionDetails" in result: - details = result["exceptionDetails"] - raise SteamUnreachable(details.get("exception", {}).get("description") or details.get("text", "error")) - return result.get("result", {}).get("value") - - def sleep_settings(): """{"ac": seconds, "battery": seconds}: when Steam puts the Frame to sleep without input, plugged in and on battery (0 = never).""" - value = _evaluate("(() => { const c = settingsStore.clientSettings; " - "return {ac: c.system_idle_suspend_ac_sec, battery: c.system_idle_suspend_battery_sec}; })()") + value = evaluate("(() => { const c = settingsStore.clientSettings; " + "return {ac: c.system_idle_suspend_ac_sec, battery: c.system_idle_suspend_battery_sec}; })()") if not isinstance(value, dict) or not all(isinstance(value.get(k), int) for k in ("ac", "battery")): raise SteamUnreachable("Steam's settings don't have the sleep timeouts") return value @@ -125,7 +34,7 @@ def set_sleep_setting(name, seconds): field, seconds = FIELDS[name], int(seconds) if seconds < 0: raise ValueError("seconds must be 0 (never) or more") - ok = _evaluate(f"""(async () => {{ + ok = evaluate(f"""(async () => {{ const bytes = []; const varint = n => {{ while (n > 127) {{ bytes.push((n & 127) | 128); n = Math.floor(n / 128); }} bytes.push(n); }}; varint({field} * 8); varint({seconds}); diff --git a/docs/controller-click-stability.md b/docs/controller-click-stability.md new file mode 100644 index 0000000..a7622ad --- /dev/null +++ b/docs/controller-click-stability.md @@ -0,0 +1,34 @@ +# Controller desktop click stability + +Trigger presses reach KDE immediately, but controller motion within 32 logical +pixels of the press stays at that position until release. Releasing without a motion outside this +zone delivers the click at the original position, even if the hand moved during +release. Moving outside the zone begins a normal drag immediately; returning to +the zone does not turn it back into a click. There is no hold-duration timer. + +This filters overlay pointer content events on desktop monitors only, and only +presses from hand controllers start it. The 3D mouse (whose laser comes from the +`ft_pointer` virtual controller), SteamVR UI, separate screen grab bars and +floating-app title-bar carrying are unaffected. Multi-button gestures keep their +existing behavior. A motion onto another desktop monitor starts a drag; +cross-monitor motion is not stabilized. + +CLI (runtime preferences, reset to 32 on desktop restart): + +```sh +input/ft-clickctl status +input/ft-clickctl threshold 32 +input/ft-clickctl threshold 0 # disable without a restart +``` + +Thresholds are 0–64 logical pixels, normalized to each panel's KDE scale. +Status reports held state, suppressed motions, stabilized clicks and drags. +Changing the threshold while a controller button is held is refused. + +This is a separate contribution from desktop mouse/controller ownership. Its +hardware validation must check small controls, intentional text selection, +long presses, cross-monitor dragging and simultaneous mouse use. The existing +renderer laser remains tracked; this change stabilizes desktop input rather +than smoothing the visual laser. The default was 8 at first. That's about 0.2° on a 3.4 m wide 3440-pixel screen 2 m away, and clicking took a very still hand, so it's 32 (about 0.9°) since 2026-10-03. + +Run `scripts/test-controller-click.sh` for the isolated gesture-state tests. diff --git a/docs/design.md b/docs/design.md index 062cffc..1e6bb16 100644 --- a/docs/design.md +++ b/docs/design.md @@ -38,7 +38,7 @@ The curved layout chains screens edge to edge, like monitors on a desk: the midd A resize handle has to be able to shrink a screen from any direction, so the dragged corner follows the laser along the screen's diagonal rather than taking the larger of its horizontal and vertical reach. Pushing and pulling a carried screen moves it along the line from your head, because the 3D mouse's virtual controller sits just in front of the bar, below the screen's centre, so the line from the device points mostly upward. -Wherever ft-screens needs to know where a laser points (showing the controls, the resize tab, the roll knob), it uses the laser's own pose, the render model's `tip` component, rather than the controller's pose. On the Frame's controllers the tip points 40° below the pose's forward axis, so rays from the pose missed what the laser was actually on. The 3D mouse's virtual controller has no tip, and its laser runs along its pose. +Wherever ft-screens needs to know where a laser points (showing the controls, the resize tab, the roll knob), it uses the laser's own pose, the render model's `tip` component, rather than the controller's pose. On the Frame's controllers the tip points 40° below the pose's forward axis, so rays from the pose missed what the laser was actually on. The 3D mouse's virtual controller has no tip, and its laser runs along its pose. ft-screens reads the tip with `GetComponentState`: `GetComponentStateForDevicePath` without an input source handle fails for every component while a VR game runs, so in games the rays came from the pose, 40° too high. `ComputeOverlayIntersection` ignores `SetOverlayIntersectionMask`, and a control can't be allowed to cover part of its screen, so the resize tab sits entirely outside the corner. @@ -62,6 +62,8 @@ A profile's screen part is the custom arrangement under a name: each screen's po `IVRApplications::GetCurrentSceneProcessId()` is 0 when no game is running (the Frame's home environment isn't a scene app) and the game's process ID while one is. ft-screens checks it twice a second, turns the flag off while a game runs, and by default hides the screens unless the dashboard is open. Flatscreen games run inside Steam's gamescope overlay and aren't scene apps, which is why "only with the dashboard open" is offered as a controller setting. +In a game, Frametop's panels work like SteamVR's own floating windows: point a controller at one and its laser comes on, point away and the game has the controllers again. ft-screens turns the flag on for a panel while a hand controller's laser pose meets it, its controls, or a floating window's popups. It finds that from the poses it already reads to show the controls, so SteamVR's laser doesn't have to be on first. Leaving takes a margin two control-sizes wide and 0.3 s, a drag or a held button keeps the flag on, and the keyboard, a single overlay, uses SteamVR's `ComputeOverlayIntersection`. The 3D mouse doesn't need any of this: it has its own laser mode. + ## Floating windows [floating-windows.md](floating-windows.md) describes the feature and its parts. Drag and drop and the clipboard only work between windows of one compositor, so a floating window stays a KWin window and gets a KWin output of its own: one of the spare outputs KWin opens after the screens, shown by ft-screens as a panel cropped to the window. What follows is how KWin 6.2.5 behaves underneath that, from its source (`src/backends/wayland/`) and from trying it on the Frametop desktop. @@ -116,9 +118,9 @@ The driver starts disconnected, because holding the right-hand role while SteamV ### The cursor -Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either. +Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either. Both tests ask SteamVR about every visible overlay, so a frame where nothing moved (the mouse, the anchor, the eye by more than 5 mm, which overlays show) reuses the last result, for up to 100 ms, since overlays can also move on their own. The dots' overlay settings go to SteamVR only when they change, and the pose goes to the driver, which keeps the last one, only when the laser would land 0.1 mm or more elsewhere, and at least every 100 ms. -OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately. +OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately. Each run is a shell and a new SteamVR client, about 30 ms of CPU, and it ran every second while the pointer was awake, which in gaze mode is all the time. Now it runs every 20 seconds, and at once when the pointer wakes, when the dashboard opens or closes, when a game starts or ends, and when a left click hits nothing (a panel that came up since). An overlay already on the list showing or hiding needs no new list: the helper checks the visibility of the ones it knows every 50 ms. The laser starts partway along your line of sight to the cursor rather than at your eye. SteamVR sizes its hit dot by distance from the laser's origin, and a laser from the eye still shows a beam in each eye. Starting it close to the target makes the beam and the dot tiny, while `POINTER_ORIGIN_MARGIN` keeps the origin in front of the small window controls, which float a few centimetres in front of their panels. The helper's own white dot is the visible cursor. In empty space it's an interactive overlay that the laser lands on, so SteamVR never draws a laser into nothing. @@ -141,7 +143,7 @@ Replacing a loaded driver's files, as re-running the installer used to do, leave The dashboard follows whichever device summoned it or last pressed its trigger. Frametop adds "last used wins": moving a real controller releases the pointer, and the next mouse movement takes the laser back. Moving means faster than 0.35 m/s or 2 rad/s (both times `POINTER_CONTROLLER_PICKUP`, 1 by default) for 100 ms in a row, while the controller is tracked normally. A single sample over the limit used to be enough, and controllers resting on a desk took the laser back on a knock or a tracking jump while the mouse was in use. Small movements don't count; waking needs `POINTER_WAKE_COUNTS` of mouse motion within a second, so desk jitter doesn't steal the laser. While the pointer is awake, a tiny transparent overlay with `MakeOverlaysInteractiveIfVisible` keeps SteamVR's laser mouse on, since otherwise the first click would only switch the laser on. -When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off. +When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off. With the pointer off it also stops running its loop every 8 ms, about 116 wakeups a second for nothing: it waits up to 250 ms for a command on its socket (20 ms while it reads mapped controller buttons, which SteamVR input only offers by polling), and leaves the overlay lookups until the pointer wakes. ### Moving floating windows @@ -153,6 +155,8 @@ SteamVR opens every input device only when it starts. When a Bluetooth mouse sle Keyboards aren't grabbed by default, because a grabbed keyboard's keys went into a virtual keyboard nothing typed from; the relay forwards them to ft-screens instead. +The relay never waits on the pointer helper. Its socket to the helper used to block, so when the helper stalled (a layout placement or `grabprobe` holds it for seconds, and the gaze service fills its socket 90 times a second meanwhile), the whole relay stopped with it: keyboards, the volume keys, and pausing. Now what the helper doesn't take waits in order and goes out on the next loops. Mouse moves add up into one while they wait, and a scroll notch is dropped, since scrolling seconds late is no use; presses and releases are kept, so no button stays down. Mouse motion goes to the helper at most every 4 ms, rather than once per report, which from a 1000 Hz mouse was 1000 datagrams a second to a helper that runs every 8 ms; a button sends the motion before it first, so the click lands where the pointer was. + An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's `InputStealer`, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's `OverlayFocusChanged` (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. So with `SHARE_KEYS=1` the relay sends a grabbed keyboard's keys to `@frametop_keys` as datagrams (`key `). It's off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first gets every key typed into the desktop, passwords included. A listener should accept only its own user (`SO_PASSCRED`) and skip any keyboard of its own that the relay grabs too. Volume keys must never reach gamescope. With the openvr backend, gamescope sends volume up and down to Steam by moving keyboard focus to Steam for the key and then back to the previously focused surface. When nothing had focus, the one it moves back to is null, and wlroots aborts on a null focus surface (`wlr_seat_keyboard_notify_enter: Assertion 'surface' failed`), which ends the whole VR session. Keyboard focus is often empty while you work in VR, so one press of the headset's volume button could take everything down. gamescope reads the headset's buttons and every keyboard itself (`InputStealer`), as do SteamVR's processes, so the relay has to stop volume keys at the device. Grabbing `gpio-keys` would also take the headset's click button, so the relay remaps the volume entries in each device's keymap (`EVIOCSKEYCODE`) and handles the stand-in codes itself. That fix covers every device at once, including keyboards that aren't grabbed. @@ -187,7 +191,15 @@ Flatpak apps need `XDG_DATA_DIRS` to include Flatpak's exports, or Plasma opens The private runtime directory also moves the session's document portal to `$XDG_RUNTIME_DIR/frametop/doc`, and that broke saving and uploading in Flatpak apps. The file picker (xdg-desktop-portal 1.18.4 on SteamOS) gives a sandboxed app the host path of the file it picked, `/run/user/1000/frametop/doc/ID/NAME`. Inside the sandbox the portal is at `/run/flatpak/doc`, and `/run/user/1000` is a private per-app folder (`.flatpak/APP/xdg-run` in the runtime directory). So Brave created the missing folder there, "finished" the download into it, and the file vanished when the session cleaned up. The session script now links that path to `/run/flatpak/doc` in each installed app's folder before Plasma starts. Upstream xdg-desktop-portal fixed this after 1.22.1 (commit `69ba5e1`) by handing Flatpak apps `/run/flatpak/doc` paths, after which the links go unused. -A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. +A podman container's monitor process (conmon) stays in the cgroup of whatever started the container, and `distrobox enter` starts it on demand. When a Frametop service happened to start the `dev` container, stopping that service stopped the container and everything in it, including the desktop's compositor. `scripts/container-up.sh` starts the container in a systemd scope of its own before anything enters it. It then waits for distrobox-init to log `container_setup_done`, as `distrobox enter` does only for containers it starts itself. A new container's first start takes a minute or more (it installs distrobox's dependencies and sets up passwordless sudo), and an install that entered right away met a sudo password prompt with no terminal to answer it ([#9](https://github.com/DeeJanuz/frametop/issues/9)). + +KWin renders with OpenGL through zink on Turnip, Vulkan on the same GPU vrcompositor needs to hit its frame time, and on the Frame that costs CPU too. The nested session started with KWin's defaults: blur and background contrast on (no `[Plugins]` group in its kwinrc) and animations at full length. Blur re-renders what's behind every translucent panel and menu each time it changes, and every animated frame is one more frame for KWin and ft-screens to draw and send. They're off by default in the Frametop desktop. The session script writes them before KWin starts, only where the desktop's own config has no value, once: System Settings deletes a setting put back to KDE's default rather than writing it, so without the marker in `frametoprc` a user who turned blur back on would lose it at the next start. The effect ids (`blur`, `contrast`) are the ones built into KWin 6.2.5 on SteamOS; KWin reads `Enabled` from `[Plugins]`. + +The nested session also runs the system's XDG autostart entries, being a KDE session. Discover's update notifier started `plasma-discover --mode update` in it (520 to 620 MB resident and about 9% of a core, plus `flatpak-system-helper` and AppStream downloads), and IBus started a daemon, the kimpanel panel and its GTK extension that nothing can use: KWin hands text input to the one input method it starts (`ft-textinput`), and the session drops `QT_IM_MODULE`, `GTK_IM_MODULE` and `XMODIFIERS`. The session hides both for this desktop only, with `Hidden=true` copies in its own autostart folder. The geoclue demo agent stays: it's what answers apps' location requests to Geoclue outside GNOME, and it costs nothing while idle. Orca's entry only starts in GNOME-family desktops. + +Plasma 6.2.5 keeps each panel on a screen number (`lastScreen` in `plasma-org.kde.plasma.desktop-appletsrc`), and the numbers rank the enabled outputs by priority, so 0 is the primary screen. A panel whose number is past the screen count gets no view, and Plasma never moves it: the remap it runs at every start only moves a panel whose number has no desktop, and this desktop keeps a desktop for every output it has seen, spares included. So the taskbar was lost when the number of screens went down, and once it was found saved on a spare output, number 8 of a desktop with three screens ([#18](https://github.com/DeeJanuz/frametop/issues/18)). Before Plasma starts, the session runs `session/fix-panels.py`, which moves any panel numbered past the screen count, with its system tray's containment, to screen 0, keeping its widgets and settings. A panel stays put when screen 0 already has one on that edge, and comes back by itself if the screens do. The file is backed up to `.ft-bak` first. Plasma's scripting can't do this while it runs (`panel.screen` is read-only in 6.2.5), so a lost taskbar comes back at the desktop's next start. `scripts/doctor.sh` and `scripts/report.sh` list the panels and their screens. + +Remote desktop is a chain (krdp, then FreeRDP inside Xvnc) because nothing on SteamOS serves KWin over VNC directly. Kept connected all the time, it cost about a core with nobody watching: krdpserver 55 to 78% (it encodes H.264 in software with openh264: VA-API finds no driver for the Frame's GPU in the container), FreeRDP 16 to 27%, Xvnc 6 to 11%, and the bridge's layout check every 5 seconds another 4%. krdp creates its screencast session per RDP connection and drops it when the connection closes (`SessionController::onNewConnection` in krdp 6.7), so an idle krdpserver costs nothing and can stay up; only the RDP connection has to go. The bridge connects FreeRDP when a VNC client appears and disconnects 45 seconds after the last one leaves. Xvnc has no hook for its clients, so the bridge counts established connections to its port with `ss`, woken early by Xvnc's log output; looking with `ss` once a second cost about 0.9% of a core in bash, against about 0.1% this way. `Xvnc -inetd` from a systemd socket would start a server per connection and lose sharing between viewers. The layout check (`ft-layout remote-view`, which scans `/proc` for plasmashell and runs `kscreen-doctor -j`) now runs only while FreeRDP runs, and then only after `kwinoutputconfig.json` or `frametop-layout.json` changes, with one check a minute in case a change touched neither. Program names stay within 15 characters, because Linux truncates process names there and the scripts find programs with `pgrep -x` and `pkill -x`. That's why the prefix is `ft-`. @@ -201,6 +213,15 @@ Movement is judged within 10-second windows. On the mount, the head pose jittere Staying awake while charging uses Steam's own setting rather than a logind sleep inhibitor. Steam suspends with `dbus-send ... login1.Manager.Suspend boolean:true`, and a block inhibitor does stop that (`CanSuspend` answers "challenge" while one is held), but it stops the power button too. `system_idle_suspend_ac_sec` is field 24004 of Steam's CMsgClientSettings. In Steam's SharedJSContext, reachable over CDP on port 8080 because Steam runs with `-cef-enable-debugging`, `SteamClient.Settings.SetSetting` takes a change as a base64 protobuf, the way Steam's Power page sends it (0 is never), and `settingsStore.clientSettings` has the current values. +## Pausing for VR games + +Hiding the screens during a game kept them out of view, but Frametop kept using the headset. Measured on 2026-10-02 with gaze mode off and no game running, in shares of one core: our eye tracker (ft-eyes) about 60%, ft-eyegrab, ft-gaze and ft-gazed about 3 to 4% each; remote desktop (krdpserver, FreeRDP, Xvnc) about 2 cores while it ran; KWin about 13%, ft-screens about 4%. The gaze service ran at full rate whether gaze mode was on or not; now it idles while the gaze isn't used (gaze/README.md), and pausing stops it outright. Reading SteamVR's eye tracking 90 times a second also made it restart every 10 to 13 seconds during Beat Saber, and each restart took input focus from the game, which paused it (PR #13; since then ft-gaze skips SteamVR's gaze action during games, but our own tracker kept running). So pausing stops what costs the most and leaves windows where they are. + +- A hidden screen still cost as much as a visible one. ft-screens sent every committed screen its frame callback at 90 Hz whether its overlay showed or not (since then, a hidden screen always gets one a second; see the frame rates in reference.md), so KWin kept drawing, and its apps with it. Paused, ft-screens sends the callbacks once a second. A Wayland client draws again only after its last frame's callback, so KWin's output stalls, KWin's own clients stop getting theirs, and the whole desktop idles, without anything losing its connection. A second's pace, rather than none, keeps any client that waits on a callback from waiting forever. Stopping KWin or the apps with SIGSTOP would free the same, but a Wayland peer that stops reading overflows the other side's 4 KB socket buffer, which ends the connection: that's how the live desktop died once when its KWin stalled (`Data too big for buffer`). They also sit in different cgroups (KWin under steam.service when the VR launcher starts it, ft-screens in the dev container's), so no single freeze stops them together. +- The relay does the pausing because it's the one part that always runs, and the pointer helper keeps running because stopping it leaves its virtual controller connected with its last pose (the driver has no staleness timeout), maybe holding a hand role, with the 3D mouse dead. Releasing it does the job. The helper already checks for a scene app twice a second, so it's what tells the relay a game started. +- The gesture has to work during a game, but SteamVR input reaches only the app with input focus, and an overlay with global input (`steamvr/globalActionSetPriority`) takes the buttons it binds from the game. vrserver's web socket on 127.0.0.1:27062, which its controller binding page uses for the live view, reports every controller component whatever has focus, and reading it takes nothing. The game sees the clicks too, so the default is a gesture games hardly use: both thumbsticks, together, twice. "Together" means within 0.3 seconds of each other, so a stick held down to sprint while the other clicks doesn't count. The stream is about 160 messages a second, nearly all capacitive sensing, so the reader parses only the few that mention a gesture's button. A controller's root path changes while the 3D mouse holds its hand role (`/devices/cv/` instead of `/user/hand/right`), so the reader looks the controllers up again (an HTTP request to vrserver): when the relay's 3D mouse connects or lets go, when a message comes from a device it doesn't know, and every 30 seconds. It used to be every 3 seconds. +- Resuming starts remote desktop through `systemd-run --scope`: started straight from the relay, it would join the relay's cgroup and end with the next relay restart. + ## SteamOS updates On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-steamvr-rel` package), next to KWin, gamescope, and the kernel, so every SteamOS update can bring a new SteamVR too. Frametop survives updates: it lives in the home folder and the `dev` container, the Bluetooth fixes are in `/etc`, which SteamOS keeps across updates, and nothing goes into `/usr`. What an update can break is what Frametop uses from the image. The public OpenVR API is versioned and stays put. The rest is less certain: `IVRIPCResourceManagerClient`, which is newer than the header SteamVR ships; the text `vrcmd --overlays` prints; the eye tracker's shared memory layout; XRService's camera buffers; KWin's nested backend; and behavior Frametop works around, such as the SteamVR Settings page that `ComputeOverlayIntersection` can't find or the scale KWin's nested backend doesn't undo. @@ -217,6 +238,5 @@ On the Frame, SteamVR is part of the OS image (`/opt/steamvr`, the `deckard-stea - A controller button that shows the screens during a game. Games own the controllers, so this needs SteamVR input actions for ft-screens. - Drawing KWin's cursor on the screens. -- Plasma can lose its panels when the number of screens goes down, because they're saved against a screen that no longer exists. Removing `plasma-org.kde.plasma.desktop-appletsrc` and `plasmashellrc` from `~/.config/frametop` brings the default panels back. - Frame pacing and GPU cost with several busy screens haven't been measured. - Real standby on a stand, with rendering and tracking paused, not just the backlight off. SteamVR has no call for it, and its activity level follows the proximity sensor. diff --git a/docs/profiles.md b/docs/profiles.md index d005ca2..61c7866 100644 --- a/docs/profiles.md +++ b/docs/profiles.md @@ -46,8 +46,8 @@ So a "Work" profile can put three screens around you with a browser, two termina ## How it works -- **Capture** (`ft-layout save NAME`, and Save as profile… in Display Settings). ft-layout captures the screens as before, then asks ft-floatd for the windows (`windows` on @frametop_float). ft-floatd has the KWin script report every window as it is now (`report-all`), then answers with every normal window: its desktop file name, the screen it's on, its rectangle there, and whether it's maximized. For floating windows it gives their panel's place, their size in pixels, and their scale. Windows with no desktop file name are kept by their process's command line (`/proc//cmdline`) and window class. Windows of Plasma itself, the Frametop settings apps, and dialogs aren't recorded. If ft-floatd doesn't answer, the profile keeps the apps it had. -- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name: single-instance and D-Bus-activated apps open their windows from a process that was already running. +- **Capture** (`ft-layout save NAME`, and Save as profile… in Display Settings). ft-layout captures the screens as before, then asks ft-floatd for the windows (`windows` on @frametop_float). ft-floatd has the KWin script report every window as it is now (`report-all`), then answers with every normal window: its desktop file name, the screen it's on, its rectangle there, and whether it's maximized. For floating windows it gives their panel's place, their size in pixels, and their scale. A window whose app id has no desktop file (a Flatpak app's X11 window can give its own: RustDesk's says `com.carriez.flutter_hbb`, its desktop file is `com.rustdesk.RustDesk`) is kept by the desktop file whose `StartupWMClass` names its window class. Windows with no desktop file name are kept by their process's command line (`/proc//cmdline`) and window class. Windows of Plasma itself, the Frametop settings apps, and dialogs aren't recorded. If ft-floatd doesn't answer, the profile keeps the apps it had. +- **Apply** (`ft-layout use NAME`, Open profile in Display Settings). ft-layout makes the profile's hidden screens the screens' own setting, arranges the screens (which hides and shows them: ft-screens' `conceal` and `reveal`), then has ft-floatd open the apps (`profile NAME`; ft-floatd reads the windows from the layout file). If the screens can't be arranged, for example with the headset off and no head pose, the apps still open: the screens stay where they are, the profile's hidden screens still hide, and floating windows go relative to the screens wherever they are. ft-floatd goes through the entries app by app. It claims windows of that app already open (oldest first, each claimed once), and moves each to its entry's place: onto its screen at its rect (or maximized), or floating at its pose. For the entries left over, it launches the app once (`ft-float launch`, the same path as Launch as Standalone) and waits up to 30 seconds for its first window. Each window that shows up goes to the next entry's place. Once the first window has been up for 3 seconds (time for an app that restores its own windows to show them), ft-floatd launches the app again for each entry still waiting, and waits up to 30 seconds more. New windows are matched to the launch by process (or a child of it), or by desktop file name (found the same way as at capture): single-instance and D-Bus-activated apps open their windows from a process that was already running. - **Default at start.** The session script runs `ft-layout start --wait 90`. That opens the profile in `FT_PROFILE` or `default_profile` (screens, then the apps once ft-floatd is up), or runs `apply --wait` if there's none. Start in profile on the Layout & profiles page sets `default_profile` (`ft-layout default NAME|none`). Plasma's session restore is turned off in the session (`ksmserverrc`: `loginMode=emptySession`). - **Launcher entries.** Each profile gets `~/.local/share/applications/frametop-profile-.desktop` ("Frametop: Work"), written when it's saved and removed when it's deleted. They show in SteamVR's Launch a program list, the Application Launcher, and KRunner. Running one (`ft-layout open NAME`) switches to that profile if the desktop runs. Otherwise it starts the desktop with `FT_PROFILE` set (`systemd-run`, as `desktops.sh start` does), which overrides `default_profile` for that start. That needs SteamVR to be running. - **The action.** `profile:NAME` in the input relay (it runs `ft-layout use NAME`) for key combinations, mouse buttons, and controller buttons, with or without pointer mode. Input Settings lists one "Open profile NAME" action per profile. diff --git a/docs/reference.md b/docs/reference.md index 7524850..e84b47c 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -20,6 +20,16 @@ When the VR launcher starts the desktop, it inherits the Steam client's environm The nested session also starts an AT-SPI accessibility registry through `session/ft-atspi` in Plasma's autostart. It discovers the bus from this session, ignores an inherited host accessibility address, and leaves an existing registry alone. Accessibility errors do not stop the desktop. This supplies the registry infrastructure for apps that expose AT-SPI trees; it does not enable gaze snapping or force Chromium/Electron accessibility. After an approved desktop restart, an AT-SPI-aware app should be visible on the nested bus. See [design.md](design.md#nested-accessibility) for native activation, fallback lifecycle, and the isolated test command. +KWin's blur and background contrast effects and its animations are off in this desktop, because KWin draws on the headset's GPU, which SteamVR needs. The session script turns them off once, the first time it starts (it leaves a setting you already have alone, and marks it done in `~/.config/frametop/frametoprc`), so turning them back on sticks. In the Frametop desktop, System Settings → Window Management → Desktop Effects has Blur and Background Contrast, and General Behavior has Animation speed. Or from a terminal, then restart the desktop: + +``` +kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key blurEnabled true +kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key contrastEnabled true +kwriteconfig6 --file ~/.config/frametop/kdeglobals --group KDE --key AnimationDurationFactor 1 +``` + +Two of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, and IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`), and skips a name you already have a file for. Delete a copy to start that program again. + Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in `~/.config/frametop-layout.json`. The backend, remote desktop, and pointer settings are in `~/.config/frametop.conf`; `session/frametop.conf.example` lists every key. Restarting the desktop closes its windows. Before the unit stops, `session/keep-apps.sh` moves every program started in the desktop into a systemd scope of its own, so background work such as servers, tmux, and builds keeps running. An app that shuts down its own helper processes when its window closes will still lose them; run that kind of work outside the desktop, for example as a systemd user service. @@ -30,12 +40,13 @@ Restarting the desktop closes its windows. Before the unit stops, `session/keep- Each KWin window is one screen. ft-screens sets its size with an `xdg_toplevel` configure and KWin resizes the output to match, live. Frames arrive as DMA-BUFs and go to SteamVR through OpenVR's `IVRIPCResourceManagerClient::ImportDmabuf`, with no copy and no size limit. -Every screen is an overlay named `frametop.screen.N` with four controls: +Every screen is an overlay named `frametop.screen.N` with five controls: - `.bar` moves the screen. Drag it with any laser or with the 3D mouse, whose right-drag tilts. Scrolling while you drag pushes the screen away or pulls it closer, along the line from your head. - `.curve` bends the screen into a cylinder around you, using your current distance as the radius, or makes it flat again. - `.roll` rolls the screen when you drag it sideways, like a knob. It snaps level within 2.5°, and scrolling on it turns 5° per notch. - `.resize`, the tab on the bottom right corner, sets the width. Screens go down to 15 cm wide. +- `.reset`, left of the bar, puts every screen back in its layout around where you are now, like Meta+Shift+R (`ft-layout apply`). The controls are sized from both the screen's width and its distance from you, follow the surface of a curved screen, and stay invisible until a laser or the 3D mouse's cursor lands on one or comes within about 1.5 times a button's size of it. While invisible they're still there, fully transparent, so SteamVR's laser can find them. They're translucent until a laser is on them, like SteamVR's own window controls. @@ -53,7 +64,7 @@ The Visibility & pins tab of Frametop Display Settings decides when the screens In the last three modes the hotkey shows the screens anyway. A screen can also be hidden on its own (Screens shown on the same tab, or `ft-layout hide N`): it stays hidden whatever the mode or the hotkey says, until it's shown again there. Windows on it stay put, and a new window that would open on it floats instead (ft-floatd). Profiles use this to show only some screens. Two more settings on the same tab cover VR games, which ft-screens detects as SteamVR scene apps: - During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up. -- Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps. +- Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. Pointing a controller at a screen, a floating window, or the keyboard still turns its laser on, like SteamVR's own floating windows, and pointing away gives the game the controllers back. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps. Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay, from pass-through keyboards and any key a pointer device passes through. Typing follows your last click: after a click on a screen it goes to the desktop, even with the SteamVR dashboard open, and after a mouse click on any other panel (the dashboard, Steam, an app like Spotify) it goes there instead. While it goes to the desktop, the relay grabs pass-through keyboards so gamescope, which reads every keyboard itself, doesn't type them into the Steam app too. A program that watches every keyboard for a hotkey loses a grabbed one; with `SHARE_KEYS=1` in `~/.config/frametop.conf`, their keys also go to `@frametop_keys` for it. That's off by default, since any local process that binds the name first would get everything typed into the desktop. Hidden screens don't take typing. @@ -68,12 +79,15 @@ place N x y z yaw pitch roll width N metres curve N radius|on|off pin N|all left|right|head [matrix] unpin N|all size N w h get N screens head state key code value scale N s vrkeyboard show|hide|toggle|close visibility always|dashboard|gesture|toggle wrist degrees gesture left|right degrees -hide | show | toggle controllers always|outside_games|dashboard ingames hide|visible +hide | show | toggle controllers always|outside_games|dashboard ingames hide|visible pause on|off|state conceal N|all reveal N|all concealed cutouts on|off|state cutouts predict on|off cutouts lead ms float N mpp x y w h title unfloat N pose N matrix sub N k x y w h | sub N k off minimized N 0|1 carry N +rates focused in_view hidden rates? watch seconds phase ms ``` -`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar. +Each screen draws at a frame rate for how much of it you see. KWin draws a screen only after ft-screens gives it a frame callback, and its apps wait for theirs, so the rate of callbacks is the screen's frame rate, for KWin and the apps on it alike. A screen is focused while you look at it (within 12 degrees of where your head points), while a laser or the mouse is on it or was in the last 1.5 seconds, while it's carried, and while you type on it; it gets every display frame. The rest of what you can see (within 60 degrees) gets 15 frames a second, and a hidden screen, one behind you, and everything while paused get one a second. A level goes up at once and comes down after a moment (1.5 s from focused, 0.5 s from in view). A video, or anything moving over a large part of a screen (6% or more of it, redrawn on 8 commits in a row, 10 or more a second), keeps every frame while in view. A floating window is a screen of its own here. Nothing that stands still costs anything at any rate: KWin sends a frame only when something on the screen changed. `rates F V H` sets the three rates in Hz (0: every display frame; default `0 15 1`, also `ft-screens --rates 0,15,1`), and `rates?` shows them, the display's rate, and for each screen its level, its milliseconds between frames, and whether it counts as a video. `watch S` gives every screen full rate for S seconds: remote desktop renews it while a VNC client is connected, since a viewer sees what KWin draws. The ticks (SteamVR events and the callbacks) come once per display frame, 1 ms after the vsync (`phase ms` changes that, for tuning), in step with the display rather than on a timer that drifted through the frame. + +`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `pause on` (from the input relay, when Frametop pauses for a VR game) hides every screen and floating window whatever else says, and slows the desktop down; `pause off` undoes it. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar. ## Input relay @@ -87,6 +101,8 @@ The relay also owns the volume keys, on every device that has them, the headset' desktops.sh relay install # enable it (starts with the next reboot or SteamVR start) desktops.sh relay status | log | uninstall input/input-relay.py --no-grab # try it without taking devices from SteamVR +input/test/keys-test.py # key combinations and modifier taps, against fake devices (safe next to the live relay) +steam/ft-steam menu # what Open Steam menu does; ft-steam check: Steam's UI still has the calls ``` The first time, the relay has to start before SteamVR, so reboot or restart SteamVR after installing it. After that it's safe to restart on its own: systemd keeps the virtual devices open in its file descriptor store (`FileDescriptorStorePreserve=yes`), so SteamVR keeps the same devices. @@ -101,7 +117,7 @@ A mouse drives SteamVR the way a controller's laser does, but shows up as a smal Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again. -To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button (or a Meta tap, with `META_DASHBOARD=1`) wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant. +To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant. Open Steam menu / close dashboard (`steam_menu`) needs no pointer: `steam/ft-steam menu` asks Steam's UI, over its debugging port (`steam/steamui.py`), to show its dashboard overlay and focus the Steam frame's menu, or to hide the dashboard if it's up. ``` pointer/driver/build.sh && pointer/driver/install.sh install # then restart SteamVR @@ -110,16 +126,17 @@ pointer/helper/run.sh status | log | restart pointer/driver/install.sh probe # devices, hand roles, who owns the dashboard pointer ``` -The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes). +The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes). ## Frametop Input Settings -A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has eight pages: +A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has nine pages: -- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, where a Meta tap toggles the dashboard if `META_DASHBOARD=1` is in `~/.config/frametop.conf`), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device. -- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep. +- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device. +- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, pause or resume Frametop ([Pausing for VR games](#pausing-for-vr-games)), Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep. - Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`. -- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, on any keyboard, to any action but passing a key through or nothing. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are Meta+J (gaze left click), Meta+K (gaze right click), and Meta+Shift+F (float window in VR or put it back); remove them or add others there. The combination's last key isn't typed, and the modifiers still reach the app. They're saved as `key_bindings` in the same file. +- Game optimization has the pause for VR games: its state with Pause now or Resume, whether VR games pause Frametop by themselves, the controller gesture (one or two buttons, pressed once or twice; one button always takes two presses), what happens to the desktop, and the sound. They're saved as `pause_auto`, `pause_gesture`, `pause_desktop`, and `pause_sound` in `~/.config/frametop-input.json`. See [Pausing for VR games](#pausing-for-vr-games). +- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), Meta+Shift+F (float window in VR or put it back), and Meta+Alt+Tab and Meta+Alt+Shift+Tab (spin the panels: every screen and floating window turns about your head, so the next one on the right or left comes to the front; ft-screens' `spin next|prev|`); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults. - Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button. - Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove. - Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu. @@ -129,7 +146,7 @@ Device rules are saved in `~/.config/frametop-input.json`. `input-settings/insta ## Frametop Display Settings and ft-layout -When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout. +When the desktop starts, its screens arrange themselves around where you're facing. You can move them by hand at any time and put them back with Meta+Shift+R, the reset button left of any screen's bar, the Reset Screen Layout menu entry, Arrange now in the app, or a mouse button mapped to Reset desktop screen layout. The desktop's own screen arrangement follows where the screens are around you, whatever their numbers: a screen you see to the left of another is to its left in Plasma too, so the pointer and dragged windows cross straight to it. Screens one above the other stack, and screens pinned to a wrist or your head come last. It's updated at startup, after arranging or saving the layout, and half a second after you let go of a screen you moved. With the headset off there's no head pose to go by, and the arrangement stays as it was. @@ -199,14 +216,41 @@ power/run.sh off | on # the displays off now, or back on power/run.sh log ``` +## Pausing for VR games + +Paused, Frametop leaves the headset's CPU and GPU to a VR game. The input relay does it (`input/game_pause.py`), since it's the one part that always runs: + +- The gaze service stops (`frametop-gaze`: ft-gazed, ft-gaze, our own eye tracker, the gaze panel), so nothing reads SteamVR's eye tracking. Our frame grabber, the root service `ft-eyegrab`, goes idle by itself 3 seconds after our eye tracker stops asking it for frames. +- Hand tracking stops if it runs (`frametop-camd`, `frametop-hands`). +- The desktop, as the Game optimization page of Frametop Input Settings says (`pause_desktop`): hidden (the default) or closed. Hidden, ft-screens hides every screen and floating window whatever the visibility mode, the hotkey, or the dashboard says, and gives KWin a frame callback once a second instead of every display frame. KWin draws a screen only after its frame callback, and its apps wait for theirs, so the desktop hardly draws, but its windows stay open. Remote desktop stops if it runs (`session/remote-ctl.sh`). Closed, `desktops.sh stop` closes the desktop and its windows, and resuming starts it again (about 12 seconds), in its start profile if it has one. +- The relay lets go of the 3D mouse and feeds pointer devices to its virtual mouse and keyboard, as with `POINTER=0`. Typing goes to Steam. Mapped buttons and key combinations do nothing but pausing, the Steam menu, and commands; a key combination that does nothing is typed as usual. + +Resuming starts again only what pausing stopped, and plays a second sound. The pointer helper and ft-powerd keep running: they cost little, the helper is what says a game started, and stopping it would leave its virtual controller connected with its last pose. + +Ways to pause and resume: + +- The controller gesture, by default both thumbsticks clicked together twice: both go down within 0.3 seconds of each other, and the second time within 0.7 seconds of the first. The relay reads it from vrserver's web socket (`input/vrws.py`), which works whatever has input focus and takes nothing from the game, so the game sees the clicks too. The Game optimization page changes it: one or two of the buttons the Controllers page lists, pressed once or twice (one button always takes two), or none. +- The Pause/resume Frametop action, on a mouse button, a key combination, or a controller button (outside games, like every mapped controller button). +- VR games, with Pause while a VR game runs on (`pause_auto`, the default). The pointer helper tells the relay when a scene app starts and ends (`vrgame 1|0`, repeated every 5 seconds). A game starting pauses Frametop. A pause that starts while a game runs ends 5 seconds after the game does, unless another game starts first. Resumed during a game, Frametop stays on until that game ends. A pause that starts outside a game lasts until you resume. Flatscreen games aren't scene apps, so they don't pause it. +- From a terminal or a script: + +``` +input/ft-pause on | off | toggle # pause or resume +input/ft-pause status # the state as JSON (the relay's "pause ?") +input/vrws.py 10 # the controllers' buttons from vrserver's web socket, for 10 s +input/test/pause-test.py # the gesture and the automatic pause, offline +``` + +The state outlives a relay restart, in `/run/user/UID/frametop-pause.json`. A SteamVR restart while paused starts the gaze service with it, and the relay stops it again when the pointer helper comes back. + ## Gaze pointer (experimental) -In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off. +In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. The service idles while the gaze isn't used: its eye tracker reader and our own eye tracker run only while gaze mode is on and someone wears the headset, while a check or the calibration runs, or while the Gaze page of Frametop Input Settings is open, and stop 30 seconds after ([gaze/README.md](../gaze/README.md)). Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off. - Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held. - The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual. - A correction before a click teaches the gaze service the tracker's error there. A correction bigger than `POINTER_GAZE_NUDGE_MAX` (55 degrees) isn't learned; it opens a quick check instead. -- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode coming on without a calibration opens Calibrate by itself. +- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it. [gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool. @@ -231,6 +275,8 @@ It listens on port 5900 on the Frame's Tailscale address only, not the LAN, so i No VNC server can capture KWin on SteamOS directly: `krfb` needs `xdg-desktop-portal-kde`, which SteamOS doesn't ship, and `wayvnc` only works with wlroots compositors. So `session/remote-desktop.sh` captures the desktop with KDE's `krdpserver --plasma` on `127.0.0.1:3390`, and `session/vnc-bridge.sh` runs TigerVNC's `Xvnc` on display `:20` with a FreeRDP client inside it and serves that. Both run in the `dev` container, and the extra hop adds a little latency. krdp streams every screen; the VNC screen is the primary's size, and the FreeRDP window is shifted so the primary fills it (`ft-layout remote-view` gives the offset). krdp's own `--monitor` would stream just one screen, but it maps the pointer as if that screen sat at 0,0, so clicks would miss. When the layout changes, the VNC screen resizes and FreeRDP reconnects within a few seconds. +FreeRDP runs only while a VNC viewer is connected, because while it's connected krdp captures and encodes every redraw. With no viewer, krdp has no RDP connection and so captures nothing, and Xvnc shows a black screen. When a viewer connects, the bridge starts FreeRDP, and the desktop appears about 3 seconds later; FreeRDP stops 45 seconds after the last viewer leaves (`VNC_IDLE_SEC` in the bridge's environment). The bridge looks for viewers with `ss` whenever Xvnc logs something, as it does for every connection, and every 5 seconds otherwise. While a viewer is connected, the bridge asks ft-screens to draw every screen at full rate (`watch 15` on `@ft_screens`, renewed every 5 seconds), so screens you aren't looking at in the headset, or a headset on a stand, don't stream at a low rate. It reads the primary screen's place again (`ft-layout remote-view`) only while FreeRDP runs, after `~/.config/frametop/kwinoutputconfig.json` or `~/.config/frametop-layout.json` changes, and once a minute. + With remote access on, the nested KWin runs with `KWIN_WAYLAND_NO_PERMISSION_CHECKS=1` and `KWIN_SCREENSHOT_NO_PERMISSION_CHECKS=1`, so any app in the Frametop desktop could capture its screens or inject input. The second one lets scripts take screenshots through KWin's `org.kde.KWin.ScreenShot2` D-Bus interface. This applies only to that desktop, not the stock one. Port 3389 is SteamOS's own `xrdp`, which starts a separate X11 session rather than showing the VR desktop. ## Limits diff --git a/float/frametop-float.js b/float/frametop-float.js index 85c0f0a..5f52059 100644 --- a/float/frametop-float.js +++ b/float/frametop-float.js @@ -378,6 +378,17 @@ function run(c) { case "activate": if (w) workspace.activeWindow = w; break; + case "activate-output": { // the top window on that output (a spin brought it to the front) + const order = workspace.stackingOrder; + for (let i = order.length - 1; i >= 0; --i) { + const o = order[i]; + if (o.deleted || o.minimized || o.hidden || !o.managed || !o.output) continue; + if (o.output.name !== c.output || !o.normalWindow || o.popupWindow) continue; + workspace.activeWindow = o; + break; + } + break; + } case "minimize": if (w) w.minimized = c.on; break; diff --git a/float/ft_floatd.py b/float/ft_floatd.py index 66a81e1..a0dc92c 100644 --- a/float/ft_floatd.py +++ b/float/ft_floatd.py @@ -18,6 +18,8 @@ command-line side). Replies go to the sender: ("float pointer" is the float key: the window under the pointer, else the active one, floated or docked; the input relay sends it for float_toggle, and "dock all" for dock_all) dock N | close N | resize N W H | scale N STEPS (ft-screens, N = its screen) + front N (ft-screens: a spin brought panel N to the front: make its window, or the top one + on a screen, KWin's active window) Spare outputs are WL- .. WL-. A floating window's output is its frame plus a margin on each side (FLOAT_MARGIN pixels), so menus have room; the panel shows @@ -229,6 +231,24 @@ def cmdline(pid): return [] +def desktop_name(app, cls): + """The desktop file name for a window's app id and class. A Flatpak app's window can give an + id with no desktop file: RustDesk's (an X11 window) says com.carriez.flutter_hbb, and its + desktop file is com.rustdesk.RustDesk. That file's StartupWMClass names the window's class + then. The app id itself when nothing matches.""" + try: + if not app or Gio.DesktopAppInfo.new(app + ".desktop"): + return app + except TypeError: # PyGObject raises for the NULL a missing desktop file returns + pass + names = {n.lower() for n in (app, cls) if n} + for info in Gio.AppInfo.get_all(): + wm = info.get_startup_wm_class() if isinstance(info, Gio.DesktopAppInfo) else None + if wm and wm.lower() in names: + return info.get_id().removesuffix(".desktop") + return app + + class Launch: """An app we started: its first window floats, or (for a profile) its windows go to the profile's entries for it, in order.""" @@ -779,8 +799,10 @@ class Daemon: return None chain = pid_chain(int(ev.get("pid") or 0)) app = ev.get("app", "") + # (An X11 window in a Flatpak gives its pid in the sandbox: only its app id can match.) + apps = {app, desktop_name(app, ev.get("cls", ""))} if app and any(la.app for la in self.launches) else {app} for la in self.launches: - if any(p in chain for p in la.pids) or (la.app and app and la.app == app): + if any(p in chain for p in la.pids) or (la.app and app and la.app in apps): if not la.entries or len(la.entries) <= 1: self.launches.remove(la) return la @@ -844,7 +866,7 @@ class Daemon: for wid, ev in self.windows.items(): if not recordable(ev): continue - entry = {"app": ev["app"]} if ev.get("app") else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")} + entry = {"app": desktop_name(ev["app"], ev.get("cls", ""))} if ev.get("app") else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")} if not entry.get("app") and not entry.get("cmd"): continue f = self.floats.get(wid) @@ -871,7 +893,7 @@ class Daemon: return out def window_key(self, ev): - return ev.get("app") or json.dumps(cmdline(ev.get("pid"))) + return desktop_name(ev.get("app", ""), ev.get("cls", "")) or json.dumps(cmdline(ev.get("pid"))) def open_profile(self, name): """Open a profile's apps (additive: nothing closes).""" @@ -887,9 +909,9 @@ class Daemon: if key != "[]": groups.setdefault(key, []).append(e) claimed = set() + keys = {wid: self.window_key(ev) for wid, ev in self.windows.items() if recordable(ev)} for key, entries in groups.items(): - have = [ev for wid, ev in self.windows.items() - if wid not in claimed and recordable(ev) and self.window_key(ev) == key] + have = [ev for wid, ev in self.windows.items() if wid not in claimed and keys.get(wid) == key] for e, ev in zip(entries, have): claimed.add(ev["id"]) self.place_entry(ev, e) @@ -1043,6 +1065,13 @@ class Daemon: for f in list(self.floats.values()): self.dock(f) return "ok" + if cmd == "front" and len(rest) == 1 and rest[0].isdigit(): + f = self.by_panel(int(rest[0])) + if f: + self.command(cmd="activate", id=f.id) + else: # a screen: its output is WL- + self.command(cmd="activate-output", output=f"WL-{int(rest[0]) - 1}") + return "ok" if cmd in ("dock", "close", "resize", "scale") and rest and rest[0].isdigit(): f = self.by_panel(int(rest[0])) if not f: diff --git a/gaze/README.md b/gaze/README.md index f42783d..2ee6937 100644 --- a/gaze/README.md +++ b/gaze/README.md @@ -2,12 +2,12 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (the pointer goes where you look, and the mouse does the last bit), and the tools to calibrate and measure it. -- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on. +- `ft-gaze` (C++, OpenVR, runs in the dev container) reads the eye tracker and prints one JSON line per sample (90 Hz). For each source, it gives the gaze direction relative to the head and the Frametop screen pixel it lands on. The gaze service asks it for only the sources it uses (`--sources`, and `sources LIST` on its stdin): our tracker and mmap set 1 with Own tracker, about 700 bytes a line instead of 1.3 KB, and every source while a check or the calibration runs. So SteamVR's gaze action, the only source that calls into vrserver (twice a sample), is read only then. The probe gets them all. - `gazecal.py` has what the probe and the gaze service share: the correction models, filters, and the reader for SteamVR's eye tracking log. - `tracker/` is our own eye tracker, an alternative to SteamVR's: `ft-eyes` finds the pupils and glints in the eye-camera frames that `ft-eyegrab` (a small root service) copies out of SteamVR's tracker. See "Our own eye tracker" below. - `probe/ft-gazeprobe` (GTK 4, host Python) is a fullscreen playground, for developing the gaze tracking: day to day, the calibration and the checks run in the headset panel (Quick check, Calibrate, and Check headset fit on the Gaze page). It runs ft-gaze, draws where you're looking, measures accuracy, and tries out hold-to-adjust clicking with a calibration that learns from your adjustments. -Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`); the probe and our own tracker are installed by hand. +Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`) and then our own tracker (`gaze/tracker/install.sh`), which gaze mode uses once it's installed; the probe is installed by hand. ``` gaze/run.sh install # the gaze service: builds ft-gaze and the panel, starts with SteamVR @@ -23,7 +23,7 @@ gaze/probe/ft-gazeprobe --screen 1 Gaze as an input method for the whole desktop, without replacing anything of SteamVR's: - `ft-gazed` (host Python, a user service: `gaze/run.sh install`) runs ft-gaze and corrects its gaze. Two settings on the Gaze page of Frametop Input Settings (`GAZE_TRACKER` and `GAZE_EYE` in `~/.config/frametop.conf`, read again when the file changes) pick whose eye tracking it uses and how it weights the eyes: - - **Eye tracker:** SteamVR's (the default), or our own (Own tracker: see "Our own eye tracker" below). The gaze service runs ours while this is on. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either. + - **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either. - **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost. With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery. @@ -33,8 +33,9 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste - **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time. - **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it. - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them. -- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. +- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`. - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always. +- **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there. Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was. @@ -62,7 +63,7 @@ Ground rules, for anyone changing it: - **Clean room.** Nothing of Valve's goes in: we don't decompile, disassemble, or patch the `eyetracking` binary or its network weights, and we don't copy their code or weights. Its public output (eye-server.mmap, read-only) is fair game as a baseline and as labels, and so are published papers and openly licensed pupil detectors (check each one's license: PuRe, PuReST, ElSe, and ExCuSe are non-commercial only). - **Root only reads.** ft-eyegrab never writes to, stops, or signals the `eyetracking` process, vrserver, or vrcompositor, never opens `/dev/adsp`, `/dev/cdsp`, or `/dev/spidev0.1`, and never writes to `/dev/shm/eye-server.mmap` (it also carries calibration clicks into SteamVR's tracker), `/opt`, or `/persist`. - **Eye images are biometric data.** Recordings live outside the repo, in `~/.local/share/frametop/eyes/captures` (0700), and `.gitignore` catches stray frame dumps. They go nowhere but the machine that runs your offline jobs. -- **Mind the headset's budget.** Finding a pupil takes about 0.4 ms a frame while ft-eyes follows it, and 1.4-2.1 ms when it searches the whole frame. Replays, scoring, and training go to a PC. +- **Mind the headset's budget.** Finding a pupil takes about 0.4 ms a frame while ft-eyes follows it, and 1.4-2.1 ms when it searches the whole frame. ft-eyes keeps OpenCV and numpy to one thread: their pools of one per core spun idle workers at about a quarter of a core, for frames this small. It also runs at nice 10 with SCHED_BATCH, and ft-gaze at nice 5 (not batch, since each sample goes on to the pointer): both run in the dev container's podman scope, out of reach of the gaze service's unit, on the cores vrcompositor and vrserver use at nice 0. Replays, scoring, and training go to a PC. ## Headset fit diff --git a/gaze/ft-gaze.cpp b/gaze/ft-gaze.cpp index 03cb285..0c9393c 100644 --- a/gaze/ft-gaze.cpp +++ b/gaze/ft-gaze.cpp @@ -4,7 +4,14 @@ // Every eye tracker sample (90 Hz) becomes one JSON line on stdout with each gaze source // hit-tested against the Frametop screens: // -// Options: -v (log action errors), --watch-stdin (quit when stdin closes). +// Options: -v (log action errors), --watch-stdin (quit when stdin closes; until then, a line +// "sources LIST" on stdin switches the sources as --sources does), --sources LIST (comma- +// separated: action, mmap1, mmap2, left, right, own, and eye for EYE; or all, the default). +// Sources left out are read not at all and print as {"ok":0} ("eye" as null), so every line +// keeps the same keys. The gaze service asks for the ones it uses (own and mmap1 with our +// tracker), and all of them while a check or the calibration runs. Only the action costs +// SteamVR anything (two calls into vrserver per sample), and a line with every source is +// about 1.5 KB, 130 KB a second through podman's relay. // // {"t":,"age":,"n":, // "head":{"yaw":..,"pitch":..,"hit":HIT}, head forward ray (for head nudging) @@ -58,6 +65,7 @@ #include #include +#include #include #include #include @@ -72,6 +80,7 @@ #include #include +#include #include #include #include @@ -389,6 +398,30 @@ std::string SrcJson(const std::vector &screens, const vr::HmdMatrix34_t return buf + extra + "\"hit\":" + HitJson(screens, head, yaw, pitch) + "}"; } +// Sources (--sources, "sources LIST" on stdin): a bit each. +enum : unsigned { kAction = 1, kMmap1 = 2, kMmap2 = 4, kLeft = 8, kRight = 16, kOwn = 32, kEye = 64, kAll = 127 }; + +bool ParseSources(const std::string &list, unsigned &mask) { + static const struct { + const char *name; + unsigned bit; + } names[] = {{"action", kAction}, {"mmap1", kMmap1}, {"mmap2", kMmap2}, {"left", kLeft}, + {"right", kRight}, {"own", kOwn}, {"eye", kEye}, {"all", kAll}}; + unsigned m = 0; + size_t at = 0; + while (at <= list.size()) { + const size_t comma = std::min(list.find(',', at), list.size()); + const std::string name = list.substr(at, comma - at); + bool known = false; + for (const auto &n : names) + if (name == n.name) m |= n.bit, known = true; + if (!known) return false; + at = comma + 1; + } + mask = m; + return true; +} + // Head poses of the last half second, so a sample can use the pose at its own time. class PoseHistory { public: @@ -426,17 +459,42 @@ std::string ExeDir() { int main(int argc, char **argv) { bool verbose = false, watchStdin = false; + std::atomic sources{kAll}; for (int i = 1; i < argc; ++i) { if (std::strcmp(argv[i], "-v") == 0) verbose = true; if (std::strcmp(argv[i], "--watch-stdin") == 0) watchStdin = true; + if (std::strcmp(argv[i], "--sources") == 0 && i + 1 < argc) { + unsigned m; + if (ParseSources(argv[++i], m)) + sources = m; + else + std::fprintf(stderr, "ft-gaze: --sources %s: unknown source (all used)\n", argv[i]); + } } + // Nice 5, before any thread starts (they inherit it): we run in the dev container's podman + // scope, beside vrcompositor and vrserver at nice 0, and the gaze service's unit doesn't + // reach us. Not SCHED_BATCH, as ft-eyes is: that would let each wakeup wait out another + // task's turn, and each sample goes on to the pointer. + errno = 0; + const int nice0 = getpriority(PRIO_PROCESS, 0); + if (errno == 0 && nice0 < 5 && setpriority(PRIO_PROCESS, 0, 5) != 0) + std::fprintf(stderr, "ft-gaze: nice: %s\n", std::strerror(errno)); // --watch-stdin: quit when stdin closes. The probe runs us through distrobox, which // passes neither its signals nor a closed stdout on to us, but does pass stdin's end. + // Lines on stdin until then: "sources LIST". std::atomic stdinClosed{false}; if (watchStdin) - std::thread([&stdinClosed] { + std::thread([&stdinClosed, &sources] { char c[256]; - while (read(0, c, sizeof c) > 0) { + std::string line; + ssize_t n; + while ((n = read(0, c, sizeof c)) > 0) { + line.append(c, size_t(n)); + for (size_t nl; (nl = line.find('\n')) != std::string::npos; line.erase(0, nl + 1)) { + unsigned m; + if (line.compare(0, 8, "sources ") == 0 && ParseSources(line.substr(8, nl - 8), m)) sources = m; + } + if (line.size() > 4096) line.clear(); // no newline in sight: not ours } stdinClosed = true; }).detach(); @@ -479,9 +537,24 @@ int main(int argc, char **argv) { double lastEmit = 0; int actionErrors = 0; vr::EVRInputError lastActionError = vr::VRInputError_None; + // During a VR game, SteamVR's gaze action is left alone. With ft-gaze reading the eyes, SteamVR + // restarted its eye tracker every 10 s or so in a game, as if the headset came off, and each + // restart took input focus from the game: Beat Saber paused (PR #13). Of what ft-gaze reads, + // only the action reaches SteamVR (the mmap and our tracker are files), so gaze still works + // over the dashboard. Games are told apart the way ft-screens does it, by the scene app. + bool inGame = false; + double nextGameCheck = 0; while (true) { const double now = NowRaw(); + if (now >= nextGameCheck) { + nextGameCheck = now + 0.5; + const bool game = vr::VRApplications()->GetCurrentSceneProcessId() != 0; + if (game != inGame) + std::fprintf(stderr, "ft-gaze: %s\n", + game ? "a VR game is running: SteamVR's gaze action left alone" : "the VR game ended"); + inGame = game; + } vr::TrackedDevicePose_t hp; sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &hp, 1); if (hp.bPoseIsValid) history.Add(now, hp.mDeviceToAbsoluteTracking); @@ -494,6 +567,7 @@ int main(int argc, char **argv) { if (fresh && hp.bPoseIsValid) { lastEmit = now; + const unsigned want = sources; const auto list = screens.Get(); const vr::HmdMatrix34_t &headNow = hp.mDeviceToAbsoluteTracking; vr::HmdMatrix34_t headThen = headNow; @@ -502,24 +576,26 @@ int main(int argc, char **argv) { // SteamVR's action: a room-space origin and fixation point, turned into the head // frame so every source reports the same kind of angles. std::string action = "{\"ok\":0}"; - vr::VRActiveActionSet_t active{}; - active.ulActionSet = set; - active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin; - input->UpdateActionState(&active, sizeof active, 1); - vr::VREyeTrackingData_t e{}; - const vr::EVRInputError ae = - input->GetEyeTrackingDataRelativeToNow(gaze, vr::TrackingUniverseStanding, 0, &e, sizeof e); - if (ae == vr::VRInputError_None && e.bActive && e.bValid) { - const Vec3 o{e.vGazeOrigin.v[0], e.vGazeOrigin.v[1], e.vGazeOrigin.v[2]}; - const Vec3 t{e.vGazeTarget.v[0], e.vGazeTarget.v[1], e.vGazeTarget.v[2]}; - const Vec3 dHead = RotateInverse(headNow, Normalize(t - o)); - char extra[96]; - std::snprintf(extra, sizeof extra, "\"tracked\":%d,\"dist\":%.3f,", int(e.bTracked), Length(t - o)); - action = SrcJson(list, headNow, dHead, extra); - } else if (ae != lastActionError || (verbose && ++actionErrors % 90 == 1)) { - std::fprintf(stderr, "ft-gaze: action: error %d active %d valid %d\n", int(ae), int(e.bActive), - int(e.bValid)); - lastActionError = ae; + if (!inGame && (want & kAction)) { + vr::VRActiveActionSet_t active{}; + active.ulActionSet = set; + active.nPriority = vr::k_nActionSetOverlayGlobalPriorityMin; + input->UpdateActionState(&active, sizeof active, 1); + vr::VREyeTrackingData_t e{}; + const vr::EVRInputError ae = + input->GetEyeTrackingDataRelativeToNow(gaze, vr::TrackingUniverseStanding, 0, &e, sizeof e); + if (ae == vr::VRInputError_None && e.bActive && e.bValid) { + const Vec3 o{e.vGazeOrigin.v[0], e.vGazeOrigin.v[1], e.vGazeOrigin.v[2]}; + const Vec3 t{e.vGazeTarget.v[0], e.vGazeTarget.v[1], e.vGazeTarget.v[2]}; + const Vec3 dHead = RotateInverse(headNow, Normalize(t - o)); + char extra[96]; + std::snprintf(extra, sizeof extra, "\"tracked\":%d,\"dist\":%.3f,", int(e.bTracked), Length(t - o)); + action = SrcJson(list, headNow, dHead, extra); + } else if (ae != lastActionError || (verbose && ++actionErrors % 90 == 1)) { + std::fprintf(stderr, "ft-gaze: action: error %d active %d valid %d\n", int(ae), int(e.bActive), + int(e.bValid)); + lastActionError = ae; + } } std::string m1 = "{\"ok\":0}", m2 = m1, left = m1, right = m1, eye = "null"; @@ -543,26 +619,33 @@ int main(int argc, char **argv) { return std::string(b); }; char extra[256]; - std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1), - s.open[0], s.open[1], lr(s.left1, s.right1)); - m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1) + unc(s.var1)); - std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2)); - m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2) + unc(s.var2)); - left = SrcJson(list, headThen, s.left2); - right = SrcJson(list, headThen, s.right2); + if (want & kMmap1) { + std::snprintf(extra, sizeof extra, "\"dist\":%.3f,\"open\":[%.3f,%.3f],\"lr\":%.3f,", Length(s.fix1), + s.open[0], s.open[1], lr(s.left1, s.right1)); + m1 = SrcJson(list, headThen, s.left1 + s.right1, extra + eyes(s.left1, s.right1) + unc(s.var1)); + } + if (want & kMmap2) { + std::snprintf(extra, sizeof extra, "\"lr\":%.3f,", lr(s.left2, s.right2)); + m2 = SrcJson(list, headThen, s.left2 + s.right2, extra + eyes(s.left2, s.right2) + unc(s.var2)); + } + if (want & kLeft) left = SrcJson(list, headThen, s.left2); + if (want & kRight) right = SrcJson(list, headThen, s.right2); + // "new" compares with the last sample, so the last measurement is kept either way. const float *m = s.meas; const bool newL = m[0] != lastMeas[0] || m[1] != lastMeas[1]; const bool newR = m[2] != lastMeas[2] || m[3] != lastMeas[3]; std::memcpy(lastMeas, m, sizeof lastMeas); - std::snprintf(extra, sizeof extra, "{\"q\":[%.3g,%.3g],\"m\":[[%.4f,%.4f],[%.4f,%.4f]],\"new\":[%d,%d]}", - (m[4] + m[5]) / 2, (m[6] + m[7]) / 2, m[0], m[1], m[2], m[3], int(newL), int(newR)); - eye = extra; + if (want & kEye) { + std::snprintf(extra, sizeof extra, "{\"q\":[%.3g,%.3g],\"m\":[[%.4f,%.4f],[%.4f,%.4f]],\"new\":[%d,%d]}", + (m[4] + m[5]) / 2, (m[6] + m[7]) / 2, m[0], m[1], m[2], m[3], int(newL), int(newR)); + eye = extra; + } } // Our tracker: its own sample time picks the head pose, like the mmap's. std::string own = "{\"ok\":0}"; OwnSample o; - if (ownFile.Read(o) && now - o.t < 0.1) { + if ((want & kOwn) && ownFile.Read(o) && now - o.t < 0.1) { vr::HmdMatrix34_t headOwn = headNow; history.At(o.t, headOwn); auto pair = [](bool ok, float a, float b) { @@ -607,7 +690,10 @@ int main(int argc, char **argv) { break; } if (stdinClosed) break; - std::this_thread::sleep_for(std::chrono::milliseconds(2)); + // 250 passes a second: a new sample is printed within 4 ms (2 on average) of appearing, + // and the pose history has a pose within 2 ms of any sample's time (a 0.2 degree head + // turn at 100 degrees a second). Every 2 ms read the pose and the events twice as often. + std::this_thread::sleep_for(std::chrono::milliseconds(4)); } screens.Stop(); vr::VR_Shutdown(); diff --git a/gaze/ft-gazectl b/gaze/ft-gazectl index 8a43637..8b4d83a 100755 --- a/gaze/ft-gazectl +++ b/gaze/ft-gazectl @@ -36,8 +36,17 @@ def main(): if r is None: sys.exit("the pointer helper isn't running") print(f"gaze mode {r.removeprefix('ok ')}" if r else "no answer (an older pointer helper without gaze mode?)") - if ask("ft_gazed", "status", 0.3) is None: + st = ask("ft_gazed", "status", 0.3) + if st is None: print("note: the gaze service (ft-gazed) isn't running, so the pointer has no gaze to follow") + elif r == "ok on": + try: + calibrated = (json.loads(st).get("checks") or {}).get("calibrated") + except ValueError: + calibrated = None + if calibrated is False: + print("note: no calibration for this eye tracker yet; it opens in the headset " + "(if it can't, ft-gazectl status says why: checks.problem)") elif cmd in ("status", "forget", "reload"): r = ask("ft_gazed", cmd) if r is None: diff --git a/gaze/ft-gazed b/gaze/ft-gazed index 157ca4d..18a768b 100755 --- a/gaze/ft-gazed +++ b/gaze/ft-gazed @@ -3,8 +3,12 @@ Two settings in ~/.config/frametop.conf (the Gaze page of Frametop Input Settings), read again when the file changes: - GAZE_TRACKER=steam|own SteamVR's eye tracker (default), or our own (gaze/tracker/ft-eyes, - ft-gaze's source "own"; this service runs it, see below) + GAZE_TRACKER=auto|own|steam + our own eye tracker (gaze/tracker/ft-eyes, ft-gaze's source "own"; + this service runs it, see below) or SteamVR's. auto (the default) + is ours when it's installed (gaze/tracker/install.sh: the frame + grabber, and ft-eyes' Python in this checkout), else SteamVR's; it + switches when ours is installed or removed. GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far off it was at your recent nudges; left or right counts that eye twice as much as the other. Either eye alone carries the gaze when the @@ -59,9 +63,24 @@ and the error moves): lessons from before count less then, so the first few afte relearn the offset. Our own tracker (gaze/tracker/ft-eyes) runs here too, in the dev container, while -GAZE_TRACKER=own or the gaze probe asks for it ("eyes SECONDS", a lease the probe renews). It -reads the eye-camera frames the root service frametop-eyegrab copies (gaze/tracker/install.sh), -which copies them only while ft-eyes runs. +GAZE_TRACKER=own and the gaze is in use (below), or the gaze probe asks for it ("eyes SECONDS", +a lease the probe renews). It reads the eye-camera frames the root service frametop-eyegrab +copies (gaze/tracker/install.sh), which copies them only while ft-eyes runs. + +Idle: ft-gaze and our own tracker run only while the gaze is in use: gaze mode on with someone +wearing the headset (the pointer helper says both: "gaze ? headset" -> "ok on|off worn|away"), a +check or calibration open or asked for, or a "wake" lease (the Gaze page of Frametop Input +Settings renews one while it's open). IDLE_AFTER after the last use they stop, and the frame +grabber goes idle with our tracker; with our tracker, that was over half a core with gaze mode +off. A check asked for while idle waits for the tracker to start (at most WAKE_SETTLE). Our +tracker's next click after it starts again re-seats it, as after the headset was off, so turning +gaze mode on after a while opens the quick check. + +ft-gaze prints only the sources this uses ("sources LIST" on its stdin, see wanted_sources): +own and mmap1 with our tracker, left, right and mmap1 with SteamVR's eyes, and every source +while a check or the calibration runs or is asked for, since those record them all. So +SteamVR's gaze action, the one source that calls into vrserver, is read only then (or with +--source action). Checks and calibration (gaze/gazecheck.py): a one-dot quick check when the headset goes on or our tracker asks for a click, and the full calibration when gaze mode comes on without one, @@ -79,6 +98,8 @@ Control socket: abstract unix datagram "@ft_gazed": reload read calibration.json and the settings again eyes keep our own tracker running that much longer (at most 120), whatever GAZE_TRACKER says: the probe's lease. Reply: "ok" + wake keep the gaze running (not idle) that much longer (at most 120), + whatever gaze mode says: Input Settings' Gaze page. Reply: "ok" quickcal the one-dot check now (the calibration if there's none) calibrate the full calibration in the panel fitcheck the headset fit check in the panel @@ -119,6 +140,7 @@ EYES_PROG = REPO / "gaze" / "tracker" / "ft-eyes" # our own tracker EYES_PYTHON = REPO / "gaze" / "tracker" / "build" / "venv" / "bin" / "python" # numpy, OpenCV (build.sh) EYES_SOCKET = "\0ft_eyes" # its control socket EYES_CAMS = Path("/dev/shm/frametop-eyes-cams") # the frames it reads (frametop-eyegrab.service) +EYEGRAB = (Path("/etc/frametop/ft-eyegrab"), Path("/etc/systemd/system/frametop-eyegrab.service")) # tracker/install.sh CONF = Path.home() / ".config" / "frametop.conf" CALIBRATION = STATE / "calibration.json" LESSONS = STATE / "pointer-lessons.json" @@ -126,6 +148,7 @@ LESSON_LOG = STATE / "pointer-lessons.jsonl" SOURCES = ("action", "mmap1", "mmap2", "left", "right") # the ones with a calibration here SIDES = ("left", "right") # ft-gaze's order, and the sources for each eye alone TRACKERS = ("steam", "own") +TRACKER_SETTINGS = ("auto",) + TRACKERS BIASES = ("auto", "left", "right") NUDGE_MAX = 55.0 # degrees: POINTER_GAZE_NUDGE_MAX's default, the largest lesson taken HISTORY = 12.0 # seconds of the gaze sent, to find a lesson's look (the helper sends it up to 10 s later) @@ -134,6 +157,9 @@ RETRY = 3.0 # seconds before starting ft-gaze again EYES_RETRY = 10.0 # seconds before starting ft-eyes again after it stopped on its own EYES_LEASE_MAX = 120.0 SETTLE = 0.3 # seconds after an eye is found again before the fallback learns from it +IDLE_AFTER = 30.0 # seconds after the gaze was last in use before ft-gaze and our tracker stop +WAKE_SETTLE = 15.0 # seconds after waking that the tracker may take to start sending +WAKE_MAX = 120.0 KEYBOARD_PITCH = -20.0 # degrees: gaze under this, on no screen, is a look at the keyboard @@ -153,8 +179,14 @@ def log(msg): print(f"ft-gazed: {msg}", file=sys.stderr, flush=True) +def own_installed(): + """Is our own tracker installed: its frame grabber, and ft-eyes' Python here?""" + return all(p.exists() for p in EYEGRAB) and EYES_PYTHON.exists() + + def read_settings(): - """(tracker, eye bias, nudge max) from frametop.conf, defaults for anything missing or unknown.""" + """(tracker, GAZE_TRACKER, eye bias, nudge max) from frametop.conf, defaults for anything + missing or unknown. The tracker is "steam" or "own": auto picks ours when it's installed.""" conf = {} try: for line in CONF.read_text().splitlines(): @@ -164,13 +196,15 @@ def read_settings(): conf[k.strip()] = v.strip().lower() except OSError: pass - tracker = conf.get("GAZE_TRACKER", "steam") + setting = conf.get("GAZE_TRACKER", "auto") + setting = setting if setting in TRACKER_SETTINGS else "auto" + tracker = setting if setting != "auto" else "own" if own_installed() else "steam" bias = conf.get("GAZE_EYE", "auto") try: nudge = min(max(float(conf.get("POINTER_GAZE_NUDGE_MAX", NUDGE_MAX)), 1.0), 110.0) # the helper's range except ValueError: nudge = NUDGE_MAX - return tracker if tracker in TRACKERS else "steam", bias if bias in BIASES else "auto", nudge + return tracker, setting, bias if bias in BIASES else "auto", nudge def mtime(path): @@ -185,7 +219,7 @@ class Service: self.override, self.verbose, self.to = source, verbose, to self.source = source or "mmap1" # the older path's source STATE.mkdir(parents=True, exist_ok=True) - self.tracker, self.bias, self.nudge_max = read_settings() + self.tracker, self.tracker_setting, self.bias, self.nudge_max = read_settings() self.conf_mtime = mtime(CONF) self.models = {name: Correction() for name in SOURCES} self.mode = DEFAULT_MODEL @@ -216,6 +250,10 @@ class Service: self.eyes_proc = None # ft-eyes, while it runs self.eyes_until = 0.0 # the probe's lease (monotonic time) self.eyes_restart_at = 0.0 + self.awake = False # ft-gaze (and our tracker) run: the gaze is in use (see the top) + self.idle_at = 0.0 # idle from then, unless it's in use again first + self.woke_at = 0.0 + self.wake_until = 0.0 # a "wake" lease self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC | socket.SOCK_NONBLOCK) self.sock.bind(ME) @@ -228,6 +266,7 @@ class Service: self.sel.register(self.eyes_sock, selectors.EVENT_READ, "own") self.checks = Checks(self, self.sel) self.proc = None + self.proc_sources = None # what ft-gaze was last told to print self.buf = b"" self.restart_at = 0.0 self.running = True @@ -246,9 +285,12 @@ class Service: def load_settings(self): self.conf_mtime = mtime(CONF) - tracker, bias, self.nudge_max = read_settings() + tracker, self.tracker_setting, bias, self.nudge_max = read_settings() if (tracker, bias) != (self.tracker, self.bias): - log(f"tracker {tracker}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else "")) + auto = "" + if self.tracker_setting == "auto": + auto = " (auto: ours is installed)" if tracker == "own" else " (auto: ours isn't installed)" + log(f"tracker {tracker}{auto}, eye bias {bias}" + (f" (--source {self.override} wins)" if self.override else "")) self.tracker, self.bias = tracker, bias for w in self.weights.values(): w.bias = bias @@ -382,8 +424,73 @@ class Service: log(f"lesson log: {e}") return rec + # --- Idle (see the top) --- + + def use_reason(self): + """Why the gaze is in use now, or None.""" + c = self.checks + if c.gaze_on and c.headset is not False: + return "gaze mode is on" + if c.active or c.pending: + return "a check" + if time.monotonic() < self.wake_until: + return "asked to stay awake" + return None + + def idle_reason(self): + c = self.checks + if c.gaze_on is None: + return "the pointer helper isn't answering (SteamVR not running?)" + if c.gaze_on and c.headset is False: + return "nobody is wearing the headset" + return "gaze mode is off" + + def waking(self): + """Idle, or awake too briefly for the tracker to be sending yet.""" + return not self.awake or time.monotonic() - self.woke_at < WAKE_SETTLE + + def update_awake(self): + now = time.monotonic() + why = self.use_reason() + if why: + self.idle_at = now + IDLE_AFTER + if why and not self.awake: + self.awake, self.woke_at, self.restart_at = True, now, 0.0 + log(f"awake: {why}") + elif not why and self.awake and now >= self.idle_at: + self.awake = False + log(f"idle: {self.idle_reason()}; ft-gaze and our tracker stop until the gaze is used again") + self.stop_helper() + if self.eyes_proc and not self.eyes_wanted(): + self.stop_eyes() + # --- ft-gaze --- + def wanted_sources(self): + """The sources ft-gaze should print (its --sources): what on_sample and the checks read. + mmap1 always (blinks, lost eyes, and the headset going on, from its openness and + variances); everything while a check runs or waits, since they record every source.""" + c = self.checks + if c.active or c.pending: + return "all" + kind = self.kind + if kind == "own": + return "own,mmap1" + if kind == "eyes": + return "left,right,mmap1" + return ",".join(dict.fromkeys((self.source, "mmap1", "mmap2"))) # mmap2: each eye, for the fallback + + def sync_sources(self): + want = self.wanted_sources() + if not self.proc or want == self.proc_sources: + return + try: + self.proc.stdin.write(f"sources {want}\n".encode()) + self.proc.stdin.flush() + except (OSError, ValueError): + return # it's stopping: read_stdout notices + self.proc_sources = want + def start_helper(self): if not HELPER.exists(): log(f"ft-gaze isn't built: run {REPO}/gaze/build.sh") @@ -394,9 +501,11 @@ class Service: subprocess.run([str(REPO / "scripts" / "container-up.sh")], env=env, check=False) distrobox = Path.home() / ".local" / "bin" / "distrobox" # ft-gaze quits when its stdin closes: the one thing distrobox passes on. - self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin"], env=env, - stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, - start_new_session=True) + sources = self.wanted_sources() + self.proc = subprocess.Popen([str(distrobox), "enter", "dev", "--", str(HELPER), "--watch-stdin", + "--sources", sources], env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, start_new_session=True) + self.proc_sources = sources os.set_blocking(self.proc.stdout.fileno(), False) os.set_blocking(self.proc.stderr.fileno(), False) self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout") @@ -424,7 +533,7 @@ class Service: self.proc = None def eyes_wanted(self): - return (self.tracker == "own" and not self.override) or time.monotonic() < self.eyes_until + return (self.tracker == "own" and not self.override and self.awake) or time.monotonic() < self.eyes_until def start_eyes(self): """Our own tracker, in the dev container, with build/venv's numpy and OpenCV. Like @@ -683,6 +792,14 @@ class Service: reply = "ok" except ValueError: reply = "error bad seconds" + elif words[:1] == ["wake"] and len(words) == 2: + try: + secs = min(max(float(words[1]), 0.0), WAKE_MAX) + self.wake_until = max(self.wake_until, time.monotonic() + secs) + self.update_awake() + reply = "ok" + except ValueError: + reply = "error bad seconds" elif words[:1] == ["reload"]: self.load_settings() self.load_calibration() @@ -733,7 +850,9 @@ class Service: st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"), lessons=max(len(m) for m in w.misses), own_running=bool(own), own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values())) - st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists()) + st.update(awake=self.awake, idle=None if self.awake else self.idle_reason()) + st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists(), + tracker_setting=self.tracker_setting, own_installed=own_installed()) st.update({"ft_gaze": self.proc is not None, "sample_age_s": round(time.monotonic() - self.last_sample, 2) if self.last_sample else None, "headset_on": self.steam.wearing(), "headset_on_since": self.steam.worn(), "last": [round(v, 2) for v in self.last] if self.last else None, @@ -749,8 +868,10 @@ class Service: if mtime(CALIBRATION) != self.cal_mtime: self.load_calibration() self.refit() - if mtime(CONF) != self.conf_mtime: + if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto" + and own_installed() != (self.tracker == "own")): self.load_settings() + self.update_awake() want = self.eyes_wanted() if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at: self.start_eyes() @@ -771,7 +892,7 @@ class Service: next_verbose = time.monotonic() + 5 while self.running: now = time.monotonic() - if not self.proc and now >= self.restart_at: + if self.awake and not self.proc and now >= self.restart_at: self.start_helper() for key, _ in self.sel.select(timeout=0.05 if self.checks.active else 0.5): if key.data == "control": @@ -789,6 +910,7 @@ class Service: elif key.data == "stderr" and self.proc: self.read_stderr() self.checks.tick() + self.sync_sources() if now >= next_periodic: self.periodic() next_periodic = now + 1.0 diff --git a/gaze/gazecal.py b/gaze/gazecal.py index 1777b92..f261ba3 100644 --- a/gaze/gazecal.py +++ b/gaze/gazecal.py @@ -640,31 +640,44 @@ def cross_validate(points, mode): return errs -def steady_samples(samples, vergence_jump=1.5): +def steady_samples(samples, vergence_jump=1.5, why=None): """The samples of one look at one spot where the tracker had both eyes: none in a blink (openness under half its median over the samples), none where it had lost an eye (its variance over EYE_LOST), and none where the angle between the eyes' directions (`lr`, the vergence) is more than `vergence_jump` degrees from its median over the samples. The vergence itself depends on distance (about 2.8 degrees for a screen 1.3 m away, a fraction of one far off), so only a jump away from what it was during this look means - the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept.""" + the tracker lost an eye. Without the mmap there's nothing to judge by: all are kept. + `why`, a dict, gets how many were dropped for each reason: "lost_left", "lost_right", + "lost_both", "blink" and "vergence" (each sample once, for the first that applies).""" + if why is None: + why = {} # Openness: a blink is a sharp drop from what it was during this look. Not a fixed # level: looking down, the upper lids come down with the eyes, and in bright light you # squint, so the reading can stay under 0.5 for the whole look while the tracker follows # the eyes fine (a calibration dot at the bottom of the bright round failed that way). opens = [min(o) for o in ((smp["src"].get("mmap1") or {}).get("open") for smp in samples) if o] floor = max(0.12, 0.5 * statistics.median(opens)) if len(opens) >= 5 else 0.12 - opened = [] + seen = [] for smp in samples: - o = (smp["src"].get("mmap1") or {}).get("open") - if not o or min(o) >= floor: - opened.append(smp) + m1 = smp["src"].get("mmap1") or {} + o = m1.get("open") + lost = [u > EYE_LOST for u in m1.get("unc") or [0, 0]] + # A lost eye's openness reads 0 too, so a lost eye is named before a blink. + key = ("lost_both" if all(lost) else "lost_left" if lost[0] else "lost_right") if any(lost) else \ + "blink" if o and min(o) < floor else None + if key: + why[key] = why.get(key, 0) + 1 + else: + seen.append(smp) def vergence(smp): return (smp["src"].get("mmap1") or {}).get("lr", (smp["src"].get("mmap2") or {}).get("lr")) - opened = [smp for smp in opened if max((smp["src"].get("mmap1") or {}).get("unc") or [0]) <= EYE_LOST] - have = [v for v in map(vergence, opened) if v is not None] + have = [v for v in map(vergence, seen) if v is not None] if len(have) < 5: - return opened + return seen med = statistics.median(have) - return [smp for smp in opened if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump] + kept = [smp for smp in seen if vergence(smp) is None or abs(vergence(smp) - med) <= vergence_jump] + if len(kept) < len(seen): + why["vergence"] = why.get("vergence", 0) + len(seen) - len(kept) + return kept diff --git a/gaze/gazecheck.py b/gaze/gazecheck.py index c37f221..a912f2e 100644 --- a/gaze/gazecheck.py +++ b/gaze/gazecheck.py @@ -10,16 +10,21 @@ directions: look at each one. once every QUICK_COOLDOWN, and on "quickcal" (Frametop Input Settings, or a mouse button or key combination mapped to Gaze quick check), and when a click's correction was past POINTER_GAZE_NUDGE_MAX (55 degrees; the helper's "recheck"): the tracker is - far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing. + far off. Ignored, it closes after QUICK_TIMEOUT and changes nothing. The headset going + on is seen only while the gaze service is awake (gaze mode on, someone wearing it: see + ft-gazed), so it also opens when gaze mode comes on after the service idled. five the middle and four around it, when the first FIVE_COUNT lessons after a quick check were all over FIVE_LIMIT degrees off: the quick check didn't fix it. full the calibration, as the gaze probe's: three rounds, dark, medium and bright (pupil size, and the tracker's error with it, changes with brightness), each the middle and a ring of six (SteamVR's tracker) or eight (ours, whose fit goes wrong past its dots) RING degrees out, half that in the middle round, turned 20 degrees a round. It opens - when gaze mode comes on without a calibration for the tracker in use, and on - "calibrate". Frametop's screens hide while it runs. Quitting it while there's still - no calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it. + whenever gaze mode is on without a calibration for the tracker in use and someone's + in the headset, and on "calibrate". One that closes unfinished (ignored, too few + dots) opens again only once the headset comes off and on, or gaze mode off and on. + Frametop's screens hide while it runs. Quitting it while there's still no + calibration turns gaze mode off (POINTER_GAZE=0); turning it on again reopens it. + Why gaze mode can't work yet goes in the status ("problem"), for Input Settings. fit the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was @@ -27,6 +32,9 @@ directions: look at each one. adjust the headset. A left click or Meta+J runs its guided check (dots, then looks down, up, left and right); a right click or Meta+K closes it, as does FIT_TIMEOUT. +A check asked for while the gaze service idles (quickcal, calibrate, fitcheck) wakes it and +waits until the tracker sends, at most ft-gazed's WAKE_SETTLE; then it opens, or logs why not. + The quick check's dot captures itself: from CHECK_SETTLE after it shows (the eyes getting there), once the gaze has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and capture time). It's the gaze holding still that counts, not where the tracker puts it, so it @@ -34,7 +42,10 @@ works however far off the tracker is; a left click or Meta+J (the pointer helper "calaccept") takes it now. The full calibration's and five's dots wait for that click: you click when you're looking at the dot (the user asked for that: a steady gaze isn't always on the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as -it takes, up to CLICK_IDLE. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1", +it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason: +gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a +click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common +reason there and in the status. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1", renewed every second; the helper shows it again by itself when that stops). What a capture teaches: @@ -90,6 +101,7 @@ ROUND_BG = (0.03, 0.33, 0.8) ROUND_NAMES = ("dark", "medium", "bright") RING_SCALE = (1.0, 0.5, 1.0) PANEL_RETRY = 10.0 +FULL_RETRY = 10.0 # seconds before an automatic calibration that failed to start tries again FIT_TIMEOUT = 300.0 # seconds the fit check stays up FIT_EVERY = 0.5 # seconds between its cards' updates (each is a new picture for the panel) FIT_HINT_WIDTH = 95 # characters a hint line holds in the panel @@ -117,6 +129,42 @@ def check_dots(kind, own): return out +# Why SteamVR's samples for a look were dropped (gazecal.steady_samples' counts) -> (short, long): +# short for the small panel, long for the calibration's. FIT_HINT goes after the ones the +# headset's fit causes, in the calibration's panel. +STEAM_REASONS = {"lost_left": ("left eye lost", "SteamVR lost your left eye"), + "lost_right": ("right eye lost", "SteamVR lost your right eye"), + "lost_both": ("both eyes lost", "SteamVR lost both eyes"), + "blink": ("blinked", "you blinked"), + "vergence": ("eyes disagreed", "SteamVR's two eyes disagreed")} +FIT_HINT = "check the headset fit" +ACCEPT_WAIT = 1.5 # seconds after a click with no capture before the panel says what it waits for + + +def reject_reason(reply, why): + """Why a dot wasn't taken -> (short, long, fit): our tracker's reply (`reply`), or SteamVR's + drop counts (`why`, when `reply` is None). `fit`: the headset's fit is the likely cause.""" + if reply is None: + if not why: + return "no reading", "SteamVR sent no reading for that look", False + key = max(why, key=why.get) + return *STEAM_REASONS[key], key.startswith("lost") + words = reply.removeprefix("fail ").split() + # ft-eyes: "fail the left eye was seen in only 3 frames", "fail the left eye moved (4.2 px)" + if len(words) >= 3 and words[0] == "the" and words[2] == "eye": + eye = words[1] + if "seen" in words: + return f"{eye} eye not seen", f"our tracker saw your {eye} eye in only {words[-2]} frames", True + if "moved" in words: + return f"{eye} eye moved", f"your {eye} eye moved while you looked", False + if not reply: + return "no answer", "our tracker didn't answer", False + if reply.startswith("our tracker isn't running"): + return "tracker not running", "our tracker isn't running", False + text = reply.removeprefix("fail ") + return text[:24], f"our tracker said: {text}", False + + def spread(points): """The median point and the spread around it (1.4826 x the median distance: a standard deviation that one stray sample can't move far).""" @@ -166,8 +214,13 @@ class Checks: sel.register(self.out, selectors.EVENT_READ, "checks") self.check = None self.gaze_on = None + self.headset = None # someone wears it (the helper's "worn"), False "away", None unknown self.gaze_heard = 0.0 - self.want_full_until = 0.0 # gaze mode came on before the tracker said whether it's calibrated + self.pending = None # (command words, when): asked for while the gaze service idled + self.full_armed = True # gaze mode on without a calibration opens the full one (need_full) + self.full_blocked = None # why it can't open now + self.full_retry_at = 0.0 + self.full_failed = None # why the last calibration failed (too few dots), for problem() self.last_quick = 0.0 self.sample_at = 0.0 # the tracker last sent anything self.seen_at = 0.0 # eyes last seen (SteamVR's tracker's variance for them, "unc") @@ -248,17 +301,22 @@ class Checks: pass def on_readable(self): - """Replies on our socket: the helper's "ok on|off" to "gaze ?"; the panel's are dropped.""" + """Replies on our socket: the helper's "ok on|off [worn|away]" to "gaze ? headset" (an + older helper leaves the headset out); the panel's are dropped.""" while True: try: - data = self.out.recv(4096).decode("utf-8", "replace") + words = self.out.recv(4096).decode("utf-8", "replace").split() except (BlockingIOError, OSError): return - if data in ("ok on", "ok off"): - on = data == "ok on" - was, self.gaze_on, self.gaze_heard = self.gaze_on, on, time.monotonic() - if on and was is False: + if words[:1] == ["ok"] and len(words) in (2, 3) and words[1] in ("on", "off"): + on = words[1] == "on" + headset = None if len(words) == 2 else words[2] == "worn" + was = (self.gaze_on, self.headset) + self.gaze_on, self.headset, self.gaze_heard = on, headset, time.monotonic() + if on and was[0] is False: self.on_gaze_on() + if was != (on, headset): + self.svc.update_awake() # --- State --- @@ -282,11 +340,51 @@ class Checks: return self.eyes_seen() and time.monotonic() - self.svc.last_sample < 2 def on_gaze_on(self): + self.full_armed = True + self.need_full("gaze mode came on without a calibration") + + def need_full(self, reason): + """Gaze mode is on without a calibration: open the full one, once per arming (see the top), + or note why it can't open.""" cal = self.calibrated() - if cal is False: - self.start("full", "gaze mode came on without a calibration") - elif cal is None: - self.want_full_until = time.monotonic() + 20 + if cal: + self.full_armed = True # missing one later (the other tracker picked) is news again + if not self.gaze_on or self.check or not self.full_armed or cal is not False: + self.full_blocked = None + return + now = time.monotonic() + if not self.can_run() and self.svc.waking(): + return # the tracker is still starting (the service idled) + if not self.can_run(): + why = ("the eye tracker isn't sending" if now - self.svc.last_sample >= 2 + else "no eyes seen (is the headset on?)") + elif now < self.full_retry_at: + return + else: + reply = self.start("full", reason) + why = None if reply == "ok" else reply.removeprefix("error ") + if why: + self.full_retry_at = now + FULL_RETRY + if why and why != self.full_blocked: + log(f"the calibration can't open: {why}") + self.full_blocked = why + if not why: + self.full_armed = False + + def problem(self): + """Why gaze mode, on, can't follow your eyes yet, or None. Our tracker not having said + yet is None: Input Settings has its own line for our tracker.""" + if not self.gaze_on or self.calibrated() is not False: + return None + if self.check and self.check["kind"] == "full": + return "Not calibrated yet: the calibration is open in the headset" + if self.full_blocked: + return f"Not calibrated, and the calibration can't open: {self.full_blocked}" + if not self.full_armed: + if self.full_failed: + return f"Not calibrated: the calibration failed (most dots: {self.full_failed}). Use Calibrate" + return "Not calibrated: the calibration closed unfinished. Use Calibrate" + return "Not calibrated: the calibration opens in the headset" def auto_quick(self, reason): now = time.monotonic() @@ -318,7 +416,7 @@ class Checks: now = time.monotonic() self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0, "started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0, - "skipped": 0, "captured": 0, "points": {}} + "skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": ""} log(f"{kind} check: {reason}") if kind == "full": st = ask(SCREENS, "state", 0.5).split() @@ -402,10 +500,12 @@ class Checks: self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look") self.last_progress = None c["shown"] = time.monotonic() - c["run"], c["accept"], c["done_at"] = [], False, None + c["run"], c["accept"], c["done_at"], c["accept_at"] = [], False, None, None def on_sample(self, s): self.sample_at = time.monotonic() + if self.pending: + self.run_pending() unc = (s["src"].get("mmap1") or {}).get("unc") if unc and min(unc) <= EYE_LOST: self.seen_at = time.monotonic() @@ -428,6 +528,7 @@ class Checks: if "hy" not in src: return now = time.monotonic() + c["gaze_at"] = now if now < c["shown"] + CHECK_SETTLE: return g = (src["hy"], src["hp"]) @@ -488,7 +589,9 @@ class Checks: if ok: svc.weights["own"].add(miss) else: - steady = steady_samples(samples) + why = {} + steady = steady_samples(samples, why=why) + rec["dropped"] = why reads = {} for name in ("action", "mmap1", "mmap2", "left", "right"): pts = [(smp["src"][name]["hy"], smp["src"][name]["hp"]) for smp in steady @@ -522,23 +625,41 @@ class Checks: if svc.kind == "eyes": svc.weights["steam"].add(miss) svc.dirty = True + if not ok: + short, long, fit = reject_reason(rec.get("reply", "") if c["own"] else None, rec.get("dropped")) + rec["reason"] = long + c["reasons"][long] = c["reasons"].get(long, 0) + 1 + if fit: + c["fit_reasons"].add(long) rec["accepted"] = ok self.log_check(rec) if not ok: c["tries"] += 1 - log(f"{c['kind']} check dot {c['i'] + 1}: not taken ({rec.get('reply', '')})") - if c["tries"] >= 2 or c["kind"] != "full": + log(f"{c['kind']} check dot {c['i'] + 1}: not taken: {rec['reason']} ({rec.get('reply', '')})") + full = c["kind"] == "full" + if c["tries"] >= 2 or not full: + self.note(f"Dot skipped: {long}" + (f" ({FIT_HINT})" if fit else "") if full else f"Skipped: {short}") self.skip() else: + self.note(f"Not taken: {long}. Look at the dot and click again") self.to_panel(f"dot {yaw:.3f} {pitch:.3f} fail") - c["run"], c["accept"] = [], False + c["run"], c["accept"], c["accept_at"] = [], False, None c["shown"] = time.monotonic() # settle again, then retry return c["captured"] += 1 c["tries"] = 0 + self.note("") self.to_panel(f"dot {yaw:.3f} {pitch:.3f} done") c["done_at"] = time.monotonic() + DONE_PAUSE + def note(self, text): + """The panel's warning line, over the instructions (empty: none). It stays until the + next dot is taken, so a skipped dot's reason is still there at the one after it.""" + c = self.check + if c.get("note") != text: + c["note"] = text + self.to_panel(f"note {text}".rstrip()) + def skip(self): c = self.check yaw, pitch, _ = c["dots"][c["i"]] @@ -567,12 +688,20 @@ class Checks: return n = len(c["dots"]) if c["captured"] < n * 2 / 3: - log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays") + reasons = c["reasons"] + main = max(reasons, key=reasons.get) if reasons else None + self.full_failed = main + log(f"calibration failed: only {c['captured']} of {n} dots; the old one stays" + + (f". Not taken: {', '.join(f'{r} ({k})' for r, k in reasons.items())}" if reasons else "")) self.to_panel(f"text Calibration failed: only {c['captured']} of {n} dots. Try again from Input Settings") + if main: + fit = main in c["fit_reasons"] + self.note(f"Most dots: {main}" + (". Check headset fit on the Gaze page" if fit else "")) self.to_panel("dot 0 0 off") - c["done_at"] = time.monotonic() + 3.0 + c["done_at"] = time.monotonic() + (8.0 if main else 3.0) # time to read why c["closing"] = True return + self.full_failed = None if c["own"]: reply = ask(EYES, "calib-fit", 10.0) log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}") @@ -629,9 +758,28 @@ class Checks: # --- From the service --- - def command(self, words): + def run_pending(self): + """A check asked for while the service idled: once the tracker sends (and our tracker has + said whether it's calibrated), or WAKE_SETTLE after waking, when its error is the real one.""" + svc = self.svc + words, _ = self.pending + ready = (time.monotonic() - svc.last_sample < 2 if words[0] == "fitcheck" else self.can_run()) \ + and (svc.kind != "own" or self.calibrated() is not None) + if not ready and svc.waking(): + return + self.pending = None + reply = self.command(words, queue=False) + if reply != "ok": + log(f"{words[0]}, asked for while idle: {reply.removeprefix('error ')}") + + def command(self, words, queue=True): """quickcal, calibrate, calaccept, calquit -> a reply.""" cmd = words[0] + if cmd in ("quickcal", "calibrate", "fitcheck") and queue and self.svc.waking() and not self.check: + # The tracker isn't running (or only just started): wake it, and do this once it sends. + self.pending = (words, time.monotonic()) + self.svc.update_awake() + return "ok waking the eye tracker first" if cmd == "quickcal": return self.start("full" if self.calibrated() is False else "quick", "asked for") if cmd == "calibrate": @@ -642,6 +790,8 @@ class Checks: if self.check and self.check["kind"] == "fit": self.check["fit"].toggle_guide(time.monotonic()) elif self.check: + if not self.check["accept"]: + self.check["accept_at"] = time.monotonic() self.check["accept"] = True return "ok" if cmd == "calquit": @@ -688,6 +838,13 @@ class Checks: return if c["done_at"]: return + if c["accept"] and c["accept_at"] and now - c["accept_at"] > ACCEPT_WAIT: + # Clicked, but no capture yet (see on_sample): say what it's waiting for. + full = c["kind"] == "full" + if now - c.get("gaze_at", 0.0) > 0.5: + self.note("Waiting: the eye tracker isn't sending a gaze" if full else "Waiting: no gaze") + else: + self.note("Waiting for your gaze to hold still on the dot" if full else "Hold your look still") if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT: self.close("ignored") elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE: @@ -698,23 +855,20 @@ class Checks: now = time.monotonic() if not self.panel_proc and now >= self.panel_restart_at: self.start_panel() - self.to_helper("gaze ?") + self.to_helper("gaze ? headset") if now - self.gaze_heard > 5: - self.gaze_on = None # the helper isn't answering + self.gaze_on = self.headset = None # the helper isn't answering + if self.pending: + self.run_pending() if self.check: self.to_helper("calpanel 1") - if self.want_full_until: - cal = self.calibrated() - if cal is not None or now > self.want_full_until: - self.want_full_until = 0.0 - if cal is False and self.gaze_on: - self.start("full", "gaze mode came on without a calibration") if not self.eyes_seen(AWAY_MIN): self.away, self.back_since = True, None elif self.back_since is not None and not self.eyes_seen(): self.back_since = None # gone again before DON_DELAY elif self.back_since is not None and now - self.back_since >= DON_DELAY: self.away, self.back_since = False, None + self.full_armed = True # a calibration that closed unfinished opens again self.auto_quick("the headset went on") if svc.kind == "own" and now - svc.own_at < 5: reseat = any(e.get("reseat") for e in (svc.own.get("eyes") or {}).values()) @@ -723,10 +877,13 @@ class Checks: elif not self.reseat_seen and self.can_run(): self.reseat_seen = True self.auto_quick("our tracker asked for a click") + self.need_full("gaze mode is on without a calibration") def status(self): c = self.check - st = {"check": None, "gaze_mode": self.gaze_on, "calibrated": self.calibrated(), "eyes": self.eyes_seen(), + st = {"check": None, "gaze_mode": self.gaze_on, "headset_worn": self.headset, "pending": self.pending[0][0] + if self.pending else None, "calibrated": self.calibrated(), "eyes": self.eyes_seen(), + "problem": self.problem(), "panel": self.panel_proc is not None, "last_quick_s": round(time.monotonic() - self.last_quick) if self.last_quick else None} if c: diff --git a/gaze/panel/ft-gazepanel.cpp b/gaze/panel/ft-gazepanel.cpp index 374694e..cbe888b 100644 --- a/gaze/panel/ft-gazepanel.cpp +++ b/gaze/panel/ft-gazepanel.cpp @@ -22,6 +22,8 @@ // look, capture (a ring filling to progress), done, // fail, off // title / text a line at the top / at the bottom (empty to clear) +// note a warning line just above the bottom one, in orange (red on +// the bright round): why a dot wasn't taken (empty to clear) // eye <0|1> // fit: an eye's card (0 left): its state in that colour, the // tracker's signal, the share of the last 10 s it was seen @@ -45,6 +47,7 @@ #include #include #include +#include #include #include #include @@ -143,7 +146,7 @@ struct Panel { double wDeg = kQuickDeg; // across std::vector px; double bg = 0.05; - std::string title, text; + std::string title, text, note; bool dotOn = false; double dotYaw = 0, dotPitch = 0, progress = 0; std::string state = "off"; @@ -268,6 +271,9 @@ void Draw(Panel &p) { const int titleSize = int(pxPerDeg * (p.full ? 1.5 : 1.1)), textSize = int(pxPerDeg * (p.full ? 1.2 : 0.9)); Text(p, p.title, titleSize, p.w / 2, int(titleSize * 1.2), faint); Text(p, p.text, textSize, p.w / 2, p.h - int(textSize * 1.3), faint); + // Under the full calibration's lowest dots (RING degrees down) and over the text. + if (light) Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 0.7, 0.12, 0.05); + else Text(p, p.note, textSize, p.w / 2, p.h - int(textSize * 2.8), 1.0, 0.62, 0.3); if (p.fit) DrawFit(p, pxPerDeg, textSize, faint); if (!p.dotOn || p.state == "off") return; double x, y; @@ -468,7 +474,7 @@ int main(int argc, char **argv) { p.w = p.full ? kFullW : p.fit ? kFitW : kQuickPx; p.h = p.full ? kFullH : p.fit ? kFitH : kQuickPx; p.wDeg = p.full ? kFullDeg : p.fit ? kFitDeg : kQuickDeg; - p.title.clear(), p.text.clear(), p.dotOn = false, p.state = "off"; + p.title.clear(), p.text.clear(), p.note.clear(), p.dotOn = false, p.state = "off"; p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear(); place(); visible = dirty = true; // shown with its first picture @@ -496,6 +502,8 @@ int main(int argc, char **argv) { p.title = buf[5] == ' ' ? buf + 6 : "", dirty = true; } else if (!std::strncmp(buf, "text", 4)) { p.text = buf[4] == ' ' ? buf + 5 : "", dirty = true; + } else if (!std::strncmp(buf, "note", 4)) { + p.note = buf[4] == ' ' ? buf + 5 : "", dirty = true; } else if (std::sscanf(buf, "%15s", word) == 1 && !std::strcmp(word, "ping")) { reply = visible ? "ok shown" : "ok hidden"; } else { @@ -517,7 +525,11 @@ int main(int argc, char **argv) { if (!shown) ov->ShowOverlay(h), shown = true; dirty = false; } - std::this_thread::sleep_for(std::chrono::milliseconds(visible ? 10 : 50)); + // Until a command comes, or 10 ms while shown (SteamVR's events). Hidden, it waits up to a + // second: it woke 20 to 30 times a second for nothing, the main cost left with gaze idle. + // A closed stdin (--watch-stdin) wakes it too, so quitting doesn't wait. + pollfd fds[2] = {{sock, POLLIN, 0}, {0, POLLIN, 0}}; + poll(fds, watchStdin ? 2 : 1, visible ? 10 : 1000); } ov->DestroyOverlay(h); buffers.Drop(); diff --git a/gaze/test/idle-test.py b/gaze/test/idle-test.py new file mode 100755 index 0000000..145becb --- /dev/null +++ b/gaze/test/idle-test.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Offline test of the gaze service idling (gaze/ft-gazed, gaze/gazecheck.py): ft-gaze runs only +while the gaze is in use, and a check asked for while it idles waits for the tracker. + +Runs ft-gazed's Service with its sockets renamed, a fake pointer helper (answers "gaze ? +headset" as the test says), and a fake ft-gaze (prints samples, quits when its stdin closes). +SteamVR's tracker is the one in use, so our own isn't started; the panel isn't either. Nothing +reaches the live gaze service, the pointer helper, or SteamVR, so it's safe next to them. + + gaze/test/idle-test.py +""" +import importlib.machinery +import importlib.util +import os +import socket +import subprocess +import sys +import tempfile +import threading +import time + +HERE = os.path.dirname(os.path.abspath(__file__)) +GAZE = os.path.join(HERE, "..") +sys.path.insert(0, GAZE) +loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(GAZE, "ft-gazed")) +gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader)) +loader.exec_module(gazed) +import gazecheck # noqa: E402 (the module ft-gazed imported) + +tag = f"ft_gaze_idle_test_{os.getpid()}" +gazed.ME = f"\0{tag}_gazed" +gazed.POINTER = gazecheck.POINTER = f"\0{tag}_helper" +gazecheck.SCREENS = f"\0{tag}_screens" +gazecheck.PANEL_PROG = gazed.REPO / "nonexistent-panel" # "isn't built": no panel +gazed.IDLE_AFTER, gazed.WAKE_SETTLE = 1.0, 3.0 +gazed.read_settings = lambda: ("steam", "steam", "auto", 55.0) +logs = [] +gazed.log = gazecheck.log = lambda msg: logs.append(msg) + +# The fake ft-gaze: 90 samples a second, both eyes seen, until its stdin closes. It logs the +# sources it was asked for: "argv LIST" (--sources), then "line LIST" for each "sources LIST". +tmp = tempfile.mkdtemp(prefix="ft-gaze-idle-test-") +FAKE = os.path.join(tmp, "ft-gaze") +SOURCES_LOG = os.path.join(tmp, "sources.log") +with open(FAKE, "w") as f: + f.write('''import json, os, select, sys, time +log = open(sys.argv[1], "a", buffering=1) +log.write("argv " + (sys.argv[sys.argv.index("--sources") + 1] if "--sources" in sys.argv else "-") + "\\n") +buf = b"" +while True: + if select.select([sys.stdin], [], [], 1 / 90)[0]: + data = os.read(0, 4096) + if not data: + break + buf += data + while b"\\n" in buf: + line, buf = buf.split(b"\\n", 1) + if line.startswith(b"sources "): + log.write("line " + line[8:].decode() + "\\n") + eye = {"hy": 1.0, "hp": 2.0} + print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001], + "open": [0.8, 0.8]}, "left": eye, "right": eye}}), flush=True) +''') +started = [] + + +def start_helper(self): + """ft-gaze, straight from here instead of the dev container.""" + import selectors + self.proc = subprocess.Popen([sys.executable, FAKE, SOURCES_LOG, "--sources", self.wanted_sources()], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + self.proc_sources = self.wanted_sources() + os.set_blocking(self.proc.stdout.fileno(), False) + os.set_blocking(self.proc.stderr.fileno(), False) + self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout") + self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr") + self.buf = b"" + started.append(time.monotonic()) + + +gazed.Service.start_helper = start_helper + +# The fake pointer helper. +helper_state = {"reply": "ok off worn"} +helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) +helper.bind(gazed.POINTER) +helper.settimeout(0.2) + + +def answer(): + while True: + try: + data, addr = helper.recvfrom(512) + except socket.timeout: + continue + except OSError: + return + if data.startswith(b"gaze ?") and addr: + helper.sendto(helper_state["reply"].encode(), addr) + + +threading.Thread(target=answer, daemon=True).start() +svc = gazed.Service(None, False, gazed.POINTER) +threading.Thread(target=svc.run, daemon=True).start() +ctl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) +ctl.bind("") +ctl.settimeout(2) + + +def ask(cmd): + ctl.sendto(cmd.encode(), gazed.ME) + return ctl.recv(4096).decode() + + +failures = [] + + +def check(label, got, want): + ok = got == want + print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True) + if not ok: + failures.append(label) + + +def wait(cond, seconds): + end = time.monotonic() + seconds + while time.monotonic() < end: + if cond(): + return True + time.sleep(0.05) + return cond() + + +time.sleep(2.5) +check("gaze mode off: idle, no ft-gaze", (svc.awake, svc.proc is None), (False, True)) +check("status says why", ask("status").count('"idle": "gaze mode is off"'), 1) + +helper_state["reply"] = "ok on worn" +check("gaze mode on, headset worn: awake within 2 s", wait(lambda: svc.awake and svc.proc is not None, 2.5), True) +check("samples come in", wait(lambda: svc.counts["samples"] > 20, 2), True) + +helper_state["reply"] = "ok on away" +check("headset off: still awake for IDLE_AFTER", wait(lambda: not svc.awake, 0.5), False) +check("then idle, ft-gaze stopped", wait(lambda: not svc.awake and svc.proc is None, 3.5), True) +check("why: nobody wears it", ask("status").count('"idle": "nobody is wearing the headset"'), 1) + +helper_state["reply"] = "ok on" +check("an older helper (no headset word): awake with gaze mode on", wait(lambda: svc.awake, 2.5), True) +helper_state["reply"] = "ok off worn" +check("gaze mode off again: idle", wait(lambda: not svc.awake and svc.proc is None, 4.5), True) + +check("wake lease", ask("wake 2"), "ok") +check("wake: awake at once", (svc.awake, wait(lambda: svc.proc is not None, 1)), (True, True)) +check("lease over (2 s + IDLE_AFTER): idle", wait(lambda: not svc.awake, 4.5), True) + +time.sleep(0.5) +logs.clear() +open(SOURCES_LOG, "w").close() +count = len(started) +check("quick check while idle: queued, waking", ask("quickcal"), "ok waking the eye tracker first") +check("it woke", wait(lambda: svc.awake and len(started) > count, 1.5), True) +check("once the tracker sends, it ran (and says why it couldn't open)", + wait(lambda: any("quickcal, asked for while idle: the panel isn't running" in m for m in logs), 3), True) +check("nothing left pending", svc.checks.pending, None) +sources = open(SOURCES_LOG).read().split("\n") +check("ft-gaze started with every source for the check", sources[0], "argv all") +in_use = svc.wanted_sources() +check("then only those in use (SteamVR's tracker: not the action, not own)", + (f"line {in_use}" in sources, in_use != "all", "action" in in_use, "own" in in_use), (True, True, False, False)) +check("idle again after", wait(lambda: not svc.awake, 3), True) + +print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True) +svc.running = False +time.sleep(0.7) +os.remove(FAKE) +os.remove(SOURCES_LOG) +os.rmdir(tmp) +os._exit(1 if failures else 0) diff --git a/gaze/tracker/eyes_pupil.py b/gaze/tracker/eyes_pupil.py index 8884363..cf3672d 100644 --- a/gaze/tracker/eyes_pupil.py +++ b/gaze/tracker/eyes_pupil.py @@ -9,7 +9,12 @@ the search runs again with a smaller closing. import cv2 import numpy as np -DARK = 30 # pupil pixels are below this (the face around it is 40-180) +# One thread: OpenCV's pool of one per core costs more than it saves on a frame this small +# (a 140-240 px window while it follows the pupil). Its idle workers spun and yielded about +# 14,000 times a second each, a quarter of a core, beside SteamVR's compositor. +cv2.setNumThreads(1) + +DARK = 30 # pupil pixels are below this (the face around it is 40-180) MIN_AREA = 150 # pupil area range in pixels MAX_AREA = 20000 MIN_FILL = 0.75 # blob area / fitted-ellipse area diff --git a/gaze/tracker/findings.md b/gaze/tracker/findings.md index d5e36f9..3d9a932 100644 --- a/gaze/tracker/findings.md +++ b/gaze/tracker/findings.md @@ -63,6 +63,13 @@ reading only. frame when its camera starts the frame after next (no slot is rewritten sooner than three frames), ignores late writes to the frame just finished, and saves from a separate thread. fit1 may hold a few percent of torn frames. + - `--share` (2026-10-03) checks only the slot each camera writes next, from the two before + (the orders above, learned again if they change; all four slots for 2 s after a frame turns + up elsewhere), sleeps until 2.5 ms before the next frame is due, then looks every 1 ms with + 0.5 ms of timer slack. Against the 0.3 ms poll of all eight slots, on simulated cameras: + 207 wakeups a second instead of 1,486, 0.8% of a core instead of 3.6% (more on the real + DMA-BUF memory), no torn or skipped frames, and a frame's start seen 1.35 ms late on + average instead of 0.76. `--rec` still polls all slots every 0.3 ms, for its times. - Eye tracking stops when the headset is off ("HMD off, stopping eye tracking"), so recordings are empty then. - **Which camera is which eye** (capture fit1, 2026-09-29, closing one eye at a time): diff --git a/gaze/tracker/ft-eyegrab.c b/gaze/tracker/ft-eyegrab.c index 6cc31c8..a976c41 100644 --- a/gaze/tracker/ft-eyegrab.c +++ b/gaze/tracker/ft-eyegrab.c @@ -46,6 +46,7 @@ #include #include #include +#include #include #include #include @@ -288,20 +289,53 @@ static int eye_buffer(void) { // Calls done(frame, slot, time) for every complete eye-camera frame until `seconds` pass // (forever if negative), a stop signal comes, the tracker process goes away, or keep() -// (checked about every 0.25 s, when given) says to stop. +// (checked about every 0.25 s, when given) says to stop. Looks every `poll_us`. // // A frame lands over several milliseconds, in bursts, and its last bursts can come after // the camera has started its next frame. A slot isn't rewritten until at least three frames // later (camera 0 cycles 3,0,1,2; camera 1 7,5,4,6,5,7,6,4), so a frame is passed on when // its camera starts the frame after next. Changes to the slot just finished are late bursts, // not a new frame. A frame's time is when its slot first changed. +// +// Each look checks only the slot each camera writes next: the one that followed the last two +// before (after[][], seeded with the orders above and learned as frames come). Every check +// reads 256 words spread over a frame in DMA-BUF memory, so all eight slots each time was +// most of the cost. A camera with nothing in its expected slot for 1.5 frames, or with no +// order known yet, has all its slots checked until a frame comes. When that finds a frame +// somewhere else (the order changed, or a frame was missed), all its slots are checked for +// SCAN_AFTER_MISS, as before, while after[][] learns the new order. A slot's fingerprint is +// taken again when it stops being one of the two in use, so a check later sees only a new frame. +// Between frames it sleeps until FRAME_EARLY before the next one is due (the cameras run at +// 90 fps, within a ms of each other), then looks every `poll_us`. +#define FRAME_DUE (1.5 / 90) // s: a camera that hasn't started a frame by then gets all its slots checked +#define SCAN_AFTER_MISS 2.0 // s of checking all slots after a frame came in an unexpected one +#define CAMS_IDLE 0.5 // s without a frame from either camera: look 4 times less often +#define FRAME_EARLY 0.0025 // s before a frame is due to start looking for it + +// When to start looking for the next frame: FRAME_EARLY before the first camera's is due. A +// camera already late (it lost its order, or stopped) means now. +static double next_due(const double last[2], double t) { + double due = 1e300; + for (int cam = 0; cam < 2; cam++) { + double d = last[cam] + 1.0 / 90 - FRAME_EARLY; + if (t - last[cam] > CAMS_IDLE) continue; // stopped: the other one sets the pace + if (d < due) due = d; + } + return due < 1e300 ? due : t; +} + static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8_t *, int, double), - int (*keep)(void)) { + int (*keep)(void), unsigned poll_us) { + static const int order[2][8] = {{3, 0, 1, 2, 3, 0, 1, 2}, {7, 5, 4, 6, 5, 7, 6, 4}}; + int after[EYE_SLOTS][EYE_SLOTS]; + memset(after, -1, sizeof after); + for (int c = 0; c < 2; c++) + for (int i = 0; i < 8; i++) after[order[c][i]][order[c][(i + 1) % 8]] = order[c][(i + 2) % 8]; uint64_t sig[EYE_SLOTS]; double first[EYE_SLOTS]; int cur[2] = {-1, -1}, prev[2] = {-1, -1}; for (int k = 0; k < EYE_SLOTS; k++) sig[k] = frame_sig(bufs[b].p + slot_start(k)), first[k] = 0; - double start = now(), checked = start, kept = start; + double start = now(), checked = start, kept = start, last[2] = {start, start}, scan_until[2] = {0, 0}; char proc[64]; snprintf(proc, sizeof proc, "/proc/%d", pid); while ((seconds < 0 || now() - start < seconds) && !stop_rec) { @@ -315,18 +349,36 @@ static void poll_frames(int b, double seconds, int pid, void (*done)(const uint8 if (!keep()) return; kept = t; } - for (int k = 0; k < EYE_SLOTS; k++) { - uint64_t s = frame_sig(bufs[b].p + slot_start(k)); - if (s == sig[k]) continue; - sig[k] = s; - int cam = k >= 4; - if (k == cur[cam] || k == prev[cam]) continue; // landing, or a late burst - if (prev[cam] >= 0) done(bufs[b].p + slot_start(prev[cam]), prev[cam], first[prev[cam]]); - prev[cam] = cur[cam]; - cur[cam] = k; - first[k] = t; + for (int cam = 0; cam < 2; cam++) { + int next = prev[cam] >= 0 ? after[prev[cam]][cur[cam]] : -1; + int all = next < 0 || t - last[cam] > FRAME_DUE || t < scan_until[cam]; + int from = all ? cam * 4 : next, to = all ? cam * 4 + 4 : next + 1; + for (int k = from; k < to; k++) { + uint64_t s = frame_sig(bufs[b].p + slot_start(k)); + if (s == sig[k]) continue; + sig[k] = s; + if (k == cur[cam] || k == prev[cam]) continue; // landing, or a late burst + if (next >= 0 && k != next) scan_until[cam] = t + SCAN_AFTER_MISS; + if (prev[cam] >= 0) { + done(bufs[b].p + slot_start(prev[cam]), prev[cam], first[prev[cam]]); + after[prev[cam]][cur[cam]] = k; + // Out of use from now on: later changes are a new frame. + sig[prev[cam]] = frame_sig(bufs[b].p + slot_start(prev[cam])); + } + prev[cam] = cur[cam]; + cur[cam] = k; + first[k] = t; + last[cam] = t; + next = prev[cam] >= 0 ? after[prev[cam]][cur[cam]] : -1; + } } - usleep(300); + double wait = poll_us * 1e-6; + if (t - last[0] > CAMS_IDLE && t - last[1] > CAMS_IDLE) { + wait *= 4; + } else if (next_due(last, t) - t > wait) { + wait = next_due(last, t) - t; + } + usleep((useconds_t)(wait * 1e6)); } } @@ -356,7 +408,7 @@ static int rec(double seconds, const char *dir, int pid) { pthread_t writer; pthread_create(&writer, NULL, ring_writer, NULL); double start = now(); - poll_frames(b, seconds, pid, rec_frame, NULL); + poll_frames(b, seconds, pid, rec_frame, NULL, 300); // 0.3 ms: recordings' times pthread_mutex_lock(&ring.mu); ring.done = 1; pthread_cond_signal(&ring.cv); @@ -383,6 +435,10 @@ static int rec(double seconds, const char *dir, int pid) { #define SHARE_SLOTS 8 #define SHARE_MAGIC 0x31434546u // "FEC1" #define WANT_FRESH 3.0 // seconds a touch of the --want file lasts +// How often --share looks for frames, with a timer slack that lets the kernel group the +// wakeups: a frame reaches ft-eyes 1 to 1.5 ms after its camera starts the next, not 0.3. +#define SHARE_POLL_US 1000 +#define SHARE_SLACK_NS 500000 typedef struct { uint32_t magic, version, width, height, slots, entry_size; @@ -462,6 +518,7 @@ static int share_loop(const char *path) { .slots = SHARE_SLOTS, .entry_size = (uint32_t)esize}; signal(SIGINT, on_stop); signal(SIGTERM, on_stop); + if (prctl(PR_SET_TIMERSLACK, SHARE_SLACK_NS, 0, 0, 0) != 0) perror("PR_SET_TIMERSLACK"); fprintf(stderr, "ft-eyegrab: sharing frames in %s%s%s\n", path, want_path ? " while wanted by " : "", want_path ? want_path : ""); int pid = -1, idle = -1, missing = 0; @@ -486,7 +543,7 @@ static int share_loop(const char *path) { if (idle != 0 || missing) fprintf(stderr, "ft-eyegrab: copying frames from eyetracking %d\n", pid); idle = 0, missing = 0; h->tracker_pid = pid; - poll_frames(eye_buffer(), -1, pid, share_frame, wanted); + poll_frames(eye_buffer(), -1, pid, share_frame, wanted, SHARE_POLL_US); struct stat st; char proc[64]; snprintf(proc, sizeof proc, "/proc/%d", pid); diff --git a/gaze/tracker/ft-eyes b/gaze/tracker/ft-eyes index 5e25799..871828e 100755 --- a/gaze/tracker/ft-eyes +++ b/gaze/tracker/ft-eyes @@ -50,6 +50,7 @@ import json import math import mmap import os +import select import socket import struct import sys @@ -58,7 +59,12 @@ import time from collections import deque from pathlib import Path -import numpy as np +# One thread for numpy's BLAS and OpenMP, set before numpy loads: it would start one per core +# (8 here) for small arrays that never need them. eyes_pupil keeps OpenCV to one as well. +for _var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS"): + os.environ.setdefault(_var, "1") + +import numpy as np # noqa: E402 sys.path.insert(0, str(Path(__file__).resolve().parent)) import eyes_model # noqa: E402 @@ -82,6 +88,15 @@ GAP = 3.0 # s without frames: the headset was off, and may sit diffe CLICK_BEFORE = 0.3 # a click's frames: the 300 ms before it (like the probe's fixation) CALIB_MIN = 15 # frames an eye needs in a calibration dot's window CALIB_SPREAD = 4.0 # px: more than this and the eye moved during the dot +# Waiting for frames. They come only as a counter in shared memory, so there's nothing to block +# on: sleep until a camera's next frame is due, then look every POLL. ft-eyegrab passes each one +# on within a ms or two of when its camera starts the one after, so they arrive 11.1 ms apart, +# give or take that. +PERIOD = 1 / 90 # s between a camera's frames +EARLY = 0.002 # start looking this long before a frame is due +POLL = 0.001 # then this often until it comes +STALLED = 0.1 # s without a frame: that camera stopped (headset off), and isn't waited for +IDLE_POLL = 0.02 # how often to look while both are stopped class Cams: @@ -352,7 +367,7 @@ def wait_for_cams(sock, tracker, want): return Cams() except (OSError, ValueError, RuntimeError): serve(sock, tracker) - time.sleep(0.2) + select.select([sock], [], [], 0.2) def serve(sock, tracker): @@ -372,7 +387,24 @@ def serve(sock, tracker): pass +def below_steamvr(): + """Nice 10 and SCHED_BATCH, for this thread and those it starts. ft-eyes runs in the dev + container's podman scope, where the gaze service's unit doesn't reach it, so it ran at + nice 0 on the cores vrcompositor and vrserver use. Batch also lets a waking ft-eyes wait + for the running task's turn instead of taking the core: a frame a few ms late costs the + gaze little, a late compositor frame costs a dropped frame in the headset.""" + try: + os.setpriority(os.PRIO_PROCESS, 0, max(os.getpriority(os.PRIO_PROCESS, 0), 10)) + except OSError as e: + print(f"ft-eyes: nice: {e}", file=sys.stderr, flush=True) + try: + os.sched_setscheduler(0, os.SCHED_BATCH, os.sched_param(0)) + except (OSError, AttributeError) as e: + print(f"ft-eyes: SCHED_BATCH: {e}", file=sys.stderr, flush=True) + + def main(): + below_steamvr() verbose = "-v" in sys.argv if "--watch-stdin" in sys.argv: # Run by ft-gazed through distrobox, which doesn't pass a stop on: quit when our @@ -398,6 +430,7 @@ def main(): print("ft-eyes: frames found, tracking", file=sys.stderr, flush=True) seen = [cams.count(0), cams.count(1)] report = time.monotonic() + arrived = [0.0, 0.0] # when each camera's newest frame was seen (monotonic) while True: serve(sock, tracker) want() @@ -407,6 +440,7 @@ def main(): if n == seen[c]: continue seen[c] = n + arrived[c] = time.monotonic() got = cams.frame(c, n - 1) # only the newest: never fall behind if got: eyes[c].feed(*got) @@ -425,8 +459,6 @@ def main(): shifts += [float(v) for v in e.shift.value] pupils += [e.gaze[3], e.gaze[4]] if fresh else [math.nan, math.nan] out.write(latest, (yaw, pitch), flags, per, shifts, pupils) - else: - time.sleep(0.001) now = time.monotonic() if now - report >= 5: if verbose: @@ -444,6 +476,12 @@ def main(): print("ft-eyes: frames went away; waiting", file=sys.stderr, flush=True) cams = wait_for_cams(sock, tracker, want) seen = [cams.count(0), cams.count(1)] + # Until the next frame is due (a command on the socket wakes us sooner). + wait = IDLE_POLL + for c in (0, 1): + if now - arrived[c] < STALLED: + wait = min(wait, max(arrived[c] + PERIOD - EARLY - now, POLL)) + select.select([sock], [], [], wait) if __name__ == "__main__": diff --git a/gaze/tracker/install.sh b/gaze/tracker/install.sh index 0cdf3c1..dec7611 100755 --- a/gaze/tracker/install.sh +++ b/gaze/tracker/install.sh @@ -2,8 +2,9 @@ # Install (or remove) the frame grabber our own eye tracker needs: ft-eyegrab, as the system # service frametop-eyegrab.service. It copies the eye-camera frames, read-only, out of # SteamVR's eyetracking process into /dev/shm/frametop-eyes-cams for ft-eyes, and only while -# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when Eye tracker -# is Own tracker or the gaze probe uses it. +# ft-eyes wants them. The gaze service (gaze/ft-gazed) runs ft-eyes itself, when ours is the +# tracker in use (GAZE_TRACKER=auto, the default, picks it once this is installed) or the gaze +# probe uses it. install.sh offers this after gaze mode. # Needs host sudo, for the binary (/etc/frametop/ft-eyegrab, root's) and the unit: it asks for # the password in the terminal, on the Frame or from a PC, or runs SUDO_ASKPASS when that's set # (frame_sudo in scripts/_env.sh, which also takes it from the repo's .env). diff --git a/hands/Makefile b/hands/Makefile index 6130d8c..a606775 100644 --- a/hands/Makefile +++ b/hands/Makefile @@ -1,6 +1,7 @@ # Hand tracking, built into build/ (hands/build.sh runs this in the dev container): # make ft-camd (camd/: runs on the host, so linked statically) and ft-hands (track/) # make tools ft-handreplay and ft-ringplay, for recordings +# make check builds and runs the C++ unit tests (tests/sides_test.cpp) # The first build fetches ncnn (NCNN_TAG) and builds it into build/ncnn, which takes a few # minutes. NCNN=DIR uses an ncnn install already built instead. NCNN_TAG = 20260526 @@ -11,7 +12,7 @@ CXXFLAGS += -std=c++17 -fopenmp -I$(NCNN)/include/ncnn LDLIBS = $(NCNN)/lib/libncnn.a -ljsoncpp -fopenmp -lpthread CAMD = camd/camd.c camd/tp.c camd/xrcams.c -TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp +TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp track/sides.cpp HDR = $(wildcard track/*.h) camd/fhring.h include/fh_hands.h include/fh_gestures.h all: build/ft-camd build/ft-hands @@ -29,6 +30,13 @@ build/ft-handreplay: track/replay.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a @mkdir -p build $(CXX) $(CXXFLAGS) -o $@ track/replay.cpp $(TRACK) $(LDLIBS) +build/sides-test: tests/sides_test.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a + @mkdir -p build + $(CXX) $(CXXFLAGS) -o $@ tests/sides_test.cpp $(TRACK) $(LDLIBS) + +check: build/sides-test + build/sides-test + build/ft-ringplay: track/ringplay.cpp track/record.h camd/fhring.h @mkdir -p build $(CXX) $(CXXFLAGS) -o $@ track/ringplay.cpp @@ -44,6 +52,6 @@ build/ncnn/install/lib/libncnn.a: cmake --build build/ncnn/build --target install > build/ncnn/build.log clean: - rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay + rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay build/sides-test -.PHONY: all tools clean +.PHONY: all tools check clean diff --git a/hands/README.md b/hands/README.md index 849d7e0..5b84901 100644 --- a/hands/README.md +++ b/hands/README.md @@ -14,6 +14,8 @@ Two programs, each a user service that stops when SteamVR does: They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` doesn't install it. +For the cutouts alone, `hands/ft-cutouts on` starts the same two programs with ft-hands' `--no-gestures`: your hands show through the screens, and no pinch or grip is detected, so nothing clicks or drags. It runs this checkout's build as transient user units, so it needs `hands/build.sh` and ft-camd's capabilities (`hands/run.sh caps`) but not `hands/run.sh install`. It and `ft-handsctl on` stop each other's services, and it stops with SteamVR too. + ``` ft-handsctl on | off # on the Frame: start or stop hand tracking (SteamVR must be running) ft-handsctl status # the services, and ft-hands' last status lines @@ -21,6 +23,8 @@ ft-handsctl log [lines] ft-handsctl cutouts on|off|state # ft-screens' hand cutouts, without stopping tracking ft-handsctl gestures # pinches and grips, live (tools/watch_gestures.py --distance) +hands/ft-cutouts on | off | status # the cutouts only: no pinches or grips + hands/run.sh install # build, give ft-camd its capabilities (sudo, once per build), install disabled hands/run.sh start|stop # start or stop the services hands/run.sh restart # after changing a setting @@ -32,7 +36,7 @@ hands/run.sh uninstall Settings in `~/.config/frametop.conf` (`FT_` in the environment overrides them), read when ft-camd and ft-hands start: -- `HANDS_SWAP_SIDES=1`: the two side cameras' names are swapped (see ft-camd below). Check with `tools/check_sides.py --ring`. +- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks and logs a warning if the hands disagree. - `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below). - `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras. - `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`. @@ -82,7 +86,14 @@ Options: It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. The service starts it again, and it attaches to the new buffers. -**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. With the headset on, looking at a room with some texture, `tools/check_sides.py --ring` says whether the names are right (exit 0), swapped (exit 3), or it can't tell (exit 2). When they're swapped, set `HANDS_SWAP_SIDES=1`. The colour cameras are video3 (`arcimx616 0-0010`) and video0 (`0-001a`); which of them is `passthrough_left` in the module's calibration is for `tools/check_color.py` to settle, on a recording with texture in view. +**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands now catches this by itself (`track/sides.h`, `HANDS_SWAP_SIDES=auto`): +- Whenever a hand's landmarks are found in two cameras at once (one of them a side camera), it intersects the rays through the 21 landmarks twice: once with the calibrations as named, once with the two side cameras exchanged. The same hand seen the right way meets within a few mm, in front of both cameras and as far away as its size says. The wrong way misses by centimetres or meets behind a camera. +- With the names wrong, the tracker never gets such pairs on its own: it hands the hand over to where the wrong calibration puts it and finds nothing there. So 5 times a second while undecided, the check places a tracked hand in 3D under the other naming and runs the landmark model where that puts it in the other side camera. +- It decides after 10 votes one way and none the other, or 20 with at most a fifth the other way, over at least 1 s. That takes about 1-2 s of hands in view. If the names are backwards, it exchanges them; the tracked views move with their images. Then it checks once more, more strictly. +- The log says what it found (`side cameras: SWAPPED, now exchanged after 3.2 s (votes ...)`). So does `/run/user/UID/frametop-hands/sides.json`, which the hand recorder reads. Recordings get a `DIR/sides.json` (hands/rec/sides.py has the rules). +- `--record-only` can't tell (it tracks nothing): it records ft-camd's names unless `--sides 0|1` says otherwise. + +`tools/check_sides.py --ring` is the independent check from the scene (ORB matches meeting under each naming): exit 0 as named, 3 swapped, 2 can't tell. `--pair upper` checks the upper pair the same way: in every recording so far (3 XRService starts, both side namings) the upper pair was named right. The colour cameras are video3 (`arcimx616 0-0010`) and video0 (`0-001a`); which of them is `passthrough_left` in the module's calibration is for `tools/check_color.py` to settle, on a recording with texture in view. ## ft-hands @@ -98,13 +109,14 @@ Options: - `--threads N`: model threads, pinned to the `--cpus` list. Default 3. - `--cpus LIST`: CPUs for the model threads and the main loop. Default `5,6,7` (`HANDS_CPUS`). SteamOS starts user processes on CPUs 0-4, and XRService's head tracking runs on 2-3. With the headset on, a step took 8.4 ms on 5-7 against 13.2 ms on 2-4, and SteamVR's frame timing didn't change (2026-09-29, three rounds of the same replayed frames). - `--contrast MODE` or `PALM/HAND`: how crops are equalized before the models see them: `clahe[:CLIP]`, `none`, or `stretch` (1st-99th percentile). Default `clahe:2/none`. In the dim recording, CLAHE let the palm search find about 10% more hands, but it made the landmarks jitter more (published median 6.9 mm, against 6.0 mm with plain landmark crops). -- `--swap-sides`: swap the two side cameras (`HANDS_SWAP_SIDES`, see ft-camd). +- `--sides auto|0|1`: which side camera is which (`HANDS_SWAP_SIDES`, see "Which camera is which"); `--swap-sides` is `--sides 1`. - `--seconds N`: stop after N seconds. - `--status S`: how often to print status, in seconds. - `--models DIR`: where the models are. - `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs. - `--no-publish`: don't write the hands and gestures files. -- `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-