diff --git a/docs/design.md b/docs/design.md index faccfd4..c51838c 100644 --- a/docs/design.md +++ b/docs/design.md @@ -91,7 +91,7 @@ A few overlays need special handling: Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping. -Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse or controller button mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze; the mouse is gaze mode's only pointer device for now. The dot shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. The gaze moving the pointer doesn't show it: you know where you're looking. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured. +Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured. Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place. diff --git a/docs/gaze-controllers.md b/docs/gaze-controllers.md new file mode 100644 index 0000000..8a39d3a --- /dev/null +++ b/docs/gaze-controllers.md @@ -0,0 +1,28 @@ +# Gaze with the controllers: a dead end + +Gaze mode is a mouse feature. On 2026-09-30 we tried to make the Frame controllers its buttons: with gaze mode on and no game running, either controller's trigger would click where you look (a tap clicks, moving the hand steers the pointer, holding still drags), the controllers' lasers would be muted, and SteamVR's dashboard would follow the gaze too. It can't be done cleanly, for the reasons below. The work is kept on the `gaze-first` branch, which isn't merged. + +## What worked + +- Our `ft_pointer` device can hold SteamVR's laser without a hand role, in the treadmill role. It has to hint that role when SteamVR activates it; a hint changed later never gets the `/user/treadmill` path. +- A trimmed copy of the Frame controller's compositor binding mutes the controllers' laser buttons. It's chosen with `POST /input/selectconfig.action` on vrserver's port 27062 (a JSON body; a form-encoded one gets "Parse failed"). The helper's global action sets can't do it, because SteamVR marks them inactive while its laser mouse has focus. +- vrserver's web socket on 127.0.0.1:27062 reports every controller component without taking it from anyone. +- Steam's UI can be kept from acting on the controllers by wrapping its gamepad input source (webpack module 17900) through Steam's CEF debugger. +- Our device takes the laser back 10 to 15 ms after a controller takes it. + +## What broke it + +- Steam reads the Frame controllers itself. They aren't devices on the host: vrserver owns their radio and passes their raw reports to Steam through SteamVR's private Steam interface. Steam's client library turns them into a virtual device ("SteamFrameVirtual", at `/steamvr/virtual`), outside every SteamVR binding. Steam's own SteamVR action manifest asks only for haptics. +- Every press and every release that Steam sees takes SteamVR's dashboard, and Frametop's panels, out of laser mode about 40 ms later. That happens whatever the bindings say and whatever Steam's UI does with the event. None of these stopped it: dropping the events in Steam's UI, removing the controller's `dualanalog` bindings, binding every button to a harmless compositor action, or setting `dashboard.modalGamepadAndLaser` to false. +- Taking the laser back after each switch leaves a gap of 20 to 40 ms, and panels treat it as the pointer leaving, so clicks and drags break. +- Steam can't be told to ignore the controllers. It won't save a controller layout for the virtual controller ("Saved Binding Selection Failed - No Identity"), and its menus don't go through the layout anyway: a live preview of the empty layout for Steam's UI (app 769) changed nothing. +- SteamVR hands the controllers to a VR app instead of Steam only while that app has the input focus, as games do. A dashboard overlay with overlay flag `1 << 4` is given the focus only in gamepad mode, and only for gamepad input. + +## Options not taken + +- Frametop as a transparent VR app (a scene application using OpenXR's alpha blend mode) whenever gaze mode is on. That would cut Steam off while the dashboard is closed, but Steam's dashboard pages would still take the controllers, and it costs a scene layer all the time. +- Patching Steam's running process, with an eBPF probe that writes its memory or an injected hook, to drop the controller reports while gaze mode is on. It would cover everything, but it changes Valve's software, needs Steam's client library reverse-engineered, and breaks with Steam updates. + +## Where the work is + +Branch `gaze-first`: the plan and the test log (`docs/gaze-first.md`), the probes (`pointer/probe/lasertest`, `focustest`, `vrsetting`), the web socket reader (`input/vrws.py`), the filter for Steam's UI (`input/steam-gamepad-filter.js`), and the relay's and helper's controller code. Only the gaze dot setting (`POINTER_GAZE_DOT`) came over. diff --git a/docs/reference.md b/docs/reference.md index 62b8f1a..9a5bc2d 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -104,7 +104,7 @@ pointer/helper/run.sh status | log | restart pointer/driver/install.sh probe # devices, hand roles, who owns the dashboard pointer ``` -The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, and `POINTER_GAZE_SHOW`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes). +The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, and `POINTER_GAZE_MOUSE`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes). ## Frametop Input Settings diff --git a/gaze/README.md b/gaze/README.md index 6ef54fb..d8a5258 100644 --- a/gaze/README.md +++ b/gaze/README.md @@ -27,7 +27,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery. Looks down past the screens (under 20 degrees down, on no Frametop screen: a glance at the keyboard) aren't sent, so the pointer stays where it was instead of following you down, and eyes lost there aren't counted. It drops blinks (both eyes closing or lost), smooths with a fixation lock, and sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset. -- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse or controller button mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on. The dot only shows while the mouse moves it, while a press is held, and as a pulse when you click. +- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse button or key combination mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on, until a controller is picked up. The dot shows all the time (`POINTER_GAZE_DOT=moving`: only while the mouse moves it, while a press is held, and as a pulse when you click). Gaze mode works with the mouse only; [docs/gaze-controllers.md](../docs/gaze-controllers.md) explains why the controllers can't click at the gaze. - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the probe's one-dot check does the same). The helper only sends nudges up to `POINTER_GAZE_NUDGE_MAX` (8 degrees), and right after putting the headset back on our tracker can be further off than that. If so, raise it for a moment, or do the probe's check. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them. - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always. diff --git a/input-settings/ft_input_settings.py b/input-settings/ft_input_settings.py index d88ef15..d44e12d 100644 --- a/input-settings/ft_input_settings.py +++ b/input-settings/ft_input_settings.py @@ -6,9 +6,9 @@ to the input relay over its control socket (@frametop_relay): - Devices: every USB/Bluetooth mouse and keyboard, a live activity light to identify them, and a role for each (3D pointer, pass through, ignore). - Buttons: press a button or key on a pointer device, then pick an action. - - Controllers: the same for the Frame controllers' buttons. They're read by the pointer - helper through SteamVR input (@ft_pointer_helper: vrstatus, vrglobal), and a mapped - button is taken from games. + - Controllers: the same for the Frame controllers' buttons, minus the gaze actions (gaze + mode is a mouse feature). They're read by the pointer helper through SteamVR input + (@ft_pointer_helper: vrstatus, vrglobal), and a mapped button is taken from games. - Pointer: speed, dot size, distance and the rest, applied live. - Ignored panels: SteamVR overlays the pointer passes through (POINTER_IGNORE), by app or one by one. The helper lists them (@ft_pointer_helper "overlays"). @@ -87,16 +87,14 @@ CONTROLLER_BUTTONS = { "right/bumper": "Right bumper", "right/trigger": "Right trigger", "right/grip": "Right grip", "right/thumbstick": "Right stick click", } -CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none")] +# Not the gaze actions: gaze mode is a mouse feature (docs/gaze-controllers.md; the relay's GAZE_ACTIONS). +CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none", "gaze_toggle", "gaze_precision", "gaze_drag")] # Gaze mode settings (pointer helper), like POINTER_SETTINGS. GAZE_SETTINGS = [ ("POINTER_GAZE_RETAKE", "Look away to hand back", 5, 1, 45, 0.5, "°"), ("POINTER_GAZE_NUDGE_MAX", "Largest nudge to learn", 8, 1, 30, 0.5, "°"), ("POINTER_GAZE_HOLD", "Hold still to drag", 0.5, 0.1, 2.0, 0.05, "s"), ("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"), - ("POINTER_PRECISION_GAIN", "Precision steering", 0.5, 0.1, 2.0, 0.05, "×"), - ("POINTER_PRECISION_DEADZONE", "Precision dead zone", 0.3, 0.0, 3.0, 0.1, "°"), - ("POINTER_GAZE_DRAG_GAIN", "Drag steering", 1.0, 0.1, 2.0, 0.05, "×"), ] # In gaze mode, what the mouse's left button does (POINTER_GAZE_MOUSE). GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, release to click", @@ -809,6 +807,17 @@ class Backend(QObject): self.pointerChanged.emit() self.message.emit(f"Mouse in gaze mode: {GAZE_MOUSE[mode].lower()}", False) + @Property(bool, notify=pointerChanged) + def gazeDotAlways(self): + return read_conf().get("POINTER_GAZE_DOT", "always") != "moving" + + @Slot(bool) + def setGazeDotAlways(self, on): + write_conf_value("POINTER_GAZE_DOT", "always" if on else "moving") + self.reload_timer.start() + self.pointerChanged.emit() + self.message.emit("Gaze dot: " + ("always shown" if on else "shown only while the mouse moves it"), False) + @Property(str, notify=pointerChanged) def pointerRole(self): v = read_conf().get("POINTER_ROLE", "right") diff --git a/input-settings/main.qml b/input-settings/main.qml index 866d7eb..7aa41d4 100644 --- a/input-settings/main.qml +++ b/input-settings/main.qml @@ -833,12 +833,18 @@ Kirigami.ApplicationWindow { } } } + Controls.Switch { + Kirigami.FormData.label: "Gaze dot:" + text: "Always shown (off: only while the mouse moves it)" + checked: backend.gazeDotAlways + onToggled: backend.setGazeDotAlways(checked) + } Controls.Label { Layout.maximumWidth: Kirigami.Units.gridUnit * 30 wrapMode: Text.WordWrap - text: "Controllers: map a button to Gaze precision (hold, point the controller to steer, release to " - + "click) or Gaze drag (the same, pressed at once) on the Controllers page. Gaze pointer on/off " - + "can go on a controller button, a mouse button, or a key combination below." + text: "Gaze works with the mouse. The Frame controllers don't take part, and moving one hands the " + + "pointer back to the controllers. Gaze pointer on/off, Gaze precision, and Gaze drag can go on " + + "a mouse button (Buttons page) or a key combination below." opacity: 0.7 font: Kirigami.Theme.smallFont } diff --git a/input/input-relay.py b/input/input-relay.py index 6a34b9b..05ce294 100755 --- a/input/input-relay.py +++ b/input/input-relay.py @@ -21,14 +21,15 @@ Buttons and keys of pointer devices go through a per-device map to actions (left, right, middle, back, scroll_up, scroll_down, dashboard, recenter, pointer_toggle, follow_toggle = head follow on or off, gaze_toggle = gaze mode on or off (the pointer goes where you look; see pointer/helper/ft-pointer.cpp), gaze_precision = while -held, the pointer stops where you look and the button's device (the mouse, or that -controller's aim) steers it, and the release clicks there, gaze_drag = the same, but pressed -at once, so it drags ("precision|gazedrag mouse|left|right|keyboard 1|0" to the helper), sens_up, sens_down, +held, the pointer stops where you look and the mouse steers it, and the release clicks there, +gaze_drag = the same, but pressed at once, so it drags ("precision|gazedrag mouse|keyboard 1|0" +to the helper), sens_up, sens_down, layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens, keyboard_toggle = open or close Frametop's keyboard, key = pass through as a key, none). Frame controller buttons can be mapped too ("controller_buttons": {"right/a": action} in the -rules file; any action but key). So can key combinations on any keyboard ("key_bindings": +rules file; any action but key and the gaze ones, GAZE_ACTIONS: gaze mode is a mouse feature, +docs/gaze-controllers.md). So can key combinations on any keyboard ("key_bindings": {"29+56+34": action}, evdev codes joined by "+", modifiers first and left-hand codes for either side, here Ctrl+Alt+G): the combination does the action, and its last key isn't typed. The controllers aren't input devices here, only SteamVR sees them, so the pointer helper reads them with SteamVR input and sends "vrbtn