Gaze mode is a mouse feature: drop its controller parts

Steam reads the Frame controllers itself, outside SteamVR's bindings, and every
press and release it sees takes SteamVR out of laser mode, so controller clicks
at the gaze can't be done cleanly (docs/gaze-controllers.md, from the gaze-first
branch's tests).

- Gaze precision and Gaze drag take a mouse button or a key combination only;
  the controller source, its aim steering, and POINTER_PRECISION_GAIN,
  POINTER_PRECISION_DEADZONE and POINTER_GAZE_DRAG_GAIN are gone.
- The gaze actions (including Gaze pointer on/off) can't be mapped to controller
  buttons: the relay ignores them there and doesn't ask the helper for those
  buttons, and Input Settings no longer offers them.
- Last used wins in gaze mode too: picking up a controller hands it the laser,
  as docs/design.md already said.
- The gaze dot shows all the time (POINTER_GAZE_DOT=always, the default;
  moving brings back the old behaviour), with a switch on the Gaze page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/design.md

@@ -91,7 +91,7 @@ A few overlays need special handling:
 
 Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
 
-Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse or controller button mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze; the mouse is gaze mode's only pointer device for now. The dot shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. The gaze moving the pointer doesn't show it: you know where you're looking. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured.
+Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. Moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: it needs accessibility (AT-SPI) on in the Frametop session, where it's off (no registry runs), plus app restarts, and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge of up to `POINTER_GAZE_NUDGE_MAX` (8 degrees) before a click is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. See `gaze/README.md` for the service, the calibration, and what was measured.
 
 Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
 

docs/gaze-controllers.md

@@ -0,0 +1,28 @@
+# Gaze with the controllers: a dead end
+
+Gaze mode is a mouse feature. On 2026-09-30 we tried to make the Frame controllers its buttons: with gaze mode on and no game running, either controller's trigger would click where you look (a tap clicks, moving the hand steers the pointer, holding still drags), the controllers' lasers would be muted, and SteamVR's dashboard would follow the gaze too. It can't be done cleanly, for the reasons below. The work is kept on the `gaze-first` branch, which isn't merged.
+
+## What worked
+
+- Our `ft_pointer` device can hold SteamVR's laser without a hand role, in the treadmill role. It has to hint that role when SteamVR activates it; a hint changed later never gets the `/user/treadmill` path.
+- A trimmed copy of the Frame controller's compositor binding mutes the controllers' laser buttons. It's chosen with `POST /input/selectconfig.action` on vrserver's port 27062 (a JSON body; a form-encoded one gets "Parse failed"). The helper's global action sets can't do it, because SteamVR marks them inactive while its laser mouse has focus.
+- vrserver's web socket on 127.0.0.1:27062 reports every controller component without taking it from anyone.
+- Steam's UI can be kept from acting on the controllers by wrapping its gamepad input source (webpack module 17900) through Steam's CEF debugger.
+- Our device takes the laser back 10 to 15 ms after a controller takes it.
+
+## What broke it
+
+- Steam reads the Frame controllers itself. They aren't devices on the host: vrserver owns their radio and passes their raw reports to Steam through SteamVR's private Steam interface. Steam's client library turns them into a virtual device ("SteamFrameVirtual", at `/steamvr/virtual`), outside every SteamVR binding. Steam's own SteamVR action manifest asks only for haptics.
+- Every press and every release that Steam sees takes SteamVR's dashboard, and Frametop's panels, out of laser mode about 40 ms later. That happens whatever the bindings say and whatever Steam's UI does with the event. None of these stopped it: dropping the events in Steam's UI, removing the controller's `dualanalog` bindings, binding every button to a harmless compositor action, or setting `dashboard.modalGamepadAndLaser` to false.
+- Taking the laser back after each switch leaves a gap of 20 to 40 ms, and panels treat it as the pointer leaving, so clicks and drags break.
+- Steam can't be told to ignore the controllers. It won't save a controller layout for the virtual controller ("Saved Binding Selection Failed - No Identity"), and its menus don't go through the layout anyway: a live preview of the empty layout for Steam's UI (app 769) changed nothing.
+- SteamVR hands the controllers to a VR app instead of Steam only while that app has the input focus, as games do. A dashboard overlay with overlay flag `1 << 4` is given the focus only in gamepad mode, and only for gamepad input.
+
+## Options not taken
+
+- Frametop as a transparent VR app (a scene application using OpenXR's alpha blend mode) whenever gaze mode is on. That would cut Steam off while the dashboard is closed, but Steam's dashboard pages would still take the controllers, and it costs a scene layer all the time.
+- Patching Steam's running process, with an eBPF probe that writes its memory or an injected hook, to drop the controller reports while gaze mode is on. It would cover everything, but it changes Valve's software, needs Steam's client library reverse-engineered, and breaks with Steam updates.
+
+## Where the work is
+
+Branch `gaze-first`: the plan and the test log (`docs/gaze-first.md`), the probes (`pointer/probe/lasertest`, `focustest`, `vrsetting`), the web socket reader (`input/vrws.py`), the filter for Steam's UI (`input/steam-gamepad-filter.js`), and the relay's and helper's controller code. Only the gaze dot setting (`POINTER_GAZE_DOT`) came over.

docs/reference.md

@@ -104,7 +104,7 @@ pointer/helper/run.sh status | log | restart
 pointer/driver/install.sh probe     # devices, hand roles, who owns the dashboard pointer
 ```
 
-The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, and `POINTER_GAZE_SHOW`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
+The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, and `POINTER_GAZE_MOUSE`, and the gaze service's `GAZE_TRACKER` (SteamVR's eye tracker or our own) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
 
 ## Frametop Input Settings
 

gaze/README.md

@@ -27,7 +27,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
   With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
 
   Looks down past the screens (under 20 degrees down, on no Frametop screen: a glance at the keyboard) aren't sent, so the pointer stays where it was instead of following you down, and eyes lost there aren't counted. It drops blinks (both eyes closing or lost), smooths with a fixation lock, and sends the result to the pointer helper 90 times a second. It follows SteamVR's eye tracking log, and when the headset goes back on (SteamVR starts its eye model over, and the error moves), older lessons count less, so the first few after relearn the offset.
-- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse or controller button mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on. The dot only shows while the mouse moves it, while a press is held, and as a pulse when you click.
+- The pointer helper's **gaze mode** (off by default: the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1` in `~/.config/frametop.conf`, or a mouse button or key combination mapped to "Gaze pointer on/off") works like MAGIC pointing (Zhai et al., 1999). The pointer goes where you look. Move the mouse and it's the mouse's, from where the gaze put it, for the last bit. Look well away (5 degrees) and the gaze takes it back. A press isn't sent at once: the pointer stops where the gaze put it, and if that's wrong, drag it onto what you meant with the button still held; the click happens where you let go. To drag something, hold the press still for half a second first (`POINTER_GAZE_HOLD`), then move. Outside games the pointer stays on while gaze mode is on, until a controller is picked up. The dot shows all the time (`POINTER_GAZE_DOT=moving`: only while the mouse moves it, while a press is held, and as a pulse when you click). Gaze mode works with the mouse only; [docs/gaze-controllers.md](../docs/gaze-controllers.md) explains why the controllers can't click at the gaze.
 - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it a little (0.2 to 8 degrees) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the probe's one-dot check does the same). The helper only sends nudges up to `POINTER_GAZE_NUDGE_MAX` (8 degrees), and right after putting the headset back on our tracker can be further off than that. If so, raise it for a moment, or do the probe's check. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
 - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
 

input-settings/ft_input_settings.py

@@ -6,9 +6,9 @@ to the input relay over its control socket (@frametop_relay):
   - Devices: every USB/Bluetooth mouse and keyboard, a live activity light to
     identify them, and a role for each (3D pointer, pass through, ignore).
   - Buttons: press a button or key on a pointer device, then pick an action.
-  - Controllers: the same for the Frame controllers' buttons. They're read by the pointer
-    helper through SteamVR input (@ft_pointer_helper: vrstatus, vrglobal), and a mapped
-    button is taken from games.
+  - Controllers: the same for the Frame controllers' buttons, minus the gaze actions (gaze
+    mode is a mouse feature). They're read by the pointer helper through SteamVR input
+    (@ft_pointer_helper: vrstatus, vrglobal), and a mapped button is taken from games.
   - Pointer: speed, dot size, distance and the rest, applied live.
   - Ignored panels: SteamVR overlays the pointer passes through (POINTER_IGNORE), by app or
     one by one. The helper lists them (@ft_pointer_helper "overlays").
@@ -87,16 +87,14 @@ CONTROLLER_BUTTONS = {
     "right/bumper": "Right bumper", "right/trigger": "Right trigger", "right/grip": "Right grip",
     "right/thumbstick": "Right stick click",
 }
-CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none")]
+# Not the gaze actions: gaze mode is a mouse feature (docs/gaze-controllers.md; the relay's GAZE_ACTIONS).
+CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none", "gaze_toggle", "gaze_precision", "gaze_drag")]
 # Gaze mode settings (pointer helper), like POINTER_SETTINGS.
 GAZE_SETTINGS = [
     ("POINTER_GAZE_RETAKE", "Look away to hand back", 5, 1, 45, 0.5, "°"),
     ("POINTER_GAZE_NUDGE_MAX", "Largest nudge to learn", 8, 1, 30, 0.5, "°"),
     ("POINTER_GAZE_HOLD", "Hold still to drag", 0.5, 0.1, 2.0, 0.05, "s"),
     ("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"),
-    ("POINTER_PRECISION_GAIN", "Precision steering", 0.5, 0.1, 2.0, 0.05, "×"),
-    ("POINTER_PRECISION_DEADZONE", "Precision dead zone", 0.3, 0.0, 3.0, 0.1, "°"),
-    ("POINTER_GAZE_DRAG_GAIN", "Drag steering", 1.0, 0.1, 2.0, 0.05, "×"),
 ]
 # In gaze mode, what the mouse's left button does (POINTER_GAZE_MOUSE).
 GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, release to click",
@@ -809,6 +807,17 @@ class Backend(QObject):
             self.pointerChanged.emit()
             self.message.emit(f"Mouse in gaze mode: {GAZE_MOUSE[mode].lower()}", False)
 
+    @Property(bool, notify=pointerChanged)
+    def gazeDotAlways(self):
+        return read_conf().get("POINTER_GAZE_DOT", "always") != "moving"
+
+    @Slot(bool)
+    def setGazeDotAlways(self, on):
+        write_conf_value("POINTER_GAZE_DOT", "always" if on else "moving")
+        self.reload_timer.start()
+        self.pointerChanged.emit()
+        self.message.emit("Gaze dot: " + ("always shown" if on else "shown only while the mouse moves it"), False)
+
     @Property(str, notify=pointerChanged)
     def pointerRole(self):
         v = read_conf().get("POINTER_ROLE", "right")

input-settings/main.qml

@@ -833,12 +833,18 @@ Kirigami.ApplicationWindow {
                         }
                     }
                 }
+                Controls.Switch {
+                    Kirigami.FormData.label: "Gaze dot:"
+                    text: "Always shown (off: only while the mouse moves it)"
+                    checked: backend.gazeDotAlways
+                    onToggled: backend.setGazeDotAlways(checked)
+                }
                 Controls.Label {
                     Layout.maximumWidth: Kirigami.Units.gridUnit * 30
                     wrapMode: Text.WordWrap
-                    text: "Controllers: map a button to Gaze precision (hold, point the controller to steer, release to "
-                          + "click) or Gaze drag (the same, pressed at once) on the Controllers page. Gaze pointer on/off "
-                          + "can go on a controller button, a mouse button, or a key combination below."
+                    text: "Gaze works with the mouse. The Frame controllers don't take part, and moving one hands the "
+                          + "pointer back to the controllers. Gaze pointer on/off, Gaze precision, and Gaze drag can go on "
+                          + "a mouse button (Buttons page) or a key combination below."
                     opacity: 0.7
                     font: Kirigami.Theme.smallFont
                 }

input/input-relay.py

@@ -21,14 +21,15 @@ Buttons and keys of pointer devices go through a per-device map to actions
 (left, right, middle, back, scroll_up, scroll_down, dashboard, recenter,
 pointer_toggle, follow_toggle = head follow on or off, gaze_toggle = gaze mode on or off
 (the pointer goes where you look; see pointer/helper/ft-pointer.cpp), gaze_precision = while
-held, the pointer stops where you look and the button's device (the mouse, or that
-controller's aim) steers it, and the release clicks there, gaze_drag = the same, but pressed
-at once, so it drags ("precision|gazedrag mouse|left|right|keyboard 1|0" to the helper), sens_up, sens_down,
+held, the pointer stops where you look and the mouse steers it, and the release clicks there,
+gaze_drag = the same, but pressed at once, so it drags ("precision|gazedrag mouse|keyboard 1|0"
+to the helper), sens_up, sens_down,
 layout_reset = put the desktop screens back in their saved layout, screens_toggle = hide or show the desktop screens,
 keyboard_toggle = open or close Frametop's keyboard, key = pass through as a key, none).
 
 Frame controller buttons can be mapped too ("controller_buttons": {"right/a": action} in the
-rules file; any action but key). So can key combinations on any keyboard ("key_bindings":
+rules file; any action but key and the gaze ones, GAZE_ACTIONS: gaze mode is a mouse feature,
+docs/gaze-controllers.md). So can key combinations on any keyboard ("key_bindings":
 {"29+56+34": action}, evdev codes joined by "+", modifiers first and left-hand codes for
 either side, here Ctrl+Alt+G): the combination does the action, and its last key isn't typed. The controllers aren't input devices here, only SteamVR sees
 them, so the pointer helper reads them with SteamVR input and sends "vrbtn <button> 1|0".
@@ -175,6 +176,8 @@ RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
 ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
            "pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "sens_up", "sens_down",
            "layout_reset", "screens_toggle", "keyboard_toggle", "key", "none")
+# Gaze mode is a mouse feature: these never come from a controller button (docs/gaze-controllers.md).
+GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag")
 # Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
 MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
 VR_KEYBOARD_MODES = ("always", "no_keyboard", "button", "never")  # when Frametop's keyboard opens
@@ -430,10 +433,13 @@ class Pointer:
 
     def action(self, name, value, now, source="mouse"):
         """A mapped button: value 1 press, 0 release, 2 autorepeat (ignored). source: what
-        pressed it (mouse, left, right for a controller, keyboard), for the gaze actions."""
+        pressed it (mouse, left, right for a controller, keyboard), for the gaze actions,
+        which take the mouse or the keyboard only."""
         if value == 2:
             return
         if name in ("gaze_precision", "gaze_drag"):
+            if source not in ("mouse", "keyboard"):
+                return  # gaze mode is a mouse feature (GAZE_ACTIONS)
             if value == 1:
                 self.wake(now)
             self.flush()
@@ -641,7 +647,8 @@ def main():
         else:
             state["vr_capture_until"] = 0.0
             buttons = " ".join(b for b, a in state["rules"]["controller_buttons"].items()
-                               if b in VR_BUTTONS and a in ACTIONS and a not in ("key", "none")) or "-"
+                               if b in VR_BUTTONS and a in ACTIONS and a not in ("key", "none")
+                               and a not in GAZE_ACTIONS) or "-"
             if state["rules"].get("controller_in_games"):
                 buttons = "+games " + buttons
         try:
@@ -658,7 +665,7 @@ def main():
                 reply(addr, {"t": "event", "id": VR_DEVICE, "path": "", "name": "Steam Frame controllers",
                              "type": "vr", "code": button, "value": value})
         action = state["rules"]["controller_buttons"].get(button)
-        if state["pointer"] and action in ACTIONS and action not in ("key", "none"):
+        if state["pointer"] and action in ACTIONS and action not in ("key", "none") and action not in GAZE_ACTIONS:
             do_action(action, value, now, button.split("/")[0])
 
     def vr_keyboard_mode():

pointer/helper/ft-pointer.cpp

@@ -148,12 +148,14 @@
 // didn't need correcting, and after a drag, the gaze has the pointer again.
 //   Outside games (no scene application), gaze mode keeps the pointer: the relay doesn't
 // release it when the mouse is idle ("gazeawake 1|0" tells it). A controller that moves
-// still releases it (the mouse is gaze mode's only pointer device), and in games the mouse
-// wakes it and idling releases it, as without gaze.
-//   The dot only shows while the mouse moves it (within POINTER_GAZE_SHOW, 1 s), while a
-// press is held, and briefly for each click (a pulse);
-// otherwise it's transparent (still there for the laser to land on). The gaze moving it
-// doesn't show it: you know where you're looking.
+// still releases it, as without gaze (last used wins), and in games the mouse wakes it and
+// idling releases it, as without gaze. Gaze mode is a mouse feature: the controllers aren't
+// part of it. Steam reads the Frame controllers itself, outside SteamVR's bindings, so
+// controller clicks at the gaze can't be done cleanly (docs/gaze-controllers.md).
+//   The dot shows all the time in gaze mode (POINTER_GAZE_DOT=always, the default). With
+// POINTER_GAZE_DOT=moving it only shows while the mouse moves it (within POINTER_GAZE_SHOW,
+// 1 s), while a press is held, and briefly for each click (a pulse); otherwise it's
+// transparent (still there for the laser to land on), since you know where you're looking.
 //   When the mouse took the pointer and you then click, the nudge was probably onto what
 // you were looking at: from the raw gaze when the mouse took over to where you clicked is
 // the tracker's error there. The helper sends it to ft-gazed as a lesson ("lesson <raw yaw>
@@ -163,24 +165,20 @@
 // press dragged onto the target is the same: from the raw gaze at the press to the release. With no
 // fresh gaze (a blink, the service stopped, the headset off), the pointer stays put.
 //
-// Gaze precision (the relay's gaze_precision and gaze_drag actions, bound to a controller
-// button, a mouse button, or a key combination; "precision|gazedrag <source> 1|0" here): gaze
-// mode aims, the button makes it exact. Pressing gaze precision stops the pointer where you
-// look, as gaze mode's mouse press does; while it's held, the button's device steers the
-// pointer: a controller by where it points, at POINTER_PRECISION_GAIN (0.5: half its turn, for
-// precision) past POINTER_PRECISION_DEADZONE (0.3 deg, the press's own jolt), and the mouse by
-// its moves. Releasing clicks where the pointer is. A correction is a lesson for the gaze
-// tracker, as with the mouse. Gaze drag is the same with a real press at once, dragging until the
-// release (at POINTER_GAZE_DRAG_GAIN, 1), for title bars, grab bars, and selections. Without
-// gaze mode both still work from wherever the pointer is.
+// Gaze precision (the relay's gaze_precision and gaze_drag actions, bound to a mouse button or
+// a key combination; "precision|gazedrag mouse|keyboard 1|0" here): gaze mode aims, the button
+// makes it exact. Pressing gaze precision stops the pointer where you look, as gaze mode's
+// mouse press does; while it's held, the mouse steers the pointer by its moves. Releasing
+// clicks where the pointer is. A correction is a lesson for the gaze tracker, as with the
+// mouse. Gaze drag is the same with a real press at once, dragging until the release, for
+// title bars, grab bars, and selections. Without gaze mode both still work from wherever the
+// pointer is.
 //   In gaze mode the mouse's left button is a gaze precision button (POINTER_GAZE_MOUSE =
-// precision, the default), or clicks at once where the pointer is (direct). And in gaze mode a
-// moving controller doesn't take the pointer away (last used wins is off): the gaze points,
-// and the controllers are its tools.
+// precision, the default), or clicks at once where the pointer is (direct).
 //   POINTER_ROLE (right, left, or stylus): the hand role our device takes while connected. A
 // Frame controller in your hand counts as used through its touch sensors and takes its hand's
-// role back, and then no click lands (see "no hand role" in the main loop): with a controller in
-// the right hand as the precision tool, the pointer needs the left hand, or the stylus role.
+// role back, and then no click lands (see "no hand role" in the main loop): with a controller
+// held in the right hand, the pointer needs the left hand, or the stylus role.
 //
 // Hands (POINTER_HANDS, off by default; needs hand tracking, hands/): ft-hands publishes
 // pinches and grips (hands/include/fh_gestures.h), read here every frame.
@@ -254,12 +252,11 @@
 // panel, and a laser that starts behind them can't hit them. POINTER_FOLLOW (0) and
 // POINTER_LEASH_DEG (10), POINTER_LEASH_DELAY (0.2 s), POINTER_LEASH_RETURN (0.2 s),
 // POINTER_FOLLOW_REACH (70 deg): head follow, above. POINTER_GAZE (0), POINTER_GAZE_RETAKE
-// (5 deg), POINTER_GAZE_NUDGE_MAX (8 deg), POINTER_GAZE_HOLD (0.5 s), POINTER_GAZE_SHOW (1 s):
-// gaze mode, above. POINTER_CONTROLLER_PICKUP (1, 0.5 to 5): how hard a controller must
+// (5 deg), POINTER_GAZE_NUDGE_MAX (8 deg), POINTER_GAZE_HOLD (0.5 s), POINTER_GAZE_DOT
+// (always), POINTER_GAZE_SHOW (1 s): gaze mode, above. POINTER_CONTROLLER_PICKUP (1, 0.5 to 5): how hard a controller must
 // move to take the laser back, above. POINTER_IGNORE (empty): ignored panels, above.
 // POINTER_HANDS (0), POINTER_PINCH_GAIN (0.5), POINTER_PINCH_DEADZONE (1.5 deg),
 // POINTER_GRIP_GAIN (1), POINTER_GRIP_BELOW (0.35 m), POINTER_PINCH_TYPING (1 s): hands, above.
-// POINTER_PRECISION_GAIN (0.5), POINTER_PRECISION_DEADZONE (0.3 deg), POINTER_GAZE_DRAG_GAIN (1),
 // POINTER_GAZE_MOUSE (precision), POINTER_ROLE (right): gaze precision, above.
 #include <openvr.h>
 
@@ -668,13 +665,12 @@ int main() {
     // Gaze mode (see the top); gazeConf is POINTER_GAZE as last read, like followConf.
     bool gazeOn = false, gazeConf = false;
     double gazeRetake = 5, gazeNudgeMax = 8, gazeHold = 0.5, gazeShow = 1;
+    bool gazeDotAlways = true;  // POINTER_GAZE_DOT (see the top)
     // Hands (see the top): POINTER_HANDS, POINTER_PINCH_GAIN, POINTER_PINCH_DEADZONE, POINTER_GRIP_GAIN.
     bool handsOn = false;
     double pinchGain = 0.5, pinchDeadzone = 1.5, gripGain = 1.0, handBelow = 0.35, typingHold = 1.0;
-    // Gaze precision (see the top): POINTER_PRECISION_GAIN, POINTER_PRECISION_DEADZONE,
-    // POINTER_GAZE_DRAG_GAIN, POINTER_GAZE_MOUSE (precision: the mouse's left button holds
-    // back its press in gaze mode; direct: it clicks at once), POINTER_ROLE.
-    double precisionGain = 0.5, precisionDeadzone = 0.3, dragGain = 1.0;
+    // Gaze precision (see the top): POINTER_GAZE_MOUSE (precision: the mouse's left button
+    // holds back its press in gaze mode; direct: it clicks at once), POINTER_ROLE.
     bool gazeMousePrecision = true;
     std::string role = "right";
     double pickupScale = 1;  // POINTER_CONTROLLER_PICKUP: scales the controller-moved limits
@@ -699,6 +695,8 @@ int main() {
         gazeNudgeMax = std::clamp(ConfDouble(conf, "POINTER_GAZE_NUDGE_MAX", 8), 1.0, 30.0);
         gazeHold = std::clamp(ConfDouble(conf, "POINTER_GAZE_HOLD", 0.5), 0.1, 5.0);
         gazeShow = std::clamp(ConfDouble(conf, "POINTER_GAZE_SHOW", 1), 0.0, 30.0);
+        const auto gd = conf.find("POINTER_GAZE_DOT");
+        gazeDotAlways = gd == conf.end() || gd->second != "moving";
         const bool wantGaze = ConfDouble(conf, "POINTER_GAZE", 0) != 0;
         if (wantGaze != gazeConf) gazeOn = gazeConf = wantGaze;
         handsOn = ConfDouble(conf, "POINTER_HANDS", 0) != 0;
@@ -707,9 +705,6 @@ int main() {
         gripGain = std::clamp(ConfDouble(conf, "POINTER_GRIP_GAIN", 1.0), 0.05, 3.0);
         handBelow = std::clamp(ConfDouble(conf, "POINTER_GRIP_BELOW", 0.35), 0.05, 1.0);
         typingHold = std::clamp(ConfDouble(conf, "POINTER_PINCH_TYPING", 1.0), 0.0, 5.0);
-        precisionGain = std::clamp(ConfDouble(conf, "POINTER_PRECISION_GAIN", 0.5), 0.05, 3.0);
-        precisionDeadzone = std::clamp(ConfDouble(conf, "POINTER_PRECISION_DEADZONE", 0.3), 0.0, 10.0);
-        dragGain = std::clamp(ConfDouble(conf, "POINTER_GAZE_DRAG_GAIN", 1.0), 0.05, 3.0);
         const auto gm = conf.find("POINTER_GAZE_MOUSE");
         gazeMousePrecision = gm == conf.end() || gm->second != "direct";
         const auto ro = conf.find("POINTER_ROLE");
@@ -844,12 +839,12 @@ int main() {
     bool aimHand = false;  // the held-back press is a hold's (below): it never turns into a real press
     Clock::time_point aimSince{}, clickReleaseAt{};
     // Holds (see "Gaze precision" and "Hands" at the top): a pinch or grip, or a gaze
-    // precision or gaze drag button, held now. What steers the pointer meanwhile (a hand, a
-    // controller's aim, or the mouse through its own moves), from where it pointed when the
-    // hold began (for a hand: seen from the eye then, in the room), and the pointer then.
+    // precision or gaze drag button, held now. What steers the pointer meanwhile (a hand, or
+    // the mouse through its own moves), from where it pointed when the hold began (for a
+    // hand: seen from the eye then, in the room), and the pointer then.
     HandGestures handFile;
     PoseHistory poses;
-    enum class Src { None, Hand, Controller, Mouse };
+    enum class Src { None, Hand, Mouse };
     struct Hold {
         Src src = Src::None;
         int side = -1;          // a hand's side (0 left, 1 right)
@@ -857,7 +852,6 @@ int main() {
         bool engaged = false;   // past the dead zone
         bool pressed = false;   // a real press went out (a grip or gaze drag, or a pinch without gaze mode)
         Vec3 origin;
-        vr::TrackedDeviceIndex_t ctrl = vr::k_unTrackedDeviceIndexInvalid;
         double refYaw = 0, refPitch = 0, startYaw = 0, startPitch = 0, lastYaw = 0, lastPitch = 0;
     } hold;
     // "precision|gazedrag <source> 1|0" from the relay, done in the frame (see Holds).
@@ -1366,10 +1360,9 @@ int main() {
             std::fflush(stdout);
         }
 
-        // Last used wins: a real controller being moved releases the pointer (see the top).
-        // Not in gaze mode: there the gaze points, and a controller is a tool for it (gaze
-        // precision), so moving one doesn't take the pointer away.
-        if (active && !gazeOn && hold.src == Src::None && tnow - lastMouse > std::chrono::milliseconds(500)) {
+        // Last used wins: a real controller being moved releases the pointer (see the top),
+        // in gaze mode too.
+        if (active && hold.src == Src::None && tnow - lastMouse > std::chrono::milliseconds(500)) {
             for (vr::TrackedDeviceIndex_t i = 1; i < vr::k_unMaxTrackedDeviceCount; ++i) {
                 if (i == ours || !all[i].bPoseIsValid || all[i].eTrackingResult != vr::TrackingResult_Running_OK ||
                     sys->GetTrackedDeviceClass(i) != vr::TrackedDeviceClass_Controller) {
@@ -1542,56 +1535,22 @@ int main() {
             if (debug) std::printf("hand %s %s began\n", side ? "right" : "left", grip ? "grip" : "pinch");
             if (debug) std::fflush(stdout);
         };
-        // A controller's aim: the yaw and pitch of its pointing direction, in the room.
-        auto controllerAngles = [&](vr::TrackedDeviceIndex_t i, double &cy, double &cp) {
-            if (i >= vr::k_unMaxTrackedDeviceCount || !all[i].bPoseIsValid) return false;
-            const auto &m = all[i].mDeviceToAbsoluteTracking.m;
-            const Vec3 f = Normalize({-m[0][2], -m[1][2], -m[2][2]});
-            cy = std::atan2(-f.x, -f.z) * 180 / M_PI;
-            cp = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
-            return true;
-        };
-        // The Frame controller in a hand (not our own device, which may hold that hand's role).
-        auto controllerFor = [&](const std::string &hand) {
-            const int32_t want = hand == "left" ? vr::TrackedControllerRole_LeftHand : vr::TrackedControllerRole_RightHand;
-            for (vr::TrackedDeviceIndex_t i = 1; i < vr::k_unMaxTrackedDeviceCount; ++i) {
-                if (i == ours || sys->GetTrackedDeviceClass(i) != vr::TrackedDeviceClass_Controller) continue;
-                if (sys->GetInt32TrackedDeviceProperty(i, vr::Prop_ControllerRoleHint_Int32) == want ||
-                    sys->GetControllerRoleForTrackedDeviceIndex(i) == want)
-                    return i;
-            }
-            return vr::k_unTrackedDeviceIndexInvalid;
-        };
-        // Gaze precision and gaze drag buttons (see the top).
+        // Gaze precision and gaze drag buttons (see the top): the mouse steers.
         for (const DevicePress &p : devicePresses) {
-            const bool fromController = p.source == "left" || p.source == "right";
+            if (p.source == "left" || p.source == "right") continue;  // no controllers in gaze mode
             if (!p.down) {
-                if (hold.src == Src::Controller || hold.src == Src::Mouse) endHold(false);
+                if (hold.src == Src::Mouse) endHold(false);
                 continue;
             }
             if (hold.src != Src::None || leftHeld || aimHeld || clickPress || clickRelease) continue;
             Hold h;
-            h.src = fromController ? Src::Controller : Src::Mouse, h.grip = p.drag;
-            if (fromController) {
-                h.ctrl = controllerFor(p.source);
-                if (!controllerAngles(h.ctrl, h.refYaw, h.refPitch)) {
-                    std::printf("gaze %s: no %s controller pose; the mouse steers instead\n", p.drag ? "drag" : "precision",
-                                p.source.c_str());
-                    std::fflush(stdout);
-                    h.src = Src::Mouse;
-                }
-            }
+            h.src = Src::Mouse, h.grip = p.drag;
             hold = h;
             startHold(p.drag);
             if (debug) std::printf("hold gaze %s began (%s)\n", p.drag ? "drag" : "precision", p.source.c_str());
             if (debug) std::fflush(stdout);
         }
         devicePresses.clear();
-        if (hold.src == Src::Controller) {
-            double cy, cp;
-            if (controllerAngles(hold.ctrl, cy, cp))
-                steer(cy, cp, precisionDeadzone, hold.grip ? dragGain : precisionGain);
-        }
         fh_gestures_t hg;
         const bool handOk = handsOn && handFile.Read(hg);
         if (handOk && (hg.seq != handSeq || handFile.opens != handOpens)) {
@@ -1894,7 +1853,7 @@ int main() {
                 // the page, with SteamVR's hit dot hidden on it.
                 const Vec3 near = eye + sight * SETTINGS_DOT, far = eye + sight * SETTINGS_CATCHER;
                 double alpha = 1;
-                if (gazeOn) {
+                if (gazeOn && !gazeDotAlways) {
                     auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
                     alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
                 }
@@ -1914,12 +1873,13 @@ int main() {
                 const vr::VROverlayHandle_t show = onPanel ? marker : cursor, hide = onPanel ? cursor : marker;
                 const Vec3 at = onPanel ? point + Normalize(eye - point) * 0.005 : onScene ? point + dir * 0.05 : point;
                 const double dist = std::sqrt(Dot(at - eye, at - eye));
-                // Gaze mode: shown only while something moves it or a press holds it, and a pulse
-                // for each click (see the top); transparent otherwise, the laser still lands on it.
+                // Gaze mode: a pulse for each click, and with POINTER_GAZE_DOT=moving, shown only
+                // while something moves it or a press holds it (see the top); transparent
+                // otherwise, the laser still lands on it.
                 double scale = 1, alpha = 1;
                 if (gazeOn) {
                     auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
-                    alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
+                    alpha = gazeDotAlways ? 1.0 : std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
                     const double pulse = secs(pulseAt);
                     if (pulse < 0.6) {
                         scale = 1 + 1.5 * std::max(0.0, 1 - pulse / 0.3);

session/frametop.conf.example

@@ -31,11 +31,9 @@ POINTER_GAZE=0             # 1 = the pointer goes where you look, the mouse does
 POINTER_GAZE_RETAKE=5      # gaze mode: how far (degrees) you look away from the pointer before the gaze takes it back from the mouse
 POINTER_GAZE_NUDGE_MAX=8   # gaze mode: a mouse nudge up to this far (degrees) before a click is learned as the eye tracker's error
 POINTER_GAZE_HOLD=0.5      # gaze mode: a press held this long (s) without moving becomes a real press (to drag); moved or released sooner, it clicks where you let go
-POINTER_GAZE_SHOW=1        # gaze mode: the dot shows this long (s) after the mouse moves it; also while a press is held, and a pulse per click
+POINTER_GAZE_DOT=always    # gaze mode: the dot shows all the time (always) | only while the mouse moves it (moving)
+POINTER_GAZE_SHOW=1        # gaze mode, with POINTER_GAZE_DOT=moving: the dot shows this long (s) after the mouse moves it; also while a press is held, and a pulse per click
 POINTER_GAZE_MOUSE=precision # gaze mode: the left button holds back its press, the mouse steers, the release clicks (precision) | clicks right away (direct)
-POINTER_PRECISION_GAIN=0.5  # gaze precision (a button mapped to gaze_precision): the pointer turns this times the controller's turn
-POINTER_PRECISION_DEADZONE=0.3 # gaze precision: degrees the controller turns before the pointer does (the press's own jolt)
-POINTER_GAZE_DRAG_GAIN=1   # gaze drag (gaze_drag): the pointer turns this times the controller's turn
 GAZE_TRACKER=steam         # gaze service: steam = SteamVR's eye tracker | own = our own (gaze/tracker: its frame grabber needs gaze/tracker/install.sh; calibrated in the gaze probe with Own tracker)
 GAZE_EYE=auto              # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
 HANDS_SWAP_SIDES=0         # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)