Keep the controllers in VR games while the screens stay up

Visible screens kept SteamVR's laser mouse on, which takes the controllers
away from a VR game. Now, by default, that's off while a scene app runs:
the screens stay over the game and the 3D mouse or the dashboard works
them. Frametop Display Settings has the choice (always, except during VR
games, only with the dashboard open); the socket command is controllers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -54,7 +54,8 @@ After the restart:
 | Drag the roll button sideways, or scroll on it | Rolls the screen; it snaps level near straight |
 | While carrying a screen, sweep its laser across your other controller's ring, then let go | Pins it to that wrist, at its size and distance, as you hold it when you let go; it shows while you see its front. Grab its bar to adjust it (it stays pinned); sweep across the ring again to take it off |
 | Meta+Shift+R in the desktop | Puts the screens back in their layout (also the **Reset Screen Layout** menu entry, and a button you can map) |
-| Meta+Shift+H in the desktop | Hides or shows all screens, for a VR game (also **Hide/Show Screens** and a mappable button). **Frametop Display Settings → Visibility & wrist** can instead show them only with the dashboard open, or while you look at your wrist |
+| Meta+Shift+H in the desktop | Hides or shows all screens (also **Hide/Show Screens** and a mappable button). **Frametop Display Settings → Visibility & wrist** can instead show them only with the dashboard open, or while you look at your wrist |
+| Play a VR game | The screens stay up over it, and your controllers stay in the game. Work the screens with the 3D mouse, or open the SteamVR dashboard. (**Visibility & wrist → Controllers on the screens** changes this.) |
 
 Map the mouse's extra buttons to actions such as **Toggle SteamVR dashboard** or **Recenter pointer** in **Frametop Input Settings → Buttons**. Speed, dot size, and the rest are on its **Pointer** page and apply immediately.
 

display-settings/ft_display_settings.py

@@ -363,6 +363,8 @@ class Backend(QObject):
                 self._ask_screens(f"visibility {value}")
             elif key == "wrist_angle":
                 self._ask_screens(f"wrist {float(value):.1f}")
+            elif key == "controllers":
+                self._ask_screens(f"controllers {value}")
             else:
                 self._ask_screens(f"gesture {v['gesture_hand']} {float(v['gesture_angle']):.1f}")
 

display-settings/main.qml

@@ -567,7 +567,7 @@ Kirigami.ApplicationWindow {
 
                     Repeater {
                         model: [
-                            { value: "always", text: "Always", help: "Meta+Shift+H (or a mapped button) hides them, for a VR game." },
+                            { value: "always", text: "Always", help: "They stay up, over VR games too. Meta+Shift+H (or a mapped button) hides them." },
                             { value: "dashboard", text: "Only with the SteamVR dashboard open", help: "They come and go with the dashboard. Meta+Shift+H shows them anyway." },
                             { value: "gesture", text: "While I look at my wrist", help: "They show while you look toward the controller below. Meta+Shift+H shows them anyway." },
                             { value: "toggle", text: "Only when I show them", help: "Hidden until Meta+Shift+H (or a mapped button) shows them." }
@@ -615,6 +615,33 @@ Kirigami.ApplicationWindow {
                         Controls.Label { text: Math.round(gesture.value) + "° of it" }
                     }
 
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Controllers on the screens" }
+
+                    Repeater {
+                        model: [
+                            { value: "outside_games", text: "Except during VR games", help: "Over a VR game the screens stay up, but the controllers stay in the game. Use the 3D mouse, or open the SteamVR dashboard, to work the screens." },
+                            { value: "always", text: "Always", help: "Controllers' lasers work the screens whenever they're visible, even over a VR game (which then can't use the controllers)." },
+                            { value: "dashboard", text: "Only with the SteamVR dashboard open", help: "Otherwise only the 3D mouse works the screens. Also for flatscreen games, which don't count as VR games." }
+                        ]
+                        delegate: ColumnLayout {
+                            required property var modelData
+                            spacing: 0
+                            Controls.RadioButton {
+                                text: modelData.text
+                                checked: (vpage.v.controllers || "outside_games") === modelData.value
+                                onToggled: if (checked) backend.setVisibility("controllers", modelData.value)
+                            }
+                            Controls.Label {
+                                text: modelData.help
+                                opacity: 0.7
+                                font: Kirigami.Theme.smallFont
+                                leftPadding: Kirigami.Units.gridUnit * 1.6
+                                wrapMode: Text.Wrap
+                                Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                            }
+                        }
+                    }
+
                     Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Screens on a wrist" }
 
                     RowLayout {

docs/reference.md

@@ -31,9 +31,9 @@ Restarting the desktop (`desktops.sh restart`, or Restart desktop in Frametop Di
 
 - Screens: each KWin window is a screen. ft-screens sizes it (`xdg_toplevel` configure) and KWin resizes that screen to match, live. Frames arrive as DMA-BUFs and go to SteamVR with `ImportDmabuf` (OpenVR's `IVRIPCResourceManagerClient`), no copy, no size limit.
 - Panels: `frametop.screen.N`, with `.bar` (move), `.curve` (the round button next to it), `.roll` (the next one: drag it sideways like a knob to roll the screen, snapping level within 2.5°, or scroll on it for 5° steps), and `.resize` (the tab on the bottom right corner; screens go down to 15 cm wide). The controls are sized from both the screen's width and its distance from you, sit on its surface when it's curved, and are translucent like SteamVR's own until a laser is on them. They're invisible until a laser or the 3D mouse's cursor comes very close to one of them (about 1.5 times a button's size), and fade out a moment after it leaves. Drag the bar with any laser (a controller, or the 3D mouse, whose right-drag tilt works too); scroll while dragging to push it away or pull it closer (along the line from your head). The curve button bends the screen into a cylinder around you (radius: your distance to it), or flat again. Pin to a wrist: while carrying a screen, sweep the laser (the line from whatever carries it to its bar) across your other controller. A ring around that controller shows the target and a dot shows where the laser passes; entering the ring arms the pin (ring and bar turn blue), entering it again disarms it. Let go while armed and the screen rides on that controller as it is then, at its size and distance (a 3.6 m screen 5 m away works; so does pinning all screens), so you can arm it first and then turn it the way you want. Grab a pinned screen's bar to adjust it: it comes back to the same wrist when you let go, unless you sweep across the ring to disarm. A pinned screen shows only while you see its front within the wrist angle, fading over the last 10°.
-- Visibility (Frametop Display Settings → Visibility & wrist tab): always (Meta+Shift+H, the Hide/Show Screens menu entry, or a mapped mouse button hides them), only with the SteamVR dashboard open, while you look at a chosen controller (the wrist gesture), or only when shown with the hotkey. In the last three, the hotkey shows them anyway. Visible screens keep SteamVR's laser on; hidden, VR games get their triggers back. (A controller button to show them isn't there yet: in a game the game owns the buttons.)
+- Visibility (Frametop Display Settings → Visibility & wrist tab): always (Meta+Shift+H, the Hide/Show Screens menu entry, or a mapped mouse button hides them), only with the SteamVR dashboard open, while you look at a chosen controller (the wrist gesture), or only when shown with the hotkey. In the last three, the hotkey shows them anyway. Controllers on the screens (same tab): visible screens can keep SteamVR's laser mouse on so controllers work them with the dashboard closed, which also takes the controllers from a VR game. By default that's off while a VR game (a scene app) runs: the screens stay over the game, the controllers stay in it, and the 3D mouse or the dashboard works the screens. The other choices are always and only with the dashboard open (also right for flatscreen games, which aren't scene apps). Control socket: `controllers always|outside_games|dashboard`. (A controller button to show them isn't there yet: in a game the game owns the buttons.)
 - Input: pointer from the panels to KWin through our seat; keys from the input relay (every keyboard it doesn't grab, and keys a pointer device passes through) to the screen that was clicked last, but not while the SteamVR dashboard is open.
-- Control socket `@ft_screens` (datagrams, replies to the sender): `place N x y z yaw pitch roll`, `width N metres`, `curve N radius|on|off`, `pin N|all left|right [12 numbers]`, `unpin N|all`, `size N w h` (live resolution), `get N`, `screens`, `head`, `visibility always|dashboard|gesture|toggle`, `wrist degrees`, `gesture left|right degrees`, `hide | show | toggle`, `state`, `key code value`.
+- Control socket `@ft_screens` (datagrams, replies to the sender): `place N x y z yaw pitch roll`, `width N metres`, `curve N radius|on|off`, `pin N|all left|right [12 numbers]`, `unpin N|all`, `size N w h` (live resolution), `get N`, `screens`, `head`, `visibility always|dashboard|gesture|toggle`, `wrist degrees`, `gesture left|right degrees`, `hide | show | toggle`, `controllers always|outside_games|dashboard`, `state`, `key code value`.
 
 ## Input relay (Bluetooth mice and keyboards)
 

layout/ft_layout.py

@@ -62,7 +62,9 @@ SCREENS = "\0ft_screens"
 LOCK_PATH = "/tmp/ft-layout.lock"
 DEFAULT_PANEL = (1.18, 0.664)  # gamescope: a floating 16:9 dashboard panel, measured on the Frame
 PIXELS_PER_METRE = 800         # ft-screens: a new screen's default size in VR (1920 px: 2.4 m)
-VISIBILITY = {"mode": "always", "wrist_angle": 60, "gesture_hand": "left", "gesture_angle": 20}
+# controllers: when controllers' lasers work the screens (always | outside_games | dashboard).
+VISIBILITY = {"mode": "always", "wrist_angle": 60, "gesture_hand": "left", "gesture_angle": 20,
+              "controllers": "outside_games"}
 DEFAULTS = {"auto": True, "mode": "preset",
             "preset": {"kind": "arc", "rows": 1, "distance": 2.0, "gap": 0.05, "height": 0.0},
             "screens": [], "panel_size": list(DEFAULT_PANEL)}
@@ -373,6 +375,7 @@ def send_visibility(sock, layout):
     sock.ask(f"visibility {v['mode']}")
     sock.ask(f"wrist {float(v['wrist_angle']):.1f}")
     sock.ask(f"gesture {v['gesture_hand']} {float(v['gesture_angle']):.1f}")
+    sock.ask(f"controllers {v['controllers']}")
 
 
 def parse_get(reply):

screens/vr.cpp

@@ -25,9 +25,14 @@
 //     its front, within the wrist angle (and fades out over the last kFade degrees).
 //   - visibility modes: always (the hide hotkey toggles), only with the SteamVR dashboard
 //     open, while you look at a chosen controller (the wrist gesture), or toggle only
-//     (hidden until the hotkey shows them). While visible the screens keep SteamVR's laser
-//     mouse on (VROverlayFlags_MakeOverlaysInteractiveIfVisible), so controllers can use
-//     them with the dashboard closed; hidden, VR games get their triggers back.
+//     (hidden until the hotkey shows them).
+//   - controllers on the screens: while visible, the screens can keep SteamVR's laser mouse
+//     on (VROverlayFlags_MakeOverlaysInteractiveIfVisible), so controllers use them with
+//     the dashboard closed. That also takes the controllers away from a VR game, so by
+//     default it's off while a game (a scene app) runs: the screens stay up over the game,
+//     the controllers stay in it, and the 3D mouse (its own laser mode) or the dashboard
+//     works the screens. Modes: always, outside_games (default), dashboard (never on its
+//     own; also for flatscreen games, which aren't scene apps).
 // OpenVR has no overlay-relative transforms here (openvr v2.15.6), so the bar, button,
 // and handle are placed whenever their screen moves.
 #include "vr.h"
@@ -133,6 +138,7 @@ const char *HandName(vr::TrackedDeviceIndex_t i) {
 
 enum class Drag { None, Move, Resize, Roll };
 enum class Mode { Always, Dashboard, Gesture, Toggle };
+enum class Lasers { Always, OutsideGames, Dashboard };
 
 constexpr double kWristZone = 0.06;  // the laser passing this close to a controller is on its wrist
 constexpr double kWristLeave = 0.09; // ...and has left it beyond this (so it doesn't flicker)
@@ -165,6 +171,7 @@ struct Screen {
     Mat rollFrom = Identity();                // roll: the pose at the press (pinRel when pinned)
     double rollAngle = 0;                     // roll: the laser's angle around the centre then
     bool hover[4] = {};                       // a laser is on the bar, curve, roll, resize control
+    bool lasers = true;                       // MakeOverlaysInteractiveIfVisible is set
     float controls = 0;                       // the controls' fade, 0 (hidden) .. 1
     bool controlsUp = false;                  // the controls' overlays are shown
     long nearUntil = 0;                       // a laser was near the controls until this tick
@@ -187,6 +194,8 @@ bool g_manual = false;
 double g_wristAngle = 60;    // a pinned screen shows while you see its front within this
 double g_gestureAngle = 20;  // gesture: look within this of the controller
 std::string g_gestureHand = "left";
+Lasers g_lasers = Lasers::OutsideGames;  // when controllers' lasers work the screens (see the top)
+bool g_gameRunning = false;              // a scene app (VR game) is running
 
 // ---------------------------------------------------------------- chrome (bar, button, handle)
 
@@ -593,6 +602,18 @@ void UpdateVisibility() {
 }
 
 
+// Controllers' lasers on the screens (see the top): the flag follows the mode and whether a
+// VR game runs (checked twice a second).
+void UpdateLasers() {
+    if (g_tick % 45 == 0) g_gameRunning = vr::VRApplications()->GetCurrentSceneProcessId() != 0;
+    const bool want = g_lasers == Lasers::Always || (g_lasers == Lasers::OutsideGames && !g_gameRunning);
+    for (auto &[i, s] : g_screens) {
+        if (s.lasers == want) continue;
+        s.lasers = want;
+        vr::VROverlay()->SetOverlayFlag(s.overlay, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, want);
+    }
+}
+
 // The controls are invisible until a laser passes very close to one of them (within
 // `reach`, about 1.5 times a button's size); they stay kControlsLinger ticks after it
 // leaves, and while in use.
@@ -892,6 +913,14 @@ uint32_t LinuxButton(uint32_t vrButton) {
     }
 }
 
+const char *LasersName() {
+    switch (g_lasers) {
+        case Lasers::Always: return "always";
+        case Lasers::Dashboard: return "dashboard";
+        default: return "outside_games";
+    }
+}
+
 const char *ModeName() {
     switch (g_mode) {
         case Mode::Dashboard: return "dashboard";
@@ -1134,6 +1163,7 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
     }
     ++g_tick;
     UpdateVisibility();
+    UpdateLasers();
     UpdateControls();
     UpdateGuides();
 }
@@ -1154,7 +1184,9 @@ void ft_vr_poll(void (*handle)(const struct ft_event *, void *), void *data) {
 //   wrist <degrees>           a pinned screen shows while you see its front within this
 //   gesture <left|right> <degrees>   the gesture mode: look within this of that controller
 //   hide | show | toggle      the manual switch (see g_manual)
-//   state         -> "ok <mode> <manual 0|1> <wrist deg> <gesture hand> <gesture deg>"
+//   controllers always|outside_games|dashboard   when controllers' lasers work the screens
+//   state         -> "ok <mode> <manual 0|1> <wrist deg> <gesture hand> <gesture deg>
+//                     <controllers> <game running 0|1>"
 // (size <screen> <w> <h> and key <code> <value> are handled in compositor.c.) Screens are
 // numbered from 1 here, like everywhere the user sees them.
 void ft_vr_command(const char *cmd, char *reply, int size) {
@@ -1264,9 +1296,17 @@ void ft_vr_command(const char *cmd, char *reply, int size) {
         g_manual = g_mode == Mode::Always ? !want : want;
         UpdateVisibility();
         std::snprintf(reply, size, "ok %s", want ? "shown" : "hidden");
+    } else if (std::sscanf(cmd, "controllers %15s", word) == 1) {
+        const std::string m = word;
+        if (m == "always") g_lasers = Lasers::Always;
+        else if (m == "outside_games") g_lasers = Lasers::OutsideGames;
+        else if (m == "dashboard") g_lasers = Lasers::Dashboard;
+        else return (void)std::snprintf(reply, size, "error modes: always outside_games dashboard");
+        UpdateLasers();
+        std::snprintf(reply, size, "ok %s", LasersName());
     } else if (std::strncmp(cmd, "state", 5) == 0) {
-        std::snprintf(reply, size, "ok %s %d %.0f %s %.0f", ModeName(), g_manual ? 1 : 0, g_wristAngle,
-                      g_gestureHand.c_str(), g_gestureAngle);
+        std::snprintf(reply, size, "ok %s %d %.0f %s %.0f %s %d", ModeName(), g_manual ? 1 : 0, g_wristAngle,
+                      g_gestureHand.c_str(), g_gestureAngle, LasersName(), g_gameRunning ? 1 : 0);
     } else {
         std::snprintf(reply, size, "error unknown command");
     }