From 19032f8963f3fad63167067ea5fe0a58714026c5 Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:10:30 -0600 Subject: [PATCH 1/8] Pointer: read the overlay list every 20 s, not every second The helper ran `vrcmd --overlays` once a second while the pointer was awake, and in gaze mode the pointer never sleeps. Each run is a shell plus vrcmd, a new SteamVR client, about 26 to 30 ms of CPU, so about 3% of a core all the time. The list is now read every 20 seconds, and at once (at most once a second) when it may have changed: the pointer waking, the dashboard opening or closing or creating an overlay, the scene app changing, an "overlays" request, and a left click that hit nothing, which may be on a panel that came up since. The thread waits on a condition variable instead of waking every 100 ms, so it sleeps while paused. The main loop looks the keys up again as soon as a new list is in, rather than at its next 1 s tick. Overlays already on the list still show and hide within 50 ms, from the IsOverlayVisible poll. Co-Authored-By: Claude Opus 5.5 --- docs/design.md | 2 +- pointer/helper/ft-pointer.cpp | 77 +++++++++++++++++++++++++++-------- 2 files changed, 62 insertions(+), 17 deletions(-) diff --git a/docs/design.md b/docs/design.md index 4725296..fb29cbf 100644 --- a/docs/design.md +++ b/docs/design.md @@ -118,7 +118,7 @@ The driver starts disconnected, because holding the right-hand role while SteamV Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either. -OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately. +OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately. Each run is a shell and a new SteamVR client, about 30 ms of CPU, and it ran every second while the pointer was awake, which in gaze mode is all the time. Now it runs every 20 seconds, and at once when the pointer wakes, when the dashboard opens or closes, when a game starts or ends, and when a left click hits nothing (a panel that came up since). An overlay already on the list showing or hiding needs no new list: the helper checks the visibility of the ones it knows every 50 ms. The laser starts partway along your line of sight to the cursor rather than at your eye. SteamVR sizes its hit dot by distance from the laser's origin, and a laser from the eye still shows a beam in each eye. Starting it close to the target makes the beam and the dot tiny, while `POINTER_ORIGIN_MARGIN` keeps the origin in front of the small window controls, which float a few centimetres in front of their panels. The helper's own white dot is the visible cursor. In empty space it's an interactive overlay that the laser lands on, so SteamVR never draws a laser into nothing. diff --git a/pointer/helper/ft-pointer.cpp b/pointer/helper/ft-pointer.cpp index a06a5fc..4820c2c 100644 --- a/pointer/helper/ft-pointer.cpp +++ b/pointer/helper/ft-pointer.cpp @@ -311,6 +311,7 @@ extern "C" { #include #include #include +#include #include #include #include @@ -500,29 +501,62 @@ std::string JsonQuote(const std::string &s) { // public call to enumerate other apps' overlays). Hidden ones are listed too: the // window controls under a floating panel only appear while something hovers the // panel, and the cursor has to find them the moment they do, not a second later. +// Each vrcmd run is a shell and a new SteamVR client, about 30 ms of CPU, so the list is read +// every 20 seconds (kEvery) while the pointer is awake, and at once when something says it may +// have changed (Kick: the pointer waking, the dashboard opening or closing, a click that hit +// nothing), at most once a second. Showing and hiding the overlays it knows doesn't need a new +// list: the main loop polls their visibility (IsOverlayVisible) every 50 ms. // Paused while the pointer is off: each vrcmd run connects to SteamVR as a new app, and a new // app every second kept SteamVR (and the headset's displays) from going to standby. // "overlays" requests (Frametop Input Settings' Ignored panels page) refresh the list even // while paused, and are answered from this thread once it's fresh. class OverlayList { public: + static constexpr auto kEvery = std::chrono::seconds(20), kGap = std::chrono::seconds(1); void Start() { out_ = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0); thread_ = std::thread([this] { + std::unique_lock lk(lock_); while (running_) { - if (!paused_ || requested_) Refresh(); - // Wait a second, or less when the pointer wakes (refresh right away then). - for (int i = 0; i < 10 && running_; ++i) { - const bool wasPaused = paused_; - std::this_thread::sleep_for(std::chrono::milliseconds(100)); - if ((wasPaused && !paused_) || requested_) break; + // Sleeps while paused (until a request or the wake); otherwise until it's time, or + // a kick once kGap has passed since the last read. + const auto due = std::max(last_ + kGap, kicked_ ? last_ : last_ + kEvery); + if (!waiting_.empty()) { + } else if (paused_) { + cv_.wait(lk); + continue; + } else if (std::chrono::steady_clock::now() < due) { + cv_.wait_until(lk, due); + continue; } + kicked_ = false; + lk.unlock(); + Refresh(); + lk.lock(); + last_ = std::chrono::steady_clock::now(); } }); } - void SetPaused(bool paused) { paused_ = paused; } + // Unpausing (the pointer woke) reads the list again at once. + void SetPaused(bool paused) { + std::lock_guard guard(lock_); + if (paused == paused_) return; + if (!paused) kicked_ = true, last_ = {}; + paused_ = paused; + cv_.notify_one(); + } + // The list may have changed: read it again soon (not while paused). + void Kick() { + std::lock_guard guard(lock_); + kicked_ = true; + cv_.notify_one(); + } void Stop() { - running_ = false; + { + std::lock_guard guard(lock_); + running_ = false; + cv_.notify_one(); + } if (thread_.joinable()) thread_.join(); } std::vector Keys() { @@ -531,12 +565,14 @@ public: for (const auto &e : entries_) keys.push_back(e.key); return keys; } + // Goes up by one each time the list is read, so the main loop knows to look the keys up again. + unsigned Generation() const { return generation_; } // Answer `to` with {"t":"overlays","list":[{"key","name","visible"}...]} after the next refresh. void Request(const sockaddr_un &to, socklen_t len) { if (len <= offsetof(sockaddr_un, sun_path)) return; std::lock_guard guard(lock_); if (waiting_.size() < 8) waiting_.push_back({to, len}); - requested_ = true; + cv_.notify_one(); } private: @@ -545,7 +581,6 @@ private: bool visible; }; void Refresh() { - requested_ = false; FILE *p = popen("LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrcmd --overlays 2>/dev/null", "r"); if (!p) return; std::vector entries; @@ -577,6 +612,7 @@ private: entries_ = std::move(entries); waiting.swap(waiting_); } + ++generation_; if (waiting.empty()) return; std::string msg = "{\"t\":\"overlays\",\"list\":["; for (size_t i = 0; i < entries_.size(); ++i) @@ -589,10 +625,11 @@ private: std::thread thread_; int out_ = -1; - std::atomic paused_{false}; - std::atomic running_{true}; - std::atomic requested_{false}; - std::mutex lock_; + std::mutex lock_; // guards everything below + std::condition_variable cv_; + bool paused_ = false, running_ = true, kicked_ = false; + std::chrono::steady_clock::time_point last_{}; // the last read + std::atomic generation_{0}; std::vector entries_; // written only by the thread; the lock guards readers std::vector> waiting_; }; @@ -965,6 +1002,8 @@ int main() { if (debug) std::fflush(stdout); } nudging = confirmLesson = false; + // A click on nothing: maybe on an overlay that came up since the list was read. + if (lastHit.empty()) overlays.Kick(); leftHeld = true; dragDistance = lastDistance; pressKey.clear(); @@ -1089,6 +1128,7 @@ int main() { !aimHeld && !leftHeld && !tilting && hold.src == Src::None && !clickPress && !clickRelease; }; auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10); + unsigned listGeneration = 0; // the overlay list the handles were looked up from // --- Panel placement (see "Placement" at the top of the file) --- // Device pose, given in the standing universe, sent to the driver in raw space. @@ -1928,10 +1968,11 @@ int main() { } followAt = tnow; - // Slow work, once a second: overlay handles, our device index, laser width. + // Slow work, once a second and when the overlay list is new: overlay handles, our device index. const auto now = std::chrono::steady_clock::now(); - if (now - lastSlow > std::chrono::seconds(1)) { + if (now - lastSlow > std::chrono::seconds(1) || overlays.Generation() != listGeneration) { lastSlow = now; + listGeneration = overlays.Generation(); handles.clear(); for (const auto &key : overlays.Keys()) { if (Ignored(ignore, key)) continue; @@ -2275,6 +2316,10 @@ int main() { vr::VREvent_t ev; while (sys->PollNextEvent(&ev, sizeof ev)) { + // The dashboard's overlays come and go with it, and a game brings its own. + if (ev.eventType == vr::VREvent_DashboardActivated || ev.eventType == vr::VREvent_DashboardDeactivated || + ev.eventType == vr::VREvent_DashboardOverlayCreated || ev.eventType == vr::VREvent_SceneApplicationChanged) + overlays.Kick(); if (ev.eventType == vr::VREvent_Quit) { sys->AcknowledgeQuit_Exiting(); From 0680297efa22b1d8d98ca7f391abbe51e906f92d Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:12:33 -0600 Subject: [PATCH 2/8] Pointer: sleep on the command socket while the pointer is off The main loop slept a fixed 8 ms, about 116 wakeups a second, whether the pointer was awake or not, and every second it looked up every overlay's handle and read a string property from all 64 device slots to find its own device. With the pointer off and hand gestures off, the loop now waits in poll() on its command socket for up to 250 ms, or 20 ms while mapped Frame controller buttons are being read (SteamVR input has no event to wait for). A mouse command ends the wait at once. The headset's activity level, the game check, and the "vrgame" and "gazeawake" repeats keep going at that pace. The 50 ms visibility poll and the 1 s handle lookups run only while the pointer is awake, and waking forces both. The device index is looked for only while it's unknown, and again after SteamVR activates or deactivates a device. The HMD pose history is kept only with hand gestures on, its one user. Co-Authored-By: Claude Opus 5.5 --- docs/design.md | 2 +- pointer/helper/ft-pointer.cpp | 42 ++++++++++++++++++++++++++--------- pointer/helper/vrbuttons.h | 8 +++++++ 3 files changed, 41 insertions(+), 11 deletions(-) diff --git a/docs/design.md b/docs/design.md index fb29cbf..10f697e 100644 --- a/docs/design.md +++ b/docs/design.md @@ -141,7 +141,7 @@ Replacing a loaded driver's files, as re-running the installer used to do, leave The dashboard follows whichever device summoned it or last pressed its trigger. Frametop adds "last used wins": moving a real controller releases the pointer, and the next mouse movement takes the laser back. Moving means faster than 0.35 m/s or 2 rad/s (both times `POINTER_CONTROLLER_PICKUP`, 1 by default) for 100 ms in a row, while the controller is tracked normally. A single sample over the limit used to be enough, and controllers resting on a desk took the laser back on a knock or a tracking jump while the mouse was in use. Small movements don't count; waking needs `POINTER_WAKE_COUNTS` of mouse motion within a second, so desk jitter doesn't steal the laser. While the pointer is awake, a tiny transparent overlay with `MakeOverlaysInteractiveIfVisible` keeps SteamVR's laser mouse on, since otherwise the first click would only switch the laser on. -When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off. +When the headset comes off, SteamVR reports its activity level as idle at once and turns the displays off 5 seconds later (`power.turnOffScreensTimeout`), unless something keeps it awake. An awake pointer did, and so did the helper's `vrcmd` runs: each is a new SteamVR client, and a new client every second kept SteamVR out of standby. The helper now releases the pointer as soon as the headset is idle, ignores the mouse until you're wearing it again, and pauses the overlay list whenever the pointer is off. With the pointer off it also stops running its loop every 8 ms, about 116 wakeups a second for nothing: it waits up to 250 ms for a command on its socket (20 ms while it reads mapped controller buttons, which SteamVR input only offers by polling), and leaves the overlay lookups until the pointer wakes. ### Moving floating windows diff --git a/pointer/helper/ft-pointer.cpp b/pointer/helper/ft-pointer.cpp index 4820c2c..a48c246 100644 --- a/pointer/helper/ft-pointer.cpp +++ b/pointer/helper/ft-pointer.cpp @@ -36,6 +36,13 @@ // pointer is off the helper also stops listing overlays with vrcmd, whose connection every // second kept SteamVR from going to standby. // +// Idle: while the pointer is off (and hand gestures are off), the main loop waits for a command +// on its socket for up to 250 ms instead of running every 8 ms, or 20 ms while it reads mapped +// Frame controller buttons (vrbuttons.h), which have no event to wait for. What has to go on +// meanwhile still does: the headset's activity level, the game check and the relay's "vrgame" +// and "gazeawake" repeats, and a mouse command wakes it at once. The overlay lookups (the 50 ms +// visibility poll, the once-a-second handles) wait for the pointer to wake, and run right then. +// // Last used wins: when a real controller moves (picked up), the pointer is released // (driver "hide", which also drops its hand role hint), so the controller gets // its role and laser back. The next mouse input reconnects and claims the laser again. @@ -329,6 +336,7 @@ extern "C" { #include #include +#include #include #include #include @@ -859,6 +867,8 @@ int main() { std::map sceneGraph; // no texture: plane test instead of ComputeOverlayIntersection std::map visible; // refreshed every 50 ms auto lastVisible = std::chrono::steady_clock::now(); + auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10); + unsigned listGeneration = 0; // the overlay list the handles were looked up from // The plane of the last panel the cursor was on, and the last point on it (panel edges). Vec3 edgePoint, edgeNormal, edgeLast; std::string edgeKey; @@ -901,6 +911,8 @@ int main() { wokeAt = t; active = true; recenter = true; + // Overlay handles and visibility weren't looked at while it was off (see "Idle" at the top). + lastSlow = lastVisible = t - std::chrono::seconds(10); SendTo(out, "ft_pointer", "role " + role); // POINTER_ROLE, before it takes it SendTo(out, "ft_pointer", "show"); claimPending = true; // take the laser without clicking, once SteamVR has bound the device @@ -1127,8 +1139,6 @@ int main() { return gazeOn && gazeMouseHeld && !inGame && !headsetOff && Clock::now() - gz.at < std::chrono::seconds(1) && !aimHeld && !leftHeld && !tilting && hold.src == Src::None && !clickPress && !clickRelease; }; - auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10); - unsigned listGeneration = 0; // the overlay list the handles were looked up from // --- Panel placement (see "Placement" at the top of the file) --- // Device pose, given in the standing universe, sent to the driver in raw space. @@ -1669,7 +1679,7 @@ int main() { } // Hands (see the top): pinches and grips from ft-hands. - { + if (handsOn) { vr::TrackedDevicePose_t h0; sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &h0, 1); if (h0.bPoseIsValid) poses.Add(tnow, h0.mDeviceToAbsoluteTracking); @@ -1968,9 +1978,10 @@ int main() { } followAt = tnow; - // Slow work, once a second and when the overlay list is new: overlay handles, our device index. + // Slow work, once a second and when the overlay list is new: overlay handles, our device + // index. Only while the pointer is awake (see "Idle" at the top). const auto now = std::chrono::steady_clock::now(); - if (now - lastSlow > std::chrono::seconds(1) || overlays.Generation() != listGeneration) { + if (active && (now - lastSlow > std::chrono::seconds(1) || overlays.Generation() != listGeneration)) { lastSlow = now; listGeneration = overlays.Generation(); handles.clear(); @@ -1992,16 +2003,16 @@ int main() { sceneGraph[key] = (tw == 0 || th == 0) && tt == vr::VROverlayTransform_Absolute && key.rfind("frametop.", 0) != 0; } - ours = vr::k_unTrackedDeviceIndexInvalid; - for (vr::TrackedDeviceIndex_t i = 0; i < vr::k_unMaxTrackedDeviceCount; ++i) { + // Our device keeps its index; it's looked for again when a device is activated or + // deactivated (the events below). + for (vr::TrackedDeviceIndex_t i = 0; ours == vr::k_unTrackedDeviceIndexInvalid && i < vr::k_unMaxTrackedDeviceCount; ++i) { char type[64] = ""; sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type); if (std::strcmp(type, "ft_pointer") == 0) ours = i; } - } - if (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size()) { + if (active && (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size())) { lastVisible = now; visible.clear(); for (const auto &[key, h] : handles) visible[key] = overlay->IsOverlayVisible(h); @@ -2320,6 +2331,8 @@ int main() { if (ev.eventType == vr::VREvent_DashboardActivated || ev.eventType == vr::VREvent_DashboardDeactivated || ev.eventType == vr::VREvent_DashboardOverlayCreated || ev.eventType == vr::VREvent_SceneApplicationChanged) overlays.Kick(); + if (ev.eventType == vr::VREvent_TrackedDeviceActivated || ev.eventType == vr::VREvent_TrackedDeviceDeactivated) + ours = vr::k_unTrackedDeviceIndexInvalid; if (ev.eventType == vr::VREvent_Quit) { sys->AcknowledgeQuit_Exiting(); @@ -2328,6 +2341,15 @@ int main() { return 0; } } - std::this_thread::sleep_for(std::chrono::milliseconds(8)); + // Idle (see the top): wait for a command instead. Not with anything still to finish (the + // claim pulse, a click's release, a press). + const bool idle = !active && !handsOn && !claimPending && !claimHeld && !clickRelease && !leftHeld && + hold.src == Src::None; + if (idle) { + pollfd p{in, POLLIN, 0}; + poll(&p, 1, controllerButtons.Watching(inGame) ? 20 : 250); + } else { + std::this_thread::sleep_for(std::chrono::milliseconds(8)); + } } } diff --git a/pointer/helper/vrbuttons.h b/pointer/helper/vrbuttons.h index 10d8f17..80d11a8 100644 --- a/pointer/helper/vrbuttons.h +++ b/pointer/helper/vrbuttons.h @@ -111,6 +111,14 @@ class ControllerButtons { } } + // Whether Poll reads any button now (the main loop can't sleep long while it does). + bool Watching(bool inGame) const { + if (!ok_ || (inGame && !games_ && !capture_)) return false; + for (int i = 0; i < kCount; ++i) + if (bound_[i]) return true; + return false; + } + // {"manifest":true,"global":false,"bound":[...],"active":[...]}: active = bound and // delivered (a controller that has the button is on, and SteamVR lets us have it). std::string Status() const { From e1f7ccee29dffad587f0b566b1e279153e89a6f3 Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:16:57 -0600 Subject: [PATCH 3/8] Pointer: skip unchanged work while the pointer is awake Every frame (about 116 a second) the helper tested the cursor ray against every visible overlay twice with ComputeOverlayIntersection, set the dot's alpha, width, transform and visibility (five calls into SteamVR), and sent the driver a pose datagram, even with the mouse and the head still. Now a frame reuses the last collision result when the mouse, the anchor (1 mm) and the eye (5 mm) haven't moved and no overlay showed, hid, or changed handle. The passes still run at least every 100 ms, since overlays move on their own (a floating window's controls follow it), and always while dragging. The dots' setters go to SteamVR only when their value changes: the placement when the dot moved 0.2 mm or the eye 5 mm, which turns or resizes it by well under 1%, and the width on a 0.5% change. The plain pose goes to the driver only when the laser's origin moved 0.2 mm or its direction 0.04 deg (0.1 mm where it lands, 15 cm on), and at least every 100 ms; the driver keeps the last pose and reports it every frame. A tilt's pose, a placement, or waking sends the next one regardless. Co-Authored-By: Claude Opus 5.5 --- docs/design.md | 2 +- pointer/helper/ft-pointer.cpp | 287 ++++++++++++++++++++++------------ 2 files changed, 186 insertions(+), 103 deletions(-) diff --git a/docs/design.md b/docs/design.md index 10f697e..c0907a3 100644 --- a/docs/design.md +++ b/docs/design.md @@ -116,7 +116,7 @@ The driver starts disconnected, because holding the right-hand role while SteamV ### The cursor -Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either. +Mouse motion turns into yaw and pitch around an anchor, the head position at the last recenter. A ray from the anchor is tested against every visible overlay with `ComputeOverlayIntersection`. On a hit, the cursor sits on that surface; otherwise it floats at `POINTER_DISTANCE`. Since the anchor isn't your current eye position, a second test runs along your line of sight to the cursor point, and anything nearer wins, so the cursor always lands on what you see under it. Overlays in `POINTER_IGNORE` are left out of both tests. A display-only panel, like a performance overlay locked to your view, has no input method, so SteamVR's laser passes through it, but `ComputeOverlayIntersection` still hits it, and the cursor stuck to it. The laser starts just before the cursor point, so an ignored panel nearer to you doesn't catch it either. Both tests ask SteamVR about every visible overlay, so a frame where nothing moved (the mouse, the anchor, the eye by more than 5 mm, which overlays show) reuses the last result, for up to 100 ms, since overlays can also move on their own. The dots' overlay settings go to SteamVR only when they change, and the pose goes to the driver, which keeps the last one, only when the laser would land 0.1 mm or more elsewhere, and at least every 100 ms. OpenVR has no call to list other programs' overlays, so the helper runs `vrcmd --overlays` in the background. It includes hidden overlays, because a floating window's controls only appear while something hovers the window, and the cursor has to find them immediately. Each run is a shell and a new SteamVR client, about 30 ms of CPU, and it ran every second while the pointer was awake, which in gaze mode is all the time. Now it runs every 20 seconds, and at once when the pointer wakes, when the dashboard opens or closes, when a game starts or ends, and when a left click hits nothing (a panel that came up since). An overlay already on the list showing or hiding needs no new list: the helper checks the visibility of the ones it knows every 50 ms. diff --git a/pointer/helper/ft-pointer.cpp b/pointer/helper/ft-pointer.cpp index a48c246..3acb9cb 100644 --- a/pointer/helper/ft-pointer.cpp +++ b/pointer/helper/ft-pointer.cpp @@ -43,6 +43,15 @@ // and "gazeawake" repeats, and a mouse command wakes it at once. The overlay lookups (the 50 ms // visibility poll, the once-a-second handles) wait for the pointer to wake, and run right then. // +// Unchanged frames: while the pointer is awake, a frame where nothing moved reuses the last one's +// result. The collision passes (ComputeOverlayIntersection on every visible overlay, twice) run +// again when the mouse or the anchor moved, the eye moved 5 mm, an overlay showed or hid, or +// 100 ms have passed (overlays also move by themselves), and always while dragging. The dots' +// setters go to SteamVR only when their value changes, and a dot's placement only when it moved +// 0.2 mm or the eye 5 mm (its facing and size then change by well under 1%). The pose goes to the +// driver, which keeps the last one, when the laser's origin moved 0.2 mm or its direction 0.04 +// deg (0.1 mm where it lands, 15 cm on), and at least every 100 ms. +// // Last used wins: when a real controller moves (picked up), the pointer is released // (driver "hide", which also drops its hand role hint), so the controller gets // its role and laser back. The next mouse input reconnects and claims the laser again. @@ -665,6 +674,36 @@ bool Ignored(const std::vector &patterns, const std::string &key) { return false; } +vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye); + +// One of our dots, with what was last set on it: each setter goes to SteamVR only when its value +// changes (see "Unchanged frames" at the top). +struct DotOverlay { + vr::VROverlayHandle_t h = vr::k_ulOverlayHandleInvalid; + int shown = -1; // unknown at first + float alpha = -1, width = -1; + bool placed = false; + Vec3 at, eye; + void Show(bool on) { + if (shown == int(on)) return; + shown = on; + if (on) vr::VROverlay()->ShowOverlay(h); + else vr::VROverlay()->HideOverlay(h); + } + void Alpha(float a) { + if (a != alpha) vr::VROverlay()->SetOverlayAlpha(h, alpha = a); + } + // Width w, at `to`, facing `from`: left as it is for a change under 0.5% in width, and while + // it moved less than 0.2 mm and the eye less than 5 mm. + void Place(Vec3 to, Vec3 from, float w) { + if (std::fabs(w - width) > 0.005f * width) vr::VROverlay()->SetOverlayWidthInMeters(h, width = w); + if (placed && Length(to - at) < 0.0002 && Length(from - eye) < 0.005) return; + placed = true, at = to, eye = from; + const auto m = Billboard(to, from); + vr::VROverlay()->SetOverlayTransformAbsolute(h, vr::TrackingUniverseStanding, &m); + } +}; + vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye) { // Overlay faces +Z; point +Z at the eye, keep +Y roughly up. const Vec3 z = Normalize(eye - at); @@ -848,6 +887,8 @@ int main() { below.m[1][3] = -50; overlay->SetOverlayTransformTrackedDeviceRelative(laserMode, vr::k_unTrackedDeviceIndex_Hmd, &below); bool laserModeShown = false; + DotOverlay cursorDot, markerDot; // the setters for `cursor` and `marker` (see "Unchanged frames" at the top) + cursorDot.h = cursor, markerDot.h = marker; // Controller beams keep the user's width; nothing here changes it any more. vr::VRSettings()->SetFloat("dashboard", "laserRayWidthScale", laserWidth); @@ -875,6 +916,26 @@ int main() { bool active = false, recenter = false, anchored = false; using Clock = std::chrono::steady_clock; + // Unchanged frames (see the top): visibleGen goes up when an overlay shows or hides or the + // handles change; pass is the last collision pass, posed the last plain pose sent. + unsigned visibleGen = 0; + struct Pass { + bool valid = false; + Clock::time_point at{}; + unsigned gen = 0; + double yaw = 0, pitch = 0; + Vec3 anchor, eye; + double best = 1e9, distance = 0; + std::string key; + bool scene = false, onEdge = false, occluded = false; + Vec3 point; + } pass; + struct Posed { + bool valid = false; + Clock::time_point at{}; + Vec3 origin; + double yaw = 0, pitch = 0; + } posed; Clock::time_point lastMouse{}, claimAt{}, claimRelease{}, wokeAt{}, noWakeUntil{}; bool claimPending = false, claimHeld = false; // Last used wins: since when each controller has been moving (zero: it isn't). @@ -913,6 +974,7 @@ int main() { recenter = true; // Overlay handles and visibility weren't looked at while it was off (see "Idle" at the top). lastSlow = lastVisible = t - std::chrono::seconds(10); + pass.valid = posed.valid = false; SendTo(out, "ft_pointer", "role " + role); // POINTER_ROLE, before it takes it SendTo(out, "ft_pointer", "show"); claimPending = true; // take the laser without clicking, once SteamVR has bound the device @@ -1154,6 +1216,7 @@ int main() { char msg[200]; std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", o.x, o.y, o.z, q[0], q[1], q[2], q[3]); SendTo(out, "ft_pointer", msg); + posed.valid = false; }; auto sleepMs = [](int ms) { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); }; auto headPos = [&] { @@ -1165,8 +1228,8 @@ int main() { auto borrow = [&] { SendTo(out, "ft_pointer", "show"); overlay->ShowOverlay(laserMode); - overlay->HideOverlay(cursor); - overlay->HideOverlay(marker); + cursorDot.Show(false); + markerDot.Show(false); sleepMs(active ? 50 : 400); // a fresh connect needs SteamVR to bind the device SendTo(out, "ft_pointer", "btn a 1"); sleepMs(60); @@ -1307,8 +1370,8 @@ int main() { if (headsetOff && active) { active = false; claimPending = claimHeld = false; - overlay->HideOverlay(cursor); - overlay->HideOverlay(marker); + cursorDot.Show(false); + markerDot.Show(false); SendTo(out, "ft_pointer", "btn a 0"); SendTo(out, "ft_pointer", "hide"); std::printf("headset off: pointer released\n"); @@ -1539,6 +1602,7 @@ int main() { } else if (std::strncmp(buf, "reload", 6) == 0) { loadConfig(); lastSlow = Clock::now() - std::chrono::seconds(10); // apply POINTER_IGNORE now + pass.valid = false; std::printf("reloaded: free distance %.2f m, dot %.2f deg, origin %.2f, head follow %s, leash %.0f deg, " "controller pickup %.1fx, %zu ignored\n", freeDistance, cursorDeg, originFraction, follow ? "on" : "off", leashDeg, pickupScale, @@ -1556,8 +1620,8 @@ int main() { if (!active) wake(Clock::now()); } else if (std::strncmp(buf, "hide", 4) == 0) { active = false; - overlay->HideOverlay(cursor); - overlay->HideOverlay(marker); + cursorDot.Show(false); + markerDot.Show(false); SendTo(out, "ft_pointer", "hide"); } else { SendTo(out, "ft_pointer", buf); // btn, scroll @@ -1599,8 +1663,8 @@ int main() { sys->GetControllerRoleForTrackedDeviceIndex(ours) == vr::TrackedControllerRole_Invalid) { active = false; claimPending = claimHeld = false; - overlay->HideOverlay(cursor); - overlay->HideOverlay(marker); + cursorDot.Show(false); + markerDot.Show(false); SendTo(out, "ft_pointer", "btn a 0"); SendTo(out, "ft_pointer", "hide"); noWakeUntil = tnow + std::chrono::seconds(2); @@ -1628,8 +1692,8 @@ int main() { if (tnow - movingSince[i] >= std::chrono::milliseconds(100)) { active = false; claimPending = claimHeld = false; - overlay->HideOverlay(cursor); - overlay->HideOverlay(marker); + cursorDot.Show(false); + markerDot.Show(false); SendTo(out, "ft_pointer", "btn a 0"); SendTo(out, "ft_pointer", "hide"); std::printf("controller %u moved (%.2f m/s, %.1f rad/s): pointer released\n", i, speed, spin); @@ -1984,6 +2048,7 @@ int main() { if (active && (now - lastSlow > std::chrono::seconds(1) || overlays.Generation() != listGeneration)) { lastSlow = now; listGeneration = overlays.Generation(); + const auto before = handles; handles.clear(); for (const auto &key : overlays.Keys()) { if (Ignored(ignore, key)) continue; @@ -2003,6 +2068,7 @@ int main() { sceneGraph[key] = (tw == 0 || th == 0) && tt == vr::VROverlayTransform_Absolute && key.rfind("frametop.", 0) != 0; } + if (handles != before) ++visibleGen; // Our device keeps its index; it's looked for again when a device is activated or // deactivated (the events below). for (vr::TrackedDeviceIndex_t i = 0; ours == vr::k_unTrackedDeviceIndexInvalid && i < vr::k_unMaxTrackedDeviceCount; ++i) { @@ -2014,8 +2080,9 @@ int main() { if (active && (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size())) { lastVisible = now; - visible.clear(); - for (const auto &[key, h] : handles) visible[key] = overlay->IsOverlayVisible(h); + std::map seen; + for (const auto &[key, h] : handles) seen[key] = overlay->IsOverlayVisible(h); + if (seen != visible) visible.swap(seen), ++visibleGen; } if (active && anchored && hmd.bPoseIsValid && tilting) { @@ -2025,8 +2092,8 @@ int main() { tiltOrigin = lastOrigin; tiltBasis = AimBasis(lastAim); tiltStart = false; // angles carry on from any earlier tilt in this drag - overlay->HideOverlay(cursor); - overlay->HideOverlay(marker); + cursorDot.Show(false); + markerDot.Show(false); } const double yr = tiltYaw * M_PI / 180, pr = tiltPitch * M_PI / 180; const Vec3 up{0, 1, 0}, side = tiltBasis.x; @@ -2042,6 +2109,7 @@ int main() { std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", originRaw.x, originRaw.y, originRaw.z, q[0], q[1], q[2], q[3]); SendTo(out, "ft_pointer", msg); + posed.valid = false; } else if (active && anchored && hmd.bPoseIsValid) { const bool dragging = leftHeld || tnow < dropHoldUntil; if (!dragging) tiltYaw = tiltPitch = 0; // drop finished: back to plain pointing @@ -2085,85 +2153,97 @@ int main() { } return h; }; - Hit first; - if (!dragging) first = nearest(anchor, dir); - double best = first.along; - std::string bestKey = first.key; - bool bestScene = first.scene; - Vec3 bestPoint = first.point, bestNormal = first.normal; - bool onEdge = false; - if (!dragging && best < 1e8 && !bestScene) { - edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint; - } else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible[edgeKey]) { - // Just off a panel: stay on its plane (see "Panel edges" at the top). - const double denom = Dot(dir, edgeNormal); - if (std::fabs(denom) > 1e-4) { - const double along = Dot(edgePoint - anchor, edgeNormal) / denom; - const Vec3 at = anchor + dir * along; - if (along > 0.05 && std::sqrt(Dot(at - edgeLast, at - edgeLast)) <= edgeReach) { - best = along; - bestKey = edgeKey; - onEdge = true; + // Unchanged frames (see the top): the last pass still holds. + const bool reuse = !dragging && pass.valid && tnow - pass.at < std::chrono::milliseconds(100) && + pass.gen == visibleGen && std::fabs(yaw - pass.yaw) < 1e-6 && + std::fabs(pitch - pass.pitch) < 1e-6 && Length(anchor - pass.anchor) < 0.001 && + Length(eye - pass.eye) < 0.005; + double best = pass.best, distance = pass.distance; + std::string bestKey = pass.key; + bool bestScene = pass.scene, onEdge = pass.onEdge, occluded = pass.occluded; + Vec3 point = pass.point; + if (!reuse) { + Hit first; + if (!dragging) first = nearest(anchor, dir); + best = first.along; + bestKey = first.key; + bestScene = first.scene; + Vec3 bestPoint = first.point, bestNormal = first.normal; + onEdge = false; + if (!dragging && best < 1e8 && !bestScene) { + edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint; + } else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible[edgeKey]) { + // Just off a panel: stay on its plane (see "Panel edges" at the top). + const double denom = Dot(dir, edgeNormal); + if (std::fabs(denom) > 1e-4) { + const double along = Dot(edgePoint - anchor, edgeNormal) / denom; + const Vec3 at = anchor + dir * along; + if (along > 0.05 && std::sqrt(Dot(at - edgeLast, at - edgeLast)) <= edgeReach) { + best = along; + bestKey = edgeKey; + onEdge = true; + } } } - } - // Held on one of ft-screens' panels that stays where it is: onto whichever of them - // the ray meets (see the top). - if (leftHeld && !pressKey.empty()) { - vr::ETrackingUniverseOrigin uo; - vr::HmdMatrix34_t now{}; - auto it = handles.find(pressKey); - bool still = it != handles.end() && - overlay->GetOverlayTransformAbsolute(it->second, &uo, &now) == vr::VROverlayError_None; - for (int i = 0; still && i < 3; ++i) - for (int j = 0; j < 4; ++j) - if (std::fabs(now.m[i][j] - pressPose.m[i][j]) > 0.001f) still = false; - if (still) { - Hit h; - for (const auto &[key, handle] : handles) { - if (!visible[key] || !FramePanel(key)) continue; - vr::VROverlayIntersectionParams_t params{}; - params.vSource = {float(anchor.x), float(anchor.y), float(anchor.z)}; - params.vDirection = {float(dir.x), float(dir.y), float(dir.z)}; - params.eOrigin = vr::TrackingUniverseStanding; - vr::VROverlayIntersectionResults_t r{}; - if (overlay->ComputeOverlayIntersection(handle, ¶ms, &r) && r.fDistance > 0.05f && - r.fDistance < h.along) - h.along = r.fDistance, h.key = key; + // Held on one of ft-screens' panels that stays where it is: onto whichever of them + // the ray meets (see the top). + if (leftHeld && !pressKey.empty()) { + vr::ETrackingUniverseOrigin uo; + vr::HmdMatrix34_t now{}; + auto it = handles.find(pressKey); + bool still = it != handles.end() && + overlay->GetOverlayTransformAbsolute(it->second, &uo, &now) == vr::VROverlayError_None; + for (int i = 0; still && i < 3; ++i) + for (int j = 0; j < 4; ++j) + if (std::fabs(now.m[i][j] - pressPose.m[i][j]) > 0.001f) still = false; + if (still) { + Hit h; + for (const auto &[key, handle] : handles) { + if (!visible[key] || !FramePanel(key)) continue; + vr::VROverlayIntersectionParams_t params{}; + params.vSource = {float(anchor.x), float(anchor.y), float(anchor.z)}; + params.vDirection = {float(dir.x), float(dir.y), float(dir.z)}; + params.eOrigin = vr::TrackingUniverseStanding; + vr::VROverlayIntersectionResults_t r{}; + if (overlay->ComputeOverlayIntersection(handle, ¶ms, &r) && r.fDistance > 0.05f && + r.fDistance < h.along) + h.along = r.fDistance, h.key = key; + } + if (h.along < 1e8) dragDistance = h.along, lastHit = h.key; + } else { + pressKey.clear(); // carried: the lock holds for the rest of this press } - if (h.along < 1e8) dragDistance = h.along, lastHit = h.key; - } else { - pressKey.clear(); // carried: the lock holds for the rest of this press } - } - // While dragging: keep the press-time distance and show the non-interactive marker. - // On a scene-graph plane or a panel's edge: the laser-catching dot goes 5 cm behind it. - double distance = dragging ? dragDistance : (best < 1e8 ? best : freeDistance); - Vec3 point = anchor + dir * distance; - bool occluded = false; - if (!dragging) { - // The cursor lands on what you see under it: the ray above starts at the anchor, - // not the eye, so after leaning it can pick a panel that something nearer - // covers from where you are now (panels close together in view, at different - // depths). Anything in front of the point on the eye's line of sight wins. - const double toPoint = std::sqrt(Dot(point - eye, point - eye)); - const Hit front = nearest(eye, Normalize(point - eye)); - if (front.along < toPoint - 0.02) { - occluded = true; - bestKey = front.key, bestScene = front.scene; - point = front.point; - if (!front.scene) edgeKey = front.key, edgePoint = front.point, edgeNormal = Normalize(front.normal), - edgeLast = front.point; - distance = std::sqrt(Dot(point - anchor, point - anchor)); - best = distance; - onEdge = false; + // While dragging: keep the press-time distance and show the non-interactive marker. + // On a scene-graph plane or a panel's edge: the laser-catching dot goes 5 cm behind it. + distance = dragging ? dragDistance : (best < 1e8 ? best : freeDistance); + point = anchor + dir * distance; + occluded = false; + if (!dragging) { + // The cursor lands on what you see under it: the ray above starts at the anchor, + // not the eye, so after leaning it can pick a panel that something nearer + // covers from where you are now (panels close together in view, at different + // depths). Anything in front of the point on the eye's line of sight wins. + const double toPoint = std::sqrt(Dot(point - eye, point - eye)); + const Hit front = nearest(eye, Normalize(point - eye)); + if (front.along < toPoint - 0.02) { + occluded = true; + bestKey = front.key, bestScene = front.scene; + point = front.point; + if (!front.scene) + edgeKey = front.key, edgePoint = front.point, edgeNormal = Normalize(front.normal), + edgeLast = front.point; + distance = std::sqrt(Dot(point - anchor, point - anchor)); + best = distance; + onEdge = false; + } + lastDistance = distance, lastHit = bestKey; } - lastDistance = distance, lastHit = bestKey; } const bool onScene = !dragging && ((bestScene && best < 1e8) || onEdge); const bool onPanel = dragging || (best < 1e8 && !onScene); const Vec3 sight = Normalize(point - eye); - if (!dragging) { + if (!dragging && !reuse) { // SteamVR Settings (see the top): the dashboard's main panel is hidden and its // scene-graph panel shows the page. onVrSettings = false; @@ -2179,6 +2259,10 @@ int main() { } } } + if (!reuse) { + pass = {!dragging, tnow, visibleGen, yaw, pitch, anchor, eye, + best, distance, bestKey, bestScene, onEdge, occluded, point}; + } if (onVrSettings != catcherHidesHit) { overlay->SetOverlayFlag(cursor, vr::VROverlayFlags_HideLaserIntersection, onVrSettings); catcherHidesHit = onVrSettings; @@ -2195,20 +2279,16 @@ int main() { alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0); } if (tnow < calPanelUntil) alpha = 0; - overlay->SetOverlayAlpha(marker, float(alpha)); - overlay->SetOverlayWidthInMeters(marker, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360))); - auto mm = Billboard(near, eye); - overlay->SetOverlayTransformAbsolute(marker, vr::TrackingUniverseStanding, &mm); - overlay->SetOverlayAlpha(cursor, 0); - overlay->SetOverlayWidthInMeters(cursor, float(2 * SETTINGS_CATCHER * std::tan(cursorDeg * M_PI / 360))); - auto mc = Billboard(far, eye); - overlay->SetOverlayTransformAbsolute(cursor, vr::TrackingUniverseStanding, &mc); - overlay->ShowOverlay(marker); - overlay->ShowOverlay(cursor); + markerDot.Alpha(float(alpha)); + markerDot.Place(near, eye, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360))); + cursorDot.Alpha(0); + cursorDot.Place(far, eye, float(2 * SETTINGS_CATCHER * std::tan(cursorDeg * M_PI / 360))); + markerDot.Show(true); + cursorDot.Show(true); } else { // On a panel: the non-interactive marker, pulled 5 mm toward the eye so it // draws on top. In free space: the interactive dot the laser lands on. - const vr::VROverlayHandle_t show = onPanel ? marker : cursor, hide = onPanel ? cursor : marker; + DotOverlay &show = onPanel ? markerDot : cursorDot, &hide = onPanel ? cursorDot : markerDot; const Vec3 at = onPanel ? point + Normalize(eye - point) * 0.005 : onScene ? point + dir * 0.05 : point; const double dist = std::sqrt(Dot(at - eye, at - eye)); // Gaze mode: a pulse for each click, and with POINTER_GAZE_DOT=moving, shown only @@ -2225,12 +2305,10 @@ int main() { } } if (tnow < calPanelUntil) alpha = 0; // the calibration panel is up (see the top) - overlay->SetOverlayAlpha(show, float(alpha)); - overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360))); - auto m = Billboard(at, eye); - overlay->SetOverlayTransformAbsolute(show, vr::TrackingUniverseStanding, &m); - overlay->ShowOverlay(show); - overlay->HideOverlay(hide); + show.Alpha(float(alpha)); + show.Place(at, eye, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360))); + show.Show(true); + hide.Show(false); } // Controller ray: from the eye, aimed at the cursor point, converted from the @@ -2275,10 +2353,15 @@ int main() { std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", oRaw.x, oRaw.y, oRaw.z, q[0], q[1], q[2], q[3]); SendTo(out, "ft_pointer", msg); - } else { + posed.valid = false; + } else if (!posed.valid || tnow - posed.at >= std::chrono::milliseconds(100) || + Length(eyeRaw - posed.origin) >= 0.0002 || std::fabs(std::remainder(ayaw - posed.yaw, 360.0)) >= 0.04 || + std::fabs(apitch - posed.pitch) >= 0.04) { + // The driver keeps the last pose: only a change goes out (see "Unchanged frames" at the top). char msg[160]; std::snprintf(msg, sizeof msg, "pose %.5f %.5f %.5f %.4f %.4f", eyeRaw.x, eyeRaw.y, eyeRaw.z, ayaw, apitch); SendTo(out, "ft_pointer", msg); + posed = {true, tnow, eyeRaw, ayaw, apitch}; } } From d5a15ebc3695c1f3b6004123dbb4b03ec11d9e69 Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:18:43 -0600 Subject: [PATCH 4/8] Input relay: never block on the pointer helper's socket The relay sent to @ft_pointer_helper on a blocking socket. When the helper stalled, a layout placement or grabprobe holds it for seconds while ft-gazed keeps filling its socket at 90 Hz, the relay's one loop blocked with it: keyboards, the volume keys (which must never reach gamescope), and pausing all stopped until the helper read again. The socket is non-blocking now. A command the helper doesn't take (EAGAIN) waits in a queue, and everything after it queues behind it so the order holds; tick() sends what it can on each loop, and the select timeout drops to 20 ms while anything waits. Mouse moves add up into one queued move. A scroll notch is dropped rather than queued, since scrolling seconds late is no use; its release still goes. Presses, releases, show, hide, and the rest are kept, so no button stays down. The queue holds at most 512 commands. While paused, the configured pointer's queue still drains, so the releases and "hide" from standing down arrive. A "vrbind" that hits a full socket is sent again on the next loop instead of being lost. Co-Authored-By: Claude Opus 5.5 --- docs/design.md | 2 ++ input/input-relay.py | 85 ++++++++++++++++++++++++++++++++++++++------ 2 files changed, 76 insertions(+), 11 deletions(-) diff --git a/docs/design.md b/docs/design.md index c0907a3..8ecb1c4 100644 --- a/docs/design.md +++ b/docs/design.md @@ -153,6 +153,8 @@ SteamVR opens every input device only when it starts. When a Bluetooth mouse sle Keyboards aren't grabbed by default, because a grabbed keyboard's keys went into a virtual keyboard nothing typed from; the relay forwards them to ft-screens instead. +The relay never waits on the pointer helper. Its socket to the helper used to block, so when the helper stalled (a layout placement or `grabprobe` holds it for seconds, and the gaze service fills its socket 90 times a second meanwhile), the whole relay stopped with it: keyboards, the volume keys, and pausing. Now what the helper doesn't take waits in order and goes out on the next loops. Mouse moves add up into one while they wait, and a scroll notch is dropped, since scrolling seconds late is no use; presses and releases are kept, so no button stays down. + An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's `InputStealer`, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's `OverlayFocusChanged` (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. So with `SHARE_KEYS=1` the relay sends a grabbed keyboard's keys to `@frametop_keys` as datagrams (`key `). It's off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first gets every key typed into the desktop, passwords included. A listener should accept only its own user (`SO_PASSCRED`) and skip any keyboard of its own that the relay grabs too. Volume keys must never reach gamescope. With the openvr backend, gamescope sends volume up and down to Steam by moving keyboard focus to Steam for the key and then back to the previously focused surface. When nothing had focus, the one it moves back to is null, and wlroots aborts on a null focus surface (`wlr_seat_keyboard_notify_enter: Assertion 'surface' failed`), which ends the whole VR session. Keyboard focus is often empty while you work in VR, so one press of the headset's volume button could take everything down. gamescope reads the headset's buttons and every keyboard itself (`InputStealer`), as do SteamVR's processes, so the relay has to stop volume keys at the device. Grabbing `gpio-keys` would also take the headset's click button, so the relay remaps the volume entries in each device's keymap (`EVIOCSKEYCODE`) and handles the stand-in codes itself. That fix covers every device at once, including keyboards that aren't grabbed. diff --git a/input/input-relay.py b/input/input-relay.py index 5d935b7..f2289cd 100755 --- a/input/input-relay.py +++ b/input/input-relay.py @@ -128,6 +128,7 @@ kernel's evdev and uinput interfaces. """ import array import atexit +import collections import errno import fcntl import json @@ -459,9 +460,15 @@ class Pointer: CLAIM_PULSE = 0.06 # seconds the claim button (switchlaserhand, no click) is held RESUME_PAUSE = 1.5 # mouse idle this long, then moving again, re-claims the laser WAKE_WINDOW = 1.0 # seconds in which WAKE_COUNTS of motion must add up + QUEUE_MAX = 512 # commands kept while the helper is behind (see send) def __init__(self, sensitivity, idle, wake_counts=40): - self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) + # Never blocks (see send): a stalled helper must not stall the keyboard, volume keys and pausing. + self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK) + # Commands the helper's full socket didn't take yet, in order: text, or [dyaw, dpitch] for + # mouse moves, which add up into one while they wait. + self.queue = collections.deque() + self.behind_logged = -60.0 self.sensitivity = sensitivity # degrees per mouse count self.idle = idle self.active = False @@ -479,11 +486,59 @@ class Pointer: self.pending_since = 0.0 self.gaze_awake_until = 0.0 # the helper's gaze mode keeps the pointer until then - def send(self, command): - try: - self.sock.sendto(command.encode(), HELPER) - except OSError: - pass # helper not running (SteamVR not running) + def send(self, command, droppable=False): + """To the helper, in order, without blocking. While the helper doesn't keep up (place and + grabprobe hold it for seconds, and ft-gazed's 90 Hz gaze fills its socket meanwhile), commands + wait in the queue and go out from tick(). A droppable one (a scroll notch: scrolling seconds + late is no use) is dropped instead; presses, releases and the rest are kept, so no button + stays down. Moves add up (_move).""" + if not self.queue: + try: + self.sock.sendto(command.encode(), HELPER) + return + except BlockingIOError: + pass + except OSError: + return # helper not running (SteamVR not running) + if not droppable: + self._queue(command) + + def _move(self, dyaw, dpitch): + if self.queue and isinstance(self.queue[-1], list): + self.queue[-1][0] += dyaw + self.queue[-1][1] += dpitch + return + if not self.queue: + try: + self.sock.sendto(f"move {dyaw:.4f} {dpitch:.4f}".encode(), HELPER) + return + except BlockingIOError: + pass + except OSError: + return + self._queue([dyaw, dpitch]) + + def _queue(self, item): + if not self.queue and time.monotonic() - self.behind_logged > 60: + self.behind_logged = time.monotonic() + log("pointer helper is behind: holding its commands (logged once a minute)") + self.queue.append(item) + if len(self.queue) > self.QUEUE_MAX: + self.queue.popleft() # stalled for long: the oldest goes + + def drain(self): + """Send what waits, in order, as far as the helper takes it.""" + while self.queue: + item = self.queue[0] + text = f"move {item[0]:.4f} {item[1]:.4f}" if isinstance(item, list) else item + try: + self.sock.sendto(text.encode(), HELPER) + except BlockingIOError: + return + except OSError: + self.queue.clear() # the helper went away: nothing to deliver to + return + self.queue.popleft() def wake(self, now): if not self.active: @@ -511,7 +566,7 @@ class Pointer: elif code == REL_Y: self.dy += value elif code == REL_WHEEL and value: - self.send(f"scroll 0 {1 if value > 0 else -1}") + self.send(f"scroll 0 {1 if value > 0 else -1}", droppable=True) self.scroll_until = now + self.SCROLL_PULSE def action(self, name, value, now, source="mouse"): @@ -543,7 +598,7 @@ class Pointer: return # the rest act on press elif name in ("scroll_up", "scroll_down"): self.wake(now) - self.send(f"scroll 0 {1 if name == 'scroll_up' else -1}") + self.send(f"scroll 0 {1 if name == 'scroll_up' else -1}", droppable=True) self.scroll_until = now + self.SCROLL_PULSE elif name == "dashboard": self.dashboard(now) @@ -585,7 +640,7 @@ class Pointer: def flush(self): if self.dx or self.dy: # Mouse right turns the ray right (negative yaw); mouse down tilts it down. - self.send(f"move {-self.dx * self.sensitivity:.4f} {-self.dy * self.sensitivity:.4f}") + self._move(-self.dx * self.sensitivity, -self.dy * self.sensitivity) self.dx = self.dy = 0 def dashboard(self, now=None): @@ -601,6 +656,7 @@ class Pointer: self.system_at = now + (0.4 if woke else 0.0) def tick(self, now): + self.drain() if self.system_at is not None and now >= self.system_at: self.send("btn system 1") self.system_at = None @@ -625,7 +681,7 @@ class Pointer: def timeout(self): pending = (self.scroll_until, self.claim_at, self.claim_release, self.system_at, self.system_release) - return 0.02 if any(t is not None for t in pending) else 0.5 + return 0.02 if self.queue or any(t is not None for t in pending) else 0.5 def stand_down(self): """Frametop is pausing: a pulse under way ends now, and the pointer lets go.""" @@ -729,7 +785,7 @@ def main(): # pointer: the 3D mouse while it's in use, pointer_conf: the one the config asks for (they # differ while Frametop is paused). state = {"pointer": None, "pointer_conf": None, "rules": {}, "share_keys": False, - "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0} + "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0, "vr_bind_retry": False} def pause_changed(paused): """Frametop paused or resumed (game_pause.py): the relay's own part.""" @@ -779,8 +835,11 @@ def main(): and a not in GAZE_ACTIONS and (not pause.paused or works_paused(a))) or "-" if state["rules"].get("controller_in_games"): buttons = "+games " + buttons + state["vr_bind_retry"] = False try: screens_sock.sendto(f"vrbind {buttons}".encode(), HELPER) + except BlockingIOError: + state["vr_bind_retry"] = True # the helper is behind: again on the next loop except OSError: pass # helper not running; it says vrhello when it starts @@ -1238,6 +1297,10 @@ def main(): now = time.monotonic() if pointer: pointer.tick(now) + elif state["pointer_conf"]: + state["pointer_conf"].drain() # paused: what waited still goes, in order + if state["vr_bind_retry"]: + vr_bind(now) volume.tick(now) pause.tick(now) if state["vr_capture_until"] and now >= state["vr_capture_until"]: From 83850bb1b7266f429e6d9e26bed7dcbc2bbeb153 Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:19:38 -0600 Subject: [PATCH 5/8] Pointer driver: parse outside the lock, report only changes Handle() held the state lock through a chain of up to a dozen sscanf calls per command, and RunFrame, which vrserver calls every frame, takes the same lock, so a burst of commands (about 116 poses a second, plus moves and buttons) could hold up vrserver's frame. Commands are now parsed into locals first, and the lock is held only to store the result. RunFrame also called UpdateBooleanComponent six times and UpdateScalarComponent twice every frame, and TrackedDevicePoseUpdated every frame even while disconnected. Components now go to SteamVR only when they change (all of them on the first frame). The pose still goes out every frame while the device is connected, as a tracked device's should; the disconnected pose goes out once. The helper now sends a pose only when it changes, so the comment says the driver keeps the last one. Co-Authored-By: Claude Opus 5.5 --- pointer/driver/driver_ft_pointer.cpp | 144 ++++++++++++++++++--------- 1 file changed, 98 insertions(+), 46 deletions(-) diff --git a/pointer/driver/driver_ft_pointer.cpp b/pointer/driver/driver_ft_pointer.cpp index 5df2609..63f6949 100644 --- a/pointer/driver/driver_ft_pointer.cpp +++ b/pointer/driver/driver_ft_pointer.cpp @@ -12,7 +12,7 @@ // aim absolute direction (yaw 0 = -Z, the SteamVR forward) // gaze follow the head (origin and direction) again // distance cursor distance from the anchor (default 1.5) -// pose exact pose, sent every frame by the ft-pointer helper +// pose exact pose, from the ft-pointer helper when it changes (it holds) // posq exact pose with a full rotation (tilting a panel while moving it) // btn <0|1> name: trigger, b, x, system, joystick, a (a = claim the laser, no click) // scroll joystick deflection -1..1 @@ -224,12 +224,23 @@ public: pose.result = ok ? TrackingResult_Running_OK : TrackingResult_Uninitialized; pose.deviceIsConnected = snapshot.visible; pose_ = pose; - VRServerDriverHost()->TrackedDevicePoseUpdated(objectId_, pose_, sizeof(DriverPose_t)); + // Connected, the pose goes out every frame, as SteamVR expects of a tracked device. + // Disconnected, once: it says the same thing every frame after. + if (snapshot.visible || !reportedOff_) { + VRServerDriverHost()->TrackedDevicePoseUpdated(objectId_, pose_, sizeof(DriverPose_t)); + reportedOff_ = !snapshot.visible; + } + // Inputs only when they change (and all of them the first time). auto input = VRDriverInput(); - for (int i = 0; i < kButtons; ++i) input->UpdateBooleanComponent(buttons_[i], snapshot.buttons[i], 0); - input->UpdateScalarComponent(scrollX_, snapshot.scrollX, 0); - input->UpdateScalarComponent(scrollY_, snapshot.scrollY, 0); + for (int i = 0; i < kButtons; ++i) + if (!inputsSent_ || snapshot.buttons[i] != sentButtons_[i]) + input->UpdateBooleanComponent(buttons_[i], sentButtons_[i] = snapshot.buttons[i], 0); + if (!inputsSent_ || snapshot.scrollX != sentScrollX_) + input->UpdateScalarComponent(scrollX_, sentScrollX_ = snapshot.scrollX, 0); + if (!inputsSent_ || snapshot.scrollY != sentScrollY_) + input->UpdateScalarComponent(scrollY_, sentScrollY_ = snapshot.scrollY, 0); + inputsSent_ = true; } private: @@ -239,6 +250,10 @@ private: int32_t role_ = TrackedControllerRole_RightHand; bool hinted_ = false; // whether the role hint currently claims role_ DriverPose_t pose_{}; + bool reportedOff_ = false; // the disconnected pose has gone out (it isn't repeated) + // What the input components were last set to. + bool inputsSent_ = false, sentButtons_[kButtons] = {}; + float sentScrollX_ = 0.f, sentScrollY_ = 0.f; VRInputComponentHandle_t buttons_[kButtons] = {}; VRInputComponentHandle_t scrollX_ = 0, scrollY_ = 0; }; @@ -293,65 +308,102 @@ private: } } + // Parsed first, then applied under the lock, which RunFrame (vrserver's frame) takes too. void Handle(const char *cmd) { - std::lock_guard guard(state_.lock); + enum { kNone, kPosq, kPose, kMove, kRecenter, kAim, kGaze, kHide, kShow, kRole, kBtn, kDistance, kScroll }; + int kind = kNone; char name[32]; - float a, b; - int v; - float x, y, z, qw, qx, qy, qz; - if (std::sscanf(cmd, "posq %f %f %f %f %f %f %f", &x, &y, &z, &qw, &qx, &qy, &qz) == 7) { + float f[7]; + int v = 0, role = 0, button = -1; + if (std::sscanf(cmd, "posq %f %f %f %f %f %f %f", &f[0], &f[1], &f[2], &f[3], &f[4], &f[5], &f[6]) == 7) { + kind = kPosq; + } else if (std::sscanf(cmd, "pose %f %f %f %f %f", &f[0], &f[1], &f[2], &f[3], &f[4]) == 5) { + kind = kPose; + } else if (std::sscanf(cmd, "move %f %f", &f[0], &f[1]) == 2) { + kind = kMove; + } else if (std::strncmp(cmd, "recenter", 8) == 0) { + kind = kRecenter; + } else if (std::sscanf(cmd, "aim %f %f", &f[0], &f[1]) == 2) { + kind = kAim; + } else if (std::strncmp(cmd, "gaze", 4) == 0) { + kind = kGaze; + } else if (std::strncmp(cmd, "hide", 4) == 0) { + kind = kHide; + } else if (std::strncmp(cmd, "show", 4) == 0) { + kind = kShow; + } else if (std::sscanf(cmd, "role %31s", name) == 1) { + role = !std::strcmp(name, "left") ? TrackedControllerRole_LeftHand + : !std::strcmp(name, "right") ? TrackedControllerRole_RightHand + : !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus + : 0; + if (role) kind = kRole; + } else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) { + for (int i = 0; i < kButtons; ++i) + if (std::strcmp(name, kButtonNames[i]) == 0) button = i; + if (button >= 0) kind = kBtn; + } else if (std::sscanf(cmd, "distance %f", &f[0]) == 1) { + kind = kDistance; + } else if (std::sscanf(cmd, "scroll %f %f", &f[0], &f[1]) == 2) { + kind = kScroll; + } + if (kind == kNone) return; + + std::lock_guard guard(state_.lock); + switch (kind) { + case kPosq: state_.explicitPose = true; state_.hasQuat = true; state_.gaze = false; - state_.pos[0] = x; - state_.pos[1] = y; - state_.pos[2] = z; - state_.quat[0] = qw; - state_.quat[1] = qx; - state_.quat[2] = qy; - state_.quat[3] = qz; - } else if (std::sscanf(cmd, "pose %f %f %f %f %f", &x, &y, &z, &a, &b) == 5) { + std::memcpy(state_.pos, f, sizeof state_.pos); + std::memcpy(state_.quat, f + 3, sizeof state_.quat); + break; + case kPose: state_.explicitPose = true; state_.hasQuat = false; state_.gaze = false; - state_.pos[0] = x; - state_.pos[1] = y; - state_.pos[2] = z; - state_.yaw = a; - state_.pitch = b; - } else if (std::sscanf(cmd, "move %f %f", &a, &b) == 2) { + std::memcpy(state_.pos, f, sizeof state_.pos); + state_.yaw = f[3]; + state_.pitch = f[4]; + break; + case kMove: state_.explicitPose = false; if (state_.gaze) state_.recenter = true; // first move starts from the gaze - state_.yaw += a; // wrap by hand: libm remainder() is GLIBC_2.43 in the build container + state_.yaw += f[0]; // wrap by hand: libm remainder() is GLIBC_2.43 in the build container while (state_.yaw > 180.f) state_.yaw -= 360.f; while (state_.yaw < -180.f) state_.yaw += 360.f; - state_.pitch = std::fmax(-85.f, std::fmin(85.f, state_.pitch + b)); - } else if (std::strncmp(cmd, "recenter", 8) == 0) { + state_.pitch = std::fmax(-85.f, std::fmin(85.f, state_.pitch + f[1])); + break; + case kRecenter: state_.explicitPose = false; state_.recenter = true; - } else if (std::sscanf(cmd, "aim %f %f", &a, &b) == 2) { + break; + case kAim: state_.gaze = false; - state_.yaw = a; - state_.pitch = b; - } else if (std::strncmp(cmd, "gaze", 4) == 0) { + state_.yaw = f[0]; + state_.pitch = f[1]; + break; + case kGaze: state_.gaze = true; - } else if (std::strncmp(cmd, "hide", 4) == 0) { + break; + case kHide: state_.visible = false; - } else if (std::strncmp(cmd, "show", 4) == 0) { + break; + case kShow: state_.visible = true; - } else if (std::sscanf(cmd, "role %31s", name) == 1) { - state_.role = !std::strcmp(name, "left") ? TrackedControllerRole_LeftHand - : !std::strcmp(name, "right") ? TrackedControllerRole_RightHand - : !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus - : state_.role; - } else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) { - for (int i = 0; i < kButtons; ++i) - if (std::strcmp(name, kButtonNames[i]) == 0) state_.buttons[i] = v != 0; - } else if (std::sscanf(cmd, "distance %f", &a) == 1) { - state_.distance = std::fmax(0.3f, std::fmin(10.f, a)); - } else if (std::sscanf(cmd, "scroll %f %f", &a, &b) == 2) { - state_.scrollX = a; - state_.scrollY = b; + break; + case kRole: + state_.role = role; + break; + case kBtn: + state_.buttons[button] = v != 0; + break; + case kDistance: + state_.distance = std::fmax(0.3f, std::fmin(10.f, f[0])); + break; + case kScroll: + state_.scrollX = f[0]; + state_.scrollY = f[1]; + break; } } From 597551cf23edff05401a12e0baa4c59c8f718349 Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:21:09 -0600 Subject: [PATCH 6/8] Pause gesture: look the controllers up every 30 s, not every 3 s The gesture reader fetched vrserver's /input/getstate.json over HTTP every 3 seconds, the whole time the relay runs, to notice a controller's root path changing when the 3D mouse takes or gives back its hand role. It now looks them up when it connects, when a message comes from a device path it doesn't know (at most every 3 s; the device is read from the message with two string searches, not a JSON parse of all 160 a second), 1.5 s after the relay's 3D mouse connects or lets go (the relay tells it through GamePause.controllers_changed), and otherwise every 30 s. The keys test's pause stub gets the new method. Co-Authored-By: Claude Opus 5.5 --- docs/design.md | 2 +- input/game_pause.py | 36 ++++++++++++++++++++++++++++++++++-- input/input-relay.py | 9 ++++++++- input/test/keys-test.py | 2 +- 4 files changed, 44 insertions(+), 5 deletions(-) diff --git a/docs/design.md b/docs/design.md index 8ecb1c4..8da91b6 100644 --- a/docs/design.md +++ b/docs/design.md @@ -197,7 +197,7 @@ Hiding the screens during a game kept them out of view, but Frametop kept using - A hidden screen still cost as much as a visible one. ft-screens sent every committed screen its frame callback at 90 Hz whether its overlay showed or not, so KWin kept drawing, and its apps with it. Paused, ft-screens sends the callbacks once a second. A Wayland client draws again only after its last frame's callback, so KWin's output stalls, KWin's own clients stop getting theirs, and the whole desktop idles, without anything losing its connection. A second's pace, rather than none, keeps any client that waits on a callback from waiting forever. Stopping KWin or the apps with SIGSTOP would free the same, but a Wayland peer that stops reading overflows the other side's 4 KB socket buffer, which ends the connection: that's how the live desktop died once when its KWin stalled (`Data too big for buffer`). They also sit in different cgroups (KWin under steam.service when the VR launcher starts it, ft-screens in the dev container's), so no single freeze stops them together. - The relay does the pausing because it's the one part that always runs, and the pointer helper keeps running because stopping it leaves its virtual controller connected with its last pose (the driver has no staleness timeout), maybe holding a hand role, with the 3D mouse dead. Releasing it does the job. The helper already checks for a scene app twice a second, so it's what tells the relay a game started. -- The gesture has to work during a game, but SteamVR input reaches only the app with input focus, and an overlay with global input (`steamvr/globalActionSetPriority`) takes the buttons it binds from the game. vrserver's web socket on 127.0.0.1:27062, which its controller binding page uses for the live view, reports every controller component whatever has focus, and reading it takes nothing. The game sees the clicks too, so the default is a gesture games hardly use: both thumbsticks, together, twice. "Together" means within 0.3 seconds of each other, so a stick held down to sprint while the other clicks doesn't count. The stream is about 160 messages a second, nearly all capacitive sensing, so the reader parses only the few that mention a gesture's button. A controller's root path changes while the 3D mouse holds its hand role (`/devices/cv/` instead of `/user/hand/right`), so the reader looks the controllers up again every 3 seconds. +- The gesture has to work during a game, but SteamVR input reaches only the app with input focus, and an overlay with global input (`steamvr/globalActionSetPriority`) takes the buttons it binds from the game. vrserver's web socket on 127.0.0.1:27062, which its controller binding page uses for the live view, reports every controller component whatever has focus, and reading it takes nothing. The game sees the clicks too, so the default is a gesture games hardly use: both thumbsticks, together, twice. "Together" means within 0.3 seconds of each other, so a stick held down to sprint while the other clicks doesn't count. The stream is about 160 messages a second, nearly all capacitive sensing, so the reader parses only the few that mention a gesture's button. A controller's root path changes while the 3D mouse holds its hand role (`/devices/cv/` instead of `/user/hand/right`), so the reader looks the controllers up again (an HTTP request to vrserver): when the relay's 3D mouse connects or lets go, when a message comes from a device it doesn't know, and every 30 seconds. It used to be every 3 seconds. - Resuming starts remote desktop through `systemd-run --scope`: started straight from the relay, it would join the relay's cgroup and end with the next relay restart. ## SteamOS updates diff --git a/input/game_pause.py b/input/game_pause.py index a31ca8a..49f1427 100644 --- a/input/game_pause.py +++ b/input/game_pause.py @@ -65,6 +65,18 @@ RESUME_DELAY = 5.0 # after a VR game ends: loading the next one can end one sce GAME_STALE = 12.0 # the helper repeats "vrgame" every 5 s; silence this long means no game (or no SteamVR) CHORD_WINDOW = 0.3 # a two-button gesture's buttons go down within this of each other DOUBLE_WINDOW = 0.7 # a double press's second press comes within this of the first +LOOKUP_EVERY = 30.0 # the controllers are looked up again this often (and sooner when they may have changed) +LOOKUP_GAP = 3.0 # but not more often than this for a message from a device it doesn't know + + +def device_of(text): + """A web socket message's "device", without parsing all of it (there are about 160 a second).""" + at = text.find('"device"') + if at < 0: + return None + start = text.find('"', text.find(":", at) + 1) + end = text.find('"', start + 1) + return text[start + 1:end] if 0 <= start < end else None def gesture_of(rules, buttons): @@ -137,6 +149,12 @@ class ControllerWatch(threading.Thread): self.changed = threading.Event() self.connected = False self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) + self.lookup_at = None # look the controllers up again then (see recheck) + + def recheck(self, delay=1.5): + """The controllers' root paths may change soon: the 3D mouse's virtual controller is taking + or giving back its hand role. SteamVR takes a moment to hand the role over.""" + self.lookup_at = time.monotonic() + delay def set_gesture(self, spec): """(buttons, presses), or None for no gesture.""" @@ -176,15 +194,22 @@ class ControllerWatch(threading.Thread): ws.open(f"frametop_pause_{os.getpid()}") sides = {} # root path -> side next_poll = 0.0 + last_poll = 0.0 while True: g = self.current() if g is None: return now = time.monotonic() + lookup_at = self.lookup_at + if lookup_at is not None and now >= lookup_at: + self.lookup_at = None + next_poll = now if now >= next_poll: # A controller connects later, or changes its root path when the 3D mouse's - # virtual controller takes or gives back its hand role. - next_poll = now + 3.0 + # virtual controller takes or gives back its hand role. Each lookup is an HTTP + # request to vrserver, so it's at connect, every LOOKUP_EVERY, when a message + # comes from a device not on the list, and when the relay says (recheck). + next_poll, last_poll = now + LOOKUP_EVERY, now found = vrws.controllers() if found != sides: for path in sides.keys() - found.keys(): @@ -199,6 +224,9 @@ class ControllerWatch(threading.Thread): f"controller{'s' if len(sides) != 1 else ''} on SteamVR's web socket") wanted = {b: (b.split("/", 1)[0], f"/input/{b.split('/', 1)[1]}/click") for b in g.buttons} for text in ws.messages(timeout=1.0): + device = device_of(text) + if device and device not in sides: + next_poll = min(next_poll, last_poll + LOOKUP_GAP) # its root path changed, maybe # About 160 messages a second, nearly all capacitive sensing: parse only the # few that carry one of the gesture's buttons. if not any(c in text for _, c in wanted.values()): @@ -470,6 +498,10 @@ class GamePause: return max(0.05, self.resume_at - now) return 1.0 if self.game else 3600.0 + def controllers_changed(self): + """The 3D mouse took or gave back its hand role, which changes a controller's root path.""" + self.watch.recheck() + def helper_started(self): """The pointer helper (re)started, so SteamVR did too, maybe with the gaze service.""" if self.paused: diff --git a/input/input-relay.py b/input/input-relay.py index f2289cd..64c2e62 100755 --- a/input/input-relay.py +++ b/input/input-relay.py @@ -785,7 +785,8 @@ def main(): # pointer: the 3D mouse while it's in use, pointer_conf: the one the config asks for (they # differ while Frametop is paused). state = {"pointer": None, "pointer_conf": None, "rules": {}, "share_keys": False, - "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0, "vr_bind_retry": False} + "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0, "vr_bind_retry": False, + "pointer_holding": False} def pause_changed(paused): """Frametop paused or resumed (game_pause.py): the relay's own part.""" @@ -1292,6 +1293,12 @@ def main(): if added: apply_roles() + # The 3D mouse connecting or letting go moves a hand role, so the pause gesture's reader + # looks the controllers up again (game_pause.py). + holding = bool(state["pointer_conf"] and state["pointer_conf"].active) + if holding != state["pointer_holding"]: + state["pointer_holding"] = holding + pause.controllers_changed() ready, _, _ = select.select(list(nodes) + [control], [], [], min(volume.timeout(now, pointer.timeout() if pointer else 0.5), pause.timeout(now))) now = time.monotonic() diff --git a/input/test/keys-test.py b/input/test/keys-test.py index 005af84..9fac4d1 100755 --- a/input/test/keys-test.py +++ b/input/test/keys-test.py @@ -56,7 +56,7 @@ class StubPause: def configure(self, *args): pass - game_state = tick = helper_started = configure + game_state = tick = helper_started = controllers_changed = configure stub = {} From fcb465a45afe34527d62463f281d55abf0992b2f Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:23:32 -0600 Subject: [PATCH 7/8] Input relay: send mouse motion at most every 4 ms The relay sent the helper one "move" per SYN_REPORT, so a 1000 Hz mouse sent 1000 datagrams a second to a helper whose loop runs every 8 ms, and each one went through a dozen sscanf and strncmp tests in the helper before reaching the move handler. In a 200 ms test at 1000 Hz, 149 reports now make 45 moves with the same total. flush() on a report now sends only once 4 ms have passed since the last move; tick() sends the rest when due, and the select timeout shrinks to match. Buttons and the gaze keys still flush first, unconditionally, so a click lands where the pointer was. In the helper, "move" is now tested first in the command dispatch, and its handling is one lambda. Co-Authored-By: Claude Opus 5.5 --- docs/design.md | 2 +- input/input-relay.py | 25 +++++++++---- pointer/helper/ft-pointer.cpp | 69 ++++++++++++++++++++--------------- 3 files changed, 59 insertions(+), 37 deletions(-) diff --git a/docs/design.md b/docs/design.md index 8da91b6..27c77ec 100644 --- a/docs/design.md +++ b/docs/design.md @@ -153,7 +153,7 @@ SteamVR opens every input device only when it starts. When a Bluetooth mouse sle Keyboards aren't grabbed by default, because a grabbed keyboard's keys went into a virtual keyboard nothing typed from; the relay forwards them to ft-screens instead. -The relay never waits on the pointer helper. Its socket to the helper used to block, so when the helper stalled (a layout placement or `grabprobe` holds it for seconds, and the gaze service fills its socket 90 times a second meanwhile), the whole relay stopped with it: keyboards, the volume keys, and pausing. Now what the helper doesn't take waits in order and goes out on the next loops. Mouse moves add up into one while they wait, and a scroll notch is dropped, since scrolling seconds late is no use; presses and releases are kept, so no button stays down. +The relay never waits on the pointer helper. Its socket to the helper used to block, so when the helper stalled (a layout placement or `grabprobe` holds it for seconds, and the gaze service fills its socket 90 times a second meanwhile), the whole relay stopped with it: keyboards, the volume keys, and pausing. Now what the helper doesn't take waits in order and goes out on the next loops. Mouse moves add up into one while they wait, and a scroll notch is dropped, since scrolling seconds late is no use; presses and releases are kept, so no button stays down. Mouse motion goes to the helper at most every 4 ms, rather than once per report, which from a 1000 Hz mouse was 1000 datagrams a second to a helper that runs every 8 ms; a button sends the motion before it first, so the click lands where the pointer was. An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's `InputStealer`, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's `OverlayFocusChanged` (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. So with `SHARE_KEYS=1` the relay sends a grabbed keyboard's keys to `@frametop_keys` as datagrams (`key `). It's off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first gets every key typed into the desktop, passwords included. A listener should accept only its own user (`SO_PASSCRED`) and skip any keyboard of its own that the relay grabs too. diff --git a/input/input-relay.py b/input/input-relay.py index 64c2e62..4a63aab 100755 --- a/input/input-relay.py +++ b/input/input-relay.py @@ -461,6 +461,7 @@ class Pointer: RESUME_PAUSE = 1.5 # mouse idle this long, then moving again, re-claims the laser WAKE_WINDOW = 1.0 # seconds in which WAKE_COUNTS of motion must add up QUEUE_MAX = 512 # commands kept while the helper is behind (see send) + MOVE_EVERY = 0.004 # mouse motion goes to the helper at most this often (see flush) def __init__(self, sensitivity, idle, wake_counts=40): # Never blocks (see send): a stalled helper must not stall the keyboard, volume keys and pausing. @@ -474,6 +475,7 @@ class Pointer: self.active = False self.last_used = 0.0 self.dx = self.dy = 0 + self.move_at = 0.0 # motion last went to the helper then self.scroll_until = None self.claim_at = None # when to press the claim button self.claim_release = None @@ -637,11 +639,16 @@ class Pointer: self.sensitivity *= 1.25 if name == "sens_up" else 0.8 log(f"sensitivity {self.sensitivity:.4f} deg/count") - def flush(self): - if self.dx or self.dy: - # Mouse right turns the ray right (negative yaw); mouse down tilts it down. - self._move(-self.dx * self.sensitivity, -self.dy * self.sensitivity) - self.dx = self.dy = 0 + def flush(self, now=None): + """Send the motion so far. With now (a mouse's SYN_REPORT), only once MOVE_EVERY has passed + since the last: a 1000 Hz mouse sent the helper, which runs every 8 ms, 1000 datagrams a + second. tick() sends the rest when it's due; a button sends it first, so it lands there.""" + if not (self.dx or self.dy) or (now is not None and now - self.move_at < self.MOVE_EVERY): + return + self.move_at = time.monotonic() if now is None else now + # Mouse right turns the ray right (negative yaw); mouse down tilts it down. + self._move(-self.dx * self.sensitivity, -self.dy * self.sensitivity) + self.dx = self.dy = 0 def dashboard(self, now=None): """Toggle the SteamVR dashboard with the virtual controller's system button. @@ -657,6 +664,7 @@ class Pointer: def tick(self, now): self.drain() + self.flush(now) if self.system_at is not None and now >= self.system_at: self.send("btn system 1") self.system_at = None @@ -681,7 +689,10 @@ class Pointer: def timeout(self): pending = (self.scroll_until, self.claim_at, self.claim_release, self.system_at, self.system_release) - return 0.02 if self.queue or any(t is not None for t in pending) else 0.5 + wait = 0.02 if self.queue or any(t is not None for t in pending) else 0.5 + if self.dx or self.dy: # motion held back by flush + wait = max(0.0, min(wait, self.move_at + self.MOVE_EVERY - time.monotonic())) + return wait def stand_down(self): """Frametop is pausing: a pulse under way ends now, and the pointer lets go.""" @@ -1389,7 +1400,7 @@ def main(): mouse.emit(etype, code, value) elif etype == EV_SYN and code == SYN_REPORT: if pointer: - pointer.flush() + pointer.flush(now) mouse.sync() keyboard.sync() diff --git a/pointer/helper/ft-pointer.cpp b/pointer/helper/ft-pointer.cpp index 3acb9cb..521c702 100644 --- a/pointer/helper/ft-pointer.cpp +++ b/pointer/helper/ft-pointer.cpp @@ -1356,6 +1356,36 @@ int main() { return msg; }; + // "move " from the relay. + auto mouseMove = [&](double a, double b) { + // A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using + // the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back. + const bool heldBack = mouseMoveHeld(); + if (!heldBack) lastMouse = Clock::now(); + // Any mouse input wakes the pointer (after a controller took over, or a helper restart). + if (!active) wake(Clock::now()); + if (heldBack) return; + if (tilting) { + tiltYaw += a; + tiltPitch = std::clamp(tiltPitch + b, -80.0, 80.0); + return; + } + lastMove = Clock::now(); // the dot shows while the mouse moves it (gaze mode) + if (gazeOn && gazeOwns) { + // The mouse takes the pointer from the gaze, from where the gaze left it. + gazeOwns = false; + nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200); + nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead; + nudgeAt = Clock::now(), nudgeMoved = 0; + } + if (nudging || aimHeld) nudgeMoved += std::hypot(a, b); + if (!anchored) recenter = true; + yaw += a; + while (yaw > 180) yaw -= 360; + while (yaw < -180) yaw += 360; + pitch = std::clamp(pitch + b, -85.0, 85.0); + }; + std::printf("ft-pointer running: free distance %.2f m, dot %.2f deg\n", freeDistance, cursorDeg); std::fflush(stdout); @@ -1421,6 +1451,12 @@ int main() { if (senderLen > offsetof(sockaddr_un, sun_path)) sendto(out, msg.data(), msg.size(), 0, reinterpret_cast(&sender), senderLen); }; + // Mouse moves first, the most frequent command. + double mx, my; + if (std::strncmp(buf, "move ", 5) == 0 && std::sscanf(buf + 5, "%lf %lf", &mx, &my) == 2) { + mouseMove(mx, my); + continue; + } // The gaze, from ft-gazed: not mouse input, it never wakes the pointer. double g[4]; if (std::sscanf(buf, "gz %lf %lf %lf %lf", &g[0], &g[1], &g[2], &g[3]) == 4) { @@ -1509,16 +1545,11 @@ int main() { reply(gazeOn ? "ok on" : "ok off"); continue; } - const bool mouseInput = std::strncmp(buf, "move", 4) == 0 || std::strncmp(buf, "btn", 3) == 0 || - std::strncmp(buf, "scroll", 6) == 0; - // A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using - // the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back. - const bool moveHeldBack = std::strncmp(buf, "move", 4) == 0 && mouseMoveHeld(); - if (mouseInput && !moveHeldBack) lastMouse = Clock::now(); + // (Moves were taken first, above.) + const bool mouseInput = std::strncmp(buf, "btn", 3) == 0 || std::strncmp(buf, "scroll", 6) == 0; + if (mouseInput) lastMouse = Clock::now(); // Any mouse input wakes the pointer (after a controller took over, or a helper restart). if (!active && mouseInput) wake(Clock::now()); - if (moveHeldBack) continue; - double a, b; char key[128]; double px, py, pz, pyaw, ppitch, proll = 0, pgrab = -1; if (std::sscanf(buf, "grabprobe %127s", key) == 1) { @@ -1577,27 +1608,7 @@ int main() { tilting = swallowedRight = false; continue; } - if (tilting && std::sscanf(buf, "move %lf %lf", &a, &b) == 2) { - tiltYaw += a; - tiltPitch = std::clamp(tiltPitch + b, -80.0, 80.0); - continue; - } - if (std::sscanf(buf, "move %lf %lf", &a, &b) == 2) { - lastMove = Clock::now(); // the dot shows while the mouse moves it (gaze mode) - if (gazeOn && gazeOwns) { - // The mouse takes the pointer from the gaze, from where the gaze left it. - gazeOwns = false; - nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200); - nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead; - nudgeAt = Clock::now(), nudgeMoved = 0; - } - if (nudging || aimHeld) nudgeMoved += std::hypot(a, b); - if (!anchored) recenter = true; - yaw += a; - while (yaw > 180) yaw -= 360; - while (yaw < -180) yaw += 360; - pitch = std::clamp(pitch + b, -85.0, 85.0); - } else if (std::strncmp(buf, "recenter", 8) == 0) { + if (std::strncmp(buf, "recenter", 8) == 0) { recenter = true; } else if (std::strncmp(buf, "reload", 6) == 0) { loadConfig(); From 60667dba1faf78f535d54525940fe6766528176e Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Sat, 3 Oct 2026 09:25:16 -0600 Subject: [PATCH 8/8] Pointer: don't put a vanished panel back in the visibility map The panel-edge test read visible[edgeKey], and when the last panel the cursor touched was gone from the overlay list, that added it back as hidden. The map then had more entries than there are handles, which made the 50 ms visibility poll run every frame, and since the last commit it also counted as a visibility change each time, so unchanged frames were never reused. The edge test now looks the key up without adding it. Co-Authored-By: Claude Opus 5.5 --- pointer/helper/ft-pointer.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pointer/helper/ft-pointer.cpp b/pointer/helper/ft-pointer.cpp index 521c702..6ca76fe 100644 --- a/pointer/helper/ft-pointer.cpp +++ b/pointer/helper/ft-pointer.cpp @@ -2183,7 +2183,8 @@ int main() { onEdge = false; if (!dragging && best < 1e8 && !bestScene) { edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint; - } else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible[edgeKey]) { + } else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible.count(edgeKey) && + visible.at(edgeKey)) { // Just off a panel: stay on its plane (see "Panel edges" at the top). const double denom = Dot(dir, edgeNormal); if (std::fabs(denom) > 1e-4) {