Map the Frame controllers' buttons; make gaze clicks correctable

The Frame controllers aren't input devices on the host, so the pointer
helper reads them with SteamVR input (vrbuttons.h, actions/), one action
set per button, and sends presses to the relay, which does the mapped
action like for a mouse button. Only mapped buttons are taken, at an
overlay-global priority (SteamVR's experimental "Enable global input from
overlays"), and only outside games unless In games is on. Frametop Input
Settings gets a Controllers page to map them.

Gaze mode: a left press while the gaze has the pointer isn't sent at
once. The pointer stops, you drag it onto what you meant with the button
held, and the release clicks there; a press held still for
POINTER_GAZE_HOLD (0.5 s) becomes a real press, for drags. The dot shows
only while the mouse moves it (POINTER_GAZE_SHOW), while a press is held,
and as a pulse per click. Outside games the pointer stays on while gaze
mode is on. ft-gazed takes a third of each lesson's offset instead of all
of it: in the first live test one 6 degree lesson moved everything and put
the next target 7 degrees off. Frametop Input Settings gets a Gaze page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

pointer/helper/ft-pointer.cpp

@@ -114,14 +114,28 @@
 // that gaze from the eye: nothing is steered, so nothing can pile up. Moving the mouse takes
 // the pointer from the gaze, and it moves from where the gaze left it, as usual. Looking
 // well away from it (more than POINTER_GAZE_RETAKE, 5 deg, for 120 ms, with the mouse still
-// for 300 ms) gives it back to the gaze; small eye movements around the pointer don't. A
-// click or a drag holds the pointer where it is, like any other press.
+// for 300 ms) gives it back to the gaze; small eye movements around the pointer don't.
+//   A left press while the gaze has the pointer isn't sent yet: the pointer stops where the
+// gaze put it, and if the gaze is off, you drag it onto what you meant with the mouse
+// (still holding the button; panels only see it hover). The release clicks there, a press
+// and a release 40 ms apart. Held still for POINTER_GAZE_HOLD (0.5 s) instead, it becomes
+// a real press where the pointer is, so drags work: hold, then move. After a click that
+// didn't need correcting, and after a drag, the gaze has the pointer again.
+//   Outside games (no scene application), gaze mode keeps the pointer: the relay doesn't
+// release it when the mouse is idle ("gazeawake 1|0" tells it). A controller that moves
+// still releases it (the mouse is gaze mode's only pointer device), and in games the mouse
+// wakes it and idling releases it, as without gaze.
+//   The dot only shows while the mouse moves it (within POINTER_GAZE_SHOW, 1 s), while a
+// press is held, and briefly for each click (a pulse);
+// otherwise it's transparent (still there for the laser to land on). The gaze moving it
+// doesn't show it: you know where you're looking.
 //   When the mouse took the pointer and you then click, the nudge was probably onto what
 // you were looking at: from the raw gaze when the mouse took over to where you clicked is
 // the tracker's error there. The helper sends it to ft-gazed as a lesson ("lesson <raw yaw>
 // <raw pitch> <true yaw> <true pitch>", the true direction relative to the head as it was
 // when the mouse took over) if the mouse moved between 0.2 deg and POINTER_GAZE_NUDGE_MAX
-// (8 deg) and the click came within 10 s; more is using the mouse, not a nudge. With no
+// (8 deg) and the click came within 10 s; more is using the mouse, not a nudge. A held
+// press dragged onto the target is the same: from the raw gaze at the press to the release. With no
 // fresh gaze (a blink, the service stopped, the headset off), the pointer stays put.
 //
 // Placement (for layout): SteamVR keeps a floating panel's position inside the
@@ -139,8 +153,9 @@
 //
 // Commands (datagrams on @ft_pointer_helper): show, hide, recenter, move <dyaw> <dpitch>,
 // follow on|off|toggle (head follow, until the next restart or a change to POINTER_FOLLOW),
-// gaze on|off|toggle (gaze mode, likewise with POINTER_GAZE), gz ... (the gaze, from ft-gazed),
+// gaze on|off|toggle|? (gaze mode, likewise with POINTER_GAZE; ? only asks), gz ... (the gaze, from ft-gazed),
 // reload (re-read the settings below), debug (toggle a twice-a-second state log),
+// vrbind/vrglobal/vrstatus (Frame controller buttons, see vrbuttons.h),
 // and btn/scroll lines, which are forwarded to the driver unchanged. For layouts, with a
 // reply datagram to the sender's (abstract) address:
 //   place <overlay> <x> <y> <z> <yaw> <pitch> [roll [grab]]: centre in the standing
@@ -160,9 +175,11 @@
 // panel, and a laser that starts behind them can't hit them. POINTER_FOLLOW (0) and
 // POINTER_LEASH_DEG (10), POINTER_LEASH_DELAY (0.2 s), POINTER_LEASH_RETURN (0.2 s),
 // POINTER_FOLLOW_REACH (70 deg): head follow, above. POINTER_GAZE (0), POINTER_GAZE_RETAKE
-// (5 deg), POINTER_GAZE_NUDGE_MAX (8 deg): gaze mode, above.
+// (5 deg), POINTER_GAZE_NUDGE_MAX (8 deg), POINTER_GAZE_HOLD (0.5 s), POINTER_GAZE_SHOW (1 s):
+// gaze mode, above.
 #include <openvr.h>
 
+#include "vrbuttons.h"
 #include "vrmath.h"
 
 #include <algorithm>
@@ -180,6 +197,8 @@
 #include <tuple>
 #include <vector>
 
+#include <climits>
+
 #include <sys/socket.h>
 #include <sys/un.h>
 #include <unistd.h>
@@ -227,6 +246,15 @@ int AbstractSocket(const char *name, bool bindIt) {
     return fd;
 }
 
+std::string ExeDir() {
+    char buf[PATH_MAX];
+    const ssize_t n = readlink("/proc/self/exe", buf, sizeof buf - 1);
+    if (n <= 0) return ".";
+    buf[n] = 0;
+    std::string p(buf);
+    return p.substr(0, p.rfind('/'));
+}
+
 void SendTo(int fd, const char *name, const std::string &msg) {
     sockaddr_un addr{};
     addr.sun_family = AF_UNIX;
@@ -361,7 +389,7 @@ int main() {
     bool follow = false, followConf = false, followReset = true;
     // Gaze mode (see the top); gazeConf is POINTER_GAZE as last read, like followConf.
     bool gazeOn = false, gazeConf = false;
-    double gazeRetake = 5, gazeNudgeMax = 8;
+    double gazeRetake = 5, gazeNudgeMax = 8, gazeHold = 0.5, gazeShow = 1;
     auto loadConfig = [&] {
         const auto conf = ReadConfig();
         freeDistance = std::clamp(ConfDouble(conf, "POINTER_DISTANCE", 1.5), 0.3, 10.0);
@@ -380,6 +408,8 @@ int main() {
         if (wantFollow != followConf) follow = followConf = wantFollow, followReset = true;
         gazeRetake = std::clamp(ConfDouble(conf, "POINTER_GAZE_RETAKE", 5), 1.0, 45.0);
         gazeNudgeMax = std::clamp(ConfDouble(conf, "POINTER_GAZE_NUDGE_MAX", 8), 1.0, 30.0);
+        gazeHold = std::clamp(ConfDouble(conf, "POINTER_GAZE_HOLD", 0.5), 0.1, 5.0);
+        gazeShow = std::clamp(ConfDouble(conf, "POINTER_GAZE_SHOW", 1), 0.0, 30.0);
         const bool wantGaze = ConfDouble(conf, "POINTER_GAZE", 0) != 0;
         if (wantGaze != gazeConf) gazeOn = gazeConf = wantGaze;
     };
@@ -427,6 +457,14 @@ int main() {
 
     const int in = AbstractSocket("ft_pointer_helper", true);
     const int out = AbstractSocket(nullptr, false);
+    // Frame controller buttons (vrbuttons.h). The build puts the binary in pointer/helper/build.
+    ControllerButtons controllerButtons;
+    {
+        const std::string manifest = ExeDir() + "/../actions/ft_pointer_actions.json";
+        char real[PATH_MAX];
+        controllerButtons.Init(realpath(manifest.c_str(), real) ? real : manifest);
+    }
+    SendTo(out, "frametop_relay", "vrhello");  // the relay answers with the mapped buttons
     OverlayList overlays;
     overlays.Start();
     std::map<std::string, vr::VROverlayHandle_t> handles;
@@ -481,7 +519,72 @@ int main() {
     vr::HmdMatrix34_t nudgeHead{}, lastHead{};
     bool haveHead = false, havePoint = false;
     Clock::time_point nudgeAt{}, retakeSince{};
+    // A held-back press (see the top): aimHeld while the button is down; then the click
+    // (clickPress at the end of the next frame, clickRelease 40 ms later). gazeBack: give
+    // the gaze the pointer again when the press or click is over.
     vr::TrackedDeviceIndex_t ours = vr::k_unTrackedDeviceIndexInvalid;
+    bool aimHeld = false, clickPress = false, clickRelease = false, gazeBack = false;
+    Clock::time_point aimSince{}, clickReleaseAt{};
+    // Gaze mode outside games, and its dot (see the top): lastMove/lastHeld/pulseAt.
+    bool inGame = false, gazeAwake = false;
+    Clock::time_point inGameAt{}, gazeAwakeAt{};
+    Clock::time_point lastMove{}, lastHeld{}, pulseAt{};
+    // The left button, as sent to the driver.
+    auto pressLeft = [&] {
+        // ft-screens sends the keyboard to the panel clicked last; it sees clicks on
+        // its own screens, but only we know when one lands on another panel.
+        SendTo(out, "ft_screens", "click " + (lastHit.empty() ? std::string("-") : lastHit));
+        // A click after nudging the gaze-placed pointer: the nudge is a lesson (see the top).
+        if (gazeOn && nudging && !gazeOwns && havePoint && Clock::now() - nudgeAt < std::chrono::seconds(10) &&
+            nudgeMoved >= 0.2 && nudgeMoved <= gazeNudgeMax) {
+            const Vec3 d = RotateInverse(nudgeHead, Normalize(lastPoint - Position(nudgeHead)));
+            char msg[160];
+            std::snprintf(msg, sizeof msg, "lesson %.3f %.3f %.3f %.3f", nudgeRawHy, nudgeRawHp,
+                          std::atan2(-d.x, -d.z) * 180 / M_PI, std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI);
+            SendTo(out, "ft_gazed", msg);
+            if (debug) std::printf("gaze %s (nudged %.2f deg)\n", msg, nudgeMoved);
+            if (debug) std::fflush(stdout);
+        }
+        nudging = false;
+        leftHeld = true;
+        dragDistance = lastDistance;
+        tiltYaw = tiltPitch = 0;  // a new drag starts untilted
+        dropHoldUntil = {};
+        pulseAt = Clock::now();
+        SendTo(out, "ft_pointer", "btn trigger 1");
+    };
+    auto releaseLeft = [&] {
+        leftHeld = false;
+        tilting = false;
+        // Hold the drag pose (tilt, frozen distance) while SteamVR finishes the drop.
+        dropHoldUntil = Clock::now() + std::chrono::milliseconds(500);
+        SendTo(out, "ft_pointer", "btn trigger 0");
+        if (gazeBack) gazeOwns = true, gazeBack = false;
+    };
+    // The left button, from the relay's "btn trigger", with gaze mode's held-back press (see
+    // the top).
+    auto leftButton = [&](bool down) {
+        if (down) {
+            if (gazeOn && gazeOwns && !aimHeld && !clickPress && !clickRelease) {
+                // Hold the press back: the pointer stops where the gaze put it.
+                gazeOwns = false;
+                nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
+                nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
+                nudgeAt = aimSince = Clock::now(), nudgeMoved = 0;
+                aimHeld = true;
+                return;
+            }
+            pressLeft();
+            return;
+        }
+        if (aimHeld) {
+            aimHeld = false;
+            clickPress = true;  // after this frame's pose, so it lands where the pointer was moved to
+            gazeBack = nudgeMoved < 0.2;
+            return;
+        }
+        if (leftHeld) releaseLeft();
+    };
     auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10);
 
     // --- Panel placement (see "Placement" at the top of the file) ---
@@ -659,6 +762,23 @@ int main() {
             std::fflush(stdout);
         }
 
+        // Outside games, gaze mode keeps the pointer (see the top); the relay needs to know.
+        {
+            const auto t = Clock::now();
+            if (t - inGameAt > std::chrono::milliseconds(500)) {
+                inGameAt = t;
+                inGame = vr::VRApplications()->GetCurrentSceneProcessId() != 0;
+            }
+            const bool awake = gazeOn && !inGame && !headsetOff;
+            if (awake != gazeAwake || t - gazeAwakeAt > std::chrono::seconds(5)) {
+                if (awake != gazeAwake) std::printf("gaze keeps the pointer: %s\n", awake ? "yes" : "no (off, in a game, or headset off)");
+                if (awake != gazeAwake) std::fflush(stdout);
+                gazeAwake = awake;
+                gazeAwakeAt = t;
+                SendTo(out, "frametop_relay", awake ? "gazeawake 1" : "gazeawake 0");
+            }
+        }
+
         // Commands from the relay.
         char buf[256];
         ssize_t n;
@@ -680,13 +800,33 @@ int main() {
                 gz = {g[0], g[1], g[2], g[3], Clock::now()};
                 continue;
             }
+            if (std::strncmp(buf, "vrbind", 6) == 0) {
+                std::printf("controller buttons: %s\n", controllerButtons.Bind(buf + 6).c_str());
+                std::fflush(stdout);
+                continue;
+            }
+            if (std::strncmp(buf, "vrglobal", 8) == 0) {
+                const char *arg = buf + 8;
+                while (*arg == ' ') ++arg;
+                ControllerButtons::SetGlobal(std::strncmp(arg, "off", 3) != 0);
+                reply(controllerButtons.Status());
+                continue;
+            }
+            if (std::strncmp(buf, "vrstatus", 8) == 0) {
+                reply(controllerButtons.Status());
+                continue;
+            }
             if (std::strncmp(buf, "gaze", 4) == 0) {
                 const char *arg = buf + 4;
                 while (*arg == ' ') ++arg;
-                gazeOn = std::strncmp(arg, "on", 2) == 0 ? true : std::strncmp(arg, "off", 3) == 0 ? false : !gazeOn;
-                gazeOwns = true, nudging = false;
-                std::printf("gaze mode %s\n", gazeOn ? "on" : "off");
-                std::fflush(stdout);
+                if (*arg != '?') {  // "gaze ?" only asks
+                    gazeOn = std::strncmp(arg, "on", 2) == 0    ? true
+                             : std::strncmp(arg, "off", 3) == 0 ? false
+                                                                : !gazeOn;
+                    gazeOwns = true, nudging = false;
+                    std::printf("gaze mode %s\n", gazeOn ? "on" : "off");
+                    std::fflush(stdout);
+                }
                 reply(gazeOn ? "ok on" : "ok off");
                 continue;
             }
@@ -738,30 +878,11 @@ int main() {
                 continue;
             }
             if (std::strncmp(buf, "btn trigger 1", 13) == 0) {
-                // ft-screens sends the keyboard to the panel clicked last; it sees clicks on
-                // its own screens, but only we know when one lands on another panel.
-                SendTo(out, "ft_screens", "click " + (lastHit.empty() ? std::string("-") : lastHit));
-                // A click after nudging the gaze-placed pointer: the nudge is a lesson (see the top).
-                if (gazeOn && nudging && !gazeOwns && havePoint && Clock::now() - nudgeAt < std::chrono::seconds(10) &&
-                    nudgeMoved >= 0.2 && nudgeMoved <= gazeNudgeMax) {
-                    const Vec3 d = RotateInverse(nudgeHead, Normalize(lastPoint - Position(nudgeHead)));
-                    char msg[160];
-                    std::snprintf(msg, sizeof msg, "lesson %.3f %.3f %.3f %.3f", nudgeRawHy, nudgeRawHp,
-                                  std::atan2(-d.x, -d.z) * 180 / M_PI, std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI);
-                    SendTo(out, "ft_gazed", msg);
-                    if (debug) std::printf("gaze %s (nudged %.2f deg)\n", msg, nudgeMoved);
-                    if (debug) std::fflush(stdout);
-                }
-                nudging = false;
-                leftHeld = true;
-                dragDistance = lastDistance;
-                tiltYaw = tiltPitch = 0;  // a new drag starts untilted
-                dropHoldUntil = {};
+                leftButton(true);
+                continue;
             } else if (std::strncmp(buf, "btn trigger 0", 13) == 0) {
-                leftHeld = false;
-                tilting = false;
-                // Hold the drag pose (tilt, frozen distance) while SteamVR finishes the drop.
-                dropHoldUntil = Clock::now() + std::chrono::milliseconds(500);
+                leftButton(false);
+                continue;
             } else if (std::strncmp(buf, "btn b 1", 7) == 0 && leftHeld) {
                 tilting = tiltStart = swallowedRight = true;  // right press while dragging: tilt, no right-click
                 continue;
@@ -775,6 +896,7 @@ int main() {
                 continue;
             }
             if (std::sscanf(buf, "move %lf %lf", &a, &b) == 2) {
+                lastMove = Clock::now();  // the dot shows while the mouse moves it (gaze mode)
                 if (gazeOn && gazeOwns) {
                     // The mouse takes the pointer from the gaze, from where the gaze left it.
                     gazeOwns = false;
@@ -782,7 +904,7 @@ int main() {
                     nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
                     nudgeAt = Clock::now(), nudgeMoved = 0;
                 }
-                if (nudging) nudgeMoved += std::hypot(a, b);
+                if (nudging || aimHeld) nudgeMoved += std::hypot(a, b);
                 if (!anchored) recenter = true;
                 yaw += a;
                 while (yaw > 180) yaw -= 360;
@@ -896,7 +1018,8 @@ int main() {
         // Gaze mode (see the top): the gaze has the pointer, or takes it back when you look
         // well away from it. Not while a press holds the pointer, and only on fresh gaze.
         if (hmd.bPoseIsValid) lastHead = hmd.mDeviceToAbsoluteTracking, haveHead = true;
-        if (gazeOn && active && hmd.bPoseIsValid && !tilting && !leftHeld && tnow >= dropHoldUntil &&
+        if (gazeOn && active && hmd.bPoseIsValid && !tilting && !leftHeld && !aimHeld && !clickPress && !clickRelease &&
+            tnow >= dropHoldUntil &&
             tnow - gz.at < std::chrono::milliseconds(150)) {
             const Vec3 g = Rotate(hmd.mDeviceToAbsoluteTracking, Direction(gz.hy, gz.hp));
             if (!gazeOwns && havePoint) {
@@ -917,6 +1040,8 @@ int main() {
             }
         }
 
+        if (aimHeld || leftHeld || clickPress || clickRelease) lastHeld = tnow;
+
         // Head follow (see the top): past the leash (for the delay), ease the reference to the
         // head's facing, and turn the cursor with it. Not in gaze mode: the gaze places it.
         if (follow && !gazeOn && active && anchored && hmd.bPoseIsValid) {
@@ -1117,7 +1242,20 @@ int main() {
                 const vr::VROverlayHandle_t show = onPanel ? marker : cursor, hide = onPanel ? cursor : marker;
                 const Vec3 at = onPanel ? point + Normalize(eye - point) * 0.005 : onScene ? point + dir * 0.05 : point;
                 const double dist = std::sqrt(Dot(at - eye, at - eye));
-                overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(cursorDeg * M_PI / 360)));
+                // Gaze mode: shown only while something moves it or a press holds it, and a pulse
+                // for each click (see the top); transparent otherwise, the laser still lands on it.
+                double scale = 1, alpha = 1;
+                if (gazeOn) {
+                    auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
+                    alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
+                    const double pulse = secs(pulseAt);
+                    if (pulse < 0.6) {
+                        scale = 1 + 1.5 * std::max(0.0, 1 - pulse / 0.3);
+                        alpha = std::max(alpha, std::clamp((0.6 - pulse) / 0.3, 0.0, 1.0));
+                    }
+                }
+                overlay->SetOverlayAlpha(show, float(alpha));
+                overlay->SetOverlayWidthInMeters(show, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360)));
                 auto m = Billboard(at, eye);
                 overlay->SetOverlayTransformAbsolute(show, vr::TrackingUniverseStanding, &m);
                 overlay->ShowOverlay(show);
@@ -1172,6 +1310,30 @@ int main() {
             }
         }
 
+        // A held-back press (see the top): held still long enough, it's a real press (a drag);
+        // released, it's a click where the pointer is now (this frame's pose has gone out).
+        if (!active) aimHeld = clickPress = false;  // released meanwhile: nothing to click
+        if (aimHeld && nudgeMoved < 0.2 && tnow - aimSince >= std::chrono::duration<double>(gazeHold)) {
+            aimHeld = false;
+            gazeBack = true;
+            pressLeft();
+        }
+        if (clickPress) {
+            clickPress = false;
+            pressLeft();
+            clickRelease = true;
+            clickReleaseAt = tnow + std::chrono::milliseconds(40);
+        } else if (clickRelease && tnow >= clickReleaseAt) {
+            clickRelease = false;
+            releaseLeft();
+        }
+
+        controllerButtons.Poll(
+            [&](const char *button, bool down) {
+                SendTo(out, "frametop_relay", std::string("vrbtn ") + button + (down ? " 1" : " 0"));
+            },
+            inGame);
+
         vr::VREvent_t ev;
         while (sys->PollNextEvent(&ev, sizeof ev)) {
             if (ev.eventType == vr::VREvent_Quit) {