Gaze panel: shared buffers, no flicker; calibration dots wait for a click

The panel drew each picture with SetOverlayRaw, a new texture upload every
time: the full calibration's 1024x768 picture flickered on every change, and
in a live test the headset kept showing the last calibration picture after the
panel had drawn the fit check (2026-10-01). Now, as screens/keyboard.cpp does,
it writes into three linear DMA-BUFs SteamVR imported once (the size of the
biggest panel; texture bounds show the part in use) and switches between them.
A show makes the overlay visible once its first picture is in.

The full calibration's and the five-dot check's dots wait for a left click or
Meta+J while you look at the dot, in place of capturing any steady gaze (which
can be a look somewhere else); no capture ring, no 8 s skip, and the check
closes after 2 minutes without a click. The quick check still captures itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gaze/README.md

@@ -31,7 +31,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
 - **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` lets the mouse take the pointer any time, as before.
 - **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, but up to 30 degrees whatever `POINTER_GAZE_NUDGE_MAX` says: a keyboard correction is always meant). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
 - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`, 55 degrees: half of the 109 the headset shows across) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the probe's one-dot check does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
-- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
+- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow. The full calibration (Calibrate on the Gaze page, or by itself when gaze mode comes on without one) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
 - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
 
 Lessons are logged to `pointer-lessons.jsonl`: the raw gaze, the true direction, the correction at the time, and how far off it was.

gaze/build.sh

@@ -14,6 +14,7 @@ g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -
   -o build/ft-gaze ft-gaze.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
 stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
 [ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
-g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include \
-  -o build/ft-gazepanel panel/ft-gazepanel.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 -lpthread
+g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
+  -o build/ft-gazepanel panel/ft-gazepanel.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64 \
+  $(pkg-config --libs gbm libdrm) -lpthread
 echo "built build/ft-gaze build/ft-gazepanel"'

gaze/gazecheck.py

@@ -27,11 +27,14 @@ directions: look at each one.
           adjust the headset. A left click or Meta+J runs its guided check (dots, then looks
           down, up, left and right); a right click or Meta+K closes it, as does FIT_TIMEOUT.
 
-A dot captures itself: from CHECK_SETTLE after it shows (the eyes getting there), once the gaze
-has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and capture time). It's
-the gaze holding still that counts, not where the tracker puts it, so it works however far off
-the tracker is. A left click or Meta+J (the pointer helper's "calaccept") takes it now; a right
-click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
+The quick check's dot captures itself: from CHECK_SETTLE after it shows (the eyes getting
+there), once the gaze has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and
+capture time). It's the gaze holding still that counts, not where the tracker puts it, so it
+works however far off the tracker is; a left click or Meta+J (the pointer helper's
+"calaccept") takes it now. The full calibration's and five's dots wait for that click: you
+click when you're looking at the dot (the user asked for that: a steady gaze isn't always on
+the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as
+it takes, up to CLICK_IDLE. A right click or Meta+K ("calquit") closes the panel. The pointer hides meanwhile ("calpanel 1",
 renewed every second; the helper shows it again by itself when that stops).
 
 What a capture teaches:
@@ -73,7 +76,7 @@ CHECK_SETTLE = 0.45
 CHECK_WINDOW = 0.6
 CHECK_SPREAD = 1.0     # degrees
 ACCEPT_SPREAD = 2.5    # degrees: a capture asked for (calaccept) takes this much
-DOT_TIMEOUT = 8.0      # seconds a dot of five or full waits; then it's skipped
+CLICK_IDLE = 120.0     # seconds a dot of five or full waits for its click; then the check closes
 QUICK_TIMEOUT = 6.0
 QUICK_COOLDOWN = 120.0
 DON_DELAY = 3.0        # seconds of eyes after AWAY_MIN without: the headset went on
@@ -391,9 +394,9 @@ class Checks:
         if c["kind"] == "full":
             self.to_panel(f"bg {ROUND_BG[rnd]}")
             self.to_panel(f"title Gaze calibration: {ROUND_NAMES[rnd]} round, {rnd + 1} of 3")
-            self.to_panel("text Look at the dot. Left click or Meta+J: now. Right click or Meta+K: stop")
+            self.to_panel("text Look at the dot and click (left click or Meta+J). Right click or Meta+K: stop")
         elif c["kind"] == "five":
-            self.to_panel(f"text Look at the dot ({c['i'] + 1} of {len(c['dots'])})")
+            self.to_panel(f"text Look at the dot and click ({c['i'] + 1} of {len(c['dots'])})")
         else:
             self.to_panel("text Look at the dot")
         self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look")
@@ -437,10 +440,10 @@ class Checks:
         run.append((now, s, g[0], g[1]))
         window = [p for p in run if p[0] >= now - CHECK_WINDOW]
         held = now - run[0][0]
-        # The ring fills in quarters: each step is a new picture for the panel to upload, so
-        # few of them keep it solid.
+        # The quick check's ring fills in quarters: each step is a new picture for the panel, so
+        # few of them keep it solid. The others wait for the click (see the top): no ring.
         progress = math.floor(min(1.0, held / CHECK_WINDOW) * 4) / 4
-        if progress != self.last_progress:
+        if c["kind"] == "quick" and progress != self.last_progress:
             yaw, pitch, _ = c["dots"][c["i"]]
             self.to_panel(f"dot {yaw:.3f} {pitch:.3f} capture {progress:.2f}")
             self.last_progress = progress
@@ -449,7 +452,7 @@ class Checks:
             if sd <= ACCEPT_SPREAD:
                 self.capture(window)
             return
-        if held >= CHECK_WINDOW and len(window) >= 20:
+        if c["kind"] == "quick" and held >= CHECK_WINDOW and len(window) >= 20:
             sd, _ = spread([(p[2], p[3]) for p in window])
             if sd <= CHECK_SPREAD:
                 self.capture(window)
@@ -687,9 +690,8 @@ class Checks:
             return
         if c["kind"] == "quick" and now - c["started"] > QUICK_TIMEOUT:
             self.close("ignored")
-        elif c["kind"] != "quick" and now - c["shown"] > DOT_TIMEOUT:
-            log(f"{c['kind']} check dot {c['i'] + 1}: skipped (the gaze didn't hold still)")
-            self.skip()
+        elif c["kind"] != "quick" and now - c["shown"] > CLICK_IDLE:
+            self.close(f"no click on dot {c['i'] + 1} for {CLICK_IDLE:.0f} s")
 
     def periodic(self):
         svc = self.svc

gaze/panel/ft-gazepanel.cpp

@@ -28,6 +28,13 @@
 //   hints <line>|<line>|...      fit: lines under the cards (empty to clear)
 //   ping
 //
+// Each picture goes into the next of three shared buffers (linear DMA-BUFs SteamVR imported
+// once, the size of the biggest panel; the texture bounds show the part in use), and the panel
+// switches to it, as screens/keyboard.cpp does. SetOverlayRaw, which uploads a new texture each
+// time, flickered on every change of the full calibration's 1024x768 picture, and in a live
+// test the headset kept showing an old picture after the panel had drawn new ones (2026-10-01).
+// It's only the fallback. A "show" makes the panel visible once its first picture is in.
+//
 // Options: --watch-stdin (quit when stdin closes: the service runs it), --socket NAME,
 // --distance METRES. Runs in the dev container (gaze/build.sh builds it into gaze/build).
 #include <openvr.h>
@@ -35,6 +42,9 @@
 #define STB_TRUETYPE_IMPLEMENTATION
 #include "stb_truetype.h"
 
+#include <drm_fourcc.h>
+#include <fcntl.h>
+#include <gbm.h>
 #include <sys/socket.h>
 #include <sys/un.h>
 #include <unistd.h>
@@ -66,6 +76,7 @@ std::atomic<bool> g_stop{false};
 stbtt_fontinfo g_font;
 std::vector<unsigned char> g_fontData;
 bool g_fontOk = false;
+constexpr int kMaxW = kFullW, kMaxH = kFullH;  // the buffers' size: the biggest panel
 struct Glyph {
     std::vector<unsigned char> bitmap;
     int w = 0, h = 0, xoff = 0, yoff = 0, advance = 0;
@@ -282,6 +293,96 @@ void Draw(Panel &p) {
 }
 
 
+// ---------------------------------------------------------------- the buffers (see the top)
+
+struct Buffer {
+    gbm_bo *bo = nullptr;
+    int fd = -1;
+    vr::SharedTextureHandle_t handle = 0;
+};
+
+struct Buffers {
+    int drm = -1;
+    gbm_device *gbm = nullptr;
+    Buffer b[3];
+    int next = 0;
+    bool ok = false;
+
+    bool Make() {
+        drm = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC);
+        if (drm >= 0) gbm = gbm_create_device(drm);
+        for (Buffer &x : b) {
+            // ABGR8888 is R, G, B, A in memory, like Panel::px.
+            if (gbm) x.bo = gbm_bo_create(gbm, kMaxW, kMaxH, GBM_FORMAT_ABGR8888, GBM_BO_USE_RENDERING | GBM_BO_USE_LINEAR);
+            if (!x.bo || (x.fd = gbm_bo_get_fd(x.bo)) < 0) break;
+            vr::DmabufAttributes_t a{};
+            a.unWidth = kMaxW, a.unHeight = kMaxH;
+            a.unDepth = a.unMipLevels = a.unArrayLayers = a.unSampleCount = 1;
+            a.unFormat = DRM_FORMAT_ABGR8888;
+            a.ulModifier = DRM_FORMAT_MOD_LINEAR;
+            a.unPlaneCount = 1;
+            a.plane[0].unOffset = gbm_bo_get_offset(x.bo, 0);
+            a.plane[0].unStride = gbm_bo_get_stride(x.bo);
+            a.plane[0].nFd = x.fd;
+            if (!vr::VRIPCResourceManager()->ImportDmabuf(vr::VRApplication_Overlay, &a, &x.handle)) x.handle = 0;
+            if (!x.handle) break;
+        }
+        ok = b[2].handle != 0;
+        if (!ok) {
+            std::fprintf(stderr, "ft-gazepanel: no shared buffers; falling back to SetOverlayRaw (it flickers)\n");
+            Drop();
+        }
+        return ok;
+    }
+
+    void Drop() {
+        for (Buffer &x : b) {
+            if (x.handle) vr::VRIPCResourceManager()->UnrefResource(x.handle);
+            if (x.fd >= 0) close(x.fd);
+            if (x.bo) gbm_bo_destroy(x.bo);
+            x = Buffer{};
+        }
+        if (gbm) gbm_device_destroy(gbm);
+        if (drm >= 0) close(drm);
+        gbm = nullptr, drm = -1, ok = false;
+    }
+
+    // p's picture to the overlay: into the next buffer, premultiplied (the overlay's flag says
+    // so), then the overlay switches to it.
+    void Present(vr::IVROverlay *ov, vr::VROverlayHandle_t h, const Panel &p) {
+        vr::EVROverlayError e;
+        if (!ok) {
+            e = ov->SetOverlayRaw(h, const_cast<uint8_t *>(p.px.data()), uint32_t(p.w), uint32_t(p.h), 4);
+        } else {
+            Buffer &x = b[next];
+            next = (next + 1) % 3;
+            uint32_t stride = 0;
+            void *mapping = nullptr;
+            auto *dst = static_cast<uint8_t *>(gbm_bo_map(x.bo, 0, 0, p.w, p.h, GBM_BO_TRANSFER_WRITE, &stride, &mapping));
+            if (!dst) {
+                std::fprintf(stderr, "ft-gazepanel: can't map a buffer\n");
+                return;
+            }
+            for (int y = 0; y < p.h; ++y) {
+                const uint8_t *src = &p.px[size_t(y) * p.w * 4];
+                uint8_t *row = dst + size_t(y) * stride;
+                for (int i = 0; i < p.w * 4; i += 4) {
+                    const unsigned a = src[i + 3];
+                    row[i] = uint8_t(src[i] * a / 255), row[i + 1] = uint8_t(src[i + 1] * a / 255);
+                    row[i + 2] = uint8_t(src[i + 2] * a / 255), row[i + 3] = uint8_t(a);
+                }
+            }
+            gbm_bo_unmap(x.bo, mapping);
+            const vr::VRTextureBounds_t bounds{0, 0, float(p.w) / kMaxW, float(p.h) / kMaxH};
+            ov->SetOverlayTextureBounds(h, &bounds);
+            vr::Texture_t tex = {&x.handle, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
+            e = ov->SetOverlayTexture(h, &tex);
+        }
+        if (e != vr::VROverlayError_None)
+            std::fprintf(stderr, "ft-gazepanel: the picture didn't go to SteamVR: %s\n", ov->GetOverlayErrorNameFromEnum(e));
+    }
+};
+
 }  // namespace
 
 int main(int argc, char **argv) {
@@ -335,9 +436,11 @@ int main(int argc, char **argv) {
     }
     ov->SetOverlaySortOrder(h, 250);  // in front of Frametop's screens and the pointer's dot
     LoadFont();
+    Buffers buffers;
+    if (buffers.Make()) ov->SetOverlayFlag(h, vr::VROverlayFlags_IsPremultiplied, true);
 
     Panel p;
-    bool visible = false, dirty = false;
+    bool visible = false, dirty = false, shown = false;  // shown: SteamVR shows it (after its first picture)
     auto place = [&] {
         const double half = std::tan(p.wDeg * M_PI / 360);
         vr::HmdMatrix34_t m{};
@@ -368,11 +471,10 @@ int main(int argc, char **argv) {
                 p.title.clear(), p.text.clear(), p.dotOn = false, p.state = "off";
                 p.eyes[0] = p.eyes[1] = EyeCard{}, p.hints.clear();
                 place();
-                ov->ShowOverlay(h);
-                visible = dirty = true;
+                visible = dirty = true;  // shown with its first picture
             } else if (!std::strcmp(buf, "hide")) {
                 ov->HideOverlay(h);
-                visible = false;
+                visible = shown = false;
             } else if (std::sscanf(buf, "bg %lf", &a) == 1) {
                 p.bg = std::clamp(a, 0.0, 1.0), dirty = true;
             } else if (std::sscanf(buf, "dot %lf %lf %15s %lf", &a, &b, state, &c) >= 3) {
@@ -411,12 +513,14 @@ int main(int argc, char **argv) {
             }
         if (visible && dirty) {
             Draw(p);
-            ov->SetOverlayRaw(h, p.px.data(), uint32_t(p.w), uint32_t(p.h), 4);
+            buffers.Present(ov, h, p);
+            if (!shown) ov->ShowOverlay(h), shown = true;
             dirty = false;
         }
         std::this_thread::sleep_for(std::chrono::milliseconds(visible ? 10 : 50));
     }
     ov->DestroyOverlay(h);
+    buffers.Drop();
     vr::VR_Shutdown();
     return 0;
 }