Hands: pick the cameras by the light, and detect grips

ft-hands tracks with the mono IR cameras in dim light and with every
camera (or the colour pair, HANDS_BRIGHT) in bright light, going by the
colour frames' mean brightness with hysteresis and a 2 s hold
(HANDS_CAMERAS=auto, the default; mono, color and all fix it). Colour
frames are placed on the mono cameras' clock by their dequeue time, and a
view in a camera a step lacks waits for that camera's next frame.

A grip (a closed hand) is a second gesture next to the pinch, in version 2
of the gestures file: every fingertip curled toward the wrist, beginning
only on a hand seen open within a second and held up in front. On the
2026-09-30 lit recording that leaves 6 false grips of 14, all with the
hands on the desk; pinch counts are unchanged. ft-handreplay logs grips
and finger curl, watch_gestures.py shows them, and tools/cut_sets.py
copies a few sets out of a recording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hands/README.md

@@ -56,9 +56,10 @@ The ring is mode 0600, in a folder only you can write. Frame handling:
 Options:
 
 - `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands.
-- `--with-color`: also publish the two Arcturus colour cameras, flagged `FH_CAM_COLOR`. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232) and at most 30 fps (`--color-fps`; the cameras run at 60). Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core.
+- `--with-color` (the service uses it): also publish the two Arcturus colour cameras, flagged `FH_CAM_COLOR`. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232). They run at `--color-idle` (2 fps), enough for ft-hands to tell how bright it is, until a reader asks for more in `/run/user/UID/frametop-hands/color-fps` (ft-hands writes 30 while it tracks or records with them), up to `--color-fps` (30; the cameras run at 60). `HANDS_CAMERAS=mono` leaves them out. Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core.
+- Each mono camera's latest near-black frame's mean goes in the ring (`dark_mean`): a short fixed exposure, so it follows the room's IR light, sunlight above all.
 - The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 colour cameras.
-- Colour isn't reliable yet. In the lit-room test of 2026-09-30, the colour cameras kept losing their buffer mapping: 30 frames in a row looked unchanged, the camera relearned, and after 5 relearns ft-camd exited. Each relearn samples all 32 colour buffers, which also made the mono cameras miss frames. Runs with the headset idle (no hands, no cutouts) had none of this, and no half-size copies either, while the failing runs had many. So the passthrough compositor may be writing into the colour buffers while Room View shows. Whether a frame is new is judged on the luma rows only: the chroma after them hardly changes in a lit room. `FT_CAMD_DEBUG=1` prints, at each colour stale frame, how many sampled words changed in every candidate buffer.
+- Colour isn't reliable yet. In the lit-room test of 2026-09-30, the colour cameras kept losing their buffer mapping while the headset was worn: 30 frames in a row looked unchanged, the camera relearned, and after 5 relearns ft-camd exited. Each relearn probed all 32 colour buffers, a whole-buffer cache sync each, which also made the mono cameras miss frames. Runs with the headset idle had none of this. So the passthrough compositor may be writing into the colour buffers while Room View shows. Since then a colour camera never takes the mono ones down: it probes at most 4 buffers a frame, and one that goes stale twice in a row is paused (10 s, doubling up to 160 s) and learned again, without ft-camd exiting. Whether a frame is new is judged on the luma rows only: the chroma after them hardly changes in a lit room. `FT_CAMD_DEBUG=1` prints, at each colour stale frame, how many sampled words changed in every candidate buffer.
 - `--sensor S`: only the mono cameras whose sensor name contains S.
 - `--status S`: a status line every S seconds (0: never).
 
@@ -90,6 +91,15 @@ Options:
 - `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
 - `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.
 - `--ring PATH`: read frames from another ring, such as `ft-ringplay`'s.
+- `--cams auto|mono|color|all` (`HANDS_CAMERAS`, default `auto`): which cameras to track with. The mono IR cameras light the hands themselves and track well in dim rooms, but in bright light they expose for the room and the hands come out dark. The colour pair is the other way round. `auto` goes by the colour frames' mean brightness: at `--bright-on` (`HANDS_BRIGHT_ON`, 40) or over for 2 s it tracks with `--bright` (`HANDS_BRIGHT`: `all`, every camera, the default, or `color`), and under `--bright-off` (`HANDS_BRIGHT_OFF`, 25) for 2 s with the mono cameras again. A dim evening room read 9. The switch is logged (`cameras: mono -> all (...)`), and the status line gives the colour level, the mono cameras' ambient IR, and how many steps had colour frames. Colour frames arrive on their own schedule, so a step holds the mono set, the colour pair, or both, and views wait in their camera for its next frame.
+- `--color-left NODE` (`HANDS_COLOR_LEFT`, `color_video0`) and `--color-crop subtract|none` (`HANDS_COLOR_CROP`, `subtract`): how the colour module's calibration maps onto the images. Not settled yet: `tools/check_color.py` on a recording with a lit, textured view tells.
+- `--grip-begin R`, `--grip-end R`: the grip detector (below).
+
+**Gestures** (`/run/user/UID/frametop-hands/gestures`, `include/fh_gestures.h`), for the pointer helper:
+
+- A pinch: the thumb and index tips within 2 cm, ending past 3.5 cm. Not begun with the palm facing down (`--pinch-palm-down`, 0.6), which is how typing looks.
+- A grip, a closed hand: every finger's tip nearer the wrist than 1.2 times its knuckle is (from the model's 3D hand, so hand size doesn't matter), ending when they open past 1.45 on average. It begins only on a hand seen open within the last second (closing it is the gesture), with the palm at most 35 degrees below straight ahead and at least 15 cm in front of the eyes. A grip ends a pinch on the same hand, as lost. In the 2026-09-30 lit recording (no deliberate fists), the checks cut false grips from 14 to 6, all with the hands on the desk while looking down at it; the pointer helper ignores grips that begin more than 30 cm below the eyes, which it can tell and ft-hands can't.
+- `tools/watch_gestures.py --distance` shows both live; `ft-handreplay --timeline` logs them and each hand's finger curl.
 
 The status line also says how often a hand was on each side (by where the wrist is), and why views and hands came and went: views lost (the landmark model stopped seeing the hand), handoff misses (a crop projected from the hand's 3D position found nothing), duplicates, splits (two views disagreed in 3D), and hands created, merged and forgotten.
 

hands/include/fh_gestures.h

@@ -1,23 +1,32 @@
 /*
- * fh_gestures - pinch state ft-hands publishes for input: look at something and pinch to
- * click it, pinch and move to drag (the Vision Pro model, with the eye tracker doing the
- * looking). /run/user/UID/frametop-hands/gestures, next to the hands
+ * fh_gestures - hand gestures ft-hands publishes for input (the pointer helper): look at
+ * something and pinch to click it, or close the hand (a grip) to press and drag it (the
+ * Vision Pro model, with the eye tracker doing the looking).
+ * /run/user/UID/frametop-hands/gestures, next to the hands
  * file, with the same sequence lock (read seq, copy, read seq again; use the copy only if
  * both reads are the same even number) and the same frame: metres in the head frame at
  * capture time, OpenVR's HMD frame (+x right, +y up, -z forward).
  *
- * One slot per side: pinch[0] is the left hand, pinch[1] the right. A pinch follows the
- * hand it began on until it ends. It begins when the thumb and index tips close within
- * begin_m and ends when they open past end_m (the gap between keeps it from flickering),
- * or when the hand stays lost too long (FH_PINCH_LOST).
+ * One slot per side and gesture: pinch[0] and grip[0] are the left hand, [1] the right.
+ * A gesture follows the hand it began on until it ends.
+ *   Pinch: begins when the thumb and index tips close within begin_m and ends when they
+ * open past end_m (the gap between keeps it from flickering). point is between the tips.
+ *   Grip: a closed hand. It begins when all four fingers are curled in (each fingertip
+ * nearer the wrist than grip_begin times its knuckle is) and ends when they open past
+ * grip_end on average. distance is that average (about 2 open, under 1.2 closed), strength
+ * 0 open .. 1 closed, and point the palm's centre. A grip ends a pinch on the same hand
+ * (closing the hand can pass through a pinch on the way), as lost.
+ * Either ends, as lost (FH_PINCH_LOST), when its hand stays lost too long.
  *
- * Don't miss short pinches: a reader that polls slower than a quick tap still sees it,
- * because begins and ends count every pinch. When begins changed, a pinch began at
- * begin_ns; when ends changed, one ended at end_ns. begins - ends is 1 while pinching.
+ * Don't miss short gestures: a reader that polls slower than a quick tap still sees it,
+ * because begins and ends count every one. When begins changed, one began at begin_ns;
+ * when ends changed, one ended at end_ns. begins - ends is 1 while it's down.
  *
- * Drags: point is where the pinch is now, begin_point where it began. Turn each into the
+ * Drags: point is where the gesture is now, begin_point where it began. Turn each into the
  * room with the HMD pose at its capture time (capture_ns, begin_ns) before subtracting,
  * so turning your head doesn't drag.
+ *
+ * Version 1 had only the pinches (192 bytes); version 2 adds the grips after them.
  */
 
 #pragma once
@@ -26,27 +35,29 @@
 #include <stdint.h>
 
 #define FH_GESTURES_MAGIC   "FHGEST01"
-#define FH_GESTURES_VERSION 1
+#define FH_GESTURES_VERSION 2
 
 enum {
     FH_PINCH_TRACKED = 1u << 0, /* the hand was tracked in this frame              */
-    FH_PINCH_DOWN    = 1u << 1, /* pinching now                                     */
-    FH_PINCH_LOST    = 1u << 2, /* the last pinch ended because the hand was lost   */
+    FH_PINCH_DOWN    = 1u << 1, /* the gesture is held now                          */
+    FH_PINCH_LOST    = 1u << 2, /* the last one ended because the hand was lost     */
+                                /* (or, for a pinch, a grip took over)              */
 };
 
 typedef struct {
     uint32_t flags;             /* FH_PINCH_*                                       */
     uint32_t hand_id;           /* fh_hand_t.id of the hand, 0 if none              */
-    uint32_t begins;            /* pinches begun so far                             */
-    uint32_t ends;              /* pinches ended so far                             */
+    uint32_t begins;            /* begun so far                                     */
+    uint32_t ends;              /* ended so far                                     */
     uint64_t begin_ns;          /* capture time (CLOCK_MONOTONIC) the current or    */
-                                /* last pinch began                                 */
-    uint64_t end_ns;            /* ... the last pinch ended                         */
-    float    distance;          /* thumb tip to index tip, m, at this user's hand   */
-                                /* size                                             */
-    float    strength;          /* 0 open (end_m or more) .. 1 closed (begin_m)     */
-    float    point[3];          /* between the thumb and index tips                 */
-    float    begin_point[3];    /* point when the current or last pinch began       */
+                                /* last one began                                   */
+    uint64_t end_ns;            /* ... the last one ended                           */
+    float    distance;          /* pinch: thumb tip to index tip, m, at this user's */
+                                /* hand size. grip: the fingers' mean curl (above)  */
+    float    strength;          /* 0 open .. 1 closed                               */
+    float    point[3];          /* pinch: between the thumb and index tips; grip:   */
+                                /* the palm's centre                                */
+    float    begin_point[3];    /* point when the current or last one began         */
 } fh_pinch_t;                   /* 64 bytes */
 
 typedef struct {
@@ -56,11 +67,14 @@ typedef struct {
     volatile uint64_t seq;
     uint64_t          capture_ns;   /* CLOCK_MONOTONIC when the cameras took the frames */
     uint64_t          publish_ns;   /* CLOCK_MONOTONIC when this was written            */
-    float             begin_m;      /* the thresholds in use                            */
+    float             begin_m;      /* the pinch thresholds in use                      */
     float             end_m;
-    uint8_t           reserved[16];
+    float             grip_begin;   /* the grip thresholds in use (curl ratios)         */
+    float             grip_end;
+    uint8_t           reserved[8];
     fh_pinch_t        pinch[2];     /* [0] left hand, [1] right hand                    */
+    fh_pinch_t        grip[2];      /* version 2                                        */
 } fh_gestures_t;
 
 static_assert(sizeof(fh_pinch_t) == 64, "fh_pinch_t layout");
-static_assert(sizeof(fh_gestures_t) == 64 + 2 * 64, "fh_gestures_t layout");
+static_assert(sizeof(fh_gestures_t) == 64 + 4 * 64, "fh_gestures_t layout");

hands/tools/cut_sets.py

@@ -0,0 +1,51 @@
+"""Copy a few frame sets out of a recording (ft-hands --record) into a small one, to look at
+or check elsewhere without moving gigabytes. Plain Python, so it runs on the Frame's host.
+
+usage: python3 tools/cut_sets.py REC_DIR OUT_DIR [--sets N] (8, spread evenly) | [--at I,J,...]
+"""
+import argparse
+import os
+import struct
+
+HDR = struct.Struct('<8sII')
+
+
+def offsets(path):
+    offs, size = [], os.path.getsize(path)
+    with open(path, 'rb') as f:
+        off = 0
+        while off + HDR.size <= size:
+            f.seek(off)
+            magic, _, nbytes = HDR.unpack(f.read(HDR.size))
+            if magic[:7] != b'FHSET01' or off + nbytes > size:
+                break
+            offs.append((off, nbytes))
+            off += nbytes
+    return offs
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('rec')
+    ap.add_argument('out')
+    ap.add_argument('--sets', type=int, default=8)
+    ap.add_argument('--at', default='')
+    a = ap.parse_args()
+    src = os.path.join(a.rec, 'sets.bin')
+    offs = offsets(src)
+    if a.at:
+        pick = [int(i) for i in a.at.split(',')]
+    else:
+        n = max(1, min(a.sets, len(offs)))
+        pick = [round(i * (len(offs) - 1) / max(n - 1, 1)) for i in range(n)]
+    os.makedirs(a.out, exist_ok=True)
+    with open(src, 'rb') as f, open(os.path.join(a.out, 'sets.bin'), 'wb') as out:
+        for i in pick:
+            off, nbytes = offs[i]
+            f.seek(off)
+            out.write(f.read(nbytes))
+    print('%d of %d sets (%s) -> %s' % (len(pick), len(offs), ','.join(map(str, pick)), a.out))
+
+
+if __name__ == '__main__':
+    main()

hands/tools/watch_gestures.py

@@ -1,13 +1,14 @@
-"""Watch the pinch gestures ft-hands publishes, live: begins, ends, and drags.
+"""Watch the gestures ft-hands publishes, live: pinches and grips, begins, ends, and drags.
 
 usage: python3 tools/watch_gestures.py [--every S] [--distance]
 
-Prints a line when a pinch begins or ends on either hand. It goes by the counters, so a
-quick tap between two reads still shows. While a pinch is held, every --every seconds
-(default 0.1) it prints how far the pinch point has moved since it began, in the head
-frame (turning your head moves it too; a real consumer turns both points into the room
+Prints a line when a pinch or a grip (a closed hand) begins or ends on either hand. It goes
+by the counters, so a quick tap between two reads still shows. While one is held, every
+--every seconds (default 0.1) it prints how far its point has moved since it began, in the
+head frame (turning your head moves it too; a real consumer turns both points into the room
 first, see include/fh_gestures.h). --distance also prints each hand's thumb-to-index
-distance, to see how close a pinch comes to the thresholds.
+distance and finger curl, to see how close a gesture comes to the thresholds.
+Version 1 files (pinches only) still work.
 """
 import argparse
 import mmap
@@ -15,10 +16,11 @@ import os
 import struct
 import time
 
-HDR = struct.Struct('<8sIIQQQff16x')         # 64 bytes
-PINCH = struct.Struct('<IIIIQQff3f3f')       # 64 bytes
+HDR = struct.Struct('<8sIIQQQffff8x')        # 64 bytes
+SLOT = struct.Struct('<IIIIQQff3f3f')        # 64 bytes
 TRACKED, DOWN, LOST = 1, 2, 4
 SIDES = ('left ', 'right')
+KINDS = ('pinch', 'grip ')
 
 
 def path():
@@ -26,57 +28,66 @@ def path():
 
 
 def read(m):
-    """(header, [left, right]) under the sequence lock, or None if it's being written."""
+    """(header, [[pinch left, right], [grip left, right]]) under the sequence lock, or None
+    if it's being written. Version 1 has no grips: they read as all zero."""
     for _ in range(10):
         s1 = struct.unpack_from('<Q', m, 16)[0]
         if s1 % 2 == 0:
             h = HDR.unpack_from(m, 0)
-            p = [PINCH.unpack_from(m, HDR.size + k * PINCH.size) for k in range(2)]
+            kinds = 2 if h[1] >= 2 and len(m) >= HDR.size + 4 * SLOT.size else 1
+            g = [[SLOT.unpack_from(m, HDR.size + (k * 2 + s) * SLOT.size) for s in range(2)] for k in range(kinds)]
+            if kinds == 1:
+                g.append([(0,) * 14, (0,) * 14])
             if struct.unpack_from('<Q', m, 16)[0] == s1:
-                return h, p
+                return h, g
         time.sleep(0.0005)
     return None
 
 
 def main():
     ap = argparse.ArgumentParser()
-    ap.add_argument('--every', type=float, default=0.1, help='seconds between drag lines while pinching')
-    ap.add_argument('--distance', action='store_true', help="print each hand's thumb-to-index distance")
+    ap.add_argument('--every', type=float, default=0.1, help='seconds between drag lines while held')
+    ap.add_argument('--distance', action='store_true', help="print each hand's pinch distance and finger curl")
     a = ap.parse_args()
     with open(path(), 'rb') as f:
         m = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
     first = read(m)
     if first is None or first[0][0] != b'FHGEST01':
         raise SystemExit('%s is not an ft-hands gestures file' % path())
-    h, p = first
-    print('thresholds: pinch begins under %.3f m, ends over %.3f m' % (h[6], h[7]))
-    seen = [(q[2], q[3]) for q in p]   # begins, ends
+    h, g = first
+    print('version %d; thresholds: pinch begins under %.3f m, ends over %.3f m; grip begins with every finger '
+          'curled under %.2f, ends over %.2f' % (h[1], h[6], h[7], h[8], h[9]))
+    seen = [[(q[2], q[3]) for q in kind] for kind in g]   # begins, ends
     last_drag = last_dist = 0.0
     while True:
         got = read(m)
         if got:
-            h, p = got
+            h, g = got
             now = time.monotonic()
-            for k, q in enumerate(p):
-                flags, hand, begins, ends, begin_ns, end_ns, dist, strength = q[:8]
-                point, begin_point = q[8:11], q[11:14]
-                if begins != seen[k][0]:
-                    print('%s pinch BEGIN (#%d, hand %d) at %+.3f %+.3f %+.3f  d %.3f' %
-                          (SIDES[k], begins, hand, *begin_point, dist), flush=True)
-                if ends != seen[k][1]:
-                    held = (end_ns - begin_ns) / 1e9 if end_ns >= begin_ns else 0
-                    print('%s pinch %s after %.2f s' % (SIDES[k], 'LOST' if flags & LOST else 'END', held), flush=True)
-                seen[k] = (begins, ends)
-                if flags & DOWN and now - last_drag >= a.every:
-                    d = [point[i] - begin_point[i] for i in range(3)]
-                    print('%s   drag %+6.1f %+6.1f %+6.1f mm  (%.0f mm)' %
-                          (SIDES[k], *(1000 * x for x in d), 1000 * sum(x * x for x in d) ** 0.5), flush=True)
-            if any(q[0] & DOWN for q in p) and now - last_drag >= a.every:
+            for k, kind in enumerate(g):
+                for s, q in enumerate(kind):
+                    flags, hand, begins, ends, begin_ns, end_ns, dist, strength = q[:8]
+                    point, begin_point = q[8:11], q[11:14]
+                    if begins != seen[k][s][0]:
+                        print('%s %s BEGIN (#%d, hand %d) at %+.3f %+.3f %+.3f  %s %.3f' %
+                              (SIDES[s], KINDS[k], begins, hand, *begin_point, 'curl' if k else 'd', dist), flush=True)
+                    if ends != seen[k][s][1]:
+                        held = (end_ns - begin_ns) / 1e9 if end_ns >= begin_ns else 0
+                        print('%s %s %s after %.2f s' % (SIDES[s], KINDS[k], 'LOST' if flags & LOST else 'END', held),
+                              flush=True)
+                    seen[k][s] = (begins, ends)
+                    if flags & DOWN and now - last_drag >= a.every:
+                        d = [point[i] - begin_point[i] for i in range(3)]
+                        print('%s %s   drag %+6.1f %+6.1f %+6.1f mm  (%.0f mm)' %
+                              (SIDES[s], KINDS[k], *(1000 * x for x in d), 1000 * sum(x * x for x in d) ** 0.5),
+                              flush=True)
+            if any(q[0] & DOWN for kind in g for q in kind) and now - last_drag >= a.every:
                 last_drag = now
             if a.distance and now - last_dist >= 0.2:
                 last_dist = now
-                print('    ' + '   '.join('%s %s' % (SIDES[k].strip(), 'd %.3f s %.2f' % (q[6], q[7]) if q[0] & TRACKED
-                                                        else '-') for k, q in enumerate(p)), flush=True)
+                print('    ' + '   '.join(
+                    '%s %s' % (SIDES[s].strip(), 'd %.3f curl %.2f' % (g[0][s][6], g[1][s][6]) if g[0][s][0] & TRACKED
+                               else '-') for s in range(2)), flush=True)
         time.sleep(0.005)
 
 

hands/track/main.cpp

@@ -1,12 +1,29 @@
 // ft-hands: hands in 3D from ft-camd's ring, published for Frametop's ft-screens (the hand
-// cutouts), and pinches for the pointer. It started as a port of frame-hands' Python
-// prototype: the same scheduling, with the models on a few threads.
+// cutouts), and pinches and grips for the pointer. It started as a port of frame-hands'
+// Python prototype: the same scheduling, with the models on a few threads.
 //
 //   ft-hands [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N]
-//            [--no-publish] [--record DIR] [--swap-sides] ... (--help lists them all)
+//            [--no-publish] [--record DIR] [--swap-sides] [--cams auto|mono|color|all] ...
+//            (--help lists them all)
+//
+// Which cameras (--cams, HANDS_CAMERAS): the four mono IR cameras light the hands with their
+// own IR and track well in dim rooms, but in bright light (a sunny room, a window behind the
+// hands) they expose for the room and the hands come out dark. The two Arcturus colour
+// cameras (the passthrough pair, forward-facing, 145 degrees) are the other way round: dark
+// and grainy in a dim room, clear in a lit one. auto (the default) picks by how bright the
+// colour cameras' frames are: at HANDS_BRIGHT_ON (mean luma) or over for 2 s, it tracks with
+// HANDS_BRIGHT (all: every camera, so hands low at the sides stay in the side cameras; or
+// color); under HANDS_BRIGHT_OFF for 2 s, with the mono cameras again. ft-camd runs the colour
+// cameras at 2 fps, enough to tell the light, until ft-hands asks for 30
+// (/run/user/UID/frametop-hands/color-fps). The colour frames' capture times are on their
+// own clock, so they're placed on the mono cameras' by when they were dequeued, less the
+// mono cameras' measured delay.
 //
 // Settings in ~/.config/frametop.conf (FT_<name> in the environment overrides them, and
-// options override both): HANDS_SWAP_SIDES (1: as --swap-sides), HANDS_CPUS (as --cpus).
+// options override both): HANDS_SWAP_SIDES (1: as --swap-sides), HANDS_CPUS (as --cpus),
+// HANDS_CAMERAS, HANDS_BRIGHT, HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT (which
+// colour camera is passthrough_left: color_video0 or color_video3), HANDS_COLOR_CROP
+// (subtract or none: tools/check_color.py tells both).
 #include "io.h"
 #include "pinch.h"
 #include "record.h"
@@ -16,10 +33,12 @@
 #include <sys/stat.h>
 #include <unistd.h>
 
+#include <cmath>
 #include <ctime>
 #include <memory>
 
 #include <csignal>
+#include <cstdlib>
 #include <cstdio>
 #include <cstring>
 #include <fstream>
@@ -88,6 +107,44 @@ double cpu_seconds() {
     return r.ru_utime.tv_sec + r.ru_stime.tv_sec + (r.ru_utime.tv_usec + r.ru_stime.tv_usec) / 1e6;
 }
 
+enum class Cams { Mono, Color, All };
+
+const char *cams_name(Cams c) { return c == Cams::Mono ? "mono" : c == Cams::Color ? "color" : "all"; }
+
+bool parse_cams(const std::string &s, Cams &out) {
+    if (s == "mono") out = Cams::Mono;
+    else if (s == "color") out = Cams::Color;
+    else if (s == "all") out = Cams::All;
+    else return false;
+    return true;
+}
+
+// How bright it is, for auto (see the top): the colour frames' mean luma, smoothed over about
+// a second, with hysteresis and a 2 s hold each way. No colour frames for 3 s (ft-camd paused
+// them, or has none) reads as dim.
+struct Lighting {
+    double on = 40, off = 25;
+    double level = -1;
+    bool bright = false;
+    uint64_t at_ns = 0, since_ns = 0;   // the last frame; since when it's wanted the other way
+
+    void add(double mean, uint64_t t_ns) {
+        const double dt = at_ns && t_ns > at_ns ? (t_ns - at_ns) / 1e9 : 1.0;
+        level = level < 0 ? mean : level + (mean - level) * std::min(1.0, dt / 1.0);
+        at_ns = t_ns;
+    }
+    // True when it switched.
+    bool update(uint64_t now_ns) {
+        if (level >= 0 && now_ns - at_ns > 3'000'000'000ull) level = -1;
+        const bool want = level >= 0 && (bright ? level > off : level >= on);
+        if (want == bright) return since_ns = 0, false;
+        if (!since_ns) since_ns = now_ns;
+        if (now_ns - since_ns < 2'000'000'000ull) return false;
+        bright = want, since_ns = 0;
+        return true;
+    }
+};
+
 }  // namespace
 
 int main(int argc, char **argv) {
@@ -104,11 +161,22 @@ int main(int argc, char **argv) {
     std::vector<int> cpus = {5, 6, 7};
     if (const auto c = parse_cpus(setting("HANDS_CPUS").c_str()); !c.empty()) cpus = c;
     swap_sides = setting("HANDS_SWAP_SIDES") == "1";
+    // Which cameras (see the top).
+    std::string cams_arg = setting("HANDS_CAMERAS"), bright_arg = setting("HANDS_BRIGHT");
+    std::string color_left = setting("HANDS_COLOR_LEFT"), color_crop = setting("HANDS_COLOR_CROP");
+    if (cams_arg.empty()) cams_arg = "auto";
+    if (bright_arg.empty()) bright_arg = "all";
+    if (color_left.empty()) color_left = "color_video0";
+    if (color_crop.empty()) color_crop = "subtract";
+    Lighting light;
+    if (const std::string v = setting("HANDS_BRIGHT_ON"); !v.empty()) light.on = std::atof(v.c_str());
+    if (const std::string v = setting("HANDS_BRIGHT_OFF"); !v.empty()) light.off = std::atof(v.c_str());
     // How crops are equalized. CLAHE helps the palm search find hands (about 10% more in the
     // dim recording), but makes the landmarks jitter, so they get plain crops.
     Contrast palm_contrast, hand_contrast{Contrast::None};
     double keep_presence = 0.5;   // landmark presence a tracked view needs to stay
     PinchParams pinch_params;
+    GripParams grip_params;
     double record_for = 120;
     for (int i = 1; i < argc; ++i) {
         const std::string a = argv[i];
@@ -124,12 +192,20 @@ int main(int argc, char **argv) {
         else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
         else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
         else if (a == "--pinch-palm-down" && more) pinch_params.palm_down_max = std::atof(argv[++i]);
+        else if (a == "--grip-begin" && more) grip_params.begin = std::atof(argv[++i]);
+        else if (a == "--grip-end" && more) grip_params.end = std::atof(argv[++i]);
         else if (a == "--swap-sides") swap_sides = true;
         else if (a == "--record-only") track = publish = false;
         else if (a == "--ring" && more) ring_path = argv[++i];
         else if (a == "--record" && more) record = argv[++i];
         else if (a == "--record-for" && more) record_for = std::atof(argv[++i]);
         else if (a == "--keep-presence" && more) keep_presence = std::atof(argv[++i]);
+        else if (a == "--cams" && more) cams_arg = argv[++i];
+        else if (a == "--bright" && more) bright_arg = argv[++i];
+        else if (a == "--bright-on" && more) light.on = std::atof(argv[++i]);
+        else if (a == "--bright-off" && more) light.off = std::atof(argv[++i]);
+        else if (a == "--color-left" && more) color_left = argv[++i];
+        else if (a == "--color-crop" && more) color_crop = argv[++i];
         else if (a == "--contrast" && more) {
             if (!Contrast::parse_pair(argv[++i], palm_contrast, hand_contrast))
                 return std::fprintf(stderr, "--contrast MODE or PALM/HAND, each clahe[:CLIP]|none|stretch\n"), 1;
@@ -141,17 +217,27 @@ int main(int argc, char **argv) {
             std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n"
                         "          [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n"
                         "          [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
+                        "          [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
+                        "          [--color-left color_video0|color_video3] [--color-crop subtract|none]\n"
                         "          [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (0.6)\n"
+                        "          [--grip-begin R] (1.2) [--grip-end R] (1.45)\n"
                         "          [--contrast MODE|PALM/HAND] (clahe[:CLIP], none, stretch; default clahe:2/none)\n"
                         "Recording saves every frame set for S seconds (120) to DIR/sets.bin, for ft-handreplay; SIGUSR1\n"
                         "starts one in ~/.local/share/frametop/hands/rec-<time>. --record-only records without tracking, so it\n"
                         "can run beside a tracking ft-hands. With ft-camd --with-dark, recordings also get each\n"
                         "camera's newest dark frame, as <name>_dk; with --with-color, the color cameras' as color_video<N>.\n"
-                        "Settings in ~/.config/frametop.conf: HANDS_SWAP_SIDES=1, HANDS_CPUS=5,6,7 (FT_<name> overrides).\n",
+                        "auto picks the cameras by the light (see the top of track/main.cpp).\n"
+                        "Settings in ~/.config/frametop.conf: HANDS_SWAP_SIDES=1, HANDS_CPUS=5,6,7, HANDS_CAMERAS, HANDS_BRIGHT,\n"
+                        "HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT, HANDS_COLOR_CROP (FT_<name> overrides).\n",
                         argv[0]);
             return a == "--help" ? 0 : 1;
         }
     }
+    const bool automatic = cams_arg == "auto";
+    Cams fixed = Cams::Mono, bright_cams = Cams::All;
+    if ((!automatic && !parse_cams(cams_arg, fixed)) || !parse_cams(bright_arg, bright_cams) || bright_cams == Cams::Mono)
+        return std::fprintf(stderr, "--cams auto|mono|color|all, --bright all|color\n"), 1;
+    if (color_crop != "subtract" && color_crop != "none") return std::fprintf(stderr, "--color-crop subtract|none\n"), 1;
     std::setvbuf(stdout, nullptr, _IOLBF, 0);   // whole lines to the journal as they come
     if (nice(niceness) < 0) std::perror("nice");   // the VR stack wins contested CPUs
     std::signal(SIGINT, [](int) { g_stop = 1; });
@@ -165,6 +251,7 @@ int main(int argc, char **argv) {
     Publisher pub;
     GesturePublisher gestures;
     Pinch pinch(pinch_params);
+    Grip grip(grip_params);
     std::unique_ptr<Recorder> rec;
     uint64_t rec_start = 0;
     auto start_recording = [&](const std::string &dir, std::string &e) {
@@ -176,21 +263,23 @@ int main(int argc, char **argv) {
         return true;
     };
     if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
-        (publish && (!pub.open(err) || !gestures.open(pinch, err))) || (!record.empty() && !start_recording(record, err))) {
+        (publish && (!pub.open(err) || !gestures.open(pinch, grip, err))) ||
+        (!record.empty() && !start_recording(record, err))) {
         std::fprintf(stderr, "%s\n", err.c_str());
         return 1;
     }
     if (!ring.alive()) return std::fprintf(stderr, "ft-camd isn't running (no heartbeat)\n"), 1;
 
-    std::map<std::string, int> index;   // calibration name -> ring camera
-    // Recorded only, not tracked: "<name>_dk" (ft-camd --with-dark) and "color_video<N>"
-    // (--with-color; which is left and right is up to tools/check_color.py). Recorded names
-    // hold 15 characters, so "upper_right_dark" wouldn't fit.
+    std::map<std::string, int> index;   // mono calibration name -> ring camera
+    std::map<std::string, int> color;   // colour calibration name (color_video<N>) -> ring camera
+    // Recorded as they are with each set: "<name>_dk" (ft-camd --with-dark) and "color_video<N>"
+    // (--with-color). Recorded names hold 15 characters, so "upper_right_dark" wouldn't fit.
     std::map<std::string, int> dark;
     std::map<std::string, Camera> used;
     for (int i = 0; i < ring.cameras(); ++i) {
         if (ring.camera(i).flags & FH_CAM_COLOR) {
-            dark["color_video" + std::to_string(ring.camera(i).node)] = i;
+            const std::string name = "color_video" + std::to_string(ring.camera(i).node);
+            dark[name] = i, color[name] = i;
             continue;
         }
         // ft-camd's cameras by capture pipe; ft-ringplay's (no device) by the name it gives
@@ -207,9 +296,34 @@ int main(int argc, char **argv) {
         if (dark.count("slam_left_dk") && dark.count("slam_right_dk")) std::swap(dark["slam_left_dk"], dark["slam_right_dk"]);
         std::printf("side cameras swapped (--swap-sides)\n");
     }
+    // The colour pair, calibrated (see the top), unless only the mono cameras are wanted.
+    if (color.size() == 2 && (automatic || fixed != Cams::Mono)) {
+        const std::string left = color.count(color_left) ? color_left : color.begin()->first;
+        const std::string right = color.begin()->first == left ? std::next(color.begin())->first : color.begin()->first;
+        const int scale = int(std::lround(1972.0 / ring.camera(color[left]).width));
+        std::map<std::string, Camera> cc;
+        std::string e;
+        if (load_color_calibration(cc, left, right, color_crop == "subtract", scale, e)) {
+            for (auto &[name, cam] : cc) used[name] = cam;
+            std::printf("colour cameras: %s is passthrough_left, crop %s, 1/%d size\n", left.c_str(), color_crop.c_str(), scale);
+        } else {
+            std::fprintf(stderr, "colour cameras left out: %s\n", e.c_str());
+            color.clear();
+        }
+    } else {
+        color.clear();
+    }
+    if (color.empty() && (automatic || fixed != Cams::Mono)) {
+        if (!automatic) std::printf("no colour cameras (ft-camd --with-color): tracking with the mono ones\n");
+        fixed = Cams::Mono;
+    }
+    const bool switching = automatic && !color.empty();
+    Cams mode = switching || color.empty() ? Cams::Mono : fixed;
     std::printf("cameras:");
     for (auto &[name, i] : index) std::printf(" %s=video%d", name.c_str(), ring.camera(i).node);
-    std::printf("  models: %s%s, %d threads on CPUs", models.c_str(), int8 ? " (int8)" : "", threads);
+    for (auto &[name, i] : color) std::printf(" %s", name.c_str());
+    std::printf("  tracking with %s%s  models: %s%s, %d threads on CPUs", cams_name(mode),
+                switching ? " (auto: by the light)" : "", models.c_str(), int8 ? " (int8)" : "", threads);
     for (int c : cpus) std::printf(" %d", c);
     std::printf("\n");
 
@@ -222,103 +336,221 @@ int main(int argc, char **argv) {
     Tracker tracker(used, nets, pool);
     tracker.set_keep_presence(keep_presence);
     std::map<std::string, std::vector<uint8_t>> pixels;
-    std::map<std::string, uint64_t> last;
+    std::map<std::string, uint64_t> last;       // per camera: the frame last used
+    std::map<std::string, uint64_t> lit_seen;   // per colour camera: the frame last counted for the light
     const uint64_t start = mono_ns();
-    uint64_t t_status = start, next_ns = 0;
+    uint64_t t_status = start, next_ns = 0, t_want = 0;
     double cpu0 = cpu_seconds();
     std::vector<double> lat;
     double hands_sum = 0, resid_sum = 0;
-    int resid_n = 0, left_sets = 0, right_sets = 0, both_sets = 0;
+    int resid_n = 0, left_sets = 0, right_sets = 0, both_sets = 0, color_steps = 0;
+    // The mono cameras' delay from capture to dequeue (their capture clock is CLOCK_MONOTONIC_RAW),
+    // to place the colour frames, whose capture clock is their own (see the top).
+    double mono_delay_ns = -1;
+    const std::string want_file = run_dir() + "/color-fps";
+
+    // The colour pair's newest frames if both are newer than last used and taken together
+    // (their own clock): their time, on the mono cameras' capture clock, else 0.
+    auto color_pair = [&](int64_t raw_off, bool copy, std::map<std::string, Image> &images) -> uint64_t {
+        fh_ring_slot_t meta[2];
+        std::string names[2];
+        uint64_t n[2];
+        int k = 0;
+        for (auto &[name, i] : color) {
+            names[k] = name, n[k] = ring.latest(i);
+            if (n[k] <= last[name] || !ring.meta(i, n[k], &meta[k])) return 0;
+            ++k;
+        }
+        if (k != 2 || mono_delay_ns < 0) return 0;
+        const int64_t apart = int64_t(meta[0].capture_ns) - int64_t(meta[1].capture_ns);
+        if (std::llabs(apart) > 3'000'000) return 0;   // one is a frame ahead: wait for the other
+        const uint64_t dq = std::min(meta[0].dqbuf_ns, meta[1].dqbuf_ns);
+        const uint64_t t = uint64_t(int64_t(dq) - int64_t(mono_delay_ns) + raw_off);
+        if (!copy) return t;
+        for (int j = 0; j < 2; ++j) {
+            const int i = color[names[j]];
+            if (!ring.read(i, n[j], pixels[names[j]], &meta[j])) return 0;
+            const auto &c = ring.camera(i);
+            images[names[j]] = {pixels[names[j]].data(), int(c.width), int(c.height), int(c.width)};
+        }
+        for (int j = 0; j < 2; ++j) last[names[j]] = n[j];
+        return t;
+    };
+    auto switch_to = [&](Cams to, const char *why) {
+        if (to == mode) return;
+        for (auto &[name, cam] : used) {
+            const bool is_color = color.count(name) > 0;
+            const bool keep = to == Cams::All || (to == Cams::Color) == is_color;
+            if (!keep) tracker.drop_camera(name);
+        }
+        std::printf("cameras: %s -> %s (%s)\n", cams_name(mode), cams_name(to), why);
+        mode = to;
+    };
+    auto ambient = [&] {   // the mono cameras' dark frames: the room's IR light
+        double sum = 0;
+        int n = 0;
+        for (auto &[name, i] : index)
+            if (ring.camera(i).dark_mean > 0) sum += ring.camera(i).dark_mean, ++n;
+        return n ? sum / n : -1;
+    };
 
     while (!g_stop && (seconds <= 0 || (mono_ns() - start) / 1e9 < seconds)) {
         if (!ring.alive()) return std::fprintf(stderr, "ft-camd stopped\n"), 2;
-        // a new frame set: every camera has a newer frame, taken at the same moment
-        std::map<std::string, uint64_t> latest;
-        bool ready = true;
-        for (auto &[name, i] : index) {
-            latest[name] = ring.latest(i);
-            ready = ready && latest[name] > last[name];
+        const uint64_t now0 = mono_ns();
+        const int64_t raw_off = raw_minus_mono_ns();
+
+        // The light, from the colour frames' brightness (their slot headers only).
+        for (auto &[name, i] : color) {
+            fh_ring_slot_t m;
+            const uint64_t n = ring.latest(i);
+            if (n && n != lit_seen[name] && ring.meta(i, n, &m)) light.add(m.mean, m.dqbuf_ns), lit_seen[name] = n;
         }
-        if (!ready) {
-            std::this_thread::sleep_for(std::chrono::milliseconds(2));
-            continue;
+        if (switching && light.update(now0)) {
+            char why[96];
+            std::snprintf(why, sizeof why, "colour frames at %.0f, ambient IR %.1f", light.level, ambient());
+            switch_to(light.bright ? bright_cams : Cams::Mono, why);
         }
-        // not needed at the current rate, and not recorded: skip it without copying images
-        if (track && !rec && !g_record) {
-            uint64_t t0 = UINT64_MAX, t1 = 0;
+        // Ask ft-camd for the colour cameras' full rate while tracking or recording with them.
+        const bool want_color = !color.empty() && (mode != Cams::Mono || rec || g_record);
+        if (!color.empty() && now0 - t_want > 1'000'000'000) {
+            t_want = now0;
+            if (want_color) {
+                if (FILE *f = std::fopen(want_file.c_str(), "w")) std::fputs("30\n", f), std::fclose(f);
+            } else {
+                unlink(want_file.c_str());
+            }
+        }
+
+        const bool use_mono = mode != Cams::Color, use_color = mode != Cams::Mono;
+        const bool mono_driven = use_mono || rec || g_record || !track;
+        std::map<std::string, Image> images;
+        uint64_t tmin = UINT64_MAX, dq = 0;
+
+        if (mono_driven) {
+            // a new frame set: every mono camera has a newer frame, taken at the same moment
+            std::map<std::string, uint64_t> latest;
+            bool ready = true;
+            for (auto &[name, i] : index) {
+                latest[name] = ring.latest(i);
+                ready = ready && latest[name] > last[name];
+            }
+            if (!ready) {
+                std::this_thread::sleep_for(std::chrono::milliseconds(2));
+                continue;
+            }
+            // not needed at the current rate, and not recorded: skip it without copying images
+            if (track && !rec && !g_record) {
+                uint64_t t0 = UINT64_MAX, t1 = 0;
+                bool ok = true;
+                for (auto &[name, i] : index) {
+                    fh_ring_slot_t meta;
+                    ok = ok && ring.meta(i, latest[name], &meta);
+                    if (ok) t0 = std::min(t0, meta.capture_ns), t1 = std::max(t1, meta.capture_ns);
+                }
+                if (ok && t1 - t0 <= 3'000'000 && t0 < next_ns) {
+                    for (auto &[name, i] : index) last[name] = latest[name];
+                    continue;
+                }
+            }
+            std::vector<SetFrame> frames;
+            uint64_t tmax = 0;
             bool ok = true;
             for (auto &[name, i] : index) {
                 fh_ring_slot_t meta;
-                ok = ok && ring.meta(i, latest[name], &meta);
-                if (ok) t0 = std::min(t0, meta.capture_ns), t1 = std::max(t1, meta.capture_ns);
+                ok = ok && ring.read(i, latest[name], pixels[name], &meta);
+                if (!ok) break;
+                const auto &c = ring.camera(i);
+                images[name] = {pixels[name].data(), int(c.width), int(c.height), int(c.width)};
+                frames.push_back({name, pixels[name].data(), c.width, c.height, meta.capture_ns, meta.dqbuf_ns});
+                tmin = std::min(tmin, meta.capture_ns), tmax = std::max(tmax, meta.capture_ns), dq = std::max(dq, meta.dqbuf_ns);
+                const double delay = double(int64_t(meta.dqbuf_ns) - (int64_t(meta.capture_ns) - raw_off));
+                mono_delay_ns = mono_delay_ns < 0 ? delay : mono_delay_ns + 0.02 * (delay - mono_delay_ns);
             }
-            if (ok && t1 - t0 <= 3'000'000 && t0 < next_ns) {
-                last = latest;
+            if (!ok || tmax - tmin > 3'000'000) {   // torn, or a camera is a frame behind
+                std::this_thread::sleep_for(std::chrono::milliseconds(1));
                 continue;
             }
-        }
-        std::map<std::string, Image> images;
-        std::vector<SetFrame> frames;
-        uint64_t tmin = UINT64_MAX, tmax = 0, dq = 0;
-        bool ok = true;
-        for (auto &[name, i] : index) {
-            fh_ring_slot_t meta;
-            ok = ok && ring.read(i, latest[name], pixels[name], &meta);
-            if (!ok) break;
-            const auto &c = ring.camera(i);
-            images[name] = {pixels[name].data(), int(c.width), int(c.height), int(c.width)};
-            frames.push_back({name, pixels[name].data(), c.width, c.height, meta.capture_ns, meta.dqbuf_ns});
-            tmin = std::min(tmin, meta.capture_ns), tmax = std::max(tmax, meta.capture_ns), dq = std::max(dq, meta.dqbuf_ns);
-        }
-        if (!ok || tmax - tmin > 3'000'000) {   // torn, or a camera is a frame behind
-            std::this_thread::sleep_for(std::chrono::milliseconds(1));
-            continue;
-        }
-        last = latest;
-        if (g_record && !rec) {
-            g_record = 0;
-            char name[64];
-            const std::time_t now = std::time(nullptr);
-            std::strftime(name, sizeof name, "rec-%Y%m%d-%H%M%S", std::localtime(&now));
-            const std::string dir = recordings_dir();
-            std::string e;
-            if (!start_recording(dir + "/" + name, e)) std::fprintf(stderr, "%s\n", e.c_str());
-        }
-        if (rec) {   // about 80 MB/s; dark frames double that, color frames add 70 MB/s
-            if ((mono_ns() - rec_start) / 1e9 < record_for) {
-                for (auto &[name, i] : dark) {   // the newest dark and color frames, as they are
-                    fh_ring_slot_t meta;
-                    const uint64_t n = ring.latest(i);
-                    const auto &c = ring.camera(i);
-                    if (n && ring.read(i, n, pixels[name], &meta))
-                        frames.push_back({name, pixels[name].data(), c.width, c.height, meta.capture_ns, meta.dqbuf_ns});
+            for (auto &[name, i] : index) last[name] = latest[name];
+            if (g_record && !rec) {
+                g_record = 0;
+                char name[64];
+                const std::time_t now = std::time(nullptr);
+                std::strftime(name, sizeof name, "rec-%Y%m%d-%H%M%S", std::localtime(&now));
+                const std::string dir = recordings_dir();
+                std::string e;
+                if (!start_recording(dir + "/" + name, e)) std::fprintf(stderr, "%s\n", e.c_str());
+            }
+            if (rec) {   // about 80 MB/s; dark frames double that, color frames add 70 MB/s
+                if ((mono_ns() - rec_start) / 1e9 < record_for) {
+                    for (auto &[name, i] : dark) {   // the newest dark and color frames, as they are
+                        fh_ring_slot_t meta;
+                        const uint64_t n = ring.latest(i);
+                        const auto &c = ring.camera(i);
+                        if (n && ring.read(i, n, pixels[name + "#rec"], &meta))
+                            frames.push_back({name, pixels[name + "#rec"].data(), c.width, c.height, meta.capture_ns,
+                                              meta.dqbuf_ns});
+                    }
+                    rec->add(frames);
+                } else {
+                    const size_t n = rec->written(), d = rec->dropped();
+                    rec.reset();   // writes out what's queued
+                    std::printf("recording done: %zu sets, %zu dropped\n", n, d);
+                    std::fflush(stdout);
+                    if (!track) break;
                 }
-                rec->add(frames);
-            } else {
-                const size_t n = rec->written(), d = rec->dropped();
-                rec.reset();   // writes out what's queued
-                std::printf("recording done: %zu sets, %zu dropped\n", n, d);
+            }
+            if (!track && rec && status > 0 && (mono_ns() - t_status) / 1e9 >= status) {
+                std::printf("%5.1fs  recorded %zu sets, dropped %zu\n", (mono_ns() - start) / 1e9, rec->written(), rec->dropped());
                 std::fflush(stdout);
-                if (!track) break;
+                t_status = mono_ns();
+            }
+            if (!track || tmin < next_ns) continue;   // not needed yet at the current rate
+            if (!use_mono) images.clear();            // colour only, driven by mono while recording
+            if (use_color && color_pair(raw_off, true, images)) ++color_steps;
+            if (images.empty()) continue;
+        } else {
+            // colour only: a new pair of colour frames
+            for (auto &[name, i] : index) {   // keep the mono cameras' delay current
+                fh_ring_slot_t meta;
+                const uint64_t n = ring.latest(i);
+                if (n && ring.meta(i, n, &meta)) {
+                    const double delay = double(int64_t(meta.dqbuf_ns) - (int64_t(meta.capture_ns) - raw_off));
+                    mono_delay_ns = mono_delay_ns < 0 ? delay : mono_delay_ns + 0.02 * (delay - mono_delay_ns);
+                }
+                break;
+            }
+            const uint64_t t = color_pair(raw_off, false, images);
+            if (!t) {
+                std::this_thread::sleep_for(std::chrono::milliseconds(2));
+                continue;
+            }
+            if (t < next_ns) {   // not needed yet: pass it by without copying
+                for (auto &[name, i] : color) last[name] = ring.latest(i);
+                continue;
+            }
+            if (!color_pair(raw_off, true, images)) continue;
+            tmin = t, ++color_steps;
+            for (auto &[name, i] : color) {
+                fh_ring_slot_t meta;
+                if (ring.meta(i, last[name], &meta)) dq = std::max(dq, meta.dqbuf_ns);
             }
         }
-        if (!track && rec && status > 0 && (mono_ns() - t_status) / 1e9 >= status) {
-            std::printf("%5.1fs  recorded %zu sets, dropped %zu\n", (mono_ns() - start) / 1e9, rec->written(), rec->dropped());
-            std::fflush(stdout);
-            t_status = mono_ns();
-        }
-        if (!track || tmin < next_ns) continue;   // not needed yet at the current rate
+
         const auto hands = tracker.step(images, int64_t(tmin));
-        const uint64_t capture = uint64_t(int64_t(tmin) - raw_minus_mono_ns());   // CLOCK_MONOTONIC
-        pinch.update(hands, tracker.views_now(), int64_t(capture));
-        // a pinch down or closing gets the full rate, even while the palm holds still
-        next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() ? 1 / 30.0 : 1.0) - 0.005) * 1e9);
-        if (publish) pub.write(hands, capture), gestures.write(pinch, capture);
-        for (const Pinch::Event &e : pinch.events) {
+        const uint64_t capture = uint64_t(int64_t(tmin) - raw_off);   // CLOCK_MONOTONIC
+        const std::vector<Seen> views = tracker.views_now();
+        grip.update(hands, views, int64_t(capture));
+        pinch.update(hands, views, int64_t(capture), grip.gripping());
+        // a gesture down or closing gets the full rate, even while the palm holds still
+        next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) -
+                                   0.005) * 1e9);
+        if (publish) pub.write(hands, capture), gestures.write(pinch, grip, capture);
+        for (const Pinch::Event &e : grip.events)
+            std::printf("grip  %s %-5s curl %.2f at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
+                        e.point[0], e.point[1], e.point[2]);
+        for (const Pinch::Event &e : pinch.events)
             std::printf("pinch %s %-5s d %.3f m at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
                         e.point[0], e.point[1], e.point[2]);
-            std::fflush(stdout);
-        }
         lat.push_back((mono_ns() - dq) / 1e6);
         hands_sum += double(hands.size());
         bool on_left = false, on_right = false;   // by where the wrist is, not the model's label
@@ -339,6 +571,11 @@ int main(int argc, char **argv) {
                         s.palm_batches ? s.palm_ms / s.palm_batches : 0, s.hand_calls,
                         s.hand_batches ? s.hand_ms / s.hand_batches : 0, s.sets ? s.step_ms / s.sets : 0,
                         lat.empty() ? 0 : lat[lat.size() / 2], resid_n ? resid_sum / resid_n : 0, 100 * (cpu1 - cpu0) / dt);
+            if (!color.empty())
+                std::printf("        cameras %s%s: colour frames at %.1f (bright at %.0f, dim under %.0f), ambient IR %.1f, "
+                            "%d steps with colour, colour placed %.1f ms after capture\n",
+                            cams_name(mode), switching ? " (auto)" : "", light.level, light.on, light.off, ambient(),
+                            color_steps, mono_delay_ns / 1e6);
             if (s.sets)
                 std::printf("        sets with a hand: left %2.0f%% right %2.0f%% both %2.0f%%  views lost %d, handoff misses %d, "
                             "dups %d, splits %d  hands new %d merged %d forgotten %d%s\n",
@@ -346,12 +583,16 @@ int main(int argc, char **argv) {
                             s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten,
                             !rec ? "" : ("  recorded " + std::to_string(rec->written()) + " dropped " +
                                          std::to_string(rec->dropped())).c_str());
-            std::printf("        pinches: left %u right %u (held back, palm down: %d %d)", pinch.side(0).begins,
-                        pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1]);
+            std::printf("        pinches: left %u right %u (held back, palm down: %d %d)  grips: left %u right %u",
+                        pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1],
+                        grip.side(0).begins, grip.side(1).begins);
             for (int k = 0; k < 2; ++k)
                 if (pinch.side(k).flags & FH_PINCH_TRACKED)
-                    std::printf("  %s %s d %.3f", k ? "right" : "left", pinch.side(k).flags & FH_PINCH_DOWN ? "DOWN" : "open",
-                                pinch.side(k).distance);
+                    std::printf("  %s %s d %.3f curl %.2f", k ? "right" : "left",
+                                grip.side(k).flags & FH_PINCH_DOWN    ? "GRIP"
+                                : pinch.side(k).flags & FH_PINCH_DOWN ? "PINCH"
+                                                                      : "open",
+                                pinch.side(k).distance, grip.curl[k]);
             std::printf("\n");
             for (const Hand *h : hands)
                 std::printf("        hand %d %-5s views %d wrist %+.3f %+.3f %+.3f m  scale %.2f  speed %.2f m/s\n", h->id,
@@ -361,12 +602,15 @@ int main(int argc, char **argv) {
             tracker.stats = Stats{};
             t_status = now, cpu0 = cpu1;
             lat.clear(), hands_sum = 0, resid_sum = 0, resid_n = 0, left_sets = right_sets = both_sets = 0;
+            color_steps = 0;
         }
     }
+    if (!color.empty()) unlink(want_file.c_str());
     if (publish) {
         pub.write({}, mono_ns());
         pinch.release(int64_t(mono_ns()));   // a drag in progress ends, as lost
-        gestures.write(pinch, mono_ns());
+        grip.release(int64_t(mono_ns()));
+        gestures.write(pinch, grip, mono_ns());
     }
     return 0;
 }

hands/track/pinch.cpp

@@ -24,8 +24,12 @@ void put3(float out[3], V3 v) {
 
 }  // namespace
 
-void Pinch::update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns) {
+void Pinch::update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns,
+                   const std::vector<int> &gripping) {
     events.clear();
+    auto grips = [&](int id) { return std::find(gripping.begin(), gripping.end(), id) != gripping.end(); };
+    for (int s = 0; s < 2; ++s)   // a grip took this pinch's hand: it's a drag now, not a click
+        if ((side_[s].flags & FH_PINCH_DOWN) && grips(follow_[s])) end(s, t_ns, true);
     // A hand a pinch is down on belongs to that side until it ends. The left/right call is a
     // running average of the model's, and when it flips mid-pinch the other side would take
     // the same hand and pinch too (4 times in the 2026-09-30 lit recording).
@@ -71,7 +75,9 @@ void Pinch::update(const std::vector<const Hand *> &hands, const std::vector<See
             // a close held back (palm down) has to open again before a pinch can begin, so
             // turning the hand with the fingers still closed doesn't start one
             if (d > p_.end_m) held_[s] = false;
-            if (d < p_.begin_m && !held_[s] && palm_down[s] > p_.palm_down_max) {
+            if (grips(h->id)) {
+                // closed: no pinch until it opens
+            } else if (d < p_.begin_m && !held_[s] && palm_down[s] > p_.palm_down_max) {
                 held_[s] = true;
                 ++held_back[s];
             } else if (d < p_.begin_m && !held_[s]) {
@@ -114,7 +120,130 @@ bool Pinch::engaged() const {
     return false;
 }
 
-bool GesturePublisher::open(const Pinch &pinch, std::string &err) {
+// ---------------------------------------------------------------------------- grip
+
+namespace {
+
+constexpr int kWrist = 0, kFingers[4][2] = {{5, 8}, {9, 12}, {13, 16}, {17, 20}};   // knuckle, tip
+
+// Each finger's curl (see GripParams): from the model's world landmarks averaged over the
+// hand's views this step, or the tracker's 3D points without any.
+bool curls(const Hand &h, const std::vector<Seen> &views, double out[4]) {
+    double sum[4] = {};
+    int n = 0;
+    for (const Seen &v : views) {
+        if (v.hand != h.id) continue;
+        auto p = [&](int i) { return V3{v.lm.world[i][0], v.lm.world[i][1], v.lm.world[i][2]}; };
+        for (int f = 0; f < 4; ++f) {
+            const double k = norm(p(kFingers[f][0]) - p(kWrist));
+            sum[f] += k > 1e-4 ? norm(p(kFingers[f][1]) - p(kWrist)) / k : 2;
+        }
+        ++n;
+    }
+    for (int f = 0; f < 4; ++f) {
+        if (n) {
+            out[f] = sum[f] / n;
+            continue;
+        }
+        const double k = norm(h.smooth[kFingers[f][0]] - h.smooth[kWrist]);
+        if (k < 1e-4) return false;
+        out[f] = norm(h.smooth[kFingers[f][1]] - h.smooth[kWrist]) / k;
+    }
+    return true;
+}
+
+V3 palm_centre(const Hand &h) {
+    return (h.smooth[0] + h.smooth[5] + h.smooth[9] + h.smooth[13] + h.smooth[17]) * 0.2;
+}
+
+}  // namespace
+
+void Grip::update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns) {
+    events.clear();
+    int taken[2] = {0, 0};
+    for (int s = 0; s < 2; ++s)
+        if (side_[s].flags & FH_PINCH_DOWN) taken[s] = follow_[s];
+    for (int s = 0; s < 2; ++s) {
+        fh_pinch_t &o = side_[s];
+        const bool down = o.flags & FH_PINCH_DOWN;
+        const Hand *h = nullptr;
+        for (const Hand *c : hands) {
+            if (down ? c->id != follow_[s] : c->right() != (s == 1) || c->id == taken[1 - s]) continue;
+            if (!h || c->frames > h->frames) h = c;
+        }
+        curl[s] = -1;
+        double f[4];
+        if (!h || !curls(*h, views, f)) {
+            o.flags &= ~FH_PINCH_TRACKED;
+            if (down && (t_ns - seen_ns_[s]) / 1e9 > p_.grace_s) end(s, t_ns, true);
+            continue;
+        }
+        seen_ns_[s] = t_ns;
+        const double mean = (f[0] + f[1] + f[2] + f[3]) / 4, most = std::max({f[0], f[1], f[2], f[3]});
+        curl[s] = mean;
+        const V3 point = palm_centre(*h);
+        o.flags |= FH_PINCH_TRACKED;
+        o.hand_id = uint32_t(h->id);
+        o.distance = float(mean);
+        o.strength = float(std::clamp((p_.end - mean) / (p_.end - p_.begin), 0.0, 1.0));
+        put3(o.point, point);
+        if (!down) {
+            if (mean > p_.end) open_ns_[s] = t_ns;
+            const bool ahead = -point[2] >= p_.min_ahead_m &&
+                               std::atan2(-point[1], -point[2]) * 180 / M_PI <= p_.max_down_deg;
+            if (most < p_.begin && ahead && open_ns_[s] && (t_ns - open_ns_[s]) / 1e9 <= p_.armed_s) {
+                o.flags = (o.flags | FH_PINCH_DOWN) & ~FH_PINCH_LOST;
+                ++o.begins;
+                o.begin_ns = uint64_t(t_ns);
+                put3(o.begin_point, point);
+                follow_[s] = h->id;
+                open_frames_[s] = 0;
+                events.push_back({s, "begin", t_ns, mean, point});
+            }
+        } else if (mean > p_.end) {
+            if (++open_frames_[s] >= p_.end_frames) {
+                end(s, t_ns, false);
+                open_ns_[s] = t_ns;
+            }
+        } else {
+            open_frames_[s] = 0;
+        }
+    }
+}
+
+void Grip::end(int s, int64_t t_ns, bool lost) {
+    fh_pinch_t &o = side_[s];
+    o.flags = (o.flags & ~FH_PINCH_DOWN) | (lost ? FH_PINCH_LOST : 0);
+    ++o.ends;
+    o.end_ns = uint64_t(t_ns);
+    follow_[s] = 0;
+    events.push_back({s, lost ? "lost" : "end", t_ns, o.distance, {o.point[0], o.point[1], o.point[2]}});
+}
+
+void Grip::release(int64_t t_ns) {
+    events.clear();
+    for (int s = 0; s < 2; ++s) {
+        side_[s].flags &= ~FH_PINCH_TRACKED;
+        if (side_[s].flags & FH_PINCH_DOWN) end(s, t_ns, true);
+    }
+}
+
+std::vector<int> Grip::gripping() const {
+    std::vector<int> out;
+    for (int s = 0; s < 2; ++s)
+        if (side_[s].flags & FH_PINCH_DOWN) out.push_back(follow_[s]);
+    return out;
+}
+
+bool Grip::engaged() const {
+    for (const fh_pinch_t &o : side_)
+        if ((o.flags & FH_PINCH_DOWN) || ((o.flags & FH_PINCH_TRACKED) && o.strength > 0.3f)) return true;
+    return false;
+}
+
+// ----------------------------------------------------------------------- publisher
+
+bool GesturePublisher::open(const Pinch &pinch, const Grip &grip, std::string &err) {
     const std::string path = run_dir() + "/gestures";
     const int fd = ::open(path.c_str(), O_RDWR | O_CREAT | O_NOFOLLOW | O_CLOEXEC, 0600);
     if (fd < 0 || ftruncate(fd, sizeof(fh_gestures_t)) < 0) return err = path + ": " + std::strerror(errno), false;
@@ -131,11 +260,13 @@ bool GesturePublisher::open(const Pinch &pinch, std::string &err) {
         out_->size = sizeof(fh_gestures_t);
     }
     seq_ = out_->seq / 2 + 1;
-    // a pinch the last tracker left down (it crashed) is over: count its end, as lost
+    // a gesture the last tracker left down (it crashed) is over: count its end, as lost
     __atomic_store_n(&out_->seq, 2 * ++seq_ - 1, __ATOMIC_RELAXED);
     __atomic_thread_fence(__ATOMIC_RELEASE);
-    for (fh_pinch_t &o : out_->pinch)
-        if (o.begins != o.ends) {
+    for (fh_pinch_t *slots : {out_->pinch, out_->grip})
+        for (int s = 0; s < 2; ++s) {
+            fh_pinch_t &o = slots[s];
+            if (o.begins == o.ends) continue;
             o.ends = o.begins;
             o.end_ns = mono_ns();
             o.flags = (o.flags & ~FH_PINCH_DOWN) | FH_PINCH_LOST;
@@ -143,22 +274,25 @@ bool GesturePublisher::open(const Pinch &pinch, std::string &err) {
     __atomic_store_n(&out_->seq, 2 * seq_, __ATOMIC_RELEASE);
     out_->begin_m = float(pinch.params().begin_m);
     out_->end_m = float(pinch.params().end_m);
+    out_->grip_begin = float(grip.params().begin);
+    out_->grip_end = float(grip.params().end);
     return true;
 }
 
-void GesturePublisher::write(const Pinch &pinch, uint64_t capture_ns) {
+void GesturePublisher::write(const Pinch &pinch, const Grip &grip, uint64_t capture_ns) {
     __atomic_store_n(&out_->seq, 2 * ++seq_ - 1, __ATOMIC_RELAXED);
     __atomic_thread_fence(__ATOMIC_RELEASE);
-    for (int s = 0; s < 2; ++s) {
-        // counters carry on from what's in the file (a restarted tracker starts its own at 0)
-        const fh_pinch_t &in = pinch.side(s);
-        fh_pinch_t &o = out_->pinch[s];
-        const uint32_t base_b = o.begins - last_begins_[s], base_e = o.ends - last_ends_[s];
-        o = in;
-        o.begins = base_b + in.begins;
-        o.ends = base_e + in.ends;
-        last_begins_[s] = in.begins, last_ends_[s] = in.ends;
-    }
+    for (int g = 0; g < 2; ++g)
+        for (int s = 0; s < 2; ++s) {
+            // counters carry on from what's in the file (a restarted tracker starts its own at 0)
+            const fh_pinch_t &in = g ? grip.side(s) : pinch.side(s);
+            fh_pinch_t &o = g ? out_->grip[s] : out_->pinch[s];
+            const uint32_t base_b = o.begins - last_begins_[g][s], base_e = o.ends - last_ends_[g][s];
+            o = in;
+            o.begins = base_b + in.begins;
+            o.ends = base_e + in.ends;
+            last_begins_[g][s] = in.begins, last_ends_[g][s] = in.ends;
+        }
     out_->capture_ns = capture_ns;
     out_->publish_ns = mono_ns();
     __atomic_store_n(&out_->seq, 2 * seq_, __ATOMIC_RELEASE);

hands/track/pinch.h

@@ -1,5 +1,6 @@
-// Pinch detection for input: look at something and pinch to click, pinch and move to drag.
-// Per side, from the tracker's hands after each step; published as fh_gestures.h.
+// Gesture detection for input: look at something and pinch to click it (pinch and move to
+// nudge the pointer first), or close the hand to press and drag it. Per side, from the
+// tracker's hands after each step; published as fh_gestures.h.
 #pragma once
 
 #include "tracker.h"
@@ -34,8 +35,10 @@ public:
     explicit Pinch(const PinchParams &p = {}) : p_(p) {}
     const PinchParams &params() const { return p_; }
     // After each processed set: the hands out of Tracker::step, the tracker's views (for
-    // the world landmarks) and the capture time.
-    void update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns);
+    // the world landmarks) and the capture time. Hands in `gripping` (their ids) are closed:
+    // no pinch begins on them, and one that's down on them ends, as lost.
+    void update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns,
+                const std::vector<int> &gripping = {});
     // Ends any pinch that's down (as lost), e.g. when the tracker stops.
     void release(int64_t t_ns);
     const fh_pinch_t &side(int s) const { return side_[s]; }   // 0 left, 1 right
@@ -66,14 +69,60 @@ private:
     bool held_[2] = {false, false};  // a close held back (palm down) that hasn't opened yet
 };
 
+struct GripParams {
+    // How curled a finger is: its tip's distance from the wrist over its knuckle's, from the
+    // model's world landmarks (so the hand's size doesn't matter). About 1.8-2.0 straight,
+    // 0.8-1.0 curled into a fist.
+    double begin = 1.2;       // every finger under this: the grip begins
+    double end = 1.45;        // their mean over this: it ends
+    int end_frames = 2;       // processed frames in a row past end before it ends
+    double grace_s = 0.25;    // a gripping hand lost this long ends its grip (FH_PINCH_LOST)
+    // A grip begins only on a hand seen open (mean over end) within this long: closing the
+    // hand is the gesture. A hand resting closed (in your lap, on a mouse) never grips.
+    double armed_s = 1.0;
+    // ...and only in front of you, where you'd hold a hand up to grab something: the palm no
+    // more than max_down_deg below straight ahead (head frame) and at least min_ahead_m in
+    // front of the eyes. Typing curls the fingers like a loose fist: in the 2026-09-30 lit
+    // recording, typing hands sat about 47 degrees down (14 false grips without this),
+    // deliberate pinches 5-15.
+    double max_down_deg = 35;
+    double min_ahead_m = 0.15;
+};
+
+// Grip (a closed hand) detection, per side like Pinch: press and drag.
+class Grip {
+public:
+    explicit Grip(const GripParams &p = {}) : p_(p) {}
+    const GripParams &params() const { return p_; }
+    void update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns);
+    void release(int64_t t_ns);
+    const fh_pinch_t &side(int s) const { return side_[s]; }
+    // The hands gripping now (for Pinch::update).
+    std::vector<int> gripping() const;
+    bool engaged() const;   // a grip is down or closing: worth tracking at the full rate
+
+    std::vector<Pinch::Event> events;
+    double curl[2] = {-1, -1};   // each side's hand: mean curl, for logs
+
+private:
+    void end(int s, int64_t t_ns, bool lost);
+    GripParams p_;
+    fh_pinch_t side_[2]{};
+    int follow_[2] = {0, 0};
+    int open_frames_[2] = {0, 0};
+    int64_t seen_ns_[2] = {0, 0};
+    int64_t open_ns_[2] = {0, 0};   // the side's hand was last seen open then
+};
+
 // Writes /run/user/UID/frametop-hands/gestures.
 class GesturePublisher {
 public:
-    bool open(const Pinch &pinch, std::string &err);
-    void write(const Pinch &pinch, uint64_t capture_ns);
+    bool open(const Pinch &pinch, const Grip &grip, std::string &err);
+    void write(const Pinch &pinch, const Grip &grip, uint64_t capture_ns);
 
 private:
     fh_gestures_t *out_ = nullptr;
     uint64_t seq_ = 0;
-    uint32_t last_begins_[2] = {0, 0}, last_ends_[2] = {0, 0};   // Pinch's counters last written
+    // the counters last written, per gesture (0 pinch, 1 grip) and side
+    uint32_t last_begins_[2][2] = {}, last_ends_[2][2] = {};
 };

hands/track/replay.cpp

@@ -15,6 +15,8 @@
 // --keep-presence P: landmark presence a tracked view needs to stay (default 0.5, as new ones).
 // --pinch-begin M, --pinch-end M, --pinch-triangulated, --pinch-palm-down MAX: the pinch detector (track/pinch.h);
 //             the timeline gets its begin/end/lost events and both distance measures per set.
+// --grip-begin R, --grip-end R: the grip detector (a closed hand; track/pinch.h); the timeline
+//             gets its events and each side's finger curl per set.
 // --cams mono|color|all: which cameras to track with (default mono). color and all need a
 //             recording made with ft-camd --with-color; --color-left NODE (color_video0 or
 //             color_video3) and --color-crop subtract|none say how its calibration maps
@@ -73,6 +75,7 @@ int main(int argc, char **argv) {
     Contrast palm_contrast, hand_contrast{Contrast::None};   // as ft-hands's
     double keep_presence = 0.5;   // landmark presence a tracked view needs to stay
     PinchParams pinch_params;
+    GripParams grip_params;
     std::string use = "mono", color_left = "color_video0", color_crop = "subtract";
     std::string timeline, depth, poses, models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
     for (int i = 2; i < argc; ++i) {
@@ -92,6 +95,8 @@ int main(int argc, char **argv) {
         else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
         else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
         else if (a == "--pinch-palm-down" && more) pinch_params.palm_down_max = std::atof(argv[++i]);
+        else if (a == "--grip-begin" && more) grip_params.begin = std::atof(argv[++i]);
+        else if (a == "--grip-end" && more) grip_params.end = std::atof(argv[++i]);
         else if (a == "--color-left" && more) color_left = argv[++i];
         else if (a == "--color-crop" && more) color_crop = argv[++i];
         else if (a == "--contrast" && more) {
@@ -155,6 +160,9 @@ int main(int argc, char **argv) {
     double pinch_begin_ts[2] = {0, 0};
     std::vector<double> pinch_len[2];   // seconds, per side
     int pinch_lost = 0;
+    Grip grip(grip_params);
+    double grip_begin_ts[2] = {0, 0};
+    std::vector<double> grip_len[2];
     do {
         std::map<std::string, Image> images;
         // the set's time: the mono cameras' when they're used (the color ones run on another
@@ -191,8 +199,17 @@ int main(int argc, char **argv) {
             busy_ms += ms;
             busy_until = t + uint64_t(ms * slow * 1e6) + 3'000'000;   // + the ring hand-off
             const std::vector<Seen> seen = tracker.views_now();
-            pinch.update(out, seen, int64_t(t));
-            next_ns = t + uint64_t((std::min(tracker.interval(), pinch.engaged() ? 1 / 30.0 : 1.0) - 0.005) * 1e9);
+            grip.update(out, seen, int64_t(t));
+            pinch.update(out, seen, int64_t(t), grip.gripping());
+            next_ns = t + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) - 0.005) * 1e9);
+            for (const Pinch::Event &e : grip.events) {
+                if (std::string(e.what) == "begin") grip_begin_ts[e.side] = ts;
+                else grip_len[e.side].push_back(ts - grip_begin_ts[e.side]);
+                if (tl) std::fprintf(tl, "%.3f grip %s %s curl %.2f point %+.3f %+.3f %+.3f hand %u\n", ts, e.side ? "R" : "L",
+                                     e.what, e.distance, e.point[0], e.point[1], e.point[2], grip.side(e.side).hand_id);
+            }
+            if (tl && (grip.curl[0] >= 0 || grip.curl[1] >= 0))
+                std::fprintf(tl, "%.3f curl L %.2f R %.2f\n", ts, grip.curl[0], grip.curl[1]);
             for (const Pinch::Event &e : pinch.events) {
                 if (std::string(e.what) == "begin") pinch_begin_ts[e.side] = ts;
                 else pinch_len[e.side].push_back(ts - pinch_begin_ts[e.side]), pinch_lost += std::string(e.what) == "lost";
@@ -333,6 +350,11 @@ int main(int argc, char **argv) {
                 pinch_len[0].size(), pinch_len[0].empty() ? 0 : pinch_len[0][pinch_len[0].size() / 2], pinch_len[1].size(),
                 pinch_len[1].empty() ? 0 : pinch_len[1][pinch_len[1].size() / 2], pinch_lost, pinch.held_back[0],
                 pinch.held_back[1]);
+    for (int k = 0; k < 2; ++k) std::sort(grip_len[k].begin(), grip_len[k].end());
+    std::printf("grips (curl under %.2f, open over %.2f): left %zu (median %.2f s), right %zu (median %.2f s)\n",
+                grip_params.begin, grip_params.end, grip_len[0].size(),
+                grip_len[0].empty() ? 0 : grip_len[0][grip_len[0].size() / 2], grip_len[1].size(),
+                grip_len[1].empty() ? 0 : grip_len[1][grip_len[1].size() / 2]);
     std::sort(jit_raw.begin(), jit_raw.end());
     std::sort(jit_sm.begin(), jit_sm.end());
     if (!jit_raw.empty())

hands/track/tracker.cpp

@@ -366,9 +366,14 @@ void Tracker::associate() {
 std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &images, int64_t t_ns) {
     const auto t_step = std::chrono::steady_clock::now();
     ++stats.sets;
-    std::vector<View> live;
-    for (View &v : views_)
-        if (images.count(v.cam->name)) live.push_back(v), live.back().fresh = false;
+    // Views in cameras without a frame in this set wait, as they are, for their camera's
+    // next one: the colour cameras run on their own clock, so a set can hold the mono
+    // cameras, the colour ones, or both (drop_camera ends them when a camera stops being used).
+    std::vector<View> live, waiting;
+    for (View &v : views_) {
+        (images.count(v.cam->name) ? live : waiting).push_back(v);
+        (images.count(v.cam->name) ? live : waiting).back().fresh = false;
+    }
 
     // 1. hand-over: give hands with too few views a crop in other cameras
     for (auto &[id, hand] : hands_) {
@@ -430,6 +435,7 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
         else ++stats.dups;
     }
     views_ = next;
+    views_.insert(views_.end(), waiting.begin(), waiting.end());
 
     // 3. search for missing hands
     std::set<int> tracked;
@@ -545,10 +551,16 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
 
 std::vector<Seen> Tracker::views_now() const {
     std::vector<Seen> out;
-    for (const View &v : views_) out.push_back({v.cam->name, v.hand, v.roi, v.lm, {}});
+    for (const View &v : views_)
+        if (v.fresh) out.push_back({v.cam->name, v.hand, v.roi, v.lm, {}});
     return out;
 }
 
+void Tracker::drop_camera(const std::string &name) {
+    views_.erase(std::remove_if(views_.begin(), views_.end(), [&](const View &v) { return v.cam->name == name; }),
+                 views_.end());
+}
+
 std::vector<Seen> Tracker::exhaustive(const std::map<std::string, Image> &images) {
     std::vector<Tile *> tiles;
     for (Tile &t : tiles_)

hands/track/tracker.h

@@ -82,7 +82,10 @@ public:
     double interval() const;
     Stats stats;
     size_t views() const { return views_.size(); }
-    std::vector<Seen> views_now() const;
+    std::vector<Seen> views_now() const;   // the views this step updated
+    // Forget this camera's views, when it stops being tracked with (the lighting switched
+    // cameras); views in cameras a set lacks otherwise wait for their next frame.
+    void drop_camera(const std::string &name);
     // Every search tile in every camera, then landmarks on every palm: slow; for checking
     // what the scheduler misses (ft-handreplay --oracle).
     std::vector<Seen> exhaustive(const std::map<std::string, Image> &images);

session/frametop.conf.example

@@ -36,6 +36,17 @@ GAZE_TRACKER=steam         # gaze service: steam = SteamVR's eye tracker | own =
 GAZE_EYE=auto              # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
 HANDS_SWAP_SIDES=0         # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
 HANDS_CPUS=5,6,7           # hand tracking: the CPUs its model threads run on
+HANDS_CAMERAS=auto         # hand tracking: auto (by the light) | mono (the IR cameras; also keeps ft-camd off the colour ones) | color | all
+HANDS_BRIGHT=all           # hand tracking, auto: the cameras in bright light, all | color
+HANDS_BRIGHT_ON=40         # hand tracking, auto: the colour frames' mean brightness (0-255) that counts as bright (a dim room reads about 9)
+HANDS_BRIGHT_OFF=25        # ... and back to the mono cameras under this
+HANDS_COLOR_LEFT=color_video0  # hand tracking: which colour camera is passthrough_left (hands/tools/check_color.py tells)
+HANDS_COLOR_CROP=subtract  # hand tracking: how the colour calibration's crop applies, subtract | none (check_color.py tells)
+POINTER_HANDS=1            # 1 = pinches click (on the release; hold and move to nudge the pointer first) and grips press and drag, with hand tracking
+POINTER_PINCH_GAIN=0.5     # hands: the pointer moves this times the pinching hand's angle (precision)
+POINTER_PINCH_DEADZONE=1.5 # hands: degrees the pinching hand moves before the pointer does (a tap's jitter)
+POINTER_GRIP_GAIN=1        # hands: the pointer moves this times the gripping hand's angle
+POINTER_GRIP_BELOW=0.3     # hands: grips beginning more than this (m) below the eyes are ignored (hands on a desk)
 META_DASHBOARD=0   # 1 = a Meta tap on a pass-through keyboard toggles the SteamVR dashboard (pointer mode only)
 SHARE_KEYS=0       # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen)
 DISPLAY_OFF_MIN=0  # minutes without use before frametop-power turns the displays off, even if the headset seems worn (0 = never)