diff --git a/docs/hands-migration.md b/docs/hands-migration.md index 9b34c7c..b4af2ff 100644 --- a/docs/hands-migration.md +++ b/docs/hands-migration.md @@ -14,10 +14,19 @@ Steps 1-5 are done: - Built in the dev container on the Frame, and on the 7i. - Checked without the headset: - `ft-handreplay` against frame-hands' `fh-replay`, both x86 with `--cost`, on the whole dim recording and the first 60 s of the bright one: identical summaries and byte-identical depth dumps. The Makefile's own ncnn build is included in that. - - `ft-ringplay` into `ft-hands` on the 7i tracked, pinched, and wrote `/run/user/UID/frametop/{hands,gestures}`. + - `ft-ringplay` into `ft-hands` on the 7i tracked, pinched, and wrote `/run/user/UID/frametop-hands/{hands,gestures}`. - On the Frame, ft-hands in the container finds the calibration through `/run/host/persist`, and ft-camd without its capabilities refuses with a clear message. -Next is step 6, with the user: `hands/run.sh install` (sudo setcap), then a desktop restart from this branch so ft-screens reads the new path. +Step 6 has started (2026-09-30 10:30): +- `hands/run.sh install` is done, and both services run from this worktree. +- The files moved to `/run/user/UID/frametop-hands/`, because `/run/user/UID/frametop` is the desktop session's own runtime folder, deleted at every desktop start. +- Until the desktop restarts from a build with this branch's ft-screens, the link `/run/user/UID/frame-hands -> frametop-hands` feeds the running one. It's tmpfs, so it's gone at reboot. + +Found in the headset: +- The side cameras were swapped (`HANDS_SWAP_SIDES=1`). +- The cutout copy shader lost resolution at `mediump` (now `highp`). +- Colour capture isn't reliable (see the README). +- Two pinch fixes: one hand no longer pinches both sides, and the palm-down limit stops typing pinches. ## What frame-hands is today @@ -66,8 +75,8 @@ Programs within 15 characters, `ft-` prefix; files under `frametop`: | `fh-camd` | `ft-camd` | | `fh-tracker` | `ft-hands` | | `fh-replay`, `fh-ringplay` | `ft-handreplay`, `ft-ringplay` | -| `/run/frame-hands/ir-ring` | `$XDG_RUNTIME_DIR/frametop/cam-ring` | -| `$XDG_RUNTIME_DIR/frame-hands/hands`, `gestures` | `$XDG_RUNTIME_DIR/frametop/hands`, `gestures` | +| `/run/frame-hands/ir-ring` | `/run/user/UID/frametop-hands/cam-ring` | +| `$XDG_RUNTIME_DIR/frame-hands/hands`, `gestures` | `/run/user/UID/frametop-hands/hands`, `gestures` | The source keeps its `fh_` identifiers and header names (`fh_hands.h`, `fh_gestures.h`, `fhring.h`), and the file formats keep their magic strings, so recordings and tools from frame-hands keep working. Programs, units and runtime paths change. diff --git a/docs/reference.md b/docs/reference.md index 745cfd3..228ce45 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -173,7 +173,7 @@ power/run.sh log Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker also detects pinches, for clicking where you look with the gaze pointer (not wired to the pointer yet). It's optional: `hands/run.sh install`, or the last step of `install.sh`. -- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. +- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. - `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches) next to the ring. - Both start and stop with SteamVR. `hands/run.sh status` and `hands/run.sh log` show how they're doing. - Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (after some SteamVR restarts the side cameras' names come out swapped, and hands land beside the holes; `hands/tools/check_sides.py --ring` tells) and `HANDS_CPUS`. diff --git a/hands/README.md b/hands/README.md index d7482b8..d0c2649 100644 --- a/hands/README.md +++ b/hands/README.md @@ -24,7 +24,7 @@ Settings in `~/.config/frametop.conf` (`FT_` in the environment overrides - `HANDS_SWAP_SIDES=1`: the two side cameras' names are swapped (see ft-camd below). Check with `tools/check_sides.py --ring`. - `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below). -Files, all in `/run/user/UID/frametop/` (private to the user): +Files, all in `/run/user/UID/frametop-hands/` (private to the user; not `/run/user/UID/frametop/`, which the desktop session deletes whenever it starts): | File | Written by | Layout | Read by | | --- | --- | --- | --- | @@ -58,6 +58,7 @@ Options: - `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands. - `--with-color`: also publish the two Arcturus colour cameras, flagged `FH_CAM_COLOR`. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232) and at most 30 fps (`--color-fps`; the cameras run at 60). Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core. - The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 colour cameras. +- Colour isn't reliable yet. In the lit-room test of 2026-09-30, the colour cameras kept losing their buffer mapping: 30 frames in a row looked unchanged, the camera relearned, and after 5 relearns ft-camd exited. Each relearn samples all 32 colour buffers, which also made the mono cameras miss frames. Runs with the headset idle (no hands, no cutouts) had none of this, and no half-size copies either, while the failing runs had many. So the passthrough compositor may be writing into the colour buffers while Room View shows. Whether a frame is new is judged on the luma rows only: the chroma after them hardly changes in a lit room. `FT_CAMD_DEBUG=1` prints, at each colour stale frame, how many sampled words changed in every candidate buffer. - `--sensor S`: only the mono cameras whose sensor name contains S. - `--status S`: a status line every S seconds (0: never). @@ -114,7 +115,9 @@ How good the depth is, measured from recordings (2026-09-30, `--depth` below): t ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gestures file. The layout, and how to read it without missing quick taps, is in `include/fh_gestures.h`. - A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost). -- The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. `--pinch-triangulated` uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. Typing still gave 2 pinches a minute, so a consumer should only act on a pinch while the gaze is on a target. +- The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. `--pinch-triangulated` uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. In the dim recording, typing still gave 2 pinches a minute before the palm check below. +- No pinch begins while the palm faces down (`--pinch-palm-down MAX`: the palm normal's share of the head's up axis, default 0.6; 1 turns it off), and a close held back that way has to open again before a pinch can begin. Typing curls the thumb onto the index. In the lit recording of 2026-09-30, typing on a keyboard in the lap began 23 pinches in about 2 minutes, all with the palm facing down (0.69-1.00), while the 26 deliberate ones read 0.00-0.50. The limit held back every typing pinch and none of the deliberate ones. Looking down tilts the head frame, which lowers the reading for a hand on a keyboard, so the consumer's gaze check stays the other guard. +- A hand a pinch is down on stays with that side until the pinch ends. The left/right call is a running average of the model's, and when it flipped mid-pinch, the other side took the same hand and both sides pinched at once. - The pinch point is midway between the thumb and index tips. A drag is the pinch point now, minus where it was when the pinch began, both turned into the room with the HMD pose at their capture times. - `tools/watch_gestures.py` prints begins, ends and drag offsets live, and `--distance` prints each hand's distance. diff --git a/hands/camd/camd.c b/hands/camd/camd.c index 6ce02dd..c6ace2a 100644 --- a/hands/camd/camd.c +++ b/hands/camd/camd.c @@ -6,7 +6,7 @@ * 2. borrows those buffers read-only with pidfd_getfd, * 3. opens the v4l2_dqbuf tracepoint (tp.c). * Then it gives up its privileges for good, creates the frame ring in the - * user's runtime folder (/run/user/UID/frametop/cam-ring, fhring.h), learns + * user's runtime folder (/run/user/UID/frametop-hands/cam-ring, fhring.h), learns * which buffer holds which V4L2 index, and copies each complete bright frame * into the ring. It exits when XRService exits or reallocates its buffers; * start it again (or let systemd) to re-attach. @@ -230,8 +230,11 @@ static void setup_camera(cam_t *c, xr_camera_t *cam, int pidfd) memset(c, 0, sizeof(*c)); c->cam = cam; xr_camera_layout(cam, &c->lay); - c->need = (size_t)c->lay.pitch * c->lay.rows; c->color = c->lay.fmt == XR_FMT_YUV420_10P; + /* Color: just the luma rows, not the chroma after them. Whether a frame is new is judged + * by the last rows sampled changing, and in a lit room the chroma has too little noise to + * change between frames, so every frame looked stale and the buffers kept being relearned. */ + c->need = (size_t)c->lay.pitch * (c->color ? c->lay.height : c->lay.rows); c->out_w = c->color ? c->lay.width / opt_color_scale : c->lay.width; c->out_h = c->color ? c->lay.height / opt_color_scale : c->lay.height; @@ -678,6 +681,23 @@ static void on_frame(cam_t *c, int64_t index, uint32_t seq, uint64_t ts, uint64_ if (found < 0 && !fresh(cur, c->samp[slot], &nchanged)) { buf_sync(c->fd[slot], DMA_BUF_SYNC_END); + if (c->color && getenv("FT_CAMD_DEBUG")) { + /* which candidate buffers changed since this camera last looked, whole and tail */ + static uint64_t d[NSAMP]; + fprintf(stderr, "stale %s t %.3f index %lld seq %u slot %d changed %d:", c->slug, evtime / 1e9, + (long long)index, seq, slot, nchanged); + for (int s = 0; s < c->nslots; s++) { + int n, tail = 0; + buf_sync(c->fd[s], DMA_BUF_SYNC_START); + sample_words(c->map[s], c->need, d); + buf_sync(c->fd[s], DMA_BUF_SYNC_END); + fresh(d, c->samp[s], &n); + for (int i = NSAMP - NSAMP / 8; i < NSAMP; i++) + tail += d[i] != c->samp[s][i]; + fprintf(stderr, " %d/%d", n, tail); + } + fprintf(stderr, "\n"); + } c->stale++; c->rc->dropped++; if (++c->stale_run >= STALE_RELEARN) { @@ -783,7 +803,7 @@ static char ring_dir[64], ring_file[96]; /* Created once privileges are gone, as the user, in a folder only the user can write. */ static uint8_t *ring_create(size_t *len_out) { - snprintf(ring_dir, sizeof(ring_dir), "/run/user/%d/frametop", (int)getuid()); + snprintf(ring_dir, sizeof(ring_dir), "/run/user/%d/frametop-hands", (int)getuid()); snprintf(ring_file, sizeof(ring_file), "/run/user/%d/" FH_RING_NAME, (int)getuid()); if (mkdir(ring_dir, 0700) < 0 && errno != EEXIST) diff --git a/hands/camd/fhring.h b/hands/camd/fhring.h index e8908e1..7fc48b4 100644 --- a/hands/camd/fhring.h +++ b/hands/camd/fhring.h @@ -1,7 +1,7 @@ /* * fhring - the shared-memory frame ring ft-camd writes and trackers read. * - * One file, /run/user/UID/frametop/cam-ring (FH_RING_NAME in the user's runtime + * One file, /run/user/UID/frametop-hands/cam-ring (FH_RING_NAME in the user's runtime * folder; the folder is private to the user), holds a header, then for each camera * a few slots, each a slot header followed by the image rows packed tightly * (stride == width for 8-bit mono). Only complete, bright frames are published. @@ -25,7 +25,7 @@ #define FH_RING_VERSION 1 #define FH_RING_MAX_CAMS 8 #define FH_RING_SLOTS 4 -#define FH_RING_NAME "frametop/cam-ring" /* in /run/user/UID */ +#define FH_RING_NAME "frametop-hands/cam-ring" /* in /run/user/UID */ enum { FH_FMT_GREY8 = 0, diff --git a/hands/frametop-hands.service b/hands/frametop-hands.service index 4596d6d..0166b68 100644 --- a/hands/frametop-hands.service +++ b/hands/frametop-hands.service @@ -8,7 +8,7 @@ PartOf=steamvr.service [Service] # In the dev container (it's built against Fedora's libraries). It reads ft-camd's ring and -# writes /run/user/UID/frametop/hands and gestures. Settings: HANDS_* in ~/.config/frametop.conf. +# writes /run/user/UID/frametop-hands/hands and gestures. Settings: HANDS_* in ~/.config/frametop.conf. ExecStartPre=-@REPO@/scripts/container-up.sh ExecStartPre=-/usr/bin/pkill -x ft-hands ExecStart=%h/.local/bin/distrobox enter dev -- @REPO@/hands/build/ft-hands --status 60 diff --git a/hands/include/fh_gestures.h b/hands/include/fh_gestures.h index d355663..bb198dd 100644 --- a/hands/include/fh_gestures.h +++ b/hands/include/fh_gestures.h @@ -1,7 +1,7 @@ /* * fh_gestures - pinch state ft-hands publishes for input: look at something and pinch to * click it, pinch and move to drag (the Vision Pro model, with the eye tracker doing the - * looking). /run/user/UID/frametop/gestures, next to the hands + * looking). /run/user/UID/frametop-hands/gestures, next to the hands * file, with the same sequence lock (read seq, copy, read seq again; use the copy only if * both reads are the same even number) and the same frame: metres in the head frame at * capture time, OpenVR's HMD frame (+x right, +y up, -z forward). diff --git a/hands/include/fh_hands.h b/hands/include/fh_hands.h index ee80acb..4637b9f 100644 --- a/hands/include/fh_hands.h +++ b/hands/include/fh_hands.h @@ -1,6 +1,6 @@ /* * fh_hands - the tracked-hands file ft-hands publishes for ft-screens' hand cutouts - * (/run/user/UID/frametop/hands, directory mode 0700), rewritten in place + * (/run/user/UID/frametop-hands/hands, directory mode 0700), rewritten in place * under a sequence lock: read seq, copy, read seq again; use the copy only if * both reads are the same even number. * diff --git a/hands/run.sh b/hands/run.sh index 6b25a5c..45d11f5 100755 --- a/hands/run.sh +++ b/hands/run.sh @@ -28,9 +28,13 @@ sudo_run() { printf '%s\n' "$pw" | on_frame "sudo -S -p '' bash -c $(printf %q "$1")" } -set_caps() { +set_caps() { # only when missing: a rebuild clears them, a reinstall doesn't local bin bin=$(printf %q "$FRAME_REPO/hands/build/ft-camd") + if on_frame "getcap $bin | grep -q cap_sys_ptrace"; then + echo "ft-camd has its capabilities" + return + fi sudo_run "setcap $caps $bin && getcap $bin" } diff --git a/hands/tools/ring.py b/hands/tools/ring.py index 8cf56e6..bb56b98 100644 --- a/hands/tools/ring.py +++ b/hands/tools/ring.py @@ -6,7 +6,7 @@ import time import numpy as np -RING_FILE = '/run/user/%d/frametop/cam-ring' % os.getuid() +RING_FILE = '/run/user/%d/frametop-hands/cam-ring' % os.getuid() MAGIC = b'FHRING01' HDR = struct.Struct('<8sIIIIQqQ16x') # 64 bytes CAM = struct.Struct('<32s32siIIIIIQQQQQ32x') # 160 bytes diff --git a/hands/tools/watch_gestures.py b/hands/tools/watch_gestures.py index 2090d86..8a14b90 100644 --- a/hands/tools/watch_gestures.py +++ b/hands/tools/watch_gestures.py @@ -22,7 +22,7 @@ SIDES = ('left ', 'right') def path(): - return '/run/user/%d/frametop/gestures' % os.getuid() + return '/run/user/%d/frametop-hands/gestures' % os.getuid() def read(m): diff --git a/hands/track/io.cpp b/hands/track/io.cpp index 2e31569..c03745b 100644 --- a/hands/track/io.cpp +++ b/hands/track/io.cpp @@ -114,7 +114,7 @@ void put(fh_capsule_t *caps, uint32_t &n, V3 a, V3 b, double ra, double rb) { } // namespace std::string run_dir() { - const std::string dir = "/run/user/" + std::to_string(getuid()) + "/frametop"; + const std::string dir = "/run/user/" + std::to_string(getuid()) + "/frametop-hands"; mkdir(dir.c_str(), 0700); return dir; } diff --git a/hands/track/io.h b/hands/track/io.h index dc8f078..b9ed618 100644 --- a/hands/track/io.h +++ b/hands/track/io.h @@ -45,7 +45,8 @@ private: uint64_t mono_ns(); int64_t raw_minus_mono_ns(); // camera timestamps are CLOCK_MONOTONIC_RAW -// /run/user/UID/frametop, created private to the user if it's missing: where ft-camd's ring +// /run/user/UID/frametop-hands, created private to the user if it's missing: where ft-camd's ring // (FH_RING_NAME) and the hands and gestures files live. Not $XDG_RUNTIME_DIR: a terminal in -// the Frametop desktop has a private one of its own. +// the Frametop desktop has a private one of its own. And not /run/user/UID/frametop: that is +// the desktop session's private runtime folder, which it deletes whenever it starts. std::string run_dir(); diff --git a/hands/track/main.cpp b/hands/track/main.cpp index 3bd6da3..49549a8 100644 --- a/hands/track/main.cpp +++ b/hands/track/main.cpp @@ -123,6 +123,7 @@ int main(int argc, char **argv) { else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]); else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]); else if (a == "--pinch-triangulated") pinch_params.triangulated = true; + else if (a == "--pinch-palm-down" && more) pinch_params.palm_down_max = std::atof(argv[++i]); else if (a == "--swap-sides") swap_sides = true; else if (a == "--record-only") track = publish = false; else if (a == "--ring" && more) ring_path = argv[++i]; @@ -140,7 +141,7 @@ int main(int argc, char **argv) { std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n" " [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n" " [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n" - " [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated]\n" + " [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (0.6)\n" " [--contrast MODE|PALM/HAND] (clahe[:CLIP], none, stretch; default clahe:2/none)\n" "Recording saves every frame set for S seconds (120) to DIR/sets.bin, for ft-handreplay; SIGUSR1\n" "starts one in ~/.local/share/frametop/hands/rec-