diff --git a/docs/hands-migration.md b/docs/hands-migration.md index 63cb2e4..9b34c7c 100644 --- a/docs/hands-migration.md +++ b/docs/hands-migration.md @@ -4,6 +4,21 @@ Hand tracking from the headset's own cameras has been built as a separate projec Builds from this worktree must sync to their own folder on the Frame, never `~/dev/frametop`: run every script with `FRAME_REPO=/home/steamos/dev/frametop-hands`. +## Status (2026-09-30) + +Steps 1-5 are done: + +- frame-hands' pending work was committed there (6c63c9e). +- Its filtered history was merged under `hands/` (1a76d15), then laid out (`trackd/` to `track/`). +- The renames, the Frametop paths, and ft-camd's file capabilities are done. So are `hands/Makefile`, `build.sh`, `run.sh`, the two units, the README, the settings, the installer step, and ft-screens on the shared header. +- Built in the dev container on the Frame, and on the 7i. +- Checked without the headset: + - `ft-handreplay` against frame-hands' `fh-replay`, both x86 with `--cost`, on the whole dim recording and the first 60 s of the bright one: identical summaries and byte-identical depth dumps. The Makefile's own ncnn build is included in that. + - `ft-ringplay` into `ft-hands` on the 7i tracked, pinched, and wrote `/run/user/UID/frametop/{hands,gestures}`. + - On the Frame, ft-hands in the container finds the calibration through `/run/host/persist`, and ft-camd without its capabilities refuses with a clear message. + +Next is step 6, with the user: `hands/run.sh install` (sudo setcap), then a desktop restart from this branch so ft-screens reads the new path. + ## What frame-hands is today | Part | What it is | Size | @@ -34,9 +49,10 @@ hands/ frametop-hands.service include/ # fhring.h, fh_hands.h, fh_gestures.h: shared with screens/ and pointer/ camd/ # ft-camd: camd.c tp.c xrcams.c, LICENSE.FrameEyeCameraFeed - track/ # ft-hands: tracker, nets, calib, pinch, publish, record + track/ # ft-hands: tracker, nets, calib, pinch, publish, record; replay.cpp + # (ft-handreplay) and ringplay.cpp (ft-ringplay) for recordings models/ # the ncnn models, with NOTICE (Apache-2.0, MediaPipe / OpenCV Zoo) - tools/ # ft-handreplay, ft-ringplay, the Python checks, watch_gestures + tools/ # the Python checks, watch_gestures, depth_report, calib.py, ring.py ``` Left behind in frame-hands, which stays as the lab: the recordings, the Python prototype (the tools that need `calib.py` or `models.py` get a trimmed copy in `hands/tools/`), `probes/`, `notes/`, `re/`, `shim/`, `camprobe`, and `vendor/`. The reverse-engineering notes don't belong in a public repo, and recordings are images of the user's hands and room, so they never go into git. @@ -59,7 +75,7 @@ The source keeps its `fh_` identifiers and header names (`fh_hands.h`, `fh_gestu - `hands/build.sh` builds in the dev container through `scripts/frame.sh --build`, into `hands/build/`, like the other components. `FRAME_BUILDER=pc` can take the ncnn build. - ncnn: fetched at a pinned tag (20260526, as now) into `hands/build/ncnn` and built once, the way `screens/build.sh` fetches the OpenVR header, with frame-hands' options so results match. `NCNN=` points the build at an existing install instead. Every net runs single-threaded (`num_threads = 1`), with the tracker spreading nets over its own pinned threads, so OpenMP could go later. -- ft-hands runs in the dev container like ft-pointer and ft-powerd (`distrobox enter dev --`, after `scripts/container-up.sh`). Today's fh-tracker runs on the host and works only because the host happens to have the same `libjsoncpp.so.25` and libgomp as the container. The calibration moves from `/persist` to `/run/host/persist` inside the container. calib.cpp already takes a prefix for this (`FRAME_JOB_DEVICE_ROOT`, to be renamed). +- ft-hands runs in the dev container like ft-pointer and ft-powerd (`distrobox enter dev --`, after `scripts/container-up.sh`). Today's fh-tracker runs on the host and works only because the host happens to have the same `libjsoncpp.so.25` and libgomp as the container. Inside the container the calibration is at `/run/host/persist`, and calib.cpp (and `tools/calib.py`) fall back to it when `/persist` isn't there. - ft-camd has to run on the host (below), so it's linked statically (only libc and libm; `glibc-static` goes into `setup/dev-container.sh`). The host has an older glibc than the container. ## Running it @@ -73,9 +89,9 @@ Either way the password is needed once at install, through the same `sudo -S` pa **ft-hands** is a user unit, `frametop-hands.service`: after `frametop-camd.service`, `PartOf=steamvr.service`, nice 5, model threads on CPUs 5-7 (measured best on 2026-09-29). -**Settings** in `~/.config/frametop.conf`: `HANDS=0|1` (off by default until it's ready for others), `HANDS_CPUS=5,6,7`, `HANDS_COLOR=0`. Later, a switch in Frametop Display Settings. +**Settings** in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES=1` and `HANDS_CPUS=5,6,7`, read by ft-hands. It's on while its services are installed (`hands/run.sh install`, `uninstall`), so there's no `HANDS` switch. There's no setting for colour yet. Later, a switch in Frametop Display Settings. -**Installer:** a step in `install.sh` that asks first, because it needs sudo. +**Installer:** an optional last step in `install.sh`, off by default, which asks first because it needs sudo. ## Interfaces diff --git a/docs/reference.md b/docs/reference.md index 4caabc5..745cfd3 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -169,6 +169,17 @@ power/run.sh off | on # the displays off now, or back on power/run.sh log ``` +## Hand tracking (experimental) + +Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker also detects pinches, for clicking where you look with the gaze pointer (not wired to the pointer yet). It's optional: `hands/run.sh install`, or the last step of `install.sh`. + +- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. +- `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches) next to the ring. +- Both start and stop with SteamVR. `hands/run.sh status` and `hands/run.sh log` show how they're doing. +- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (after some SteamVR restarts the side cameras' names come out swapped, and hands land beside the holes; `hands/tools/check_sides.py --ring` tells) and `HANDS_CPUS`. + +Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md). + ## Remote desktop over VNC With `REMOTE=1` in the config (`desktops.sh remote on`), the desktop is also served over VNC, for RealVNC Viewer or macOS Screen Sharing. `desktops.sh remote info` prints the address and password. diff --git a/hands/.gitignore b/hands/.gitignore index e699c3f..96e366a 100644 --- a/hands/.gitignore +++ b/hands/.gitignore @@ -1,23 +1,3 @@ -# Recordings (tens of GB) and the Python venv -captures/ -.venv/ -__pycache__/ -# Upstream clones: ncnn (see trackd/README.md) and FrameEyeCameraFeed (MIT, adapted into camd/) -vendor/ -# Disassembly of Valve's vrclient.so, for reverse engineering only -re/*.dis -# Downloadable model sources (tools/convert_models.py); the converted ncnn models are kept +# Model sources that tools/convert_models.py downloads; the converted ncnn models are kept models/onnx/ models/*.task -# Build output -camd/fh-camd -camd/fh-camprobe -trackd/fh-tracker -trackd/fh-replay -trackd/fh-ringplay -trackd/nettest -probes/fh-frametime -probes/mgrvt -probes/ptstate -probes/refprobe -probes/*.bin diff --git a/hands/Makefile b/hands/Makefile new file mode 100644 index 0000000..6130d8c --- /dev/null +++ b/hands/Makefile @@ -0,0 +1,49 @@ +# Hand tracking, built into build/ (hands/build.sh runs this in the dev container): +# make ft-camd (camd/: runs on the host, so linked statically) and ft-hands (track/) +# make tools ft-handreplay and ft-ringplay, for recordings +# The first build fetches ncnn (NCNN_TAG) and builds it into build/ncnn, which takes a few +# minutes. NCNN=DIR uses an ncnn install already built instead. +NCNN_TAG = 20260526 +NCNN ?= build/ncnn/install +CFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter +CXXFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter -Wno-psabi +CXXFLAGS += -std=c++17 -fopenmp -I$(NCNN)/include/ncnn +LDLIBS = $(NCNN)/lib/libncnn.a -ljsoncpp -fopenmp -lpthread + +CAMD = camd/camd.c camd/tp.c camd/xrcams.c +TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp +HDR = $(wildcard track/*.h) camd/fhring.h include/fh_hands.h include/fh_gestures.h + +all: build/ft-camd build/ft-hands +tools: build/ft-handreplay build/ft-ringplay + +build/ft-camd: $(CAMD) camd/tp.h camd/xrcams.h camd/fhring.h + @mkdir -p build + $(CC) $(CFLAGS) -static -o $@ $(CAMD) -lm + +build/ft-hands: track/main.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a + @mkdir -p build + $(CXX) $(CXXFLAGS) -o $@ track/main.cpp $(TRACK) $(LDLIBS) + +build/ft-handreplay: track/replay.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a + @mkdir -p build + $(CXX) $(CXXFLAGS) -o $@ track/replay.cpp $(TRACK) $(LDLIBS) + +build/ft-ringplay: track/ringplay.cpp track/record.h camd/fhring.h + @mkdir -p build + $(CXX) $(CXXFLAGS) -o $@ track/ringplay.cpp + +# ncnn as frame-hands built it (the models were converted and quantized for it), minus its tools +build/ncnn/install/lib/libncnn.a: + rm -rf build/ncnn && mkdir -p build/ncnn + git clone -q --depth 1 --branch $(NCNN_TAG) -c advice.detachedHead=false https://github.com/Tencent/ncnn.git build/ncnn/src + cmake -S build/ncnn/src -B build/ncnn/build -G Ninja -Wno-dev -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=$(CURDIR)/build/ncnn/install -DCMAKE_INSTALL_LIBDIR=lib -DNCNN_VULKAN=OFF \ + -DNCNN_OPENMP=ON -DNCNN_INT8=ON -DNCNN_SIMPLEOCV=ON -DNCNN_BUILD_TOOLS=OFF -DNCNN_BUILD_EXAMPLES=OFF \ + -DNCNN_BUILD_BENCHMARK=OFF -DNCNN_BUILD_TESTS=OFF -DNCNN_PYTHON=OFF > build/ncnn/cmake.log + cmake --build build/ncnn/build --target install > build/ncnn/build.log + +clean: + rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay + +.PHONY: all tools clean diff --git a/hands/README.md b/hands/README.md new file mode 100644 index 0000000..d7482b8 --- /dev/null +++ b/hands/README.md @@ -0,0 +1,156 @@ +# Hands (experimental) + +Hand tracking from the headset's own cameras. It serves two things in Frametop: + +- **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro. +- **Pinches:** look at something and pinch to click it, pinch and move to drag, with the eye tracker doing the looking (`gaze/`). The tracker publishes the pinches. The pointer helper doesn't read them yet. + +Two programs, each a user service that starts and stops with SteamVR: + +- `ft-camd` (`camd/`, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host. +- `ft-hands` (`track/`, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container. + +``` +hands/run.sh install # build, give ft-camd its capabilities (sudo, once per build), enable +hands/run.sh status # the services, and ft-hands' last status lines +hands/run.sh log [lines] +hands/run.sh restart # after changing a setting +hands/run.sh caps # after rebuilding ft-camd (a rebuild clears its capabilities) +hands/run.sh uninstall +``` + +Settings in `~/.config/frametop.conf` (`FT_` in the environment overrides them): + +- `HANDS_SWAP_SIDES=1`: the two side cameras' names are swapped (see ft-camd below). Check with `tools/check_sides.py --ring`. +- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below). + +Files, all in `/run/user/UID/frametop/` (private to the user): + +| File | Written by | Layout | Read by | +| --- | --- | --- | --- | +| `cam-ring` | ft-camd | `camd/fhring.h` | ft-hands, `tools/ring.py` | +| `hands` | ft-hands | `include/fh_hands.h` | ft-screens (`screens/handcut.cpp`) | +| `gestures` | ft-hands | `include/fh_gestures.h` | `tools/watch_gestures.py`; the pointer helper, later | + +The source keeps the `fh_` names and magic strings of frame-hands, where this was developed (`~/Desktop/Projects/frame-hands` on the developer's Frame, which keeps the recordings, probes and Python prototype). So its recordings and tools still work. + +## ft-camd + +XRService owns the headset cameras. ft-camd borrows its DMA-BUFs read-only with `pidfd_getfd`, the same way FrameEyeCameraFeed does. It never touches XRService's V4L2 descriptors. `discovery` in `camd/xrcams.c` is adapted from FrameEyeCameraFeed (MIT, see `camd/LICENSE.FrameEyeCameraFeed`). + +Polling buffers for changes can catch a frame while the camera is still writing it. Instead, ft-camd listens to the `v4l2:v4l2_dqbuf` tracepoint, which fires when XRService takes a buffer. It gives the buffer index, the sequence number and the capture timestamp. ft-camd learns which DMA-BUF holds each V4L2 index by watching which buffer changes at each dequeue: + +- Right after XRService allocates its buffers, the mapping is allocation order. +- After XRService restarts streaming, the order is shuffled, and the mapping is learned index by index. +- The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart. +- It also re-maps an index on the fly when its buffer holds no new frame. + +**Privileges.** Setting up needs three things. `pidfd_getfd` on XRService needs `CAP_SYS_PTRACE`, because the Frame has `ptrace_scope=1`. The system-wide tracepoint needs `CAP_PERFMON`, because `perf_event_paranoid` is 2. Its format files are root-only, which needs `CAP_DAC_READ_SEARCH`. `hands/run.sh install` gives the binary those capabilities with `sudo setcap`. ft-camd drops them all once it has set up, before it reads a frame, and then runs as you. XRService runs as you too. It also runs under `sudo`, for trying it by hand, and then drops to the user who ran sudo. It reads nothing from the ring's readers. + +The ring is mode 0600, in a folder only you can write. Frame handling: + +- Only complete, bright frames are published. The cameras alternate a normal exposure with a near-black one, so each camera gets 30 of its 60 fps. +- A copy torn by the camera overwriting the buffer is dropped. +- Each copy takes about 0.1 ms, and a cache sync about 0.15 ms. + +Options: + +- `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands. +- `--with-color`: also publish the two Arcturus colour cameras, flagged `FH_CAM_COLOR`. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232) and at most 30 fps (`--color-fps`; the cameras run at 60). Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core. +- The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 colour cameras. +- `--sensor S`: only the mono cameras whose sensor name contains S. +- `--status S`: a status line every S seconds (0: never). + +It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. The service starts it again, and it attaches to the new buffers. + +**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. With the headset on, looking at a room with some texture, `tools/check_sides.py --ring` says whether the names are right (exit 0), swapped (exit 3), or it can't tell (exit 2). When they're swapped, set `HANDS_SWAP_SIDES=1`. The colour cameras are video3 (`arcimx616 0-0010`) and video0 (`0-001a`); which of them is `passthrough_left` in the module's calibration is for `tools/check_color.py` to settle, on a recording with texture in view. + +## ft-hands + +``` +hands/build/ft-hands # status every 5 s; Ctrl+C to stop +hands/build/ft-hands --int8 # the 8-bit models (models/ncnn/*-int8.ncnn.*) +``` + +Run it in the dev container (`distrobox enter dev -- ...`). It reads the factory calibration from `/persist` (`/run/host/persist` in the container). + +Options: + +- `--threads N`: model threads, pinned to the `--cpus` list. Default 3. +- `--cpus LIST`: CPUs for the model threads and the main loop. Default `5,6,7` (`HANDS_CPUS`). SteamOS starts user processes on CPUs 0-4, and XRService's head tracking runs on 2-3. With the headset on, a step took 8.4 ms on 5-7 against 13.2 ms on 2-4, and SteamVR's frame timing didn't change (2026-09-29, three rounds of the same replayed frames). +- `--contrast MODE` or `PALM/HAND`: how crops are equalized before the models see them: `clahe[:CLIP]`, `none`, or `stretch` (1st-99th percentile). Default `clahe:2/none`. In the dim recording, CLAHE let the palm search find about 10% more hands, but it made the landmarks jitter more (published median 6.9 mm, against 6.0 mm with plain landmark crops). +- `--swap-sides`: swap the two side cameras (`HANDS_SWAP_SIDES`, see ft-camd). +- `--seconds N`: stop after N seconds. +- `--status S`: how often to print status, in seconds. +- `--models DIR`: where the models are. +- `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs. +- `--no-publish`: don't write the hands and gestures files. +- `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-