From 067a03ce38ea13ee61ed0deb247bc09a9c9938f0 Mon Sep 17 00:00:00 2001 From: DeeJanuz <45082401+DeeJanuz@users.noreply.github.com> Date: Tue, 29 Sep 2026 22:36:32 -0600 Subject: [PATCH] Hand tracking for Frametop's hand cutouts fh-camd (camd/) borrows XRService's camera buffers and publishes the tracking cameras' frames to a shared ring. fh-tracker (trackd/) finds hands in them with MediaPipe's palm and landmark models on ncnn, triangulates them in 3D, and publishes them for ft-screens. fh-replay replays recordings offline. tracker/ is the earlier Python version; tools/ and probes/ hold the checks and experiments. As of this commit: crop contrast defaults to CLAHE for the palm search and plain crops for the landmarks, --swap-sides works around fh-camd naming the side cameras backwards after some XRService restarts (tools/check_sides.py detects it), and --record-only, --with-dark, --cpus and --keep-presence support the bright-light and CPU-placement tests. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 22 + camd/LICENSE.FrameEyeCameraFeed | 21 + camd/Makefile | 15 + camd/README.md | 56 ++ camd/camd.c | 926 +++++++++++++++++++++++++++++++ camd/fhring.h | 83 +++ camd/tp.c | 427 ++++++++++++++ camd/tp.h | 76 +++ camd/xrcams.c | 783 ++++++++++++++++++++++++++ camd/xrcams.h | 82 +++ include/fh_hands.h | 58 ++ models/ncnn/hand-int8.ncnn.bin | Bin 0 -> 1093108 bytes models/ncnn/hand-int8.ncnn.param | 79 +++ models/ncnn/hand.ncnn.bin | Bin 0 -> 2046732 bytes models/ncnn/hand.ncnn.param | 79 +++ models/ncnn/palm-int8.ncnn.bin | Bin 0 -> 1029796 bytes models/ncnn/palm-int8.ncnn.param | 151 +++++ models/ncnn/palm.ncnn.bin | Bin 0 -> 1949940 bytes models/ncnn/palm.ncnn.param | 151 +++++ tools/check_sides.py | 129 +++++ tools/convert_models.py | 81 +++ tools/make_int8_calib.py | 79 +++ tools/nettest_compare.py | 90 +++ tools/show_set.py | 120 ++++ trackd/Makefile | 25 + trackd/README.md | 66 +++ trackd/calib.cpp | 166 ++++++ trackd/calib.h | 31 ++ trackd/geom.h | 22 + trackd/io.cpp | 169 ++++++ trackd/io.h | 46 ++ trackd/main.cpp | 295 ++++++++++ trackd/nets.cpp | 255 +++++++++ trackd/nets.h | 66 +++ trackd/nettest.cpp | 75 +++ trackd/record.cpp | 101 ++++ trackd/record.h | 69 +++ trackd/replay.cpp | 240 ++++++++ trackd/tracker.cpp | 607 ++++++++++++++++++++ trackd/tracker.h | 129 +++++ tracker/calib.py | 151 +++++ tracker/models.py | 300 ++++++++++ 42 files changed, 6321 insertions(+) create mode 100644 .gitignore create mode 100644 camd/LICENSE.FrameEyeCameraFeed create mode 100644 camd/Makefile create mode 100644 camd/README.md create mode 100644 camd/camd.c create mode 100644 camd/fhring.h create mode 100644 camd/tp.c create mode 100644 camd/tp.h create mode 100644 camd/xrcams.c create mode 100644 camd/xrcams.h create mode 100644 include/fh_hands.h create mode 100644 models/ncnn/hand-int8.ncnn.bin create mode 100644 models/ncnn/hand-int8.ncnn.param create mode 100644 models/ncnn/hand.ncnn.bin create mode 100644 models/ncnn/hand.ncnn.param create mode 100644 models/ncnn/palm-int8.ncnn.bin create mode 100644 models/ncnn/palm-int8.ncnn.param create mode 100644 models/ncnn/palm.ncnn.bin create mode 100644 models/ncnn/palm.ncnn.param create mode 100644 tools/check_sides.py create mode 100644 tools/convert_models.py create mode 100644 tools/make_int8_calib.py create mode 100644 tools/nettest_compare.py create mode 100644 tools/show_set.py create mode 100644 trackd/Makefile create mode 100644 trackd/README.md create mode 100644 trackd/calib.cpp create mode 100644 trackd/calib.h create mode 100644 trackd/geom.h create mode 100644 trackd/io.cpp create mode 100644 trackd/io.h create mode 100644 trackd/main.cpp create mode 100644 trackd/nets.cpp create mode 100644 trackd/nets.h create mode 100644 trackd/nettest.cpp create mode 100644 trackd/record.cpp create mode 100644 trackd/record.h create mode 100644 trackd/replay.cpp create mode 100644 trackd/tracker.cpp create mode 100644 trackd/tracker.h create mode 100644 tracker/calib.py create mode 100644 tracker/models.py diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..6ab0749 --- /dev/null +++ b/.gitignore @@ -0,0 +1,22 @@ +# Recordings (tens of GB) and the Python venv +captures/ +.venv/ +__pycache__/ +# Upstream clones: ncnn (see trackd/README.md) and FrameEyeCameraFeed (MIT, adapted into camd/) +vendor/ +# Disassembly of Valve's vrclient.so, for reverse engineering only +re/*.dis +# Downloadable model sources (tools/convert_models.py); the converted ncnn models are kept +models/onnx/ +models/*.task +# Build output +camd/fh-camd +camd/fh-camprobe +trackd/fh-tracker +trackd/fh-replay +trackd/nettest +probes/fh-frametime +probes/mgrvt +probes/ptstate +probes/refprobe +probes/*.bin diff --git a/camd/LICENSE.FrameEyeCameraFeed b/camd/LICENSE.FrameEyeCameraFeed new file mode 100644 index 0000000..b9fa147 --- /dev/null +++ b/camd/LICENSE.FrameEyeCameraFeed @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Curtis English + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/camd/Makefile b/camd/Makefile new file mode 100644 index 0000000..6918b24 --- /dev/null +++ b/camd/Makefile @@ -0,0 +1,15 @@ +CFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter +LDLIBS = -lm + +all: fh-camprobe fh-camd + +fh-camprobe: camprobe.c tp.c xrcams.c tp.h xrcams.h + $(CC) $(CFLAGS) -o $@ camprobe.c tp.c xrcams.c $(LDLIBS) + +fh-camd: camd.c tp.c xrcams.c tp.h xrcams.h fhring.h + $(CC) $(CFLAGS) -o $@ camd.c tp.c xrcams.c $(LDLIBS) + +clean: + rm -f fh-camprobe fh-camd + +.PHONY: all clean diff --git a/camd/README.md b/camd/README.md new file mode 100644 index 0000000..3745b2e --- /dev/null +++ b/camd/README.md @@ -0,0 +1,56 @@ +# camd + +Root-side camera access for frame-hands: + +- `fh-camd`: the frame broker. It publishes the four IR tracking cameras to a shared-memory ring that the unprivileged tracker reads. +- `fh-camprobe`: a test tool that records timestamped frames from every camera, including the Arcturus color pair, with CSV logs. + +## How it gets frames + +XRService owns the headset cameras. Both tools borrow its DMA-BUFs read-only with `pidfd_getfd`, the same way FrameEyeCameraFeed does. They never touch XRService's V4L2 descriptors. + +Polling buffers for changes can catch a frame while the camera is still writing it. Instead, they listen to the `v4l2:v4l2_dqbuf` tracepoint, which fires when XRService takes a buffer. It gives the buffer index, the sequence number and the capture timestamp. + +The tools learn which DMA-BUF holds each V4L2 index by watching which buffer changes at each dequeue. + +- Right after XRService allocates its buffers, the mapping is allocation order. +- After XRService restarts streaming, the order is shuffled, and the mapping is learned index by index. +- The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart. +- `fh-camd` also re-maps an index on the fly when its buffer holds no new frame. + +## fh-camd + +``` +make +sudo ./fh-camd # runs until stopped or XRService exits +``` + +It needs root only to set up: to borrow the buffers (`ptrace_scope=1` blocks `pidfd_getfd`) and to open the root-only tracepoints. Then it drops to the invoking user for good; XRService itself runs as that user. It reads nothing from the ring's readers. + +Frames go to `/run/frame-hands/ir-ring`. The file is mode 0600 and owned by the user. It sits in a root-owned directory, so no other account can plant a file or link there. The layout is in `fhring.h`, and `tracker/ring.py` reads it. + +- Only complete, bright frames are published. The cameras alternate a normal exposure with a near-black one, so each camera gets 30 of its 60 fps. +- A copy torn by the camera overwriting the buffer is dropped. +- Each copy takes about 0.1 ms, and a cache sync about 0.15 ms. + +It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. Start it again to re-attach. + +## fh-camprobe + +Wear the headset (cameras only stream while it's worn), then run: + +``` +sudo ./fh-camprobe # records 15 s +sudo ./fh-camprobe --list # discovery and tracepoints only +``` + +Output goes to `~/Pictures/framecap/stereo-