diff --git a/hands/.gitignore b/hands/.gitignore new file mode 100644 index 0000000..e699c3f --- /dev/null +++ b/hands/.gitignore @@ -0,0 +1,23 @@ +# Recordings (tens of GB) and the Python venv +captures/ +.venv/ +__pycache__/ +# Upstream clones: ncnn (see trackd/README.md) and FrameEyeCameraFeed (MIT, adapted into camd/) +vendor/ +# Disassembly of Valve's vrclient.so, for reverse engineering only +re/*.dis +# Downloadable model sources (tools/convert_models.py); the converted ncnn models are kept +models/onnx/ +models/*.task +# Build output +camd/fh-camd +camd/fh-camprobe +trackd/fh-tracker +trackd/fh-replay +trackd/fh-ringplay +trackd/nettest +probes/fh-frametime +probes/mgrvt +probes/ptstate +probes/refprobe +probes/*.bin diff --git a/hands/camd/LICENSE.FrameEyeCameraFeed b/hands/camd/LICENSE.FrameEyeCameraFeed new file mode 100644 index 0000000..b9fa147 --- /dev/null +++ b/hands/camd/LICENSE.FrameEyeCameraFeed @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Curtis English + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/hands/camd/Makefile b/hands/camd/Makefile new file mode 100644 index 0000000..6918b24 --- /dev/null +++ b/hands/camd/Makefile @@ -0,0 +1,15 @@ +CFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter +LDLIBS = -lm + +all: fh-camprobe fh-camd + +fh-camprobe: camprobe.c tp.c xrcams.c tp.h xrcams.h + $(CC) $(CFLAGS) -o $@ camprobe.c tp.c xrcams.c $(LDLIBS) + +fh-camd: camd.c tp.c xrcams.c tp.h xrcams.h fhring.h + $(CC) $(CFLAGS) -o $@ camd.c tp.c xrcams.c $(LDLIBS) + +clean: + rm -f fh-camprobe fh-camd + +.PHONY: all clean diff --git a/hands/camd/README.md b/hands/camd/README.md new file mode 100644 index 0000000..63c1060 --- /dev/null +++ b/hands/camd/README.md @@ -0,0 +1,63 @@ +# camd + +Root-side camera access for frame-hands: + +- `fh-camd`: the frame broker. It publishes the four IR tracking cameras, and optionally the Arcturus color pair, to a shared-memory ring that the unprivileged tracker reads. +- `fh-camprobe`: a test tool that records timestamped frames from every camera, including the Arcturus color pair, with CSV logs. + +## How it gets frames + +XRService owns the headset cameras. Both tools borrow its DMA-BUFs read-only with `pidfd_getfd`, the same way FrameEyeCameraFeed does. They never touch XRService's V4L2 descriptors. + +Polling buffers for changes can catch a frame while the camera is still writing it. Instead, they listen to the `v4l2:v4l2_dqbuf` tracepoint, which fires when XRService takes a buffer. It gives the buffer index, the sequence number and the capture timestamp. + +The tools learn which DMA-BUF holds each V4L2 index by watching which buffer changes at each dequeue. + +- Right after XRService allocates its buffers, the mapping is allocation order. +- After XRService restarts streaming, the order is shuffled, and the mapping is learned index by index. +- The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart. +- `fh-camd` also re-maps an index on the fly when its buffer holds no new frame. + +## fh-camd + +``` +make +sudo ./fh-camd # runs until stopped or XRService exits +``` + +It needs root only to set up: to borrow the buffers (`ptrace_scope=1` blocks `pidfd_getfd`) and to open the root-only tracepoints. Then it drops to the invoking user for good; XRService itself runs as that user. It reads nothing from the ring's readers. + +Frames go to `/run/frame-hands/ir-ring`. The file is mode 0600 and owned by the user. It sits in a root-owned directory, so no other account can plant a file or link there. The layout is in `fhring.h`, and `tracker/ring.py` reads it. + +- Only complete, bright frames are published. The cameras alternate a normal exposure with a near-black one, so each camera gets 30 of its 60 fps. +- A copy torn by the camera overwriting the buffer is dropped. +- Each copy takes about 0.1 ms, and a cache sync about 0.15 ms. + +Options: + +- `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands. +- `--with-color`: also publish the two Arcturus color cameras, flagged `FH_CAM_COLOR`. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232) and at most 30 fps (`--color-fps`; the cameras run at 60). Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the color module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core. +- The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 color cameras. +- `--sensor S`: only the mono cameras whose sensor name contains S. + +It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. Start it again to re-attach. + +## fh-camprobe + +Wear the headset (cameras only stream while it's worn), then run: + +``` +sudo ./fh-camprobe # records 15 s +sudo ./fh-camprobe --list # discovery and tracepoints only +``` + +Output goes to `~/Pictures/framecap/stereo-