mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-06 00:00:07 +02:00
Make hand tracking a Frametop component
- Programs: ft-camd (the camera broker), ft-hands (the tracker), and ft-handreplay and ft-ringplay for recordings, built by hands/build.sh into hands/build/ with one Makefile. The first build fetches ncnn at frame-hands' pinned tag and builds it with the same options. - ft-camd gets its privileges from file capabilities (CAP_SYS_PTRACE, CAP_PERFMON, CAP_DAC_READ_SEARCH) that hands/run.sh install sets with sudo, and drops them once set up. It still works under sudo. It runs on the host, linked statically, as frametop-camd.service. ft-hands runs in the dev container as frametop-hands.service. Both start and stop with SteamVR. - Files move to /run/user/UID/frametop/ (cam-ring, hands, gestures), not $XDG_RUNTIME_DIR, which a terminal in the Frametop desktop has its own of. SIGUSR1 recordings go to ~/.local/share/frametop/hands. - The calibration is read through /run/host in the container. - Settings: HANDS_SWAP_SIDES and HANDS_CPUS in frametop.conf. - install.sh offers hand tracking as an optional last step. - The container gets jsoncpp-devel, glibc-static, and NumPy and OpenCV for the Python tools. - tools/ring.py reads the ring, and models/NOTICE credits the Apache-2.0 models. Checked: ft-handreplay gives identical summaries and byte-identical depth dumps to frame-hands' fh-replay on both 2026-09-29 recordings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed
1 parent
3e3d31c728
commit
499035216c
41 files changed
+727
-402
No files matched your search
+21
-5
@@ -4,6 +4,21 @@ Hand tracking from the headset's own cameras has been built as a separate projec
|
||||
|
||||
Builds from this worktree must sync to their own folder on the Frame, never `~/dev/frametop`: run every script with `FRAME_REPO=/home/steamos/dev/frametop-hands`.
|
||||
|
||||
## Status (2026-09-30)
|
||||
|
||||
Steps 1-5 are done:
|
||||
|
||||
- frame-hands' pending work was committed there (6c63c9e).
|
||||
- Its filtered history was merged under `hands/` (1a76d15), then laid out (`trackd/` to `track/`).
|
||||
- The renames, the Frametop paths, and ft-camd's file capabilities are done. So are `hands/Makefile`, `build.sh`, `run.sh`, the two units, the README, the settings, the installer step, and ft-screens on the shared header.
|
||||
- Built in the dev container on the Frame, and on the 7i.
|
||||
- Checked without the headset:
|
||||
- `ft-handreplay` against frame-hands' `fh-replay`, both x86 with `--cost`, on the whole dim recording and the first 60 s of the bright one: identical summaries and byte-identical depth dumps. The Makefile's own ncnn build is included in that.
|
||||
- `ft-ringplay` into `ft-hands` on the 7i tracked, pinched, and wrote `/run/user/UID/frametop/{hands,gestures}`.
|
||||
- On the Frame, ft-hands in the container finds the calibration through `/run/host/persist`, and ft-camd without its capabilities refuses with a clear message.
|
||||
|
||||
Next is step 6, with the user: `hands/run.sh install` (sudo setcap), then a desktop restart from this branch so ft-screens reads the new path.
|
||||
|
||||
## What frame-hands is today
|
||||
|
||||
| Part | What it is | Size |
|
||||
@@ -34,9 +49,10 @@ hands/
|
||||
frametop-hands.service
|
||||
include/ # fhring.h, fh_hands.h, fh_gestures.h: shared with screens/ and pointer/
|
||||
camd/ # ft-camd: camd.c tp.c xrcams.c, LICENSE.FrameEyeCameraFeed
|
||||
track/ # ft-hands: tracker, nets, calib, pinch, publish, record
|
||||
track/ # ft-hands: tracker, nets, calib, pinch, publish, record; replay.cpp
|
||||
# (ft-handreplay) and ringplay.cpp (ft-ringplay) for recordings
|
||||
models/ # the ncnn models, with NOTICE (Apache-2.0, MediaPipe / OpenCV Zoo)
|
||||
tools/ # ft-handreplay, ft-ringplay, the Python checks, watch_gestures
|
||||
tools/ # the Python checks, watch_gestures, depth_report, calib.py, ring.py
|
||||
```
|
||||
|
||||
Left behind in frame-hands, which stays as the lab: the recordings, the Python prototype (the tools that need `calib.py` or `models.py` get a trimmed copy in `hands/tools/`), `probes/`, `notes/`, `re/`, `shim/`, `camprobe`, and `vendor/`. The reverse-engineering notes don't belong in a public repo, and recordings are images of the user's hands and room, so they never go into git.
|
||||
@@ -59,7 +75,7 @@ The source keeps its `fh_` identifiers and header names (`fh_hands.h`, `fh_gestu
|
||||
|
||||
- `hands/build.sh` builds in the dev container through `scripts/frame.sh --build`, into `hands/build/`, like the other components. `FRAME_BUILDER=pc` can take the ncnn build.
|
||||
- ncnn: fetched at a pinned tag (20260526, as now) into `hands/build/ncnn` and built once, the way `screens/build.sh` fetches the OpenVR header, with frame-hands' options so results match. `NCNN=` points the build at an existing install instead. Every net runs single-threaded (`num_threads = 1`), with the tracker spreading nets over its own pinned threads, so OpenMP could go later.
|
||||
- ft-hands runs in the dev container like ft-pointer and ft-powerd (`distrobox enter dev --`, after `scripts/container-up.sh`). Today's fh-tracker runs on the host and works only because the host happens to have the same `libjsoncpp.so.25` and libgomp as the container. The calibration moves from `/persist` to `/run/host/persist` inside the container. calib.cpp already takes a prefix for this (`FRAME_JOB_DEVICE_ROOT`, to be renamed).
|
||||
- ft-hands runs in the dev container like ft-pointer and ft-powerd (`distrobox enter dev --`, after `scripts/container-up.sh`). Today's fh-tracker runs on the host and works only because the host happens to have the same `libjsoncpp.so.25` and libgomp as the container. Inside the container the calibration is at `/run/host/persist`, and calib.cpp (and `tools/calib.py`) fall back to it when `/persist` isn't there.
|
||||
- ft-camd has to run on the host (below), so it's linked statically (only libc and libm; `glibc-static` goes into `setup/dev-container.sh`). The host has an older glibc than the container.
|
||||
|
||||
## Running it
|
||||
@@ -73,9 +89,9 @@ Either way the password is needed once at install, through the same `sudo -S` pa
|
||||
|
||||
**ft-hands** is a user unit, `frametop-hands.service`: after `frametop-camd.service`, `PartOf=steamvr.service`, nice 5, model threads on CPUs 5-7 (measured best on 2026-09-29).
|
||||
|
||||
**Settings** in `~/.config/frametop.conf`: `HANDS=0|1` (off by default until it's ready for others), `HANDS_CPUS=5,6,7`, `HANDS_COLOR=0`. Later, a switch in Frametop Display Settings.
|
||||
**Settings** in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES=1` and `HANDS_CPUS=5,6,7`, read by ft-hands. It's on while its services are installed (`hands/run.sh install`, `uninstall`), so there's no `HANDS` switch. There's no setting for colour yet. Later, a switch in Frametop Display Settings.
|
||||
|
||||
**Installer:** a step in `install.sh` that asks first, because it needs sudo.
|
||||
**Installer:** an optional last step in `install.sh`, off by default, which asks first because it needs sudo.
|
||||
|
||||
## Interfaces
|
||||
|
||||
|
||||
@@ -169,6 +169,17 @@ power/run.sh off | on # the displays off now, or back on
|
||||
power/run.sh log
|
||||
```
|
||||
|
||||
## Hand tracking (experimental)
|
||||
|
||||
Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker also detects pinches, for clicking where you look with the gaze pointer (not wired to the pointer yet). It's optional: `hands/run.sh install`, or the last step of `install.sh`.
|
||||
|
||||
- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`.
|
||||
- `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches) next to the ring.
|
||||
- Both start and stop with SteamVR. `hands/run.sh status` and `hands/run.sh log` show how they're doing.
|
||||
- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (after some SteamVR restarts the side cameras' names come out swapped, and hands land beside the holes; `hands/tools/check_sides.py --ring` tells) and `HANDS_CPUS`.
|
||||
|
||||
Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md).
|
||||
|
||||
## Remote desktop over VNC
|
||||
|
||||
With `REMOTE=1` in the config (`desktops.sh remote on`), the desktop is also served over VNC, for RealVNC Viewer or macOS Screen Sharing. `desktops.sh remote info` prints the address and password.
|
||||
|
||||
+1
-21
@@ -1,23 +1,3 @@
|
||||
# Recordings (tens of GB) and the Python venv
|
||||
captures/
|
||||
.venv/
|
||||
__pycache__/
|
||||
# Upstream clones: ncnn (see trackd/README.md) and FrameEyeCameraFeed (MIT, adapted into camd/)
|
||||
vendor/
|
||||
# Disassembly of Valve's vrclient.so, for reverse engineering only
|
||||
re/*.dis
|
||||
# Downloadable model sources (tools/convert_models.py); the converted ncnn models are kept
|
||||
# Model sources that tools/convert_models.py downloads; the converted ncnn models are kept
|
||||
models/onnx/
|
||||
models/*.task
|
||||
# Build output
|
||||
camd/fh-camd
|
||||
camd/fh-camprobe
|
||||
trackd/fh-tracker
|
||||
trackd/fh-replay
|
||||
trackd/fh-ringplay
|
||||
trackd/nettest
|
||||
probes/fh-frametime
|
||||
probes/mgrvt
|
||||
probes/ptstate
|
||||
probes/refprobe
|
||||
probes/*.bin
|
||||
@@ -0,0 +1,49 @@
|
||||
# Hand tracking, built into build/ (hands/build.sh runs this in the dev container):
|
||||
# make ft-camd (camd/: runs on the host, so linked statically) and ft-hands (track/)
|
||||
# make tools ft-handreplay and ft-ringplay, for recordings
|
||||
# The first build fetches ncnn (NCNN_TAG) and builds it into build/ncnn, which takes a few
|
||||
# minutes. NCNN=DIR uses an ncnn install already built instead.
|
||||
NCNN_TAG = 20260526
|
||||
NCNN ?= build/ncnn/install
|
||||
CFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter
|
||||
CXXFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter -Wno-psabi
|
||||
CXXFLAGS += -std=c++17 -fopenmp -I$(NCNN)/include/ncnn
|
||||
LDLIBS = $(NCNN)/lib/libncnn.a -ljsoncpp -fopenmp -lpthread
|
||||
|
||||
CAMD = camd/camd.c camd/tp.c camd/xrcams.c
|
||||
TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp
|
||||
HDR = $(wildcard track/*.h) camd/fhring.h include/fh_hands.h include/fh_gestures.h
|
||||
|
||||
all: build/ft-camd build/ft-hands
|
||||
tools: build/ft-handreplay build/ft-ringplay
|
||||
|
||||
build/ft-camd: $(CAMD) camd/tp.h camd/xrcams.h camd/fhring.h
|
||||
@mkdir -p build
|
||||
$(CC) $(CFLAGS) -static -o $@ $(CAMD) -lm
|
||||
|
||||
build/ft-hands: track/main.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
|
||||
@mkdir -p build
|
||||
$(CXX) $(CXXFLAGS) -o $@ track/main.cpp $(TRACK) $(LDLIBS)
|
||||
|
||||
build/ft-handreplay: track/replay.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
|
||||
@mkdir -p build
|
||||
$(CXX) $(CXXFLAGS) -o $@ track/replay.cpp $(TRACK) $(LDLIBS)
|
||||
|
||||
build/ft-ringplay: track/ringplay.cpp track/record.h camd/fhring.h
|
||||
@mkdir -p build
|
||||
$(CXX) $(CXXFLAGS) -o $@ track/ringplay.cpp
|
||||
|
||||
# ncnn as frame-hands built it (the models were converted and quantized for it), minus its tools
|
||||
build/ncnn/install/lib/libncnn.a:
|
||||
rm -rf build/ncnn && mkdir -p build/ncnn
|
||||
git clone -q --depth 1 --branch $(NCNN_TAG) -c advice.detachedHead=false https://github.com/Tencent/ncnn.git build/ncnn/src
|
||||
cmake -S build/ncnn/src -B build/ncnn/build -G Ninja -Wno-dev -DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_INSTALL_PREFIX=$(CURDIR)/build/ncnn/install -DCMAKE_INSTALL_LIBDIR=lib -DNCNN_VULKAN=OFF \
|
||||
-DNCNN_OPENMP=ON -DNCNN_INT8=ON -DNCNN_SIMPLEOCV=ON -DNCNN_BUILD_TOOLS=OFF -DNCNN_BUILD_EXAMPLES=OFF \
|
||||
-DNCNN_BUILD_BENCHMARK=OFF -DNCNN_BUILD_TESTS=OFF -DNCNN_PYTHON=OFF > build/ncnn/cmake.log
|
||||
cmake --build build/ncnn/build --target install > build/ncnn/build.log
|
||||
|
||||
clean:
|
||||
rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay
|
||||
|
||||
.PHONY: all tools clean
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
# Hands (experimental)
|
||||
|
||||
Hand tracking from the headset's own cameras. It serves two things in Frametop:
|
||||
|
||||
- **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro.
|
||||
- **Pinches:** look at something and pinch to click it, pinch and move to drag, with the eye tracker doing the looking (`gaze/`). The tracker publishes the pinches. The pointer helper doesn't read them yet.
|
||||
|
||||
Two programs, each a user service that starts and stops with SteamVR:
|
||||
|
||||
- `ft-camd` (`camd/`, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host.
|
||||
- `ft-hands` (`track/`, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container.
|
||||
|
||||
```
|
||||
hands/run.sh install # build, give ft-camd its capabilities (sudo, once per build), enable
|
||||
hands/run.sh status # the services, and ft-hands' last status lines
|
||||
hands/run.sh log [lines]
|
||||
hands/run.sh restart # after changing a setting
|
||||
hands/run.sh caps # after rebuilding ft-camd (a rebuild clears its capabilities)
|
||||
hands/run.sh uninstall
|
||||
```
|
||||
|
||||
Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides them):
|
||||
|
||||
- `HANDS_SWAP_SIDES=1`: the two side cameras' names are swapped (see ft-camd below). Check with `tools/check_sides.py --ring`.
|
||||
- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
|
||||
|
||||
Files, all in `/run/user/UID/frametop/` (private to the user):
|
||||
|
||||
| File | Written by | Layout | Read by |
|
||||
| --- | --- | --- | --- |
|
||||
| `cam-ring` | ft-camd | `camd/fhring.h` | ft-hands, `tools/ring.py` |
|
||||
| `hands` | ft-hands | `include/fh_hands.h` | ft-screens (`screens/handcut.cpp`) |
|
||||
| `gestures` | ft-hands | `include/fh_gestures.h` | `tools/watch_gestures.py`; the pointer helper, later |
|
||||
|
||||
The source keeps the `fh_` names and magic strings of frame-hands, where this was developed (`~/Desktop/Projects/frame-hands` on the developer's Frame, which keeps the recordings, probes and Python prototype). So its recordings and tools still work.
|
||||
|
||||
## ft-camd
|
||||
|
||||
XRService owns the headset cameras. ft-camd borrows its DMA-BUFs read-only with `pidfd_getfd`, the same way FrameEyeCameraFeed does. It never touches XRService's V4L2 descriptors. `discovery` in `camd/xrcams.c` is adapted from FrameEyeCameraFeed (MIT, see `camd/LICENSE.FrameEyeCameraFeed`).
|
||||
|
||||
Polling buffers for changes can catch a frame while the camera is still writing it. Instead, ft-camd listens to the `v4l2:v4l2_dqbuf` tracepoint, which fires when XRService takes a buffer. It gives the buffer index, the sequence number and the capture timestamp. ft-camd learns which DMA-BUF holds each V4L2 index by watching which buffer changes at each dequeue:
|
||||
|
||||
- Right after XRService allocates its buffers, the mapping is allocation order.
|
||||
- After XRService restarts streaming, the order is shuffled, and the mapping is learned index by index.
|
||||
- The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart.
|
||||
- It also re-maps an index on the fly when its buffer holds no new frame.
|
||||
|
||||
**Privileges.** Setting up needs three things. `pidfd_getfd` on XRService needs `CAP_SYS_PTRACE`, because the Frame has `ptrace_scope=1`. The system-wide tracepoint needs `CAP_PERFMON`, because `perf_event_paranoid` is 2. Its format files are root-only, which needs `CAP_DAC_READ_SEARCH`. `hands/run.sh install` gives the binary those capabilities with `sudo setcap`. ft-camd drops them all once it has set up, before it reads a frame, and then runs as you. XRService runs as you too. It also runs under `sudo`, for trying it by hand, and then drops to the user who ran sudo. It reads nothing from the ring's readers.
|
||||
|
||||
The ring is mode 0600, in a folder only you can write. Frame handling:
|
||||
|
||||
- Only complete, bright frames are published. The cameras alternate a normal exposure with a near-black one, so each camera gets 30 of its 60 fps.
|
||||
- A copy torn by the camera overwriting the buffer is dropped.
|
||||
- Each copy takes about 0.1 ms, and a cache sync about 0.15 ms.
|
||||
|
||||
Options:
|
||||
|
||||
- `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands.
|
||||
- `--with-color`: also publish the two Arcturus colour cameras, flagged `FH_CAM_COLOR`. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232) and at most 30 fps (`--color-fps`; the cameras run at 60). Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the colour module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core.
|
||||
- The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 colour cameras.
|
||||
- `--sensor S`: only the mono cameras whose sensor name contains S.
|
||||
- `--status S`: a status line every S seconds (0: never).
|
||||
|
||||
It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. The service starts it again, and it attaches to the new buffers.
|
||||
|
||||
**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. With the headset on, looking at a room with some texture, `tools/check_sides.py --ring` says whether the names are right (exit 0), swapped (exit 3), or it can't tell (exit 2). When they're swapped, set `HANDS_SWAP_SIDES=1`. The colour cameras are video3 (`arcimx616 0-0010`) and video0 (`0-001a`); which of them is `passthrough_left` in the module's calibration is for `tools/check_color.py` to settle, on a recording with texture in view.
|
||||
|
||||
## ft-hands
|
||||
|
||||
```
|
||||
hands/build/ft-hands # status every 5 s; Ctrl+C to stop
|
||||
hands/build/ft-hands --int8 # the 8-bit models (models/ncnn/*-int8.ncnn.*)
|
||||
```
|
||||
|
||||
Run it in the dev container (`distrobox enter dev -- ...`). It reads the factory calibration from `/persist` (`/run/host/persist` in the container).
|
||||
|
||||
Options:
|
||||
|
||||
- `--threads N`: model threads, pinned to the `--cpus` list. Default 3.
|
||||
- `--cpus LIST`: CPUs for the model threads and the main loop. Default `5,6,7` (`HANDS_CPUS`). SteamOS starts user processes on CPUs 0-4, and XRService's head tracking runs on 2-3. With the headset on, a step took 8.4 ms on 5-7 against 13.2 ms on 2-4, and SteamVR's frame timing didn't change (2026-09-29, three rounds of the same replayed frames).
|
||||
- `--contrast MODE` or `PALM/HAND`: how crops are equalized before the models see them: `clahe[:CLIP]`, `none`, or `stretch` (1st-99th percentile). Default `clahe:2/none`. In the dim recording, CLAHE let the palm search find about 10% more hands, but it made the landmarks jitter more (published median 6.9 mm, against 6.0 mm with plain landmark crops).
|
||||
- `--swap-sides`: swap the two side cameras (`HANDS_SWAP_SIDES`, see ft-camd).
|
||||
- `--seconds N`: stop after N seconds.
|
||||
- `--status S`: how often to print status, in seconds.
|
||||
- `--models DIR`: where the models are.
|
||||
- `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
|
||||
- `--no-publish`: don't write the hands and gestures files.
|
||||
- `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
|
||||
- `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
|
||||
- `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.
|
||||
- `--ring PATH`: read frames from another ring, such as `ft-ringplay`'s.
|
||||
|
||||
The status line also says how often a hand was on each side (by where the wrist is), and why views and hands came and went: views lost (the landmark model stopped seeing the hand), handoff misses (a crop projected from the hand's 3D position found nothing), duplicates, splits (two views disagreed in 3D), and hands created, merged and forgotten.
|
||||
|
||||
### Scheduling
|
||||
|
||||
- Each hand is tracked in its best two cameras, the way MediaPipe tracks: the landmark model runs on a crop placed from the previous landmarks, with no palm detection.
|
||||
- A hand seen in too few cameras is projected into the others through the calibration. Where it lands well inside a camera, that camera gets a crop to try. This is how a hand raised out of the side cameras reaches the upper ones.
|
||||
- The palm detector runs only while fewer than two hands are tracked, at most 5 times a second, on a few zoomed tiles per search. Tiles are picked in proportion to how likely hands are there. Each tile is turned so the expected shoulder-to-hand direction points up.
|
||||
- Frame sets are processed at 30 Hz while a hand moves faster than 0.25 m/s (or a pinch is down or closing), at 15 Hz otherwise, and at 5 Hz while no hand is in view.
|
||||
|
||||
### 3D
|
||||
|
||||
- **Two or more views:** each landmark is triangulated from the camera rays, weighted by the model's presence score. The median ray distance is reported as the residual.
|
||||
- **Pairing views across cameras.** The side cameras sit side by side, so two hands next to each other at the same height fall on the same epipolar lines, and rays to two different hands can nearly meet close to the cameras. That made phantom hands 12-15 cm in front of the eyes, which tore holes through the screens. Each step now scores every way of pairing the views in two cameras and keeps the best. A pair scores well when its rays meet, when each view's apparent size matches the triangulated distance, and when the model calls both the same hand. The size check uses a fixed prior: with the model's average hand, clean pairs measure 0.71-1.51 times the one-view distance, and mismatched pairs mostly far less.
|
||||
- **One view:** depth comes from the model's metric world landmarks, their spread across the palm against the angle it covers in the image, scaled by the user's hand size (learned while two views are available). That distance is off by 10-30% and wanders about 10% between frames, so a hand that drops to one camera keeps its last distance and drifts toward the one-view guess by 10% a frame.
|
||||
- **Smoothing.** The published landmarks go through a One Euro filter: it smooths hard while the hand is still (tracking noise is several mm per frame) and hardly at all while it moves fast. The palm speed that sets the update rate is the filtered one; the raw speed read about 0.25 m/s from noise alone.
|
||||
- **Capsules.** Forearms follow the hand's own axis, and nothing within 12 cm in front of the eyes is published.
|
||||
|
||||
How good the depth is, measured from recordings (2026-09-30, `--depth` below): the two lower cameras see the hands about 77% of the time, a lower and an upper camera 7-12%, and one camera 12-15%. Depth is the noisy direction. With the lower pair, it jitters 4-6 times as much as sideways position (published: 3-7 mm against 1-2 mm). The one-camera guess is a median 2-6 cm off. When a camera drops out, drifting 10% a frame toward that guess is worse than keeping the last distance (after 0.5 s a median 23-30 mm off, against 11-12 mm).
|
||||
|
||||
## Pinch
|
||||
|
||||
ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gestures file. The layout, and how to read it without missing quick taps, is in `include/fh_gestures.h`.
|
||||
|
||||
- A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost).
|
||||
- The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. `--pinch-triangulated` uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. Typing still gave 2 pinches a minute, so a consumer should only act on a pinch while the gaze is on a target.
|
||||
- The pinch point is midway between the thumb and index tips. A drag is the pinch point now, minus where it was when the pinch began, both turned into the room with the HMD pose at their capture times.
|
||||
- `tools/watch_gestures.py` prints begins, ends and drag offsets live, and `--distance` prints each hand's distance.
|
||||
|
||||
The pointer helper is the natural consumer. Its gaze mode already treats a press as "stop where the gaze put it, drag onto the target, click on release", and "hold still for half a second, then move" as a drag. A pinch begin would be the press, the end the release, and the pinch point's movement the drag.
|
||||
|
||||
## Recordings
|
||||
|
||||
`hands/build.sh --tools` also builds the offline tools.
|
||||
|
||||
`ft-handreplay DIR` runs a recording through the tracker with the live scheduling and reports how well it kept the hands: hands per set, left and right coverage, track lengths, pinches, jitter, and the same reasons as the status line.
|
||||
|
||||
```
|
||||
hands/build/ft-handreplay ~/.local/share/frametop/hands/rec-20260929-120000 --cost --oracle 10 --timeline /tmp/tl.txt
|
||||
```
|
||||
|
||||
- `--cost`: instead of timing the steps, charge each round of model calls what it typically costs live (10 ms landmarks, 18 ms palms), so results repeat exactly.
|
||||
- `--oracle N`: every N-th set, also search every tile of every camera, and report how often the tracker had the hands that full search could find.
|
||||
- `--slow F`: live, the tracker skips sets that arrive while it's busy. Replay counts each step's time times F as busy (default 1; the headset is busier live).
|
||||
- `--timeline FILE`: a line per processed set and hand, with pinch events and distances.
|
||||
- `--cams mono|color|all`: which cameras to track with (default `mono`). `color` tracks with the Arcturus pair alone, for comparing it with the IR cameras on the same recording. It needs a recording made with `ft-camd --with-color`. `--color-left NODE` (`color_video0` or `color_video3`) and `--color-crop subtract|none` say how the module's calibration maps onto the images; `tools/check_color.py` finds out.
|
||||
- `--depth FILE`: a line per hand per processed set for `tools/depth_report.py`, which measures the depth without ground truth: how the hands were seen, the noise along the line of sight against across it, each camera's one-view distance against the triangulated one, and what a camera dropping out would do.
|
||||
- The pinch, contrast and presence options are ft-hands'.
|
||||
|
||||
`ft-ringplay DIR --ring PATH [--from S] [--to S] [--loop]` publishes a recording into a ring file in real time, as ft-camd would, so `ft-hands --ring PATH --no-publish` runs the same frames run after run. It needs no privileges, and it skips the dark frames.
|
||||
|
||||
## Tools
|
||||
|
||||
Python, with NumPy and OpenCV (in the dev container: `python3-numpy`, `python3-opencv`, which `setup/dev-container.sh` installs). Off the Frame, `FRAME_JOB_DEVICE_ROOT` can point at a folder with copies of the headset's calibration files.
|
||||
|
||||
- `tools/check_sides.py --ring` (or a recording): are the side cameras named right?
|
||||
- `tools/check_color.py REC`: how the colour module's calibration maps onto its images.
|
||||
- `tools/show_set.py REC`: a recording's frame sets as images.
|
||||
- `tools/watch_gestures.py [--distance]`: pinches, live.
|
||||
- `tools/depth_report.py DEPTH`: the depth measures above.
|
||||
- `tools/convert_models.py`: how `models/ncnn` was made from the OpenCV Zoo ONNX ports of MediaPipe's models (see `models/NOTICE`).
|
||||
|
||||
## Build
|
||||
|
||||
`hands/build.sh` builds in the dev container on the Frame, into `hands/build/`, with `hands/Makefile`. The first build fetches ncnn at a pinned tag and builds it into `hands/build/ncnn`, which takes a few minutes; `NCNN=DIR` points at an ncnn install already built instead. ft-camd is linked statically, because it runs on the host, which has an older glibc than the container.
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build hand tracking in the dev container on the Frame, into hands/build/: ft-camd and ft-hands,
|
||||
# and with --tools also ft-handreplay and ft-ringplay. The first build fetches ncnn and builds
|
||||
# it (a few minutes); NCNN=DIR, an ncnn install already on the Frame, skips that.
|
||||
# A rebuilt ft-camd has lost its capabilities: hands/run.sh install sets them again.
|
||||
set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
||||
targets=all
|
||||
[ "${1:-}" = --tools ] && targets="all tools"
|
||||
"$root/scripts/sync.sh" >/dev/null
|
||||
exec "$root/scripts/frame.sh" -C hands "make -s ${NCNN:+NCNN=$NCNN} $targets && echo built \$(ls build/ft-* | tr '\n' ' ')"
|
||||
@@ -1,15 +0,0 @@
|
||||
CFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter
|
||||
LDLIBS = -lm
|
||||
|
||||
all: fh-camprobe fh-camd
|
||||
|
||||
fh-camprobe: camprobe.c tp.c xrcams.c tp.h xrcams.h
|
||||
$(CC) $(CFLAGS) -o $@ camprobe.c tp.c xrcams.c $(LDLIBS)
|
||||
|
||||
fh-camd: camd.c tp.c xrcams.c tp.h xrcams.h fhring.h
|
||||
$(CC) $(CFLAGS) -o $@ camd.c tp.c xrcams.c $(LDLIBS)
|
||||
|
||||
clean:
|
||||
rm -f fh-camprobe fh-camd
|
||||
|
||||
.PHONY: all clean
|
||||
@@ -1,63 +0,0 @@
|
||||
# camd
|
||||
|
||||
Root-side camera access for frame-hands:
|
||||
|
||||
- `fh-camd`: the frame broker. It publishes the four IR tracking cameras, and optionally the Arcturus color pair, to a shared-memory ring that the unprivileged tracker reads.
|
||||
- `fh-camprobe`: a test tool that records timestamped frames from every camera, including the Arcturus color pair, with CSV logs.
|
||||
|
||||
## How it gets frames
|
||||
|
||||
XRService owns the headset cameras. Both tools borrow its DMA-BUFs read-only with `pidfd_getfd`, the same way FrameEyeCameraFeed does. They never touch XRService's V4L2 descriptors.
|
||||
|
||||
Polling buffers for changes can catch a frame while the camera is still writing it. Instead, they listen to the `v4l2:v4l2_dqbuf` tracepoint, which fires when XRService takes a buffer. It gives the buffer index, the sequence number and the capture timestamp.
|
||||
|
||||
The tools learn which DMA-BUF holds each V4L2 index by watching which buffer changes at each dequeue.
|
||||
|
||||
- Right after XRService allocates its buffers, the mapping is allocation order.
|
||||
- After XRService restarts streaming, the order is shuffled, and the mapping is learned index by index.
|
||||
- The two upper cameras share one run of buffers. For them, only allocation order can tell the cameras apart.
|
||||
- `fh-camd` also re-maps an index on the fly when its buffer holds no new frame.
|
||||
|
||||
## fh-camd
|
||||
|
||||
```
|
||||
make
|
||||
sudo ./fh-camd # runs until stopped or XRService exits
|
||||
```
|
||||
|
||||
It needs root only to set up: to borrow the buffers (`ptrace_scope=1` blocks `pidfd_getfd`) and to open the root-only tracepoints. Then it drops to the invoking user for good; XRService itself runs as that user. It reads nothing from the ring's readers.
|
||||
|
||||
Frames go to `/run/frame-hands/ir-ring`. The file is mode 0600 and owned by the user. It sits in a root-owned directory, so no other account can plant a file or link there. The layout is in `fhring.h`, and `tracker/ring.py` reads it.
|
||||
|
||||
- Only complete, bright frames are published. The cameras alternate a normal exposure with a near-black one, so each camera gets 30 of its 60 fps.
|
||||
- A copy torn by the camera overwriting the buffer is dropped.
|
||||
- Each copy takes about 0.1 ms, and a cache sync about 0.15 ms.
|
||||
|
||||
Options:
|
||||
|
||||
- `--with-dark`: also publish the near-black frames, as extra ring cameras flagged `FH_CAM_DARK`. They show only light sources, so they're no use for hands.
|
||||
- `--with-color`: also publish the two Arcturus color cameras, flagged `FH_CAM_COLOR`. Each is the luma of the 10-bit frame's valid 1972x2464 (the top 8 bits), at half size (`--color-scale 2`: 986x1232) and at most 30 fps (`--color-fps`; the cameras run at 60). Frames that carry the module's warped half-size copy are dropped. Their `capture_ns` is on the color module's clock (2.2 s off the mono cameras' on 2026-09-29), so line them up with the mono cameras by `dqbuf_ns`. Each frame costs about 0.65 ms of cache sync and 1.1 ms of decoding, so both cameras at 30 fps take about 11% of a core.
|
||||
- The ring holds 8 cameras: 4 mono, plus 4 dark twins or 2 color cameras.
|
||||
- `--sensor S`: only the mono cameras whose sensor name contains S.
|
||||
|
||||
It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. Start it again to re-attach.
|
||||
|
||||
## fh-camprobe
|
||||
|
||||
Wear the headset (cameras only stream while it's worn), then run:
|
||||
|
||||
```
|
||||
sudo ./fh-camprobe # records 15 s
|
||||
sudo ./fh-camprobe --list # discovery and tracepoints only
|
||||
```
|
||||
|
||||
Output goes to `~/Pictures/framecap/stereo-<time>/`:
|
||||
|
||||
- `summary.txt`: delays, buffer mapping, exposure pattern, stereo sync, and torn or stale copies.
|
||||
- `frames.csv`: one row per frame. `events.csv`: every tracepoint sample.
|
||||
- `<model>_NNNN_<camera>.pgm`: saved bright pairs. The color cameras are saved raw as `.yuv420_10p` (`tools/decode.py` reads them).
|
||||
- `plane1_*.bin`: raw plane 1 of a few frames, which may hold sensor metadata.
|
||||
|
||||
Which camera is which: video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But fh-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards: check with `tools/check_sides.py --ring` and run the tracker with `--swap-sides` when it says swapped. The color cameras are video3 (`arcimx616 0-0010`) and video0 (`0-001a`); which of them is `passthrough_left` in the module's calibration is for `tools/check_color.py` to settle on a recording with texture in view. `tracker/live.py` maps them by capture pipe (`/sys/class/video4linux/videoN/name`).
|
||||
|
||||
`discovery` in `xrcams.c` is adapted from FrameEyeCameraFeed (MIT, see `LICENSE.FrameEyeCameraFeed`).
|
||||
+100
-59
@@ -1,21 +1,25 @@
|
||||
/*
|
||||
* fh-camd - publish the headset's IR camera frames to unprivileged trackers.
|
||||
* ft-camd - publish the headset's IR camera frames to unprivileged trackers.
|
||||
*
|
||||
* Start it with sudo. As root it:
|
||||
* Privileged only while it sets up:
|
||||
* 1. finds the mono tracking cameras and XRService's buffer queues (xrcams.c),
|
||||
* 2. borrows those buffers read-only with pidfd_getfd,
|
||||
* 3. opens the v4l2_dqbuf tracepoint (tp.c),
|
||||
* 4. creates the frame ring in /run/frame-hands (fhring.h), owned by the user.
|
||||
* Then it drops to that user for good. From then on it only learns which
|
||||
* buffer holds which V4L2 index (as fh-camprobe does), and copies each
|
||||
* complete bright frame into the ring. It exits when XRService exits or
|
||||
* reallocates its buffers; start it again (or let systemd) to re-attach.
|
||||
* 3. opens the v4l2_dqbuf tracepoint (tp.c).
|
||||
* Then it gives up its privileges for good, creates the frame ring in the
|
||||
* user's runtime folder (/run/user/UID/frametop/cam-ring, fhring.h), learns
|
||||
* which buffer holds which V4L2 index, and copies each complete bright frame
|
||||
* into the ring. It exits when XRService exits or reallocates its buffers;
|
||||
* start it again (or let systemd) to re-attach.
|
||||
*
|
||||
* XRService itself runs as the same user; root is needed only because
|
||||
* ptrace_scope=1 blocks pidfd_getfd and the tracepoints are root-only.
|
||||
* Nothing is read from the ring's readers.
|
||||
* XRService itself runs as the same user. The privileges are needed only
|
||||
* because ptrace_scope=1 limits pidfd_getfd to CAP_SYS_PTRACE, and the
|
||||
* tracepoints need CAP_PERFMON and CAP_DAC_READ_SEARCH (their format files are
|
||||
* root-only). They come from file capabilities (hands/run.sh install sets
|
||||
* them, and ft-camd then runs as a user service), or from starting it with
|
||||
* sudo (it then drops to the user who ran sudo). Nothing is read from the
|
||||
* ring's readers.
|
||||
*
|
||||
* Build: make
|
||||
* Build: hands/build.sh (it runs on the host: linked statically)
|
||||
*/
|
||||
|
||||
#define _GNU_SOURCE
|
||||
@@ -27,10 +31,10 @@
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <grp.h>
|
||||
#include <linux/capability.h>
|
||||
#include <linux/dma-buf.h>
|
||||
#include <math.h>
|
||||
#include <poll.h>
|
||||
#include <pwd.h>
|
||||
#include <signal.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdio.h>
|
||||
@@ -52,6 +56,10 @@
|
||||
#define SYS_pidfd_getfd 438
|
||||
#endif
|
||||
|
||||
#ifndef CAP_PERFMON
|
||||
#define CAP_PERFMON 38
|
||||
#endif
|
||||
|
||||
#define MAX_CAMS FH_RING_MAX_CAMS
|
||||
#define MAX_SLOTS 64
|
||||
#define MAX_INDEX 32
|
||||
@@ -59,8 +67,6 @@
|
||||
#define SEAM_LIMIT 2.0 /* seam_score above this: the frame carries the half-size copy */
|
||||
#define STALE_RELEARN 30 /* consecutive unchanged frames: the mapping changed */
|
||||
#define MAX_RELEARNS 5 /* then assume XRService has new buffers, and exit */
|
||||
#define RING_DIR FH_RING_DIR
|
||||
#define RING_FILE FH_RING_PATH
|
||||
|
||||
typedef struct {
|
||||
xr_camera_t *cam;
|
||||
@@ -117,7 +123,6 @@ static tp_event_t ev_dqbuf;
|
||||
static int f_dq_minor, f_dq_index, f_dq_ts, f_dq_seq;
|
||||
|
||||
static const char *opt_sensor = "";
|
||||
static const char *opt_user = NULL;
|
||||
static double opt_dark = 0.4;
|
||||
static bool opt_with_dark; /* also publish the near-black frames */
|
||||
static bool opt_with_color; /* also publish the Arcturus color cameras */
|
||||
@@ -561,7 +566,7 @@ static double decode_luma(const cam_t *c, const uint8_t *src, uint8_t *dst, unsi
|
||||
* The color module sometimes writes a warped half-size copy of the image into the
|
||||
* top-left quarter of its buffers. Its bottom edge is a seam between the middle rows
|
||||
* in the left half: this is ~1 for a clean frame and well above for one carrying the
|
||||
* copy (from fh-camprobe --seam).
|
||||
* copy (measured with frame-hands' camera probe).
|
||||
*/
|
||||
static double seam_score(const cam_t *c, const uint8_t *p)
|
||||
{
|
||||
@@ -773,16 +778,21 @@ static int ring_layout(int cam_of[], bool dark_of[], int max)
|
||||
return n;
|
||||
}
|
||||
|
||||
/* The ring lives in a root-owned directory, so nobody can plant a file or link there. */
|
||||
static uint8_t *ring_create(uid_t uid, gid_t gid, size_t *len_out)
|
||||
static char ring_dir[64], ring_file[96];
|
||||
|
||||
/* Created once privileges are gone, as the user, in a folder only the user can write. */
|
||||
static uint8_t *ring_create(size_t *len_out)
|
||||
{
|
||||
if (mkdir(RING_DIR, 0755) < 0 && errno != EEXIST)
|
||||
die("mkdir %s: %s", RING_DIR, strerror(errno));
|
||||
snprintf(ring_dir, sizeof(ring_dir), "/run/user/%d/frametop", (int)getuid());
|
||||
snprintf(ring_file, sizeof(ring_file), "/run/user/%d/" FH_RING_NAME, (int)getuid());
|
||||
|
||||
if (mkdir(ring_dir, 0700) < 0 && errno != EEXIST)
|
||||
die("mkdir %s: %s", ring_dir, strerror(errno));
|
||||
|
||||
struct stat st;
|
||||
|
||||
if (lstat(RING_DIR, &st) < 0 || !S_ISDIR(st.st_mode) || st.st_uid != 0 || (st.st_mode & 022))
|
||||
die("%s must be a directory owned by root and writable only by root", RING_DIR);
|
||||
if (lstat(ring_dir, &st) < 0 || !S_ISDIR(st.st_mode) || st.st_uid != getuid() || (st.st_mode & 022))
|
||||
die("%s must be a directory owned by uid %d and writable only by it", ring_dir, (int)getuid());
|
||||
|
||||
size_t len = sizeof(fh_ring_hdr_t);
|
||||
int cam_of[FH_RING_MAX_CAMS];
|
||||
@@ -795,22 +805,22 @@ static uint8_t *ring_create(uid_t uid, gid_t gid, size_t *len_out)
|
||||
len += FH_RING_SLOTS * ((slot + 63) & ~(size_t)63);
|
||||
}
|
||||
|
||||
if (unlink(RING_FILE) < 0 && errno != ENOENT)
|
||||
die("unlink %s: %s", RING_FILE, strerror(errno));
|
||||
if (unlink(ring_file) < 0 && errno != ENOENT)
|
||||
die("unlink %s: %s", ring_file, strerror(errno));
|
||||
|
||||
int fd = open(RING_FILE, O_RDWR | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC, 0600);
|
||||
int fd = open(ring_file, O_RDWR | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC, 0600);
|
||||
|
||||
if (fd < 0)
|
||||
die("create %s: %s", RING_FILE, strerror(errno));
|
||||
die("create %s: %s", ring_file, strerror(errno));
|
||||
|
||||
if (fchown(fd, uid, gid) < 0 || ftruncate(fd, (off_t)len) < 0)
|
||||
die("prepare %s: %s", RING_FILE, strerror(errno));
|
||||
if (ftruncate(fd, (off_t)len) < 0)
|
||||
die("prepare %s: %s", ring_file, strerror(errno));
|
||||
|
||||
uint8_t *m = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
|
||||
close(fd);
|
||||
|
||||
if (m == MAP_FAILED)
|
||||
die("mmap %s: %s", RING_FILE, strerror(errno));
|
||||
die("mmap %s: %s", ring_file, strerror(errno));
|
||||
|
||||
fh_ring_hdr_t *h = (fh_ring_hdr_t *)m;
|
||||
size_t off = sizeof(*h);
|
||||
@@ -821,7 +831,7 @@ static uint8_t *ring_create(uid_t uid, gid_t gid, size_t *len_out)
|
||||
fh_ring_cam_t *rc = &h->cams[i];
|
||||
bool dark = dark_of[i];
|
||||
|
||||
snprintf(rc->sensor, sizeof(rc->sensor), "%s", c->cam->sensor);
|
||||
snprintf(rc->sensor, sizeof(rc->sensor), "%.31s", c->cam->sensor);
|
||||
snprintf(rc->name, sizeof(rc->name), "%.26s%s", c->slug, dark ? "-dark" : "");
|
||||
rc->flags = dark ? FH_CAM_DARK : c->color ? FH_CAM_COLOR : 0;
|
||||
rc->node = c->cam->node;
|
||||
@@ -857,21 +867,14 @@ static uint8_t *ring_create(uid_t uid, gid_t gid, size_t *len_out)
|
||||
|
||||
static void target_user(uid_t *uid, gid_t *gid)
|
||||
{
|
||||
if (opt_user) {
|
||||
struct passwd *pw = getpwnam(opt_user);
|
||||
if (!pw)
|
||||
die("unknown user %s", opt_user);
|
||||
*uid = pw->pw_uid;
|
||||
*gid = pw->pw_gid;
|
||||
} else if (getenv("SUDO_UID") && getenv("SUDO_GID")) {
|
||||
*uid = (uid_t)atoi(getenv("SUDO_UID"));
|
||||
*gid = (gid_t)atoi(getenv("SUDO_GID"));
|
||||
} else {
|
||||
die("run through sudo or pass --user NAME: fh-camd drops root once set up");
|
||||
}
|
||||
if (!getenv("SUDO_UID") || !getenv("SUDO_GID"))
|
||||
die("started as root without sudo: ft-camd drops to the user who ran sudo once set up");
|
||||
|
||||
*uid = (uid_t)atoi(getenv("SUDO_UID"));
|
||||
*gid = (gid_t)atoi(getenv("SUDO_GID"));
|
||||
|
||||
if (*uid == 0)
|
||||
die("refusing to keep running as root; pass --user NAME");
|
||||
die("refusing to keep running as root: run it with sudo from your own account");
|
||||
}
|
||||
|
||||
static void drop_root(uid_t uid, gid_t gid)
|
||||
@@ -885,6 +888,39 @@ static void drop_root(uid_t uid, gid_t gid)
|
||||
prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
|
||||
}
|
||||
|
||||
/* Started as the user with file capabilities (hands/run.sh install): are they all there? */
|
||||
static bool have_caps(void)
|
||||
{
|
||||
struct __user_cap_header_struct h = { _LINUX_CAPABILITY_VERSION_3, 0 };
|
||||
struct __user_cap_data_struct d[2];
|
||||
|
||||
if (syscall(SYS_capget, &h, d) < 0)
|
||||
return false;
|
||||
|
||||
uint64_t eff = d[0].effective | (uint64_t)d[1].effective << 32;
|
||||
uint64_t need = 1ull << CAP_SYS_PTRACE | 1ull << CAP_PERFMON | 1ull << CAP_DAC_READ_SEARCH;
|
||||
|
||||
return (eff & need) == need;
|
||||
}
|
||||
|
||||
static void drop_caps(void)
|
||||
{
|
||||
struct __user_cap_header_struct h = { _LINUX_CAPABILITY_VERSION_3, 0 };
|
||||
struct __user_cap_data_struct d[2];
|
||||
|
||||
memset(d, 0, sizeof(d));
|
||||
|
||||
if (syscall(SYS_capset, &h, d) < 0)
|
||||
die("dropping capabilities: %s", strerror(errno));
|
||||
|
||||
prctl(PR_CAP_AMBIENT, PR_CAP_AMBIENT_CLEAR_ALL, 0, 0, 0);
|
||||
|
||||
if (have_caps())
|
||||
die("dropping capabilities failed");
|
||||
|
||||
prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------- main */
|
||||
|
||||
static void status(double secs)
|
||||
@@ -922,8 +958,8 @@ static void on_signal(int sig)
|
||||
|
||||
static void usage(const char *argv0)
|
||||
{
|
||||
printf("Usage: sudo %s [options]\n"
|
||||
" --user NAME user to run as after setup and to own the ring (default: $SUDO_USER)\n"
|
||||
printf("Usage: %s [options]\n"
|
||||
"It needs its file capabilities (hands/run.sh install sets them) or sudo.\n"
|
||||
" --sensor S only cameras whose sensor name contains S (default: all mono cameras)\n"
|
||||
" --dark R a frame dimmer than R x the camera's recent brightest is dark (default 0.4)\n"
|
||||
" --with-dark also publish the dark frames, as extra ring cameras flagged FH_CAM_DARK\n"
|
||||
@@ -931,16 +967,14 @@ static void usage(const char *argv0)
|
||||
" --color-scale N ... at 1/N size (default 2: 986x1232)\n"
|
||||
" --color-fps F ... at most F frames a second (default 30; they run at 60)\n"
|
||||
" --status S print a status line every S seconds, 0 for never (default 10)\n"
|
||||
"Frames go to " RING_FILE " (layout in fhring.h).\n", argv0);
|
||||
"Frames go to /run/user/UID/" FH_RING_NAME " (layout in fhring.h).\n", argv0);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv)
|
||||
{
|
||||
for (int i = 1; i < argc; i++) {
|
||||
|
||||
if (!strcmp(argv[i], "--user") && i + 1 < argc)
|
||||
opt_user = argv[++i];
|
||||
else if (!strcmp(argv[i], "--sensor") && i + 1 < argc)
|
||||
if (!strcmp(argv[i], "--sensor") && i + 1 < argc)
|
||||
opt_sensor = argv[++i];
|
||||
else if (!strcmp(argv[i], "--dark") && i + 1 < argc)
|
||||
opt_dark = atof(argv[++i]);
|
||||
@@ -960,12 +994,15 @@ int main(int argc, char **argv)
|
||||
}
|
||||
}
|
||||
|
||||
if (geteuid() != 0)
|
||||
die("run with sudo: borrowing XRService's buffers and reading tracepoints need root");
|
||||
bool root = geteuid() == 0;
|
||||
uid_t uid = getuid();
|
||||
gid_t gid = getgid();
|
||||
|
||||
uid_t uid;
|
||||
gid_t gid;
|
||||
target_user(&uid, &gid);
|
||||
if (root)
|
||||
target_user(&uid, &gid);
|
||||
else if (!have_caps())
|
||||
die("no privileges to borrow XRService's buffers: run hands/run.sh install (it sets ft-camd's "
|
||||
"capabilities; a rebuild clears them), or start it with sudo");
|
||||
|
||||
char err[512];
|
||||
|
||||
@@ -1022,11 +1059,15 @@ int main(int argc, char **argv)
|
||||
if (!tp_open(&tp, evs, 1, err, sizeof(err)))
|
||||
die("%s", err);
|
||||
|
||||
size_t ring_len;
|
||||
fh_ring_hdr_t *ring = (fh_ring_hdr_t *)ring_create(uid, gid, &ring_len);
|
||||
if (root)
|
||||
drop_root(uid, gid);
|
||||
else
|
||||
drop_caps();
|
||||
|
||||
drop_root(uid, gid);
|
||||
printf("ring %s (%.1f MB), running as uid %d\n", RING_FILE, ring_len / 1e6, (int)getuid());
|
||||
size_t ring_len;
|
||||
fh_ring_hdr_t *ring = (fh_ring_hdr_t *)ring_create(&ring_len);
|
||||
|
||||
printf("ring %s (%.1f MB), running as uid %d without privileges\n", ring_file, ring_len / 1e6, (int)getuid());
|
||||
fflush(stdout);
|
||||
|
||||
signal(SIGINT, on_signal);
|
||||
|
||||
+7
-7
@@ -1,7 +1,8 @@
|
||||
/*
|
||||
* fhring - the shared-memory frame ring fh-camd writes and trackers read.
|
||||
* fhring - the shared-memory frame ring ft-camd writes and trackers read.
|
||||
*
|
||||
* One file (FH_RING_PATH) holds a header, then for each camera
|
||||
* One file, /run/user/UID/frametop/cam-ring (FH_RING_NAME in the user's runtime
|
||||
* folder; the folder is private to the user), holds a header, then for each camera
|
||||
* a few slots, each a slot header followed by the image rows packed tightly
|
||||
* (stride == width for 8-bit mono). Only complete, bright frames are published.
|
||||
*
|
||||
@@ -12,7 +13,7 @@
|
||||
* changed the copy is torn, retry with the new latest.
|
||||
*
|
||||
* All multi-byte fields are little-endian; offsets are fixed so Python can read
|
||||
* them with struct (tracker/ring.py mirrors this file).
|
||||
* them with struct (tools/ring.py mirrors this file).
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
@@ -24,8 +25,7 @@
|
||||
#define FH_RING_VERSION 1
|
||||
#define FH_RING_MAX_CAMS 8
|
||||
#define FH_RING_SLOTS 4
|
||||
#define FH_RING_DIR "/run/frame-hands"
|
||||
#define FH_RING_PATH FH_RING_DIR "/ir-ring"
|
||||
#define FH_RING_NAME "frametop/cam-ring" /* in /run/user/UID */
|
||||
|
||||
enum {
|
||||
FH_FMT_GREY8 = 0,
|
||||
@@ -33,9 +33,9 @@ enum {
|
||||
|
||||
enum {
|
||||
FH_CAM_DARK = 1u << 0, /* the near-black exposures between this node's */
|
||||
/* normal frames (fh-camd --with-dark) */
|
||||
/* normal frames (ft-camd --with-dark) */
|
||||
FH_CAM_COLOR = 1u << 1, /* an Arcturus color camera's luma, downscaled */
|
||||
/* (fh-camd --with-color). Not synced with the */
|
||||
/* (ft-camd --with-color). Not synced with the */
|
||||
/* mono cameras, and capture_ns is on its own */
|
||||
/* clock: line it up with them by dqbuf_ns */
|
||||
};
|
||||
|
||||
+1
-1
@@ -434,7 +434,7 @@ static bool scan_xr_fds(pid_t pid, char *err, size_t errn)
|
||||
} else if (strncmp(target, "/dev/video", 10) == 0) {
|
||||
|
||||
ent.kind = FD_VIDEO;
|
||||
snprintf(ent.path, sizeof(ent.path), "%s", target);
|
||||
snprintf(ent.path, sizeof(ent.path), "%.63s", target);
|
||||
|
||||
} else if (strncmp(target, "/dev/v4l-subdev", 15) == 0) {
|
||||
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# Template: the installer replaces @REPO@ with the repo path on the Frame.
|
||||
[Unit]
|
||||
Description=Frametop camera broker: the headset cameras' frames, for hand tracking
|
||||
Documentation=file://@REPO@/hands/README.md
|
||||
# It borrows XRService's camera buffers, so it comes and goes with SteamVR.
|
||||
After=steamvr.service
|
||||
PartOf=steamvr.service
|
||||
|
||||
[Service]
|
||||
# On the host: the dev container can't reach XRService. Its file capabilities (set by
|
||||
# hands/run.sh install) let it borrow the buffers; it drops them once set up. It exits when
|
||||
# XRService restarts, and comes back to attach to the new one.
|
||||
ExecStart=@REPO@/hands/build/ft-camd --status 60
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
TimeoutStopSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=steamvr.service
|
||||
@@ -0,0 +1,20 @@
|
||||
# Template: the installer replaces @REPO@ with the repo path on the Frame.
|
||||
[Unit]
|
||||
Description=Frametop hand tracking: hands for the screens' hand cutouts, pinches for the pointer
|
||||
Documentation=file://@REPO@/hands/README.md
|
||||
After=steamvr.service frametop-camd.service
|
||||
Wants=frametop-camd.service
|
||||
PartOf=steamvr.service
|
||||
|
||||
[Service]
|
||||
# In the dev container (it's built against Fedora's libraries). It reads ft-camd's ring and
|
||||
# writes /run/user/UID/frametop/hands and gestures. Settings: HANDS_* in ~/.config/frametop.conf.
|
||||
ExecStartPre=-@REPO@/scripts/container-up.sh
|
||||
ExecStartPre=-/usr/bin/pkill -x ft-hands
|
||||
ExecStart=%h/.local/bin/distrobox enter dev -- @REPO@/hands/build/ft-hands --status 60
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
TimeoutStopSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=steamvr.service
|
||||
@@ -1,7 +1,7 @@
|
||||
/*
|
||||
* fh_gestures - pinch state frame-hands' tracker publishes for input: look at something
|
||||
* and pinch to click it, pinch and move to drag (the Vision Pro model, with the eye
|
||||
* tracker doing the looking). $XDG_RUNTIME_DIR/frame-hands/gestures, next to the hands
|
||||
* fh_gestures - pinch state ft-hands publishes for input: look at something and pinch to
|
||||
* click it, pinch and move to drag (the Vision Pro model, with the eye tracker doing the
|
||||
* looking). /run/user/UID/frametop/gestures, next to the hands
|
||||
* file, with the same sequence lock (read seq, copy, read seq again; use the copy only if
|
||||
* both reads are the same even number) and the same frame: metres in the head frame at
|
||||
* capture time, OpenVR's HMD frame (+x right, +y up, -z forward).
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
/*
|
||||
* fh_hands - the tracked-hands file frame-hands' tracker publishes
|
||||
* ($XDG_RUNTIME_DIR/frame-hands/hands, directory mode 0700), rewritten in place
|
||||
* fh_hands - the tracked-hands file ft-hands publishes for ft-screens' hand cutouts
|
||||
* (/run/user/UID/frametop/hands, directory mode 0700), rewritten in place
|
||||
* under a sequence lock: read seq, copy, read seq again; use the copy only if
|
||||
* both reads are the same even number.
|
||||
*
|
||||
* Positions are metres in the head frame at capture time, which is OpenVR's HMD
|
||||
* frame (+x right, +y up, -z forward). Turn them into the room with the HMD pose
|
||||
* at capture_ns (CLOCK_MONOTONIC). Writers: trackd (fh-tracker), tracker/publish.py.
|
||||
* at capture_ns (CLOCK_MONOTONIC). Writer: ft-hands (hands/track/io.cpp).
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
The models in ncnn/ are converted from the OpenCV Zoo ONNX ports of Google's MediaPipe hand
|
||||
models, by tools/convert_models.py:
|
||||
|
||||
- palm.ncnn.*: palm_detection_mediapipe_2023feb (https://huggingface.co/opencv/palm_detection_mediapipe)
|
||||
- hand.ncnn.*: handpose_estimation_mediapipe_2023feb (https://huggingface.co/opencv/handpose_estimation_mediapipe)
|
||||
|
||||
MediaPipe is Copyright Google LLC. The models and the OpenCV Zoo ports are licensed under the
|
||||
Apache License, Version 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
|
||||
|
||||
Changes made here: converted to ncnn with pnnx, with the palm detector's channel pads
|
||||
rewritten as ncnn Padding layers, and quantized to 8 bits (the *-int8.ncnn.* files) with
|
||||
ncnn's tools.
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install, start, stop, or inspect hand tracking on the Frame: ft-camd (the camera broker) and
|
||||
# ft-hands (the tracker), user services that start and stop with SteamVR.
|
||||
# Usage: hands/run.sh install|uninstall
|
||||
# hands/run.sh caps # give ft-camd its capabilities again (a rebuild clears them)
|
||||
# hands/run.sh start|stop|restart|status|log [lines]
|
||||
# install and caps need the password (sudo setcap, once per build of ft-camd). On the Frame,
|
||||
# sudo asks in the terminal. From a PC (or with no terminal), the password comes from
|
||||
# steamos_root_pwd in the repo's .env and is sent to sudo -S on stdin, never on a command line.
|
||||
set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
||||
. "$root/scripts/_env.sh"
|
||||
frame="$root/scripts/frame.sh"
|
||||
units="frametop-camd.service frametop-hands.service"
|
||||
# pidfd_getfd on XRService (ptrace_scope=1), system-wide tracepoints, and their root-only
|
||||
# format files. ft-camd drops them all once it has set up.
|
||||
caps=cap_sys_ptrace,cap_perfmon,cap_dac_read_search+ep
|
||||
|
||||
sudo_run() {
|
||||
if [ "$FRAME_LOCAL" = 1 ] && [ -t 0 ]; then
|
||||
sudo bash -c "$1" # asks for the password here
|
||||
return
|
||||
fi
|
||||
local pw
|
||||
pw=$(sed -n 's/^steamos_root_pwd=//p' "$root/.env" 2>/dev/null)
|
||||
pw=${pw#[\"\']}; pw=${pw%[\"\']} # .env values may be quoted
|
||||
[ -n "$pw" ] || { echo "no terminal for sudo, and steamos_root_pwd is missing from $root/.env" >&2; exit 1; }
|
||||
printf '%s\n' "$pw" | on_frame "sudo -S -p '' bash -c $(printf %q "$1")"
|
||||
}
|
||||
|
||||
set_caps() {
|
||||
local bin
|
||||
bin=$(printf %q "$FRAME_REPO/hands/build/ft-camd")
|
||||
sudo_run "setcap $caps $bin && getcap $bin"
|
||||
}
|
||||
|
||||
states="for u in $units; do echo \"\$u: \$(systemctl --user is-active \$u)\"; done"
|
||||
|
||||
case ${1:-status} in
|
||||
install)
|
||||
"$root/hands/build.sh"
|
||||
set_caps
|
||||
for u in $units; do
|
||||
fill_template "$root/hands/$u" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$u"
|
||||
done
|
||||
"$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable $units
|
||||
if systemctl --user -q is-active steamvr.service; then systemctl --user restart $units; sleep 5; fi
|
||||
$states" ;;
|
||||
caps) set_caps ;;
|
||||
uninstall) "$frame" --host "systemctl --user disable --now $units 2>/dev/null
|
||||
for u in $units; do rm -f ~/.config/systemd/user/\$u; done; systemctl --user daemon-reload; echo removed" ;;
|
||||
start|stop|restart) "$frame" --host "systemctl --user $1 $units; $states" ;;
|
||||
status) "$frame" --host "$states; journalctl --user -u frametop-hands.service --no-pager -o cat -n 4" || true ;;
|
||||
log) "$frame" --host "journalctl --user -u frametop-camd.service -u frametop-hands.service --no-pager -o short -n ${2:-30}" ;;
|
||||
*) echo "usage: $0 install|uninstall|caps|start|stop|restart|status|log [lines]" >&2; exit 2 ;;
|
||||
esac
|
||||
+18
-5
@@ -13,6 +13,7 @@ the image.
|
||||
Everything here returns metres in the head frame.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
|
||||
import numpy as np
|
||||
|
||||
@@ -91,11 +92,23 @@ class Camera:
|
||||
return np.degrees(np.arccos(np.clip(self.unproject(uv)[:, 2], -1, 1)))
|
||||
|
||||
|
||||
def device_path(path):
|
||||
"""A headset file such as /persist/xrservice.json. In the dev container the host's / is
|
||||
at /run/host (distrobox doesn't mount /persist); off the Frame, FRAME_JOB_DEVICE_ROOT can
|
||||
point at a folder with copies of them."""
|
||||
root = os.environ.get('FRAME_JOB_DEVICE_ROOT')
|
||||
if root:
|
||||
return root + path
|
||||
if not os.access(path, os.R_OK) and os.access('/run/host' + path, os.R_OK):
|
||||
return '/run/host' + path
|
||||
return path
|
||||
|
||||
|
||||
def load(xrservice=XRSERVICE_JSON, device=DEVICE_JSON):
|
||||
"""{calibration name: Camera} for the tracking cameras, posed in the head frame."""
|
||||
with open(xrservice) as f:
|
||||
with open(device_path(xrservice)) as f:
|
||||
rig = json.load(f)
|
||||
with open(device) as f:
|
||||
with open(device_path(device)) as f:
|
||||
dev = json.load(f)
|
||||
cad_from_cam0 = _pose(dev['cv']['cad_from_cal'])
|
||||
head_from_cad = np.linalg.inv(_pose(dev['head']))
|
||||
@@ -110,18 +123,18 @@ def load(xrservice=XRSERVICE_JSON, device=DEVICE_JSON):
|
||||
|
||||
def load_color(eeprom=ARCTURUS_EEPROM, device=DEVICE_JSON, scale=2, crop='subtract'):
|
||||
"""{"passthrough_left"/"passthrough_right": Camera} for the Arcturus color cameras, posed in
|
||||
the head frame, for fh-camd --with-color's images (luma at 1/scale size).
|
||||
the head frame, for ft-camd --with-color's images (luma at 1/scale size).
|
||||
|
||||
Their calibration is in the CAD frame (mm) with pixel coordinates on the full 2464x2464
|
||||
sensor; each camera also has a cropRegion. crop says how that maps to the delivered
|
||||
image: 'subtract' (image x = sensor x - cropRegion.x) or 'none'. tools/check_color.py
|
||||
tells which fits.
|
||||
"""
|
||||
with open(eeprom, 'rb') as f:
|
||||
with open(device_path(eeprom), 'rb') as f:
|
||||
raw = f.read()
|
||||
i = raw.rfind(b'{', 0, raw.find(b'"alignment_method"'))
|
||||
rig, _ = json.JSONDecoder().raw_decode(raw[i:].decode('latin1'))
|
||||
with open(device) as f:
|
||||
with open(device_path(device)) as f:
|
||||
dev = json.load(f)
|
||||
head_from_cad = np.linalg.inv(_pose(dev['head']))
|
||||
cams = {}
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
"""Which color camera is which, and how their calibration maps onto fh-camd's images.
|
||||
"""Which color camera is which, and how their calibration maps onto ft-camd's images.
|
||||
|
||||
usage: python tools/check_color.py REC_DIR [--sets N]
|
||||
|
||||
A recording made with fh-camd --with-color holds color_video<N> frames with each set.
|
||||
A recording made with ft-camd --with-color holds color_video<N> frames with each set.
|
||||
This matches features between the two color images and scores every reading of the
|
||||
calibration: which video node is passthrough_left, and whether the calibration's
|
||||
cropRegion is subtracted from x ('subtract') or not ('none'). Only the right reading
|
||||
@@ -19,12 +19,11 @@ import numpy as np
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
|
||||
from tools.check_sides import load_cams, matches, score # noqa: E402
|
||||
from tools.show_set import index, read_set # noqa: E402
|
||||
from tracker import calib # noqa: E402
|
||||
from tools import calib # noqa: E402
|
||||
|
||||
|
||||
def load_color(crop):
|
||||
root = os.environ.get('FRAME_JOB_DEVICE_ROOT', '') # frame-job's copy of the device files
|
||||
return calib.load_color(root + calib.ARCTURUS_EEPROM, root + calib.DEVICE_JSON, crop=crop)
|
||||
return calib.load_color(crop=crop)
|
||||
|
||||
|
||||
def main():
|
||||
@@ -37,7 +36,7 @@ def main():
|
||||
sets = [read_set(path, offs[n]) for n in np.linspace(0, len(offs) - 1, a.sets).astype(int)]
|
||||
nodes = sorted(k for k in sets[0] if k.startswith('color_video'))
|
||||
if len(nodes) != 2:
|
||||
sys.exit('need two color_video<N> cameras in the recording (fh-camd --with-color); found %s' % nodes)
|
||||
sys.exit('need two color_video<N> cameras in the recording (ft-camd --with-color); found %s' % nodes)
|
||||
pairs = [matches(s[nodes[0]][0], s[nodes[1]][0]) for s in sets]
|
||||
print('%d sets, %d matches between %s and %s' % (len(sets), sum(len(p[0]) for p in pairs), *nodes))
|
||||
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
"""Check that the side cameras' images carry the right names (slam_left vs slam_right).
|
||||
|
||||
usage: python tools/check_sides.py REC_DIR [--sets N]
|
||||
python tools/check_sides.py --ring [--sets N] (live, from fh-camd's ring)
|
||||
python tools/check_sides.py --ring [--sets N] (live, from ft-camd's ring)
|
||||
|
||||
With --ring it exits 0 when the names are right, 3 when they're swapped (run fh-tracker
|
||||
With --ring it exits 0 when the names are right, 3 when they're swapped (run ft-hands
|
||||
with --swap-sides), and 2 when it can't tell (too little texture in view, or the headset
|
||||
isn't worn).
|
||||
|
||||
fh-camd tells the two side cameras' buffers apart by the order XRService allocated them,
|
||||
ft-camd tells the two side cameras' buffers apart by the order XRService allocated them,
|
||||
and after some XRService restarts that order puts each camera's images under the other's
|
||||
name. The tracker then sees every hand in one camera only, at the wrong depth. This
|
||||
matches features between the two images and measures how close each pair's rays pass
|
||||
@@ -23,15 +23,14 @@ import numpy as np
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
|
||||
from tools.show_set import index, read_set # noqa: E402
|
||||
from tracker import calib # noqa: E402
|
||||
from tools import calib # noqa: E402
|
||||
|
||||
|
||||
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right'} # as fh-tracker maps them
|
||||
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right'} # as ft-hands maps them
|
||||
|
||||
|
||||
def load_cams():
|
||||
root = os.environ.get('FRAME_JOB_DEVICE_ROOT', '') # frame-job's copy of /persist off the Frame
|
||||
return calib.load(root + calib.XRSERVICE_JSON, root + calib.DEVICE_JSON)
|
||||
return calib.load()
|
||||
|
||||
|
||||
def matches(a, b):
|
||||
@@ -80,12 +79,12 @@ def recorded_pairs(rec, count):
|
||||
|
||||
|
||||
def live_pairs(count):
|
||||
"""(label, slam_left image, slam_right image) from fh-camd's ring, half a second apart."""
|
||||
"""(label, slam_left image, slam_right image) from ft-camd's ring, half a second apart."""
|
||||
import time
|
||||
from tracker.ring import Ring
|
||||
from tools.ring import Ring
|
||||
ring = Ring()
|
||||
if not ring.alive():
|
||||
sys.exit('fh-camd isn\'t running (no heartbeat)')
|
||||
sys.exit('ft-camd isn\'t running (no heartbeat)')
|
||||
cams = {}
|
||||
for c in ring.cams:
|
||||
name = PIPES.get(open('/sys/class/video4linux/video%d/name' % c.node).read().strip())
|
||||
|
||||
@@ -2,13 +2,13 @@
|
||||
|
||||
usage: python3 tools/depth_report.py DEPTH [DEPTH...] [--still M/S]
|
||||
|
||||
DEPTH comes from `trackd/fh-replay DIR --depth DEPTH`. Every measure is split by how the
|
||||
DEPTH comes from `hands/build/ft-handreplay DIR --depth DEPTH`. Every measure is split by how the
|
||||
hand was seen: by the two lower cameras ("lower pair"), by a lower and an upper camera on
|
||||
one side ("lower+upper"), or by one camera. Distances are from the head (between the eyes).
|
||||
|
||||
1. How the hands were seen: the share of hand updates in each way, by distance.
|
||||
2. Noise along the line of sight against across it. Each update's palm is compared with a
|
||||
straight line through the two updates before it (fh-replay's jitter measure), and the
|
||||
straight line through the two updates before it (ft-handreplay's jitter measure), and the
|
||||
miss is split along the line from the hand's cameras to the palm and across it. Given
|
||||
as a robust sigma per axis, measured (as triangulated) and published (after the One Euro
|
||||
filter), on updates where the published palm moved slower than --still (default 0.15
|
||||
@@ -20,7 +20,7 @@ one side ("lower+upper"), or by one camera. Distances are from the head (between
|
||||
distance from that camera.
|
||||
4. A camera lost: from two-camera updates, what the tracker would have had if one of the
|
||||
two cameras dropped out there. It keeps the last distance and moves a share of the way
|
||||
to the one-view guess each update (0.1 now, kMonoDepthGain in trackd/tracker.cpp);
|
||||
to the one-view guess each update (0.1 now, kMonoDepthGain in track/tracker.cpp);
|
||||
also shown with other shares, 0 (keep the distance) and 1 (take each guess), and with
|
||||
the guess first scaled by how far off it was while both cameras saw the hand. Compared with the
|
||||
triangulated distance from that camera, 0.1-2 s after the loss.
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
"""Read frames from ft-camd's shared-memory ring (layout: camd/fhring.h)."""
|
||||
import mmap
|
||||
import os
|
||||
import struct
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
|
||||
RING_FILE = '/run/user/%d/frametop/cam-ring' % os.getuid()
|
||||
MAGIC = b'FHRING01'
|
||||
HDR = struct.Struct('<8sIIIIQqQ16x') # 64 bytes
|
||||
CAM = struct.Struct('<32s32siIIIIIQQQQQ32x') # 160 bytes
|
||||
SLOT = struct.Struct('<QQQQQIf16x') # 64 bytes
|
||||
MAX_CAMS = 8
|
||||
LATEST_OFF = 32 + 32 + 4 * 6 + 8 * 2 # cam.latest within fh_ring_cam_t
|
||||
HEARTBEAT_OFF = 40
|
||||
|
||||
|
||||
class Frame:
|
||||
__slots__ = ('cam', 'frame', 'capture_ns', 'dqbuf_ns', 'publish_ns', 'v4l2_seq', 'mean', 'image')
|
||||
|
||||
def __init__(self, cam, fields, image):
|
||||
self.cam = cam
|
||||
(_, self.frame, self.capture_ns, self.dqbuf_ns, self.publish_ns, self.v4l2_seq, self.mean) = fields
|
||||
self.image = image
|
||||
|
||||
|
||||
class RingCamera:
|
||||
def __init__(self, index, fields):
|
||||
(sensor, name, self.node, self.format, self.width, self.height, self.stride, self.nslots,
|
||||
self.slot_offset, self.slot_bytes, _latest, _pub, _drop) = fields
|
||||
self.index = index
|
||||
self.sensor = sensor.split(b'\0', 1)[0].decode()
|
||||
self.name = name.split(b'\0', 1)[0].decode()
|
||||
self.latest_off = HDR.size + index * CAM.size + LATEST_OFF
|
||||
|
||||
def __repr__(self):
|
||||
return 'RingCamera(video%d %s %dx%d)' % (self.node, self.sensor, self.width, self.height)
|
||||
|
||||
|
||||
class Ring:
|
||||
def __init__(self, path=RING_FILE):
|
||||
fd = os.open(path, os.O_RDONLY)
|
||||
try:
|
||||
self.map = mmap.mmap(fd, 0, mmap.MAP_SHARED, mmap.PROT_READ)
|
||||
finally:
|
||||
os.close(fd)
|
||||
magic, version, hdr_bytes, ncams, _, file_bytes, self.writer_pid, _ = HDR.unpack_from(self.map, 0)
|
||||
if magic != MAGIC or version != 1:
|
||||
raise RuntimeError('%s is not an ft-camd ring (magic %r version %d)' % (path, magic, version))
|
||||
self.cams = [RingCamera(i, CAM.unpack_from(self.map, HDR.size + i * CAM.size)) for i in range(ncams)]
|
||||
|
||||
def heartbeat_ns(self):
|
||||
return struct.unpack_from('<Q', self.map, HEARTBEAT_OFF)[0]
|
||||
|
||||
def alive(self, max_age=1.0):
|
||||
hb = self.heartbeat_ns()
|
||||
return hb != 0 and (time.clock_gettime_ns(time.CLOCK_MONOTONIC) - hb) / 1e9 < max_age
|
||||
|
||||
def latest(self, cam):
|
||||
return struct.unpack_from('<Q', self.map, cam.latest_off)[0]
|
||||
|
||||
def read(self, cam, n=None):
|
||||
"""Copy frame n (default: the newest) of a camera, or None if it's gone or being written."""
|
||||
for _ in range(3):
|
||||
if n is None or n == 0:
|
||||
n = self.latest(cam)
|
||||
if n == 0:
|
||||
return None
|
||||
off = cam.slot_offset + (n % cam.nslots) * cam.slot_bytes
|
||||
fields = SLOT.unpack_from(self.map, off)
|
||||
if fields[0] != 2 * n + 2:
|
||||
return None
|
||||
start = off + SLOT.size
|
||||
image = np.frombuffer(self.map, np.uint8, cam.stride * cam.height, start).reshape(cam.height, cam.stride)
|
||||
image = image[:, :cam.width].copy()
|
||||
if struct.unpack_from('<Q', self.map, off)[0] == fields[0]:
|
||||
return Frame(cam, fields, image)
|
||||
n = None # overwritten while copying: take the newest
|
||||
return None
|
||||
@@ -1,12 +1,12 @@
|
||||
"""Draw frame sets from a recording (fh-tracker --record) with what the tracker saw.
|
||||
"""Draw frame sets from a recording (ft-hands --record) with what the tracker saw.
|
||||
|
||||
usage: python tools/show_set.py REC_DIR SET [SET...] [--timeline TL] [--out DIR]
|
||||
|
||||
SET is a set index (fh-replay's timeline gives them). With --timeline (fh-replay
|
||||
SET is a set index (ft-handreplay's timeline gives them). With --timeline (ft-handreplay
|
||||
--timeline), each camera shows the tracker's views at that set: the crop for the next
|
||||
frame, labelled with the hand and presence. Recordings made with fh-camd --with-dark get
|
||||
frame, labelled with the hand and presence. Recordings made with ft-camd --with-dark get
|
||||
a second row: each camera's latest dark frame (<name>_dk), stretched to be visible and
|
||||
labelled with its mean brightness. Recordings made with fh-camd --with-color get a row of
|
||||
labelled with its mean brightness. Recordings made with ft-camd --with-color get a row of
|
||||
the color cameras (color_video<N>). Writes OUT/set_<n>.jpg (default /tmp).
|
||||
"""
|
||||
import argparse
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Watch the pinch gestures fh-tracker publishes, live: begins, ends, and drags.
|
||||
"""Watch the pinch gestures ft-hands publishes, live: begins, ends, and drags.
|
||||
|
||||
usage: python3 tools/watch_gestures.py [--every S] [--distance]
|
||||
|
||||
@@ -22,8 +22,7 @@ SIDES = ('left ', 'right')
|
||||
|
||||
|
||||
def path():
|
||||
run = os.environ.get('XDG_RUNTIME_DIR', '/run/user/%d' % os.getuid())
|
||||
return os.path.join(run, 'frame-hands', 'gestures')
|
||||
return '/run/user/%d/frametop/gestures' % os.getuid()
|
||||
|
||||
|
||||
def read(m):
|
||||
@@ -48,7 +47,7 @@ def main():
|
||||
m = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
|
||||
first = read(m)
|
||||
if first is None or first[0][0] != b'FHGEST01':
|
||||
raise SystemExit('%s is not an fh-tracker gestures file' % path())
|
||||
raise SystemExit('%s is not an ft-hands gestures file' % path())
|
||||
h, p = first
|
||||
print('thresholds: pinch begins under %.3f m, ends over %.3f m' % (h[6], h[7]))
|
||||
seen = [(q[2], q[3]) for q in p] # begins, ends
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
# fh-tracker, the C++ tracker. Needs ncnn built into ../vendor/ncnn/build/install
|
||||
# (see README.md) and jsoncpp.
|
||||
NCNN ?= ../vendor/ncnn/build/install
|
||||
CXXFLAGS ?= -O2 -g -Wall -Wextra -Wno-unused-parameter
|
||||
CXXFLAGS += -std=c++17 -fopenmp -I$(NCNN)/include/ncnn
|
||||
LDLIBS = $(NCNN)/lib/libncnn.a -ljsoncpp -fopenmp -lpthread
|
||||
|
||||
SRC = main.cpp calib.cpp nets.cpp tracker.cpp io.cpp record.cpp pinch.cpp
|
||||
HDR = calib.h geom.h nets.h tracker.h io.h record.h pinch.h ../camd/fhring.h ../include/fh_hands.h ../include/fh_gestures.h
|
||||
|
||||
all: fh-tracker fh-replay fh-ringplay
|
||||
|
||||
fh-tracker: $(SRC) $(HDR)
|
||||
$(CXX) $(CXXFLAGS) -o $@ $(SRC) $(LDLIBS)
|
||||
|
||||
fh-replay: replay.cpp calib.cpp nets.cpp tracker.cpp record.cpp pinch.cpp io.cpp $(HDR)
|
||||
$(CXX) $(CXXFLAGS) -o $@ replay.cpp calib.cpp nets.cpp tracker.cpp record.cpp pinch.cpp io.cpp $(LDLIBS)
|
||||
|
||||
fh-ringplay: ringplay.cpp record.h ../camd/fhring.h
|
||||
$(CXX) $(CXXFLAGS) -o $@ ringplay.cpp
|
||||
|
||||
clean:
|
||||
rm -f fh-tracker fh-replay fh-ringplay nettest
|
||||
|
||||
.PHONY: all clean
|
||||
|
||||
nettest: nettest.cpp nets.cpp nets.h geom.h
|
||||
$(CXX) $(CXXFLAGS) -o $@ nettest.cpp nets.cpp $(LDLIBS)
|
||||
@@ -1,86 +0,0 @@
|
||||
# trackd
|
||||
|
||||
`fh-tracker` is the C++ version of `tracker/live.py`. It uses the same scheduling and the same models. The models run on a few threads, with no Python in the loop. It reads `fh-camd`'s ring and publishes hands to `$XDG_RUNTIME_DIR/frame-hands/hands` (`include/fh_hands.h`), where Frametop's ft-screens picks them up for the hand cutouts.
|
||||
|
||||
```
|
||||
sudo camd/fh-camd &
|
||||
trackd/fh-tracker # status every 5 s; Ctrl+C to stop
|
||||
trackd/fh-tracker --int8 # the 8-bit models (models/ncnn/*-int8.ncnn.*)
|
||||
```
|
||||
|
||||
Options:
|
||||
|
||||
- `--threads N`: model threads, pinned to the `--cpus` list. Default 3.
|
||||
- `--cpus LIST`: CPUs for the model threads and the main loop. Default `5,6,7`. With the headset on, that ran a step in 8.4 ms against 13.2 ms on `2,3,4`, where XRService's head tracking also runs, and SteamVR's frame timing didn't change (`probes/core_ab.py`, 2026-09-29).
|
||||
- `--contrast MODE` or `PALM/HAND`: how crops are equalized before the models see them: `clahe[:CLIP]`, `none`, or `stretch` (1st-99th percentile). Default `clahe:2/none`. In the dim recording, CLAHE let the palm search find about 10% more hands, but it made the landmarks jitter more (published median 6.9 mm, against 6.0 mm with plain landmark crops).
|
||||
- `--swap-sides`: swap the two side cameras (`slam_left`, `slam_right`). fh-camd tells their buffers apart by XRService's allocation order, and after some XRService restarts that order is reversed. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. With the headset on and looking at a room with some texture, `tools/check_sides.py --ring` tells whether the names are right (exit 0), swapped (exit 3), or it can't tell (exit 2).
|
||||
- `--seconds N`: stop after N seconds.
|
||||
- `--status S`: how often to print status, in seconds.
|
||||
- `--models DIR`: where the models are.
|
||||
- `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
|
||||
- `--no-publish`: don't write the hands file.
|
||||
- `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x fh-tracker`) starts a recording in `captures/rec-<time>` without a restart.
|
||||
- `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`; SIGUSR1 would reach both trackers. With `fh-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate.
|
||||
- `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.
|
||||
- `--ring PATH`: read frames from another ring, such as `fh-ringplay`'s.
|
||||
- Recording from a ring with color cameras (`fh-camd --with-color`) also saves each color camera's newest frame with every set, as `color_video<N>`. That adds about 70 MB/s. Run the recorder at normal I/O priority (not under `frame-job`, whose `ionice -c 3` stalled a 165 MB/s recording).
|
||||
|
||||
The status line also says how often a hand was on each side (by where the wrist is), and why views and hands came and went: views lost (the landmark model stopped seeing the hand), handoff misses (a crop projected from the hand's 3D position found nothing), duplicates, splits (two views disagreed in 3D), and hands created, merged and forgotten.
|
||||
|
||||
## Beyond the Python tracker
|
||||
|
||||
fh-tracker started as a port of `tracker/hands.py`. Replaying recordings (below) showed where it went wrong, and it now differs in these ways:
|
||||
|
||||
- **Pairing views across cameras.** The side cameras sit side by side, so two hands next to each other at the same height fall on the same epipolar lines, and rays to two different hands can nearly meet close to the cameras. That made phantom hands 12-15 cm in front of the eyes, which tore holes through the screens. Each step now scores every way of pairing the views in two cameras and keeps the best. A pair scores well when its rays meet, when each view's apparent size matches the triangulated distance, and when the model calls both the same hand. The size check uses a fixed prior: with the model's average hand, clean pairs measure 0.71-1.51 times the one-view distance, and mismatched pairs mostly far less. It doesn't use the learned hand size, which bad pairs had corrupted.
|
||||
- **One view.** From how big the hand looks, its distance is off by 10-30% and wanders about 10% between frames. So a hand that drops to one camera keeps its last distance and drifts toward the one-view guess by 10% a frame.
|
||||
- **Smoothing.** The published landmarks go through a One Euro filter: it smooths hard while the hand is still (tracking noise is several mm per frame) and hardly at all while it moves fast. The palm speed that sets the update rate (15 or 30 Hz) is the filtered one; the raw speed read about 0.25 m/s from noise alone.
|
||||
- **Capsules.** Forearms follow the hand's own axis, and nothing within 12 cm in front of the eyes is published.
|
||||
|
||||
## Pinch
|
||||
|
||||
For input, the Vision Pro way: look at something and pinch to click, pinch and move to drag, with the eye tracker doing the looking. fh-tracker detects a pinch per hand (`pinch.h`) and publishes it to `$XDG_RUNTIME_DIR/frame-hands/gestures`, next to the hands file. The layout, and how to read it without missing quick taps, is in `include/fh_gestures.h`.
|
||||
|
||||
- A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost). While a pinch is down or closing, the tracker runs at the full 30 Hz.
|
||||
- The distance comes from MediaPipe's world landmarks (the model's own 3D hand pose, averaged over the hand's views, at the user's hand size). `--pinch-triangulated` uses the triangulated tips instead. On the two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. Typing still gave 2 pinches a minute, so a consumer should only act on a pinch while the gaze is on a target.
|
||||
- The pinch point is midway between the thumb and index tips. A drag is the pinch point now, minus where it was when the pinch began, both turned into the room with the HMD pose at their capture times.
|
||||
- `tools/watch_gestures.py` prints begins, ends and drag offsets live, and `--distance` prints each hand's distance. `fh-replay` runs the same detector and reports pinch counts. Its `--timeline` gets each begin, end and lost event, and both distance measures per set.
|
||||
|
||||
Frametop's pointer helper is the natural consumer. Its gaze mode already treats a press as "stop where the gaze put it, drag onto the target, click on release", and "hold still for half a second, then move" as a drag. A pinch begin would be the press, the end the release, and the pinch point's movement the drag.
|
||||
|
||||
## Replay
|
||||
|
||||
`fh-replay DIR` runs a recording through the tracker with the live scheduling and reports how well it kept the hands: hands per set, left and right coverage, track lengths, and the same reasons as the status line.
|
||||
|
||||
```
|
||||
trackd/fh-replay captures/rec-20260929-120000 --oracle 10 --timeline /tmp/tl.txt
|
||||
```
|
||||
|
||||
- `--oracle N`: every N-th set, also search every tile of every camera, and report how often the tracker had the hands that full search could find.
|
||||
- `--slow F`: live, the tracker skips sets that arrive while it's busy. Replay counts each step's time times F as busy (default 1; the headset is busier live).
|
||||
- `--timeline FILE`: a line per processed set and hand.
|
||||
- `--cams mono|color|all`: which cameras to track with (default `mono`). `color` tracks with the Arcturus pair alone, for comparing it with the IR cameras on the same recording. It needs a recording made with `fh-camd --with-color`. `--color-left NODE` (`color_video0` or `color_video3`) and `--color-crop subtract|none` say how the module's calibration maps onto the images; `tools/check_color.py` finds out. Color frames repeat across sets (the newest one is saved with each), and repeats are skipped.
|
||||
- `--depth FILE`: a line per hand per processed set for `tools/depth_report.py`, which measures the depth without ground truth. It reports how the hands were seen (by the two lower cameras, a lower and an upper one, or one camera), the noise along the line of sight against across it, each camera's one-view distance against the triangulated one, and what a camera dropping out would do to the distance.
|
||||
|
||||
## Playing a recording live
|
||||
|
||||
`fh-ringplay DIR --ring PATH [--from S] [--to S] [--loop]` publishes a recording into a ring file in real time, as fh-camd would, so `fh-tracker --ring PATH --no-publish` runs the same frames run after run. It needs no root, and it skips the dark frames. `probes/core_ab.py` uses it to compare CPU placements (A: CPUs 2-4, B: 5-7, C: no tracker), with the headset on so SteamVR's compositor is running.
|
||||
|
||||
## Build
|
||||
|
||||
ncnn is built from source into `vendor/ncnn/build/install`. The build also provides the `ncnn2table` and `ncnn2int8` quantization tools.
|
||||
|
||||
```
|
||||
git clone --depth 1 --branch 20260526 https://github.com/Tencent/ncnn.git vendor/ncnn
|
||||
cd vendor/ncnn && mkdir build && cd build
|
||||
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DNCNN_VULKAN=OFF -DNCNN_BUILD_TOOLS=ON -DNCNN_SIMPLEOCV=ON \
|
||||
-DNCNN_BUILD_EXAMPLES=OFF -DNCNN_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$PWD/install ..
|
||||
nice ninja install
|
||||
cd ../../../trackd && make # fh-tracker and fh-replay; `make nettest` for the model check
|
||||
```
|
||||
|
||||
It needs jsoncpp for the calibration files (the host has it).
|
||||
|
||||
## Checks
|
||||
|
||||
- `tools/nettest_compare.py CAPTURE` runs the C++ model code (`nettest`) and `tracker/models.py` on the same crops from a recording, and compares the palms, ROIs and landmarks. Add `--int8` to check the 8-bit models.
|
||||
- `tools/make_int8_calib.py CAPTURE OUT` cuts the calibration crops that `ncnn2table` needs to quantize the models.
|
||||
@@ -2,6 +2,7 @@
|
||||
#include "calib.h"
|
||||
|
||||
#include <json/json.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
@@ -91,10 +92,13 @@ V2 Camera::project(V3 head, double *depth) const {
|
||||
|
||||
double Camera::off_axis(V2 uv) const { return std::acos(std::clamp(unproject(uv)[2], -1.0, 1.0)) * 180 / M_PI; }
|
||||
|
||||
// Off the Frame (frame-job on the 7i), the /persist files are copies under FRAME_JOB_DEVICE_ROOT.
|
||||
// A headset file such as /persist/xrservice.json. In the dev container the host's / is at
|
||||
// /run/host (distrobox doesn't mount /persist); off the Frame, FRAME_JOB_DEVICE_ROOT can
|
||||
// point at a folder with copies of them.
|
||||
static std::string device_path(const char *path) {
|
||||
const char *root = std::getenv("FRAME_JOB_DEVICE_ROOT");
|
||||
return root ? std::string(root) + path : std::string(path);
|
||||
if (const char *root = std::getenv("FRAME_JOB_DEVICE_ROOT")) return std::string(root) + path;
|
||||
const std::string host = std::string("/run/host") + path;
|
||||
return access(path, R_OK) != 0 && access(host.c_str(), R_OK) == 0 ? host : path;
|
||||
}
|
||||
|
||||
bool load_calibration(std::map<std::string, Camera> &out, std::string &err) {
|
||||
|
||||
+3
-3
@@ -1,4 +1,4 @@
|
||||
// Tracking-camera calibration from the headset's factory files (see tracker/calib.py
|
||||
// Tracking-camera calibration from the headset's factory files (see tools/calib.py
|
||||
// for the conventions): Kannala-Brandt fisheye intrinsics, and each camera's pose in the
|
||||
// head frame (OpenVR's: +x right, +y up, -z forward), metres.
|
||||
#pragma once
|
||||
@@ -27,8 +27,8 @@ struct Camera {
|
||||
// name: slam_left, slam_right, upper_left, upper_right.
|
||||
bool load_calibration(std::map<std::string, Camera> &out, std::string &err);
|
||||
|
||||
// The Arcturus color cameras (tracker/calib.py load_color has the conventions), for
|
||||
// fh-camd --with-color's images: luma at 1/scale size, recorded as color_video<N>. They're
|
||||
// The Arcturus color cameras (tools/calib.py load_color has the conventions), for
|
||||
// ft-camd --with-color's images: luma at 1/scale size, recorded as color_video<N>. They're
|
||||
// keyed by those recorded names: left_node is passthrough_left, right_node
|
||||
// passthrough_right. crop_subtract: image x = sensor x - the calibration's cropRegion.x.
|
||||
// tools/check_color.py tells which node is which and which crop reading fits.
|
||||
|
||||
+9
-7
@@ -39,7 +39,7 @@ bool Ring::open(const char *path, std::string &err) {
|
||||
map_ = static_cast<const uint8_t *>(m);
|
||||
hdr_ = reinterpret_cast<const fh_ring_hdr_t *>(map_);
|
||||
if (std::memcmp(hdr_->magic, FH_RING_MAGIC, 8) || hdr_->version != FH_RING_VERSION || hdr_->file_bytes > len_)
|
||||
return err = std::string(path) + " is not an fh-camd ring", false;
|
||||
return err = std::string(path) + " is not an ft-camd ring", false;
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -81,7 +81,7 @@ bool Ring::meta(int i, uint64_t n, fh_ring_slot_t *meta) const {
|
||||
|
||||
namespace {
|
||||
|
||||
// The hand's shape to cut out, as capsules (tracker/publish.py has the same model).
|
||||
// The hand's shape to cut out, as capsules.
|
||||
// Radii are a real hand's half-widths plus a small margin for tracking noise.
|
||||
const int kThumb[][2] = {{0, 1}, {1, 2}, {2, 3}, {3, 4}};
|
||||
const int kFingers[][2] = {{5, 6}, {6, 7}, {7, 8}, {9, 10}, {10, 11}, {11, 12}, {13, 14}, {14, 15}, {15, 16},
|
||||
@@ -113,12 +113,14 @@ void put(fh_capsule_t *caps, uint32_t &n, V3 a, V3 b, double ra, double rb) {
|
||||
|
||||
} // namespace
|
||||
|
||||
bool Publisher::open(std::string &err) {
|
||||
const char *run = std::getenv("XDG_RUNTIME_DIR");
|
||||
const std::string dir = std::string(run ? run : "/run/user/" + std::to_string(getuid())) + "/frame-hands";
|
||||
std::string run_dir() {
|
||||
const std::string dir = "/run/user/" + std::to_string(getuid()) + "/frametop";
|
||||
mkdir(dir.c_str(), 0700);
|
||||
chmod(dir.c_str(), 0700);
|
||||
const std::string path = dir + "/hands";
|
||||
return dir;
|
||||
}
|
||||
|
||||
bool Publisher::open(std::string &err) {
|
||||
const std::string path = run_dir() + "/hands";
|
||||
const int fd = ::open(path.c_str(), O_RDWR | O_CREAT | O_NOFOLLOW | O_CLOEXEC, 0600);
|
||||
if (fd < 0 || ftruncate(fd, sizeof(fh_hands_t)) < 0) return err = path + ": " + std::strerror(errno), false;
|
||||
void *m = mmap(nullptr, sizeof(fh_hands_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
|
||||
|
||||
+6
-1
@@ -1,4 +1,4 @@
|
||||
// Frames in from fh-camd's ring (camd/fhring.h), hands out to the hands file
|
||||
// Frames in from ft-camd's ring (camd/fhring.h), hands out to the hands file
|
||||
// (include/fh_hands.h, read by Frametop's ft-screens).
|
||||
#pragma once
|
||||
|
||||
@@ -44,3 +44,8 @@ private:
|
||||
|
||||
uint64_t mono_ns();
|
||||
int64_t raw_minus_mono_ns(); // camera timestamps are CLOCK_MONOTONIC_RAW
|
||||
|
||||
// /run/user/UID/frametop, created private to the user if it's missing: where ft-camd's ring
|
||||
// (FH_RING_NAME) and the hands and gestures files live. Not $XDG_RUNTIME_DIR: a terminal in
|
||||
// the Frametop desktop has a private one of its own.
|
||||
std::string run_dir();
|
||||
+66
-28
@@ -1,9 +1,12 @@
|
||||
// fh-tracker: hands in 3D from fh-camd's ring, published for Frametop's ft-screens.
|
||||
// The C++ version of tracker/live.py: the same scheduling, with the models on a few
|
||||
// threads and no Python in the loop.
|
||||
// ft-hands: hands in 3D from ft-camd's ring, published for Frametop's ft-screens (the hand
|
||||
// cutouts), and pinches for the pointer. It started as a port of frame-hands' Python
|
||||
// prototype: the same scheduling, with the models on a few threads.
|
||||
//
|
||||
// fh-tracker [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N]
|
||||
// [--no-publish] [--record DIR] [--swap-sides] ... (--help lists them all)
|
||||
// ft-hands [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N]
|
||||
// [--no-publish] [--record DIR] [--swap-sides] ... (--help lists them all)
|
||||
//
|
||||
// Settings in ~/.config/frametop.conf (FT_<name> in the environment overrides them, and
|
||||
// options override both): HANDS_SWAP_SIDES (1: as --swap-sides), HANDS_CPUS (as --cpus).
|
||||
#include "io.h"
|
||||
#include "pinch.h"
|
||||
#include "record.h"
|
||||
@@ -40,6 +43,45 @@ const char *camera_for_pipe(int node) {
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
// A setting from ~/.config/frametop.conf, or FT_<key> from the environment; "" if unset.
|
||||
std::string setting(const std::string &key) {
|
||||
if (const char *v = std::getenv(("FT_" + key).c_str())) return v;
|
||||
const char *home = std::getenv("HOME");
|
||||
std::ifstream in(std::string(home ? home : "") + "/.config/frametop.conf");
|
||||
std::string line, value;
|
||||
auto trim = [](std::string s) {
|
||||
s.erase(0, s.find_first_not_of(" \t\"'"));
|
||||
s.erase(s.find_last_not_of(" \t\"'") + 1);
|
||||
return s;
|
||||
};
|
||||
while (std::getline(in, line)) {
|
||||
line = line.substr(0, line.find('#'));
|
||||
const auto eq = line.find('=');
|
||||
if (eq != std::string::npos && trim(line.substr(0, eq)) == key) value = trim(line.substr(eq + 1));
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
std::vector<int> parse_cpus(const char *p) {
|
||||
std::vector<int> out;
|
||||
while (*p) {
|
||||
char *end;
|
||||
const long c = std::strtol(p, &end, 10);
|
||||
if (end == p) break;
|
||||
out.push_back(int(c));
|
||||
p = *end == ',' ? end + 1 : end;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Where SIGUSR1 puts recordings: $XDG_DATA_HOME/frametop/hands (~/.local/share/...).
|
||||
std::string recordings_dir() {
|
||||
const char *data = std::getenv("XDG_DATA_HOME"), *home = std::getenv("HOME");
|
||||
std::string dir = data && *data ? data : std::string(home ? home : "") + "/.local/share";
|
||||
for (const char *part : {"/frametop", "/hands"}) mkdir((dir += part).c_str(), 0700);
|
||||
return dir;
|
||||
}
|
||||
|
||||
double cpu_seconds() {
|
||||
rusage r;
|
||||
getrusage(RUSAGE_SELF, &r);
|
||||
@@ -53,13 +95,15 @@ int main(int argc, char **argv) {
|
||||
int threads = 3, niceness = 5;
|
||||
bool int8 = false, publish = true, track = true, swap_sides = false;
|
||||
std::string models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
|
||||
std::string record, ring_path = FH_RING_PATH;
|
||||
std::string record, ring_path = "/run/user/" + std::to_string(getuid()) + "/" FH_RING_NAME;
|
||||
// SteamOS starts user processes on CPUs 0-4 and keeps 5-7 (two A720s and the X4) for
|
||||
// SteamVR's compositor, whose threads there run at real-time priority, so they always
|
||||
// win. XRService pins its head tracking to 2-3. probes/core_ab.py (2026-09-29, headset
|
||||
// on, 3 rounds): on 5-7 a step took 8.4 ms against 13.2 on 2-4, latency 9.6 against
|
||||
// 14.1 ms, and the compositor's late frames and CPU/GPU time didn't change.
|
||||
// win. XRService pins its head tracking to 2-3. frame-hands' probes/core_ab.py
|
||||
// (2026-09-29, headset on, 3 rounds): on 5-7 a step took 8.4 ms against 13.2 on 2-4,
|
||||
// latency 9.6 against 14.1 ms, and the compositor's late frames and CPU/GPU time didn't change.
|
||||
std::vector<int> cpus = {5, 6, 7};
|
||||
if (const auto c = parse_cpus(setting("HANDS_CPUS").c_str()); !c.empty()) cpus = c;
|
||||
swap_sides = setting("HANDS_SWAP_SIDES") == "1";
|
||||
// How crops are equalized. CLAHE helps the palm search find hands (about 10% more in the
|
||||
// dim recording), but makes the landmarks jitter, so they get plain crops.
|
||||
Contrast palm_contrast, hand_contrast{Contrast::None};
|
||||
@@ -89,24 +133,20 @@ int main(int argc, char **argv) {
|
||||
if (!Contrast::parse_pair(argv[++i], palm_contrast, hand_contrast))
|
||||
return std::fprintf(stderr, "--contrast MODE or PALM/HAND, each clahe[:CLIP]|none|stretch\n"), 1;
|
||||
} else if (a == "--cpus" && more) {
|
||||
cpus.clear();
|
||||
for (char *p = argv[++i]; *p;) {
|
||||
cpus.push_back(int(std::strtol(p, &p, 10)));
|
||||
if (*p == ',') ++p;
|
||||
else if (*p) break;
|
||||
}
|
||||
cpus = parse_cpus(argv[++i]);
|
||||
if (cpus.empty()) cpus = {5, 6, 7};
|
||||
}
|
||||
else {
|
||||
std::printf("usage: %s [--seconds N] [--threads N] [--int8] [--status S] [--models DIR] [--nice N] [--no-publish]\n"
|
||||
" [--record DIR] [--record-for S] [--record-only] [--cpus 5,6,7] [--swap-sides]\n"
|
||||
" [--keep-presence P] (0.5) [--ring PATH] (fh-camd's, or fh-ringplay's)\n"
|
||||
" [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
|
||||
" [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated]\n"
|
||||
" [--contrast MODE|PALM/HAND] (clahe[:CLIP], none, stretch; default clahe:2/none)\n"
|
||||
"Recording saves every frame set for S seconds (120) to DIR/sets.bin, for fh-replay; SIGUSR1\n"
|
||||
"starts one in captures/rec-<time> next to trackd. --record-only records without tracking, so it\n"
|
||||
"can run beside a tracking fh-tracker. With fh-camd --with-dark, recordings also get each\n"
|
||||
"camera's newest dark frame, as <name>_dk; with --with-color, the color cameras' as color_video<N>.\n",
|
||||
"Recording saves every frame set for S seconds (120) to DIR/sets.bin, for ft-handreplay; SIGUSR1\n"
|
||||
"starts one in ~/.local/share/frametop/hands/rec-<time>. --record-only records without tracking, so it\n"
|
||||
"can run beside a tracking ft-hands. With ft-camd --with-dark, recordings also get each\n"
|
||||
"camera's newest dark frame, as <name>_dk; with --with-color, the color cameras' as color_video<N>.\n"
|
||||
"Settings in ~/.config/frametop.conf: HANDS_SWAP_SIDES=1, HANDS_CPUS=5,6,7 (FT_<name> overrides).\n",
|
||||
argv[0]);
|
||||
return a == "--help" ? 0 : 1;
|
||||
}
|
||||
@@ -138,10 +178,10 @@ int main(int argc, char **argv) {
|
||||
std::fprintf(stderr, "%s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
if (!ring.alive()) return std::fprintf(stderr, "fh-camd isn't running (no heartbeat)\n"), 1;
|
||||
if (!ring.alive()) return std::fprintf(stderr, "ft-camd isn't running (no heartbeat)\n"), 1;
|
||||
|
||||
std::map<std::string, int> index; // calibration name -> ring camera
|
||||
// Recorded only, not tracked: "<name>_dk" (fh-camd --with-dark) and "color_video<N>"
|
||||
// Recorded only, not tracked: "<name>_dk" (ft-camd --with-dark) and "color_video<N>"
|
||||
// (--with-color; which is left and right is up to tools/check_color.py). Recorded names
|
||||
// hold 15 characters, so "upper_right_dark" wouldn't fit.
|
||||
std::map<std::string, int> dark;
|
||||
@@ -151,14 +191,14 @@ int main(int argc, char **argv) {
|
||||
dark["color_video" + std::to_string(ring.camera(i).node)] = i;
|
||||
continue;
|
||||
}
|
||||
// fh-camd's cameras by capture pipe; fh-ringplay's (no device) by the name it gives
|
||||
// ft-camd's cameras by capture pipe; ft-ringplay's (no device) by the name it gives
|
||||
const char *name = camera_for_pipe(ring.camera(i).node);
|
||||
if (!name && ring.camera(i).node < 0) name = ring.camera(i).name;
|
||||
if (!name || !calib.count(name)) continue;
|
||||
if (ring.camera(i).flags & FH_CAM_DARK) dark[std::string(name) + "_dk"] = i;
|
||||
else index[name] = i, used[name] = calib[name];
|
||||
}
|
||||
// fh-camd tells the side cameras' buffers apart by XRService's allocation order, which
|
||||
// ft-camd tells the side cameras' buffers apart by XRService's allocation order, which
|
||||
// some XRService restarts reverse; tools/check_sides.py --ring tells when.
|
||||
if (swap_sides && index.count("slam_left") && index.count("slam_right")) {
|
||||
std::swap(index["slam_left"], index["slam_right"]);
|
||||
@@ -189,7 +229,7 @@ int main(int argc, char **argv) {
|
||||
int resid_n = 0, left_sets = 0, right_sets = 0, both_sets = 0;
|
||||
|
||||
while (!g_stop && (seconds <= 0 || (mono_ns() - start) / 1e9 < seconds)) {
|
||||
if (!ring.alive()) return std::fprintf(stderr, "fh-camd stopped\n"), 2;
|
||||
if (!ring.alive()) return std::fprintf(stderr, "ft-camd stopped\n"), 2;
|
||||
// a new frame set: every camera has a newer frame, taken at the same moment
|
||||
std::map<std::string, uint64_t> latest;
|
||||
bool ready = true;
|
||||
@@ -238,9 +278,7 @@ int main(int argc, char **argv) {
|
||||
char name[64];
|
||||
const std::time_t now = std::time(nullptr);
|
||||
std::strftime(name, sizeof name, "rec-%Y%m%d-%H%M%S", std::localtime(&now));
|
||||
const std::string here = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1);
|
||||
const std::string dir = here + "../captures";
|
||||
mkdir(dir.c_str(), 0755);
|
||||
const std::string dir = recordings_dir();
|
||||
std::string e;
|
||||
if (!start_recording(dir + "/" + name, e)) std::fprintf(stderr, "%s\n", e.c_str());
|
||||
}
|
||||
|
||||
+1
-2
@@ -1,5 +1,4 @@
|
||||
// MediaPipe's palm detector and hand landmark model on ncnn (see tracker/models.py for
|
||||
// the conventions). A crop is a square region of a camera image: centre and size in
|
||||
// MediaPipe's palm detector and hand landmark model on ncnn. A crop is a square region of a camera image: centre and size in
|
||||
// pixels, and a rotation that turns the crop's "up" toward the image direction
|
||||
// (sin r, -cos r). Crops are contrast-equalized (CLAHE) before the models see them.
|
||||
// Everything here may run on several threads at once.
|
||||
|
||||
@@ -99,11 +99,7 @@ bool Pinch::engaged() const {
|
||||
}
|
||||
|
||||
bool GesturePublisher::open(const Pinch &pinch, std::string &err) {
|
||||
const char *run = std::getenv("XDG_RUNTIME_DIR");
|
||||
const std::string dir = std::string(run ? run : "/run/user/" + std::to_string(getuid())) + "/frame-hands";
|
||||
mkdir(dir.c_str(), 0700);
|
||||
chmod(dir.c_str(), 0700);
|
||||
const std::string path = dir + "/gestures";
|
||||
const std::string path = run_dir() + "/gestures";
|
||||
const int fd = ::open(path.c_str(), O_RDWR | O_CREAT | O_NOFOLLOW | O_CLOEXEC, 0600);
|
||||
if (fd < 0 || ftruncate(fd, sizeof(fh_gestures_t)) < 0) return err = path + ": " + std::strerror(errno), false;
|
||||
void *m = mmap(nullptr, sizeof(fh_gestures_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
|
||||
|
||||
+1
-1
@@ -58,7 +58,7 @@ private:
|
||||
int64_t seen_ns_[2] = {0, 0};
|
||||
};
|
||||
|
||||
// Writes $XDG_RUNTIME_DIR/frame-hands/gestures.
|
||||
// Writes /run/user/UID/frametop/gestures.
|
||||
class GesturePublisher {
|
||||
public:
|
||||
bool open(const Pinch &pinch, std::string &err);
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// Recordings of frame sets, for replaying live sessions through the tracker offline
|
||||
// (fh-replay). A recording is DIR/sets.bin: one record per frame set, each
|
||||
// (ft-handreplay). A recording is DIR/sets.bin: one record per frame set, each
|
||||
// fh_set_hdr_t, then per camera fh_set_cam_t, then each camera's pixels (w x h, packed)
|
||||
// in the same camera order.
|
||||
#pragma once
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// fh-replay: run a recording (fh-tracker --record) through the tracker offline, with the
|
||||
// ft-handreplay: run a recording (ft-hands --record) through the tracker offline, with the
|
||||
// live scheduling, and report how well it kept the hands.
|
||||
//
|
||||
// fh-replay DIR [--oracle N] [--slow F] [--timeline FILE] [--threads N] [--models DIR]
|
||||
// ft-handreplay DIR [--oracle N] [--slow F] [--timeline FILE] [--threads N] [--models DIR]
|
||||
// [--from S] [--to S] [--contrast MODE|PALM/HAND] (clahe[:CLIP], none, stretch)
|
||||
//
|
||||
// --oracle N: every N-th set, also search every tile of every camera (slow), to see
|
||||
@@ -13,14 +13,14 @@
|
||||
// --timeline: per processed set, a line per hand (time, id, side, views, wrist) and per view
|
||||
// (hand, camera, presence, next crop, set index).
|
||||
// --keep-presence P: landmark presence a tracked view needs to stay (default 0.5, as new ones).
|
||||
// --pinch-begin M, --pinch-end M, --pinch-triangulated: the pinch detector (trackd/pinch.h);
|
||||
// --pinch-begin M, --pinch-end M, --pinch-triangulated: the pinch detector (track/pinch.h);
|
||||
// the timeline gets its begin/end/lost events and both distance measures per set.
|
||||
// --cams mono|color|all: which cameras to track with (default mono). color and all need a
|
||||
// recording made with fh-camd --with-color; --color-left NODE (color_video0 or
|
||||
// recording made with ft-camd --with-color; --color-left NODE (color_video0 or
|
||||
// color_video3) and --color-crop subtract|none say how its calibration maps
|
||||
// (tools/check_color.py).
|
||||
// --contrast: how the palm search's and the landmark model's crops are equalized
|
||||
// (default clahe:2/none, as fh-tracker).
|
||||
// (default clahe:2/none, as ft-hands).
|
||||
// --depth FILE: per processed set, a line per hand for tools/depth_report.py: its views'
|
||||
// cameras, triangulation residual, hand scale, measured and published palm, and
|
||||
// each view's one-view palm (Tracker::single_view at the hand's scale). The
|
||||
@@ -66,11 +66,11 @@ int main(int argc, char **argv) {
|
||||
int oracle = 0, threads = 2;
|
||||
double slow = 1.0, from = 0, to = 1e9;
|
||||
bool cost = false;
|
||||
Contrast palm_contrast, hand_contrast{Contrast::None}; // as fh-tracker's
|
||||
Contrast palm_contrast, hand_contrast{Contrast::None}; // as ft-hands's
|
||||
double keep_presence = 0.5; // landmark presence a tracked view needs to stay
|
||||
PinchParams pinch_params;
|
||||
std::string use = "mono", color_left = "color_video0", color_crop = "subtract";
|
||||
std::string timeline, depth, models =std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
|
||||
std::string timeline, depth, models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
|
||||
for (int i = 2; i < argc; ++i) {
|
||||
const std::string a = argv[i];
|
||||
const bool more = i + 1 < argc;
|
||||
@@ -113,7 +113,7 @@ int main(int argc, char **argv) {
|
||||
std::vector<std::string> nodes;
|
||||
for (auto &c : cams)
|
||||
if (std::string(c.name).rfind("color_video", 0) == 0) nodes.push_back(c.name);
|
||||
if (nodes.size() != 2) return std::fprintf(stderr, "%s: no color cameras (fh-camd --with-color)\n", dir.c_str()), 1;
|
||||
if (nodes.size() != 2) return std::fprintf(stderr, "%s: no color cameras (ft-camd --with-color)\n", dir.c_str()), 1;
|
||||
const std::string right = nodes[0] == color_left ? nodes[1] : nodes[0];
|
||||
if (!load_color_calibration(calib, color_left, right, color_crop == "subtract", 2, err))
|
||||
return std::fprintf(stderr, "%s\n", err.c_str()), 1;
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
// fh-ringplay: play a recording (fh-tracker --record) into a frame ring in real time, the
|
||||
// way fh-camd publishes live cameras, so fh-tracker --ring PATH processes the same frames
|
||||
// run after run. For A/B tests of how the tracker runs (probes/core-ab.sh).
|
||||
// ft-ringplay: play a recording (ft-hands --record) into a frame ring in real time, the
|
||||
// way ft-camd publishes live cameras, so ft-hands --ring PATH processes the same frames
|
||||
// run after run. For A/B tests of how the tracker runs.
|
||||
//
|
||||
// fh-ringplay DIR --ring PATH [--from S] [--to S] [--loop] [--cpus 0,1]
|
||||
// ft-ringplay DIR --ring PATH [--from S] [--to S] [--loop] [--cpus 0,1]
|
||||
//
|
||||
// Frames are stamped as they're published, so the tracker's latency figures stay
|
||||
// meaningful. Cameras carry their calibration name and no device node (fh-tracker maps
|
||||
// meaningful. Cameras carry their calibration name and no device node (ft-hands maps
|
||||
// them by name), so a recording made with the right names needs no --swap-sides.
|
||||
// Dark frames (<name>_dk) are skipped. Needs no root: the ring is an ordinary file.
|
||||
#include "record.h"
|
||||
@@ -139,7 +139,7 @@ int main(int argc, char **argv) {
|
||||
while (have && set_time(cams) - rec0 < from && !g_stop) have = in.next(cams, px, want);
|
||||
if (!have) return std::fprintf(stderr, "%s: nothing after %.1f s\n", dir.c_str(), from), 1;
|
||||
|
||||
// the ring: the recording's bright cameras, as fh-camd lays them out
|
||||
// the ring: the recording's bright cameras, as ft-camd lays them out
|
||||
std::vector<int> pub; // set camera index of each ring camera
|
||||
for (size_t k = 0; k < cams.size() && pub.size() < FH_RING_MAX_CAMS; ++k)
|
||||
if (!is_dark(cams[k].name)) pub.push_back(int(k));
|
||||
@@ -162,7 +162,7 @@ int main(int argc, char **argv) {
|
||||
for (size_t r = 0; r < pub.size(); ++r) {
|
||||
const fh_set_cam_t &c = cams[pub[r]];
|
||||
fh_ring_cam_t &rc = hdr->cams[r];
|
||||
std::snprintf(rc.sensor, sizeof rc.sensor, "fh-ringplay");
|
||||
std::snprintf(rc.sensor, sizeof rc.sensor, "ft-ringplay");
|
||||
std::snprintf(rc.name, sizeof rc.name, "%s", c.name);
|
||||
rc.node = -1;
|
||||
rc.format = FH_FMT_GREY8;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// Multi-camera hand tracking (a port of tracker/hands.py; the scheduling is described
|
||||
// there and in tracker/README.md). All 3D is metres in the head frame.
|
||||
// Multi-camera hand tracking (a port of frame-hands' Python prototype; the scheduling is
|
||||
// described in hands/README.md). All 3D is metres in the head frame.
|
||||
#pragma once
|
||||
|
||||
#include "calib.h"
|
||||
@@ -84,7 +84,7 @@ public:
|
||||
size_t views() const { return views_.size(); }
|
||||
std::vector<Seen> views_now() const;
|
||||
// Every search tile in every camera, then landmarks on every palm: slow; for checking
|
||||
// what the scheduler misses (fh-replay --oracle).
|
||||
// what the scheduler misses (ft-handreplay --oracle).
|
||||
std::vector<Seen> exhaustive(const std::map<std::string, Image> &images);
|
||||
// Landmark presence a tracked view needs to stay (new views need min presence, 0.5). In
|
||||
// bright rooms the camera exposes for the room, the hands come out dim, and presence
|
||||
|
||||
+17
-10
@@ -1,8 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install everything on the Steam Frame: the build container, Frametop (multi-screen
|
||||
# desktop, input relay, universal 3D mouse, settings app), and optionally the Bluetooth
|
||||
# fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run, for
|
||||
# example after `git pull`.
|
||||
# fixes and hand tracking. Run it on the headset in a terminal, from this repo. It's safe
|
||||
# to re-run, for example after `git pull`.
|
||||
# (It also works from a PC over SSH; see "Developing from a PC" in the README.)
|
||||
#
|
||||
# Usage: ./install.sh [--yes] [--no-bluetooth]
|
||||
@@ -45,7 +45,7 @@ else
|
||||
"$root/scripts/sync.sh" >/dev/null
|
||||
fi
|
||||
|
||||
step "1/8 distrobox (container tool, installed in your home folder)"
|
||||
step "1/9 distrobox (container tool, installed in your home folder)"
|
||||
if on_frame 'test -x ~/.local/bin/distrobox'; then
|
||||
echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')"
|
||||
else
|
||||
@@ -55,25 +55,25 @@ else
|
||||
cd ~/dev/src/distrobox && ./install --prefix ~/.local'
|
||||
fi
|
||||
|
||||
step "2/8 build container (Fedora 44 'dev', about 1-2 GB the first time)"
|
||||
step "2/9 build container (Fedora 44 'dev', about 1-2 GB the first time)"
|
||||
"$root/setup/dev-container.sh"
|
||||
|
||||
step "3/8 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
|
||||
step "3/9 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
|
||||
"$root/desktops.sh" relay install
|
||||
|
||||
step "4/8 3D mouse: SteamVR driver"
|
||||
step "4/9 3D mouse: SteamVR driver"
|
||||
"$root/pointer/driver/build.sh"
|
||||
"$root/pointer/driver/install.sh" install 2>&1 | grep -v xdg-open
|
||||
|
||||
step "5/8 3D mouse: pointer helper service"
|
||||
step "5/9 3D mouse: pointer helper service"
|
||||
"$root/pointer/helper/build.sh"
|
||||
"$root/pointer/helper/run.sh" install
|
||||
|
||||
step "6/8 power service (turns the displays off while the headset isn't used, even on a stand)"
|
||||
step "6/9 power service (turns the displays off while the headset isn't used, even on a stand)"
|
||||
"$root/power/build.sh"
|
||||
"$root/power/run.sh" install
|
||||
|
||||
step "7/8 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
|
||||
step "7/9 multi-screen desktop (ft-screens), Frametop Input Settings, and Frametop Display Settings"
|
||||
"$root/screens/build.sh"
|
||||
"$root/desktops.sh" install >/dev/null
|
||||
"$root/input-settings/install.sh"
|
||||
@@ -81,13 +81,20 @@ step "7/8 multi-screen desktop (ft-screens), Frametop Input Settings, and Framet
|
||||
on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
|
||||
echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
|
||||
|
||||
step "8/8 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
|
||||
step "8/9 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
|
||||
if [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; then
|
||||
"$root/setup/bluetooth/install.sh" install
|
||||
else
|
||||
echo "skipped. Install later with: setup/bluetooth/install.sh install"
|
||||
fi
|
||||
|
||||
step "9/9 hand tracking (optional, experimental: your hands show over the screens)"
|
||||
if ask "Install hand tracking? It needs your password (sudo) to let its camera service read the headset cameras." n; then
|
||||
"$root/hands/run.sh" install
|
||||
else
|
||||
echo "skipped. Install later with: hands/run.sh install"
|
||||
fi
|
||||
|
||||
step "Done"
|
||||
cat <<'EOF'
|
||||
SteamVR has to restart once, to load the 3D mouse driver and to start the input relay
|
||||
|
||||
@@ -30,6 +30,8 @@ POINTER_GAZE_RETAKE=5 # gaze mode: how far (degrees) you look away from the
|
||||
POINTER_GAZE_NUDGE_MAX=8 # gaze mode: a mouse nudge up to this far (degrees) before a click is learned as the eye tracker's error
|
||||
POINTER_GAZE_HOLD=0.5 # gaze mode: a press held this long (s) without moving becomes a real press (to drag); moved or released sooner, it clicks where you let go
|
||||
POINTER_GAZE_SHOW=1 # gaze mode: the dot shows this long (s) after the mouse moves it; also while a press is held, and a pulse per click
|
||||
HANDS_SWAP_SIDES=0 # hand tracking (hands/run.sh install): 1 = the side cameras' names are swapped, which some SteamVR restarts cause (hands/tools/check_sides.py --ring tells)
|
||||
HANDS_CPUS=5,6,7 # hand tracking: the CPUs its model threads run on
|
||||
META_DASHBOARD=0 # 1 = a Meta tap on a pass-through keyboard toggles the SteamVR dashboard (pointer mode only)
|
||||
SHARE_KEYS=0 # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen)
|
||||
DISPLAY_OFF_MIN=0 # minutes without use before frametop-power turns the displays off, even if the headset seems worn (0 = never)
|
||||
|
||||
@@ -18,6 +18,9 @@ packages=(
|
||||
mesa-libgbm-devel wayland-devel vulkan-loader-devel vulkan-headers plasma-wayland-protocols wlroots-devel
|
||||
# ft_pointer SteamVR driver: static C++ runtime (the host has an older glibc)
|
||||
libstdc++-static
|
||||
# hand tracking: ft-hands reads the calibration with jsoncpp; ft-camd runs on the host, linked
|
||||
# statically; the Python tools (hands/tools) need NumPy and OpenCV
|
||||
jsoncpp-devel glibc-static python3-numpy python3-opencv
|
||||
# Frametop Input Settings app (Kirigami, PySide6)
|
||||
python3-pyside6 kf6-kirigami kf6-qqc2-desktop-style qt6-qtwayland breeze-icon-theme plasma-breeze
|
||||
# Frametop remote desktop (VNC bridge through krdp)
|
||||
|
||||
Reference in new issue
Block a user