diff --git a/hands/Makefile b/hands/Makefile index 6130d8c..a606775 100644 --- a/hands/Makefile +++ b/hands/Makefile @@ -1,6 +1,7 @@ # Hand tracking, built into build/ (hands/build.sh runs this in the dev container): # make ft-camd (camd/: runs on the host, so linked statically) and ft-hands (track/) # make tools ft-handreplay and ft-ringplay, for recordings +# make check builds and runs the C++ unit tests (tests/sides_test.cpp) # The first build fetches ncnn (NCNN_TAG) and builds it into build/ncnn, which takes a few # minutes. NCNN=DIR uses an ncnn install already built instead. NCNN_TAG = 20260526 @@ -11,7 +12,7 @@ CXXFLAGS += -std=c++17 -fopenmp -I$(NCNN)/include/ncnn LDLIBS = $(NCNN)/lib/libncnn.a -ljsoncpp -fopenmp -lpthread CAMD = camd/camd.c camd/tp.c camd/xrcams.c -TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp +TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp track/sides.cpp HDR = $(wildcard track/*.h) camd/fhring.h include/fh_hands.h include/fh_gestures.h all: build/ft-camd build/ft-hands @@ -29,6 +30,13 @@ build/ft-handreplay: track/replay.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a @mkdir -p build $(CXX) $(CXXFLAGS) -o $@ track/replay.cpp $(TRACK) $(LDLIBS) +build/sides-test: tests/sides_test.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a + @mkdir -p build + $(CXX) $(CXXFLAGS) -o $@ tests/sides_test.cpp $(TRACK) $(LDLIBS) + +check: build/sides-test + build/sides-test + build/ft-ringplay: track/ringplay.cpp track/record.h camd/fhring.h @mkdir -p build $(CXX) $(CXXFLAGS) -o $@ track/ringplay.cpp @@ -44,6 +52,6 @@ build/ncnn/install/lib/libncnn.a: cmake --build build/ncnn/build --target install > build/ncnn/build.log clean: - rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay + rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay build/sides-test -.PHONY: all tools clean +.PHONY: all tools check clean diff --git a/hands/README.md b/hands/README.md index cfbca6d..5b84901 100644 --- a/hands/README.md +++ b/hands/README.md @@ -36,7 +36,7 @@ hands/run.sh uninstall Settings in `~/.config/frametop.conf` (`FT_` in the environment overrides them), read when ft-camd and ft-hands start: -- `HANDS_SWAP_SIDES=1`: the two side cameras' names are swapped (see ft-camd below). Check with `tools/check_sides.py --ring`. +- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks and logs a warning if the hands disagree. - `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below). - `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras. - `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`. @@ -86,7 +86,14 @@ Options: It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. The service starts it again, and it attaches to the new buffers. -**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. With the headset on, looking at a room with some texture, `tools/check_sides.py --ring` says whether the names are right (exit 0), swapped (exit 3), or it can't tell (exit 2). When they're swapped, set `HANDS_SWAP_SIDES=1`. The colour cameras are video3 (`arcimx616 0-0010`) and video0 (`0-001a`); which of them is `passthrough_left` in the module's calibration is for `tools/check_color.py` to settle, on a recording with texture in view. +**Which camera is which:** video9 is `slam_left`, video13 is `slam_right`, video6 is `upper_left` and video7 is `upper_right`. This was checked by rendering the same view from each camera with the factory calibration. But ft-camd tells the side cameras' buffers apart only by XRService's allocation order, and after some XRService restarts it gets them backwards. Then every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands now catches this by itself (`track/sides.h`, `HANDS_SWAP_SIDES=auto`): +- Whenever a hand's landmarks are found in two cameras at once (one of them a side camera), it intersects the rays through the 21 landmarks twice: once with the calibrations as named, once with the two side cameras exchanged. The same hand seen the right way meets within a few mm, in front of both cameras and as far away as its size says. The wrong way misses by centimetres or meets behind a camera. +- With the names wrong, the tracker never gets such pairs on its own: it hands the hand over to where the wrong calibration puts it and finds nothing there. So 5 times a second while undecided, the check places a tracked hand in 3D under the other naming and runs the landmark model where that puts it in the other side camera. +- It decides after 10 votes one way and none the other, or 20 with at most a fifth the other way, over at least 1 s. That takes about 1-2 s of hands in view. If the names are backwards, it exchanges them; the tracked views move with their images. Then it checks once more, more strictly. +- The log says what it found (`side cameras: SWAPPED, now exchanged after 3.2 s (votes ...)`). So does `/run/user/UID/frametop-hands/sides.json`, which the hand recorder reads. Recordings get a `DIR/sides.json` (hands/rec/sides.py has the rules). +- `--record-only` can't tell (it tracks nothing): it records ft-camd's names unless `--sides 0|1` says otherwise. + +`tools/check_sides.py --ring` is the independent check from the scene (ORB matches meeting under each naming): exit 0 as named, 3 swapped, 2 can't tell. `--pair upper` checks the upper pair the same way: in every recording so far (3 XRService starts, both side namings) the upper pair was named right. The colour cameras are video3 (`arcimx616 0-0010`) and video0 (`0-001a`); which of them is `passthrough_left` in the module's calibration is for `tools/check_color.py` to settle, on a recording with texture in view. ## ft-hands @@ -102,14 +109,14 @@ Options: - `--threads N`: model threads, pinned to the `--cpus` list. Default 3. - `--cpus LIST`: CPUs for the model threads and the main loop. Default `5,6,7` (`HANDS_CPUS`). SteamOS starts user processes on CPUs 0-4, and XRService's head tracking runs on 2-3. With the headset on, a step took 8.4 ms on 5-7 against 13.2 ms on 2-4, and SteamVR's frame timing didn't change (2026-09-29, three rounds of the same replayed frames). - `--contrast MODE` or `PALM/HAND`: how crops are equalized before the models see them: `clahe[:CLIP]`, `none`, or `stretch` (1st-99th percentile). Default `clahe:2/none`. In the dim recording, CLAHE let the palm search find about 10% more hands, but it made the landmarks jitter more (published median 6.9 mm, against 6.0 mm with plain landmark crops). -- `--swap-sides`: swap the two side cameras (`HANDS_SWAP_SIDES`, see ft-camd). +- `--sides auto|0|1`: which side camera is which (`HANDS_SWAP_SIDES`, see "Which camera is which"); `--swap-sides` is `--sides 1`. - `--seconds N`: stop after N seconds. - `--status S`: how often to print status, in seconds. - `--models DIR`: where the models are. - `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs. - `--no-publish`: don't write the hands and gestures files. - `--no-gestures`: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. `ft-cutouts` runs it this way. -- `--record DIR`, `--record-for S`: save every frame set for S seconds (default 120) to `DIR/sets.bin`. That's about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-