Make hand tracking a Frametop component

- Programs: ft-camd (the camera broker), ft-hands (the tracker), and
  ft-handreplay and ft-ringplay for recordings, built by hands/build.sh
  into hands/build/ with one Makefile. The first build fetches ncnn at
  frame-hands' pinned tag and builds it with the same options.
- ft-camd gets its privileges from file capabilities (CAP_SYS_PTRACE,
  CAP_PERFMON, CAP_DAC_READ_SEARCH) that hands/run.sh install sets with
  sudo, and drops them once set up. It still works under sudo. It runs
  on the host, linked statically, as frametop-camd.service. ft-hands
  runs in the dev container as frametop-hands.service. Both start and
  stop with SteamVR.
- Files move to /run/user/UID/frametop/ (cam-ring, hands, gestures),
  not $XDG_RUNTIME_DIR, which a terminal in the Frametop desktop has
  its own of. SIGUSR1 recordings go to ~/.local/share/frametop/hands.
- The calibration is read through /run/host in the container.
- Settings: HANDS_SWAP_SIDES and HANDS_CPUS in frametop.conf.
- install.sh offers hand tracking as an optional last step.
- The container gets jsoncpp-devel, glibc-static, and NumPy and OpenCV
  for the Python tools.
- tools/ring.py reads the ring, and models/NOTICE credits the
  Apache-2.0 models.

Checked: ft-handreplay gives identical summaries and byte-identical
depth dumps to frame-hands' fh-replay on both 2026-09-29 recordings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed 2026-09-30 09:15:07 -06:00
1 parent 3e3d31c728
commit 499035216c
41 files changed
+727 -402

No files matched your search

+9 -10
View File
@@ -1,13 +1,13 @@
"""Check that the side cameras' images carry the right names (slam_left vs slam_right).
usage: python tools/check_sides.py REC_DIR [--sets N]
python tools/check_sides.py --ring [--sets N] (live, from fh-camd's ring)
python tools/check_sides.py --ring [--sets N] (live, from ft-camd's ring)
With --ring it exits 0 when the names are right, 3 when they're swapped (run fh-tracker
With --ring it exits 0 when the names are right, 3 when they're swapped (run ft-hands
with --swap-sides), and 2 when it can't tell (too little texture in view, or the headset
isn't worn).
fh-camd tells the two side cameras' buffers apart by the order XRService allocated them,
ft-camd tells the two side cameras' buffers apart by the order XRService allocated them,
and after some XRService restarts that order puts each camera's images under the other's
name. The tracker then sees every hand in one camera only, at the wrong depth. This
matches features between the two images and measures how close each pair's rays pass
@@ -23,15 +23,14 @@ import numpy as np
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
from tools.show_set import index, read_set # noqa: E402
from tracker import calib # noqa: E402
from tools import calib # noqa: E402
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right'} # as fh-tracker maps them
PIPES = {'msm_vfe3_video0': 'slam_left', 'msm_vfe4_video0': 'slam_right'} # as ft-hands maps them
def load_cams():
root = os.environ.get('FRAME_JOB_DEVICE_ROOT', '') # frame-job's copy of /persist off the Frame
return calib.load(root + calib.XRSERVICE_JSON, root + calib.DEVICE_JSON)
return calib.load()
def matches(a, b):
@@ -80,12 +79,12 @@ def recorded_pairs(rec, count):
def live_pairs(count):
"""(label, slam_left image, slam_right image) from fh-camd's ring, half a second apart."""
"""(label, slam_left image, slam_right image) from ft-camd's ring, half a second apart."""
import time
from tracker.ring import Ring
from tools.ring import Ring
ring = Ring()
if not ring.alive():
sys.exit('fh-camd isn\'t running (no heartbeat)')
sys.exit('ft-camd isn\'t running (no heartbeat)')
cams = {}
for c in ring.cams:
name = PIPES.get(open('/sys/class/video4linux/video%d/name' % c.node).read().strip())