Defer hand tracking; the README leads with what Frametop does now

- install.sh no longer offers hand tracking (it's heavy on the CPU and
  needs more work); hands/ still builds and installs by hand
- the build container drops python3-opencv and python3-numpy, which
  only hand tracking's tools used: Fedora's OpenCV pulls in over a GB
- README: a new opening and feature list, the Use section by topic
  (screens, mouse, floating windows, profiles, gaze), and new known
  limitations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed 2026-10-01 15:20:39 -06:00
1 parent c257000a23
commit 7bcda94267
5 files changed
+88 -54

No files matched your search

+6 -4
View File
@@ -1,6 +1,8 @@
# Hands (experimental)
# Hands (experimental, deferred)
Hand tracking from the headset's own cameras. It serves two things in Frametop:
Hand tracking from the headset's own cameras. It's deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it and the README doesn't list it. It still builds and runs, installed by hand (below), for working on it.
It serves two things in Frametop:
- **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro.
- **Pinches and grips:** with `POINTER_HANDS=1`, the pointer helper takes them as clicks and drags. Look at something and pinch to click it, with the eye tracker doing the looking (`gaze/`), or close your hand to press and drag what the pointer is on. See "Pinches and grips in the pointer" below.
@@ -10,7 +12,7 @@ Two programs, each a user service that stops when SteamVR does:
- `ft-camd` (`camd/`, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host.
- `ft-hands` (`track/`, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container.
They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` offers the install as its last, optional step.
They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` doesn't install it.
```
ft-handsctl on | off # on the Frame: start or stop hand tracking (SteamVR must be running)
@@ -188,7 +190,7 @@ hands/build/ft-handreplay ~/.local/share/frametop/hands/rec-20260929-120000 --co
## Tools
Python, with NumPy and OpenCV (in the dev container: `python3-numpy`, `python3-opencv`, which `setup/dev-container.sh` installs). Off the Frame, `FRAME_JOB_DEVICE_ROOT` can point at a folder with copies of the headset's calibration files.
Python, with NumPy and OpenCV. `setup/dev-container.sh` doesn't install them, because Fedora's `python3-opencv` pulls in over a gigabyte; in the dev container, run `sudo dnf install python3-numpy python3-opencv` once. Off the Frame, `FRAME_JOB_DEVICE_ROOT` can point at a folder with copies of the headset's calibration files.
- `tools/check_sides.py --ring` (or a recording): are the side cameras named right?
- `tools/check_color.py REC`: how the colour module's calibration maps onto its images.