mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-14 16:00:43 +02:00
Compare commits
2
Commits
v0.3.0-exp.8
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 DeeJanuzandClaude Opus 5.5
|
24a86dc026 | ||
|
DeeJanuzandClaude Opus 5.5
|
194eed71c9 |
No files matched your search
@@ -53,13 +53,12 @@ jobs:
|
||||
run: |
|
||||
podman run --rm --entrypoint sh "$FT_IMAGE" -c '
|
||||
set -e
|
||||
for p in ft-screens ft-spotpanel ft-pointer ft-powerd ft-gaze ft-gazepanel ft-stream ft-eyegrab whisper-server; do
|
||||
for p in ft-screens ft-pointer ft-powerd ft-gaze ft-gazepanel ft-stream ft-eyegrab; do
|
||||
test -x /opt/frametop/bin/$p
|
||||
done
|
||||
test -f /src/frametop/pointer/driver/build/driver_ft_pointer.so
|
||||
test -x /src/frametop/pack/build/distrobox/install
|
||||
/src/frametop/gaze/tracker/build/venv/bin/python -c "import numpy, cv2"
|
||||
/src/frametop/voice/build/venv/bin/python -c "import numpy, sentencepiece, sherpa_onnx"
|
||||
python3 -c "import PySide6"'
|
||||
|
||||
# The FrameDrop manifest names the zip's URL in this repo's release (a fork's, in a fork).
|
||||
|
||||
@@ -4,7 +4,6 @@ Frametop is a desktop for the Steam Frame that runs on the headset itself, with
|
||||
|
||||
- **Screens that are real monitors.** Each KDE Plasma screen has its own resolution and shape: an ultrawide in front, portrait screens beside it. Move, resize, curve, and roll them, or pin one to your wrist or your head. They come back to your layout when the desktop starts.
|
||||
- **Windows that float on their own.** Take any app window off the screens into a panel of its own with Meta+Shift+F, its title bar button, or Launch as Standalone. It stays part of the desktop, so drag and drop and the clipboard still work between floating windows and the screens.
|
||||
- **Your other computers' monitors.** Stream a Windows PC's or a Mac's monitors, or a virtual monitor at any size, as screens of their own, with the same controls. See [Remote displays](#remote-displays).
|
||||
- **Profiles.** Save where your screens are, which ones show, and which apps are open where. Switch to a profile from Display Settings, a key, or its launcher entry, or start the desktop in one.
|
||||
- **One mouse for all of SteamVR.** A Bluetooth mouse drives a small dot anchored in the room. It works the screens, the dashboard, Steam, and overlays, and hands the laser back when you pick up a controller.
|
||||
- **Look and click (experimental).** In gaze mode the pointer goes where you look. Meta+J and Meta+K, or the mouse buttons, click and fine-tune, and each correction teaches the tracker. Calibration runs in the headset.
|
||||
@@ -36,7 +35,7 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
|
||||
|
||||
The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.1 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
|
||||
|
||||
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you five things along the way: whether to install voice dictation (yes by default; it downloads about 320 MB of speech models), gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
|
||||
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you four things along the way: whether to install gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. The eye tracker and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
|
||||
|
||||
After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings. SteamOS's own single-screen desktop is still there, as Native Desktop in the same list.
|
||||
|
||||
@@ -54,7 +53,7 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
|
||||
|
||||
| Do this | To get this |
|
||||
| --- | --- |
|
||||
| Point near the bottom of a screen | Its controls fade in: the bar, the curve and roll buttons, and, near a corner, a soft glow round it: press there and drag to resize |
|
||||
| Point near the bottom of a screen | Its controls fade in: the bar, the curve and roll buttons, and the resize tab on the corner |
|
||||
| Drag the bar under a screen | Moves the screen; scroll while dragging to push it away or pull it closer. With the mouse, hold right while dragging to tilt it |
|
||||
| Drag the tab on a screen's bottom right corner | Resizes the screen |
|
||||
| Click the curve button (next to the bar) | Curves the screen around you, or flattens it |
|
||||
@@ -63,7 +62,6 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
|
||||
| Set a screen to On your head (Frametop Display Settings, Visibility & pins) | Pins it to your head where it is, like a HUD. Grab its bar to move it; it stays on your head |
|
||||
| Meta+Shift+R in the desktop | Puts the screens back in their layout (also in the menu as Reset Screen Layout, and mappable to a mouse button) |
|
||||
| Meta+Alt+Tab, or Meta+Alt+Shift+Tab | Spins every screen and floating window around you together, like a lazy susan, so the next one on your right (or left) glides to straight ahead, with the pointer and typing going to it. Their arrangement stays the same: the room turns instead of you. Tap again to keep going; Meta+Shift+R puts the screens back. Pinned screens stay where they are |
|
||||
| Meta+Space, on any keyboard | Opens spotwatch, a search pill in front of you: type an app's or a remote display's name and press Enter. A running app's window spins to the front, an app that isn't running opens floating, and a remote display connects and spins to the front. Shift+Enter (or a right click) has more: a private window, a new window, docked on this screen, or one of its windows. Esc closes it. See [docs/spotwatch.md](docs/spotwatch.md) |
|
||||
| Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
|
||||
| Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own |
|
||||
| Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead |
|
||||
@@ -102,22 +100,6 @@ A profile is a named setup: where the screens are, with their sizes and pins, wh
|
||||
| Pick a profile under Arrangement and press Open profile | Switches to it: the screens move, open windows of its apps go to their places, and the apps that aren't open start. Nothing closes |
|
||||
| Pick a profile under Start in profile, or run its entry (Frametop: NAME) from SteamVR's Launch a program list | The desktop starts in that profile, or switches to it if it's running. A profile can also go on a key combination, mouse button, or controller button in Frametop Input Settings |
|
||||
|
||||
### Remote displays
|
||||
|
||||
Remote displays bring another computer's monitors into Frametop. Each one is a panel with a screen's controls: move it, resize it, curve it, pin it, and save it in a profile. The mouse, the keyboard and the lasers work on it, and its sound plays on the Frame. The computer streams with Sunshine's protocol, and the Frame decodes the streams in hardware.
|
||||
|
||||
Each computer needs a host kit once:
|
||||
|
||||
| Computer | What it streams | Set up with |
|
||||
| --- | --- | --- |
|
||||
| Windows PC | Its monitors, and virtual monitors at any size | Vibepollo with Frametop's build: [host/windows/README.md](host/windows/README.md) |
|
||||
| Mac with Apple silicon | Its monitors | Sunshine, one instance per monitor, and Frametop Streams in the menu bar to turn them on and off: [host/macos/README.md](host/macos/README.md) |
|
||||
| Linux PC (Arch, CachyOS) | Its monitors, and virtual monitors | Vibepollo with Frametop's build, not tested yet: [host/linux/README.md](host/linux/README.md) |
|
||||
|
||||
Then, on the Frame, open Frametop Remote Displays (in the app menu, or Remote displays in Display Settings), choose Add computer, sign in with the computer's Web UI login, and pick its displays. Each display has a Connected switch, a Shown switch, and its stream's resolution, frame rate and bitrate. Meta+Space finds a remote display by name and brings it to the front. A PC with a Steam Link dongle can stream over it, which skips your router.
|
||||
|
||||
A monitor you stream switches to the stream's resolution while it streams (Windows), so pick a size the monitor has, or one with the same shape. A computer that's asleep can't be reached. [docs/reference.md](docs/reference.md#frametop-remote-displays) has the details.
|
||||
|
||||
### Pause for VR games
|
||||
|
||||
Frametop pauses while a VR game runs, so the game gets the headset's CPU and GPU, and comes back a few seconds after the game ends. Paused, the screens hide and the desktop nearly stops drawing, but its windows stay open. Gaze mode's eye tracking stops, and so do remote desktop and hand tracking if they run. The mouse works as a plain mouse in SteamVR.
|
||||
@@ -232,14 +214,10 @@ A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland com
|
||||
| `input/` | The input relay (Bluetooth mice and keyboards, button maps). |
|
||||
| `pointer/` | The 3D mouse: SteamVR driver, helper service, and a probe tool. |
|
||||
| `power/` | ft-powerd: turns the displays off while the headset isn't used. |
|
||||
| `voice/` | Frametop Voice: dictation and wake words, recognised on the headset; the input relay types the text. See [voice/README.md](voice/README.md). |
|
||||
| `gaze/` | Gaze mode (experimental): the gaze service, its calibration panel, our own eye tracker, and the gaze probe. See [gaze/README.md](gaze/README.md). |
|
||||
| `hands/` | Hand tracking (experimental, deferred: the installer doesn't offer it). See [hands/README.md](hands/README.md). |
|
||||
| `display-settings/`, `input-settings/` | The two settings apps (Kirigami, Python). |
|
||||
| `remote/` | Frametop Remote Access, the app that turns remote desktop over VNC on and off. |
|
||||
| `remote-displays/` | Frametop Remote Displays, the app that adds other computers' monitors as screens. |
|
||||
| `stream/` | ft-stream, one per remote display: receives and decodes its stream for ft-screens. |
|
||||
| `host/` | The host kits for remote displays: Windows, macOS and Linux. |
|
||||
| `setup/` | The build container and the Bluetooth fixes. See [setup/README.md](setup/README.md). |
|
||||
| `scripts/` | Helpers the installers use. They run commands locally on the Frame, or over SSH from a PC. |
|
||||
|
||||
@@ -284,9 +262,4 @@ Programs built in the `dev` container link against its glibc, which is newer tha
|
||||
|
||||
## License
|
||||
|
||||
MIT. See [LICENSE](LICENSE). The rendered hands' 3D models in `screens/assets/hands` (see its [README](screens/assets/hands/README.md); each folder has its licence) are from:
|
||||
|
||||
- the W3C Immersive Web Working Group's WebXR Input Profiles, `generic-hand` (MIT, Copyright (c) 2019 Amazon), the default;
|
||||
- Babylon.js's hand meshes by Patrick Ryan, https://github.com/BabylonJS/Assets (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/), unchanged;
|
||||
- the Mixed Reality Toolkit: MRTK3's OpenXR hand (BSD-3-Clause, Copyright (c) 2025, Mixed Reality Toolkit Contributors) and MRTK2's rigged hand (MIT, Copyright (c) Microsoft Corporation), converted to glTF;
|
||||
- Godot XR Tools' hands by DigitalN8m4r3 (Miodrag Sejic), CC0 1.0.
|
||||
MIT. See [LICENSE](LICENSE).
|
||||
@@ -10,9 +10,8 @@ the dev container:
|
||||
1920x1080 worth of pixels, rotation for portrait.)
|
||||
- Visibility (ft-screens): when the screens show (always, only with the SteamVR
|
||||
dashboard open, while you look at a controller, or only when toggled), the wrist
|
||||
angle within which a pinned screen shows, pinning each screen to a wrist or your
|
||||
head, and whether the desktop's notifications go to the Steam session
|
||||
(NOTIFY_FORWARD, session/ft-notifyfwd; its state from its log).
|
||||
angle within which a pinned screen shows, and pinning each screen to a wrist or
|
||||
your head.
|
||||
- Layout: a preset (curved or flat, rows, distance, gap, height) or a named layout
|
||||
saved from where the screens are, with a preview; arrange now; save the current
|
||||
arrangement under a name; rename and delete; arrange automatically when the
|
||||
@@ -63,8 +62,6 @@ FT_POWERD = "\0ft_powerd"
|
||||
# nothing was saved.
|
||||
STEAM_SLEEP_AC_DEFAULT = 3600
|
||||
SCALES = [0.75, 1.0, 1.25, 4 / 3, 1.5, 1.75, 2.0]
|
||||
NOTIFY_FORWARD = ("auto", "on", "off") # session/ft-notifyfwd; frametop.conf's NOTIFY_FORWARD
|
||||
NOTIFYFWD_LOG = "/tmp/frametop-notifyfwd.log" # its last line is its state (/tmp is the host's)
|
||||
ROTATIONS = [("normal", "Landscape"), ("left", "Portrait"), ("right", "Portrait (flipped)")]
|
||||
|
||||
|
||||
@@ -124,7 +121,6 @@ class Backend(QObject):
|
||||
self._steam_busy = False
|
||||
self._steamDone.connect(self._steam_done, Qt.QueuedConnection)
|
||||
self._pins = [] # each running screen's pin: none | left | right | head
|
||||
self._notify_state = ""
|
||||
self.poll = QTimer(interval=3000, timeout=self._check_running)
|
||||
self.poll.start()
|
||||
self._check_running()
|
||||
@@ -137,10 +133,8 @@ class Backend(QObject):
|
||||
return cast(conf.get(key, default))
|
||||
except ValueError:
|
||||
return default
|
||||
notify = conf.get("NOTIFY_FORWARD", "auto")
|
||||
return {"screens": num("SCREENS", 2, int), "width": num("WIDTH", 1920, int),
|
||||
"height": num("HEIGHT", 1080, int), "physWidth": num("PHYS_WIDTH", 1.6),
|
||||
"notifyForward": notify if notify in NOTIFY_FORWARD else "off"}
|
||||
"height": num("HEIGHT", 1080, int), "physWidth": num("PHYS_WIDTH", 1.6)}
|
||||
|
||||
def _check_running(self):
|
||||
# The session's own Wayland socket (host processes' environments aren't readable
|
||||
@@ -148,15 +142,12 @@ class Backend(QObject):
|
||||
running = os.path.exists(f"/run/user/{os.getuid()}/frametop/wayland-0")
|
||||
count = self._screens_running() if running and ft_layout.backend() == "screens" else 0
|
||||
pins = self._read_pins(count)
|
||||
notify_state = self._read_notify_state() if running else ""
|
||||
if (running != self._running or count != self._running_count or pins != self._pins
|
||||
or notify_state != self._notify_state):
|
||||
if running != self._running or count != self._running_count or pins != self._pins:
|
||||
if running != self._running or count != self._running_count:
|
||||
self._started = self._conf() if running else {}
|
||||
self._running = running
|
||||
self._running_count = count
|
||||
self._pins = pins
|
||||
self._notify_state = notify_state
|
||||
self.changed.emit()
|
||||
self._check_powerd()
|
||||
|
||||
@@ -168,22 +159,6 @@ class Backend(QObject):
|
||||
pins.append(f[16] if len(f) > 16 else "none")
|
||||
return pins
|
||||
|
||||
@staticmethod
|
||||
def _read_notify_state():
|
||||
"""ft-notifyfwd's: forwarding | waiting (no usable server on the user bus) | "" (not running)."""
|
||||
try:
|
||||
with open(NOTIFYFWD_LOG) as f:
|
||||
lines = f.read().splitlines()
|
||||
except OSError:
|
||||
return ""
|
||||
# the last state line: errors it logs in between (a failed Notify) don't change it
|
||||
for line in reversed(lines):
|
||||
if line == "ft-notifyfwd: forwarding":
|
||||
return "forwarding"
|
||||
if line.startswith("ft-notifyfwd: not forwarding"):
|
||||
return "waiting"
|
||||
return ""
|
||||
|
||||
def _ask_screens(self, text):
|
||||
"""Request/reply to ft-screens; None if it isn't running."""
|
||||
try:
|
||||
@@ -408,26 +383,6 @@ class Backend(QObject):
|
||||
def toggleScreens(self):
|
||||
self._ask_screens("toggle")
|
||||
|
||||
# --- the desktop's notifications in the Steam session (session/ft-notifyfwd) ---
|
||||
@Property(str, notify=changed)
|
||||
def notifyForward(self):
|
||||
return self._conf()["notifyForward"]
|
||||
|
||||
@Property(str, notify=changed)
|
||||
def notifyForwardState(self):
|
||||
return self._notify_state
|
||||
|
||||
@Property(bool, notify=changed)
|
||||
def notifyRestartNeeded(self):
|
||||
"""The session writes ft-notifyfwd's autostart entry when it starts."""
|
||||
return self._running and bool(self._started) and self._started["notifyForward"] != self._conf()["notifyForward"]
|
||||
|
||||
@Slot(str)
|
||||
def setNotifyForward(self, mode):
|
||||
if mode in NOTIFY_FORWARD:
|
||||
write_conf_value("NOTIFY_FORWARD", mode)
|
||||
self.changed.emit()
|
||||
|
||||
# --- ft-screens: visibility and pinning ---
|
||||
@Property("QVariantMap", notify=changed)
|
||||
def visibility(self):
|
||||
|
||||
@@ -958,60 +958,6 @@ Kirigami.ApplicationWindow {
|
||||
+ "your head where you let go. Choosing a pin above keeps the screen where it is now, so place "
|
||||
+ "it first. Save as profile… (Layout) keeps pins."
|
||||
}
|
||||
|
||||
Kirigami.FormLayout {
|
||||
Layout.fillWidth: true
|
||||
|
||||
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Notifications" }
|
||||
|
||||
Repeater {
|
||||
model: [
|
||||
{ value: "auto", text: "Show them in the Steam session", help: "The desktop's notifications go to the Steam session's notification server, such as one that shows them in VR, with their buttons. Plasma's own popups stay off meanwhile; its history keeps them. Not to SteamOS's own server, which can't show them on the Frame." },
|
||||
{ value: "on", text: "Always send them to the Steam session", help: "Whatever serves notifications there, SteamOS's own server too, which can't show them on the Frame: they're lost then." },
|
||||
{ value: "off", text: "Keep them in the desktop", help: "Plasma shows them on the desktop's screens." }
|
||||
]
|
||||
delegate: ColumnLayout {
|
||||
required property var modelData
|
||||
spacing: 0
|
||||
// "on" can lose them (SteamOS's server): set in frametop.conf, shown only then
|
||||
visible: modelData.value !== "on" || backend.notifyForward === "on"
|
||||
Controls.RadioButton {
|
||||
text: modelData.text
|
||||
checked: backend.notifyForward === modelData.value
|
||||
onToggled: if (checked) backend.setNotifyForward(modelData.value)
|
||||
}
|
||||
Controls.Label {
|
||||
text: modelData.help
|
||||
opacity: 0.7
|
||||
font: Kirigami.Theme.smallFont
|
||||
leftPadding: Kirigami.Units.gridUnit * 1.6
|
||||
wrapMode: Text.Wrap
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
|
||||
}
|
||||
}
|
||||
}
|
||||
Controls.Label {
|
||||
Kirigami.FormData.label: "Now:"
|
||||
visible: backend.desktopRunning
|
||||
text: backend.notifyForwardState === "forwarding" ? "In the Steam session"
|
||||
: backend.notifyForwardState === "waiting" ? "In the desktop: the Steam session has no notification server that can show them"
|
||||
: "In the desktop"
|
||||
wrapMode: Text.Wrap
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
|
||||
}
|
||||
RowLayout {
|
||||
visible: backend.notifyRestartNeeded
|
||||
Controls.Label {
|
||||
text: "Applies when the desktop starts again."
|
||||
wrapMode: Text.Wrap
|
||||
}
|
||||
Controls.Button {
|
||||
text: "Restart desktop"
|
||||
icon.name: "system-reboot"
|
||||
onClicked: restartDialog.open()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+4
-20
@@ -74,7 +74,7 @@ In a game, Frametop's panels work like SteamVR's own floating windows: point a c
|
||||
|
||||
- Disabling a nested output keeps its host window. `Output::applyChanges` only flips `enabled`, KWin stops rendering it (no more commits), and Plasma drops its desktop view. So ft-screens keeps the same toplevel, and its screen numbers stay put. Enabling the output again resumes on the same toplevel.
|
||||
- Each output's host window is titled `KDE Wayland Compositor WL-<n>`, with `- Output disabled` appended while it's disabled (`WaylandOutput::updateWindowTitle`, on every `enabledChanged`). ft-screens reads the title to tell screens (`WL-0` to `WL-<SCREENS-1>`) from spares, and to see a spare turn on and off.
|
||||
- A spare resized while it's disabled comes up at the new size on its first frame, so floating a window needn't blink. Outputs with gaps between them are accepted, so ft-floatd places spares apart from the screens and from each other: a grid below them, 4000 logical pixels a cell. Xwayland's coordinates are the layout's times its largest output scale, and they wrap past 32767, so an X11 window on a spare past that took no clicks (Brave on the 4th spare at x 27000 and scale 1.25 sat at x -31506; KWin still focused it). The grid ends at 20000 for 16 spares, within the limit up to scale 1.6.
|
||||
- A spare resized while it's disabled comes up at the new size on its first frame, so floating a window needn't blink. Outputs with gaps between them are accepted, so ft-floatd places spares apart from the screens and from each other, within Xwayland's 32767-pixel limit.
|
||||
- KWin keeps a Wayland popup inside its parent's output (`XdgPopupWindow::updateRelativePlacement` uses the output's placement area), and X11 apps place their menus within the monitor. That's why a floating window's output has a margin around the window: menus and dropdowns open past the window's edges, into the margin.
|
||||
- KWin makes a nested output the size it's configured to times its scale, rounded (at 1.5 it lays out 1067 × 667 on a 1600 × 1000 buffer), and gives the buffer a whole buffer scale (1.2 becomes 2). A buffer whose size isn't a multiple of that is a protocol error that disconnects KWin, so ft-floatd sizes spares in multiples of it. After a scale change, ft-floatd asks for the output's size again in the new scale's terms, or the next configure would make it the old size times the scale.
|
||||
- **Virtual outputs don't work.** `createVirtualOutput` makes an output window but never adds it to the backend's `m_outputs`, so `findOutput()` returns null when the pointer enters it, and the next line dereferences it (`Q_ASSERT` is compiled out). A click on such a panel would crash KWin. This rules out virtual outputs (`stream_virtual_output`) for floating windows without a patched KWin.
|
||||
@@ -117,13 +117,7 @@ Driver poses are in SteamVR's raw tracking space, and client programs work in th
|
||||
|
||||
Frametop's SteamVR clients (ft-pointer, ft-screens, ft-gaze) connect as a background app first and switch to an overlay app only once that works. `VR_Init` as an overlay app starts vrserver itself when none is running, and one started that way from the dev container never finds the headset. At a boot where the gamescope session timed out, systemd dropped `steamvr.service`'s start job, the pointer service (ordered only `After=` it) started anyway, and its vrserver made every SteamVR launch fail with `HmdNotFound`. SteamOS's health check then kept resetting the Steam client and tried to fall back to the previous OS slot. The units also say `Requisite=steamvr.service`, so they don't start at all when SteamVR's start fails.
|
||||
|
||||
The driver starts disconnected, because holding the right-hand role while SteamVR starts leaves the Steam UI stuck on its loading icon. It connects when the mouse is used. With a hand role (`POINTER_ROLE=right`, the default before the treadmill role) it claims that hand: SteamVR keeps a hand role reserved for a disconnected device that still asks for it, so the driver switches its role hint between the hand (connected) and opt-out (not connected). The treadmill role (below), now the default, needs no switching.
|
||||
|
||||
#### Treadmill role
|
||||
|
||||
With a hand role, a Frame controller held in that hand takes the role back (its touch sensors count as use), and the virtual controller is left with no role and no clicks. So mixing a controller in one hand with the 3D mouse or a bare hand's ray in the other fails whenever that controller is merely held. The treadmill role (4) is no hand: the real controllers keep both hand roles, and the virtual controller never fights them. It held the dashboard laser in the gaze-first tests (2026-09-30, branch `gaze-first`), with two conditions. SteamVR gives a device the `/user/treadmill` path only when the device is added, so the driver must hint the role from `Activate`, not on connect: the `role` in its `default.vrsettings` (or `driver_ft_pointer/role` in SteamVR's settings), and switching it needs a SteamVR restart. And the compositor binding needs `/user/treadmill` entries next to the hand ones (`ft_pointer_vrcompositor.json`). A treadmill reports no controller role (`GetControllerRoleForTrackedDeviceIndex` is invalid), so the helper's "no hand role" release skips it. The laser claim (`/input/a`, bound to `switchlaserhand`) works the same; SteamVR took the laser back to our device 11-15 ms after a controller took it.
|
||||
|
||||
A treadmill can't move SteamVR's own floating windows by their grab bars, though: SteamVR's dashboard (a web page in vrwebhelper) starts a window's move in `UndockedOverlay.startFloatingWindowMove`, which reads the pressing device from the mouse event's CapsLock and NumLock bits (head, left hand, right hand) and, for anything else, logs "mouse event carried no input path" and does nothing. Clicks and resizes don't ask, so only moves broke, from 2026-10-09, when treadmill became the default (found 2026-10-13, SteamVR 2.18.2; with role 2 the same laser moved them). `pointer/dashpatch` puts that back. `treadmill-move.js` wraps the grab bar's handler: a press with no device runs it as the right hand, with that one pose query answered by `/user/treadmill`, and the move then follows `/user/treadmill`. `ft-dashpatch.py run` (`frametop-dashpatch.service`, started with SteamVR) puts it into the dashboard over the Chrome DevTools Protocol, again whenever the page is new, and checks every 30 s that it's still in. The dashboard's debugger is off unless `VRWebHelper/DebuggerEnabled` is on; the installer turns it on (`ft-dashpatch.py on`, from SteamVR's next start) on port 8089, bound to 127.0.0.1. Not the usual 8087: SteamOS's `steamvr-web-debug-portforward.service` forwards 0.0.0.0:8088 to 127.0.0.1:8087, which would open the dashboard to the network. Like Steam's own debugger on 8080, any local program can use it. The uninstaller turns it off again if Frametop turned it on. The patch finds the handler by its name and React by its module, so a SteamVR update that renames them leaves windows unmovable for the treadmill again, without breaking anything else; `ft-dashpatch.py status` shows whether it's in and how many moves it made.
|
||||
The driver starts disconnected, because holding the right-hand role while SteamVR starts leaves the Steam UI stuck on its loading icon. It connects when the mouse is used and claims the right hand. SteamVR keeps a hand role reserved for a disconnected device that still asks for it, so the driver switches its role hint between right hand (connected) and opt-out (not connected).
|
||||
|
||||
### The cursor
|
||||
|
||||
@@ -166,7 +160,7 @@ Keyboards aren't grabbed by default, because a grabbed keyboard's keys went into
|
||||
|
||||
The relay never waits on the pointer helper. Its socket to the helper used to block, so when the helper stalled (a layout placement or `grabprobe` holds it for seconds, and the gaze service fills its socket 90 times a second meanwhile), the whole relay stopped with it: keyboards, the volume keys, and pausing. Now what the helper doesn't take waits in order and goes out on the next loops. Mouse moves add up into one while they wait, and a scroll notch is dropped, since scrolling seconds late is no use; presses and releases are kept, so no button stays down. Mouse motion goes to the helper at most every 4 ms, rather than once per report, which from a 1000 Hz mouse was 1000 datagrams a second to a helper that runs every 8 ms; a button sends the motion before it first, so the click lands where the pointer was.
|
||||
|
||||
An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's `InputStealer`, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's `OverlayFocusChanged` (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. Until 2026-10-10, `SHARE_KEYS=1` sent a grabbed keyboard's keys to `@frametop_keys` as datagrams for frame-voice. It was off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first got every key typed into the desktop, passwords included. Now frame-voice is part of Frametop (Frametop Voice), and the relay does its keyboard side itself: its push-to-talk key is a key combination, and its text comes to the relay as `type TEXT`, taken only from the relay's own user (`SO_PASSCRED`), and goes where typing goes, like a keyboard's keys: to ft-screens for the desktop, or the virtual keyboard for Steam, never both. Its own uinput keyboard went with it, so SteamVR no longer has to have opened a second keyboard for dictation to reach games.
|
||||
An ungrabbed keyboard reaches both sides at once. In VR, gamescope reads every input device itself (the SteamOS build's `InputStealer`, libinput with udev hotplug, so new devices too) and types into its focused app, and ft-screens types the same keys into the desktop. So Space in the desktop also paused Spotify on the dashboard. Typing now follows the last click. ft-screens sees clicks on its own screens, from the mouse or a controller. A click anywhere else is only visible for the mouse: overlay apps get SteamVR's `OverlayFocusChanged` (which panel the laser is on) but no controller button events, so the pointer helper reports the panel under the dot on each left press. ft-screens tells the relay where typing goes every second, from an unbound socket so the relay's replies can't loop back into its control socket, and the relay grabs pass-through keyboards while it's the desktop. A grab waits until the keyboard has no key down, so no key stays held on either side, and the relay lets go if ft-screens stops reporting. A program that reads every keyboard for a hotkey (a dictation tool, say) loses a grabbed keyboard. Repeating the keys on another input device doesn't work: gamescope reads that device too, whether it's the relay's virtual keyboard or one created later, and every Space, typed or dictated, paused Spotify again. So with `SHARE_KEYS=1` the relay sends a grabbed keyboard's keys to `@frametop_keys` as datagrams (`key <code> <value> <device name>`). It's off by default, because the relay can't tell who is listening: abstract sockets have no permissions, and any local process that binds the name first gets every key typed into the desktop, passwords included. A listener should accept only its own user (`SO_PASSCRED`) and skip any keyboard of its own that the relay grabs too.
|
||||
|
||||
A mouse button that pointer mode passes through as a key (a side button for Back) is a pointer button to ft-screens, not a key, so it goes to the screen the pointer is on rather than where typing goes. As a key, its release was lost whenever typing moved while it was held (the dashboard closing, a click on another panel, a pause), and wlroots counts presses per button: from then on that button, the lasers' left click included, did nothing in the desktop, and KWin kept it held. So ft-screens tracks the relay's buttons itself, lets a release through whenever it took the press, and releases them when the pointer leaves the screens, its screen hides, or Frametop pauses.
|
||||
|
||||
@@ -174,7 +168,7 @@ Volume keys must never reach gamescope. With the openvr backend, gamescope sends
|
||||
|
||||
Frametop's keyboard opens by itself for a text field on the desktop. The apps run inside the nested KWin, so only KWin knows when a text field has focus, and the way it tells anyone is its input method protocol (`zwp_input_method_v1`): KWin starts one input method program and activates it whenever the focused app turns on text input. `input/ft-textinput` is that program, speaking the Wayland wire protocol directly so it needs nothing but Python on the host. It only reports focus. The gamescope session puts `QT_IM_MODULE=xim` and `GTK_IM_MODULE=xim` in the systemd user environment; with those, Qt and GTK apps use X input methods and never turn on Wayland text input, so the session script drops them.
|
||||
|
||||
The keyboard itself is ft-screens' own panel (`screens/keyboard.cpp`). We tried SteamVR's first (`ShowKeyboardForOverlay`), and on the Frame it doesn't fit a desktop. It's Steam's own panel (`valve.steam.gamepadui.keyboard`), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (`SetKeyboardTransformAbsolute`) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, as the old `@frametop_keys` could), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU and uploaded with `SetOverlayRaw` when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.
|
||||
The keyboard itself is ft-screens' own panel (`screens/keyboard.cpp`). We tried SteamVR's first (`ShowKeyboardForOverlay`), and on the Frame it doesn't fit a desktop. It's Steam's own panel (`valve.steam.gamepadui.keyboard`), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (`SetKeyboardTransformAbsolute`) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, like `@frametop_keys`), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU and uploaded with `SetOverlayRaw` when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.
|
||||
|
||||
The Frame controllers can be mapped like mouse buttons, but they aren't input devices on the host: they reach SteamVR over the headset's own radio, and no evdev or hidraw node exists for them. So only a SteamVR client can read them. Overlay apps normally get controller input only while they have input focus, which a background helper never has. SteamVR's experimental global action set priority (`steamvr/globalActionSetPriority`, "Enable global input from overlays") lets an overlay's action set receive input anyway, and takes the inputs it binds from the scene app. Binding every button would take them all from games, so the helper's action manifest puts each button in an action set of its own, and it activates only the sets of mapped buttons. The mapping itself stays in the relay, which does the action, so mice and controllers share one list of actions.
|
||||
|
||||
@@ -194,16 +188,6 @@ The fallback registry does not exit merely when its bus disconnects in the isola
|
||||
|
||||
Run the isolated checks on the host with `/usr/bin/python3 session/test/test_accessibility.py`. They use private D-Bus buses, Xvfb and a GTK3 app, never the production display or input. Native activation uses a small `org.a11y.Bus` test provider pointing to a real private dbus-daemon with the installed registry service; the SteamOS fallback uses the installed bus launcher and broker. The tests check real app-tree discovery, existing owners, concurrent starts, session stop/restart, and teardown. They require test-only PyGObject (Gio and GTK3), Xvfb, and at-spi2-core; the runtime helper uses Python's standard library and the existing host `gdbus`. Actual Plasma autostart and VR desktop restart still require an approved hardware test.
|
||||
|
||||
### Notifications outside the desktop
|
||||
|
||||
The private bus keeps the desktop's notifications in it: apps notify Plasma, and Plasma's popups appear on the desktop's screens, whose panels may be hidden, or behind a VR game. Putting the desktop on the user bus instead would collide with the Steam session: Steam owns `org.freedesktop.ScreenSaver` there, the Steam session's portals and accessibility bus are there, Plasma would start its parts as units of the user's systemd and push its environment (`DISPLAY`, `WAYLAND_DISPLAY`, its runtime directory) into the bus's and systemd's activation environment, which programs started later in the Steam session would inherit.
|
||||
|
||||
`session/ft-notifyfwd` forwards instead. Taking `org.freedesktop.Notifications` from Plasma isn't reliable: plasmashell, the bus's "D-Bus master", registers it with `ReplaceExistingService` and `DontAllowReplacement` (plasma-workspace's `libnotificationmanager/server_p.cpp`), so another owner only keeps it by starting first, and loses it to a Plasma restart in between. Plasma 6.2.5 has a watcher API for this: `org.kde.NotificationManager.RegisterWatcher` on Plasma's notification object, after which Plasma calls `org.kde.NotificationWatcher.Notify` (its own id first, then the spec's arguments with the body as sent and the app's hints, image data included) and `CloseNotification` on the watcher's `/NotificationWatcher`, and `InvokeAction(id, key)`, which makes Plasma emit `ActionInvoked` to the app. The app only accepts that signal from its notification server's owner, which is why a click has to go back through Plasma. The helper copies each notification to the user bus's server keeping the hints' D-Bus types, maps the two servers' ids (a replacement reuses Plasma's), and maps that server's `ActionInvoked` to `InvokeAction`. Its `NotificationClosed` with reason 2 (dismissed) closes the notification in Plasma, so the app sees it closed (as revoked, reason 3); an expired one stays in Plasma's history. While it forwards, it holds an `Inhibit` on Plasma, which turns its popups off as Do not disturb does, and ends with the helper's connection; watchers get every notification regardless. It follows both servers' owners: it registers and inhibits again when Plasma restarts, and in `auto` mode checks `GetServerInformation` whenever the user bus's server changes. SteamOS's `steam_notif_daemon` reports itself as `simple_notif_daemon`; it passes notifications to `~/.steam/root/ubuntu12_32/steam` with a `steam://open_xdg_notification` URL, an x86 program that can't run on the Frame, and the Frame's Steam client ignores that URL anyway, so forwarding there would lose them. The helper is event-driven (GLib's main loop, PyGObject), so it costs nothing between notifications, and exits when the desktop's bus closes.
|
||||
|
||||
Notifications sent through the notification portal (`org.freedesktop.portal.Notification.AddNotification`, as Flatpak apps do) reach the watchers too: xdg-desktop-portal-kde passes them to Plasma's `Notify`. Plasma sends no activation token with `InvokeAction`, so KWin's focus stealing prevention may keep a window that an action raises from coming to the front. KDE's inline reply needs the `inline-reply` capability from the other server; without it, it's an action like any other.
|
||||
|
||||
Run the isolated checks on the host with `/usr/bin/python3 session/test/test_notifyfwd.py`. Two private buses stand in for the desktop's and the user bus, with a fake Plasma (the watcher API, Inhibit) on one and a fake notification server on the other. They check forwarding with the hints' types, actions coming back, dismissed and expired notifications, replacements and closes mapping ids, `auto` skipping `simple_notif_daemon` until another server replaces it and giving Plasma its popups back when none is left, registering again after a Plasma restart, exiting with the desktop's bus, and doing nothing outside the Frametop desktop.
|
||||
|
||||
### Other session behavior
|
||||
|
||||
Steam, not systemd, suspends the Frame: after `system_idle_suspend_ac_sec` (an hour by default) without input on AC power, it logs `Switching to power state: k_ESystemPowerState_Sleep` and suspends, even while charging. It's a Steam setting (Settings → Power → When Plugged In and Idle → Sleep after), which the Stay awake while plugged in switch in Frametop Display Settings sets to Never. SteamVR's standby, which turns the displays off when the headset comes off, is separate; see below.
|
||||
|
||||
@@ -28,7 +28,7 @@ The numbers are cited in the code, so they stay as they are. Struck-out text was
|
||||
|
||||
| # | Question | Decision |
|
||||
|---|---|---|
|
||||
| 1 | How many windows can float at once | 16 spare outputs by default (also the most), configurable with `FLOAT_SLOTS`; a change needs a desktop restart. A Display Settings control isn't built |
|
||||
| 1 | How many windows can float at once | 8 spare outputs by default, configurable with `FLOAT_SLOTS` (at most 16); a change needs a desktop restart. A Display Settings control isn't built |
|
||||
| 2 | Menus and dropdowns | Each floating output has a margin around the window. The panel shows only the window, and each open popup gets a small overlay of its own, cut from the same buffer |
|
||||
| 3 | Tearing off | Not built. Drag the title bar past a screen's edge and let go in the air, with a small dead zone past the edge |
|
||||
| 4 | Docking by dragging | Not built. Push the window flush against a screen (within about 10 cm), with the landing spot highlighted, and let go |
|
||||
@@ -73,7 +73,7 @@ Alternatives considered:
|
||||
|
||||
### Where the extra outputs come from: spare outputs
|
||||
|
||||
KWin's nested backend opens its outputs at start (`--output-count`). The session starts KWin with the screen count plus `FLOAT_SLOTS` outputs (default and most 16). ft-floatd turns off the spares nothing floats on with `kscreen-doctor` once it starts. Floating a window enables a spare, and docking the window disables it again. `FLOAT_SLOTS` limits how many windows can float at once, and changing it means restarting the desktop. How KWin's nested backend treats disabled outputs, and why its virtual outputs can't be used instead, is in [design.md](design.md#floating-windows).
|
||||
KWin's nested backend opens its outputs at start (`--output-count`). The session starts KWin with the screen count plus `FLOAT_SLOTS` outputs (default 8, at most 16). ft-floatd turns off the spares nothing floats on with `kscreen-doctor` once it starts. Floating a window enables a spare, and docking the window disables it again. `FLOAT_SLOTS` limits how many windows can float at once, and changing it means restarting the desktop. How KWin's nested backend treats disabled outputs, and why its virtual outputs can't be used instead, is in [design.md](design.md#floating-windows).
|
||||
|
||||
ft-screens creates a `screen` for each toplevel in the order they appear, and indexes its settings by that order. Spares come after the screens, so they get indices `SCREENS` and up. Their panels are hidden while their output is disabled.
|
||||
|
||||
|
||||
@@ -1,87 +0,0 @@
|
||||
# Hands while typing on a real keyboard
|
||||
|
||||
Branch `hands-typing` (2026-10-10). The problem, from the user's headset session that day (hand
|
||||
tracking with student-v10a): typing on the real keyboard made hands click (pinches, middle pinches,
|
||||
grips, pokes), and the hand ray took the pointer from the mouse or the gaze during or right after
|
||||
typing.
|
||||
|
||||
## What the logs showed (2026-10-10)
|
||||
|
||||
Sources: `journalctl --user -u frametop-pointer` (the helper), `-u frametop-hands`, `-u
|
||||
frametop-input-relay`, and the four flags the user said "flag that" for
|
||||
(`frame-hands/captures/flags/flag-20261010-145753`, `-145841`, `-145857`, `-150115`: 16 s of hands
|
||||
in the head frame plus head poses). Keys aren't logged anywhere, so how long after a key a gesture
|
||||
came is known only as "within 1 s" (the old guard's "ignored (no ray up; typing)") or not.
|
||||
|
||||
- The old guard (`POINTER_PINCH_TYPING`, 1 s) caught 23 pinches today (09:12-09:42 and
|
||||
14:59-15:02), always "no ray up" ones. It never looked at ray pinches, middle pinches, grips,
|
||||
ft-screens' pokes and grabs, or the ray showing.
|
||||
- 14:57:10-14:57:30, typing into a text field on the TK-KB006 keyboard (connected 14:56:27, field
|
||||
focused 14:56:34): 11 pinches became "no ray up: a press where the pointer is, on nothing" and 2
|
||||
more were "ignored (a press is already held)". All began at 1.7-2.0 cm (deliberate pinches read
|
||||
0.4-1.0 cm), so typing thumbs. None was within 1 s of a key: typing pauses are longer than that.
|
||||
- 14:57:37 the right hand's ray came up ("hand ray: none -> right (raised)") with the hand on the
|
||||
keyboard, and at 14:57:38 a typing pinch became a ray press on `frametop.remote.103.bar`, held
|
||||
1.89 s. Flag 145753 shows that moment: both palms 0.29-0.33 m under the eyes, 0.26-0.34 m ahead,
|
||||
palms down (0.90-1.00 of the normal pointing down), 0.17-0.22 m apart. The knuckles sat 0.29-0.31 m
|
||||
under the eyes, right at the ray's "raised" limit (0.30 m to show).
|
||||
- 15:00:37 and 15:02:24: two more presses at the pointer between "ignored (typing)" pinches.
|
||||
- 15:01:15 (flag 150115): both hands typing at 0.40-0.42 m under the eyes, 0.18-0.21 m ahead, palms
|
||||
down 0.84-0.97, 0.15-0.16 m apart; the right hand rose, gripped, dragged a panel, then came back
|
||||
to 0.28 m under the eyes, palm down 0.84, and a ray pinch clicked `frametop.screen.1`.
|
||||
- Flags 145841 and 145857 are deliberate ray use: the pointing hand 0.15-0.30 m under the eyes and
|
||||
0.33-0.46 m ahead, palm anywhere from sideways to down; the other hand resting low (0.41-0.52 m
|
||||
under the eyes, -0.1 to 0.2 m ahead), 0.35-0.6 m away.
|
||||
- "pinch right: left to ft-screens (the hand is near its panel)" bursts (14:58:26-14:58:42,
|
||||
09:14:51-09:15:02) are hands near a low panel; with typing nearby those are ft-screens' to judge.
|
||||
|
||||
The hand recorder's desk take (PR #3, `frame-hands/dumps/pr3-04-desk-work`, head poses known):
|
||||
|
||||
| phase | palms under the eyes | ahead | palm down | apart | height difference |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| typing | 0.36-0.39 m | 0.22-0.29 m | 0.85-0.99 | 0.17 m (p90 0.40) | 0.00-0.01 m |
|
||||
| mouse (one hand on the keys) | 0.38-0.41 m | 0.21-0.29 m | 0.82-0.99 | 0.41 m | 0.00-0.02 m |
|
||||
| switching | 0.37-0.40 m | 0.21-0.29 m | 0.82-1.00 | 0.17-0.41 m | 0.00-0.02 m |
|
||||
|
||||
Deliberate gestures in the same recording (`pr3-03-gestures`): the hand 0.0-0.10 m under the eyes,
|
||||
0.26-0.37 m ahead. The touch take (`pr3-06-touch`, low dots) had the touching hand 0.24-0.30 m
|
||||
under the eyes, palm down 0.7-0.9, but 0.10-0.17 m higher than the other hand.
|
||||
|
||||
So typing hands sit anywhere from 0.29 to 0.42 m under the eyes depending on the desk and chair,
|
||||
which is also where deliberate gaze pinches with the elbow resting sit (0.35-0.45 m): no fixed
|
||||
height tells them apart. What does: both hands down together (level within a few cm, 0.15-0.25 m
|
||||
apart, palms down), and a hand that types rising well above where it typed when it's meant.
|
||||
|
||||
## What the branch does
|
||||
|
||||
`pointer/helper/typing.h` (tests: `pointer/tests/typing-test.cpp`) decides, per side, whether typing
|
||||
holds a hand back; the pointer helper feeds it the relay's keys and each hands frame, and ft-screens
|
||||
hears the result (`typing <L> <R>` on @ft_screens).
|
||||
|
||||
- Keys: for `POINTER_PINCH_TYPING` (2 s, was 1) after a key, every hand is held back, except one
|
||||
that has left the keyboard or is busy out of the keyboard zone (its ray up, a gesture held:
|
||||
pointing with one hand and pressing Enter with the other).
|
||||
- On the keyboard: a hand in the keyboard zone (palm at least `POINTER_TYPING_BELOW`, 0.2 m, under
|
||||
the eyes, 0.05-0.45 m ahead, palm down) while typing is parked, and stays held back through any
|
||||
pause until it rises `POINTER_TYPING_RAISE` (0.08 m) above where it typed, or turns its palm up,
|
||||
for 0.1 s. Freed, it works at once, keys or not. A hand on the mouse beside the keyboard is parked
|
||||
too once a key comes (its pinches are mouse clicks). Unseen 3 s, it's forgotten.
|
||||
- The posture: both hands in the zone, level within 8 cm and at most 32 cm apart, for
|
||||
`POINTER_TYPING_POSTURE` (0.3 s), is typing before any key, within `POINTER_TYPING_RECENT` (600 s)
|
||||
of a key. Without that limit, both hands resting on a desk would hold gaze pinches back.
|
||||
- Held back: no pinch, middle pinch or grip (ray, gaze or plain) in the helper; no ray (an owner
|
||||
drops the pointer to the mouse or the gaze, "hand ray: right -> none (typing)"); no poke, near grab
|
||||
or fingertip cursor in ft-screens. A gesture already held carries on.
|
||||
|
||||
Replaying the four flags and the PR #3 takes through typing.h (`/tmp` driver, not in the repo; the
|
||||
flags have no keys, so a key a minute before the window was assumed):
|
||||
|
||||
| recording | result |
|
||||
| --- | --- |
|
||||
| flag 145753 (typing, the ray click at 14:57:38) | both hands parked by the posture 0.3 s into the window, held back 98% of it; the 14:57:38 pinch came 0.27 s into the window, before the replay's posture (live, the hands had been on the keys since 14:57:10) |
|
||||
| flag 150115 (typing, then the right hand up for a grip, a drag and a click) | posture at once; the right hand freed by its raise (0.18 m), its grip and two pinches allowed; the left held back 98% |
|
||||
| flags 145841, 145857 (deliberate ray use, the other hand low) | nothing held back, all 11 pinches allowed |
|
||||
| desk take, keys during the typing prompt | typing, mouse and switching: both hands held back 100% |
|
||||
| desk take, no keys | the posture alone: typing 84-100%, mouse phase: the mouse hand free (0.41 m from the other hand) |
|
||||
| gestures take (pinches, grabs, pointing) | nothing held back |
|
||||
| touch take (low dots) | the posture took both hands while they waited low; the touching hand freed by its first raise, then held back 9% |
|
||||
+1
-1
@@ -25,6 +25,6 @@ ft-screens drops keys while no screen has focus or the SteamVR dashboard is open
|
||||
|
||||
## Typing and grabbed keyboards
|
||||
|
||||
- **Programs that watch every keyboard lose grabbed ones.** While typing goes to the desktop, the relay grabs pass-through keyboards, so a hotkey tool reading them directly stops seeing their keys. Bind the tool's key as a Frametop key combination that runs a command (`command:CMD`) instead: the relay sees every keyboard. Frametop Voice's push-to-talk key works this way. (Until 2026-10-10, `SHARE_KEYS=1` sent every key typed into the desktop to the abstract socket `@frametop_keys`, which any local process could bind first.)
|
||||
- **Programs that watch every keyboard lose grabbed ones.** While typing goes to the desktop, the relay grabs pass-through keyboards, so a hotkey tool reading them directly stops seeing their keys. `SHARE_KEYS=1` in `~/.config/frametop.conf` sends their keys to the abstract socket `@frametop_keys` instead. It's off by default: abstract sockets have no permissions, and any local process that binds the name first receives every key typed into the desktop, passwords included.
|
||||
- **Typing starts out going to Steam.** After the desktop starts, keys go to Steam until you click a screen.
|
||||
- **Controller clicks don't move typing.** Overlay apps don't see controller clicks on other panels, so after one typing stays where it was. A mouse click, or a click on a screen, moves it.
|
||||
+17
-60
@@ -30,8 +30,6 @@ kwriteconfig6 --file ~/.config/frametop/kwinrc --group Plugins --key contrastEna
|
||||
kwriteconfig6 --file ~/.config/frametop/kdeglobals --group KDE --key AnimationDurationFactor 1
|
||||
```
|
||||
|
||||
The desktop's notifications can show outside it, in the Steam session. Plasma there runs on the desktop's own D-Bus, so its popups only appear on the desktop's screens. `session/ft-notifyfwd` (Plasma's autostart; log: `/tmp/frametop-notifyfwd.log`) hands each one to the notification server on the user bus, the one the Steam session's apps use, for example a VR notification server that draws them as panels. Its buttons work: a click there reaches the app as if made in Plasma, and dismissing it there closes it in Plasma. While it forwards, Plasma's own popups are off (do not disturb, held by "frametop"), and Plasma's notification history still has every one. `NOTIFY_FORWARD` in `~/.config/frametop.conf` (or Notifications on the Visibility & pins tab of Frametop Display Settings) sets it: `auto` (the default) forwards only when the user bus's server isn't SteamOS's own `steam_notif_daemon`, which hands notifications to a Steam client that can't show them on the Frame, so with stock SteamOS nothing changes; `on` forwards to any server, SteamOS's too, where they're lost (it's in the file only; the settings app shows it while it's set); `off` keeps them in the desktop. Restart the desktop after a change. Notifications Flatpak apps send through the notification portal go along too. KDE's inline reply shows as a plain Reply button there; see [design.md](design.md#notifications-outside-the-desktop).
|
||||
|
||||
Three of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`, and Steam (`steam`), which is already running. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`; Steam's was added later and is hidden once on desktops that already had the other two), and skips a name you already have a file for. Delete a copy to start that program again.
|
||||
|
||||
Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in `~/.config/frametop-layout.json`. The backend, remote desktop, and pointer settings are in `~/.config/frametop.conf`; `session/frametop.conf.example` lists every key.
|
||||
@@ -49,7 +47,7 @@ Every screen is an overlay named `frametop.screen.N` with five controls:
|
||||
- `.bar` moves the screen. Drag it with any laser or with the 3D mouse, whose right-drag tilts. Scrolling while you drag pushes the screen away or pulls it closer, along the line from your head.
|
||||
- `.curve` bends the screen into a cylinder around you, using your current distance as the radius, or makes it flat again.
|
||||
- `.roll` rolls the screen when you drag it sideways, like a knob. It snaps level within 2.5°, and scrolling on it turns 5° per notch.
|
||||
- `.resize`, the handle on a corner, sets the size from there, the opposite corner staying put (a screen keeps its aspect; a floating window gets that many pixels). It sits on the bottom right, and goes to whichever corner a laser, the gaze or a hand's pinch comes near, brightening as it comes; lit blue while resizing. Screens go down to 15 cm wide.
|
||||
- `.resize`, the tab on the bottom right corner, sets the width. Screens go down to 15 cm wide.
|
||||
- `.reset`, left of the bar, is the quick reset, like Meta+Shift+R (`ft-layout reset`): with a profile in use it opens that profile again, as Open profile does, and otherwise it puts every screen back in its layout around where you are now.
|
||||
|
||||
The controls are sized from both the screen's width and its distance from you, follow the surface of a curved screen, and stay invisible until a laser or the 3D mouse's cursor lands on one or comes within about 1.5 times a button's size of it. While invisible they're still there, fully transparent, so SteamVR's laser can find them. They're translucent until a laser is on them, like SteamVR's own window controls.
|
||||
@@ -70,7 +68,7 @@ In the last three modes the hotkey shows the screens anyway. A screen can also b
|
||||
- During VR games, the Always mode hides the screens unless the dashboard is open (the default), or leaves them up.
|
||||
- Controllers on the screens. Visible screens can keep SteamVR's laser mouse on, so controllers work them with the dashboard closed, but that also takes the controllers away from a game. By default this is off while a VR game runs, and the 3D mouse or the dashboard works the screens. Pointing a controller at a screen, a floating window, or the keyboard still turns its laser on, like SteamVR's own floating windows, and pointing away gives the game the controllers back. The other choices are always on, or only with the dashboard open, which also suits flatscreen games since they aren't scene apps.
|
||||
|
||||
Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay: from pass-through keyboards, a keyboard's media keys (its Consumer Control node), and any key a pointer device passes through. A mouse button passed through as a key in pointer mode (a side button for Back) goes to the screen the pointer is on instead, wherever typing goes, and only while the pointer is on a screen that shows and Frametop isn't paused. Its release always goes through, and ft-screens releases it itself when the pointer leaves the screens, its screen hides, or Frametop pauses (`scripts/test-relay-buttons.sh` checks this offline). Typing follows your last click: after a click on a screen it goes to the desktop, even with the SteamVR dashboard open, and after a mouse click on any other panel (the dashboard, Steam, an app like Spotify) it goes there instead. While it goes to the desktop, the relay grabs pass-through keyboards so gamescope, which reads every keyboard itself, doesn't type them into the Steam app too. A program that watches every keyboard for a hotkey loses a grabbed one; bind its key as a key combination with a command instead (Frametop Input Settings, Keyboard). Frametop Voice types its text through the relay the same way, so it lands where typing goes. Hidden screens don't take typing.
|
||||
Input from the lasers reaches KWin through ft-screens' own seat. Keys come from the input relay: from pass-through keyboards, a keyboard's media keys (its Consumer Control node), and any key a pointer device passes through. A mouse button passed through as a key in pointer mode (a side button for Back) goes to the screen the pointer is on instead, wherever typing goes, and only while the pointer is on a screen that shows and Frametop isn't paused. Its release always goes through, and ft-screens releases it itself when the pointer leaves the screens, its screen hides, or Frametop pauses (`scripts/test-relay-buttons.sh` checks this offline). Typing follows your last click: after a click on a screen it goes to the desktop, even with the SteamVR dashboard open, and after a mouse click on any other panel (the dashboard, Steam, an app like Spotify) it goes there instead. While it goes to the desktop, the relay grabs pass-through keyboards so gamescope, which reads every keyboard itself, doesn't type them into the Steam app too. A program that watches every keyboard for a hotkey loses a grabbed one; with `SHARE_KEYS=1` in `~/.config/frametop.conf`, their keys also go to `@frametop_keys` for it. That's off by default, since any local process that binds the name first would get everything typed into the desktop. Hidden screens don't take typing.
|
||||
|
||||
Frametop's keyboard opens by itself when a text field on the desktop gets focus, and stays open until its Close key, a layout reset, or a mapped button closes it (or, with Keep it open off in Frametop Input Settings, until the text field loses focus). While the Steam menu (the dashboard) or Steam's own keyboard is up, it steps aside, and it comes back where it was when they're gone; one asked for meanwhile appears then. In the "only with the dashboard" visibility mode, the dashboard doesn't count. It doesn't open without a head pose (the headset in standby). It's a panel of keys (a US laptop layout, with Esc where Caps Lock would be, arrows, and a Close key) that ft-screens shows 0.7 m in front of you and below your eyes, facing you. It stays where it opened, and its grab bar (the pill along the top) moves it like a screen's. Type on it with a controller's laser or the 3D mouse. Shift, Ctrl and Alt latch for the next key, and a held key repeats. KWin starts `input/ft-textinput` as the desktop's input method, and KWin activates it whenever the focused app turns on text input for a field. It tells the relay (`textfield 1` or `0`), the relay decides by the Keyboard setting in Frametop Input Settings, and ft-screens opens the keyboard for the screen that has keyboard focus (`vrkeyboard show`, `hide`, or `toggle` from a mapped button). Its keys reach the focused screen as key presses, so it works in every app, but only apps that use Wayland text input (Qt, GTK, Firefox) open it by themselves; Chromium, Electron and X11 apps need the button. The session drops the `QT_IM_MODULE=xim` and `GTK_IM_MODULE=xim` that the gamescope session sets, or Qt and GTK apps wouldn't use Wayland text input either.
|
||||
|
||||
@@ -85,18 +83,13 @@ get N screens head state key code value scale N s vrkeyboard s
|
||||
visibility always|dashboard|gesture|toggle wrist degrees gesture left|right degrees
|
||||
hide | show | toggle controllers always|outside_games|dashboard ingames hide|visible pause on|off|state
|
||||
conceal N|all reveal N|all concealed cutouts on|off|state cutouts predict on|off cutouts lead ms
|
||||
cutouts matte circle|mesh|capsules cutouts margin deg [ms [deg]] cutouts shift off|spec cutouts roomview auto|on|off
|
||||
hands L R allowed handinput on|off|force|state palmmenu on|off|state grabthat on|off|state
|
||||
float N mpp x y w h title unfloat N pose N matrix sub N k x y w h | sub N k off minimized N 0|1 carry N
|
||||
rates focused in_view hidden rates? watch seconds phase ms
|
||||
spin next|prev|DEGREES|to N front N spotwatch on|off
|
||||
```
|
||||
|
||||
Each screen draws at a frame rate for how much of it you see. KWin draws a screen only after ft-screens gives it a frame callback, and its apps wait for theirs, so the rate of callbacks is the screen's frame rate, for KWin and the apps on it alike. A screen is focused while you look at it (within 12 degrees of where your head points), while a laser or the mouse is on it or was in the last 1.5 seconds, while it's carried, and while you type on it; it gets every display frame. The rest of what you can see (within 60 degrees) gets 15 frames a second, and a hidden screen, one behind you, and everything while paused get one a second. A level goes up at once and comes down after a moment (1.5 s from focused, 0.5 s from in view). A video, or anything moving over a large part of a screen (6% or more of it, redrawn on 8 commits in a row, 10 or more a second), keeps every frame while in view. A floating window is a screen of its own here. Nothing that stands still costs anything at any rate: KWin sends a frame only when something on the screen changed. `rates F V H` sets the three rates in Hz (0: every display frame; default `0 15 1`, also `ft-screens --rates 0,15,1`), and `rates?` shows them, the display's rate, and for each screen its level, its milliseconds between frames, and whether it counts as a video. `watch S` gives every screen full rate for S seconds: remote desktop renews it while a VNC client is connected, since a viewer sees what KWin draws. The ticks (SteamVR events and the callbacks) come once per display frame, 1 ms after the vsync (`phase ms` changes that, for tuning), in step with the display rather than on a timer that drifted through the frame.
|
||||
|
||||
`spin` is the lazy susan. By default it brings panels to a front slot at reading distance and leaves the rest where they are: `next` and `prev` bring the next one round on the right or left (the one in the slot goes home), and `to N` brings panel N. With `SCREEN_SPIN_BRING=0` in `~/.config/frametop.conf` it turns every panel in the room about your head instead: `next` and `prev` bring the next one on the right or left to straight ahead, and `to N` turns the short way to panel N. Either way, a number of degrees turns the room that far. Typing goes to the panel that ends up in front. `front N` gives panel N the typing where it is, as a spin to it would. `spotwatch on`, renewed every second, gives spotwatch's search the typing, and `spotwatch off` gives it back ([Spotwatch](#spotwatch)).
|
||||
|
||||
`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `pause on` (from the input relay, when Frametop pauses for a VR game) hides every screen and floating window whatever else says, and slows the desktop down; `pause off` undoes it. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`), and sets their matte, its margin and the per-distance correction (hands/README.md, "Hand cutouts and near input"); they cut only with Room View on, detected or forced (`cutouts roomview`, hands/README.md "Rendered hands"). `hands` is the pointer helper's word on hand input (each side `hand`, `controller` or `off`, and whether it's allowed now); `handinput` turns near input on (following that word), off, or on without it, and `handinput state` reports what each hand is doing, its counts (presses, one-frame presses, pinch clicks, grabs by card and fist, poke attempts and those refused, crossings off the content, presses at the edge, flicks and throws, holds through tracking loss, second hands joining a held panel, pinches that waited to become fists, buttons pinched) and what held presses back; each press, release, refused poke and grab also goes to ft-screens' log as a `hands:` line. `palmmenu` turns the palm menu (a bare hand palm up, pinch: the Steam menu) on and off, and `palmmenu state` reports it. `grabthat` turns "grab that" (an open hand held out to a panel farther than arm's reach, then a fist pulled back: it comes to arm's reach) on and off, and `grabthat state` reports each hand's state, its counts and the panels that came. The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar.
|
||||
`conceal` and `reveal` hide and show one screen on its own (`ft-layout hide` and `show` send them), and `concealed` lists those screens. `pause on` (from the input relay, when Frametop pauses for a VR game) hides every screen and floating window whatever else says, and slows the desktop down; `pause off` undoes it. `cutouts` turns the hand cutouts on and off (`ft-handsctl cutouts`). The last line is ft-floatd's, for floating windows: N is a floating window's panel, numbered on from the screens, one per spare output. `float` gives the window's rectangle in its output, metres per pixel, and the title bar's height, and shows the panel; `unfloat` hides it. `pose` places it (a 3x4 matrix, standing universe), `sub` shows popup or dialog k over it, `minimized` hides it while its window is minimized, and `carry` moves it with the laser that pressed the window's own title bar.
|
||||
|
||||
## Input relay
|
||||
|
||||
@@ -122,10 +115,9 @@ A mouse drives SteamVR the way a controller's laser does, but shows up as a smal
|
||||
|
||||
- The input relay, in pointer mode (`POINTER=1`), sends mouse motion, clicks, and scrolling to the helper. A deliberate movement or a click wakes the pointer, and 30 seconds without mouse input releases it.
|
||||
- The helper, `pointer/helper/ft-pointer`, runs in the `dev` container as `frametop-pointer.service` and starts with SteamVR. It keeps the cursor, tests it against every visible overlay, draws the dot, and sends the driver an exact pose.
|
||||
- The driver, `pointer/driver/` (`ft_pointer`), is loaded by SteamVR. It's an invisible virtual controller whose laser follows the cursor. It takes the treadmill role, no hand, so the real controllers keep theirs and one can be used beside it (`POINTER_ROLE` and the driver's `role` setting; see docs/design.md, "Treadmill role").
|
||||
- The dashboard patch, `pointer/dashpatch/` (`frametop-dashpatch.service`, starts with SteamVR), lets that treadmill move SteamVR's own windows by their grab bars, which SteamVR allows only for the head or a hand. It goes into SteamVR's dashboard page through the dashboard's debugger, which the installer turns on at 127.0.0.1:8089 (`VRWebHelper/DebuggerEnabled`; the uninstaller turns it off). `pointer/dashpatch/run.sh status` shows whether it's in.
|
||||
- The driver, `pointer/driver/` (`ft_pointer`), is loaded by SteamVR. It's an invisible virtual right-hand controller whose laser follows the cursor.
|
||||
|
||||
Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. With hand interaction (`POINTER_HANDS=1`), this goes per hand: a controller used recently wins its hand, and the other hand can stay bare. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again.
|
||||
Whichever device you used last wins. Picking up a controller hands the laser back at once, and moving the mouse takes it again. When the headset comes off, the pointer lets go, so the displays can sleep, and it stays off until you're wearing the headset again.
|
||||
|
||||
To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pulls it while you drag. Hold the right button while dragging and move the mouse to tilt the panel around the grab point; the right press isn't sent as a click. The tilt stays for the rest of the drag, and releasing the left button drops the panel as it is. A mapped Toggle dashboard button wakes the pointer if needed and holds the virtual system button for 0.12 s, because SteamVR ignores a press and release in the same instant. Open Steam menu / close dashboard (`steam_menu`) needs no pointer: `steam/ft-steam menu` asks Steam's UI, over its debugging port (`steam/steamui.py`), to show its dashboard overlay and focus the Steam frame's menu, or to hide the dashboard if it's up.
|
||||
|
||||
@@ -133,22 +125,20 @@ To move a floating panel, left-drag its grab bar. The scroll wheel pushes and pu
|
||||
pointer/driver/build.sh && pointer/driver/install.sh install # then restart SteamVR
|
||||
pointer/helper/build.sh && pointer/helper/run.sh install
|
||||
pointer/helper/run.sh status | log | restart
|
||||
pointer/driver/install.sh probe # devices, hand roles, activity, charging, who owns the dashboard pointer
|
||||
pointer/probe/build.sh # probes: vrprobe, lasertest (who has the laser), ctlnoise (controller noise)
|
||||
pointer/driver/install.sh probe # devices, hand roles, who owns the dashboard pointer
|
||||
```
|
||||
|
||||
The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_CONTROL_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
|
||||
The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`) and `GAZE_EYE` (the eye bias). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its two again when the file changes).
|
||||
|
||||
## Frametop Input Settings
|
||||
|
||||
A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has ten pages:
|
||||
A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has nine pages:
|
||||
|
||||
- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
|
||||
- Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, pause or resume Frametop ([Pausing for VR games](#pausing-for-vr-games)), Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
|
||||
- Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
|
||||
- Game optimization has the pause for VR games: its state with Pause now or Resume, whether VR games pause Frametop by themselves, the controller gesture (one or two buttons, pressed once or twice; one button always takes two presses), what happens to the desktop, and the sound. They're saved as `pause_auto`, `pause_gesture`, `pause_desktop`, and `pause_sound` in `~/.config/frametop-input.json`. See [Pausing for VR games](#pausing-for-vr-games).
|
||||
- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), Meta+Shift+F (float window in VR or put it back), and Meta+Alt+Tab and Meta+Alt+Shift+Tab (spin the panels: every screen and floating window turns about your head, so the next one on the right or left comes to the front; ft-screens' `spin next|prev|<degrees>`); remove them or add others there. With Frametop Voice installed, a Right Ctrl tap starts and stops dictation too. A tap is a press with no other modifier held, then a release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. Tapping a left modifier binds either side's; tapping a right one (Right Ctrl) binds that key alone. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
|
||||
- Voice has Frametop Voice's settings (see Voice dictation): on or off, the dictation keys (set on the Keyboard page), the controller buttons and their mode, typing as you talk, sounds, and the wake word and its phrases. They're saved in `~/.config/frametop-voice.json`, and the service applies them within 2 seconds.
|
||||
- Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput, like frame-voice's, don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), Meta+Shift+F (float window in VR or put it back), and Meta+Alt+Tab and Meta+Alt+Shift+Tab (spin the panels: every screen and floating window turns about your head, so the next one on the right or left comes to the front; ft-screens' `spin next|prev|<degrees>`); remove them or add others there. A tap is a press and release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
|
||||
- Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
|
||||
- Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
|
||||
- Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
|
||||
@@ -166,7 +156,7 @@ Frametop Display Settings has four tabs (three with the gamescope backend, which
|
||||
|
||||
- Screens: add and remove screens, and set each one's resolution (presets from 1080p to 4K, ultrawide, super ultrawide, portrait, or custom), its width in VR (0.5 to 6 m), its scale, whether it's curved, and whether it has the taskbar. Resolution, width, and curve apply at once. Adding or removing a screen takes a desktop restart, which the app offers.
|
||||
- Layout (the Layout & profiles page): the Arrangement list starts with two presets: Curved around you, with the screens hinged edge to edge like monitors on a desk and each turned to face you, and Flat wall. Both take rows, distance, gap, and height, and Arrange now applies them. Save as profile… saves where the screens are now (positions, sizes, curves, and pins), which ones are hidden, and the open apps and where their windows are, under a name. Profiles are listed in Arrangement after the presets: pick one and Open profile switches to it, and the buttons next to the list rename or delete it. Below it, Apps lists the profile's windows and where they go, and Leave out drops one. A profile saved with fewer screens than you have now leaves the others where they were saved last, or where the preset would put them. Start in profile picks the profile the desktop starts in; with None, a switch turns auto-arrange at startup on or off. A preview shows the layout from above and from the front. See [profiles.md](profiles.md).
|
||||
- Visibility & pins: the visibility, game, and controller settings described above, the wrist angle, where each screen is pinned (in the room, a wrist, or your head), and buttons to pin all screens or unpin them. Its Notifications section sets `NOTIFY_FORWARD` (whether the desktop's notifications show in the Steam session, see [The desktop](#the-desktop)), shows where they go now, and offers a desktop restart after a change.
|
||||
- Visibility & pins: the visibility, game, and controller settings described above, the wrist angle, where each screen is pinned (in the room, a wrist, or your head), and buttons to pin all screens or unpin them.
|
||||
- Power: when the displays turn off while the headset isn't used, their state now, Turn displays off now (to try it), and Stay awake while plugged in. See [Displays off and sleep](#displays-off-and-sleep).
|
||||
|
||||
`layout/ft-layout` does the arranging. It's a Python script that uses only the standard library and runs on the host:
|
||||
@@ -191,7 +181,7 @@ The layout is stored relative to your head when it's applied. `/run/user/<uid>/f
|
||||
|
||||
## Frametop Remote Displays
|
||||
|
||||
Other computers' monitors as Frametop screens (ft-screens only), streamed from Vibepollo or Sunshine with Moonlight's protocol. Frametop Remote Displays (`remote-displays/`, also opened by the Remote displays button on Display Settings' Screens page) finds Vibepollo and Sunshine computers on the network, signs in to one with its Web UI login (it keeps a narrow API token, not the password, and pins the host's certificate), shows whether it answers, and adds its displays: its monitors, or a virtual one at any size. A computer with a Steam Link dongle on the Frame's hotspot streams over it (Connection: auto, network or dongle only); the others use the network. Each display has a Connected switch (and the host one for all of its displays), Shown, its stream's resolution, frame rate and bitrate, and its width in VR. On Windows, a monitor switches to its stream's resolution while it streams (or its closest mode with the same shape), and back when the stream ends. That takes Frametop's build `frametop-2.0.0-2` or newer; `frametop-2.0.0-1` scaled the monitor's own resolution into the stream. A disconnected display keeps its settings, and within the same desktop run, its place. In VR each one is a panel with a screen's controls, and profiles keep where they are. See [remote-displays.md](remote-displays.md).
|
||||
Other computers' monitors as Frametop screens (ft-screens only), streamed from Vibepollo with Moonlight's protocol. Frametop Remote Displays (`remote-displays/`, also opened by the Remote displays button on Display Settings' Screens page) finds Vibepollo computers on the network, signs in to one with its Web UI login (it keeps a narrow API token, not the password, and pins the host's certificate), shows whether it answers, and adds its displays: its monitors, or a virtual one at any size. A computer with a Steam Link dongle on the Frame's hotspot streams over it (Connection: auto, network or dongle only); the others use the network. Each display has a Connected switch (and the host one for all of its displays), Shown, its stream's resolution, frame rate and bitrate, and its width in VR. A disconnected display keeps its settings, and within the same desktop run, its place. In VR each one is a panel with a screen's controls, and profiles keep where they are. See [remote-displays.md](remote-displays.md).
|
||||
|
||||
```
|
||||
layout/ft-layout remote list # the remote displays and their streams' state
|
||||
@@ -199,15 +189,13 @@ layout/ft-layout remote connect|disconnect ID...
|
||||
remote-displays/install.sh # its menu entry
|
||||
```
|
||||
|
||||
On the PC, `host/windows/Setup Frametop host.cmd` sets it up for Frametop: Vibepollo 2.0.0 (installed if missing), Frametop's build of its `sunshine.exe`, the settings Frametop needs, the Web UI login you sign in with from the Frame, and a firewall check (`-Check` to see what it would change, `-Undo` to put things back). [host/windows/README.md](../host/windows/README.md) has the steps, and the [`windows-host-1`](https://github.com/Frametop/frametop/releases/tag/windows-host-1) release has the kit as a zip. A Linux PC (Arch or CachyOS) installs Frametop's Vibepollo package with Vibepollo's installer, as [host/linux/README.md](../host/linux/README.md) says; it hasn't been tested yet.
|
||||
|
||||
On a Mac, `host/macos/frametop-host-setup.sh` (or `Setup Frametop host.command`) installs Sunshine and runs one instance per real display, each on its own port (47989 + 100·N), started at login. You allow Sunshine Screen Recording and Accessibility once in System Settings. Without Accessibility the Frame's pointer moves on the Mac but clicks, scrolling and typing don't arrive; the setup checks both, and offers to ask again if Accessibility is missing. Remote Displays finds the Mac as one computer; signing in with the Web UI login pairs the displays you pick, and the password isn't kept, so adding more later signs in again. Macs have no virtual displays (`--check`, `--undo` as on the PC). Frametop Streams in the menu bar turns each display on or off with a click, or all of them, and `ft-sunshine off|on|status` does the same in Terminal. Off lasts through logins until on. [host/macos/README.md](../host/macos/README.md) has the steps for setting up a Mac, and the [`macos-host-1`](https://github.com/Frametop/frametop/releases/tag/macos-host-1) release has the kit as a zip.
|
||||
On the PC, `host/windows/Setup Frametop host.cmd` sets it up for Frametop: Vibepollo 2.0.0 (installed if missing), Frametop's build of its `sunshine.exe`, the settings Frametop needs, the Web UI login you sign in with from the Frame, and a firewall check (`-Check` to see what it would change, `-Undo` to put things back).
|
||||
|
||||
## Floating windows
|
||||
|
||||
A desktop window can float in VR as a panel of its own, away from the screens. Meta+Shift+F floats the window under the pointer (or the active one, over the wallpaper), or puts it back on its screen if it floats. So do Float in VR in every window's menu (Alt+F3; Back to Desktop on a floating one), the button left of Close in its title bar, and a mouse button, controller button, or key combination mapped to Float window in VR in Frametop Input Settings; Put all floating windows back is mappable too. Launch as Standalone, in an app's right-click menu in the Application Launcher or the taskbar, starts the app with its first window floating, where that app last floated or in front of you. [floating-windows.md](floating-windows.md) explains how it works.
|
||||
|
||||
`float/ft-floatd` does this. It runs inside the desktop's Plasma session (log: `/tmp/frametop-floatd.log`), loads the KWin script `float/frametop-float.js`, and moves each floating window to a spare KWin output of its own, which ft-screens shows as a panel cropped to the window. The settings are in `~/.config/frametop.conf`: `FLOAT_SLOTS` is how many windows can float at once (16, the most; 0 turns floating off; restart the desktop after a change), and `FLOAT_MARGIN` the pixels around each window on its output, so menus have room past its edges (300). Each app's last floating place, size, and scale are kept in `~/.config/frametop-float.json` by desktop file name, relative to the primary screen, so they move with the screens. `FT_FLOAT_DEBUG=1` in ft-floatd's environment logs every event from the KWin script.
|
||||
`float/ft-floatd` does this. It runs inside the desktop's Plasma session (log: `/tmp/frametop-floatd.log`), loads the KWin script `float/frametop-float.js`, and moves each floating window to a spare KWin output of its own, which ft-screens shows as a panel cropped to the window. The settings are in `~/.config/frametop.conf`: `FLOAT_SLOTS` is how many windows can float at once (8, at most 16; 0 turns floating off; restart the desktop after a change), and `FLOAT_MARGIN` the pixels around each window on its output, so menus have room past its edges (300). Each app's last floating place, size, and scale are kept in `~/.config/frametop-float.json` by desktop file name, relative to the primary screen, so they move with the screens. `FT_FLOAT_DEBUG=1` in ft-floatd's environment logs every event from the KWin script.
|
||||
|
||||
With floating on, the session gives the desktop's windows Frametop's own decoration (`decoration/`): Breeze's look plus the float button. Apps that draw their own title bar, like Chromium and Electron apps, don't have the button. `decoration/apply.sh` puts a changed copy into the running desktop, and `decoration/apply.sh --off` goes back to Breeze until the next desktop start.
|
||||
|
||||
@@ -219,20 +207,6 @@ float/ft-float launch org.kde.dolphin # start an app (its desktop file name) flo
|
||||
float/ft-float run COMMAND [ARG...] # the same for a command
|
||||
float/ft-float close ID # close a window
|
||||
float/ft-float list # the spare outputs and what floats on them
|
||||
float/ft-float launch firefox action=new-private-window # one of the app's own actions, floating
|
||||
float/ft-float launch org.kde.dolphin screen=2 # its first window docked on screen 2
|
||||
float/ft-float activate ID # make a window the active one, unminimized
|
||||
```
|
||||
|
||||
## Spotwatch
|
||||
|
||||
Meta+Space opens spotwatch, a search pill straight ahead of where your head points, for apps and remote displays (`{host} - {display}`). Tab, Shift+Tab, and the arrow keys move through what it finds, and Enter or a click opens it. A running app's most recent window spins to the front with the lazy susan. An app that isn't running starts floating. A remote display connects if needed and spins to the front. Shift+Enter or a right click opens an item's menu: an app's own actions (a private window), Open new window, Open docked on this screen, and its windows; a remote display's Go to and Disconnect. Esc, Meta+Space again, or a click elsewhere closes it. It's the `spotwatch` action in Frametop Input Settings, bindable to mouse and controller buttons too, and opened that way it brings Frametop's keyboard. [spotwatch.md](spotwatch.md) has the decisions and how it works.
|
||||
|
||||
`spotwatch/ft-spotwatch` runs inside the desktop's Plasma session (log: `/tmp/frametop-spotwatch.log`) and runs the pill, `ft-spotpanel` (built by `screens/build.sh`), in the container. While the search is open, ft-screens sends it every key. Use counts are in `~/.config/frametop-spotwatch.json` and icons in `~/.cache/frametop/spotwatch`.
|
||||
|
||||
```
|
||||
spotwatch/test/spotwatch-test.py # the ranking and the search against a fake ft-screens and ft-floatd (offline)
|
||||
spotwatch/tools/render.py OUT.png # the panel's picture, without a headset
|
||||
```
|
||||
|
||||
## Displays off and sleep
|
||||
@@ -263,7 +237,7 @@ Paused, Frametop leaves the headset's CPU and GPU to a VR game. The input relay
|
||||
- The gaze service stops (`frametop-gaze`: ft-gazed, ft-gaze, our own eye tracker, the gaze panel), so nothing reads SteamVR's eye tracking. Our frame grabber, the root service `ft-eyegrab`, goes idle by itself 3 seconds after our eye tracker stops asking it for frames.
|
||||
- Hand tracking stops if it runs (`frametop-camd`, `frametop-hands`).
|
||||
- The desktop, as the Game optimization page of Frametop Input Settings says (`pause_desktop`): hidden (the default) or closed. Hidden, ft-screens hides every screen and floating window whatever the visibility mode, the hotkey, or the dashboard says, and gives KWin a frame callback once a second instead of every display frame. KWin draws a screen only after its frame callback, and its apps wait for theirs, so the desktop hardly draws, but its windows stay open. Remote desktop stops if it runs (`session/remote-ctl.sh`). Closed, `desktops.sh stop` closes the desktop and its windows, and resuming starts it again (about 12 seconds), in its start profile if it has one.
|
||||
- The relay lets go of the 3D mouse, releasing any click still held on it (a mouse button, a mapped controller button, or a key combination), and feeds pointer devices to its virtual mouse and keyboard, as with `POINTER=0`. Typing goes to Steam. Mapped buttons and key combinations do nothing but pausing, the Steam menu, dictation, and commands; a key combination that does nothing is typed as usual.
|
||||
- The relay lets go of the 3D mouse, releasing any click still held on it (a mouse button, a mapped controller button, or a key combination), and feeds pointer devices to its virtual mouse and keyboard, as with `POINTER=0`. Typing goes to Steam. Mapped buttons and key combinations do nothing but pausing, the Steam menu, and commands; a key combination that does nothing is typed as usual.
|
||||
|
||||
Resuming starts again only what pausing stopped, and plays a second sound. The pointer helper and ft-powerd keep running: they cost little, the helper is what says a game started, and stopping it would leave its virtual controller connected with its last pose.
|
||||
|
||||
@@ -284,23 +258,6 @@ input/test/pause-buttons-test.py # a click held into a pause comes up, offline
|
||||
|
||||
The state outlives a relay restart, in `/run/user/UID/frametop-pause.json`. A SteamVR restart while paused starts the gaze service with it, and the relay stops it again when the pointer helper comes back.
|
||||
|
||||
## Voice dictation
|
||||
|
||||
Frametop Voice (`voice/`, [voice/README.md](../voice/README.md)) types what you say where you type, recognised on the headset; nothing leaves it. `voice/ft-voiced` runs in Frametop's container as `frametop-voice.service` (installed by `voice/run.sh install`, which `install.sh` offers, yes by default), starting at login. It records with `pw-record` and transcribes live with Moonshine on the CPU; `frametop-voice-whisper.service` (whisper.cpp on the GPU) runs only while the wake word is on, for its commands. Its models, about 320 MB, are downloaded on install into `~/.local/share/frametop/voice/models` (`voice/models.sh`: fixed URLs, checked by sha256).
|
||||
|
||||
The input relay does the keyboard side. Dictation's keys are key combinations with the actions `dictate` (a press starts or stops) and `dictate_hold` (records while held); with Frametop Voice installed, the default key combinations add a Right Ctrl tap (`"97": "dictate"`, Right Ctrl alone: a right-hand modifier's own code is that key's tap only). Like every tap, it doesn't count with another modifier held, or with a key, click or scroll in between. The relay sends `toggle|start|stop <source>` to `@frametop_voice`, and dictation works while Frametop is paused. ft-voiced sends its text back as `type TEXT` on `@frametop_relay`; the relay types it, one key every 4 ms on a US layout, to ft-screens while typing goes to the desktop, otherwise on its virtual keyboard for Steam and games, and holds a character back while a modifier is held. Both sockets take these only from their own user. Frame controller buttons (left d-pad down, held, by default) are read by ft-voiced itself from vrserver's web socket (`input/vrws.py`), so games still get them.
|
||||
|
||||
Frametop Input Settings' Voice page turns it on and off and sets the controller buttons and their mode (hold or press to toggle), typing as you talk, sounds, and the wake word (off by default) and its phrases, in `~/.config/frametop-voice.json`; the keys are on its Keyboard page. Wake word commands that run a program, and advanced settings, are edited in that file by hand (voice/README.md).
|
||||
|
||||
```
|
||||
voice/build.sh && voice/run.sh install
|
||||
voice/run.sh status # the services, and ft-voiced's state as JSON
|
||||
voice/ft-voice toggle # start or stop dictation; also start, stop, cancel, type TEXT, wake on|off, events
|
||||
voice/run.sh log
|
||||
voice/models.sh check # every model's sha256
|
||||
voice/test/test_daemon.py # typing through a stand-in relay, push-to-talk, offline
|
||||
```
|
||||
|
||||
## Gaze pointer (experimental)
|
||||
|
||||
In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the keyboard does the last bit. It needs the gaze service, which `install.sh` offers (yes by default) and `gaze/run.sh install` installs on its own: it builds it and runs `gaze/ft-gazed` as `frametop-gaze.service`, which starts with SteamVR. The service idles while the gaze isn't used: its eye tracker reader and our own eye tracker run only while gaze mode is on and someone wears the headset, while a check or the calibration runs, or while the Gaze page of Frametop Input Settings is open, and stop 30 seconds after ([gaze/README.md](../gaze/README.md)). Turn gaze mode on with the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, `POINTER_GAZE=1`, or a button or key combination mapped to Gaze pointer on/off.
|
||||
@@ -316,12 +273,12 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke
|
||||
|
||||
Deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it. It still builds and runs, installed by hand, for working on it.
|
||||
|
||||
Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen (a round hole per hand, `HANDS_MATTE=circle`; `mesh` and `capsules` are the older shapes). ft-handview (`frametop-handview.service`) draws your hands as a 3D hand model posed by the tracking: over games and the void with Room View off, and over the hole with it on (`HANDS_VIEW_ROOMVIEW=1`, the default). `HANDS_VIEW_MODEL` picks the model: `asset` (the default), `babylon`, `mrtk3`, `godot`, `mrtk2`, `robot` (a robot hand built in code from the tracked skeleton: light shells, dark joints, cyan accents), or `capsules` for the older tubes; `HANDS_VIEW_STYLE=ghost|solid` the look. The models, in `screens/assets/hands` with their licences: the WebXR Input Profiles' generic hand (`asset`; MIT, Copyright (c) 2019 Amazon), Babylon.js's hand meshes by Patrick Ryan (`babylon`; CC BY 4.0, https://github.com/BabylonJS/Assets), MRTK3's OpenXR hand (`mrtk3`; BSD-3-Clause, Copyright (c) 2025, Mixed Reality Toolkit Contributors), Godot XR Tools' hands by DigitalN8m4r3 (`godot`; CC0 1.0) and MRTK2's rigged hand (`mrtk2`; MIT, Copyright (c) Microsoft Corporation). The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode your eyes aim and your hands click, Vision Pro style: a pinch clicks where you were looking as it began, and the hand ray stays off (`POINTER_GAZE_HAND_RAY=1` brings it back beside the gaze). Pinch, hold and move the hand a few centimetres to nudge the pointer slowly onto the target, then let go to click there; the correction teaches the gaze tracker, as the mouse's do. Pinch and hold still for half a second to press and drag with the hand. A middle pinch right-clicks the same way. Hold a pinch with one hand and pinch with the other to hold both buttons, left and right, each until its pinch opens. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`. With `POINTER_HANDS=1` and gaze mode off, a bare hand raised and pointing also shows a ray that drives the pointer like a controller's laser (a pinch clicks, a middle pinch right-clicks or scrolls, reaching out and in pushes and pulls a carried panel), a controller used recently takes over its own hand only, and during a paused VR game hand tracking is parked rather than stopped, waking when the dashboard opens. Details: [hands/README.md](../hands/README.md), "Pinches and grips in the pointer". When tracking goes wrong, `ft-handsctl flag [note]` (or a voice command through frame-voice, "flag that") saves the last 12 seconds of hand crops and what the tracker made of them for the offline tools: ft-hands keeps them in RAM (`HANDS_FLAG_BUFFER=1`, about 50 MB), the drawn hands pulse cyan when a flag is taken (`HANDS_FLAG_FLASH`, `HANDS_FLAG_TONE`), and flags go to `HANDS_FLAG_DIR`. See [hands/README.md](../hands/README.md), "Flags".
|
||||
Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode a pinch clicks where you look when it opens; hold it and move the hand to correct the pointer first. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`.
|
||||
|
||||
- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. The Hand Recorder's installer (`hands/rec/install.sh`) sets them the same way. `hands/run.sh uncaps` takes them back while neither the services nor the Hand Recorder is installed. `hands/run.sh uninstall` and `hands/rec/install.sh uninstall` run it after removing their own part, and `uninstall.sh` always takes them back.
|
||||
- `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring.
|
||||
- `HANDS_MODE` says when they run: `off` (the default) only after `ft-handsctl on`, until `ft-handsctl off` or SteamVR stops; `frametop` while the Frametop desktop runs; `always` whenever SteamVR runs, in games too. Frametop Input Settings' Hands page sets it and applies it at once (`hands/handsmode.py`), next to the hand cutouts' margin (`HANDS_MATTE_MARGIN`, live). The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live.
|
||||
- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (`auto`, the default: ft-hands tells from the hands when some SteamVR restart has swapped the side cameras' names, and fixes them; `0` or `1` force them, and `hands/tools/check_sides.py --ring` tells which is right), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, typing's `POINTER_PINCH_TYPING`, `POINTER_TYPING_RAISE`, `POINTER_TYPING_BELOW`, `POINTER_TYPING_POSTURE`, `POINTER_TYPING_RECENT`, the gaze pinch's `POINTER_GAZE_PINCH`, `POINTER_GAZE_HAND_RAY`, `POINTER_GAZE_PINCH_DEADZONE`, and `POINTER_GAZE_PINCH_GAIN`, the hand ray's `POINTER_HAND_RAY`, `POINTER_RAY_DEADZONE`, `POINTER_RAY_PUSH`, `POINTER_RAY_SCROLL`, `POINTER_RAY_RELEASE`, `POINTER_RAY_GRACE`, its smoothing and taps (`POINTER_RAY_SMOOTH_MINCUTOFF`, `POINTER_RAY_SMOOTH_BETA`, `POINTER_RAY_SMOOTH_HOLD`, `POINTER_RAY_STEADY`, `POINTER_RAY_TAP`, `POINTER_RAY_TAP_DEADZONE`), its body and hand models (`POINTER_RAY_TORSO_WRISTS`, `POINTER_RAY_HAND_WEIGHT`, `POINTER_RAY_HAND_PITCH`, `POINTER_RAY_HAND_YAW`, `POINTER_RAY_ANCHOR`), and ft-screens' near reach (`HANDS_NEAR_REACH`, 0.075 m), pokes (`HANDS_POKE_ANGLE`, `HANDS_POKE_TRAVEL`, `HANDS_POKE_REST_BELOW`, `HANDS_POKE_REST_NEAR`) and near grabs (the border band `HANDS_GRAB_BAND`, a pinch on the content clicking `HANDS_PINCH_CLICK`, the card pinch `HANDS_CARD_PINCH`, the fist grab `HANDS_FIST_GRAB` and its reach `HANDS_FIST_REACH`, a second hand joining a held panel `HANDS_JOIN_REACH`, two hands turning a panel on every axis `HANDS_STRETCH_TURN`, the round buttons' hit area for hands `HANDS_BUTTON_REACH` and pinches pressing them `HANDS_PINCH_BUTTONS`, `HANDS_GRAB_SMOOTH_MINCUTOFF`, `HANDS_GRAB_SMOOTH_BETA`, `HANDS_GRAB_DEADZONE`, the wrist turning a carried panel: `HANDS_GRAB_ROTATE`, `HANDS_GRAB_ROTATE_DEADZONE`, holding through tracking loss `HANDS_HOLD_LOST_MS`, ignoring tracking no hand makes `HANDS_TRACK_GATE`), throws (the flick `HANDS_FLICK`, `HANDS_FLICK_SPEED`, `HANDS_FLICK_OPENING`, a carry let go fast `HANDS_THROW`, `HANDS_THROW_SPEED`, the push `HANDS_PUSH`, `HANDS_PUSH_SPEED`, where they go, `HANDS_THROW_DISTANCE`, and at what width, `HANDS_SEND_AWAY_WIDTH`), its "grab that" gesture (`HANDS_GRAB_THAT`, `HANDS_GRAB_THAT_DISTANCE`, `HANDS_GRAB_THAT_WIDTH`, which panels: `HANDS_GRAB_THAT_FROM`, `HANDS_GRAB_THAT_MARGIN`, `HANDS_GRAB_THAT_MARGIN_DEG`, `HANDS_GRAB_THAT_FACING`), the soft white halo round a gesture's target panel (`HANDS_TARGET_GLOW`), and the soft glow round the corner, edge or button a hand is at (`HANDS_EDGE_GLOW`). The example config explains each.
|
||||
- The install leaves both off, and they don't start with SteamVR. `ft-handsctl on` starts them while SteamVR runs, and `ft-handsctl off` stops them; they also stop with SteamVR. The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live.
|
||||
- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (`auto`, the default: ft-hands tells from the hands when some SteamVR restart has swapped the side cameras' names, and fixes them; `0` or `1` force them, and `hands/tools/check_sides.py --ring` tells which is right), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, and `POINTER_PINCH_TYPING`. The example config explains each.
|
||||
|
||||
Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md).
|
||||
|
||||
|
||||
+18
-33
@@ -1,6 +1,6 @@
|
||||
# Remote displays: design notes
|
||||
# Remote displays (exploration)
|
||||
|
||||
These are the notes from building remote displays, from the first exploration (2026-10-06) on: what the Frame's hardware gives us, the spikes, and the decisions. To set a computer up, see [Remote displays in the README](../README.md#remote-displays) and the host kits: [Windows](../host/windows/README.md), [macOS](../host/macos/README.md), [Linux](../host/linux/README.md). How the pieces work now is in [reference.md](reference.md#frametop-remote-displays).
|
||||
Status: exploration. Spike S1 is done; nothing in Frametop has changed yet. Branch `remote-displays`, written 2026-10-06.
|
||||
|
||||
The idea: show desktops streamed from other machines as Frametop panels that behave like the native screens. They get placement, curve, pinning, layouts, lasers, gaze, and attention-based frame rates. The target is up to 5 remote displays from 2 hosts, with no more headset overhead than 5 native screens.
|
||||
|
||||
@@ -33,7 +33,7 @@ Five 1440p displays fit at full rate with room left. Five 4K displays only fit a
|
||||
|
||||
### The planned setup
|
||||
|
||||
Two hosts that share the same two desk monitors: a Mac (14-inch MacBook Pro) with those two plus its built-in display, and the test PC (Windows, RTX 5090) with the two. Windows and Linux hosts run Vibepollo, which streams real displays and creates virtual ones on demand. The Mac runs Sunshine, which began with one display; it now runs one instance per display (see "Host side"). At 60 fps:
|
||||
Two hosts that share the same two desk monitors: a Mac (14-inch MacBook Pro) with those two plus its built-in display, and the test PC (Windows, RTX 5090) with the two. Windows and Linux hosts run Vibepollo, which streams real displays and creates virtual ones on demand. The Mac runs Sunshine and is limited to one display for now (see "Host side"). At 60 fps:
|
||||
|
||||
| Display | Share of the decoder |
|
||||
|---|---|
|
||||
@@ -175,7 +175,7 @@ The requirement: each host streams its real displays, and virtual displays it cr
|
||||
|---|---|---|
|
||||
| Windows | Vibepollo | 1 main session (real or virtual) + up to 4 virtual Remote Monitors |
|
||||
| Linux (Arch, CachyOS; beta) | Vibepollo | The same, with at most 4 virtual displays at once |
|
||||
| macOS | Sunshine | Every real display, one Sunshine instance each (`host/macos`); no virtual displays |
|
||||
| macOS | Sunshine | 1, the Mac's main display, for now |
|
||||
|
||||
### Windows and Linux: Vibepollo
|
||||
|
||||
@@ -210,15 +210,19 @@ The config folder is admin-only, so the change is a script for the user to run (
|
||||
|
||||
**Risk.** Vibepollo is new, moves fast, and has one maintainer, and its README says about 99% of its code is AI-generated. Frametop depends only on its Moonlight-protocol behaviour and the Remote Monitor app ids, so on Windows Apollo (virtual displays through the SudoVDA driver) or plain Sunshine (real displays) stay as fallbacks.
|
||||
|
||||
### macOS: Sunshine, one instance per display
|
||||
### macOS: Sunshine, one display for now
|
||||
|
||||
Vibepollo has no macOS build. Sunshine (labelled experimental on macOS) is the only Moonlight-protocol host for the Mac. One Sunshine streams one display, so the Mac runs one instance per real display, each with its own config file, ports (`port` 47989 + 100·N) and `output_name` (the display's CGDirectDisplayID). `host/macos/frametop-host-setup.sh` sets them up (see "The Mac" below). Sunshine can't create virtual displays on macOS, and no PR is open for it.
|
||||
Vibepollo has no macOS build. Sunshine (v2026.914, labelled experimental on macOS) is the only Moonlight-protocol host for the Mac, so the Mac streams one display: its main display, from one Sunshine instance. The reasons:
|
||||
|
||||
- Absolute mouse input reached only the main display from v2026.906 (Sunshine #5733). libvirtualhid #145 and Sunshine #5739 fixed it on 2026-10-09, first in the pre-release v2026.1010.141104, which the host setup pins until a release has it.
|
||||
- Capture is at backing pixel size, so the built-in display streams at 3024x1964. Closing the lid removes the built-in display, so a stream of it ends. Sunshine now keeps a display awake while it captures it (#5509 was the stall when one slept).
|
||||
- Sound comes from a Core Audio tap (macOS 14+), so nothing like BlackHole is needed. Every instance carries it; ft-stream's per-address lock plays one.
|
||||
- Since v2026.906, absolute mouse input only reaches the main display (Sunshine #5733). Any other display would show but couldn't be clicked.
|
||||
- Nobody reports running several Sunshine instances on one Mac yet.
|
||||
- Sunshine can't create virtual displays on macOS.
|
||||
|
||||
Full-motion video can still reach the encoder's limit. Sunshine forces VideoToolbox's low-latency mode, which roughly halves throughput (#5814); an M4 Pro managed 0.36-0.46 Gpix/s in that mode. The super ultrawide at 60 fps is 0.44 Gpix/s; the test Mac is an M3 Max, with two encode engines. Open PR #5817 (ScreenCaptureKit + OBS's VideoToolbox encoder) may raise the limit. Its capture (AVFoundation) also sends 30-55 fps of a still desktop, not `minimum_fps_target`'s 1 fps, at a low bitrate (1-6 Mbit/s).
|
||||
The main display is the one with the menu bar (System Settings → Displays). Capture is at backing pixel size, so the built-in display streams at 3024x1964. Closing the lid removes the built-in display, so a stream of it ends. Capture stalls if the display sleeps mid-stream (#5509).
|
||||
|
||||
One display can still reach the encoder's limit. Sunshine forces VideoToolbox's low-latency mode, which roughly halves throughput (#5814); an M4 Pro managed 0.36-0.46 Gpix/s in that mode. The super ultrawide at 60 fps is 0.44 Gpix/s, so full-motion video on it may need a lower fps or a smaller stream. The other two displays need 0.30 and 0.36 Gpix/s.
|
||||
|
||||
**What would lift the limit.** The mouse fix is libvirtualhid PR #145 ("target configured mouse viewport", open), which Sunshine's draft PR #5739 pulls in. Once it ships, spike S4 tries one Sunshine instance per display (own config file, `port` about 100 apart, `output_name`) and BetterDisplay virtual displays switched on by `global_prep_cmd`. Open PR #5817 (ScreenCaptureKit + OBS's VideoToolbox encoder) may raise the encoder limit. Helping #145 along upstream is the one Mac contribution worth making.
|
||||
|
||||
### What stock hosts cost us
|
||||
|
||||
@@ -228,8 +232,8 @@ Full-motion video can still reach the encoder's limit. Sunshine forces VideoTool
|
||||
| One stream per display | Five ft-stream processes, connections, and client pairings on the Frame | Measure the CPU (S3) |
|
||||
| One real display per Vibepollo install | A second real display needs a second host instance | Use virtual displays; S3 tries a second instance on Windows |
|
||||
| Cursor drawn into the video | Lags the laser by a round trip; each mouse move over a still page becomes a video frame | Accept |
|
||||
| Mac: an instance per display | Each display pairs on its own, with the Web UI login (no tokens); no virtual displays | Remote Displays pairs all the picked displays from one sign-in |
|
||||
| Mac encoder | Full-motion video on the super ultrawide at 60 fps is at the limit; a still desktop still streams 30-55 fps | Lower fps or size for that stream; ScreenCaptureKit (#5817) upstream |
|
||||
| Mac: one display | Sunshine's mouse reaches only the main display, and there are no virtual displays | Upstream fix in progress, then S4 |
|
||||
| Mac encoder | Full-motion video on the super ultrawide at 60 fps is at the limit | Lower fps or size for that stream |
|
||||
|
||||
What Frametop maintains: ft-stream (pairing, decoder, keyframe sampling, input mapping, Remote Monitor launch and resume; the protocol itself is moonlight-common-c's), the remote screen type in ft-screens, the layout and settings changes, and host setup notes or scripts.
|
||||
|
||||
@@ -447,30 +451,11 @@ The user decided remote displays are Frametop displays, with the same controls a
|
||||
- After Vibepollo restarted, a stream checked the dongle in 300 ms, too soon, and went over the network; ft-stream now gives it a second.
|
||||
|
||||
**Frametop's build released (2026-10-09):**
|
||||
- Release [`frametop-2.0.0-2`](https://github.com/Frametop/frametop-vibepollo/releases/tag/frametop-2.0.0-2) of Frametop/frametop-vibepollo (2026-10-14; `frametop-2.0.0-1` was the first) has `sunshine.exe` and the Arch package (`pacman -U`), each with its `.sha256`. The fork's CI (`frametop-build.yml`, Depot's runners) builds them from the tag and publishes them; a `frametop-*` tag is what makes a release. The tag is the source.
|
||||
- Release [`frametop-2.0.0-1`](https://github.com/Frametop/frametop-vibepollo/releases/tag/frametop-2.0.0-1) of Frametop/frametop-vibepollo has `sunshine.exe` and the Arch package (`pacman -U`), each with its `.sha256`. The fork's CI (`frametop-build.yml`, Depot's runners) builds them from the tag and publishes them; a `frametop-*` tag is what makes a release. The tag is the source.
|
||||
- The host setup's `$BuildUrl` points at that `sunshine.exe`, so a PC needs only `host/windows`. It replaces the original, the hand-built `2f032252`, and the first CI build (`a84b6cfc`).
|
||||
- New in it: a Frametop display stream that asks for SDR turns the display's HDR off while it captures it, and back on when the stream ends (with `dd_hdr_option` automatic, Vibepollo's default). The test PC's HDR monitor looked washed out in SDR: Vibepollo's conversion clips at 80 nits while Windows draws SDR content at its SDR white level (240 nits there), and scaling for that left the colours heavily oversaturated. Displays it turned off are listed in `config\frametop_display_hdr.json` until they're back on, so the next start turns them back on after a crash or a forced stop. On the test PC's dev build: HDR off 0.4 s after the stream started, 8-bit capture, back on at the end, off again on a reconnect.
|
||||
- Tested on the test PC: with the first CI build installed and only `host/windows` in a folder, `-Check` downloaded the release and matched its SHA-256, and the run backed up, swapped and restarted Vibepollo; the Web UI answered.
|
||||
|
||||
**The monitor follows the stream's resolution (frametop-vibepollo, 2026-10-10):**
|
||||
- Before, a Frametop display stream captured the monitor at its own resolution and Vibepollo fit that picture into the stream's size, so a stream smaller than the monitor, or of another shape, never showed its desktop pixel for pixel.
|
||||
- Now the stream switches the monitor to the stream's width and height, at the monitor's own refresh rate, before capture starts, and puts it back when the stream ends, the way `dd_resolution_option` automatic (Vibepollo's default) matches a main session's display to the client. With it set to manual or disabled, nothing changes. A monitor without that mode gets its closest mode with the same shape, and the stream scales that; one with no mode of that shape keeps its own and is scaled as before.
|
||||
- The change is temporary: Windows' saved display settings stay as they were, so a reboot undoes it too. Windows may move the other monitors to keep them touching one that changed size, so every monitor's place is noted before the first change and put back after the last, in one change that either fully applies or doesn't.
|
||||
- Two streams of the same monitor share the first one's mode. HDR (above) goes off after the mode changes and back on before it's put back.
|
||||
- Monitors it changed are listed in `config\frametop_display_mode.json` (each one's own mode and the one it set, and the places) until they're back, so the next start puts them back after a crash or a forced stop. One someone has changed since keeps theirs.
|
||||
- In Frametop nothing changes: ft-stream already asks for the size set in Remote Displays, and changing it starts the stream over, which switches the monitor again. The note under a monitor's stream settings says what happens.
|
||||
|
||||
**The Mac: one Sunshine instance per display (2026-10-10, branch `mac-host`):**
|
||||
- `host/macos/frametop-host-setup.sh` (or `Setup Frametop host.command`): installs the pinned Sunshine pre-release (SHA-256 checked, notarized by LizardByte), lists the displays (`displays.js`, JavaScript for Automation, no compiler), and writes an instance per display in `~/Library/Application Support/Frametop/sunshine/instance-N`, the main display's first, each started at login by the LaunchAgent `frametop.sunshine.N`. Its settings: `port`, `sunshine_name` "COMPUTER: DISPLAY (WxH)", its own state and apps files, `minimum_fps_target = 1`, and the menu bar icon on the first only. The rest of each `sunshine.conf` (Web UI changes) is kept on a rerun. `--check`, `--undo` (removes the LaunchAgents; settings, pairings and Sunshine stay), `--set-login`, `--skip-login`.
|
||||
- Turning them off and on (2026-10-12): Frametop Streams, a menu bar icon, lists the displays with a check by each one that's on. A click turns one on or off, Turn All On and Turn All Off do every one, and the icon dims while they're all off. It's `Frametop Streams.applescript`, a stay-open AppleScriptObjC app (`NSStatusItem`, so no compiler) that the setup compiles into `~/Applications`, marks `LSUIElement` (no Dock icon), signs again ad hoc after that change, and starts at login with the LaunchAgent `frametop.streams`. Quit from its menu, it stays quit until the next login. It runs `ft-sunshine off|on|status [--tsv] [N...]`, in the background for a change, with the output in `~/Library/Logs/Frametop/streams.log`. `ft-sunshine` is installed next to `ft-sunshine-instance` and linked into `~/.local/bin` when there is one. Off disables each instance's LaunchAgent (`launchctl disable`) and stops it, so it stays off through logins. On enables it and starts it, or starts one quit from Sunshine's own menu bar icon, which launchd leaves stopped after a clean exit. Rerunning the setup turns them all on. The Frame's streams of the Mac keep trying while it's off and reconnect after on. Tested with throwaway LaunchAgents serving fake instances, through the menu (UI scripting) and `ft-sunshine`: on, on again (the running one kept its pid), off, on after off, one display, a stopped one, unknown numbers, a display that isn't connected, and the app started by its LaunchAgent and quit from its menu. On the M3 Max: the setup from the kit zip, unpacked with a download's quarantine (the installed scripts keep it, and run anyway: launchd and the app run them through bash and sh), which turned the three on; then Turn All Off, Turn All On and Turn All Off again from the menu, each done within 3 s.
|
||||
- The kit as a release (2026-10-12): [`macos-host-1`](https://github.com/Frametop/frametop/releases/tag/macos-host-1) of Frametop/frametop has `frametop-macos-host-1.zip`, `host/macos` of 188f107 in a `frametop-macos-host` folder (`zip -qr -X`), and its `.sha256`. Its notes are `host/macos/README.md` without the title, plus the files. It isn't marked Latest and has no `Frametop.zip`, so get.sh skips it: stable takes `releases/latest`, experimental the newest release with a `Frametop.zip`. Its tag doesn't start with `v`, so `release.yml` doesn't build it. A new kit is a new release, `macos-host-N`, with the zip renamed to match.
|
||||
- An instance is tied to its monitor by vendor, model and serial, not to a display id: `ft-sunshine-instance` looks the id up at every start and passes `output_name=ID`. While the monitor is unplugged it doesn't start Sunshine, which would fall back to the main display, and launchd tries again.
|
||||
- One certificate for all the instances, so Remote Displays pins one key per Mac; one Web UI login file for all, kept from an earlier Sunshine install if there was one. An earlier Homebrew Sunshine's LaunchAgent (the same ports) is turned off, and back on with `--undo`.
|
||||
- Screen Recording and Accessibility are granted by hand, once for all the instances: macOS lists them as Sunshine (the LaunchAgent runs a bash script that execs it; `AssociatedBundleIdentifiers` makes Login Items say Sunshine too). Sunshine started from a terminal gets that terminal's grants instead.
|
||||
- Without Accessibility (Post Event access) the pointer still moves, since libvirtualhid also warps it (`CGWarpMouseCursorPosition`, no permission needed), but clicks, scrolling and keys go only through `CGEventPost`, which macOS drops without a word: every Frame input method moved the pointer and clicked nothing, while the Mac's own trackpad worked. Sunshine notes at each start whether it had each permission (`defaults read dev.lizardbyte.app.Sunshine inputPostEventPermissionState`, 2 = granted, 1 = asked and not granted) and asks only once, so a grant macOS lost track of, or one given to an older Homebrew Sunshine (a separate app to macOS), stays broken with nothing asking again. The setup reports both permissions at the end and with `--check`. After the first instance's restart, if Accessibility is still missing, it offers to clear Sunshine's records (`tccutil reset Accessibility` and `PostEvent` for its bundle id, and that note) and restart the instance so macOS asks again, which fixed the test Mac (2026-10-10).
|
||||
- Frametop side: a display has a `port` when its instance isn't on 47989, and ft-stream takes `HOST:PORT`, so ft-screens passes it through as it is. The token, pin and audio lock stay per address. Given HOST alone, ft-stream takes its display's port from the layout, by its client: an ft-layout from before ports (an older release's Reset Screen Layout and profile launchers) restarted the Mac's streams at HOST, the instance on 47989, where they froze unpaired and their pointer went to the main display. `host_answers` checks the displays' ports. Remote Displays tries `/api/token` at sign-in; a bare 400 "Bad Request" (upstream Sunshine's answer to a path it doesn't have, before any login check) or a 404 means plain Sunshine, and the login is checked on `/api/clients/list`. It's kept in memory only, passed on stdin to `ft-layout remote add --port P --login` and on to `ft-stream pair --login` for each display picked, then forgotten. Upstream Sunshine (v2026.1010) lists a waiting pairing request with an id (`GET /api/pin`); ft-stream approves its own (libgamestream's device name "roth", from its own address) with `POST /api/pin {pairing_id, pin, name}`. The login is checked first: a wrong password's request, left waiting, blocked the next try. There are no permissions to grant. Add computer merges the instances' mDNS entries ("Mac (2)") into one computer, and Add displays lists the instances from their serverinfo on ports 47989 + 100·N.
|
||||
- Tested on the M3 Max (macOS 26.6.2: built-in 3024x1964, MPG491CX OLED 5120x1440, LC34G55T 3440x1440): the setup with `--check`, a first run and a rerun; the three instances answer from the Frame. With throwaway test instances (their own login, removed after): pairing with a wrong password then the right one; `ft-layout remote add --port --login` into a scratch layout; streams through a stand-in for ft-screens, the three at once (no losses; 35, 52 and 36 fps of still desktops, 3-9% of a Frame core each, about 6% CPU for Sunshine on the Mac for the OLED); the host's sound; a pointer move to two corners of the LC34G55T, not the main display, landing within a pixel of where it should. Not tested: the Remote Displays app itself, the live desktop, drags across the Mac's displays (separate processes; the release goes through the stream the press went down on), full-motion video's encode rate, replugging a monitor.
|
||||
|
||||
**Tested in the live desktop (frame-testbench, the 3D mouse through `@ft_pointer_helper`, 2026-10-07):**
|
||||
- An Explorer window carried by its title bar from the Remote Monitor to the OLED stayed there, and carried back, stayed there too. The log showed one change of screen each way.
|
||||
- Disconnect, connect: the panel came back where it was, at its width. Connect in a new run with the headset off: placed from screen 1's anchor.
|
||||
@@ -496,6 +481,6 @@ The user decided remote displays are Frametop displays, with the same controls a
|
||||
## Open questions
|
||||
|
||||
- Which of the test PC's two desk monitors is the real one (the main session)?
|
||||
- Mac virtual displays: BetterDisplay switched on by `global_prep_cmd`, or a small helper on the private `CGVirtualDisplay` API. Not needed for v1.
|
||||
- The Mac's chip (Max chips have two video encode engines) and BetterDisplay Pro matter only once the Mac goes past one display.
|
||||
- Audio: none, the focused display's host, or a fixed one?
|
||||
- Remote displays outside the desktop: should they also show over games, where the decoder is shared with Steam Link VR?
|
||||
@@ -1,76 +0,0 @@
|
||||
# Spotwatch
|
||||
|
||||
Spotwatch is Frametop's search for apps and remote displays, like Spotlight on a Mac. Press Meta+Space and a glass search pill opens straight ahead of where your head points. Type part of an app's name, or a remote display's `{host} - {display}`, and pick from what it finds:
|
||||
|
||||
- **An app that's already running:** its most recent window comes to you. The lazy susan spins the panel it's on, a screen or its own floating window, to straight ahead, and typing goes to it.
|
||||
- **An app that isn't running:** it starts as a floating window, where that app last floated or in front of you.
|
||||
- **A remote display:** spotwatch connects it if needed, then spins it to the front.
|
||||
|
||||
Use Tab, Shift+Tab, or the arrow keys to move through the results, and Enter or a click to open one. Shift+Enter or a right click opens the item's menu:
|
||||
|
||||
- **For an app:** its own launch options, such as a browser's New Private Window, plus Open new window (even when one is open), Open docked on this screen, and each of its open windows.
|
||||
- **For a remote display:** Go to (connecting it first if needed) and Disconnect.
|
||||
|
||||
Esc goes back from a menu and closes the search. Meta+Space again, or a click anywhere else, closes it too.
|
||||
|
||||
Spotwatch works while the Frametop desktop is running and Frametop isn't paused for a VR game. It's the `spotwatch` action in Frametop Input Settings, so a mouse button, a controller button, or another key combination can open it too. When it opens from a mouse or controller, Frametop's keyboard opens with it.
|
||||
|
||||
## Not built
|
||||
|
||||
- Searching open windows by title, Frametop's own actions ("Reset layout"), and profiles (decision 4).
|
||||
- Moving the caret within the query. Typing and Backspace work at the end only.
|
||||
- A setting for its distance and size (`--distance` and `--width` on ft-spotpanel; 1 m and 0.56 m).
|
||||
|
||||
## Decisions
|
||||
|
||||
These were settled with the user on 2026-10-10. The numbers may be cited in the code, so they stay as they are.
|
||||
|
||||
| # | Question | Decision |
|
||||
|---|---|---|
|
||||
| 1 | The look | A rounded glass pill with a results card under it. The glass is faux, because SteamVR's overlays have alpha but no backdrop blur: a translucent dark tint, a soft sheen from the top, a bright rim fading down the edges, and a shadow outside. Real blur would mean reading the panels back every frame on the Frame's GPU |
|
||||
| 2 | Where it opens | Fixed in the room, centred where the head points (not the eye gaze), at 1 m, upright, tilted no more than 60° up or down. The card hangs below the pill, so the pill stays put as results change |
|
||||
| 3 | An app with several windows | One row per app. Enter goes to its most recently active window, and Shift+Enter or a right click lists the others with the menu items |
|
||||
| 4 | What it searches | Apps and remote displays only. Hidden remote displays are left out, and disconnected ones show with Connect |
|
||||
| 5 | Disconnected remote displays | Enter connects the display (`ft-layout remote connect`), waits for its panel, then spins to it |
|
||||
| 6 | When it works | Only while Frametop is running and active: the session starts it, and the input relay doesn't run the action while Frametop is paused. A pause while it's open closes it |
|
||||
| 7 | Without a keyboard | `spotwatch` is an action like the others, so it can be bound to a mouse button or a controller button. Opened that way, Frametop's keyboard opens too and closes with it |
|
||||
| 8 | The menu | The app's desktop actions, Open new window (the app's own new-window action when it has one, else a plain launch), Open docked on this screen (the screen nearest straight ahead), and its windows |
|
||||
| 9 | An app that isn't running | It floats, where ft-floatd puts any launched app (where it last floated, or in front of you) |
|
||||
| 10 | A window on a pinned panel | Pinned panels don't spin. Typing goes to it where it is (`front N`), and a short note says where it is ("on your left wrist") |
|
||||
| 11 | Shortcut | Meta+Space by default. Plasma's KRunner has Alt+Space, and nothing in Frametop used Meta+Space |
|
||||
| 12 | Branch | `spotwatch`, off `experimental`, merged into `experimental` |
|
||||
|
||||
## How it works
|
||||
|
||||
Spotwatch has three parts, and needs changes in ft-screens, ft-floatd and the input relay:
|
||||
|
||||
- **`spotwatch/ft-spotwatch`** (`ft_spotwatch.py`) runs inside the desktop's Plasma session, started from its autostart like ft-floatd, so it's up exactly while the desktop is. It keeps the app index (Gio's, so it sees what the desktop's menus see), the remote displays from the layout, the usage counts, and the search state. It decides what each key and click does. Its log is `/tmp/frametop-spotwatch.log`.
|
||||
- **`spotwatch/ft-spotpanel.cpp`** is the pill: a SteamVR overlay drawn on the CPU, like ft-gazepanel, built by `screens/build.sh` and run by ft-spotwatch in the container. It holds no state beyond the picture. ft-spotwatch sends it the whole state each time, and it sends back hovers, clicks, right clicks, scrolls, and clicks beside the card. Its clear area catches those clicks.
|
||||
- **`spotwatch/spotsearch.py`** ranks the results with plain Python, so its test runs anywhere. Name matches come first, from the strongest kind of match to the weakest: the name's start, a word's start, initials, anywhere in the name, then letters in order. Generic names, keywords, and program names count for less and never match fuzzily. Every word of the query must match. Use counts and use in the last day raise an item, and so does an app that's running.
|
||||
|
||||
The search has to take the typing from wherever it went, Steam included. While it's open, ft-screens sends every key to `@ft_spotwatch` instead of KWin or a remote display's stream, from keyboards and Frametop's keyboard alike, and tells the input relay that typing goes to the desktop, so the relay grabs the keyboards. ft-spotwatch holds this with `spotwatch on`, renewed every second. ft-screens lets it lapse after 3 seconds, so a crashed ft-spotwatch can't keep the keys. A click on any other panel, or a pause, ends it, and ft-screens says `dismiss` so the pill closes.
|
||||
|
||||
ft-spotwatch turns key codes into text on the desktop's keyboard layout (Plasma's `kxkbrc`) through libxkbcommon, and falls back to a US layout without it. The input relay sends no key repeats, so ft-spotwatch repeats a held Backspace or arrow key itself.
|
||||
|
||||
Bringing a window to the front takes three commands:
|
||||
|
||||
1. ft-floatd's `activate ID` makes the window KWin's active window, unminimized.
|
||||
2. ft-screens' `spin to N` brings panel N to the front slot at reading distance (or, with `SCREEN_SPIN_BRING=0`, turns the ring of panels the short way so it ends straight ahead). Screens are 1 to S, floating windows come after them, and remote displays start at 101.
|
||||
3. When the spin settles, ft-screens gives that panel the typing, as any spin does. When the panel can't spin (pinned, or a floating window that's minimized), `front N` gives it the typing where it is.
|
||||
|
||||
ft-floatd's `spotlist` lists every app window with its desktop file name, title, panel, and when it was last active. The KWin script keeps that timestamp.
|
||||
|
||||
Launching uses ft-floatd's `launch APP.desktop`, which now takes `action=ACTION` (one of the desktop file's actions) and `screen=N` (open docked on screen N; the KWin script's `to-output`). Without ft-floatd (`FLOAT_SLOTS=0`), spotwatch launches through Gio itself, and nothing counts as running.
|
||||
|
||||
Apps' icons come from the desktop's icon theme (`kdeglobals`, then breeze and hicolor). GdkPixbuf rasterizes them to 64x64 RGBA files in `~/.cache/frametop/spotwatch` for the panel, one at a time in idle moments after the desktop starts. Use counts are kept in `~/.config/frametop-spotwatch.json`.
|
||||
|
||||
```
|
||||
spotwatch/test/spotwatch-test.py # the ranking, then ft-spotwatch against a fake ft-screens and ft-floatd (offline)
|
||||
spotwatch/tools/render.py OUT.png # what the panel draws, without a headset (ft-spotpanel --render)
|
||||
```
|
||||
|
||||
## Risks
|
||||
|
||||
- **Stray key releases.** A key pressed before the search opened is released to wherever it went, so nothing stays held. A key pressed in the search and released after it closes (the Enter that opened something) reaches the window that now has the typing as a release without a press. Apps ignore those.
|
||||
- **Laser input.** While the pill shows, it makes SteamVR's lasers active (`MakeOverlaysInteractiveIfVisible`), as Frametop's keyboard does. The search is short-lived, but during that time a game loses its controllers.
|
||||
- **Matching windows to apps.** Matching uses KWin's desktop file name, then the window class. An app whose windows report neither, such as some X11 apps under Flatpak, doesn't show as running, and Enter starts it again.
|
||||
+1
-11
@@ -17,7 +17,6 @@ const settled = {}; // id -> {output, frame, fullScreen, maximized}: where a
|
||||
const held = {}; // id -> {w, h, until, asked}: a size asked for, for a second (see hold)
|
||||
const moved = {}; // id -> true, or "back" once put back: it moved while KWin changed the outputs
|
||||
let layout = "", layoutOutputs = {}, layoutSince = 0; // the outputs at the last screensChanged
|
||||
const activeAt = {}; // id -> when it was last KWin's active window (ms), for spotwatch's most recent
|
||||
|
||||
function send(ev) {
|
||||
callDBus(SERVICE, PATH, IFACE, "Event", JSON.stringify(ev));
|
||||
@@ -53,7 +52,7 @@ function info(w) {
|
||||
frame: rect(w.frameGeometry), client: rect(w.clientGeometry), popup: w.popupWindow,
|
||||
transient: w.transient, parent: w.transientFor ? String(w.transientFor.internalId) : "",
|
||||
normal: w.normalWindow, dialog: w.dialog, fullScreen: w.fullScreen, minimized: w.minimized,
|
||||
onAllDesktops: w.onAllDesktops, maximized: isMaximized(w), activeAt: activeAt[String(w.internalId)] || 0
|
||||
onAllDesktops: w.onAllDesktops, maximized: isMaximized(w)
|
||||
};
|
||||
}
|
||||
|
||||
@@ -269,10 +268,8 @@ workspace.windowRemoved.connect(w => {
|
||||
delete settled[id];
|
||||
delete held[id];
|
||||
delete moved[id];
|
||||
delete activeAt[id];
|
||||
});
|
||||
workspace.windowActivated.connect(w => {
|
||||
if (w) activeAt[String(w.internalId)] = Date.now();
|
||||
if (w && isSpare(w.output)) send({ev: "activated", id: String(w.internalId)});
|
||||
});
|
||||
takeLayout();
|
||||
@@ -379,15 +376,8 @@ function run(c) {
|
||||
if (w) w.closeWindow();
|
||||
break;
|
||||
case "activate":
|
||||
if (w && w.minimized) w.minimized = false;
|
||||
if (w) workspace.activeWindow = w;
|
||||
break;
|
||||
case "to-output": { // a launched window that opens docked, onto that screen (spotwatch)
|
||||
const o = outputByName(c.output);
|
||||
if (w && o && (!w.output || w.output.name !== o.name)) workspace.sendClientToScreen(w, o);
|
||||
if (w) workspace.activeWindow = w;
|
||||
break;
|
||||
}
|
||||
case "activate-output": { // the top window on that output (a spin brought it to the front)
|
||||
const order = workspace.stackingOrder;
|
||||
for (let i = order.length - 1; i >= 0; --i) {
|
||||
|
||||
@@ -8,10 +8,7 @@
|
||||
ft-float dock all put every floating window back
|
||||
ft-float launch APP start an app (its desktop file name, e.g. org.kde.dolphin) and
|
||||
float its first window where that app last floated
|
||||
ft-float launch APP action=ACTION ...one of its desktop file's actions (a private window)
|
||||
ft-float launch APP screen=N ...its first window docked on screen N instead
|
||||
ft-float run COMMAND [ARG...] the same for a command
|
||||
ft-float activate ID make a window KWin's active one, unminimized
|
||||
ft-float close ID close a window
|
||||
ft-float list the spare outputs and what floats on them
|
||||
FT_FLOAT_SOCKET names ft-floatd's socket (default frametop_float).
|
||||
|
||||
+46
-152
@@ -12,12 +12,8 @@ which window floats on which spare output and panel, and connects three parts:
|
||||
Commands and ft-screens' events arrive as datagrams on @frametop_float (ft-float is the
|
||||
command-line side). Replies go to the sender:
|
||||
float [ID|active|pointer] dock [ID|active|all] close ID list quit (ft-float)
|
||||
launch APP.desktop [action=ACTION] [screen=N] run ["argv", ...]
|
||||
(start an app, or one of its desktop file's actions, and float its first window;
|
||||
with screen=N its first window opens docked on screen N instead)
|
||||
launch APP.desktop run ["argv", ...] (start an app and float its first window)
|
||||
windows (-> JSON: the open apps' windows and where they are, for a profile; ft-layout save)
|
||||
spotlist (-> JSON: every app window, its title, panel, and when it was last active; spotwatch)
|
||||
activate ID (make that window KWin's active one, unminimized; spotwatch)
|
||||
profile NAME (open a profile's apps: move the matching windows, launch the missing ones)
|
||||
("float pointer" is the float key: the window under the pointer, else the active one,
|
||||
floated or docked; the input relay sends it for float_toggle, and "dock all" for dock_all)
|
||||
@@ -28,15 +24,15 @@ command-line side). Replies go to the sender:
|
||||
Spare outputs are WL-<screens> .. WL-<screens + slots - 1>. A floating window's output is its
|
||||
frame plus a margin on each side (FLOAT_MARGIN pixels), so menus have room; the panel shows
|
||||
only the window. Spares are placed apart from the screens and from each other in KWin's
|
||||
layout, within Xwayland's 32767-pixel limit at scales up to 1.6.
|
||||
layout, all within Xwayland's 32767-pixel limit.
|
||||
|
||||
Usage: ft-floatd [--screens N] [--slots N] [--margin PX] [--control NAME] [--socket NAME]
|
||||
Defaults: FT_SCREEN_COUNT (from the session) or the layout's count, FLOAT_SLOTS and
|
||||
FLOAT_MARGIN from ~/.config/frametop.conf (8 and 300), @ft_screens, @frametop_float.
|
||||
|
||||
Launching floating: "launch" starts a desktop file's app (Gio), "run" a command. The next
|
||||
normal window from that process (or a child), or of that desktop file (desktop_name: by its app
|
||||
id, else its class, so Flatpak and X11 apps match too), within LAUNCH_SECONDS floats: where that app last floated, or in front of you. Single-instance and
|
||||
normal window from that process (or a child), or with that desktop file name, within
|
||||
LAUNCH_SECONDS floats: where that app last floated, or in front of you. Single-instance and
|
||||
D-Bus-activated apps open their window from a process that was already running, hence the
|
||||
name. Each app's last floating place (its pose relative to the primary screen, so it moves
|
||||
with the screens' layout), size in pixels, and scale are kept in PLACES_PATH, by desktop
|
||||
@@ -77,11 +73,7 @@ SERVICE = IFACE = "org.frametop.Float"
|
||||
PATH = "/Float"
|
||||
SCRIPT = "frametop-float"
|
||||
POLL_SECONDS = 20 # NextCommand answers empty after this (KWin's D-Bus timeout is 25 s)
|
||||
# Spares in KWin's layout: a grid 4 across, below the screens. X11 windows get Xwayland
|
||||
# coordinates of the layout's times its largest scale, which wrap past 32767: a window there
|
||||
# takes no clicks. 16 spares end at 20000, fine up to scale 1.6 (12000 across had the 4th
|
||||
# column past it at 1.25).
|
||||
SPARE_Y, SPARE_CELL = 4000, 4000
|
||||
SPARE_X, SPARE_CELL = 12000, 5000 # spares in KWin's layout: a grid from here, 4 across
|
||||
PULL_OUT = 0.3 # a floated window starts this far in front of its screen (metres), clear of it
|
||||
DEFAULT_MPP = 1.6 / 1920 # metres per pixel when ft-screens can't say (no SteamVR)
|
||||
PLACES_PATH = os.path.expanduser("~/.config/frametop-float.json") # each app's last floating place
|
||||
@@ -231,14 +223,6 @@ def recordable(ev):
|
||||
return app not in SKIP_APPS and not any(n.startswith(("ft-", "frametop", "ksplash")) for n in (app, cls))
|
||||
|
||||
|
||||
def listable(ev):
|
||||
"""An app's top-level window, the kind spotwatch finds: Frametop's own settings apps too,
|
||||
which profiles leave out (an open Input Settings launched a second copy, 2026-10-13)."""
|
||||
if not ev.get("normal") or ev.get("popup") or ev.get("transient"):
|
||||
return False
|
||||
return ev.get("app", "") not in SKIP_APPS and not any(n.startswith("ksplash") for n in (ev.get("app", ""), ev.get("cls", "")))
|
||||
|
||||
|
||||
def cmdline(pid):
|
||||
try:
|
||||
with open(f"/proc/{int(pid)}/cmdline", "rb") as f:
|
||||
@@ -247,56 +231,22 @@ def cmdline(pid):
|
||||
return []
|
||||
|
||||
|
||||
_names = {} # (app, cls) -> (desktop file name, when): desktop_name's answers for a minute
|
||||
|
||||
|
||||
def desktop_name(app, cls):
|
||||
now = time.monotonic()
|
||||
hit = _names.get((app, cls))
|
||||
if hit and now - hit[1] < 60:
|
||||
return hit[0]
|
||||
if len(_names) > 512:
|
||||
_names.clear()
|
||||
name = _desktop_name(app, cls)
|
||||
_names[(app, cls)] = (name, now)
|
||||
return name
|
||||
|
||||
|
||||
def _desktop_name(app, cls):
|
||||
"""The desktop file name (without .desktop) for a window's app id and class, or "" when no
|
||||
desktop file is its. Most windows give their desktop file name as the app id. The others:
|
||||
- A Flatpak app's window can give an id with no desktop file: RustDesk's (an X11 window)
|
||||
says com.carriez.flutter_hbb, and its desktop file is com.rustdesk.RustDesk.
|
||||
- An X11 window can give no app id at all, only its class: Brave's Flatpak says
|
||||
"Brave-browser" (2026-10-13), and its desktop file is com.brave.Browser.
|
||||
Then, by the class (or the id), in this order: a desktop file's StartupWMClass; a desktop
|
||||
file whose name, or its last dotted part, is that name; the program a desktop file runs
|
||||
(Flatpak's --command=, else the Exec's own), by its name."""
|
||||
"""The desktop file name for a window's app id and class. A Flatpak app's window can give an
|
||||
id with no desktop file: RustDesk's (an X11 window) says com.carriez.flutter_hbb, and its
|
||||
desktop file is com.rustdesk.RustDesk. That file's StartupWMClass names the window's class
|
||||
then. The app id itself when nothing matches."""
|
||||
try:
|
||||
if app and Gio.DesktopAppInfo.new(app + ".desktop"):
|
||||
if not app or Gio.DesktopAppInfo.new(app + ".desktop"):
|
||||
return app
|
||||
except TypeError: # PyGObject raises for the NULL a missing desktop file returns
|
||||
pass
|
||||
names = {n.lower() for n in (app, cls) if n}
|
||||
if not names:
|
||||
return ""
|
||||
infos = [i for i in Gio.AppInfo.get_all() if isinstance(i, Gio.DesktopAppInfo) and i.get_id()]
|
||||
for info in infos:
|
||||
wm = info.get_startup_wm_class()
|
||||
for info in Gio.AppInfo.get_all():
|
||||
wm = info.get_startup_wm_class() if isinstance(info, Gio.DesktopAppInfo) else None
|
||||
if wm and wm.lower() in names:
|
||||
return info.get_id().removesuffix(".desktop")
|
||||
for info in infos:
|
||||
did = info.get_id().removesuffix(".desktop").lower()
|
||||
if did in names or did.rsplit(".", 1)[-1] in names:
|
||||
return info.get_id().removesuffix(".desktop")
|
||||
for info in infos:
|
||||
line = info.get_commandline() or ""
|
||||
m = re.search(r"--command=(\S+)", line)
|
||||
prog = os.path.basename(m.group(1) if m else (info.get_executable() or ""))
|
||||
if prog and prog.lower() not in ("flatpak", "env", "sh", "bash", "python3", "snap") and \
|
||||
any(n == prog.lower() or n.startswith(prog.lower() + "-") for n in names):
|
||||
return info.get_id().removesuffix(".desktop")
|
||||
return ""
|
||||
return app
|
||||
|
||||
|
||||
class Launch:
|
||||
@@ -309,7 +259,6 @@ class Launch:
|
||||
self.pids = [pid] if pid else [] # the processes we started (none when D-Bus started it)
|
||||
self.entries = entries # a profile's entries still waiting for a window, or None
|
||||
self.relaunched = False
|
||||
self.dock = None # its first window opens docked on this output (launch screen=N)
|
||||
self.until = time.monotonic() + LAUNCH_SECONDS
|
||||
|
||||
|
||||
@@ -338,7 +287,7 @@ class Slot:
|
||||
self.k = k
|
||||
self.output = f"WL-{screens + k}"
|
||||
self.index = screens + k + 1 # ft-screens' number (1-based)
|
||||
self.pos = ((k % 4) * SPARE_CELL, SPARE_Y + (k // 4) * SPARE_CELL)
|
||||
self.pos = (SPARE_X + (k % 4) * SPARE_CELL, (k // 4) * SPARE_CELL)
|
||||
self.size = None # its output's size in pixels, as last set
|
||||
self.want = None # the size in pixels asked for (size is at least that)
|
||||
self.kscale = 1.0 # its output's scale in KWin, as last set
|
||||
@@ -359,7 +308,7 @@ class Daemon:
|
||||
self.sub_numbers = {} # popup/dialog id -> (window id, number)
|
||||
self.next_sub = 1
|
||||
self.launches = [] # Launch: apps we started, waiting for their window
|
||||
self.captures = {} # token -> (sender, what): "windows" and "spotlist" requests waiting for the script's report
|
||||
self.captures = {} # token -> sender: "windows" requests waiting for the script's report
|
||||
self.next_token = 1
|
||||
self.sock = None # our socket (set by main), for deferred replies
|
||||
self.places = load_places() # app -> {"rel": pose by the primary screen, "pixels", "scale"}
|
||||
@@ -445,11 +394,6 @@ class Daemon:
|
||||
return
|
||||
if kind == "added" and self.launches and not f:
|
||||
launch = self.launched(ev)
|
||||
if launch and launch.dock:
|
||||
log(f"{wid[:9]} ({ev.get('cls')}) opens docked on {launch.dock}")
|
||||
self.command(cmd="to-output", id=wid, output=launch.dock)
|
||||
self.seen(ev)
|
||||
return
|
||||
if launch and launch.entries is None:
|
||||
self.float_launched(ev, launch)
|
||||
return
|
||||
@@ -525,7 +469,7 @@ class Daemon:
|
||||
return
|
||||
# Floating when ft-floatd (re)started: take it over where it is.
|
||||
f = Float(wid, slot, None)
|
||||
f.app = desktop_name(ev.get("app", ""), ev.get("cls", "")) or ev.get("app", "")
|
||||
f.app = ev.get("app", "")
|
||||
slot.window = f
|
||||
self.floats[wid] = f
|
||||
f.scale = slot.kscale = output_scales().get(slot.output, 1.0)
|
||||
@@ -613,7 +557,7 @@ class Daemon:
|
||||
"to float another.")
|
||||
return None
|
||||
f = Float(wid, slot, {"output": ev["output"], "frame": ev["frame"], "onAllDesktops": ev.get("onAllDesktops")})
|
||||
f.app = desktop_name(ev.get("app", ""), ev.get("cls", "")) or ev.get("app", "")
|
||||
f.app = ev.get("app", "")
|
||||
slot.window = f
|
||||
self.floats[wid] = f
|
||||
scales = output_scales()
|
||||
@@ -816,11 +760,9 @@ class Daemon:
|
||||
c[1] -= 0.1 * d # a little below eye level, like a screen
|
||||
return c, ft_layout.turn_yaw((1.0, 0.0, 0.0), heading), (0.0, 1.0, 0.0), ft_layout.turn_yaw((0.0, 0.0, 1.0), heading)
|
||||
|
||||
def launch(self, app=None, argv=None, action=None, dock=None):
|
||||
"""Start an app (a desktop file name) or a command; its first window will float, or
|
||||
with dock (an output name) open docked there. action: one of the desktop file's
|
||||
actions instead (no pid: its window is known by the app's name). Returns (reply, pid or
|
||||
None)."""
|
||||
def launch(self, app=None, argv=None):
|
||||
"""Start an app (a desktop file name) or a command; its first window will float.
|
||||
Returns (reply, pid or None)."""
|
||||
if app:
|
||||
app = app.removesuffix(".desktop")
|
||||
try:
|
||||
@@ -829,22 +771,14 @@ class Daemon:
|
||||
info = None
|
||||
if info is None:
|
||||
return f"error no app {app}", None
|
||||
if action and action not in info.list_actions():
|
||||
return f"error {app} has no action {action}", None
|
||||
pids = []
|
||||
try:
|
||||
if action:
|
||||
info.launch_action(action, Gio.AppLaunchContext())
|
||||
else:
|
||||
info.launch_uris_as_manager([], Gio.AppLaunchContext(), GLib.SpawnFlags.SEARCH_PATH,
|
||||
None, None, lambda _info, pid, *_: pids.append(pid), None)
|
||||
info.launch_uris_as_manager([], Gio.AppLaunchContext(), GLib.SpawnFlags.SEARCH_PATH,
|
||||
None, None, lambda _info, pid, *_: pids.append(pid), None)
|
||||
except GLib.Error as e:
|
||||
return f"error {app}: {e.message}", None
|
||||
la = Launch(app, pids[0] if pids else None)
|
||||
la.dock = dock
|
||||
self.launches.append(la)
|
||||
log(f"launched {app}{' (' + action + ')' if action else ''}"
|
||||
f"{' docked on ' + dock if dock else ''} (pid {pids[0] if pids else 'by D-Bus'})")
|
||||
self.launches.append(Launch(app, pids[0] if pids else None))
|
||||
log(f"launched {app} (pid {pids[0] if pids else 'by D-Bus'})")
|
||||
return "ok", (pids[0] if pids else None)
|
||||
else:
|
||||
try:
|
||||
@@ -864,26 +798,16 @@ class Daemon:
|
||||
if not ev.get("normal") or ev.get("popup") or ev.get("transient"):
|
||||
return None
|
||||
chain = pid_chain(int(ev.get("pid") or 0))
|
||||
# A window in a sandbox (Flatpak, Snap) gives its pid in there, and a single-instance app
|
||||
# opens it from a process that was already running: then only its desktop file can match.
|
||||
name = desktop_name(ev.get("app", ""), ev.get("cls", "")) if any(la.app for la in self.launches) else ""
|
||||
app = ev.get("app", "")
|
||||
# (An X11 window in a Flatpak gives its pid in the sandbox: only its app id can match.)
|
||||
apps = {app, desktop_name(app, ev.get("cls", ""))} if app and any(la.app for la in self.launches) else {app}
|
||||
for la in self.launches:
|
||||
if any(p in chain for p in la.pids) or (la.app and name == la.app):
|
||||
return self.took(la)
|
||||
# Last: a window no desktop file claims, while exactly one app launch waits and none of
|
||||
# its processes made it (an app whose window names itself after nothing we can find).
|
||||
waiting = [la for la in self.launches if la.app]
|
||||
if not name and len(waiting) == 1 and not any(p in chain for la in self.launches for p in la.pids):
|
||||
log(f"{ev.get('id', '')[:9]} ({ev.get('cls', '') or '?'}): no desktop file claims it; "
|
||||
f"taken as {waiting[0].app}'s, the one launch waiting")
|
||||
return self.took(waiting[0])
|
||||
if any(p in chain for p in la.pids) or (la.app and app and la.app in apps):
|
||||
if not la.entries or len(la.entries) <= 1:
|
||||
self.launches.remove(la)
|
||||
return la
|
||||
return None
|
||||
|
||||
def took(self, la):
|
||||
if not la.entries or len(la.entries) <= 1:
|
||||
self.launches.remove(la)
|
||||
return la
|
||||
|
||||
def concealed(self):
|
||||
"""The screens (1-based) hidden on their own (ft-layout hide N)."""
|
||||
reply = self.screens.ask("concealed", quiet=True)
|
||||
@@ -901,7 +825,7 @@ class Daemon:
|
||||
def float_launched(self, ev, launch):
|
||||
"""Float a launched app's window (or one that opened on a hidden screen) where that app
|
||||
last floated (its size too), or in front of you, at the primary screen's density."""
|
||||
app = (launch.app if launch and launch.app else "") or desktop_name(ev.get("app", ""), ev.get("cls", "")) or ev.get("app", "")
|
||||
app = ev.get("app") or (launch.app if launch else "")
|
||||
known = self.places.get(app) if app else None
|
||||
ref = self.reference()
|
||||
place = to_world(ref, known["rel"]) if known and ref and len(known.get("rel", [])) == 12 else self.in_front()
|
||||
@@ -914,42 +838,25 @@ class Daemon:
|
||||
|
||||
# ------------------------------------------------------------ profiles (docs/profiles.md)
|
||||
|
||||
def capture(self, sender, what="windows"):
|
||||
""""windows" and "spotlist": have the script report every window as it is now, and
|
||||
answer when the report is in (or after 3 seconds with what we know)."""
|
||||
def capture(self, sender):
|
||||
""""windows": have the script report every window as it is now, and answer when the
|
||||
report is in (or after 3 seconds with what we know)."""
|
||||
if not sender:
|
||||
return f"error {what} needs a reply address"
|
||||
return "error windows needs a reply address"
|
||||
token = self.next_token
|
||||
self.next_token += 1
|
||||
self.captures[token] = (sender, what)
|
||||
self.captures[token] = sender
|
||||
self.command(cmd="report-all", token=token)
|
||||
GLib.timeout_add(3000, lambda: self.captured(token) and False)
|
||||
return None # answered by captured
|
||||
|
||||
def captured(self, token):
|
||||
sender, what = self.captures.pop(token, (None, None))
|
||||
sender = self.captures.pop(token, None)
|
||||
if sender and self.sock:
|
||||
entries = self.spot_entries() if what == "spotlist" else self.window_entries()
|
||||
try:
|
||||
self.sock.sendto(("ok " + json.dumps(entries)).encode(), sender)
|
||||
self.sock.sendto(("ok " + json.dumps(self.window_entries())).encode(), sender)
|
||||
except OSError as e:
|
||||
log(f"{what}: {e}")
|
||||
|
||||
def spot_entries(self):
|
||||
"""Every app window for spotwatch's search: its desktop file name, title, the panel
|
||||
it's on (a screen or a floating window's; None elsewhere), and when it was last active."""
|
||||
out = []
|
||||
for wid, ev in self.windows.items():
|
||||
if not listable(ev):
|
||||
continue
|
||||
f = self.floats.get(wid)
|
||||
m = re.match(r"WL-(\d+)$", ev.get("output", ""))
|
||||
panel = f.slot.index if f else (int(m.group(1)) + 1 if m and int(m.group(1)) < self.screens_n else None)
|
||||
out.append({"id": wid, "app": desktop_name(ev.get("app", ""), ev.get("cls", "")),
|
||||
"class": ev.get("cls", ""), "pid": ev.get("pid") or 0, "title": ev.get("caption", ""),
|
||||
"panel": panel, "floating": bool(f), "minimized": bool(ev.get("minimized")),
|
||||
"activeAt": ev.get("activeAt") or 0})
|
||||
return out
|
||||
log(f"windows: {e}")
|
||||
|
||||
def window_entries(self):
|
||||
"""Every app window and where it is, as a profile keeps it."""
|
||||
@@ -959,8 +866,7 @@ class Daemon:
|
||||
for wid, ev in self.windows.items():
|
||||
if not recordable(ev):
|
||||
continue
|
||||
app = desktop_name(ev.get("app", ""), ev.get("cls", ""))
|
||||
entry = {"app": app} if app else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")}
|
||||
entry = {"app": desktop_name(ev["app"], ev.get("cls", ""))} if ev.get("app") else {"cmd": cmdline(ev.get("pid")), "class": ev.get("cls", "")}
|
||||
if not entry.get("app") and not entry.get("cmd"):
|
||||
continue
|
||||
f = self.floats.get(wid)
|
||||
@@ -1130,15 +1036,8 @@ class Daemon:
|
||||
if not words:
|
||||
return "error empty"
|
||||
cmd, rest = words[0], words[1:]
|
||||
if cmd == "launch" and rest:
|
||||
opts = dict(w.split("=", 1) for w in rest[1:] if "=" in w)
|
||||
if len(opts) != len(rest) - 1 or set(opts) - {"action", "screen"}:
|
||||
return "error launch APP.desktop [action=ACTION] [screen=N]"
|
||||
screen = opts.get("screen")
|
||||
if screen is not None and not (screen.isdigit() and 1 <= int(screen) <= self.screens_n):
|
||||
return f"error no screen {screen}"
|
||||
return self.launch(app=rest[0], action=opts.get("action"),
|
||||
dock=f"WL-{int(screen) - 1}" if screen else None)[0]
|
||||
if cmd == "launch" and len(rest) == 1:
|
||||
return self.launch(app=rest[0])[0]
|
||||
if cmd == "run" and rest:
|
||||
try:
|
||||
argv = json.loads(text.split(None, 1)[1])
|
||||
@@ -1147,13 +1046,8 @@ class Daemon:
|
||||
if not isinstance(argv, list) or not argv or not all(isinstance(a, str) for a in argv):
|
||||
return "error run takes a JSON list"
|
||||
return self.launch(argv=argv)[0]
|
||||
if cmd in ("windows", "spotlist"):
|
||||
return self.capture(sender, cmd)
|
||||
if cmd == "activate" and len(rest) == 1:
|
||||
if rest[0] not in self.windows:
|
||||
return f"error no window {rest[0]}"
|
||||
self.command(cmd="activate", id=rest[0])
|
||||
return "ok"
|
||||
if cmd == "windows":
|
||||
return self.capture(sender)
|
||||
if cmd == "profile" and rest:
|
||||
return self.open_profile(text.split(None, 1)[1])
|
||||
if cmd == "list":
|
||||
@@ -1266,7 +1160,7 @@ def main():
|
||||
conf = ft_layout.read_conf()
|
||||
p = argparse.ArgumentParser(description="Floating windows for the Frametop desktop")
|
||||
p.add_argument("--screens", type=int, default=int(os.environ.get("FT_SCREEN_COUNT") or 0))
|
||||
p.add_argument("--slots", type=int, default=int(conf.get("FLOAT_SLOTS") or 16))
|
||||
p.add_argument("--slots", type=int, default=int(conf.get("FLOAT_SLOTS") or 8))
|
||||
p.add_argument("--margin", type=int, default=int(conf.get("FLOAT_MARGIN") or 300))
|
||||
p.add_argument("--control", default="ft_screens")
|
||||
p.add_argument("--socket", default="frametop_float")
|
||||
|
||||
@@ -46,7 +46,7 @@ RELEASE = (HERE / "install-release.sh").exists() and (HERE / "frametop-image.tar
|
||||
SOCK_DIR = Path(f"/run/user/{os.getuid()}/frametop-install")
|
||||
SOCK = SOCK_DIR / "askpass"
|
||||
ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
|
||||
# get.sh and install.sh mark each step with "== N/M what it is"
|
||||
# get.sh and install.sh mark each step with "== N/10 what it is"
|
||||
STEP = re.compile(r"^== (\d+)/(\d+) (.*)$")
|
||||
DONE_TEXT = ("Frametop is installed. Restart SteamVR once, or reboot the headset, so it loads "
|
||||
"Frametop's driver. Then open Launch a program, then Desktop.")
|
||||
@@ -101,7 +101,7 @@ def in_unit(pid):
|
||||
return False
|
||||
|
||||
|
||||
def install_command(eye_tracker, bluetooth, voice=True):
|
||||
def install_command(eye_tracker, bluetooth):
|
||||
if RELEASE:
|
||||
args = [str(HERE / "install-release.sh"), "--yes"]
|
||||
if DRY_RUN:
|
||||
@@ -110,8 +110,6 @@ def install_command(eye_tracker, bluetooth, voice=True):
|
||||
args = [str(HERE / "get.sh"), "--yes"]
|
||||
if DRY_RUN:
|
||||
args += ["--clone-only", "--dir", str(STATE / "dry-run")]
|
||||
if not voice:
|
||||
args.append("--no-voice")
|
||||
if not eye_tracker:
|
||||
args.append("--no-eye-tracker")
|
||||
if bluetooth:
|
||||
@@ -304,10 +302,8 @@ class Window(Adw.ApplicationWindow):
|
||||
def show_choices(self):
|
||||
what = f"Frametop {release_version()}" if RELEASE else "Frametop from GitHub"
|
||||
self.detail.set_label(f"This installs {what}: the multi-screen desktop, the 3D mouse, gaze "
|
||||
"mode, and Frametop's settings apps. Optional: voice dictation, and two "
|
||||
"parts that need your SteamOS password (sudo):")
|
||||
self.voice = Gtk.CheckButton(label="Voice dictation: a Right Ctrl tap or holding the left d-pad down, and "
|
||||
"what you say is typed (downloads about 320 MB)", active=True)
|
||||
"mode, and Frametop's settings apps. Two optional parts need your "
|
||||
"SteamOS password (sudo):")
|
||||
self.eye = Gtk.CheckButton(label="Our own eye tracker for gaze mode (more accurate than SteamVR's)",
|
||||
active=True)
|
||||
self.bt = Gtk.CheckButton(label="Bluetooth fixes (LE mice and keyboards, like the Swiftpoint Z3, "
|
||||
@@ -323,7 +319,7 @@ class Window(Adw.ApplicationWindow):
|
||||
self.go.connect("clicked", self.install)
|
||||
self.pw.connect("activate", self.install)
|
||||
pw_row, keypad = keypad_row(self.pw)
|
||||
self.choices = [self.voice, self.eye, self.bt, pw_row, keypad, pw_note, self.error, self.go]
|
||||
self.choices = [self.eye, self.bt, pw_row, keypad, pw_note, self.error, self.go]
|
||||
if not password_set():
|
||||
for c in (self.eye, self.bt):
|
||||
c.set_active(False)
|
||||
@@ -375,7 +371,7 @@ class Window(Adw.ApplicationWindow):
|
||||
self.error.set_visible(True)
|
||||
|
||||
def begin(self, pw):
|
||||
args = install_command(self.eye.get_active(), self.bt.get_active(), self.voice.get_active())
|
||||
args = install_command(self.eye.get_active(), self.bt.get_active())
|
||||
if pw is not None:
|
||||
try:
|
||||
self.askpass = Askpass(self.ask_again)
|
||||
|
||||
+1
-4
@@ -9,11 +9,8 @@ The Steam Frame's eye tracking as pointer input: a gaze mode for the 3D mouse (t
|
||||
|
||||
Day to day, install the gaze service, then turn gaze mode on and calibrate on the Gaze page of Frametop Input Settings (Calibrate). The installer offers the gaze service (`gaze/run.sh install`) and then our own tracker (`gaze/tracker/install.sh`), which gaze mode uses once it's installed; the probe is installed by hand.
|
||||
|
||||
The gaze service's install also turns on SteamVR's `steamvr.gazeDepEyeToHeadEnable` (`gaze/eye-rate.py on`), which counts from SteamVR's next start. The Frame's SteamVR driver runs the eye cameras at 15 to 90 fps (`eyeTrackingRateMin` and `eyeTrackingRateMax` in its `frame_hmd_additional.vrsettings`) and picks the rate as SteamVR starts: with its defaults, 15 (`eyetracking.txt`: "Set framerate 15"). That's enough for SteamVR's own tracker, but our tracker then sees 8 or 9 frames in a calibration dot's 0.6 s, where it needs 15, so it can never calibrate (a 2026-10-10 report: "the right eye was seen in only 8 frames" on every dot, while SteamVR's fit screen showed both eyes tracked). With that setting on, SteamVR asks for 90 at every start (vrserver.txt: "cv: Request eye tracking framerate: 90"). Valve's driver leaves it off and SteamVR's settings have no switch for it; what else it changes isn't documented, and the headset Frametop is developed on runs with it on. Uninstalling turns it off again only if Frametop turned it on. `scripts/doctor.sh` and `scripts/gaze-report.py` report the cameras' rate.
|
||||
|
||||
```
|
||||
gaze/run.sh install # the gaze service: builds ft-gaze and the panel, starts with SteamVR
|
||||
gaze/eye-rate.py status # SteamVR's eye cameras: the setting that keeps them at 90 fps, and their rate now
|
||||
gaze/ft-gazectl on # the pointer follows your gaze (off: the mouse alone)
|
||||
gaze/tracker/install.sh # our own eye tracker's frame grabber (asks for sudo)
|
||||
gaze/build.sh # build ft-gaze and the panel by hand
|
||||
@@ -59,7 +56,7 @@ The tracker stops when the headset is off your head. SteamVR also calibrates gaz
|
||||
|
||||
`gaze/tracker/` is an eye tracker of our own, because SteamVR's is about 1.5 degrees off after the best correction the gaze service can learn, and what's left is mostly look-to-look noise that no correction on top of its output can remove. Ours processes the eye cameras itself: 0.59 degrees (median) in its best live session against 0.83 for SteamVR's with the probe's correction, and after the headset was taken off and put back without recalibrating, 0.58 once your first clicks had taught it where the headset sat (`tracker/findings.md` has the measurements).
|
||||
|
||||
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye, or 15 without `gaze/eye-rate.py on`) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too). It also adds it to SteamOS's update keep list (`/etc/atomic-update.conf.d/frametop-eyegrab.conf`): an update deletes `/etc` files that list doesn't name, and without the grabber gaze mode falls back to SteamVR's tracker and its calibration. `scripts/doctor.sh` reports a grabber an update deleted.
|
||||
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too). It also adds it to SteamOS's update keep list (`/etc/atomic-update.conf.d/frametop-eyegrab.conf`): an update deletes `/etc` files that list doesn't name, and without the grabber gaze mode falls back to SteamVR's tracker and its calibration. `scripts/doctor.sh` reports a grabber an update deleted.
|
||||
- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`.
|
||||
- `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones are meant for a PC: if you have `frame-job` (a personal tool, not in this repo), `gaze/tracker/.frame-job` sends them there. `lab/py` runs them with that Python (in the dev container on the Frame; on a PC, the same venv from `requirements.txt`, which frame-job's setup makes).
|
||||
|
||||
|
||||
@@ -1,141 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Keep SteamVR's eye cameras at 90 fps, for gaze mode.
|
||||
|
||||
The Frame's SteamVR driver runs the eye cameras at 15 to 90 fps (steamvr.eyeTrackingRateMin
|
||||
and eyeTrackingRateMax, in drivers/frame_hmd/resources/frame_hmd_additional.vrsettings),
|
||||
and asks for the rate once, as SteamVR starts (vrserver.txt: "cv: Request eye tracking
|
||||
framerate"). With its defaults it asks for 15 (eyetracking.txt: "Set framerate 15"). SteamVR's
|
||||
own tracker copes, but our tracker gets 8 or 9 frames in a calibration dot's 0.6 s, where it
|
||||
needs 15, so it can never calibrate, and the gaze pointer would move 15 times a second.
|
||||
With steamvr.gazeDepEyeToHeadEnable on, SteamVR asks for 90 (seen 2026-10-10: a headset with
|
||||
it on asks for 90 at every start; a user's with it off, at 15, calibrated once it was on).
|
||||
SteamVR's settings have no switch for it.
|
||||
|
||||
gaze/eye-rate.py on # turn it on (gaze/run.sh install); it counts from SteamVR's next start
|
||||
gaze/eye-rate.py off # turn it off again, if this turned it on (gaze/run.sh uninstall)
|
||||
gaze/eye-rate.py status # the setting, and the rate the eye cameras run at now
|
||||
|
||||
With SteamVR running it goes through vrcmd, since SteamVR keeps its settings in memory and
|
||||
writes over the file; without it, it edits SteamVR's settings file.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HOME = Path.home()
|
||||
SECTION, KEY = "steamvr", "gazeDepEyeToHeadEnable"
|
||||
STEAMVR_BIN = "/opt/steamvr/bin/linuxarm64"
|
||||
# The first that exists is SteamVR's (as gaze/gazecal.py reads them).
|
||||
STEAMVR_SETTINGS = (HOME / ".config/openvr/config/steamvr.vrsettings",
|
||||
HOME / ".steam/steam/config/steamvr.vrsettings")
|
||||
# There when this turned the setting on (it was off), so `off` leaves a setting someone chose alone.
|
||||
MARK = HOME / ".local/state/frametop/gaze/steamvr-eye-rate"
|
||||
EYE_LOG = HOME / ".local/share/Steam/logs/eyetracking.txt"
|
||||
# A terminal in the desktop has the desktop's own config folder, where SteamVR's path registry
|
||||
# isn't (see scripts/update-check.py): vrcmd needs the real one.
|
||||
ENV = dict(os.environ, XDG_CONFIG_HOME=str(HOME / ".config"), LD_LIBRARY_PATH=STEAMVR_BIN)
|
||||
|
||||
|
||||
def steamvr_up():
|
||||
return subprocess.run(["pgrep", "-x", "vrserver"], capture_output=True).returncode == 0
|
||||
|
||||
|
||||
def vrcmd(*args):
|
||||
p = subprocess.run([f"{STEAMVR_BIN}/vrcmd", *args], capture_output=True, text=True, timeout=15, env=ENV)
|
||||
return p.stdout
|
||||
|
||||
|
||||
def settings_file():
|
||||
return next((p for p in STEAMVR_SETTINGS if p.exists()), STEAMVR_SETTINGS[0])
|
||||
|
||||
|
||||
def get():
|
||||
"""The setting: True, False, or None when SteamVR isn't running and its file doesn't say
|
||||
(then the driver's default, off)."""
|
||||
if steamvr_up():
|
||||
# vrcmd prints "steamvr.gazeDepEyeToHeadEnable=true"
|
||||
m = re.search(rf"^{SECTION}\.{KEY}=(\w+)", vrcmd("--settings-bool", f"{SECTION}.{KEY}"), re.M)
|
||||
return None if not m else m.group(1) in ("true", "1")
|
||||
try:
|
||||
value = json.loads(settings_file().read_text()).get(SECTION, {}).get(KEY)
|
||||
except (OSError, ValueError, AttributeError):
|
||||
return None
|
||||
return value if isinstance(value, bool) else None
|
||||
|
||||
|
||||
def put(on):
|
||||
if steamvr_up():
|
||||
# 1 or 0: vrcmd reads "true" as 0.
|
||||
vrcmd("--set-settings-bool", f"{SECTION}.{KEY}", "1" if on else "0")
|
||||
return get() is on
|
||||
path = settings_file()
|
||||
try:
|
||||
d = json.loads(path.read_text())
|
||||
except FileNotFoundError:
|
||||
d = {}
|
||||
except (OSError, ValueError) as e:
|
||||
print(f"can't read {path}: {e}", file=sys.stderr)
|
||||
return False
|
||||
d.setdefault(SECTION, {})[KEY] = on
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".frametop-tmp")
|
||||
tmp.write_text(json.dumps(d, indent=3) + "\n")
|
||||
tmp.replace(path)
|
||||
return True
|
||||
|
||||
|
||||
def camera_rate():
|
||||
"""The rate the eye cameras run at since the eye tracker last started: the last "Set
|
||||
framerate" after its last start, or 90 when it set none (it starts at 90). None without
|
||||
the log or a start in it."""
|
||||
try:
|
||||
lines = EYE_LOG.read_text(errors="replace").splitlines()
|
||||
except OSError:
|
||||
return None
|
||||
starts = [i for i, line in enumerate(lines) if " startup with PID" in line]
|
||||
if not starts:
|
||||
return None
|
||||
rates = re.findall(r"CStereoAdspCams: Set framerate ([\d.]+)", "\n".join(lines[starts[-1]:]))
|
||||
return float(rates[-1]) if rates else 90.0
|
||||
|
||||
|
||||
def main():
|
||||
what = sys.argv[1] if len(sys.argv) == 2 else ""
|
||||
if what == "on":
|
||||
if get():
|
||||
print(f"SteamVR's eye cameras: {SECTION}.{KEY} is on already, so they run at 90 fps")
|
||||
return 0
|
||||
if not put(True):
|
||||
print(f"couldn't turn on {SECTION}.{KEY}: the eye cameras may stay at 15 fps, and our eye "
|
||||
"tracker can't calibrate at that. Try again with: gaze/eye-rate.py on", file=sys.stderr)
|
||||
return 1
|
||||
MARK.parent.mkdir(parents=True, exist_ok=True)
|
||||
MARK.write_text(f"{SECTION}.{KEY} was off; Frametop turned it on\n")
|
||||
print(f"SteamVR's eye cameras: turned on {SECTION}.{KEY}, so they run at 90 fps, not 15, "
|
||||
"once SteamVR restarts")
|
||||
return 0
|
||||
if what == "off":
|
||||
if not MARK.exists():
|
||||
print(f"SteamVR's eye cameras: left as they are (Frametop didn't turn on {SECTION}.{KEY})")
|
||||
return 0
|
||||
if not put(False):
|
||||
print(f"couldn't turn off {SECTION}.{KEY}. Try again with: gaze/eye-rate.py off", file=sys.stderr)
|
||||
return 1
|
||||
MARK.unlink()
|
||||
print(f"SteamVR's eye cameras: turned {SECTION}.{KEY} off again (back to 15 fps once SteamVR restarts)")
|
||||
return 0
|
||||
if what == "status":
|
||||
value = get()
|
||||
rate = camera_rate()
|
||||
print(f"{SECTION}.{KEY}: {'on' if value else 'off'}" + (" (set by Frametop)" if MARK.exists() else ""))
|
||||
print("eye cameras: " + (f"{rate:g} fps since the eye tracker started" if rate else "unknown (no eyetracking.txt)"))
|
||||
return 0
|
||||
print("usage: gaze/eye-rate.py on|off|status", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+3
-7
@@ -1,7 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install, start, stop, or inspect the gaze service (ft-gazed) on the Frame.
|
||||
# Usage: gaze/run.sh install|uninstall # user service, starts with SteamVR; install also keeps
|
||||
# # SteamVR's eye cameras at 90 fps (gaze/eye-rate.py)
|
||||
# Usage: gaze/run.sh install|uninstall # user service, starts with SteamVR
|
||||
# gaze/run.sh start|stop|restart|status|log [lines]
|
||||
set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
||||
@@ -12,13 +11,10 @@ case ${1:-status} in
|
||||
install)
|
||||
[ "$FRAME_RELEASE" = 1 ] || "$root/gaze/build.sh"
|
||||
fill_template "$root/gaze/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit"
|
||||
on_frame "chmod +x gaze/ft-gazed gaze/ft-gazectl gaze/eye-rate.py"
|
||||
on_frame "/usr/bin/python3 gaze/eye-rate.py on" || true
|
||||
on_frame "chmod +x gaze/ft-gazed gaze/ft-gazectl"
|
||||
"$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable $unit
|
||||
$(start_with_steamvr $unit)" ;;
|
||||
uninstall)
|
||||
"$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed"
|
||||
on_frame "/usr/bin/python3 gaze/eye-rate.py off" || true ;;
|
||||
uninstall) "$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed" ;;
|
||||
start|stop|restart) "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit" ;;
|
||||
status) "$frame" --host "systemctl --user is-active $unit" || true; on_frame "gaze/ft-gazectl status" || true ;;
|
||||
log) "$frame" --host "journalctl --user -u $unit --no-pager -o cat -n ${2:-30}" ;;
|
||||
|
||||
@@ -1,138 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline test of gaze/eye-rate.py (SteamVR's eye cameras at 90 fps): on, off, and the camera
|
||||
rate, with SteamVR off (its settings file) and on (vrcmd). HOME is a temporary folder and
|
||||
vrcmd and pgrep are stand-ins, so the live SteamVR settings are never read or changed.
|
||||
|
||||
gaze/test/eye-rate-test.py
|
||||
"""
|
||||
import contextlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
tmp = Path(tempfile.mkdtemp(prefix="ft-eye-rate-test-"))
|
||||
os.environ["HOME"] = str(tmp) # eye-rate.py's paths come from it
|
||||
|
||||
loader = importlib.machinery.SourceFileLoader("eyerate", os.path.join(HERE, "..", "eye-rate.py"))
|
||||
er = importlib.util.module_from_spec(importlib.util.spec_from_loader("eyerate", loader))
|
||||
loader.exec_module(er)
|
||||
assert str(er.HOME) == str(tmp), "eye-rate.py must take HOME from the environment"
|
||||
|
||||
failures = []
|
||||
|
||||
|
||||
def check(what, got, want):
|
||||
if got != want:
|
||||
failures.append(what)
|
||||
print(f"FAIL {what}: got {got!r}, want {want!r}", flush=True)
|
||||
|
||||
|
||||
def main_with(*args):
|
||||
"""eye-rate.py's main() with these arguments: (exit code, what it printed)."""
|
||||
er.sys.argv = ["eye-rate.py", *args]
|
||||
out = io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(out):
|
||||
code = er.main()
|
||||
return code, out.getvalue()
|
||||
|
||||
|
||||
settings = er.STEAMVR_SETTINGS[0]
|
||||
|
||||
|
||||
def file_value():
|
||||
return json.loads(settings.read_text()).get("steamvr", {}).get("gazeDepEyeToHeadEnable")
|
||||
|
||||
|
||||
# --- SteamVR off: the settings file ---
|
||||
er.steamvr_up = lambda: False
|
||||
settings.parent.mkdir(parents=True)
|
||||
settings.write_text(json.dumps({"steamvr": {"analogGain": 0.25}, "dashboard": {"x": 1}}, indent=3))
|
||||
check("off by default (not in the file)", er.get(), None)
|
||||
code, _ = main_with("on")
|
||||
check("on: exit 0", code, 0)
|
||||
check("on: written to the file", file_value(), True)
|
||||
check("on: SteamVR's other settings kept", json.loads(settings.read_text())["steamvr"]["analogGain"], 0.25)
|
||||
check("on: other sections kept", json.loads(settings.read_text())["dashboard"], {"x": 1})
|
||||
check("on: marked as Frametop's", er.MARK.exists(), True)
|
||||
code, out = main_with("on")
|
||||
check("on again: says it's on already", "already" in out, True)
|
||||
code, _ = main_with("off")
|
||||
check("off: written to the file", file_value(), False)
|
||||
check("off: mark gone", er.MARK.exists(), False)
|
||||
|
||||
# Someone turned it on themselves: off leaves it.
|
||||
settings.write_text(json.dumps({"steamvr": {"gazeDepEyeToHeadEnable": True}}))
|
||||
code, out = main_with("on")
|
||||
check("on, set by the user: no mark", er.MARK.exists(), False)
|
||||
code, out = main_with("off")
|
||||
check("off, set by the user: left on", file_value(), True)
|
||||
check("off, set by the user: says so", "left as they are" in out, True)
|
||||
|
||||
# No settings file yet (SteamVR never ran): on makes one.
|
||||
settings.unlink()
|
||||
code, _ = main_with("on")
|
||||
check("no file: on makes it", file_value(), True)
|
||||
main_with("off")
|
||||
|
||||
# A broken file isn't overwritten.
|
||||
settings.write_text("{not json")
|
||||
code, _ = main_with("on")
|
||||
check("broken file: on fails", code, 1)
|
||||
check("broken file: left alone", settings.read_text(), "{not json")
|
||||
check("broken file: no mark", er.MARK.exists(), False)
|
||||
settings.unlink()
|
||||
|
||||
# --- SteamVR on: vrcmd ---
|
||||
live = {"value": False}
|
||||
calls = []
|
||||
|
||||
|
||||
def fake_vrcmd(*args):
|
||||
calls.append(args)
|
||||
if args[0] == "--settings-bool":
|
||||
return f"steamvr.gazeDepEyeToHeadEnable={'true' if live['value'] else 'false'}\n"
|
||||
if args[0] == "--set-settings-bool":
|
||||
live["value"] = args[2] == "1" # as vrcmd does: "true" would read as 0
|
||||
return f"steamvr.gazeDepEyeToHeadEnable={args[2]}\n"
|
||||
return ""
|
||||
|
||||
|
||||
er.steamvr_up = lambda: True
|
||||
er.vrcmd = fake_vrcmd
|
||||
code, _ = main_with("on")
|
||||
check("SteamVR on: set through vrcmd", live["value"], True)
|
||||
check("SteamVR on: sets 1, never 'true'", [c for c in calls if c[0] == "--set-settings-bool"][-1][2], "1")
|
||||
check("SteamVR on: the file isn't touched", settings.exists(), False)
|
||||
check("SteamVR on: marked", er.MARK.exists(), True)
|
||||
code, _ = main_with("off")
|
||||
check("SteamVR on: off through vrcmd", live["value"], False)
|
||||
check("SteamVR on: mark gone", er.MARK.exists(), False)
|
||||
|
||||
# vrcmd that doesn't take it: on fails, no mark.
|
||||
er.vrcmd = lambda *args: "" if args[0] == "--set-settings-bool" else "steamvr.gazeDepEyeToHeadEnable=false\n"
|
||||
code, _ = main_with("on")
|
||||
check("vrcmd didn't take it: on fails", code, 1)
|
||||
check("vrcmd didn't take it: no mark", er.MARK.exists(), False)
|
||||
|
||||
# --- The camera rate, from eyetracking.txt ---
|
||||
check("no eyetracking.txt: unknown", er.camera_rate(), None)
|
||||
er.EYE_LOG.parent.mkdir(parents=True)
|
||||
head = "Sat Oct 10 2026 22:35:43.000000 [Info] - "
|
||||
er.EYE_LOG.write_text("\n".join([
|
||||
head + "eyetracking 2.18.2 startup with PID=1, config=x",
|
||||
head + "CStereoAdspCams: Set framerate 15.000000",
|
||||
head + "Eye tracking shutting down",
|
||||
head + "eyetracking 2.18.2 startup with PID=2, config=x",
|
||||
head + "Eye tracking started"]) + "\n")
|
||||
check("no rate set since the last start: 90", er.camera_rate(), 90.0)
|
||||
with er.EYE_LOG.open("a") as f:
|
||||
f.write(head + "CStereoAdspCams: Set framerate 15.000000\n")
|
||||
check("15 set since the last start", er.camera_rate(), 15.0)
|
||||
|
||||
print("eye-rate-test:", "FAILED " + ", ".join(failures) if failures else "all passed")
|
||||
raise SystemExit(1 if failures else 0)
|
||||
@@ -23,16 +23,16 @@
|
||||
# --version V with --release: that release (tag vV), not the newest
|
||||
# --zip FILE|URL with --release: this Frametop.zip instead of GitHub's newest
|
||||
# --any-steamos with --release: install even on a SteamOS build the release breaks on
|
||||
# --yes, --no-voice, --no-eye-tracker, --no-bluetooth, --bluetooth passed to install.sh (--yes also
|
||||
# --yes, --no-eye-tracker, --no-bluetooth, --bluetooth passed to install.sh (--yes also
|
||||
# answers this script's questions: the version already there, or stable)
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
usage: get.sh [--stable | --experimental | --branch NAME] [--dir DIR] [--clone-only] [--yes]
|
||||
[--no-voice] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
|
||||
[--no-eye-tracker] [--no-bluetooth | --bluetooth]
|
||||
get.sh --release [--stable | --experimental | --version V | --zip FILE|URL]
|
||||
[--any-steamos] [--dir DIR] [--clone-only] [--yes] [--no-voice] [--no-eye-tracker]
|
||||
[--any-steamos] [--dir DIR] [--clone-only] [--yes] [--no-eye-tracker]
|
||||
[--no-bluetooth | --bluetooth]
|
||||
piped: curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- [options]
|
||||
EOF
|
||||
@@ -129,7 +129,7 @@ main() {
|
||||
--version) want=${2:?--version needs a version}; shift ;;
|
||||
--any-steamos) any=1 ;;
|
||||
--yes) yes=1; pass+=("$1") ;;
|
||||
--no-bluetooth|--bluetooth|--no-eye-tracker|--no-voice) pass+=("$1") ;;
|
||||
--no-bluetooth|--bluetooth|--no-eye-tracker) pass+=("$1") ;;
|
||||
-h|--help) usage; return 0 ;;
|
||||
*) echo "unknown option: $1" >&2; usage >&2; return 2 ;;
|
||||
esac
|
||||
|
||||
@@ -3,9 +3,3 @@ models/onnx/
|
||||
models/*.task
|
||||
# frame-job settings (where replays run, and the lab's capture folder): personal, not shipped
|
||||
.frame-job
|
||||
dumps/
|
||||
# tap replays' output and their per-host drivers (frame-hands notes/taps.md)
|
||||
taps-out/
|
||||
tools/taps-*.sh
|
||||
# recorded demo clips (hands/rec/demo/camgifs.py) show the room around the recorder: never shipped
|
||||
rec/demos-recorded/
|
||||
+4
-20
@@ -1,8 +1,7 @@
|
||||
# Hand tracking, built into build/ (hands/build.sh runs this in the dev container):
|
||||
# make ft-camd (camd/: runs on the host, so linked statically) and ft-hands (track/)
|
||||
# make tools ft-handreplay and ft-ringplay, for recordings
|
||||
# make check builds and runs the C++ unit tests (tests/sides_test.cpp, tests/skeleton_test.cpp,
|
||||
# tests/pinch_test.cpp, tests/flagbuf_test.cpp)
|
||||
# make check builds and runs the C++ unit tests (tests/sides_test.cpp)
|
||||
# The first build fetches ncnn (NCNN_TAG) and builds it into build/ncnn, which takes a few
|
||||
# minutes. NCNN=DIR uses an ncnn install already built instead.
|
||||
NCNN_TAG = 20260526
|
||||
@@ -15,7 +14,7 @@ LDLIBS = $(NCNN)/lib/libncnn.a -ljsoncpp -fopenmp -lpthread
|
||||
LDFLAGS ?=
|
||||
|
||||
CAMD = camd/camd.c camd/tp.c camd/xrcams.c
|
||||
TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp track/sides.cpp track/skeleton.cpp track/flagbuf.cpp
|
||||
TRACK = track/calib.cpp track/nets.cpp track/tracker.cpp track/io.cpp track/record.cpp track/pinch.cpp track/sides.cpp
|
||||
HDR = $(wildcard track/*.h) camd/fhring.h include/fh_hands.h include/fh_gestures.h
|
||||
|
||||
all: build/ft-camd build/ft-hands
|
||||
@@ -37,23 +36,8 @@ build/sides-test: tests/sides_test.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
|
||||
@mkdir -p build
|
||||
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ tests/sides_test.cpp $(TRACK) $(LDLIBS)
|
||||
|
||||
build/skeleton-test: tests/skeleton_test.cpp track/skeleton.cpp track/skeleton.h track/geom.h
|
||||
@mkdir -p build
|
||||
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ tests/skeleton_test.cpp track/skeleton.cpp
|
||||
|
||||
build/pinch-test: tests/pinch_test.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
|
||||
@mkdir -p build
|
||||
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ tests/pinch_test.cpp $(TRACK) $(LDLIBS)
|
||||
|
||||
build/flagbuf-test: tests/flagbuf_test.cpp $(TRACK) $(HDR) $(NCNN)/lib/libncnn.a
|
||||
@mkdir -p build
|
||||
$(CXX) $(CXXFLAGS) $(LDFLAGS) -o $@ tests/flagbuf_test.cpp $(TRACK) $(LDLIBS)
|
||||
|
||||
check: build/sides-test build/skeleton-test build/pinch-test build/flagbuf-test
|
||||
check: build/sides-test
|
||||
build/sides-test
|
||||
build/skeleton-test
|
||||
build/pinch-test
|
||||
build/flagbuf-test
|
||||
|
||||
build/ft-ringplay: track/ringplay.cpp track/record.h camd/fhring.h
|
||||
@mkdir -p build
|
||||
@@ -70,6 +54,6 @@ build/ncnn/install/lib/libncnn.a:
|
||||
cmake --build build/ncnn/build --target install > build/ncnn/build.log
|
||||
|
||||
clean:
|
||||
rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay build/sides-test build/skeleton-test build/pinch-test build/flagbuf-test
|
||||
rm -f build/ft-camd build/ft-hands build/ft-handreplay build/ft-ringplay build/sides-test
|
||||
|
||||
.PHONY: all tools check clean
|
||||
+21
-195
@@ -4,9 +4,7 @@ Hand tracking from the headset's own cameras. It's deferred: it costs a lot of t
|
||||
|
||||
It serves two things in Frametop:
|
||||
|
||||
- **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro. The hole is round, one per hand, until the tracking is good enough for a hand-shaped one (also there, built from the skeleton; see "Hand cutouts and near input" below).
|
||||
- **Rendered hands:** ft-handview draws your hands as a 3D hand model posed by the tracking, Quest-style, over everything (`screens/handview.cpp`; see "Rendered hands" below): with Room View off (a VR game, SteamVR Home, the dashboard, or the panels over the void), where ft-screens doesn't cut, and by default with Room View on too, floating over the hole.
|
||||
- **Near input on Frametop's panels:** poke a screen with your index finger to click, pinch its bar, edges or any corner to move or resize it, hold it like a card or in a fist to carry and turn it, hold it with both hands to stretch it, flick it away or throw it (ft-screens, `screens/handinput.cpp`; see "Hand cutouts and near input"). SteamVR's own panels (the dashboard, the Steam menu, other apps' overlays) take pokes and pinch clicks through the pointer helper's laser, and the hand ray's pinch on a floating one's grab bar moves it.
|
||||
- **Hand cutouts:** where your hand is between an eye and a screen, that eye sees the room through the screen (ft-screens, `screens/handcut.cpp`), so your hands show over the screens the way they do on a Vision Pro.
|
||||
- **Pinches and grips:** with `POINTER_HANDS=1`, the pointer helper takes them as clicks and drags. Look at something and pinch to click it, with the eye tracker doing the looking (`gaze/`), or close your hand to press and drag what the pointer is on. See "Pinches and grips in the pointer" below.
|
||||
|
||||
Two programs, each a user service that stops when SteamVR does:
|
||||
@@ -14,28 +12,20 @@ Two programs, each a user service that stops when SteamVR does:
|
||||
- `ft-camd` (`camd/`, C) borrows XRService's camera buffers and publishes the four IR tracking cameras' frames to a shared-memory ring. It runs on the host.
|
||||
- `ft-hands` (`track/`, C++) finds hands in those frames with MediaPipe's palm and landmark models on ncnn, triangulates them, and publishes them. It runs in the dev container.
|
||||
|
||||
`hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services, and links `hands/ft-handsctl` into `~/.local/bin`. `install.sh` doesn't install it. When they run is `HANDS_MODE` in `~/.config/frametop.conf`, set on the Hands page of Frametop Input Settings or with `ft-handsctl mode`; `hands/handsmode.py` does the starting and stopping, for these two and the hand view (`frametop-handview.service`, when it's installed):
|
||||
|
||||
- `off` (the default): nothing starts them. `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it, as before; it stops with SteamVR.
|
||||
- `frametop`: they run while the Frametop desktop does. The session (`session/frametop-session.sh`, screens backend) starts them once ft-screens is up and stops them as it ends.
|
||||
- `always`: they're wanted by `steamvr.service` (`systemctl --user add-wants`), so they run whenever SteamVR does, in games and the SteamVR dashboard too, with or without the desktop. Pausing Frametop for a VR game leaves them running and unparked.
|
||||
|
||||
Choosing a mode applies it at once: the services start or stop, with no desktop restart. Without ft-camd's capabilities nothing starts, and the Hands page says to run `hands/run.sh caps` (it asks for the password in the terminal; the page never does). `hands/run.sh install` and `caps` apply the mode too, but stop nothing, so what `ft-handsctl on` or `ft-cutouts on` started keeps running.
|
||||
They don't start with SteamVR. `hands/run.sh install` builds them, gives ft-camd its capabilities, installs both services disabled, and links `hands/ft-handsctl` into `~/.local/bin`. Then `ft-handsctl on` starts hand tracking and `ft-handsctl off` stops it. `install.sh` doesn't install it.
|
||||
|
||||
For the cutouts alone, `hands/ft-cutouts on` starts the same two programs with ft-hands' `--no-gestures`: your hands show through the screens, and no pinch or grip is detected, so nothing clicks or drags. It runs this checkout's build as transient user units, so it needs `hands/build.sh` and ft-camd's capabilities (`hands/run.sh caps`) but not `hands/run.sh install`. It and `ft-handsctl on` stop each other's services, and it stops with SteamVR too.
|
||||
|
||||
```
|
||||
ft-handsctl on | off # on the Frame: start or stop hand tracking (SteamVR must be running)
|
||||
ft-handsctl mode [frametop|always|off] # HANDS_MODE: show it, or set and apply it
|
||||
ft-handsctl status # the services, and ft-hands' last status lines
|
||||
ft-handsctl log [lines]
|
||||
ft-handsctl cutouts on|off|state # ft-screens' hand cutouts, without stopping tracking
|
||||
ft-handsctl gestures # pinches, grips and middle pinches, live (tools/watch_gestures.py --distance)
|
||||
ft-handsctl flag [note...] # flag a tracking failure: the last seconds go to a folder (see "Flags")
|
||||
ft-handsctl gestures # pinches and grips, live (tools/watch_gestures.py --distance)
|
||||
|
||||
hands/ft-cutouts on | off | status # the cutouts only: no pinches or grips
|
||||
|
||||
hands/run.sh install # build, give ft-camd its capabilities (sudo, once per build), install, apply HANDS_MODE
|
||||
hands/run.sh install # build, give ft-camd its capabilities (sudo, once per build), install disabled
|
||||
hands/run.sh start|stop # start or stop the services
|
||||
hands/run.sh restart # after changing a setting
|
||||
hands/run.sh status
|
||||
@@ -52,9 +42,6 @@ Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides
|
||||
- `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
|
||||
- `HANDS_MODELS` (unset: `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to run fine-tuned models, such as the ones trained on the hand dataset, without passing options to every launcher. Those folders usually lack the `*-int8` files, so `--int8` won't load them.
|
||||
- `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
|
||||
- `HANDS_SKELETON=1`: fit a skeleton with your bone lengths and publish it (see "Skeleton" below); `0` publishes the filtered landmarks as before. As `--skeleton`.
|
||||
- `HANDS_PINCH_DIP=0`: `1` also starts a pinch on a quick dip of the thumb-index distance that never reaches 2 cm (see the pinch under "Gestures" below). As `--pinch-dip`.
|
||||
- `HANDS_FLAG_BUFFER=1`: keep the last seconds of hand crops in RAM for `ft-handsctl flag` (see "Flags"); `0` keeps nothing. As `--flag-buffer`. `HANDS_FLAG_SECONDS` (12), `HANDS_FLAG_CROP` (224 px), `HANDS_FLAG_MB` (80) size it. `HANDS_FLAG_DIR` (`~/.local/share/frametop/hands/flags`) is where flags go; `HANDS_FLAG_FLASH=1` pulses the drawn hands when a flag is taken, `HANDS_FLAG_TONE=1` adds a short tone, `HANDS_FLAG_VIEW=1` saves a frame of your view.
|
||||
|
||||
The pointer helper's `POINTER_HANDS` and `POINTER_PINCH_*`/`POINTER_GRIP_*` settings are in "Pinches and grips in the pointer" below.
|
||||
|
||||
@@ -63,9 +50,8 @@ Files, all in `/run/user/UID/frametop-hands/` (private to the user; not `/run/us
|
||||
| File | Written by | Layout | Read by |
|
||||
| --- | --- | --- | --- |
|
||||
| `cam-ring` | ft-camd | `camd/fhring.h` | ft-hands, `tools/ring.py` |
|
||||
| `hands` | ft-hands | `include/fh_hands.h` (version 2: version 1, then a skeleton per hand) | ft-screens (`screens/handcut.cpp`, version 1's part), the hand recorder (`rec/session.py`) |
|
||||
| `hands` | ft-hands | `include/fh_hands.h` | ft-screens (`screens/handcut.cpp`) |
|
||||
| `gestures` | ft-hands | `include/fh_gestures.h` | the pointer helper (`pointer/helper/ft-pointer.cpp`), `tools/watch_gestures.py` |
|
||||
| `flag` | `ft-handsctl flag` (`handsflag.py`) | key=value lines: `dir`, `note`, `flag_unix`, `said_unix` | ft-hands, which removes it when it takes it |
|
||||
|
||||
The source keeps the `fh_` names and magic strings of frame-hands, the project it started as, so recordings made with it still work.
|
||||
|
||||
@@ -100,7 +86,7 @@ Options:
|
||||
- `--sensor S`: only the mono cameras whose sensor name contains S.
|
||||
- `--status S`: a status line every S seconds (0: never).
|
||||
|
||||
It exits when XRService exits, when a camera's buffers keep going stale, which means XRService has reallocated them, or when a camera stops while the others keep running (3 s), which means XRService has moved it to another device (the colour module was unplugged or plugged in). The service starts it again, and it attaches to the new buffers.
|
||||
It exits when XRService exits, or when a camera's buffers keep going stale, which means XRService has reallocated them. The service starts it again, and it attaches to the new buffers.
|
||||
|
||||
**Which camera is which:** video9 is `slam_right` (sensor `og01a1b 4-0036`), video13 is `slam_left` (`4-0060`), video6 is `upper_left` and video7 is `upper_right`; without the colour module the side pair is on video0 (`slam_right`) and video3. XRService says so in its log: `Found camera 'slam_left': ... v4l_subdev=/dev/v4l-subdev30` names each sensor, and each `TrackingCameraInit` line gives the device and subdev it opened. ft-hands and `camcheck.py` name the devices that way. Until 2026-10-05 they named them by the `TrackingCameraInit` index instead, which is only the order XRService opens them in (index 0 was `slam_right` on every start logged), and ft-camd bound each buffer queue to a device by the order of XRService's file descriptors, which changes when XRService restarts its cameras. The two mistakes made the side names come out swapped on most starts and right on some. ft-camd now asks each device which buffers it holds (`VIDIOC_QUERYBUF` names the descriptor XRService queued at each index) and only falls back to the order if that fails. With the names swapped, every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands still checks by itself, in case (`track/sides.h`, `HANDS_SWAP_SIDES=auto`):
|
||||
- Whenever a hand's landmarks are found in two cameras at once (one of them a side camera), it intersects the rays through the 21 landmarks twice: once with the calibrations as named, once with the two side cameras exchanged. The same hand seen the right way meets within a few mm, in front of both cameras and as far away as its size says. The wrong way misses by centimetres or meets behind a camera.
|
||||
@@ -131,29 +117,21 @@ Options:
|
||||
- `--models DIR`: where the models are.
|
||||
- `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
|
||||
- `--no-publish`: don't write the hands and gestures files.
|
||||
- Parked: while `/run/user/UID/frametop-hands/park` exists, ft-hands processes no frames (the models stay loaded), publishes no hands, and ends any gesture down, as lost; it writes `0` to `hands-want` there every second, and ft-camd stops copying the mono cameras' frames while that file is fresh (3 s) and says 0. The input relay parks it while Frametop is paused for a VR game with hand interaction on (`POINTER_HANDS=1`), instead of stopping the services, and unparks it while the dashboard is open, so the hands work there; it tracks again from a palm search. Not while recording.
|
||||
- `--no-gestures`: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. `ft-cutouts` runs it this way.
|
||||
- `--record DIR`, `--record-for S`, `--record-hz N`: save every frame set for S seconds (default 120) to `DIR/sets.bin`, or at most N sets a second with `--record-hz` (the hand recorder uses 10). Every set is about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
|
||||
- `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
|
||||
- `--reacquire S` (`HANDS_REACQUIRE`, default 1.5): how long after losing a hand the palm search aims at where it should be (see "Scheduling"). 0 turns it off.
|
||||
- `--pair-sides 0|1` (`HANDS_PAIR_SIDES`, default 1): two hands seen together are one left and one right (see "Scheduling").
|
||||
- `--hold S` (default 0.6): how long a lost hand tracked for 30+ frames is held before it's forgotten (0.3 s with `--reacquire 0`, as before).
|
||||
- `--keep-presence P`: the landmark presence a tracked view needs to stay tracked. New views always need 0.5. Default 0.5. Lowering it to 0.2 barely helped in the bright recording, because lost hands drop to near-zero presence.
|
||||
- `--ring PATH`: read frames from another ring, such as `ft-ringplay`'s.
|
||||
- `--cams auto|mono|color|all` (`HANDS_CAMERAS`, default `auto`): which cameras to track with. The mono IR cameras light the hands themselves and track well in dim rooms, but in bright light they expose for the room and the hands come out dark. The colour pair is the other way round. `auto` goes by the colour frames' mean brightness: at `--bright-on` (`HANDS_BRIGHT_ON`, 40) or over for 2 s it tracks with `--bright` (`HANDS_BRIGHT`: `all`, every camera, the default, or `color`), and under `--bright-off` (`HANDS_BRIGHT_OFF`, 25) for 2 s with the mono cameras again. A dim evening room read 9. The switch is logged (`cameras: mono -> all (...)`), and the status line gives the colour level, the mono cameras' ambient IR, and how many steps had colour frames. Colour frames arrive on their own schedule, so a step holds the mono set, the colour pair, or both, and views wait in their camera for its next frame. With no colour cameras in the ring (ft-camd without `--with-color`, as the service runs it), ft-hands tracks with the mono cameras whatever this says.
|
||||
- `--color-left NODE` (`HANDS_COLOR_LEFT`, `color_video0`) and `--color-crop subtract|none` (`HANDS_COLOR_CROP`, `subtract`): how the colour module's calibration maps onto the images. Not settled yet: `tools/check_color.py` on a recording with a lit, textured view tells.
|
||||
- `--skeleton 0|1` (`HANDS_SKELETON`, 1): the skeleton fit (below). With 0 no skeleton is published, and version 1's points and capsules are the filtered landmarks, as before 2026-10-08.
|
||||
- `--pinch-dip 0|1` (`HANDS_PINCH_DIP`, 0): the dip rule for pinches (below).
|
||||
- `--grip-begin R`, `--grip-end R`: the grip detector (below). Defaults 1.2 and 1.45.
|
||||
- `--gesture-log`: print what the pinch and grip detectors measure, 10 times a second: each hand's thumb-to-index distance (world and triangulated), its palm-down reading and its finger curl.
|
||||
|
||||
**Gestures** (`/run/user/UID/frametop-hands/gestures`, `include/fh_gestures.h`), for the pointer helper:
|
||||
|
||||
- A pinch: the thumb and index tips within 2 cm, ending past 3.5 cm (see "Pinch" below).
|
||||
- With `HANDS_PINCH_DIP=1`, also on a sharp dip that doesn't reach 2 cm. Quick taps with the hand raised often read 25-30 mm at contact and spring back, so they never click. The dip rule begins a pinch when, within 150 ms, the distance falls by 40% of its recent highest to under 3.2 cm, with the palm moving under 0.3 m/s and the thumb at least 4 cm from the middle fingertip (a hand closing into a grab brings both in). It ends once the distance is 6 mm over its lowest, and becomes an ordinary pinch (drags and all) if the tips get within 2 cm. It is off by default. In ft-handreplay on frame-hands' guided tap capture (2026-10-09, student v4), it caught 25 of 30 quick taps on a 2 Hz click instead of 10, and 74% of the teacher's clear taps instead of 62% at the same precision (0.91 against 0.93). The cost is that fingers stopped about 1 cm apart click too: 16 pinches in the near-miss segment instead of 5. Pointing and grabs in the hand recorder's takes gave 22 pinches, against 21 without it (`PinchParams::dip`; frame-hands notes/taps.md, Part D).
|
||||
- A grip, a closed hand: every finger's tip nearer the wrist than 1.2 times its knuckle is (from the model's 3D hand, so hand size doesn't matter), ending when they open past 1.45 on average. It begins only on a hand seen open within the last second (closing it is the gesture), with the palm at most 35 degrees below straight ahead and at least 15 cm in front of the eyes, and not with the thumb within 3 cm of the index tip (that's a pinch with the other fingers curled). A grip ends a pinch on the same hand, as lost. In the 2026-09-30 lit recording (no deliberate fists), the checks cut false grips from 14 to 6, all with the hands on the desk while looking down at it. The pointer helper ignores grips that begin more than `POINTER_GRIP_BELOW` (0.35 m) below the eyes, which it can tell and ft-hands can't.
|
||||
- A middle pinch (gestures version 3): the thumb on the middle fingertip, with the pinch's thresholds. It begins only with the thumb at least 1 cm nearer the middle tip than the index tip, the index extended (its tip at least 1.5 times as far from the wrist as its knuckle), and the hand tracked for 5 frames; a pinch and a middle pinch on one hand exclude each other, and a pinch begins only with the thumb nearer the index tip than the middle one. The pointer helper's hand ray turns it into a right click or scrolling. Closing the hand into a grab brought the middle tip to the thumb with the index curled: before the index and frames rules, the recorder's gestures take (PR #8, about 95 s of pinches and grabs, none of them middle pinches) gave 6 false middle pinches; after, 1. The 2026-09-30 lit recording (typing and 26 deliberate pinches) gave none, and the same 48 pinches as without the middle pinch.
|
||||
- `tools/watch_gestures.py --distance` shows them live; `ft-handreplay --timeline` logs them (middle pinches as `pinch2`) and each hand's finger curl.
|
||||
- `tools/watch_gestures.py --distance` shows both live; `ft-handreplay --timeline` logs them and each hand's finger curl.
|
||||
|
||||
The status line also says how often a hand was on each side (by where the wrist is), and why views and hands came and went: views lost (the landmark model stopped seeing the hand), handoff misses (a crop projected from the hand's 3D position found nothing), duplicates, splits (two views disagreed in 3D), and hands created, merged and forgotten.
|
||||
|
||||
@@ -161,63 +139,26 @@ The status line also says how often a hand was on each side (by where the wrist
|
||||
|
||||
- Each hand is tracked in its best two cameras, the way MediaPipe tracks: the landmark model runs on a crop placed from the previous landmarks, with no palm detection.
|
||||
- A hand seen in too few cameras is projected into the others through the calibration. Where it lands well inside a camera, that camera gets a crop to try. This is how a hand raised out of the side cameras reaches the upper ones.
|
||||
- The palm detector runs only while fewer than two hands are tracked, at most 5 times a second, on a few zoomed tiles per search. Tiles are picked in proportion to how likely hands are there. Each tile is turned so the expected shoulder-to-hand direction points up. There are 42 tiles and a search tries 2 (3 with no hand tracked), so even the likeliest tile comes round only every 10-11 searches: about 2 s at 5 searches a second, and 4-5 s live, where a search round takes about 50 ms and the tracker manages 2-3 a second.
|
||||
- **Re-acquiring a lost hand** (`--reacquire`, on by default since 2026-10-08). A hand that stops being seen is first held, not published: 0.6 s if it was tracked for 30 frames or more (`--hold`), else 0.3 s. Meanwhile its crops are retried where it was heading (its last position plus its velocity). From the first step it's missing and for 1.5 s after it's forgotten, the palm search aims up to 2 tiles at where it should be, in the cameras that see that spot best. The aimed tiles grow with the time since the loss, and they run every processed set while such a hand is missing, not only 5 times a second. The rotation keeps at least one tile per search. In replays this brought a lost hand back on the next processed set (median 0.05 s, against 0.12-0.25 s), and the slowest cases from seconds down to under 1 s (see frame-hands' `notes/reacquire.md`).
|
||||
- **Left and right** (`--pair-sides`, on by default since 2026-10-08). Each hand used to be called left or right by a running average of the landmark model's call, and both hands could get the same side for seconds. Now each hand also accumulates the model's calls as fading log-odds, and two hands seen together are one left and one right: the hand with more right-hand evidence is the right one, and which wrist is further right only tips a close call.
|
||||
- Frame sets are processed at 30 Hz while a hand moves faster than 0.25 m/s (or a pinch is down or closing), at 15 Hz otherwise, and at 5 Hz while no hand is in view. A reader that needs the full rate writes it to `/run/user/UID/frametop-hands/want-hz` and keeps the file fresh: ft-screens writes 30 there while a hand is near one of its panels, because at 15 Hz a poke's press would come up to 67 ms later. A file older than 1 s lapses.
|
||||
- The palm detector runs only while fewer than two hands are tracked, at most 5 times a second, on a few zoomed tiles per search. Tiles are picked in proportion to how likely hands are there. Each tile is turned so the expected shoulder-to-hand direction points up.
|
||||
- Frame sets are processed at 30 Hz while a hand moves faster than 0.25 m/s (or a pinch is down or closing), at 15 Hz otherwise, and at 5 Hz while no hand is in view.
|
||||
|
||||
### 3D
|
||||
|
||||
- **Two or more views:** each landmark is triangulated from the camera rays, weighted by the model's presence score. The median ray distance is reported as the residual.
|
||||
- **Pairing views across cameras.** The side cameras sit side by side, so two hands next to each other at the same height fall on the same epipolar lines, and rays to two different hands can nearly meet close to the cameras. That made phantom hands 12-15 cm in front of the eyes, which tore holes through the screens. Each step now scores every way of pairing the views in two cameras and keeps the best. A pair scores well when its rays meet, when each view's apparent size matches the triangulated distance, and when the model calls both the same hand. The size check uses a fixed prior: with the model's average hand, clean pairs measure 0.71-1.51 times the one-view distance, and mismatched pairs mostly far less.
|
||||
- **One view:** depth comes from the model's metric world landmarks, their spread across the palm against the angle it covers in the image, scaled by the user's hand size (learned while two views are available). That distance is off by 10-30% and wanders about 10% between frames, so a hand that drops to one camera keeps its last distance and drifts toward the one-view guess by 10% a frame.
|
||||
- **Smoothing.** The landmarks go through a One Euro filter: it smooths hard while the hand is still (tracking noise is several mm per frame) and hardly at all while it moves fast. The palm speed that sets the update rate is the filtered one; the raw speed read about 0.25 m/s from noise alone. The gestures (pinch point, palm-down reading) use these filtered landmarks. What's published is the skeleton's (below), unless `--skeleton 0`.
|
||||
- **Smoothing.** The published landmarks go through a One Euro filter: it smooths hard while the hand is still (tracking noise is several mm per frame) and hardly at all while it moves fast. The palm speed that sets the update rate is the filtered one; the raw speed read about 0.25 m/s from noise alone.
|
||||
- **Capsules.** Forearms follow the hand's own axis, and nothing within 12 cm in front of the eyes is published.
|
||||
|
||||
How good the depth is, measured from recordings (2026-09-30, `--depth` below): the two lower cameras see the hands about 77% of the time, a lower and an upper camera 7-12%, and one camera 12-15%. Depth is the noisy direction. With the lower pair, it jitters 4-6 times as much as sideways position (published: 3-7 mm against 1-2 mm). The one-camera guess is a median 2-6 cm off. When a camera drops out, drifting 10% a frame toward that guess is worse than keeping the last distance (after 0.5 s a median 23-30 mm off, against 11-12 mm).
|
||||
|
||||
## Skeleton
|
||||
|
||||
After the 3D step, ft-hands fits a skeleton with your own bone lengths to each hand (`track/skeleton.h`), so fingers keep their length and don't wobble. It's the base for a hand mesh, colliders and the hand ray. `--skeleton 0` (`HANDS_SKELETON=0`) turns it off.
|
||||
|
||||
- **Bone lengths, per side.** From clean stereo frames (residual under 4 mm, the model sure of the side), each of 30 numbers is learned: the thumb's CMC and the four knuckles' places in the palm, and the 15 finger bones. A landmark whose two rays miss each other by more than 4 mm, a finger bent more than 90 degrees, or a bone far off the average hand's proportions (under 0.7 or over 1.5 times them) gives no sample. Each settles on the median of its first 45 samples (3 s of a clean hand at 15 Hz), then moves at most 0.02 mm a sample. Until then the estimate is pulled toward a prior: the other side's lengths if it has more, else MediaPipe's average hand times `Hand::scale`. Nothing is saved: each start learns again.
|
||||
- **The fit, every frame.** The palm is rigid: its frame from the wrist and knuckles, then Gauss-Newton on its rotation and position. Then the thumb and each finger as a chain from its base: the thumb's CMC turns three ways (±50 degrees from a rest pose, ±35 twist), its MCP bends and spreads a little, its IP is a hinge; a finger's knuckle bends (-35 to 95 degrees), spreads (±25, the little finger ±30) and twists a little (±25), the two joints above it are hinges (-10 to 110, -20 to 90). A chain's base may sit up to 15 mm off the rigid palm, pulled back with a 4 mm prior: the measured knuckles wander far more than a palm bends (the little finger's by 3 cm across the palm). Each chain runs 4 Gauss-Newton steps (the limits as clamps) twice, from the measured bone directions and from last frame's pose, and keeps the better fit, so it doesn't hop between two poses that fit about as well.
|
||||
- **Weights.** Stereo: each landmark by its own ray miss (2 mm at least), and 3.5 times less sure along the line of sight. One view: 3 mm across, 15 mm along it.
|
||||
- **Which side.** The model's, but a tracked hand keeps its side while the model's running call is between 0.25 and 0.75. The mirror image is tried when the fit is worse than 15 mm rms, and taken after it fits at least twice as well 3 frames in a row; it then sticks while the hand is tracked (a finger can't bend backward, so a hand fitted as the wrong side fits badly once its fingers bend). Switching restarts the filter.
|
||||
- **Filtering the pose, not the points.** The palm centre and the palm's rotation go through One Euro filters (2 Hz still, plus 30 Hz per m/s and 10 Hz per rad/s); each finger's angles and base through one whose cutoff follows that finger's own speed (2 Hz plus 3 Hz per rad/s), so a pinch on a still palm doesn't lag the way the One Euro on the points does. A chain that fits its landmarks worse than 15 mm this frame moves its filter less (by the square of 15 mm over its error): that's mostly a hidden landmark misplaced.
|
||||
- **Elbow.** frame-hands' arm model (`notes/forearm.md`): a neck pivot fixed in the head frame, shoulders 18 cm to each side and 6 cm down, upper arm 1.5 times the forearm, the forearm 2.8 times the palm (wrist to middle knuckle), the elbow on the circle the two bones allow, turned 3 degrees from straight down, and never more than 75 degrees off the hand's own axis. ft-hands has no HMD pose, so "down" is the head frame's -y: published with `FH_FIT_ELBOW_HEAD_UP`. The model's other parameters (neck and shoulder offsets) are its starting values: the fitted ones are on the training PC.
|
||||
- **Published** in the hands file's version 2 (`include/fh_hands.h`): after version 1's struct, which is unchanged, an `fh_hand_fit_t` per hand with the 21 joints (position, a rough radius, and a quaternion in OpenXR's hand-joint axes), the bone lengths, the joint angles, the elbow, the pinch point (between the index and middle knuckles) and the thumb-index contact point, the thumb-index and thumb-middle distances, the fit's rms and how settled the calibration is, and the palm's velocity and angular velocity. Those two are in the head frame and include the head's own motion; for motion in the room, difference room-frame points across publishes as `screens/handcut.cpp` does. Version 1's points and capsules are the skeleton's too, so ft-screens' cutouts get it without a change. The hand recorder's reader takes version 1 or 2.
|
||||
- The gestures don't use it: pinches still measure the model's world landmarks, and the pinch point and palm-down reading still come from the filtered landmarks. `ft-handreplay --pinch-fitted` counts pinches on the fitted tips for comparison: they begin about as often as on the triangulated tips, more than on the world landmarks.
|
||||
|
||||
What it measured, `ft-handreplay` on three recordings and three takes of the 2026-10-03 hand-recorder session (frame-hands `notes/skeleton-fit.md` has every number):
|
||||
|
||||
| 2026-09-29/30 recordings, 30 Hz | dim (typing) | lit | bright |
|
||||
| --- | --- | --- | --- |
|
||||
| Bone length spread per track (median CV): measured / One Euro / skeleton | 0.41 / 0.37 / 0.05 | 0.24 / 0.22 / 0.03 | 0.39 / 0.35 / 0.01 |
|
||||
| Index tip jitter, hand and fingers still, mm (median; 90%): One Euro / skeleton | 4.5 / 3.9; 16.9 / 24.3 | 3.4 / 2.8; 11.0 / 14.7 | 4.2 / 3.2; 11.5 / 10.2 |
|
||||
| Thumb tip jitter, still, mm (median; 90%) | 3.0 / 2.7; 12.0 / 11.8 | 2.4 / 2.7; 7.8 / 12.0 | 3.0 / 2.2; 10.1 / 8.7 |
|
||||
| Index tip jitter, palm over 0.3 m/s, median mm | 14.0 / 11.2 | 11.0 / 8.6 | 17.7 / 14.7 |
|
||||
| Palm jitter, still, median mm | 1.8 / 1.8 | 1.9 / 1.8 | 1.9 / 1.8 |
|
||||
| Tips trailing, moving over 0.5 m/s, median mm | 6.5 / 5.7 | 5.1 / 4.9 | 5.7 / 4.9 |
|
||||
| Tips trailing, fingers moving on the palm, median mm | 7.8 / 3.5 | 5.9 / 4.7 | 4.6 / 3.5 |
|
||||
| Palm normal's turn per update, palm still, fingers moving, median degrees | 2.8 / 1.7 | 2.2 / 1.3 | 2.3 / 2.0 |
|
||||
| Fit rms against the measured landmarks, stereo, median mm | 10.4 | 9.0 | 7.6 |
|
||||
| Calibration settled after each side's first sample, s (left, right) | 20.5, 10.9 | 10.8, 3.5 | 15.6, 8.9 |
|
||||
|
||||
- **Against the auto-labels** (frame-hands' teacher, triangulated, smoothed and fitted: not ground truth; three takes of the PR #3 session, 10 Hz, stereo hands): mean landmark error measured 10.2, 12.1 and 7.0 mm, the fit before its filter 10.4, 12.2 and 6.5 mm (pose sweeps, gestures, desk work). The index tip: 8.1, 7.7 and 7.9 mm measured, 8.7, 8.7 and 5.5 mm fitted. The labels' ring and little knuckles sit far from MediaPipe's (their wrist to little knuckle is 110-140 mm against our 80-99), so part of every number is the two models' conventions.
|
||||
- **Cost:** 41-43 us a hand (median; 90% 61-70 us, 99% 91-164 us) on an A720 core, the fit alone replayed on CPU 3; 320-350 us on an A520 (CPU 0). ft-hands runs on CPUs 5-7, which are A720s and the X4.
|
||||
- **Pinches** don't change: the same counts in every replay, because they don't use the skeleton. On the fitted tips they'd begin more often (lit: 77 against 51; dim: 50 against 35), which is why they stay on the world landmarks.
|
||||
- **Worse:** the fingertips' jitter tail with the hand still in the dim and lit recordings (index 90% 24 against 17 mm, 15 against 11 mm; thumb in lit 12 against 8 mm). That's the price of each finger's own cutoff: a misplaced tip that the skeleton can reach (a few frames of a hidden finger) gets through where the palm-speed One Euro held it. The fitted landmarks also sit further from a centred average of the measured ones (moving fast: 11.7 against 9.3 mm in lit), mostly the fit's own residual: the labels don't say the fit is the worse of the two.
|
||||
- **Elbow:** the forearm comes out 231-247 mm, a median 21-27 degrees off the hand's own axis. With gravity's up from the takes' HMD poses instead of the head's, it moves a median 0.6-2.2 degrees (90%: 2.2-5.0): the head-up assumption costs little at a desk. No elbow labels are here to say how right it is.
|
||||
|
||||
## Pinch
|
||||
|
||||
ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gestures file. The layout, and how to read it without missing quick taps, is in `include/fh_gestures.h`.
|
||||
|
||||
- A pinch begins when the thumb and index tips come within `--pinch-begin` (default 0.020 m). It ends when they open past `--pinch-end` (0.035 m) for 2 processed frames in a row, or when the hand stays lost for 0.25 s (flagged lost).
|
||||
- A pinch held 0.3 s or longer is a drag, and holds on harder: it ends only once the tips are 5 cm apart (for the same 2 frames), or 0.12 s after they first went past 3.5 cm unless they closed back under 3 cm meanwhile. In the 2026-10-09 headset test drags ended with the tips 3.5-4.0 cm apart, a thumb hidden for a frame or fingers loosened a little mid-drag. On the lit recording's replayed distances, pinches held 0.3 s or more (typing among them) ended 33-134 ms later than before, 200 at most, and one of 25 ran into the next pinch's begin. A grip ends a pinch (as lost) only within 0.3 s of its begin, when closing the hand passes through a pinch; after that no grip begins on that hand, so curling the other fingers around a held pinch doesn't drop it (3 of 28 pinches in the same test). `tests/pinch_test.cpp` (`make check`).
|
||||
- The distance comes from MediaPipe's world landmarks: the model's own 3D hand pose, averaged over the hand's views, at the user's hand size. `--pinch-triangulated` uses the triangulated tips instead. On two recordings without deliberate pinches, the world landmarks came under 2 cm in 0.2-1% of frames, against 3.3-4.5% for the triangulated tips. In the dim recording, typing still gave 2 pinches a minute (see the next point).
|
||||
- `--pinch-palm-down MAX` holds back pinches begun with the palm facing down (MAX is the palm normal's share of the head's up axis). The default, 1, turns it off. A close held back that way has to open again before a pinch can begin. Typing curls the thumb onto the index: in the lit recording of 2026-09-30, typing on a keyboard in the lap began 23 pinches in about 2 minutes, all with the palm facing down (0.69-1.00), while the 26 deliberate ones read 0.00-0.50. But in the headset, deliberate pinches with the hand raised in front read 0.90-0.99 too, so the limit is off. Typing is caught by the pointer helper instead: the input relay tells it when you press a key, and the hands on the keys are held back (`POINTER_PINCH_TYPING` and the settings after it, below).
|
||||
- `--pinch-palm-down MAX` holds back pinches begun with the palm facing down (MAX is the palm normal's share of the head's up axis). The default, 1, turns it off. A close held back that way has to open again before a pinch can begin. Typing curls the thumb onto the index: in the lit recording of 2026-09-30, typing on a keyboard in the lap began 23 pinches in about 2 minutes, all with the palm facing down (0.69-1.00), while the 26 deliberate ones read 0.00-0.50. But in the headset, deliberate pinches with the hand raised in front read 0.90-0.99 too, so the limit is off. Typing is caught by the pointer helper instead: the input relay tells it when you press a key, and no pinch begins within `POINTER_PINCH_TYPING` of one.
|
||||
- A hand a pinch is down on stays with that side until the pinch ends. The left/right call is a running average of the model's, and when it flipped mid-pinch, the other side took the same hand and both sides pinched at once.
|
||||
- The pinch point is between the index and middle knuckles, which hold still while the fingers open and close. The tips' midpoint moved 1-2 cm as a pinch opened, which dragged every release off its press. A drag is the pinch point now, minus where it was when the pinch began, both turned into the room with the HMD pose at their capture times.
|
||||
- `tools/watch_gestures.py` prints begins, ends and drag offsets live, and `--distance` prints each hand's distance.
|
||||
@@ -226,45 +167,26 @@ ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gest
|
||||
|
||||
With `POINTER_HANDS=1`, the pointer helper (`pointer/helper/ft-pointer.cpp`) reads the gestures file every frame. It's off by default.
|
||||
|
||||
**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless `POINTER_GAZE_HAND_RAY=1`. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`).
|
||||
|
||||
- **Pinch to click, in gaze mode.** Your eyes aim and your hands click, Vision Pro style, and the hand ray stays off (`POINTER_GAZE_HAND_RAY`). A pinch locks the pointer where you were looking just before the fingers began to close (75 ms before ft-hands saw the pinch, since the eyes often move on as the hand closes), and a quick tap clicks there. Pinch, hold and move the hand to nudge the pointer: nothing moves until the hand has moved 4 degrees (about 3 cm at arm's length), then the pointer follows at 0.3 times the hand's movement. Letting go clicks where the pointer is, and the correction is a lesson for the gaze tracker, as with the mouse. Held still for `POINTER_GAZE_HOLD` (0.5 s) instead, the pinch is a real press at that point, and the hand drags at `POINTER_GRIP_GAIN`. A middle pinch is a right click the same way. Hold a pinch with one hand and pinch with the other: both buttons are down, left for the first hand and right for the second, each until its own pinch opens (a first pinch not yet pressed becomes a press at once, where the pointer is), as when you hold both of a mouse's buttons. The rules and timings are in `pointer/helper/gazehand.h`.
|
||||
- **Pinch to click.** In gaze mode a pinch works like the mouse's press: the pointer stops where the gaze put it, and the click comes when the pinch opens, where the pointer is then. A quick tap clicks where you looked. Held, the pinching hand moves the pointer to correct the gaze, and the correction is a lesson for the gaze tracker, as with the mouse.
|
||||
- **Without gaze mode,** a pinch is a real press, like the mouse's button: pressed when it closes, released when it opens, and while it's held the hand drags the pointer. A tap is still a click where the pointer is.
|
||||
- **Grip to drag.** Closing the hand presses where the pointer is, the hand moves the pointer, and opening the hand releases. So a title bar moves its window, a panel's grab bar carries the panel, and text gets selected. A panel the hand ray carries by its bar (or a floating window by its title bar) stays upright and facing you wherever the ray takes it (2026-10-13: the ray can't tilt it, so the wrist's turn shouldn't either); a controller's laser still carries it rigidly.
|
||||
- **Grip to drag.** Closing the hand presses where the pointer is, the hand moves the pointer, and opening the hand releases. So a title bar moves its window, a panel's grab bar carries the panel, and text gets selected.
|
||||
- A pinch ended by losing the hand, or by a grip taking over, doesn't click.
|
||||
- The hand's movement is taken in the room, from where the eye was when the gesture began, so turning your head doesn't move the pointer. The first gesture while the pointer is off only wakes it. Gestures are ignored in a VR game (unless the dashboard is up), with the headset off, and while the mouse's button is held.
|
||||
|
||||
Settings in `~/.config/frametop.conf`:
|
||||
|
||||
- `POINTER_HANDS` (0): 1 turns pinches and grips on.
|
||||
- `POINTER_GAZE_PINCH` (1): in gaze mode, 0 turns clicks at the gaze off (pinches and grips with no ray do nothing), leaving the ray when gaze is off and direct touch. Frametop Input Settings' Hands page has the three hand inputs as switches of their own (`hands/handsmode.py`, `methods_of`).
|
||||
- `POINTER_PINCH_GAIN` (0.5): without gaze mode, a held pinch moves the pointer this many times the hand's angle, seen from the eye. Under 1 gives precision.
|
||||
- `POINTER_PINCH_DEADZONE` (1.5): without gaze mode, how many degrees the pinching hand moves before the pointer does, so a tap's jitter and the pinch point shifting as the fingers close don't move it.
|
||||
- `POINTER_GAZE_HAND_RAY` (0): in gaze mode, 1 keeps the hand ray beside the gaze (a raised, pointing hand's pinches are then ray pinches).
|
||||
- `POINTER_GAZE_PINCH_DEADZONE` (4): in gaze mode, how many degrees a held pinch moves before it nudges the pointer. A tap's jitter and a hand drifting while you look stay inside it.
|
||||
- `POINTER_GAZE_PINCH_GAIN` (0.3): in gaze mode, past the dead zone the pointer moves this many times the hand's angle. Low, so a nudge is a fine correction.
|
||||
- `POINTER_PINCH_GAIN` (0.5): a held pinch moves the pointer this many times the hand's angle, seen from the eye. Under 1 gives precision.
|
||||
- `POINTER_PINCH_DEADZONE` (1.5): how many degrees the pinching hand moves before the pointer does, so a tap's jitter and the pinch point shifting as the fingers close don't move it.
|
||||
- `POINTER_GRIP_GAIN` (1): a grip moves the pointer this many times the hand's angle.
|
||||
- `POINTER_GRIP_BELOW` (0.35): grips that begin more than this many metres below the eyes are ignored, because hands resting on a desk curl like a loose fist. Pinches have no such limit: deliberate ones sat 0.35-0.45 m below the eyes with the elbow resting.
|
||||
- `POINTER_PINCH_TYPING` (2), `POINTER_TYPING_RAISE` (0.08 m), `POINTER_TYPING_BELOW` (0.2 m), `POINTER_TYPING_POSTURE` (0.3 s), `POINTER_TYPING_RECENT` (600 s): typing on a real keyboard. For `POINTER_PINCH_TYPING` seconds after a key, no hand pinches, grips, pokes or shows a ray, except one lifted off the keys; a hand that was on the keys stays held back through any pause until it rises `POINTER_TYPING_RAISE` above where it typed (or turns its palm up); both hands down on the keys together count as typing before the first key. `pointer/helper/typing.h` and `docs/hands-typing.md` have the rules and the measurements.
|
||||
- `POINTER_HAND_RAY` (1), `POINTER_RAY_DEADZONE` (2.5 degrees the ray turns during a pinch before the press drags), `POINTER_RAY_PUSH` (1), `POINTER_RAY_SCROLL` (1; -1 turns scrolling round), `POINTER_RAY_RELEASE` (2 s after the ray goes before a pointer it woke lets go), `POINTER_RAY_GRACE` (0.5 s a hand keeps the laser after its ray drops; the other hand's ray takes it after that, or at once by pinching or raising): the hand ray.
|
||||
- The ray's smoothing (`include/fh_smooth.h`, through `fh_ray.h`): One Euro filters run in continuous time, so the pointer's pose moves at the helper's rate (about 100 Hz) instead of stepping at the tracker's 15-30 Hz; the direction is filtered on the sphere. `POINTER_RAY_SMOOTH_MINCUTOFF` (0.8 Hz with the hand still), `POINTER_RAY_SMOOTH_BETA` (0.06 Hz more per degree a second it turns), `POINTER_RAY_SMOOTH_HOLD` (0.5 Hz still while a pinch is held: a drag's precision). The reach, push and pull's input, has a heavier one of its own. `POINTER_RAY_STEADY` (1): while a pinch is held (and 0.25 s after), the ray pivots on where the shoulder was at the press, then eases back to the body model over 0.5 s; the model's shoulder follows the head, which made looking around move what was dragged. `POINTER_RAY_TAP` (0.3 s) and `POINTER_RAY_TAP_DEADZONE` (6 degrees): a pinch's first 0.3 s takes 6 degrees of turn before it drags, so a quick tap whose hand swung stays a click; after that `POINTER_RAY_DEADZONE` applies. When the press starts dragging the pointer carries on from where it was, no jump. On the probe-moves replay (`pointer/tools/rayeval` on `frame-hands/dumps/probe-moves-v4.dump`, read at 90 Hz) the pose's second difference per tick went from 0.90-0.96 degrees rms (p90 1.4) to 0.24-0.26 (p90 0.24) at about the same lag (110 ms behind the capture, against 105).
|
||||
- The ray's body and hand models (`include/fh_ray.h`; ideas and formulas from StereoKit, MIT): `POINTER_RAY_TORSO_WRISTS` (1): the shoulders face between the head (counted twice) and the wrists seen, so looking aside while pointing swings the ray less. `POINTER_RAY_HAND_WEIGHT` (0; 0..1): how far the ray turns from the shoulder's line toward the hand's own direction (`POINTER_RAY_HAND_PITCH` -30 and `POINTER_RAY_HAND_YAW` 8 degrees off the palm's axes). `POINTER_RAY_ANCHOR` (0): the knuckle point pinned to the palm while a pinch is held. The last two measured worse on replays (`pointer/tools/rayeval`; numbers in `fh_ray.h`), so they're off.
|
||||
|
||||
## Flags
|
||||
|
||||
When tracking goes wrong in real use, flag it: `ft-handsctl flag [note]`, or by voice with frame-voice's voice commands (its `commands` setting: "flag that" runs `ft-handsctl flag` with the words after it as the note, e.g. "flag that, fingers crossed"). The offline tools (frame-hands' `train/flags`) relabel what was saved with the teacher, find what went wrong and in which situations, and make guided capture sessions for the weakest ones.
|
||||
|
||||
ft-hands keeps a rolling buffer in RAM (`track/flagbuf.h`, `HANDS_FLAG_BUFFER=1`): per processed set, a mono crop around each view the tracker used (its landmark crop's centre, 1.6 times its size, axis aligned, at most `HANDS_FLAG_CROP` px a side), with where it is in the image, so labels made on it lift back to the image and triangulate; a lost view is still cropped where it was for a second, so the teacher can say whether the hand was there; and its hands, gestures and each view's landmarks. It keeps `HANDS_FLAG_SECONDS` and at most `HANDS_FLAG_MB` of crops; nothing is written until a flag. Cost, measured (2026-10-09, a replayed ring with two hands in view): about 80 crops a second at 20 sets a second, 56 MB of crops for 14 s at 224 px (ft-hands' peak RSS 101 -> 159 MB), and 0.1 ms a crop on a tracking core (A720), so about 1% of a core; ft-hands' CPU time was the same with and without it. At 30 sets a second with four views the 80 MB cap can trim the oldest seconds.
|
||||
|
||||
`ft-handsctl flag` makes `HANDS_FLAG_DIR/flag-<time>/` and asks ft-hands for the buffer through `run_dir/flag`; ft-hands keeps going for 2 s more, then a thread of its own (off the tracking cores, lowest priority) writes `manifest.json` (written last), `sets.jsonl`, `crops.jsonl` and `crops.bin`. Once ft-hands has taken the request, the drawn hands pulse cyan for half a second (ft-handview's `flash`; `HANDS_FLAG_FLASH=0` turns it off; `HANDS_FLAG_TONE=1` also plays a short tone). It returns then, and in the background saves one frame of your view from `/dev/video99` as `view.jpg` (`HANDS_FLAG_VIEW=0` skips it), the head's poses over the window from ft-handview (`poses`) as `hmd.jsonl`, and `ctl.json`. ft-hands logs each flag ("flag:", "flag written:"), and the status line has the buffer's size and cost. A flag while one is being taken adds its note to it. The format is in frame-hands' `notes/flag-failures.md`. The crops show your hands and some of the room: keep flags with your recordings.
|
||||
|
||||
`ft-handreplay REC --flag-at S1,S2 --flag-out DIR` (or `--flag-every S`) makes the same flags from a recording, with the same buffer, for testing the offline tools before real flags exist; `--flag-crop PX`, `--flag-scale F`, `--flag-mb MB` and `--flag-seconds S` change the buffer.
|
||||
- `POINTER_PINCH_TYPING` (1): no pinch begins within this many seconds of a key press, because typing touches thumb to index.
|
||||
|
||||
## Recordings
|
||||
|
||||
`hands/build.sh --tools` also builds the offline tools.
|
||||
|
||||
`ft-handreplay DIR` runs a recording through the tracker with the live scheduling and reports how well it kept the hands: hands per set, left and right coverage, track lengths, pinches, jitter, and the same reasons as the status line. `--publish-dump FILE` also writes what ft-hands would have published after each processed set (the hands and gestures files' bytes, and the set's dequeue time), for ft-screens' offline tests; a hand-recorder take's `poses.jsonl` and `prompts.jsonl` copied next to the dump as `NAME.poses.jsonl` and `NAME.prompts.jsonl` give them the head's poses and the touch take's targets (`scripts/test-handinput.sh`, `scripts/bench-matte.sh`). To match the live tracker, replay with `--reacquire 1.5 --reacquire-fast --pair-sides --hold 0.6` and the models in use.
|
||||
`ft-handreplay DIR` runs a recording through the tracker with the live scheduling and reports how well it kept the hands: hands per set, left and right coverage, track lengths, pinches, jitter, and the same reasons as the status line.
|
||||
|
||||
```
|
||||
hands/build/ft-handreplay ~/.local/share/frametop/hands/rec-20260929-120000 --cost --oracle 10 --timeline /tmp/tl.txt
|
||||
@@ -274,16 +196,9 @@ hands/build/ft-handreplay ~/.local/share/frametop/hands/rec-20260929-120000 --co
|
||||
- `--oracle N`: every N-th set, also search every tile of every camera, and report how often the tracker had the hands that full search could find.
|
||||
- `--slow F`: live, the tracker skips sets that arrive while it's busy. Replay counts each step's time times F as busy (default 1; the headset is busier live).
|
||||
- `--timeline FILE`: a line per processed set and hand, with pinch events and distances.
|
||||
- `--ray FILE`: a line per processed set and hand with what the pointer helper's hand ray reads (time, side, stereo, pinch and grip state, the published points, the wrist's frame, the elbow), for `pointer/tools/rayeval` with the same session's `ft-handtest` probe log (head poses).
|
||||
- `--cams mono|color|all`: which cameras to track with (default `mono`). `color` tracks with the Arcturus pair alone, for comparing it with the IR cameras on the same recording. It needs a recording made with `ft-camd --with-color`. `--color-left NODE` (`color_video0` or `color_video3`) and `--color-crop subtract|none` say how the module's calibration maps onto the images; `tools/check_color.py` finds out.
|
||||
- The skeleton (see "Skeleton"): the report ends with its measures for the measured, One Euro filtered and fitted landmarks (and the fit before its filter): its time per hand, also on its own pinned to a little and a big core; its rms; each side's calibration and when it settled; how much bone lengths spread per track; jitter at rest and moving for the palm and the thumb and index tips; how far the tips trail during fast motion (against a centred average of the measured tips); how steady the palm's normal is; and the elbow. `--no-skeleton` replays without it; `--skel NAME=VALUE` changes a setting (`skel::Params`); `--labels FILE` compares with frame-hands' auto-labels (`labels.jsonl`), `--skeleton-dump FILE` writes the matched hands; `--hmd-poses FILE` (a take's `poses.jsonl`) compares the elbow with gravity's up against the head's; `--pinch-fitted` counts pinches on the fitted tips too.
|
||||
- A hand-recorder take is a recording too: its `sets.bin` is followed by `sets-2.bin` and later parts, which are read in order. It has no `sides.json`: whether its side cameras need exchanging is `take.json`'s `names_swapped` against `session.json`'s `sides` (`rec/sides.py`), so give `--sides 0` or `1` (`--sides auto` checks from the hands).
|
||||
- `--depth FILE`: a line per hand per processed set for `tools/depth_report.py`, which measures the depth without ground truth: how the hands were seen, the noise along the line of sight against across it, each camera's one-view distance against the triangulated one, and what a camera dropping out would do.
|
||||
- The pinch, contrast and presence options are ft-hands'. `--reacquire S` (off unless given; `--reacquire-fast` searches every set as ft-hands does, `--reacquire-tiles N`, `--reacquire-clear M`), `--pair-sides`, `--hold S`, `--near F` and `--search-budget N` are the tracker's re-acquire settings, off or at their old values by default so old replays repeat.
|
||||
- `--cost-hand MS`, `--cost-palm MS`: with `--cost`, what a round of landmark and palm calls costs. The defaults (10, 18) are the headset's idle costs. The 2026-10-08 live probe (CPUs 0,1,4 beside SteamVR, 3 threads) measured about 14 and 52, and the live tracker loses hands far more at those costs.
|
||||
- `--drop-every S`: the re-acquire test. Every S s, while two hands are tracked, one of them is made invisible for 0.35 s (just past the 0.3 s the tracker holds a lost hand), and the report gives how long it then took to have two hands again: the time to find a hand that never left the view.
|
||||
- `--flag-at S1,S2`, `--flag-every S`, `--flag-out DIR`: flags as `ft-handsctl flag` makes them live (see "Flags"), at those times into the recording.
|
||||
- With `--timeline`, the tracker also writes why views and hands came and went (`ev` lines: lost, miss, split, search and its tiles, palms kept or rejected, new, forget, pair-sides), and `--oracle` writes what the full search found. `tools/gap_report.py TIMELINE --list` turns that into every gap of 0.3 s or more per side, with its cause. `tools/replay_gaps.sh OUT MODELS "ARGS" REC...` runs several replays at once (on the 7i through frame-job) and reports them. For pinches, `--tips FILE` logs each hand's thumb, index and middle tips and the pinch detector's distance measures per processed set, with its pinch events; `--rate HZ` fixes the processing rate (15 or 30, against the live rule) and `--want-hz HZ` plays a reader asking for it; `tools/replay_tips.sh OUT NAME=MODELS "ARGS" REC...` runs them (frame-hands' `train/taps/tapeval.py` scores them against auto-labels).
|
||||
- The pinch, contrast and presence options are ft-hands'.
|
||||
|
||||
`ft-ringplay DIR --ring PATH [--from S] [--to S] [--loop]` publishes a recording into a ring file in real time, as ft-camd would, so `ft-hands --ring PATH --no-publish` runs the same frames run after run. It needs no privileges, and it skips the dark frames.
|
||||
|
||||
@@ -299,94 +214,7 @@ Python, with NumPy and OpenCV. `setup/dev-container.sh` doesn't install them, be
|
||||
- `tools/cut_sets.py REC OUT [--sets N | --at I,J,...]`: copies a few frame sets (by default 8, spread evenly) out of a recording into a small one, to look at or check elsewhere without moving gigabytes. Plain Python, so it also runs on the Frame's host.
|
||||
- `tools/convert_models.py`: how `models/ncnn` was made from the OpenCV Zoo ONNX ports of MediaPipe's models (see `models/NOTICE`).
|
||||
|
||||
To try the hand cutouts without restarting the desktop, `screens/build/ft-handtest [--distance m] [--width m] [--seconds s] [--matte circle|mesh|capsules] [--margin deg]` (built by `screens/build.sh`, run in the dev container, with hand tracking on) shows a test panel of its own, a light grid 1 m wide and 0.8 m ahead by default, and cuts your hands out of it the way ft-screens cuts them out of the screens. Every 2 s it prints the composite's cost.
|
||||
|
||||
## Hand cutouts and near input
|
||||
|
||||
ft-screens reads the hands file (version 2's skeleton when it's there, else version 1's points) and the gestures file once per tick (`screens/handcut.cpp`). It cuts only with Room View on: with it off, a hole would show the void or the game, not your hand (see "Rendered hands"). `cutouts state` ends with `room view on|off (why)`; `cutouts roomview auto|on|off` forces it.
|
||||
|
||||
**Off by default since 2026-10-09** (`HANDS_CUTOUTS=0`): ft-handview draws the hands over the panels instead (see "Rendered hands"). `HANDS_CUTOUTS=1`, or `ft-handsctl cutouts on` for the session, brings the holes back.
|
||||
|
||||
**The matte** (`HANDS_MATTE=circle`, the default since 2026-10-09) is a round hole per hand: a disk facing the eye, centred on the middle of what the head sees of the hand (the box around its joints, each with its half-width, the wrist's 2.5 cm), wide enough to reach the hand's furthest edge from there, plus the margin. Each eye sees the disk on the screen as an ellipse (a circle on a screen square to it), drawn as the stadium around it with the capsules' soft edge (15% of its radius). It hides more of the screen than the hand needs, and with it the hand's tracking error and the few degrees Room View's picture of the hand sits off it, until the tracking is good enough for a hand-shaped hole. The forearm isn't in it. Building and projecting it costs 4 us median on the desk-work replay (the mesh 26 us), the composite the same as the mesh's or less (`scripts/bench-matte.sh` on probe-moves: 2 pieces per eye against 35).
|
||||
|
||||
`HANDS_MATTE=mesh` is hand-shaped: a tapered capsule per finger bone, with real fingers' half-widths (10.5 mm at the index's knuckle down to 7.5 mm at its tip, scaled by the hand's palm length); the palm as a 2.4 cm slab through the wrist and the knuckles, widened to the hand's edges; the thumb's web; and an 18 cm stub of the forearm toward the skeleton's elbow (along the hand's axis without a skeleton). The palm, web and forearm are convex pieces, each drawn as its outline grown by the margin with a soft 2 px edge. `HANDS_MATTE=capsules` draws ft-hands' capsules instead, as before, for comparison (`cutouts matte circle|mesh|capsules` switches it live).
|
||||
|
||||
Its margin is an angle seen from the eyes, so it covers the same share of the tracking error at any distance: `HANDS_MATTE_MARGIN` (1.5 degrees) with the hand still, plus `HANDS_MATTE_MOTION` (25 ms) of the hand's motion, at most `HANDS_MATTE_MOTION_MAX` (1.5 degrees) more. The cutout placement probes of 2026-10-08 (frame-hands `captures/probe-2026100*/video_analysis`) measured the cutout points against the hand Room View shows: 1.8-2.4 degrees median still (p90 4.1-4.7), 2.3-3.1 moving at 0.1-0.8 m/s, 4-6 above 0.8 m/s. Part of that is the video landmark model's own error, and an edge needs only the error across it (about 70% of a point's). A margin covering the p90 (4.5 degrees, 3 cm at 40 cm) would turn each finger into a 7 cm blob; 1.5 covers about the median. `cutouts margin DEG [MS [MAX]]` changes it live, and so does the Hand cutout margin slider on the Hands page of Frametop Input Settings (0-3 degrees in quarter steps; it sends the still margin only and writes `HANDS_MATTE_MARGIN`). The prediction lead stays 36 ms (`HANDS_MATTE_LEAD`, `cutouts lead`): the probes' best lead for fast hands was 6-14 ms, but 36 costs only 0.1-0.2 degrees.
|
||||
|
||||
The probes also saw the hand's picture off by an amount that depended on its distance, but not the same way in any two runs, so the correction is a hook, off: `HANDS_MATTE_SHIFT="0.35:0.5,-0.3 0.6:-1,0"` would turn the matte's hands by those yaw and pitch angles (degrees, right and up positive) at those distances from the head, in between linearly (`cutouts shift ...|off` live).
|
||||
|
||||
Cost, offline (`scripts/bench-matte.sh` on the probe-moves and lit replays, a 3440x1935 screen 0.6 m ahead, GPU time to finish each buffer on an otherwise idle GPU): the mesh 3.9-4.1 ms per full buffer against 3.8-3.9 for the capsules, and 3.4-3.6 against 3.0-3.4 when only the hands' boxes are redrawn (the margin is wider than the capsules' 4 mm). CPU 0.2 ms per composite either way; building and projecting the matte 14-24 us median.
|
||||
|
||||
**Near input** (`screens/handinput.cpp`, frame-hands `notes/hand-interaction.md` sections 3, 5, 16, 17, 18 and 19): a bare hand whose index tip, thumb-index contact, knuckles or palm is over a panel (or within the grab band outside it) and within `HANDS_NEAR_REACH` (0.075 m, about 3 inches; 0.30 on 2026-10-10 before the live test, 0.15 before that) in front of it touches it: ft-screens tells the pointer helper so ("near ... 1"), which then hides that hand's ray and leaves its pinches and grips to ft-screens. Beyond it a pinch is the ray's or the gaze's. A hand stays near until it's 2.5 cm farther out, then 200 ms on, so the ray doesn't flicker at the edge; in that margin the ray stays hidden ("near ... 2") but the hand's pinches are the helper's again. The ray shows again once its pointing pose has held 120 ms, about 0.35 s after the fingertip passes 10 cm. A hand resting low by the body (the poke's resting rule below: typing, the lap) touches only within 4 cm, so a low panel within reach doesn't take its gaze pinches. While a hand is within 0.30 m of a panel, ft-screens asks ft-hands for its full rate (want-hz), so a tap from outside the zone is seen at 30 Hz. Near a panel:
|
||||
- **Poke to click:** the index fingertip presses when it comes at the panel's face (user, 2026-10-08): past the surface for two hands frames in a row, or for one when it came in decisively (1.7 cm in on average over the last 0.3 s, on its way in on the frame before too, at 0.1 m/s or more, and under 2.5 cm in coming nearer on each of the last 3 frames: `HANDS_POKE_TRAVEL`, 0 = always two frames; 2.5 cm until the second live test of 2026-10-10, which refused taps that came 1.7 and 2.0 cm). A quick tap at 30 Hz is past the surface for one or two frames, and one the tracker has a centimetre short for none: two frames in a row lost most shallow taps in the live test of 2026-10-10. It also needs: seen by two cameras, moving into the surface at 4 cm/s or more when it crossed (over the last 100 ms), the finger (knuckle to tip) within `HANDS_POKE_ANGLE` (50) degrees of the panel's normal, the index straight (1.5) and either no other fingertip more than 6 mm deeper than its tip or the hand pointing (the index straighter than each other finger by 0.2; until the second live test the index had to be 5 mm nearer than the others, and the tracker often reads a pointing hand's middle finger nearly as straight, its tip within -7 to +4 mm of the index tip's depth: 14 of its 39 refusals), not past the surface for more than 5 frames or 4 cm (reaching through), and not from a hand resting low by the body (palm more than `HANDS_POKE_REST_BELOW`, 0.30 m, under the eyes and within `HANDS_POKE_REST_NEAR`, 0.32 m, of the head across the floor; it stays resting until 5 cm higher or further out). So typing under a low panel doesn't poke it, and reaching out to one does. It lets go 1.5 cm out, or 1 cm back from the deepest point (so a tracker that has the finger behind the panel still lets go). The click stays where it was pressed until the fingertip slides 1.5 cm; then it drags. No new press for 150 ms after a release, nor until the tip has been 1 cm out (a noisy pull-back pressed twice); a finger 8 cm through the panel lets go and needs to come back out first. Within 5 cm the pointer follows the fingertip. A tap up to 1.5 cm past the content's edge presses at the edge (the desktop's taskbar is on the bottom edge). The curve, reset, dock and close buttons take pokes too (below: Buttons). The index crossing the plane farther off the content (a hand reaching for the bar or a corner) isn't a poke: ft-screens logs it once, as "not a poke", with where it crossed. On synthetic taps with 3-6 mm of tracking noise, 50-56 of 60 shallow taps (0.5 cm past the surface) press, against 34-36 with the two-frame rule; a finger held still 1.5-2.5 cm out presses no more often than before (`handinput-test`, TestPokeNoisy).
|
||||
- **The hand owns the pointer:** while a hand holds the button, SteamVR's laser moves (the pointer helper's head or gaze pointer, a controller) go nowhere, and on a panel whose pointer a fingertip moved in the last 250 ms neither do they. Before, the helper's pointer a few cm off the fingertip pulled the remote host's pointer away between the poke's press and release: a drag, not a click.
|
||||
- **Why a poke didn't press:** ft-screens' log has a line for each press and release ("hands: right poke pressed screen 104 at 1932,869 (-0.6 cm, in at 0.25 m/s from 3.1 cm out, 1 frame(s) past, the finger 12 deg off)"), and for each time the fingertip crossed a panel's surface without a press, the rules that held it back ("hands: right poke on screen 104 didn't press (0.4 cm deep at most): past the surface for 1 frame(s); ..."). `handinput state` counts them.
|
||||
- **Pinch to move or resize:** a pinch where the thumb and index meet (not the gestures file's point, the knuckles, 5-8 cm behind them) within the border band takes the panel: 6 cm outside its edges to 2.5 cm inside (`HANDS_GRAB_BAND`), anywhere in the touch zone's depth (in front of it, or a little behind), or up to 3 cm beyond it when the touch zone has the hand (the third live test: "the fingers +9.0 cm from it, out of the touch zone: nothing", the pointer helper having left the pinch to ft-screens). By any corner (6 cm) it resizes, and the opposite corner stays put: a screen or remote display along its diagonal (its aspect is its resolution's), a floating window in pixels from that corner (every corner of a window since 2026-10-10, its title bar's ends too). Anywhere else on the border, the bar, a floating window's title bar, it moves. Moved, it follows the knuckles 1:1 (closer and farther too) and keeps facing you. A soft white glow shows where a pinch would take it (below: Glows at corners, edges and buttons); with `HANDS_EDGE_GLOW=0` the resize handle goes to the corner a pinch is near instead, brighter the nearer, lit blue while resizing. With `HANDS_CUTOUTS=1` a glow on the panel also shows what a pinch would do there: amber at the corner (resize), blue on the border (move), green for a card pinch; a dot while it holds. (The glows are drawn into the cutout buffer, so with the cutouts off, the default since the drawn hands, they don't show.)
|
||||
- **Resize from any corner, by hand, gaze or laser** (2026-10-10, as on a Vision Pro): each panel has one resize handle, and it goes to the corner something aims at: a laser's ray (a controller's, or the pointer helper's, which the gaze and the hand ray drive) landing within a fifth of the panel's height (5-20 cm) of a corner (or a hand's pinch at one, with `HANDS_EDGE_GLOW=0`; hands have their own glow since the fourth live test). Since 2026-10-13 (user: "remove the radio icon thing on corners for resizing and replace them with the glow that is used for grabbing with hand tracking") the handle draws nothing: the same soft white L round the corner as a hand's pinch gets shows it, brighter as the aim comes to the corner, brightest while resizing (the handle is still there, just outside the corner, to take the laser's press); with no aim for 0.4 s it goes back to the bottom right. Press it with the laser, or look at it and pinch (a gaze pinch on a resize handle or a grab bar is a press after 0.12 s held still, `POINTER_GAZE_CONTROL_HOLD`, not the 0.5 s elsewhere), and drag: the corner follows, the opposite one stays put.
|
||||
- **Pinch on the content to click** (`HANDS_PINCH_CLICK`, 2026-10-10): a pinch within the touch zone, not in the band, presses where the fingertip's pointer was (if it was there in the last half second, within 4 cm of the fingers), else where the fingers meet; it's held while the pinch is, and moving the pinched hand 1.5 cm drags. Not from a hand resting low by the body or seen by one camera.
|
||||
- **Card pinch** (`HANDS_CARD_PINCH`, 2026-10-10: "pinch from underneath the display and manipulate it like a card"): a pinch closing on the panel's edge with the panel between the thumb and the index (each tip at least 3 mm from its plane on some frame in the last 0.3 s, the line between them within 60 degrees of its normal, the hand along the panel: at least 45 degrees off its normal, the fingers within 5 cm of its border), and the index and middle fingers together against the thumb (thumb to middle tip within 4.5 cm, index to middle tip within 3.5 cm) or all four fingertips bunched at it (each within 5 cm: the "pinched fingers" hand), holds it like a card (since the second live test; the index alone was enough before, and is now the border's move or a corner's resize; the middle finger joining within 0.3 s of the pinch makes the move a card from there): it goes rigidly with the hand, its place and its turn about all three axes, 1:1, smoothed, past 1 cm and 3 degrees; no facing you, no level snap. An edge pinch from the front or the side has both tips on one side of the plane, or the hand pointing into it: that's the border's move.
|
||||
- **Fist grab** (`HANDS_FIST_GRAB`, 2026-10-10: "use the fist to expand and rotate the display, as well as change its angle"): a grip (ft-hands' closed hand) made with the palm, knuckles or curled middle finger within the touch zone over a panel (or within 6 cm of its edge) takes it; so does a pinch by a closed hand (the index 1.15 extended or less, the other three 1.05 or less on average: ft-hands calls a fist with the thumb on the index a pinch, and in the third live test two fists clicked the content instead of grabbing it). It's held while either the grip or that pinch is, until the hand opens: the fist carries it 1:1 and turning the fist turns it about all three axes, as the wrist turns a pinched one (below); open the hand to let go. Two fists (or a fist and a pinch) on one panel stretch it and turn it, as two pinches do. Since the fourth live test (2026-10-10, "I can't grab using fists"): a fist is at a panel with its palm, knuckles or curled middle finger up to `HANDS_FIST_REACH` (0.11 m) in front of it or 10 cm behind it (a live fist went 8.5 cm through and wasn't "at it"), and a hand curled that far (the three fingers under 1.2) is ft-screens' that far out, so the pointer helper leaves its grip; a pinch on the content by a hand that's closing (the three fingers under 1.4 and down by 0.1 in the last 150 ms: a fist's thumb meets the index first) waits up to 150 ms to become a fist (then a fist grab) before it clicks, where it began. A fist farther away is the pointer helper's grip (a gaze drag). A fist or a pinch by a hand "grab that" has (held out open, or closing into its fist, aiming at a panel it would take) waits until that gesture fires, and then does nothing, or lets the hand go, at most 0.6 s, and then it's what the hand is by then (a fist, a second anchor, or a pinch): an open hand held out to a panel then a fist pulled back and turned over is "grab that", a fist made at the panel without that is a fist grab. At a panel within arm's reach "grab that" doesn't aim, so nothing waits.
|
||||
- **Turn it with the wrist** (2026-10-10): while one hand carries a panel (by its border or bar, or a fist), turning the wrist turns the panel the same way, 1:1, about all three axes (yaw, pitch, roll), to line displays up. The hand's turn is read from its wrist and index and little knuckles (they hold still while the fingers pinch), smoothed (One Euro, 1 Hz still), relative to facing your head across the floor from the hand, and applied on top of the panel facing you: carrying a panel round you still only keeps it facing you, and a wrist turn adds to that. Nothing turns until the hand has turned `HANDS_GRAB_ROTATE_DEADZONE` (5 degrees; the hand frame jitters about 1 degree a frame and drifts up to 6 over a second and a half held still), then it turns from there without a jump. Within 2.5 degrees of level it snaps level. Let go and it keeps the turn; the next grab starts from it. Corner and tab grabs only resize (letting go with one of two hands, the other's turn starts afresh). `HANDS_GRAB_ROTATE=0` keeps the old facing-only carry. A panel held near the eyes by a point far from its centre (closer than twice that distance) faces you from a point between the hand and its centre: facing from the centre turned it about the hand, which moved the centre, which turned it, and 0.6 m panels held by an edge at arm's reach spun and flipped between frames on the third live test's replays.
|
||||
- **Two hands:** both hands holding one panel (pinches or fists) stretch it (its width follows their distance, past a 1 cm dead zone) and turn it with the line between them on every axis: swing the pair round (yaw), raise one hand (roll), bring one nearer, and tip both wrists the same way about the line between them to pitch it (the mean of the two wrists' turns about that line, past `HANDS_GRAB_ROTATE_DEADZONE`); it snaps level within 2.5 degrees. `HANDS_STRETCH_TURN=0`: about the vertical only (before 2026-10-10's third live test). Let go with one and the other carries on. The second hand doesn't have to find a grab zone (since the fourth live test, where it clicked the content instead): while one hand holds a panel, the other's pinch or fist anywhere on it, up to `HANDS_JOIN_REACH` (0.15 m) in front of it, 12 cm behind or 10 cm round it, joins as the second anchor; it never clicks, pokes, pushes or flicks a panel the other hand holds. Replaying the user's two-fist flags (215945 and 215955) with the panel where the live log had it, and 4 cm nearer or 4-8 cm farther: before, a click and no two-hand hold; now both hands hold it 9.4 s, stretching and turning it. A carried panel (one hand or two) turns at most 6 degrees a tick about the hand's point (a fist pulled in toward the face flipped it between ticks in a replay).
|
||||
- **Holding through tracking loss** (2026-10-10: losing the hand dropped what it held): a pinch, card or fist grab, and a poke's or pinch's press, last until a clear release: the hand seen with its pinch or grip up and its fingers open (thumb to index tip over 3.5 cm; a fist's fingers over 1.35 extended) on 2 hands frames in a row (67 ms), or up for 6 frames whatever the fingers say. A pinch that flickers up for a frame or two in fast motion, or that ft-hands ended as lost while the hand comes back still pinched, holds on. Unseen for `HANDS_HOLD_LOST_MS` (1.75 s), it lets go where it is. Meanwhile the panel coasts on the hand's last velocity, slowing to a stop over 150 ms; a frame whose point jumped more than 12 cm counts as unseen (up to 2 in a row). When the hand is back, the smoothing restarts moving at the gap's average velocity and the panel blends to the hand over 100 ms (longer when that's more than 0.6 m/s on average). Offline, carries at 0.6 m/s losing the hand for 3, 6 and 10 frames hold, the panel's largest step 12-17 mm a 90 Hz tick (6.7 moving). A hold also lasts while the pointer helper's word says that hand is "off" (not a controller's) for up to `HANDS_HOLD_LOST_MS`: the word flickered off 24 times in the third live test's 7 minutes and dropped three holds.
|
||||
- **Tracking no hand makes** (`HANDS_TRACK_GATE`, 2026-10-10: "grabbing the side of the monitor, my hand moves off to the left and tracking gets lost: the grabbed monitor is jittery"): on that flag the hand, 45-60 degrees off the head's forward, read alternately at two places 10-20 cm apart and its frame flipped 100-180 degrees between frames. A holding hand's frame that moves faster than 4 m/s from the last one kept, or lands more than 0.5 x 100 m/s² x dt² + 2 cm from where its velocity had it, is dropped (three in a row at one new place: it moved there, and the panel blends over); its frame turning faster than 540 degrees a second likewise. Frames seen by one camera, or 50-80 degrees off the head's forward, are smoothed harder (the cutoffs down to 0.35 of theirs) and don't turn the panel; past 80 degrees the hand counts as unseen, so a panel carried out of view at the side holds still until it's back. The turn's smoothing follows at most 4 Hz. On the third live test's ten flags, each pinch or fist held 0.4 s or more replayed against a 0.6 m panel at the hand (`handinput-test --carries`): the carried panel's centre moved with a per-tick second difference of 159 mm rms (99th percentile 898) on hands-throw, 9.7 mm (29) now; its turn 33 degrees rms (187) then, 1.6 (5.2) now; the side-grab flag alone 96 mm (599) to 6.2 (23). All 23 flags: 119 mm (719) to 7.6 (26), 26 degrees (157) to 1.3 (4.8).
|
||||
- **Flick it away** (`HANDS_FLICK`, 2026-10-10): with the hand at a panel (its palm within 12 cm in front of it, over it, nearly still), flick it out fast with the fingers opening: away from you (within 40 degrees) at `HANDS_FLICK_SPEED` (1.2 m/s) or more, 12 cm or more within 0.2 s, the fingers ending open and opening by `HANDS_FLICK_OPENING` (0.15 of mean extension; 0 lets an open hand pushed out flick too). The panel goes out along the line from your eyes through it, to `HANDS_THROW_DISTANCE` (3 m) from them, facing you, over 0.4 s easing out. A pointing hand doesn't flick (its fingers read about 1.3), and an open hand coming through a panel faster than 0.8 m/s doesn't poke it. On the recordings, with a panel 0.3-0.45 m ahead, once in 11 minutes (6 times without the opening rule); none in the 13 "grab that" flags.
|
||||
- **Throw it** (`HANDS_THROW`, 2026-10-10): let go of a panel you carry with one hand (a card, a fist, a pinch on its border, bar or title; not a resize or two hands) while the hand moves away from you (within 60 degrees) at `HANDS_THROW_SPEED` (1.0 m/s) or more, over the 0.1 s before the fingers opened, and it flies on that way to 3 m from your eyes, facing you. Let go slower and it stays. "Grab that" brings a thrown or flicked panel back, and the gesture again sends it back out there.
|
||||
- **Push it away** (`HANDS_PUSH`, 2026-10-10: "pushing against the monitor or window does not make it go away"): an open hand (the four fingers 1.6 extended or more on average, the middle, ring and little too), palm to the panel (within 50 degrees), over it, pushed straight into it (within 30 degrees of its normal) from in front of it or resting on it: at `HANDS_PUSH_SPEED` (0.5 m/s) or more and 5 cm or more in, or slowly 15 cm past its surface. It goes 3 m out like a flick. A pointing index pokes; a flat open hand (1.8 or more) or an open one coming in at the push's speed doesn't. Not from a hand resting low, typing, pinching or gripping, nor within 0.6 s of a hold or a press. On the recordings with a panel 0.3-0.45 m ahead: probe-moves (open hands pushed and pulled to probe the ray's reach) 2-15 pushes in 3.5 minutes, the 2026-09-29 recording none, lit 1; on 23 flags with a panel at arm's reach, one (both hands pushed out 11-19 cm through it at 0.5-0.6 m/s); "grab that"'s open hand held out to a panel at arm's reach (5-10 cm in at 0.14-0.43 m/s) doesn't push.
|
||||
- **Sent away, a panel keeps its size** (`HANDS_SEND_AWAY_WIDTH`, user, 2026-10-10: "windows should not resize to be smaller when they're sent back"): thrown, flicked, pushed or sent back by "grab that", it keeps its width; one "grab that" shrank to bring it to arm's reach (screens wider than `HANDS_GRAB_THAT_WIDTH`) and still that wide gets the width it had back (`restore`, the default; `keep`: never). In the third live test the 2.5 m desktop came at 0.6 m and was thrown 3 m out at 0.6.
|
||||
- **The target's glow** (`HANDS_TARGET_GLOW`, user, 2026-10-10: "a glow over the window that grab that is targeting"; after the fourth live test: "too powerful. It should be a white glow and far more subtle. It should only be around the outside of the window"): while an open hand is held out to a panel for "grab that", a soft white halo shows outside that panel's edges, about 1.3 degrees wide (1 cm at arm's reach), at most half opaque at the edge and fading out, nothing over the content; it fades in as the gesture arms (0.2 s) and goes as it fires or is let go. An open palm close in front of a panel, toward it, shows the same halo dimmer (a push or a flick would throw it). One overlay (`frametop.guide.target`) sized and curved to the panel, in its plane, so it shows with the cutouts off.
|
||||
- **Buttons** (the fourth live test: "it's extremely difficult to interact with the button to make the curve curve and not"): at arm's reach the curve button is 8 mm across, 3.5 mm past the bar's end, and the bar (first in the list, with a 1 cm margin) took the six pokes the log has at it, 3-5 mm left of its centre. Now the nearest control wins, the round buttons (curve, reset, dock, close) take a fingertip or a pinch at least `HANDS_BUTTON_REACH` (0.015 m) each way from their centre, and a pinch there presses the button (`HANDS_PINCH_BUTTONS`; 0: the bar's grab zone takes it, as before). A hand at one lights it and shows a soft ring round it. The log names the control pressed.
|
||||
- **Glows at corners, edges and buttons** (`HANDS_EDGE_GLOW`, user, after the fourth live test: "almost like a cutout of the soft glow we had for highlighting, just around the edges and corners when you're within the interaction zone"): as a hand comes to a panel (in its touch zone), the same soft white halo shows only where a pinch would take it: an L round the corner it would resize from (along both edges from it), a short segment along the edge it would move the panel by, or a ring round a button; brighter as the fingers come closer, brightest while resizing from that corner or pressing that button. One overlay per hand (`frametop.guide.hand.left`, `.right`), so it shows with the cutouts off. With it the resize handle no longer follows hands (lasers and the gaze keep it); `HANDS_EDGE_GLOW=0` brings back the handle at the hand's corner and the cutout buffer's rings.
|
||||
- **Remote displays** take all of it as screens do: pokes, pinches, the card and fist, corner resize, throws and "grab that".
|
||||
- **SteamVR's own panels** (2026-10-13, user: "work on the native SteamVR overlay windows"; `screens/vr.cpp`'s "SteamVR's panels by hand", `screens/steampanels.cpp`): the dashboard, the Steam menu, Steam's keyboard and other apps' overlays take a poke or a pinch on the content as a click. Pinches on the border, at a corner or like a card do nothing there and show no glow (user, 2026-10-13: out of scope for SteamVR's panels; handinput's `GrabZone` gives them none). In the headset tests they never moved one, with the pointer's device as the right hand or as a treadmill. The code for those grabs, described below, stays in `screens/vr.cpp` for a later try. The hand ray's pinch on a floating window's grab bar still moves it, like the 3D mouse's laser. ft-screens can't click them itself, so the pointer helper's laser does what the hand does ("touch" lines, the helper's "Touch"): it goes 10 cm in front of the point, aimed in, and presses there. OpenVR doesn't say where a dashboard panel is or how big, so for each hands frame ft-screens casts 7 x 7 rays from the eyes past each index tip (12 degrees each way), keeps the hits within 40 cm of the tip, and fits the panel to their texture coordinates; the overlay keys come from the helper's overlay list ("steamvr"), which it keeps while hands are allowed. When grabs were on, a move or resize first hovered the laser just inside the edge: a floating panel then showed its window controls (`valve.steam.gamepadui.floatingfooter`) under it, and only then did the laser press its grab bar (7.5 cm under the bottom edge) or its resize margin (1 cm past the corner) and follow the hand. A docked dashboard or another app's overlay showed none: the pinch did nothing, and ft-screens' log said so ("no window controls under it"). No fist grabs, two-hand stretch, throws, flicks, pushes, rewinds or two-finger scroll there. One hand at a time: the laser is one device.
|
||||
- **Smoothing:** a grab carries the knuckles as the cameras saw them, smoothed (`include/fh_smooth.h`: One Euro in continuous time, read each frame ft-screens draws), not the point moved ahead along the hand's velocity as the cutouts use: that velocity is mostly noise times the lead. It moves the panel only once it's 1 cm from where the pinch began, then carries on from there (no jump), so a still hand leaves the panel still. The head the panel turns to face is smoothed too. `HANDS_GRAB_SMOOTH_MINCUTOFF` (1 Hz), `HANDS_GRAB_SMOOTH_BETA` (15 Hz per m/s), `HANDS_GRAB_DEADZONE` (0.01 m). On the probe-moves replay (`scripts/test-handinput.sh`, 90 Hz ticks), the carried point's per-tick second difference went from 13-17 mm rms (50 mm p99) to 2.3-2.6 mm (11 mm), its still wobble from 1.3-1.5 to 0.8-1.0 mm, and it trails the capture by 85 ms while moving (it was 10 ms ahead, by the prediction, and 18 mm off any delay).
|
||||
- With the cutouts on, a glow in the cutout buffer shows where the fingertip is on the screen (a ring that tightens as it comes to the surface, a dot while pressed); what a pinch would do where the fingers are shows as the soft white glow (above) either way; the bar and the buttons brighten under a pinch or a fingertip.
|
||||
- It takes the helper's word on which hands are bare and whether hand input is allowed ("hands"); without that word for 3 s, near input is off. The helper sends it with `POINTER_HANDS=1`. A hand typing holds back (the helper's "typing" word, `POINTER_PINCH_TYPING` above) doesn't poke, grab, move the pointer with its fingertip, or make "grab that" (below). `HANDS_NEAR` (1): `on` has the helper send it without `POINTER_HANDS` too (direct touch alone, no pinches for the pointer), 0 turns it off, `force` turns it on without the helper (testing). `ft-screens` command `handinput on|off|force|state`.
|
||||
|
||||
**The palm menu** (`screens/palmmenu.cpp`, frame-hands `notes/hand-interaction.md` sections 11 and 18), like the Vision Pro's home indicator: turn a bare hand palm up toward your face, open, and look at it, still. After 0.3 s a small ring (2.5 cm) shows 4 cm above the palm and follows it. A plain thumb-index pinch tap by that hand opens the Steam menu (`steam/ft-steam menu`, what the input relay's `steam_menu` runs), or closes the SteamVR dashboard if it's open. The ring turns blue while the pinch is held, and the release does it; a pinch that's lost, or let go after the hand turned over, does nothing. Turn the hand away for 0.15 s and the ring goes.
|
||||
- Palm up means: the palm's normal within 45 degrees of the line to your head (it leaves past 60; 60 and 75 before the third live test), the palm within 45 degrees of where your head points (leaves past 55), 15-70 cm from the head, the middle, ring and little fingers open (tip-to-wrist over knuckle-to-wrist 1.5 on average, leaves under 1.3), and the palm moving under 0.25 m/s. A palm up in your lap while you look at a screen doesn't count. The hand ray needs the palm turned away, so the two never overlap.
|
||||
- Not by a panel, nor busy (the third live test: the Steam menu opened as the user took a panel's bottom edge from below, palm up, fingers bunched): no ring while the hand is near a panel (in its touch zone or margin, or its fingertip, fingers or palm within 12 cm in front of one and 8 cm round it), while near input presses or holds with it or "grab that" has it, nor for a second after any of those. The pinch must be a plain thumb-index one by a still hand: the middle finger out of it (its tip 4.5 cm or more from the thumb's, the finger 1.35 extended or straighter); a card's index and middle or the bunched fingertips do nothing on the ring.
|
||||
- While the ring shows, ft-screens reports that hand near ("near"), so the pointer helper leaves its pinches alone (no gaze click), and near input leaves it too. A pinch that began before the ring showed stays the helper's.
|
||||
- Where: a bare hand (not one holding a controller), with hand input allowed: on the desktop, and during a VR game while its dashboard is open or Frametop's panels are turned back on. So in a game it closes the Steam menu it opened; with the dashboard closed, the game keeps your hands. `HANDS_PALM_MENU` (1): 0 turns it off; `ft-screens` command `palmmenu on|off|state`.
|
||||
|
||||
**Grab that** (`screens/grabthat.cpp`, 2026-10-10): hold an open hand out toward a screen, a remote display or a floating window, palm to it, then pull the arm back while making a fist, and the panel comes to arm's reach over a quarter of a second: 45 cm ahead of your eyes (`HANDS_GRAB_THAT_DISTANCE`), at the fist's height (at most 45 cm under the eyes), facing you, so the hands can poke it, carry it and turn it. A screen wider than `HANDS_GRAB_THAT_WIDTH` (0.6 m; 0 keeps the size) comes at that width; floating windows keep their size. It doesn't take a panel within arm's reach (below: Which panel); with `HANDS_GRAB_THAT_FROM=0`, as before the fourth live test, the same gesture at a panel that came that way and is still within arm's reach (its centre within 0.8 m of the eyes, wherever you moved it about there) sends it back where it was (and its width), and at a panel within arm's reach that didn't come that way, it centres it at arm's reach. One at a time: a panel that came and is still where it came goes back as another one comes, so they don't stack up (three remote displays did in the live test of 2026-10-10). The reset button sends floating windows back too (the layout reset puts the screens and remote displays back itself). With the panel that near, the open hand held out to it is in its touch zone: its fingers pointing up don't poke it, and its fist or a pinch as it closes waits on the gesture (above).
|
||||
- The gesture, from the user's own example (flag 2026-10-10 15:01): the fingers open (each tip from the wrist at least 1.55 times its knuckle's, on average), the palm facing ahead (its normal's forward part at least 0.45), the wrist at least 28 cm ahead of the eyes, all for 0.2 s; then within 0.45 s of the hand last held out: the middle, ring and little fingers close with the palm still toward the panel, and on one frame a fist (under 1.0), the wrist 8 cm or more back from its farthest in the last half second, the palm turned away from the panel. Only two-camera frames count. Not again for a second.
|
||||
- Which panel (since 2026-10-13, user: "The glow should be where the palm is facing"; the line below lit panels off to the side of a palm that wasn't facing them): the one the open palm faces, the line out of the palm along its normal, with the margins below seen from the palm (`HANDS_GRAB_THAT_AIM=palm`; `eyes` brings back the line from the eyes through the palm). The palm has to face what it takes: in the user's example below the palm was held half down, 41 and 51 degrees off the eyes' line, and that line took both fires where the palm's takes none. With `eyes` (since the fourth live test, user: "make the trigger more precise and only activate if the window is not already held and closed up"): the one the line from your eyes through the open palm meets (`HANDS_GRAB_THAT_MARGIN`, 0: the panel itself; 10 cm or 3 degrees and a 20-degree head cone before), the nearest; else, since the fifth live test ("it couldn't be pulled back in": the palm is held out under the line of sight, so the line runs 12-15 degrees under where the head points and passed just under a desktop thrown 3 m out), the one whose outline it passes nearest within `HANDS_GRAB_THAT_MARGIN_DEG` (10 degrees since 2026-10-11, seen from the eyes; a "no panel there" log line names the nearest and how far off), curved ones on their curve, farther than `HANDS_GRAB_THAT_FROM` (0.8 m: its centre from the eyes). Pinned panels, panels held by a laser or a hand, and panels a hand presses on don't come. A panel at arm's reach is the hands' already: push, flick or throw it to send it away (`HANDS_GRAB_THAT_FROM=0` brings back the gesture sending a near one back, or centring it). The arming hold has the palm facing along that line within `HANDS_GRAB_THAT_FACING` (80 degrees since 2026-10-11) on average and still (under 0.5 m/s over the 0.2 s), else it starts again; the user's example holds 41 and 51 degrees off, at 0.25 and 0.46 m/s. For a second after a panel is sent away (a push, a flick, a throw) the gesture doesn't fire: the push's open hand pulled back closing has its shape.
|
||||
- The closing hand is ours: from when its fingers close (the thumb clear of the index tip, so not a pinch) until 0.4 s after it fires, ft-screens reports it near, so the pointer helper doesn't take the fist as a gaze grip. While a hand is held out it asks ft-hands for its full rate (want-hz).
|
||||
- Offline (`scripts/test-handinput.sh` and `handinput-test --grabthat FLAG_DIR...`, which also replays each flag through near input with a panel at arm's reach): the example flag fires twice, right hand, at -11.35 and -5.89 s (the user's -11.4 and -5.9), and twice at a panel 1.6 m out along its aim (with a panel 0.35-0.45 m ahead, where the gesture brings one, it doesn't take it since the fourth live test: the closing fist grabs it); the other 12 flags and 15 minutes of recordings (probe-moves, the 2026-09-29 and lit recordings, the PR #3 takes) none; 1 of 143 pinches and 3 of 12 grips there began on a hand it claimed. Synthetic waves, pushes, fists without the pull, a slow pull, a hand turned over before closing, pinches held out, a hand dropped to the lap and typing don't fire.
|
||||
- `HANDS_GRAB_THAT` (1): 0 turns it off; `ft-screens` command `grabthat on|off|state`.
|
||||
|
||||
The "approach from front" rules, on recordings: in the hand recorder's desk take (PR #3, with head poses), typing under panels 0.30-0.40 m ahead (at eye height, or 0.45 m low and tilted up) made no press; the hand on the mouse (0.37-0.40 m out, past the resting zone) made 2 presses in 13 s under the nearest ones. In its touch-the-dot take the fingertip stopped 0-2 cm short of the dot in the air, its last bone 44-63 degrees off the line from the head: with a panel 3 cm nearer, 2 of 6 touches pressed at 50 degrees, 3 at 65. `HANDS_POKE_REST_NEAR=0.42` takes the mouse hand out too, but then a low dot 25 cm ahead no longer takes a poke.
|
||||
|
||||
Offline, `scripts/test-handinput.sh [DUMP...]` runs its unit tests and plays recorded hands (`ft-handreplay --publish-dump`) against panels placed around them. On the probe-moves (v4), lit and dim recordings, in 1 s and longer stretches where a hand held still and pointed (184 s in all; the fingertip still moved 19-23 mm in depth within each, median), a panel 3 cm beyond the fingertip got no press, 2 cm one (in 42 s of probe-moves), 1 cm 7, 0.5 cm 18. frame-hands `notes/hand-interaction.md` section 10 has the rest.
|
||||
|
||||
## Rendered hands
|
||||
|
||||
`ft-handview` (`screens/handview.cpp`, built by `screens/build.sh`) draws each tracked hand: when Room View is off, and with `HANDS_VIEW_ROOMVIEW=1` (the default) when it's on too, floating over the cutout's hole (the tracking is good enough to look right on its own; `0` draws nothing with Room View on, as before).
|
||||
|
||||
**The model** (`HANDS_VIEW_MODEL`): `asset` (the default), the WebXR Input Profiles' `generic-hand` (MIT), a rigged low-poly hand of 1360 vertices and 2314 triangles per hand with the 25 WebXR hand joints as its skeleton; or `babylon` (Babylon.js's hand meshes, CC BY 4.0, 1808 triangles), `mrtk3` (MRTK3's OpenXR hand, BSD-3-Clause, 2794), `godot` (Godot XR Tools' hands, CC0, 3052) or `mrtk2` (MRTK2's rigged hand, MIT, 2664). All are in `screens/assets/hands/`, each with its source, version, licence and conversion in a README (the MRTK ones were FBX, converted with assimp and `screens/tools/hand-glb-clean.py`; MRTK3 has only a right hand, mirrored for the left). `screens/tools/hand-asset.py` maps each one's skeleton onto WebXR's 25 joints by name (MRTK3's names sit a joint along; MRTK2 has no metacarpals but the little finger's, so the others go a third of the way to the knuckle, unweighted; extra joints such as a root or a palm give their weights to the wrist) and turns them all into `screens/handasset.h` (quantized vertices, normals, four joints and weights per vertex, the triangles, the joints' bind pose), compiled in, so there's no glTF code at run time. generic-hand's fingertip joints sit about a centimetre inside its fingertips, so its fingers reach about that far past the tracked tips; Babylon's and MRTK2's end within a few millimetres of them. `screens/handskin.cpp` poses it: each of the model's joints gets a frame, the same way in its bind pose and on the tracked hand (version 2's fitted skeleton, else the landmarks): its bone's direction toward the next joint, and a roll from the palm (across it for the fingers' first bones, its normal for the thumb's; each bone after takes its parent's sideways axis, the axis a finger bends about). WebXR's metacarpal joints, which MediaPipe doesn't have, follow the palm (wrist, index and little knuckles). The model is scaled by the palm's size and each bone along its length to the tracked bone (0.6-1.6 times). A left hand is drawn with the model's left hand, so nothing is mirrored. The vertex shader blends up to four joints' matrices per vertex (linear blend skinning on the GPU); the CPU only works out the 25 matrices and skins a third of the vertices to place the hand's square. The model ends just past the wrist: its heel fades out over the 3 cm before the tracked wrist, and the rest past it isn't drawn. That leaves it open there: with the hand pointing away from you, you'd look into it (a hollow tube, a fist's fingertips through the palm). So it's capped: a stencil pass counts the surfaces along each view (+1 for each front face, -1 for each back face; the GPU's early depth test would drop hidden ones, so it's a pass of its own with the depth test off), and where one more back face than front says the view went in through the cut, a disc where the fade begins (an ellipse fitted to the skinned points there) covers it, solid in the middle and fading toward its edge. Godot's hand has a longer, wider wrist, so more of it is faded, and from above a hand pointing away shows a darker patch there. `HANDS_VIEW_MODEL=robot` draws a robot hand built in code from the tracked skeleton (`screens/handrobot.cpp`; no asset, so no licence): rigid pieces, each fixed to one bone, so nothing bends or stretches out of shape. Each finger bone is a light shell (a tapered box with rounded edges and ends) with a gap at each joint; a dark ball sits at each knuckle and at the thumb's base, a dark hinge (a short cylinder along the axis the bone bends about) at the other joints. The palm is a dark core between a palm plate and a back plate (a dark seam shows round its sides, a panel line across each), ending in a short closed cuff past the wrist with a dark plate in its end. Small accents glow cyan: a pad on each fingertip, a line across the back of the knuckles, a dot in the palm, a ring round the cuff. 2373 vertices and 4360 triangles, one mesh for both hands, built once as a right hand on a reference hand (`handskin::DemoPose`'s open one) in each segment's frame: the frames are handskin's (`handskin::TrackedFrames`: the bone's direction and its roll from the palm, by the models' rule), a left hand's mirrored from a right one's, so the same mesh draws the left mirrored. Posed, each of its 16 segments (the palm, three thumb bones, three for each finger) is scaled by the palm's size against the reference's, and the palm's width by the knuckles' span; each vertex is fixed to its bone's start or end, so a shell's rounded ends keep their shape, its straight middle takes up the tracked bone's length, and a fingertip ends at the tracked tip. The girth follows the palm's size, not ft-hands' joint radii (rough, mostly its defaults). The vertex shader applies one segment's matrix per vertex; the CPU works out 16 matrices and poses a quarter of the vertices to place the square. Nothing fades and nothing is open, so there's no cap pass. In `solid` the shells are matte ceramic white and the joints darLine truncated
|
||||
|
||||
**The look** (`HANDS_VIEW_STYLE`): `ghost` (the default) is a light, see-through hand (about a third opaque) that glows white toward its edges, a soft fresnel rim where the surface turns away from you (no drawn outline, so a finger over the palm shows only its own soft edge), with the fingertips a little brighter; `solid` is a matte light grey hand, nearly opaque, lit from above (wrapped, so the underside is dimmer, not dark) with a soft cool rim light, easier to see over bright games. Both shade only the surface nearest the eye (the body pass keeps depth), so a see-through hand never shows its inside or its far side. `model <name>|capsules`, `style ghost|solid` and `inroom 1|0` on `@ft_handview` switch them live until a restart.
|
||||
|
||||
Each hand is an overlay of its own (sort order 100, over the panels and the dashboard): a square placed at the hand, facing your head, side by side, each half what one eye sees of the hand projected onto the square. The compositor keeps a placed overlay still in the room as your head turns, so a hand only waits for its own motion; a head-locked layer would have to be redrawn at the display's rate to stay put. It's drawn for each new hands frame and, while the hand moves, at most every 20 ms in between (moved ahead like the cutouts), on the GPU: the body into a scratch buffer (and the model's cap), then the colour into a DMA-BUF SteamVR imports. The `--demo` hands (two hands 45 cm ahead, drawn 30 times a second each) cost 0.5-0.7 ms of CPU per hand per draw (0.3 of it the mesh) and about 4.5% of one core in all on 2026-10-09 with the capsules; idle it's 0.2%, and with Room View on and `HANDS_VIEW_ROOMVIEW=0` it wakes 10 times a second. Offscreen (`--snapshot`, two waving hands at 30 draws a second each, no SteamVR, capped to one core by frame-job, 2026-10-09) the model took 0.36-0.40 ms of CPU per draw (0.08-0.09 of it posing and placing) and 2.2-2.4% of a core, the capsules 0.55-0.62 ms (0.27-0.34) and 3.2-3.5%. With the fresnel look and the cap (the same day, the user in the headset with SteamVR running): `asset` 0.34 ms of CPU against 0.32-0.33 before, and 0.61-0.62 ms of waiting for the GPU per draw (each frame's glFinish, shared between its two draws) against 0.57; the other models 0.27-0.41 ms of CPU and 0.61-0.68 ms of GPU wait, the capsules 0.44-0.45 and 0.53-0.54. The robot (2026-10-09, frame-job at 4 cores, the same run as the others' numbers here): 0.21-0.24 ms of CPU per draw (0.04-0.05 posing and placing) and 0.55-0.57 ms of GPU wait, against `asset` 0.28 and 0.66, `babylon` 0.26 and 0.66, `mrtk3` 0.25 and 0.66.
|
||||
|
||||
**Between frames and past them** (`screens/handhold.h`, 2026-10-13, user: "implement interpolation for the rendered hands so that they feel smoother", and "if the hand is lost it's preferable the rendered hand display freeze instead of make impossible movements, spaz out, or vanish until a few seconds have passed"). The tracking comes about 30 times a second. Each new frame blends in, joint by joint, from the pose drawn now over about the time since the frame before (15-70 ms), drawn every 15 ms meanwhile. A hand the tracking loses stays where it was last drawn, still, for `HANDS_VIEW_HOLD` seconds (3), then goes. A frame whose palm is further from the last one taken than a hand moves (15 cm, or 6 m/s over the time since; the lazy susan's strokes peak at 4.5) or whose palm turned over (100 degrees) in under 0.2 s isn't drawn until two frames in a row agree with each other by the same test (or 0.25 s passes): the drawn hand holds still meanwhile, then goes straight there. A hand first seen, or back after a loss, also waits for two frames that agree, about 33 ms. A hand taken off for a controller, Room View or a lost head pose goes at once. `HANDS_VIEW_SMOOTH=0` draws each frame as it comes and a hand only while it's tracked, as before. The status line counts the losses held and the jumps held off.
|
||||
|
||||
`--snapshot FILE.png` draws demo poses of both hands offscreen (open palm toward you, palm away, palm up, fist, point, pinch, spread; each row the left hand's two eye halves over dark, the right's over light) with no SteamVR or headset, then the cost loop above (and how long the GPU took after it): for checking the look and the skinning after a change. `FT_HANDS_VIEW_MODEL` and `FT_HANDS_VIEW_STYLE` in its environment pick what it draws.
|
||||
|
||||
A hand holding a controller isn't drawn: the pointer helper sends ft-handview its word ("hands L R allowed" on `@ft_handview`, as to ft-screens); while the helper is silent (3 s), a hand within 15 cm of its side's controller counts as holding it. The hands draw in VR games too (hand input keeps its own rules), but only while hand tracking runs: the relay parks it while Frametop is paused for a game, unless the dashboard is open.
|
||||
|
||||
**Room View** (`screens/roomview.h`): SteamVR has no public "is Room View on". In a VR game, the compositor's `VREvent_RoomViewShown` and `VREvent_RoomViewHidden` tell it; with no game, the construct shows Room View when `camera/enableConstructRoomView` is on (the default on the Frame). The passthrough shared state (`VR_CameraPassthroughState`, read only) has an `enabled` byte; it read 1 on 2026-10-09 and isn't yet known to follow the dashboard's Room View button, so only a 0 counts (off). `HANDS_ROOMVIEW=auto|on|off` in `~/.config/frametop.conf` forces it for both programs.
|
||||
|
||||
```
|
||||
screens/build/ft-handview [--status S] [--res PX] [--demo] [--seconds S] # in the dev container (scripts/in-box)
|
||||
screens/build/ft-handview --snapshot /tmp/hands.png [--res PX] [--seconds S] # offscreen, no SteamVR
|
||||
echo -n state | socat -t 1 - ABSTRACT-SENDTO:ft_handview,bind= # the status line; also: roomview auto|on|off, model, style, inroom
|
||||
screens/tools/hand-asset.py # screens/handasset.h from screens/assets/hands/*.glb (--check: is it up to date)
|
||||
screens/tools/hand-glb-clean.py # a converted (FBX -> glTF) hand rewritten for hand-asset.py (screens/assets/hands/*/README.md)
|
||||
```
|
||||
|
||||
`screens/frametop-handview.service` runs it as a user service (`--status 60`), started and stopped with SteamVR; the installer doesn't install it on its own.
|
||||
To try the hand cutouts without restarting the desktop, `screens/build/ft-handtest [--distance m] [--width m] [--seconds s]` (built by `screens/build.sh`, run in the dev container, with hand tracking on) shows a test panel of its own, a light grid 1 m wide and 0.8 m ahead by default, and cuts your hands out of it the way ft-screens cuts them out of the screens.
|
||||
|
||||
## Camera check
|
||||
|
||||
@@ -441,8 +269,6 @@ Guesses, not tested:
|
||||
- **The colour cameras can't be used while the headset is worn.** The colour module then writes only a half-size image into the top-left quarter of its buffers, and ft-camd drops those frames. So the service runs the mono cameras only, and tracking in bright light, where the mono cameras see dark hands, doesn't get the colour pair's help.
|
||||
- **The colour calibration mapping isn't settled.** Which colour camera is `passthrough_left` (`HANDS_COLOR_LEFT`) and how the module's crop applies (`HANDS_COLOR_CROP`) still need `tools/check_color.py` on a recording with a lit, textured view.
|
||||
- **Depth when one camera loses the hand.** A hand seen in one camera drifts 10% per update toward the one-camera depth guess (`kMonoDepthGain`, 0.1, in `track/tracker.cpp`). In the 2026-09-30 replays that was worse than keeping the last distance (see "3D" above). A smaller gain, such as 0.02, is the next thing to try.
|
||||
- **Pinches aren't reliable enough for everyday use yet.** That's why hand tracking is off by default (`HANDS_MODE=off`: only `ft-handsctl on` starts it), and `POINTER_HANDS` is 0 by default.
|
||||
- **Pinches aren't reliable enough for everyday use yet.** That's why hand tracking stays off until `ft-handsctl on`, and `POINTER_HANDS` is 0 by default.
|
||||
- **SteamVR can leave the upper cameras and the IR light off after a wake**, and then no hands are found. See "Camera check" above: `camcheck.py` tells, the hand recorder won't start a session, and the fix is a SteamVR restart or a reboot.
|
||||
- **The skeleton (2026-10-08) is tested on recordings only**, not in the headset: the cutouts now follow it (version 1's points and capsules), so check them there; `HANDS_SKELETON=0` brings back the filtered landmarks.
|
||||
- **Floating windows don't get hand cutouts or glows.** Their panels show crops of the client buffer, which the cutouts' side-by-side buffer doesn't match (`screens/vr.cpp`, `UpdateCutouts`). Near input works on them.
|
||||
- **The matte and near input are tested offline only** (2026-10-08): `HEADSET-TESTS.md` has the first headset tests.
|
||||
- **Floating windows don't get hand cutouts.** Their panels show crops of the client buffer, which the cutouts' side-by-side buffer doesn't match (`screens/vr.cpp`, `UpdateCutouts`).
|
||||
+4
-15
@@ -430,23 +430,12 @@ def check(log=None, proc=True, ring=True, ring_path=None):
|
||||
names = " ".join(c["name"] for c in r["mono"]) or "none"
|
||||
evidence.append("ft-camd (pid %d, %s): %d mono cameras: %s" % (
|
||||
r["writer_pid"], "running" if r["alive"] else "stale ring", len(r["mono"]), names))
|
||||
have = {c["node"] for c in r["mono"]}
|
||||
want = state.tracking_nodes() if state else SIDE_NODES + UPPER_NODES
|
||||
missing = [n for n in want if n not in have]
|
||||
if r["alive"] and len(r["mono"]) < TRACKING and status == "ok":
|
||||
out["ring_missing"] = missing
|
||||
have = {c["node"] for c in r["mono"]}
|
||||
want = state.tracking_nodes() if state else SIDE_NODES + UPPER_NODES
|
||||
out["ring_missing"] = [n for n in want if n not in have]
|
||||
status, reason = "degraded", ("ft-camd publishes only %d of %d mono cameras, missing %s" % (
|
||||
len(r["mono"]), TRACKING, " ".join("video%d" % n for n in missing) or "?"))
|
||||
elif r["alive"] and missing and status == "ok":
|
||||
# XRService moved the cameras after ft-camd started (the colour module was
|
||||
# unplugged or plugged in: the side pair goes between video9/13 and video0/3).
|
||||
# ft-camd keeps publishing the old devices, and ft-hands leaves them out.
|
||||
out["ring_missing"] = missing
|
||||
old = sorted(have - set(want))
|
||||
status, reason = "degraded", ("ft-camd publishes %s, but XRService runs those cameras on %s now "
|
||||
"(it moved them after ft-camd started)" % (
|
||||
" ".join("video%d" % n for n in old) or "?",
|
||||
" ".join("video%d" % n for n in missing)))
|
||||
len(r["mono"]), TRACKING, " ".join("video%d" % n for n in out["ring_missing"]) or "?"))
|
||||
out.update(status=status, reason=reason, evidence=evidence,
|
||||
summary="ok" if status == "ok" else "%s: %s" % (status, reason))
|
||||
return out
|
||||
|
||||
+7
-191
@@ -8,8 +8,7 @@
|
||||
* Then it gives up its privileges for good, creates the frame ring in the
|
||||
* user's runtime folder (/run/user/UID/frametop-hands/cam-ring, fhring.h), learns
|
||||
* which buffer holds which V4L2 index, and copies each complete bright frame
|
||||
* into the ring. It exits when XRService exits, reallocates its buffers or moves
|
||||
* a camera to another device (SILENT_S);
|
||||
* into the ring. It exits when XRService exits or reallocates its buffers;
|
||||
* start it again (or let systemd) to re-attach.
|
||||
*
|
||||
* XRService itself runs as the same user. The privileges are needed only
|
||||
@@ -68,19 +67,6 @@
|
||||
#define SEAM_LIMIT 2.0 /* seam_score above this: the frame carries the half-size copy */
|
||||
#define STALE_RELEARN 30 /* consecutive unchanged frames: the mapping changed */
|
||||
#define MAX_RELEARNS 5 /* then assume XRService has new buffers, and exit */
|
||||
/*
|
||||
* A mono camera that has delivered frames and then stays silent for SILENT_S while another
|
||||
* keeps going: XRService has moved it to another device. Unplugging or plugging in the colour
|
||||
* module does that (the side pair goes between video9/13 and video0/3, seen 2026-10-12): the
|
||||
* old device just stops, nothing gets reallocated, and ft-camd would publish two dead cameras
|
||||
* for as long as it runs. It exits instead (status 4), so systemd starts it on the new devices.
|
||||
* The cameras all stop together when the headset is taken off; that isn't this. Nor is one that
|
||||
* starts later than the others when it goes back on (seen 2026-10-13: video13 just behind, then
|
||||
* video0 over 3 s behind): for RESUME_GRACE_S after the cameras start again, a camera may stay
|
||||
* silent. One the module moved while the headset was off is caught after that.
|
||||
*/
|
||||
#define SILENT_S 3
|
||||
#define RESUME_GRACE_S 10
|
||||
/*
|
||||
* The color cameras never take the mono ones down with them. Each color buffer is ~9 MB,
|
||||
* and every probe for a fresh one syncs a whole buffer's cache, so a color camera looks
|
||||
@@ -116,8 +102,6 @@ typedef struct {
|
||||
int slot_of[MAX_INDEX]; /* buffer holding each V4L2 index */
|
||||
int relearns;
|
||||
bool shared; /* its buffers sit among another camera's */
|
||||
bool exact; /* its own run, bound to indices by VIDIOC_QUERYBUF */
|
||||
bool each; /* mapped one by one (resolve_each) */
|
||||
|
||||
double recent[8]; /* means of the last frames, for dark detection */
|
||||
int nrecent;
|
||||
@@ -132,7 +116,6 @@ typedef struct {
|
||||
unsigned out_w, out_h; /* image size in the ring */
|
||||
uint64_t last_pub_ns; /* --color-fps pacing */
|
||||
uint64_t paced, seams; /* color frames skipped: pacing, the half-size copy */
|
||||
uint64_t idled; /* mono frames skipped while ft-hands is parked (hands_want) */
|
||||
uint64_t paused_until; /* color: going stale, left alone until then */
|
||||
unsigned pause_s; /* ... for this long the next time */
|
||||
uint64_t pauses;
|
||||
@@ -143,8 +126,6 @@ typedef struct {
|
||||
uint64_t events, bright, dark, stale, torn, repairs;
|
||||
uint64_t sync_ns, copy_ns, nsync; /* cost of cache sync and copy */
|
||||
uint64_t last_bright; /* for the status line */
|
||||
uint64_t last_event_ns; /* its last v4l2_dqbuf, 0 before the first (see SILENT_S) */
|
||||
int vfd; /* our own handle on its node, for format_moved, or -1 */
|
||||
} cam_t;
|
||||
|
||||
static cam_t cams[MAX_CAMS];
|
||||
@@ -169,7 +150,6 @@ static double opt_color_idle = 2; /* ... and this often while no reader as
|
||||
* saves most of the decoding (about 11% of a core at 30 fps).
|
||||
*/
|
||||
static double color_fps = 2;
|
||||
static bool hands_parked = false; /* ft-hands is parked: the mono frames aren't copied (hands_want) */
|
||||
static double opt_status = 10.0;
|
||||
|
||||
static volatile sig_atomic_t stop;
|
||||
@@ -271,7 +251,6 @@ static void setup_camera(cam_t *c, xr_camera_t *cam, int pidfd)
|
||||
{
|
||||
memset(c, 0, sizeof(*c));
|
||||
c->cam = cam;
|
||||
c->vfd = open(cam->path, O_RDWR | O_CLOEXEC | O_NONBLOCK); /* opened while privileged */
|
||||
xr_camera_layout(cam, &c->lay);
|
||||
c->color = c->lay.fmt == XR_FMT_YUV420_10P;
|
||||
/* Color: just the luma rows, not the chroma after them. Whether a frame is new is judged
|
||||
@@ -290,7 +269,6 @@ static void setup_camera(cam_t *c, xr_camera_t *cam, int pidfd)
|
||||
for (int g = 0; g < xr.ngroups; g++)
|
||||
if (xr.groups[g].cam == cam && group_fits(&xr.groups[g], cam, c->need))
|
||||
own = true, exact = exact || xr.groups[g].exact;
|
||||
c->exact = own && exact;
|
||||
|
||||
char model[16];
|
||||
model_of(cam->sensor, model, sizeof(model));
|
||||
@@ -407,15 +385,6 @@ static bool resolve_block(cam_t *c)
|
||||
* the same every time, so their dequeues change no buffer and get no votes. Such
|
||||
* an index is left unmapped (slot -1): on_frame counts its frames as dark without
|
||||
* reading them. Most of a camera's indices silent means it isn't streaming yet.
|
||||
*
|
||||
* Which indices are the dark ones doesn't stay put, though: measured 2026-10-13 with
|
||||
* the headset worn, the bright half moved to other indices every few seconds. With a
|
||||
* fixed split, bright frames landing on a silent index were dropped unread, and
|
||||
* find_fresh then published them at the next dequeue under that frame's timestamp,
|
||||
* 8 ms late. The side cameras lost about 4 of 10 sets with the upper ones, tracking
|
||||
* ran at half rate with 3x the stereo error. So a camera whose buffers are bound by
|
||||
* VIDIOC_QUERYBUF maps its silent indices to their own buffers too, and on_frame
|
||||
* tells bright from dark by each frame's pixels, as for the upper cameras.
|
||||
*/
|
||||
static bool resolve_each(cam_t *c)
|
||||
{
|
||||
@@ -450,18 +419,11 @@ static bool resolve_each(cam_t *c)
|
||||
if (silent * 2 > depth)
|
||||
return false;
|
||||
|
||||
for (int i = 0; i < depth && c->exact; i++) /* see above: silent indices get their own buffers */
|
||||
for (int s = 0; s < c->nslots && c->slot_of[i] < 0; s++)
|
||||
if (c->slot_pos[s] == i && !taken[s])
|
||||
taken[s] = true, c->slot_of[i] = s, silent--;
|
||||
|
||||
printf("%s: queue depth %d, buffers mapped one by one:", c->slug, depth);
|
||||
for (int i = 0; i < depth; i++)
|
||||
c->slot_of[i] < 0 ? printf(" -") : printf(" %d", c->slot_of[i]);
|
||||
if (silent)
|
||||
printf(" (- : %d indices whose frames are all zeros, skipped as dark)", silent);
|
||||
else if (c->exact)
|
||||
printf(" (bright and dark told apart frame by frame)");
|
||||
printf("\n");
|
||||
return true;
|
||||
}
|
||||
@@ -482,8 +444,7 @@ static void resolve_map(cam_t *c)
|
||||
|
||||
/* a shuffled shared run can't be split by change votes alone */
|
||||
c->shared = shared;
|
||||
c->each = false;
|
||||
c->mapped = resolve_block(c) || (!shared && (c->each = resolve_each(c)));
|
||||
c->mapped = resolve_block(c) || (!shared && resolve_each(c));
|
||||
|
||||
if (c->mapped) {
|
||||
fflush(stdout);
|
||||
@@ -727,11 +688,6 @@ static void on_frame(cam_t *c, int64_t index, uint32_t seq, uint64_t ts, uint64_
|
||||
return;
|
||||
}
|
||||
|
||||
if (hands_parked && !c->color) { /* nobody tracks: no copies (the colour ones still tell the light) */
|
||||
c->idled++;
|
||||
return;
|
||||
}
|
||||
|
||||
if (index > c->maxindex) {
|
||||
reset_learning(c); /* the queue grew */
|
||||
return;
|
||||
@@ -762,14 +718,6 @@ static void on_frame(cam_t *c, int64_t index, uint32_t seq, uint64_t ts, uint64_
|
||||
int nchanged;
|
||||
int found = -1;
|
||||
|
||||
if (c->each && c->exact && !fresh(cur, c->samp[slot], &nchanged)) {
|
||||
/* the same all-zero dark frame again (see resolve_each); never another buffer's frame */
|
||||
buf_sync(c->fd[slot], DMA_BUF_SYNC_END);
|
||||
c->dark++;
|
||||
c->rc->dropped++;
|
||||
return;
|
||||
}
|
||||
|
||||
if (!fresh(cur, c->samp[slot], &nchanged)) {
|
||||
buf_sync(c->fd[slot], DMA_BUF_SYNC_END);
|
||||
found = find_fresh(c, cur);
|
||||
@@ -883,39 +831,10 @@ static void on_sample(void *ctx, const tp_sample_t *s)
|
||||
|
||||
cam_t *c = cam_by_minor(tp_get(s->ev, f_dq_minor, s->raw, s->rawlen));
|
||||
|
||||
if (c) {
|
||||
c->last_event_ns = mono_ns();
|
||||
if (c)
|
||||
on_frame(c, tp_get(s->ev, f_dq_index, s->raw, s->rawlen),
|
||||
(uint32_t)tp_get(s->ev, f_dq_seq, s->raw, s->rawlen),
|
||||
(uint64_t)tp_get(s->ev, f_dq_ts, s->raw, s->rawlen), s->time);
|
||||
}
|
||||
}
|
||||
|
||||
/* A mono camera gone silent while another runs (see SILENT_S), or NULL. */
|
||||
static const cam_t *silent_camera(uint64_t now)
|
||||
{
|
||||
static uint64_t running_since; /* when a camera last started again after none ran */
|
||||
uint64_t newest = 0;
|
||||
|
||||
for (int i = 0; i < ncams; i++)
|
||||
if (!cams[i].color && cams[i].last_event_ns > newest)
|
||||
newest = cams[i].last_event_ns;
|
||||
|
||||
if (!newest || now - newest > 1000000000ull) {
|
||||
running_since = 0;
|
||||
return NULL; /* none running: the headset is off or asleep */
|
||||
}
|
||||
if (!running_since)
|
||||
running_since = now;
|
||||
|
||||
if (now - running_since < RESUME_GRACE_S * 1000000000ull)
|
||||
return NULL;
|
||||
for (int i = 0; i < ncams; i++) {
|
||||
const cam_t *c = &cams[i];
|
||||
if (!c->color && c->last_event_ns && now - c->last_event_ns > SILENT_S * 1000000000ull)
|
||||
return c;
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------ ring + user */
|
||||
@@ -924,49 +843,6 @@ static const cam_t *silent_camera(uint64_t now)
|
||||
* Ring cameras: every camera (mono and color) at its own index, then with --with-dark a
|
||||
* dark twin for each mono camera. Returns how many, with which camera each one shows.
|
||||
*/
|
||||
/*
|
||||
* A camera whose node now has another format: XRService gave the node to another camera.
|
||||
* Plugging the colour module in with passthrough working moves the side cameras off the
|
||||
* ISP nodes (video0/3) and puts the colour cameras there; the nodes keep streaming, so
|
||||
* silent_camera never sees it (2026-10-13: tracker fed 2464x2464 NV12 as side frames).
|
||||
*/
|
||||
static const cam_t *format_moved(void)
|
||||
{
|
||||
for (int i = 0; i < ncams; i++) {
|
||||
|
||||
const cam_t *c = &cams[i];
|
||||
struct v4l2_format fmt;
|
||||
|
||||
if (c->vfd < 0)
|
||||
continue;
|
||||
|
||||
memset(&fmt, 0, sizeof(fmt));
|
||||
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
|
||||
|
||||
unsigned w, h, bpl;
|
||||
uint32_t pix;
|
||||
|
||||
if (ioctl(c->vfd, VIDIOC_G_FMT, &fmt) == 0) {
|
||||
w = fmt.fmt.pix_mp.width, h = fmt.fmt.pix_mp.height;
|
||||
pix = fmt.fmt.pix_mp.pixelformat, bpl = fmt.fmt.pix_mp.plane_fmt[0].bytesperline;
|
||||
} else {
|
||||
memset(&fmt, 0, sizeof(fmt));
|
||||
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
|
||||
if (ioctl(c->vfd, VIDIOC_G_FMT, &fmt) < 0) {
|
||||
if (errno == ENODEV)
|
||||
return c;
|
||||
continue;
|
||||
}
|
||||
w = fmt.fmt.pix.width, h = fmt.fmt.pix.height;
|
||||
pix = fmt.fmt.pix.pixelformat, bpl = fmt.fmt.pix.bytesperline;
|
||||
}
|
||||
|
||||
if (w != c->cam->width || h != c->cam->height || pix != c->cam->pixfmt || bpl != c->cam->bytesperline)
|
||||
return c;
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static int ring_layout(int cam_of[], bool dark_of[], int max)
|
||||
{
|
||||
int n = 0;
|
||||
@@ -1022,39 +898,6 @@ static void color_want(void)
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Whether ft-hands is parked (hand interaction in VR games, hands/track/main.cpp "Parked"): it writes
|
||||
* "0" to ring_dir/hands-want every second while parked. A file newer than 3 s saying 0 stops the mono
|
||||
* frames' copies; anything else (no file, a stale one: ft-hands gone) copies them as usual, so other
|
||||
* readers of the ring (the hand recorder, ft-cutouts) are never starved by a parked tracker that died.
|
||||
*/
|
||||
static void hands_want(void)
|
||||
{
|
||||
char path[128], buf[32] = "";
|
||||
struct stat st;
|
||||
struct timespec now;
|
||||
bool parked = false;
|
||||
|
||||
snprintf(path, sizeof(path), "%s/hands-want", ring_dir);
|
||||
clock_gettime(CLOCK_REALTIME, &now);
|
||||
|
||||
if (stat(path, &st) == 0 && now.tv_sec - st.st_mtim.tv_sec <= 3) {
|
||||
FILE *f = fopen(path, "r");
|
||||
if (f) {
|
||||
if (fgets(buf, sizeof(buf), f) && buf[0] == '0')
|
||||
parked = true;
|
||||
fclose(f);
|
||||
}
|
||||
}
|
||||
|
||||
if (parked != hands_parked) {
|
||||
printf("hand tracking %s: %s the mono cameras' frames\n", parked ? "parked" : "back",
|
||||
parked ? "not copying" : "copying");
|
||||
fflush(stdout);
|
||||
hands_parked = parked;
|
||||
}
|
||||
}
|
||||
|
||||
/* A setting from ~/.config/frametop.conf, or FT_<key> from the environment; "" if unset. */
|
||||
static void setting(const char *key, char *out, size_t n)
|
||||
{
|
||||
@@ -1256,10 +1099,10 @@ static void status(double secs)
|
||||
c->last_bright = c->bright;
|
||||
continue;
|
||||
}
|
||||
printf(" %s %s%s%.1f fps (dark %llu at %.1f stale %llu torn %llu remapped %llu idled %llu, sync %.2f ms copy %.2f ms)",
|
||||
c->slug, hands_parked ? "PARKED " : "", c->mapped ? "" : "learning ", (double)(c->bright - c->last_bright) / opt_status,
|
||||
printf(" %s %s%.1f fps (dark %llu at %.1f stale %llu torn %llu remapped %llu, sync %.2f ms copy %.2f ms)",
|
||||
c->slug, c->mapped ? "" : "learning ", (double)(c->bright - c->last_bright) / opt_status,
|
||||
(unsigned long long)c->dark, c->rc->dark_mean, (unsigned long long)c->stale,
|
||||
(unsigned long long)c->torn, (unsigned long long)c->repairs, (unsigned long long)c->idled,
|
||||
(unsigned long long)c->torn, (unsigned long long)c->repairs,
|
||||
c->nsync ? c->sync_ns / 1e6 / c->nsync : 0.0, c->bright ? c->copy_ns / 1e6 / c->bright : 0.0);
|
||||
c->last_bright = c->bright;
|
||||
}
|
||||
@@ -1404,7 +1247,7 @@ int main(int argc, char **argv)
|
||||
signal(SIGINT, on_signal);
|
||||
signal(SIGTERM, on_signal);
|
||||
|
||||
uint64_t start = mono_ns(), last_status = start, last_want = 0, last_hands = 0, last_fmt = start;
|
||||
uint64_t start = mono_ns(), last_status = start, last_want = 0;
|
||||
int rc = 0;
|
||||
|
||||
while (!stop) {
|
||||
@@ -1422,33 +1265,6 @@ int main(int argc, char **argv)
|
||||
last_want = now;
|
||||
}
|
||||
|
||||
if (now - last_hands >= 200000000ull) { /* a stat: often, so an unparked tracker gets frames soon */
|
||||
hands_want();
|
||||
last_hands = now;
|
||||
}
|
||||
|
||||
const cam_t *silent = silent_camera(now);
|
||||
if (silent) {
|
||||
fprintf(stderr, "%s (video%d): no frames for %.1f s while the others run: XRService has moved "
|
||||
"the cameras (colour module unplugged or plugged in?); exiting so ft-camd starts on the "
|
||||
"new devices\n", silent->slug, silent->cam->node,
|
||||
(double)(now - silent->last_event_ns) / 1e9);
|
||||
rc = 4;
|
||||
break;
|
||||
}
|
||||
|
||||
if (now - last_fmt >= 1000000000ull) {
|
||||
last_fmt = now;
|
||||
const cam_t *moved = format_moved();
|
||||
if (moved) {
|
||||
fprintf(stderr, "%s (video%d): its node has another format now: XRService has moved the "
|
||||
"cameras (colour module plugged in or unplugged?); exiting so ft-camd starts on the "
|
||||
"new devices\n", moved->slug, moved->cam->node);
|
||||
rc = 4;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
struct pollfd pf = { .fd = pidfd, .events = POLLIN };
|
||||
|
||||
if (poll(&pf, 1, 0) > 0) {
|
||||
|
||||
@@ -707,80 +707,6 @@ static void split_groups(xr_state_t *st)
|
||||
}
|
||||
}
|
||||
|
||||
static const fdent_t *fdent_of(int xfd)
|
||||
{
|
||||
for (int i = 0; i < nfdents; i++)
|
||||
if (fdents[i].xfd == xfd)
|
||||
return &fdents[i];
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* Another allocation can land between a buffer's two planes (a 16 MB udmabuf, seen after a
|
||||
* SteamVR restart), and the pairing in build_groups then loses that buffer: the camera had
|
||||
* 15 of its 16, and one index's frames were never found ("still learning" for good). A run
|
||||
* bound by VIDIOC_QUERYBUF knows all its camera's buffers, so the missing ones go back in, in
|
||||
* fd order (the allocation order).
|
||||
*/
|
||||
static void complete_exact_groups(xr_state_t *st)
|
||||
{
|
||||
for (int i = 0; i < st->ngroups; i++) {
|
||||
|
||||
xr_group_t *g = &st->groups[i];
|
||||
xr_camera_t *cam = g->cam;
|
||||
|
||||
bool first = g->exact;
|
||||
|
||||
for (int j = 0; j < i && first; j++)
|
||||
first = !(st->groups[j].exact && st->groups[j].cam == cam);
|
||||
|
||||
if (!first) /* the camera's first run takes them all */
|
||||
continue;
|
||||
|
||||
for (int k = 0; k < cam->nqbuf && g->nbufs < XR_MAX_RUNBUFS; k++) {
|
||||
|
||||
int xfd = cam->qbuf_xfd[k];
|
||||
bool have = false;
|
||||
|
||||
for (int b = 0; b < g->nbufs && !have; b++)
|
||||
have = g->buf[b].xfd == xfd;
|
||||
|
||||
const fdent_t *e = fdent_of(xfd);
|
||||
|
||||
if (have || !e || e->kind != FD_DMABUF || e->size != g->planesize[0])
|
||||
continue;
|
||||
|
||||
/* the run the interruption cut off starts a group of its own: empty it */
|
||||
for (int j = 0; j < st->ngroups; j++)
|
||||
for (int b = 0; j != i && b < st->groups[j].nbufs; b++)
|
||||
if (st->groups[j].buf[b].xfd == xfd) {
|
||||
xr_group_t *o = &st->groups[j];
|
||||
memmove(&o->buf[b], &o->buf[b + 1], (size_t)(o->nbufs - b - 1) * sizeof(o->buf[0]));
|
||||
o->nbufs--;
|
||||
break;
|
||||
}
|
||||
|
||||
int at = g->nbufs;
|
||||
|
||||
while (at > 0 && g->buf[at - 1].xfd > xfd)
|
||||
at--;
|
||||
|
||||
memmove(&g->buf[at + 1], &g->buf[at], (size_t)(g->nbufs - at) * sizeof(g->buf[0]));
|
||||
g->buf[at] = (xr_bufref_t){ .xfd = xfd, .xfd1 = -1, .size = e->size };
|
||||
g->nbufs++;
|
||||
}
|
||||
}
|
||||
|
||||
int keep = 0;
|
||||
|
||||
for (int i = 0; i < st->ngroups; i++)
|
||||
if (st->groups[i].nbufs >= 4)
|
||||
st->groups[keep++] = st->groups[i];
|
||||
|
||||
st->ngroups = keep;
|
||||
}
|
||||
|
||||
/*
|
||||
* XRService allocates one udmabuf per plane, plane 0 then plane 1, a whole
|
||||
* queue at a time right after opening the sensor's subdev. Plane 1 matches
|
||||
@@ -878,8 +804,6 @@ static void build_groups(xr_state_t *st)
|
||||
break;
|
||||
}
|
||||
|
||||
complete_exact_groups(st);
|
||||
|
||||
/*
|
||||
* The rest by the sensor marker and sizes. The sensor marker alone can be wrong: XRService
|
||||
* sometimes opens another sensor's subdev (e.g. the idle color camera)
|
||||
|
||||
@@ -7,12 +7,11 @@ Wants=frametop-camd.service
|
||||
PartOf=steamvr.service
|
||||
|
||||
[Service]
|
||||
# In the container it was built in (scripts/in-box: dev for a clone, the release's own for a
|
||||
# release; it's built against Fedora's libraries). It reads ft-camd's ring and
|
||||
# In the dev container (it's built against Fedora's libraries). It reads ft-camd's ring and
|
||||
# writes /run/user/UID/frametop-hands/hands and gestures. Settings: HANDS_* in ~/.config/frametop.conf.
|
||||
ExecStartPre=-@REPO@/scripts/container-up.sh
|
||||
ExecStartPre=-/usr/bin/pkill -x ft-hands
|
||||
ExecStart=@REPO@/scripts/in-box @REPO@/hands/build/ft-hands --status 60
|
||||
ExecStopPost=-/usr/bin/pkill -x ft-hands
|
||||
ExecStart=%h/.local/bin/distrobox enter dev -- @REPO@/hands/build/ft-hands --status 60
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
TimeoutStopSec=5
|
||||
|
||||
+3
-9
@@ -1,15 +1,12 @@
|
||||
#!/usr/bin/env bash
|
||||
# ft-handsctl: turn hand tracking on and off by hand, on the Frame. With it on, your hands show
|
||||
# through Frametop's screens (the hand cutouts); pinches and grips only move the pointer with
|
||||
# POINTER_HANDS=1. HANDS_MODE says when it starts by itself (off, the default: never); on lasts
|
||||
# until SteamVR stops, or with HANDS_MODE=frametop, the desktop.
|
||||
# POINTER_HANDS=1. It doesn't start with SteamVR (hands/run.sh install leaves it off), and it
|
||||
# stops when SteamVR does.
|
||||
#
|
||||
# ft-handsctl on | off | status | log [lines]
|
||||
# ft-handsctl mode [frametop|always|off] HANDS_MODE: with the Frametop desktop, with SteamVR, or off
|
||||
# ft-handsctl cutouts on|off|state ft-screens' hand cutouts, without stopping tracking
|
||||
# ft-handsctl gestures watch pinches and grips live (Ctrl+C to stop)
|
||||
# ft-handsctl flag [note...] flag a tracking failure: the last seconds of hand crops and
|
||||
# ft-hands' output go to HANDS_FLAG_DIR/flag-<time> (handsflag.py)
|
||||
#
|
||||
# Needs the services installed once: hands/run.sh install (it sets ft-camd's capabilities).
|
||||
set -euo pipefail
|
||||
@@ -43,12 +40,9 @@ case ${1:-status} in
|
||||
for u in $units; do echo "$u: $(systemctl --user is-active $u || true)"; done
|
||||
journalctl --user -u frametop-hands.service --no-pager -o cat -n 40 |
|
||||
grep -E '^ *[0-9.]+s |sets with a hand|cameras:' | tail -2 | cut -c1-160 || true ;;
|
||||
mode) # hands/handsmode.py; the Hands page of Frametop Input Settings does the same
|
||||
if [ -n "${2:-}" ]; then python3 "$here/handsmode.py" set "$2"; else python3 "$here/handsmode.py" status; fi ;;
|
||||
log) journalctl --user -u frametop-camd.service -u frametop-hands.service --no-pager -o short -n "${2:-30}" ;;
|
||||
cutouts)
|
||||
ask_screens "cutouts ${2:-state}" ;;
|
||||
gestures) exec python3 "$here/tools/watch_gestures.py" --distance ;;
|
||||
flag) shift; exec python3 "$here/handsflag.py" "$@" ;;
|
||||
*) sed -n '2,14p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
|
||||
*) sed -n '2,11p' "$0" | sed 's/^# \{0,1\}//'; exit 2 ;;
|
||||
esac
|
||||
@@ -1,208 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""handsflag: `ft-handsctl flag [note...]`: flag a hand tracking failure (hands/README.md, Flags).
|
||||
|
||||
Asks ft-hands for its flag buffer (the last HANDS_FLAG_SECONDS of hand crops and what it made of
|
||||
them, and the next 2 s) through its request file, run_dir/flag, and makes the folder it writes them
|
||||
to: HANDS_FLAG_DIR/flag-<time>/ (default ~/.local/share/frametop/hands/flags). Once ft-hands has
|
||||
taken the request (it looks 4 times a second), the drawn hands pulse cyan (ft-handview's flash
|
||||
command; HANDS_FLAG_FLASH=0 turns it off) and, with HANDS_FLAG_TONE=1, a short tone plays. Then it
|
||||
returns; in the background it saves one frame of what you saw (/dev/video99, the 60th: the
|
||||
capture's first frames are black; HANDS_FLAG_VIEW=0 skips it) as view.jpg, ft-handview's head poses
|
||||
for the window as hmd.jsonl, and what it did as ctl.json.
|
||||
|
||||
The note is the rest of the command line; frame-voice's voice commands pass the words after
|
||||
"flag that" ("fingers crossed"), and FRAME_VOICE_WAKE_UNIX, when the wake word was heard
|
||||
(said_unix in the manifest).
|
||||
|
||||
python3 hands/handsflag.py [note...]
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
import handsmode # noqa: E402
|
||||
|
||||
RUN_DIR = "/run/user/%d/frametop-hands" % os.getuid()
|
||||
DEFAULT_DIR = os.path.join(os.environ.get("XDG_DATA_HOME") or os.path.expanduser("~/.local/share"), "frametop/hands/flags")
|
||||
VIEW_SOCKET = "ft_handview"
|
||||
TAKE_WAIT_S = 1.5 # ft-hands looks every 0.25 s
|
||||
AFTER_S = 2.0 # ft-hands keeps taking this long after the flag
|
||||
VIEW_DEVICE = "/dev/video99"
|
||||
|
||||
|
||||
def setting(key, default="", conf=None):
|
||||
"""FT_<key> from the environment, else ~/.config/frametop.conf's, else default."""
|
||||
if "FT_" + key in os.environ:
|
||||
return os.environ["FT_" + key]
|
||||
conf = handsmode.read_conf() if conf is None else conf
|
||||
return conf.get(key, default) or default
|
||||
|
||||
|
||||
def flag_dir(base, now=None):
|
||||
"""A new folder base/flag-YYYYMMDD-HHMMSS (-2, -3... if taken), made private to the user."""
|
||||
os.makedirs(base, exist_ok=True)
|
||||
stamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(now))
|
||||
for n in range(1, 100):
|
||||
path = os.path.join(base, "flag-" + stamp + ("" if n == 1 else "-%d" % n))
|
||||
try:
|
||||
os.mkdir(path, 0o700)
|
||||
return path
|
||||
except FileExistsError:
|
||||
continue
|
||||
raise RuntimeError("no free flag folder in " + base)
|
||||
|
||||
|
||||
def one_line(s):
|
||||
return " ".join(str(s).split())
|
||||
|
||||
|
||||
def request(run_dir, path, note, said_unix, flag_unix):
|
||||
"""Writes run_dir/flag (key=value lines) whole, through a temporary file."""
|
||||
os.makedirs(run_dir, mode=0o700, exist_ok=True)
|
||||
lines = ["dir=" + path, "note=" + one_line(note), "flag_unix=%.3f" % flag_unix]
|
||||
if said_unix:
|
||||
lines.append("said_unix=%.3f" % float(said_unix))
|
||||
tmp = os.path.join(run_dir, "flag.tmp")
|
||||
with open(tmp, "w") as f:
|
||||
f.write("\n".join(lines) + "\n")
|
||||
os.replace(tmp, os.path.join(run_dir, "flag"))
|
||||
|
||||
|
||||
def wait_taken(run_dir, timeout=TAKE_WAIT_S):
|
||||
"""True once ft-hands has taken the request (the file is gone); else removes it, False."""
|
||||
path = os.path.join(run_dir, "flag")
|
||||
end = time.monotonic() + timeout
|
||||
while time.monotonic() < end:
|
||||
if not os.path.exists(path):
|
||||
return True
|
||||
time.sleep(0.05)
|
||||
try:
|
||||
os.unlink(path)
|
||||
except FileNotFoundError:
|
||||
return True # taken at the last moment
|
||||
return False
|
||||
|
||||
|
||||
def ask_view(command, sock_name=VIEW_SOCKET, timeout=1.0, size=65536):
|
||||
"""A command to ft-handview's socket and its reply, or None if it didn't answer."""
|
||||
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
try:
|
||||
s.bind("")
|
||||
s.settimeout(timeout)
|
||||
s.sendto(command.encode(), "\0" + sock_name)
|
||||
return s.recv(size).decode(errors="replace")
|
||||
except OSError:
|
||||
return None
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
def poses_lines(reply):
|
||||
"""ft-handview's `poses` reply as hmd.jsonl lines ({"t_ns", "m": 12})."""
|
||||
out = []
|
||||
if not reply or not reply.startswith("ok"):
|
||||
return out
|
||||
for line in reply.split("\n")[1:]:
|
||||
parts = line.split()
|
||||
if len(parts) == 13:
|
||||
try:
|
||||
out.append(json.dumps({"t_ns": int(parts[0]), "m": [float(x) for x in parts[1:]]}))
|
||||
except ValueError:
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def tone():
|
||||
"""A short soft tone (HANDS_FLAG_TONE=1), if the session has pw-play."""
|
||||
import math
|
||||
import struct
|
||||
import tempfile
|
||||
rate, secs = 48000, 0.18
|
||||
n = int(rate * secs)
|
||||
frames = b"".join(struct.pack("<h", int(6000 * math.sin(2 * math.pi * 1175 * i / rate) *
|
||||
min(1, i / 400, (n - i) / 2000))) for i in range(n))
|
||||
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
|
||||
f.write(b"RIFF" + struct.pack("<I", 36 + len(frames)) + b"WAVEfmt " +
|
||||
struct.pack("<IHHIIHH", 16, 1, 1, rate, rate * 2, 2, 16) + b"data" + struct.pack("<I", len(frames)) + frames)
|
||||
try:
|
||||
subprocess.Popen(["pw-play", f.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def grab_view(path, device=VIEW_DEVICE):
|
||||
"""One frame of what the user sees, the 60th (the capture's first frames are black), as path.
|
||||
On the low cores at the lowest priority; a few seconds at most."""
|
||||
if not os.path.exists(device):
|
||||
return "no " + device
|
||||
cmd = ["nice", "-n", "19", "taskset", "-c", "0-4", "timeout", "15", "ffmpeg", "-loglevel", "error", "-y", "-f", "v4l2",
|
||||
"-i", device, "-vf", r"select=gte(n\,59)", "-frames:v", "1", "-q:v", "3", path]
|
||||
try:
|
||||
r = subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=20)
|
||||
except (OSError, subprocess.TimeoutExpired) as e:
|
||||
return str(e)
|
||||
return "ok" if r.returncode == 0 and os.path.exists(path) else (r.stderr.decode(errors="replace").strip()[-200:] or "failed")
|
||||
|
||||
|
||||
def background(path, ctl, grab=True, sock_name=VIEW_SOCKET, seconds=14.0, wait_s=AFTER_S + 0.7):
|
||||
"""After the flag: the view frame, then (once ft-hands' window has closed) the head poses, and ctl.json."""
|
||||
if grab:
|
||||
ctl["view"] = grab_view(os.path.join(path, "view.jpg"))
|
||||
ctl["view_unix"] = time.time()
|
||||
rest = ctl["flag_unix"] + wait_s - time.time()
|
||||
if rest > 0:
|
||||
time.sleep(rest)
|
||||
lines = poses_lines(ask_view("poses %.1f" % (seconds + 1), sock_name, size=1 << 18))
|
||||
if lines:
|
||||
with open(os.path.join(path, "hmd.jsonl"), "w") as f:
|
||||
f.write("\n".join(lines) + "\n")
|
||||
ctl["hmd_poses"] = len(lines)
|
||||
with open(os.path.join(path, "ctl.json"), "w") as f:
|
||||
json.dump(ctl, f, indent=1)
|
||||
f.write("\n")
|
||||
|
||||
|
||||
def main(argv, run_dir=RUN_DIR, sock_name=VIEW_SOCKET, detach=True):
|
||||
note = " ".join(argv).strip()
|
||||
conf = handsmode.read_conf()
|
||||
base = os.path.expanduser(setting("HANDS_FLAG_DIR", DEFAULT_DIR, conf))
|
||||
flag_unix = time.time()
|
||||
path = flag_dir(base, flag_unix)
|
||||
said = os.environ.get("FRAME_VOICE_WAKE_UNIX") or None
|
||||
request(run_dir, path, note, said, flag_unix)
|
||||
taken = wait_taken(run_dir)
|
||||
ctl = {"flag_unix": flag_unix, "note": note or None, "said_unix": float(said) if said else None, "taken": taken}
|
||||
if taken:
|
||||
if setting("HANDS_FLAG_FLASH", "1", conf) != "0":
|
||||
ctl["flash"] = ask_view("flash", sock_name, timeout=0.5)
|
||||
if setting("HANDS_FLAG_TONE", "0", conf) == "1":
|
||||
tone()
|
||||
print("flag: %s (written in about %.0f s)" % (path, AFTER_S + 0.5))
|
||||
else:
|
||||
print("flag: ft-hands didn't take it (is hand tracking on? ft-handsctl status); saving the view only: " + path,
|
||||
file=sys.stderr)
|
||||
sys.stdout.flush()
|
||||
grab = setting("HANDS_FLAG_VIEW", "1", conf) != "0"
|
||||
seconds = float(setting("HANDS_FLAG_SECONDS", "12", conf) or 12) + AFTER_S
|
||||
if detach and os.fork() > 0:
|
||||
return 0 if taken else 1
|
||||
if detach:
|
||||
os.setsid()
|
||||
devnull = os.open(os.devnull, os.O_RDWR)
|
||||
for fd in (0, 1, 2):
|
||||
os.dup2(devnull, fd)
|
||||
try:
|
||||
background(path, ctl, grab, sock_name, seconds)
|
||||
finally:
|
||||
if detach:
|
||||
os._exit(0)
|
||||
return 0 if taken else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -1,358 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""handsmode: when hand tracking runs, by HANDS_MODE in ~/.config/frametop.conf.
|
||||
|
||||
frametop ft-camd, ft-hands and the hand view run while the Frametop desktop does: the session
|
||||
(session/frametop-session.sh) says "desktop up" once ft-screens is up and "desktop
|
||||
down" as it ends.
|
||||
always they're wanted by steamvr.service (systemctl add-wants), so they run whenever SteamVR
|
||||
does, in games and the SteamVR dashboard too, with or without the desktop. The input
|
||||
relay then leaves them running and unparked when Frametop pauses for a game.
|
||||
off (the default, and anything else) nothing starts them. ft-handsctl on still starts hand
|
||||
tracking by hand, until SteamVR stops; choosing off stops it, and ft-cutouts' units too.
|
||||
|
||||
The units are the ones hands/run.sh install puts in ~/.config/systemd/user, and the hand view's
|
||||
frametop-handview.service when it's installed; a unit that isn't installed is left out.
|
||||
ft-camd needs its file capabilities (hands/run.sh caps, with sudo): without them this starts
|
||||
nothing and says so ("problem"), and never asks for a password.
|
||||
|
||||
python3 hands/handsmode.py status [--json] # the mode, what runs, what's wrong
|
||||
python3 hands/handsmode.py apply # start what HANDS_MODE wants now, fix the SteamVR wants
|
||||
python3 hands/handsmode.py set MODE # write HANDS_MODE, then apply it, stopping what it
|
||||
# doesn't want (Frametop Input Settings, ft-handsctl mode)
|
||||
python3 hands/handsmode.py desktop up|down # the session's hooks; only HANDS_MODE=frametop acts
|
||||
|
||||
It also reads and writes the three hand input switches Frametop Input Settings shows (METHODS:
|
||||
gaze clicks, the hand ray, direct touch), which are pointer and ft-screens settings, not services.
|
||||
|
||||
On the host, system Python, standard library only (the settings app runs it through
|
||||
distrobox-host-exec, and imports it for the settings' defaults).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
CONF_PATH = os.path.expanduser("~/.config/frametop.conf")
|
||||
MODES = ("frametop", "always", "off")
|
||||
DEFAULT_MODE = "off"
|
||||
# The cutouts' margin (HANDS_MATTE_MARGIN, degrees seen from the eyes; ft-screens' default) and the
|
||||
# range Frametop Input Settings offers. ft-screens takes 0-10.
|
||||
MARGIN_DEFAULT = 1.5
|
||||
MARGIN_MIN, MARGIN_MAX, MARGIN_STEP = 0.0, 3.0, 0.25
|
||||
STEAMVR = "steamvr.service"
|
||||
TRACKING = ("frametop-camd.service", "frametop-hands.service")
|
||||
VIEW = "frametop-handview.service" # draws the hands where Room View is off (hands-view)
|
||||
CUTOUTS = ("frametop-cutouts-camd.service", "frametop-cutouts-hands.service") # ft-cutouts' transient units
|
||||
UNITS = TRACKING + (VIEW,)
|
||||
CAMD = "frametop-camd.service"
|
||||
WANTS_DIR = os.path.expanduser("~/.config/systemd/user/" + STEAMVR + ".wants")
|
||||
|
||||
|
||||
def read_conf(path=CONF_PATH):
|
||||
"""KEY=value lines, as the shell that sources the file would mostly see them."""
|
||||
conf = {}
|
||||
try:
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
line = line.split("#", 1)[0].strip()
|
||||
if "=" in line:
|
||||
k, v = line.split("=", 1)
|
||||
v = v.strip()
|
||||
if len(v) >= 2 and v[0] == v[-1] and v[0] in "'\"":
|
||||
v = v[1:-1]
|
||||
conf[k.strip()] = v
|
||||
except OSError:
|
||||
pass
|
||||
return conf
|
||||
|
||||
|
||||
def write_value(key, value, path=CONF_PATH):
|
||||
"""Set KEY=value, keeping comments and the rest of the file (as Frametop Input Settings does)."""
|
||||
try:
|
||||
with open(path) as f:
|
||||
lines = f.read().splitlines()
|
||||
except OSError:
|
||||
lines = []
|
||||
for i, line in enumerate(lines):
|
||||
if line.split("#", 1)[0].strip().startswith(f"{key}="):
|
||||
comment = line[line.index("#"):] if "#" in line else ""
|
||||
lines[i] = f"{key}={value}" + (f" {comment}" if comment else "")
|
||||
break
|
||||
else:
|
||||
lines.append(f"{key}={value}")
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w") as f:
|
||||
f.write("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
def mode_of(conf):
|
||||
v = conf.get("HANDS_MODE", "")
|
||||
return v if v in MODES else DEFAULT_MODE
|
||||
|
||||
|
||||
def margin_of(conf):
|
||||
"""HANDS_MATTE_MARGIN as ft-screens reads it: a number 0-10, else its default."""
|
||||
try:
|
||||
v = float(conf.get("HANDS_MATTE_MARGIN", ""))
|
||||
except ValueError:
|
||||
return MARGIN_DEFAULT
|
||||
return v if 0 <= v <= 10 else MARGIN_DEFAULT
|
||||
|
||||
|
||||
# How the hands work the desktop, as three switches that don't depend on each other (Frametop Input
|
||||
# Settings' Hands page; user, 2026-10-10). In frametop.conf:
|
||||
# gaze_clicks gaze mode's hand clicks: a pinch clicks where you look, a middle pinch right-clicks, a
|
||||
# pinch by each hand holds both buttons. POINTER_HANDS=1 and POINTER_GAZE_PINCH (1).
|
||||
# ray with gaze mode off, a raised pointing hand's ray drives the pointer (and a low hand's
|
||||
# pinch presses where it is). POINTER_HANDS=1 and POINTER_HAND_RAY (1).
|
||||
# touch ft-screens' direct touch, poke and grab its panels, gaze or not. HANDS_NEAR: 1 (the
|
||||
# default) with POINTER_HANDS=1, on without it, force always, 0 off.
|
||||
# POINTER_HANDS turns the pointer's hand input on for either of the first two, so the others' keys
|
||||
# are pinned when it changes, and an old config reads as it always worked (HANDS_NEAR=1 and
|
||||
# POINTER_HANDS=1: all three on; POINTER_HANDS=0, the default: all three off).
|
||||
METHODS = ("gaze_clicks", "ray", "touch")
|
||||
|
||||
|
||||
def _flag(conf, key, default):
|
||||
"""A 0/1 setting as the pointer helper reads it (a number, not 0), else the default."""
|
||||
v = conf.get(key)
|
||||
if v is None or v == "":
|
||||
return default
|
||||
try:
|
||||
return float(v) != 0
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def near_of(conf):
|
||||
"""Whether ft-screens' direct touch is on: HANDS_NEAR as ft-screens and the helper read it."""
|
||||
v = conf.get("HANDS_NEAR", "1")
|
||||
if v == "0":
|
||||
return False
|
||||
if v in ("on", "force"):
|
||||
return True
|
||||
return _flag(conf, "POINTER_HANDS", False)
|
||||
|
||||
|
||||
def methods_of(conf):
|
||||
"""{method: on} for METHODS."""
|
||||
hands = _flag(conf, "POINTER_HANDS", False)
|
||||
return {"gaze_clicks": hands and _flag(conf, "POINTER_GAZE_PINCH", True),
|
||||
"ray": hands and _flag(conf, "POINTER_HAND_RAY", True),
|
||||
"touch": near_of(conf)}
|
||||
|
||||
|
||||
def method_writes(conf, method, on):
|
||||
"""The KEY: value writes that turn one method on or off and leave the other two as they are."""
|
||||
want = methods_of(conf)
|
||||
want[method] = bool(on)
|
||||
writes = {}
|
||||
hands = want["gaze_clicks"] or want["ray"]
|
||||
if _flag(conf, "POINTER_HANDS", False) != hands:
|
||||
writes["POINTER_HANDS"] = "1" if hands else "0"
|
||||
if hands:
|
||||
for key, m in (("POINTER_GAZE_PINCH", "gaze_clicks"), ("POINTER_HAND_RAY", "ray")):
|
||||
if _flag(conf, key, True) != want[m]:
|
||||
writes[key] = "1" if want[m] else "0"
|
||||
if near_of(dict(conf, **writes)) != want["touch"]:
|
||||
writes["HANDS_NEAR"] = "on" if want["touch"] else "0"
|
||||
return writes
|
||||
|
||||
|
||||
class Host:
|
||||
"""systemctl --user, pgrep and getcap. Tests replace it."""
|
||||
|
||||
def __init__(self):
|
||||
uid = os.getuid()
|
||||
# Inside the Frametop session XDG_RUNTIME_DIR is the session's own folder, and the bus is
|
||||
# its private one: systemctl needs the user manager's.
|
||||
self.env = dict(os.environ, XDG_RUNTIME_DIR=f"/run/user/{uid}",
|
||||
DBUS_SESSION_BUS_ADDRESS=f"unix:path=/run/user/{uid}/bus")
|
||||
|
||||
def run(self, cmd, timeout=20):
|
||||
try:
|
||||
return subprocess.run(cmd, env=self.env, stdin=subprocess.DEVNULL, capture_output=True,
|
||||
text=True, timeout=timeout)
|
||||
except (OSError, subprocess.TimeoutExpired) as e:
|
||||
return subprocess.CompletedProcess(cmd, 1, "", str(e))
|
||||
|
||||
def systemctl(self, *args):
|
||||
return self.run(["systemctl", "--user", *args])
|
||||
|
||||
def units(self, names):
|
||||
"""{unit: {"load": LoadState, "active": ActiveState}} for each."""
|
||||
out = self.systemctl("show", "-p", "Id,LoadState,ActiveState", *names).stdout
|
||||
found = {}
|
||||
for block in out.strip().split("\n\n"):
|
||||
props = dict(line.split("=", 1) for line in block.splitlines() if "=" in line)
|
||||
if props.get("Id"):
|
||||
found[props["Id"]] = {"load": props.get("LoadState", ""), "active": props.get("ActiveState", "")}
|
||||
return {u: found.get(u, {"load": "not-found", "active": "inactive"}) for u in names}
|
||||
|
||||
def exec_path(self, unit):
|
||||
m = re.search(r"path=(\S+)", self.systemctl("show", "-p", "ExecStart", "--value", unit).stdout)
|
||||
return m.group(1) if m else ""
|
||||
|
||||
def wanted(self, unit):
|
||||
return os.path.lexists(os.path.join(WANTS_DIR, unit))
|
||||
|
||||
def exists(self, path):
|
||||
return os.path.exists(path)
|
||||
|
||||
def caps(self, path):
|
||||
"""Does ft-camd have its capabilities? None: can't tell (no getcap)."""
|
||||
if not shutil.which("getcap"):
|
||||
return None
|
||||
return "cap_sys_ptrace" in self.run(["getcap", path]).stdout
|
||||
|
||||
def desktop_up(self):
|
||||
return self.run(["pgrep", "-x", "ft-screens"]).returncode == 0
|
||||
|
||||
def cameras(self):
|
||||
"""camcheck's summary ("ok", "degraded: ...", "unknown: ...")."""
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import camcheck
|
||||
return camcheck.check()["summary"]
|
||||
except Exception as e: # a status line, not a crash
|
||||
return f"unknown: camcheck failed: {e}"
|
||||
|
||||
|
||||
def status(host, conf=None, cameras=True):
|
||||
"""What the settings app shows: the mode, what's installed and running, and "problem", a
|
||||
sentence saying why hand tracking can't run (empty when nothing's wrong)."""
|
||||
conf = read_conf() if conf is None else conf
|
||||
states = host.units((STEAMVR,) + UNITS + CUTOUTS)
|
||||
|
||||
def loaded(u):
|
||||
return states[u]["load"] == "loaded"
|
||||
|
||||
def running(u):
|
||||
return states[u]["active"] in ("active", "activating", "reloading")
|
||||
|
||||
st = {"mode": mode_of(conf), "steamvr": running(STEAMVR), "desktop": host.desktop_up(),
|
||||
"installed": all(loaded(u) for u in TRACKING), "view": loaded(VIEW),
|
||||
"tracking": all(running(u) for u in TRACKING), "view_running": running(VIEW),
|
||||
"cutouts": any(running(u) for u in CUTOUTS), "wanted": [u for u in UNITS if host.wanted(u)],
|
||||
"active": [u for u in UNITS + CUTOUTS if running(u)],
|
||||
"camd": "", "built": False, "caps": None, "cameras": "", "problem": ""}
|
||||
if st["installed"]:
|
||||
st["camd"] = host.exec_path(CAMD)
|
||||
st["built"] = bool(st["camd"]) and host.exists(st["camd"])
|
||||
if st["built"]:
|
||||
st["caps"] = host.caps(st["camd"])
|
||||
repo = os.path.dirname(os.path.dirname(os.path.dirname(st["camd"]))) if st["camd"] else "the Frametop folder"
|
||||
if not st["installed"]:
|
||||
st["problem"] = ("Hand tracking isn't installed. To install it, run hands/run.sh install in the "
|
||||
"Frametop folder, in a terminal (it asks for your password once).")
|
||||
elif not st["built"]:
|
||||
st["problem"] = f"ft-camd isn't built ({st['camd'] or 'no path'}). Run hands/run.sh install in {repo}."
|
||||
elif st["caps"] is False:
|
||||
st["problem"] = (f"ft-camd is missing the permissions it needs to read the cameras (a rebuild clears "
|
||||
f"them). In a terminal, run {repo}/hands/run.sh caps (it asks for your password).")
|
||||
elif cameras and st["steamvr"]:
|
||||
st["cameras"] = host.cameras()
|
||||
if st["cameras"].startswith("degraded"):
|
||||
st["problem"] = "The headset's cameras aren't all running: " + st["cameras"].split(": ", 1)[-1]
|
||||
st["summary"] = summary(st)
|
||||
return st
|
||||
|
||||
|
||||
def summary(st):
|
||||
"""One plain line: what runs, or when it will."""
|
||||
if st["tracking"]:
|
||||
how = {"off": " (started by hand: ft-handsctl)"}.get(st["mode"], "")
|
||||
return "Hand tracking is running" + how
|
||||
if st["cutouts"]:
|
||||
return "The hand cutouts are running (started by hand: ft-cutouts)"
|
||||
if st["mode"] == "off":
|
||||
return "Hand tracking is off"
|
||||
when = "with the Frametop desktop" if st["mode"] == "frametop" else "with SteamVR"
|
||||
if st["problem"]:
|
||||
return "Hand tracking can't start " + when
|
||||
if not st["steamvr"]:
|
||||
return "Hand tracking starts " + when
|
||||
return "Hand tracking is starting" if st["mode"] == "always" or st["desktop"] else "Hand tracking starts " + when
|
||||
|
||||
|
||||
def plan(mode, st, desktop=None, strict=False):
|
||||
"""The systemctl calls that bring the units to MODE: a list of argument lists. Starting and the
|
||||
SteamVR wants always follow the mode; stopping only when the desktop just ended (desktop=False,
|
||||
the session's hook) or with strict (the mode was just chosen), so an install or a caps run
|
||||
doesn't stop what ft-handsctl or ft-cutouts started by hand.
|
||||
desktop: True/False from the session's hooks, None to look (st["desktop"])."""
|
||||
units = [u for u in TRACKING if st["installed"]] + ([VIEW] if st["view"] else [])
|
||||
up = st["desktop"] if desktop is None else desktop
|
||||
can_run = st["installed"] and st["built"] and st["caps"] is not False
|
||||
calls = []
|
||||
if mode == "always":
|
||||
missing = [u for u in units if u not in st["wanted"]]
|
||||
if missing:
|
||||
calls.append(["add-wants", STEAMVR, *missing])
|
||||
else:
|
||||
unwanted = [u for u in units if u in st["wanted"]]
|
||||
if unwanted:
|
||||
calls.append(["disable", *unwanted])
|
||||
run = st["steamvr"] and can_run and (mode == "always" or (mode == "frametop" and up))
|
||||
if run and units:
|
||||
calls.append(["start", "--no-block", *units])
|
||||
elif not (strict or desktop is False):
|
||||
pass
|
||||
elif mode == "off" and st["active"]:
|
||||
calls.append(["stop", "--no-block", *st["active"]]) # ft-handsctl's or ft-cutouts' too
|
||||
elif mode == "frametop" and not up:
|
||||
active = [u for u in units if u in st["active"]]
|
||||
if active:
|
||||
calls.append(["stop", "--no-block", *active])
|
||||
return calls
|
||||
|
||||
|
||||
def apply(host, conf=None, desktop=None, strict=False):
|
||||
"""Bring the units to HANDS_MODE (plan); returns the systemctl calls made, and the status they
|
||||
started from."""
|
||||
conf = read_conf() if conf is None else conf
|
||||
st = status(host, conf, cameras=False)
|
||||
calls = plan(st["mode"], st, desktop, strict)
|
||||
for args in calls:
|
||||
host.systemctl(*args)
|
||||
return calls, st
|
||||
|
||||
|
||||
def main(argv):
|
||||
host = Host()
|
||||
cmd = argv[0] if argv else "status"
|
||||
if cmd == "status":
|
||||
st = status(host)
|
||||
if "--json" in argv:
|
||||
print(json.dumps(st))
|
||||
else:
|
||||
print(f"HANDS_MODE={st['mode']}: {st['summary']}")
|
||||
if st["problem"]:
|
||||
print(st["problem"])
|
||||
return 0
|
||||
if (cmd == "set" and len(argv) == 2 and argv[1] in MODES) or argv == ["apply"]:
|
||||
if cmd == "set":
|
||||
write_value("HANDS_MODE", argv[1])
|
||||
calls, st = apply(host, strict=cmd == "set")
|
||||
for args in calls:
|
||||
print("systemctl --user " + " ".join(args))
|
||||
if st["problem"]:
|
||||
print(st["problem"], file=sys.stderr)
|
||||
return 0
|
||||
if cmd == "desktop" and len(argv) == 2 and argv[1] in ("up", "down"):
|
||||
conf = read_conf()
|
||||
if mode_of(conf) == "frametop":
|
||||
calls, st = apply(host, conf, desktop=argv[1] == "up")
|
||||
for args in calls: # into the session's log
|
||||
print("frametop: hand tracking (HANDS_MODE=frametop): systemctl --user " + " ".join(args))
|
||||
if st["problem"] and argv[1] == "up":
|
||||
print("frametop: hand tracking: " + st["problem"])
|
||||
return 0
|
||||
print("usage: handsmode.py status [--json] | apply | set frametop|always|off | desktop up|down", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -27,15 +27,7 @@
|
||||
* room with the HMD pose at its capture time (capture_ns, begin_ns) before subtracting,
|
||||
* so turning your head doesn't drag.
|
||||
*
|
||||
* Middle pinch (version 3): the thumb on the middle fingertip instead of the index one, with
|
||||
* the pinch's thresholds and point. distance is thumb tip to middle tip. It begins only while
|
||||
* the thumb is clearly nearer the middle tip than the index tip, and an index pinch and a middle
|
||||
* pinch on one hand exclude each other (whichever began first holds until it ends). The pointer
|
||||
* helper's hand ray turns it into a right click (a tap) or scrolling (held and moved).
|
||||
*
|
||||
* Version 1 had only the pinches (192 bytes); version 2 adds the grips after them, version 3
|
||||
* the middle pinches after those. A reader of an older version maps and copies the size it
|
||||
* knows: the file only grows at the end.
|
||||
* Version 1 had only the pinches (192 bytes); version 2 adds the grips after them.
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
@@ -44,7 +36,7 @@
|
||||
#include <stdint.h>
|
||||
|
||||
#define FH_GESTURES_MAGIC "FHGEST01"
|
||||
#define FH_GESTURES_VERSION 3
|
||||
#define FH_GESTURES_VERSION 2
|
||||
|
||||
enum {
|
||||
FH_PINCH_TRACKED = 1u << 0, /* the hand was tracked in this frame */
|
||||
@@ -83,8 +75,7 @@ typedef struct {
|
||||
uint8_t reserved[8];
|
||||
fh_pinch_t pinch[2]; /* [0] left hand, [1] right hand */
|
||||
fh_pinch_t grip[2]; /* version 2 */
|
||||
fh_pinch_t pinch2[2]; /* version 3: thumb to middle fingertip */
|
||||
} fh_gestures_t;
|
||||
|
||||
static_assert(sizeof(fh_pinch_t) == 64, "fh_pinch_t layout");
|
||||
static_assert(sizeof(fh_gestures_t) == 64 + 6 * 64, "fh_gestures_t layout");
|
||||
static_assert(sizeof(fh_gestures_t) == 64 + 4 * 64, "fh_gestures_t layout");
|
||||
@@ -7,10 +7,6 @@
|
||||
* Positions are metres in the head frame at capture time, which is OpenVR's HMD
|
||||
* frame (+x right, +y up, -z forward). Turn them into the room with the HMD pose
|
||||
* at capture_ns (CLOCK_MONOTONIC). Writer: ft-hands (hands/track/io.cpp).
|
||||
*
|
||||
* Version 2 appends a skeleton per hand (fh_hand_fit_t) after version 1's fh_hands_t, which is
|
||||
* unchanged: a version 1 reader that maps sizeof(fh_hands_t) and checks the magic keeps
|
||||
* working. Readers of the skeleton check version >= 2 and the file's size >= sizeof(fh_hands_v2_t).
|
||||
*/
|
||||
|
||||
#pragma once
|
||||
@@ -19,7 +15,7 @@
|
||||
#include <stdint.h>
|
||||
|
||||
#define FH_HANDS_MAGIC "FHHANDS1"
|
||||
#define FH_HANDS_VERSION 2
|
||||
#define FH_HANDS_VERSION 1
|
||||
#define FH_HANDS_MAX_HANDS 2
|
||||
#define FH_HANDS_MAX_CAPSULES 64
|
||||
|
||||
@@ -57,69 +53,6 @@ typedef struct {
|
||||
fh_capsule_t capsules[FH_HANDS_MAX_CAPSULES];
|
||||
} fh_hands_t;
|
||||
|
||||
/* ---- version 2: the fitted skeleton (hands/track/skeleton.h, hands/README.md "Skeleton") ----
|
||||
*
|
||||
* Bone lengths are learned per user and side, so fitted fingers keep their length; the pose
|
||||
* (palm and joint angles) is filtered, not the points. Everything is in the head frame at
|
||||
* capture time, like pts. Joint frames follow OpenXR's hand joints: -Z along the bone that
|
||||
* starts at the joint (toward the fingertip), +Y out of the back of the hand, +X = Y x Z
|
||||
* (so +X points to the little finger on a right hand, to the thumb on a left one). The
|
||||
* wrist's frame is the palm's: -Z toward the middle knuckle, +Y the back of the hand's normal.
|
||||
* A tip has its bone's frame. Joints are indexed like MediaPipe's 21 landmarks. */
|
||||
|
||||
enum {
|
||||
FH_FIT_VALID = 1u << 0, /* this hand has a skeleton */
|
||||
FH_FIT_RIGHT = 1u << 1, /* fitted as a right hand (else left) */
|
||||
FH_FIT_SETTLED = 1u << 2, /* its side's bone lengths have settled (else partly the prior) */
|
||||
FH_FIT_FLIPPED = 1u << 3, /* fitted as the other side than the model called it */
|
||||
FH_FIT_ELBOW_HEAD_UP = 1u << 4, /* elbow from the arm model with the head's +y as up */
|
||||
};
|
||||
|
||||
typedef struct {
|
||||
float p[3]; /* position, m */
|
||||
float radius; /* rough, m: for colliders and a mesh's thickness */
|
||||
float q[4]; /* orientation x, y, z, w: the joint's frame (above) */
|
||||
} fh_joint_t; /* 32 bytes */
|
||||
|
||||
typedef struct {
|
||||
uint32_t id; /* hands[k].id; this is hands[k]'s skeleton */
|
||||
uint32_t flags; /* FH_FIT_* */
|
||||
uint64_t tracked_ns; /* CLOCK_MONOTONIC capture time of the last frame the hand was
|
||||
measured in (today always capture_ns) */
|
||||
fh_joint_t joints[21];
|
||||
float bone_len[20]; /* m; bone b ends at joint b + 1 (bones 0, 4, 8, 12, 16 start
|
||||
at the wrist) */
|
||||
float angles[26]; /* rad: thumb CMC abduction, flexion, twist, MCP abduction,
|
||||
flexion, IP flexion; then per finger (index, middle, ring,
|
||||
little) MCP abduction, MCP flexion, MCP twist, PIP, DIP.
|
||||
Flexion is positive toward the palm, abduction toward the
|
||||
thumb's side */
|
||||
float elbow[3]; /* the arm model's elbow (notes: frame-hands forearm.md) */
|
||||
float elbow_conf; /* 0: inferred by the arm model only (as now) */
|
||||
float pinch_point[3]; /* between the index and middle knuckles: the gestures file's
|
||||
pinch point, from the skeleton */
|
||||
float contact[3]; /* midway between the thumb and index tips */
|
||||
float pinch_d[2]; /* thumb tip to index tip, thumb tip to middle tip, m */
|
||||
float fit_rms; /* the fit against this frame's measured landmarks, m */
|
||||
float settled; /* 0..1: how far its side's bone-length calibration is */
|
||||
float vel_head[3]; /* palm centre velocity in the HEAD frame, m/s, filtered. It
|
||||
includes the head's own motion: turning the head reads as
|
||||
hand motion. For motion in the room, difference room-frame
|
||||
points across publishes (capture_ns) as handcut.cpp does. */
|
||||
float angvel_head[3]; /* the palm's angular velocity in the head frame, rad/s; it
|
||||
includes the head's rotation, as vel_head */
|
||||
float reserved[18];
|
||||
} fh_hand_fit_t; /* 1024 bytes */
|
||||
|
||||
typedef struct {
|
||||
fh_hands_t v1; /* version 1, unchanged; v1.version is 2 */
|
||||
fh_hand_fit_t fits[FH_HANDS_MAX_HANDS]; /* fits[k] goes with v1.hands[k]; zero when
|
||||
k >= nhands or the hand has no skeleton */
|
||||
} fh_hands_v2_t;
|
||||
|
||||
static_assert(sizeof(fh_hand_t) == 272, "fh_hand_t layout");
|
||||
static_assert(sizeof(fh_joint_t) == 32, "fh_joint_t layout");
|
||||
static_assert(sizeof(fh_hand_fit_t) == 1024, "fh_hand_fit_t layout");
|
||||
static_assert(sizeof(fh_capsule_t) == 32, "fh_capsule_t layout");
|
||||
static_assert(sizeof(fh_hands_t) == 64 + 2 * 272 + 64 * 32, "fh_hands_t layout");
|
||||
static_assert(sizeof(fh_hands_v2_t) == sizeof(fh_hands_t) + 2 * 1024, "fh_hands_v2_t layout");
|
||||
@@ -1,538 +0,0 @@
|
||||
// fh_ray - the hand ray: where a bare hand points, Quest-style, and when it counts as pointing.
|
||||
// Shared by the pointer helper (the ray drives the ft_pointer SteamVR device: pointer/helper)
|
||||
// and ft-screens (a second hand's ray for two-hand stretch), so both agree.
|
||||
// Design: frame-hands/notes/hand-interaction.md, sections 2.4 and 2.5.
|
||||
//
|
||||
// Header-only C++17, no dependencies. Everything is in the room (OpenVR's standing universe:
|
||||
// +x right, +y up, -z forward), metres, radians inside and degrees in Params, seconds on one
|
||||
// steady clock (CLOCK_MONOTONIC: ft-hands' capture times, as seconds).
|
||||
//
|
||||
// Body model (no torso tracking): the torso faces the head's yaw, low-passed (torsoTau, 0.6 s) and
|
||||
// never more than torsoMaxLag (30 deg) off the head's. With torsoWrists (on), what it faces is the
|
||||
// head's forward counted twice plus the direction from the neck to each wrist seen, levelled (the
|
||||
// formula is StereoKit's, MIT: systems/input.cpp, body_make_shoulders): you point where your body
|
||||
// faces, so looking aside while pointing turns the shoulders less, and the ray with them (with both
|
||||
// hands ahead, a third as far). On the probe-moves replay (pointer/tools/rayeval) the share of a
|
||||
// still hand's ray motion that came from the shoulder went from 20 to 18% (p90 1.7-3.6 deg/s from
|
||||
// 2.0-3.7); on the guided tap capture, where the head mostly faced the panel, nothing got worse.
|
||||
// The neck is the head moved by neck (10 cm down, 8 cm back), turned by the head's yaw only; a
|
||||
// shoulder is the neck moved by (+-shoulderX, shoulderY, 0) (17 cm out, 13 cm down), turned by the
|
||||
// torso's yaw. Head pitch and roll are ignored, so nodding doesn't swing the ray.
|
||||
//
|
||||
// Ray: from the shoulder S through the knuckle point K (the index and middle knuckles: the pinch
|
||||
// point, which holds still as the fingers close, so pinching barely turns the ray). The direction
|
||||
// is D = normalize(K - S); the origin is K + originAhead (3 cm) along D, in front of the fingers.
|
||||
// K is predicted predict (20 ms) ahead with its room velocity (differenced across frames).
|
||||
//
|
||||
// Hand direction (handWeight, 0 = off, the default): D turned toward where the hand itself points
|
||||
// by that fraction of the angle between them (StereoKit's aim, MIT: xr_backends/extensions/
|
||||
// hand_tracking.cpp, which slerps half way). The hand's axes come from the joints that hold still in
|
||||
// a thumb-index pinch (HandAxes: the wrist and the middle, ring and little knuckles), and its
|
||||
// direction is their forward (wrist to knuckles) tilted handPitch toward the back of the hand
|
||||
// (negative: the palm) and handYaw out, away from the thumb. The gate's tests still read the
|
||||
// shoulder's ray. Off because every weight measured worse (rayeval, the guided tap capture and
|
||||
// probe-moves, fitted skeleton): our palm's orientation is far noisier than K seen from the
|
||||
// shoulder 45 cm away. At 0.3 the 90 Hz judder went from 0.07-0.14 to 0.11-0.33 deg rms, a still
|
||||
// hand's wobble more than doubled, the ray's turn 150 ms into a pinch went from 0.4-0.5 to 1.2-1.7
|
||||
// deg (p50), and frames where the raw ray turned over 3 deg from 159 to 330. Where the hand's
|
||||
// direction sits against the shoulder's ray depends on how the hand is held: pitch -15 to -41, yaw
|
||||
// -4 to 19 (the recordings' medians); handPitch and handYaw are in the middle of that. StereoKit's
|
||||
// -50 and 20 suit its own axes.
|
||||
//
|
||||
// Pinch anchor (pinchAnchor, off by default): while a ray pinch is held, K is pinned to the hand's
|
||||
// axes where it was at the press, rebuilt from them each frame (StereoKit's pinch point, MIT:
|
||||
// hands/input_hand.cpp, pinned to the index finger's root while the pinch is active), and eased
|
||||
// back to the measured K on the steady shoulder's schedule (below). StereoKit pins a point between
|
||||
// the fingertips, which travels as they open; ours is the knuckles already, and on the guided tap
|
||||
// capture it held stiller than the axes do: pinned, a held pinch's point (as a near grab carries
|
||||
// it) shook as much or more (0.15-0.29 against 0.12-0.19 mm per 90 Hz tick, p50) and its release
|
||||
// kick was no smaller (1.4-2.0 against 1.2-1.6 mm, p50, palm still), from the fitted skeleton and
|
||||
// from the filtered landmarks alike. The palm's turn, times the 7 cm from its centre to K, is the
|
||||
// noise it adds.
|
||||
//
|
||||
// Pointing (Gate): the ray shows once every "show" test has held for showFor (120 ms), and hides
|
||||
// once some "keep" test has failed for hideFor (250 ms):
|
||||
// test show keep (hide past it)
|
||||
// side the arbiter says hand (and the caller (at once otherwise: "off")
|
||||
// allows it: in a game, the mouse)
|
||||
// tracked a two-camera (STEREO) frame within a frame within lostFor (250 ms); one-camera
|
||||
// stereoWithin (200 ms) depth is 2-6 cm off, fine for a short gap
|
||||
// raised K at most raisedShow (0.30 m) below raisedHide (0.40 m): gaze pinches sat
|
||||
// the eyes 0.35-0.45 m below the eyes, elbow resting
|
||||
// forward K at least forwardShow (0.25 m) ahead forwardHide (0.20 m)
|
||||
// of the eyes, along the torso's yaw
|
||||
// palm away palm normal . D >= palmShow (0.3) palmHide (0.0): palm toward you is reserved
|
||||
// forearm elbow-to-wrist pitch >= forearmShow forearmHide (-50 deg); only with an elbow
|
||||
// (-35 deg) of elbowConfMin (0.5) confidence
|
||||
// aim D's pitch in [pitchMin, pitchMax] [pitchMinHide, pitchMaxHide] (-60, 75),
|
||||
// (-45, 60), yaw within yawShow (70) yawHide (80)
|
||||
// of the head's
|
||||
// not near not near a panel (ft-screens' near near: at once (near wins)
|
||||
// claim, or the fingertip at an overlay)
|
||||
// not a fist no grip down a grip down
|
||||
// ready the thumb and index tips within readyHide (0.10 m) apart: the laser is
|
||||
// readyShow (0.06 m) (user, 2026-10-11: for pinching, not a hand held up (0: off;
|
||||
// "the aiming laser only activates when frames without the gap pass)
|
||||
// your fingers get within a distance")
|
||||
// open not a fist: HandFrame::curl (the more fistCurl - 0.05: a hand closing into a
|
||||
// open of the index and the other three fist drops its laser, so the fist's
|
||||
// fingers) at least fistCurl (1.1) thumb-to-index touch isn't a click
|
||||
// While a ray pinch is held (Hold), the ray stays up whatever the tests say, until hideFor after
|
||||
// the release. A pinch may also bring a ray up (ShowForPinch): one that hid within a moment (the
|
||||
// pointer helper's grace; not one hidden for "off", "near" or a reset), or one whose show tests
|
||||
// all pass but not yet for showFor. Closing the fingers turns the hand a little, so a ray near a
|
||||
// hide limit can drop just as the pinch begins.
|
||||
//
|
||||
// Filter (RayFilter, hands/include/fh_smooth.h): One Euro filters (Casiez, Roussel and Vogel, CHI
|
||||
// 2012) run in continuous time, so the beam moves at the display's rate rather than in steps at
|
||||
// the tracker's: the direction on the sphere (minCutoff, beta per deg/s, dCutoff), the origin
|
||||
// (originMinCutoff, originBeta per m/s): the aim matters more than where the beam starts. While a
|
||||
// ray pinch is held (a drag), the direction's minimum cutoff drops to holdMinCutoff: a slow drag
|
||||
// is steadier, a fast one still keeps up. The reach (|K - S|, push and pull's input) has its own
|
||||
// (reachMinCutoff, reachBeta per m/s): push and pull turns a reach's change into 3 times its share
|
||||
// of the panel's distance (13 times it for a panel 2 m away). The caller reads the ray at
|
||||
// the time it draws (At(now)); between tracker frames it glides toward the newest one.
|
||||
//
|
||||
// Steady shoulder (holdShoulder): while a ray pinch is held, and hideFor after, the ray's shoulder
|
||||
// stays where it was in the room at the press; then it eases back to the body model's over
|
||||
// shoulderBlend (0.5 s). The model's shoulder moves with the head (the neck follows its position,
|
||||
// the torso its yaw): on a recording, with the hand still, a fifth of the ray's motion came from
|
||||
// there, up to 2-4 deg/s (p90). Looking around while dragging moved what was dragged. The pinch
|
||||
// anchor (when on) holds and eases back with it.
|
||||
#pragma once
|
||||
|
||||
#include "fh_smooth.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <cstdio>
|
||||
#include <string>
|
||||
|
||||
namespace fhray {
|
||||
|
||||
using fhsmooth::Dot;
|
||||
using fhsmooth::Len;
|
||||
using fhsmooth::Norm;
|
||||
using fhsmooth::V3;
|
||||
constexpr double kDeg = M_PI / 180;
|
||||
// Yaw: 0 along -z, positive turning left (about +y), as OpenVR; pitch: up positive.
|
||||
inline double YawOf(V3 d) { return std::atan2(-d.x, -d.z); }
|
||||
inline double PitchOf(V3 d) { return std::asin(std::clamp(Norm(d).y, -1.0, 1.0)); }
|
||||
inline V3 Dir(double yaw, double pitch) {
|
||||
return {-std::sin(yaw) * std::cos(pitch), std::sin(pitch), -std::cos(yaw) * std::cos(pitch)};
|
||||
}
|
||||
// v turned by yaw about +y.
|
||||
inline V3 TurnYaw(V3 v, double yaw) {
|
||||
const double c = std::cos(yaw), s = std::sin(yaw);
|
||||
return {v.x * c + v.z * s, v.y, -v.x * s + v.z * c};
|
||||
}
|
||||
inline double WrapPi(double a) { return std::remainder(a, 2 * M_PI); }
|
||||
|
||||
struct Params {
|
||||
// body
|
||||
double torsoTau = 0.6, torsoMaxLag = 30;
|
||||
bool torsoWrists = true;
|
||||
V3 neck{0, -0.10, 0.08};
|
||||
double shoulderX = 0.17, shoulderY = -0.13;
|
||||
// ray
|
||||
double originAhead = 0.03, predict = 0.020;
|
||||
double handWeight = 0, handPitch = -30, handYaw = 8;
|
||||
bool pinchAnchor = false;
|
||||
// gate
|
||||
double showFor = 0.12, hideFor = 0.25;
|
||||
double stereoWithin = 0.2, lostFor = 0.25;
|
||||
double raisedShow = 0.30, raisedHide = 0.40;
|
||||
double forwardShow = 0.25, forwardHide = 0.20;
|
||||
double palmShow = 0.3, palmHide = 0.0;
|
||||
double forearmShow = -35, forearmHide = -50, elbowConfMin = 0.5;
|
||||
double pitchMin = -45, pitchMax = 60, pitchMinHide = -60, pitchMaxHide = 75;
|
||||
double yawShow = 70, yawHide = 80;
|
||||
double readyShow = 0.06, readyHide = 0.10; // the thumb and index tips apart (0: no "ready" test)
|
||||
double fistCurl = 1.1; // HandFrame::curl under this: a fist (0: no "open" test)
|
||||
// filter (see the top)
|
||||
double minCutoff = 0.8, beta = 0.06, dCutoff = 1.0, originMinCutoff = 1.5, originBeta = 0.5;
|
||||
double holdMinCutoff = 0.5;
|
||||
bool holdShoulder = true;
|
||||
double shoulderBlend = 0.5;
|
||||
double reachMinCutoff = 1.0, reachBeta = 5.0;
|
||||
};
|
||||
|
||||
// The torso and shoulders, from the head (and the wrists: torsoWrists).
|
||||
class Body {
|
||||
public:
|
||||
explicit Body(const Params &p = {}) : p_(p) {}
|
||||
void SetParams(const Params &p) { p_ = p; }
|
||||
// The head's position and forward direction (-z of its pose) at time t, and the wrists seen in
|
||||
// the same frame (none: nullptr, n 0).
|
||||
void Update(double t, V3 head, V3 forward, const V3 *wrists = nullptr, int n = 0) {
|
||||
head_ = head;
|
||||
headYaw_ = YawOf(forward);
|
||||
// what the torso turns toward: the head's yaw, or with the wrists (see the top)
|
||||
double target = headYaw_;
|
||||
if (p_.torsoWrists && n > 0) {
|
||||
V3 face = Dir(headYaw_, 0) * 2;
|
||||
for (int i = 0; i < n; ++i) face = face + Norm(wrists[i] - Neck());
|
||||
face.y = 0;
|
||||
if (Len(face) > 0.1) target = YawOf(face);
|
||||
}
|
||||
if (!init_) {
|
||||
torso_ = target, t_ = t, init_ = true;
|
||||
} else {
|
||||
const double dt = std::clamp(t - t_, 0.0, 0.5);
|
||||
t_ = t;
|
||||
const double lag = WrapPi(target - torso_) * std::exp(-dt / std::max(1e-3, p_.torsoTau));
|
||||
torso_ = WrapPi(target - lag);
|
||||
}
|
||||
const double max = p_.torsoMaxLag * kDeg;
|
||||
torso_ = WrapPi(headYaw_ - std::clamp(WrapPi(headYaw_ - torso_), -max, max));
|
||||
}
|
||||
bool Ready() const { return init_; }
|
||||
V3 Head() const { return head_; }
|
||||
double HeadYaw() const { return headYaw_; }
|
||||
double TorsoYaw() const { return torso_; }
|
||||
V3 TorsoForward() const { return Dir(torso_, 0); }
|
||||
V3 Neck() const { return head_ + TurnYaw(p_.neck, headYaw_); }
|
||||
V3 Shoulder(int side) const {
|
||||
return Neck() + TurnYaw(V3{side ? p_.shoulderX : -p_.shoulderX, p_.shoulderY, 0}, torso_);
|
||||
}
|
||||
|
||||
private:
|
||||
Params p_;
|
||||
V3 head_;
|
||||
double headYaw_ = 0, torso_ = 0, t_ = 0;
|
||||
bool init_ = false;
|
||||
};
|
||||
|
||||
struct Ray {
|
||||
V3 origin, dir{0, 0, -1};
|
||||
};
|
||||
|
||||
// A hand's axes from the joints that hold still in a thumb-index pinch: the wrist and the middle,
|
||||
// ring and little knuckles (MediaPipe 0, 9, 13, 17). f: forward, from the wrist to the ring and
|
||||
// little knuckles; u: out of the back of the hand; r = f x u (toward the little finger on a right
|
||||
// hand palm away, toward the thumb on a left one: to the right on both). o: their centre. side: 0
|
||||
// left, 1 right. Built as StereoKit's hand orientation (MIT: xr_backends/extensions/hand_tracking.cpp),
|
||||
// with the middle knuckle for the index one and the wrist for the metacarpals' roots.
|
||||
struct Axes {
|
||||
V3 o, r, u, f;
|
||||
bool ok = false;
|
||||
};
|
||||
inline Axes HandAxes(V3 wrist, V3 middle, V3 ring, V3 little, int side) {
|
||||
Axes a;
|
||||
const V3 fwd = (ring + little) * 0.5 - wrist, across = (little - middle) * (side ? 1.0 : -1.0);
|
||||
const V3 back = Cross(across, fwd);
|
||||
if (Len(fwd) < 0.02 || Len(across) < 0.005 || Len(back) < 1e-6) return a; // a collapsed hand
|
||||
a.f = Norm(fwd);
|
||||
a.u = Norm(back - a.f * Dot(back, a.f));
|
||||
a.r = Cross(a.f, a.u);
|
||||
a.o = (wrist + middle + ring + little) * 0.25;
|
||||
a.ok = true;
|
||||
return a;
|
||||
}
|
||||
inline V3 ToAxes(const Axes &a, V3 p) {
|
||||
const V3 d = p - a.o;
|
||||
return {Dot(d, a.r), Dot(d, a.u), Dot(d, a.f)};
|
||||
}
|
||||
inline V3 FromAxes(const Axes &a, V3 l) { return a.o + a.r * l.x + a.u * l.y + a.f * l.z; }
|
||||
// Where the hand points (see "Hand direction" at the top): the axes' forward turned pitch degrees
|
||||
// toward the back of the hand (negative: toward the palm) and yaw degrees out, away from the thumb.
|
||||
// StereoKit's is pitch -50, yaw 20 from its own axes (MIT, quat_from_angles(-50, +-20, 0)).
|
||||
inline V3 HandDir(const Axes &a, int side, double pitch, double yaw) {
|
||||
const double p = pitch * kDeg, y = (side ? -yaw : yaw) * kDeg;
|
||||
return Norm(a.r * (-std::cos(p) * std::sin(y)) + a.u * std::sin(p) + a.f * (std::cos(p) * std::cos(y)));
|
||||
}
|
||||
|
||||
// A point pinned to a hand's axes (see "Pinch anchor" at the top): Pin it, then At rebuilds it from
|
||||
// the axes now, weighted against the point as measured (1: pinned, 0: as measured, between: easing
|
||||
// back). Axes that can't be built leave it as measured.
|
||||
class Anchor {
|
||||
public:
|
||||
void Pin(const Axes &a, V3 p) {
|
||||
pinned_ = a.ok;
|
||||
if (a.ok) local_ = ToAxes(a, p);
|
||||
}
|
||||
void Unpin() { pinned_ = false; }
|
||||
bool Pinned() const { return pinned_; }
|
||||
V3 At(const Axes &a, V3 measured, double weight) const {
|
||||
if (!pinned_ || !a.ok || weight <= 0) return measured;
|
||||
return measured + (FromAxes(a, local_) - measured) * std::min(weight, 1.0);
|
||||
}
|
||||
|
||||
private:
|
||||
V3 local_;
|
||||
bool pinned_ = false;
|
||||
};
|
||||
|
||||
// One hand at one capture time, in the room.
|
||||
struct HandFrame {
|
||||
double t = 0; // capture time, s
|
||||
bool stereo = false; // triangulated from two or more cameras
|
||||
V3 knuckle; // K: the index and middle knuckles
|
||||
V3 wrist;
|
||||
V3 middle, ring, little; // their knuckles (palm: set), for the hand's axes
|
||||
bool palm = false;
|
||||
V3 palmNormal; // unit, out of the palm
|
||||
V3 elbow;
|
||||
double elbowConf = 0; // 0: the arm model's guess
|
||||
bool grip = false; // a grip (closed hand) is down
|
||||
bool near = false; // near a panel (see "not near" at the top)
|
||||
double gap = -1; // the thumb and index tips apart (m); < 0 unknown (the "ready" test passes)
|
||||
double curl = -1; // how open the hand is (FistCurl): about 1.9 open, under 1.1 a fist; < 0 unknown
|
||||
};
|
||||
|
||||
// HandFrame::curl from MediaPipe's 21 landmarks (any 3D type with operator-, and len its length): a
|
||||
// finger's tip-to-wrist distance over its knuckle's; the larger of the index's and the mean of the
|
||||
// middle, ring and little's. A fist has both low; a pinch with the other fingers curled keeps the index
|
||||
// out (the flags of 2026-10-09..11, 47k hands: the other three alone read under 1.25 in a fifth of the
|
||||
// pinches with the index out; both under 1.1 in 70% of the frames with the index curled, a fist).
|
||||
template <class P, class L> double FistCurl(const P *pts, L len) {
|
||||
double sum = 0;
|
||||
for (int tip : {12, 16, 20}) {
|
||||
const double k = len(pts[tip - 3] - pts[0]);
|
||||
if (k < 1e-4) return -1;
|
||||
sum += len(pts[tip] - pts[0]) / k;
|
||||
}
|
||||
const double k = len(pts[5] - pts[0]);
|
||||
if (k < 1e-4) return -1;
|
||||
return std::max(sum / 3, len(pts[8] - pts[0]) / k);
|
||||
}
|
||||
|
||||
// What each test said for one frame (degrees and metres, for logs and the offline evaluation).
|
||||
struct Tests {
|
||||
double below = 0, ahead = 0, palm = 0, forearm = 0, pitch = 0, yawOff = 0;
|
||||
bool forearmUsed = false;
|
||||
bool show = false, keep = false; // every show test, every keep test (not counting side, tracking)
|
||||
std::string fails; // the show tests that failed
|
||||
};
|
||||
|
||||
inline Tests Evaluate(const Params &p, const Body &body, const HandFrame &f, const Ray &ray) {
|
||||
Tests r;
|
||||
const V3 eye = body.Head();
|
||||
r.below = eye.y - f.knuckle.y;
|
||||
const V3 fwd = body.TorsoForward();
|
||||
r.ahead = Dot(f.knuckle - eye, fwd);
|
||||
r.palm = Dot(f.palmNormal, ray.dir);
|
||||
r.pitch = PitchOf(ray.dir) / kDeg;
|
||||
r.yawOff = std::fabs(WrapPi(YawOf(ray.dir) - body.HeadYaw())) / kDeg;
|
||||
r.forearmUsed = f.elbowConf >= p.elbowConfMin;
|
||||
if (r.forearmUsed) r.forearm = PitchOf(f.wrist - f.elbow) / kDeg;
|
||||
bool show = true, keep = true;
|
||||
auto test = [&](bool s, bool k, const char *name) {
|
||||
if (!s) r.fails += r.fails.empty() ? name : std::string(",") + name;
|
||||
show = show && s, keep = keep && k;
|
||||
};
|
||||
test(r.below <= p.raisedShow, r.below <= p.raisedHide, "raised");
|
||||
test(r.ahead >= p.forwardShow, r.ahead >= p.forwardHide, "forward");
|
||||
test(r.palm >= p.palmShow, r.palm >= p.palmHide, "palm");
|
||||
if (r.forearmUsed) test(r.forearm >= p.forearmShow, r.forearm >= p.forearmHide, "forearm");
|
||||
test(r.pitch >= p.pitchMin && r.pitch <= p.pitchMax, r.pitch >= p.pitchMinHide && r.pitch <= p.pitchMaxHide, "aim");
|
||||
test(r.yawOff <= p.yawShow, r.yawOff <= p.yawHide, "yaw");
|
||||
test(!f.grip, !f.grip, "grip");
|
||||
if (p.readyShow > 0 && f.gap >= 0) test(f.gap <= p.readyShow, f.gap <= std::max(p.readyShow, p.readyHide), "ready");
|
||||
if (p.fistCurl > 0 && f.curl >= 0) test(f.curl >= p.fistCurl, f.curl >= p.fistCurl - 0.05, "fist");
|
||||
r.show = show, r.keep = keep;
|
||||
return r;
|
||||
}
|
||||
|
||||
// The ray's filter (see the top).
|
||||
class RayFilter {
|
||||
public:
|
||||
explicit RayFilter(const Params &p = {}) { SetParams(p); }
|
||||
void SetParams(const Params &p) {
|
||||
p_ = p;
|
||||
dir_.Set({held_ ? p.holdMinCutoff : p.minCutoff, p.beta, p.dCutoff});
|
||||
origin_.Set({p.originMinCutoff, p.originBeta, p.dCutoff});
|
||||
reach_.Set({p.reachMinCutoff, p.reachBeta, p.dCutoff});
|
||||
}
|
||||
// A ray pinch held: the direction's precision (holdMinCutoff), from now on.
|
||||
void SetHeld(bool on) {
|
||||
if (on == held_) return;
|
||||
held_ = on;
|
||||
SetParams(p_);
|
||||
}
|
||||
void Reset() { dir_.Reset(), origin_.Reset(), reach_.Reset(); }
|
||||
// A tracker frame's ray and reach, captured at t, arrived at now.
|
||||
void Sample(const Ray &raw, double reach, double t, double now) {
|
||||
dir_.Sample(raw.dir, t, now), origin_.Sample(raw.origin, t, now), reach_.Sample(reach, t, now);
|
||||
}
|
||||
// The ray at now (the filters' timeline moves there).
|
||||
Ray At(double now) {
|
||||
Ray r;
|
||||
r.dir = dir_.At(now), r.origin = origin_.At(now);
|
||||
reach_.At(now);
|
||||
return r;
|
||||
}
|
||||
Ray Last() const {
|
||||
Ray r;
|
||||
r.dir = dir_.Last(), r.origin = origin_.Last();
|
||||
return r;
|
||||
}
|
||||
double Reach() const { return reach_.Last(); }
|
||||
|
||||
private:
|
||||
Params p_;
|
||||
fhsmooth::Direction dir_;
|
||||
fhsmooth::Point3 origin_;
|
||||
fhsmooth::Scalar reach_;
|
||||
bool held_ = false;
|
||||
};
|
||||
|
||||
// One side's ray: prediction, the gate and the filter.
|
||||
class HandRay {
|
||||
public:
|
||||
explicit HandRay(int side, const Params &p = {}) : side_(side), p_(p), filter_(p) {}
|
||||
void SetParams(const Params &p) { p_ = p, filter_.SetParams(p); }
|
||||
int Side() const { return side_; }
|
||||
|
||||
// A new frame of this side's hand. sideIsHand: the ray may show (the arbiter says this side is a
|
||||
// bare hand, and the caller allows it); without it, a ray shown hides at once ("hide off").
|
||||
// now: when it arrived, on the capture times' clock (< 0: at its capture time).
|
||||
void Frame(const Body &body, const HandFrame &f, bool sideIsHand, double now = -1) {
|
||||
if (now < 0) now = f.t;
|
||||
const bool wasShown = shown_;
|
||||
// K: pinned to the hand's axes during a pinch (see "Pinch anchor" at the top)
|
||||
axes_ = f.palm ? HandAxes(f.wrist, f.middle, f.ring, f.little, side_) : Axes{};
|
||||
lastMeasured_ = f.knuckle;
|
||||
if (pinAt_) anchor_.Pin(axes_, f.knuckle), pinAt_ = false;
|
||||
const double pinned = Steady(f.t);
|
||||
const V3 kNow = anchor_.At(axes_, f.knuckle, pinned);
|
||||
if (pinned <= 0) anchor_.Unpin();
|
||||
// K's room velocity, for the prediction
|
||||
V3 vk;
|
||||
if (haveLast_ && f.t > lastT_ && f.t - lastT_ < 0.2) vk = (kNow - lastK_) * (1 / (f.t - lastT_));
|
||||
lastK_ = kNow, lastT_ = f.t, haveLast_ = true;
|
||||
const V3 k = kNow + vk * p_.predict;
|
||||
const V3 s = Shoulder(body, f.t);
|
||||
raw_.dir = Norm(k - s);
|
||||
raw_.origin = k + raw_.dir * p_.originAhead;
|
||||
reach_ = Len(kNow - s);
|
||||
lastFrame_ = f.t, arrived_ = now;
|
||||
if (f.stereo) lastStereo_ = f.t;
|
||||
tests_ = Evaluate(p_, body, f, raw_); // the shoulder's ray: the tests were tuned on it
|
||||
// turned toward the hand's own direction (see "Hand direction" at the top)
|
||||
if (p_.handWeight > 0 && axes_.ok) {
|
||||
const V3 hand = HandDir(axes_, side_, p_.handPitch, p_.handYaw);
|
||||
raw_.dir = fhsmooth::Slerp(raw_.dir, hand, std::min(p_.handWeight, 1.0));
|
||||
raw_.origin = k + raw_.dir * p_.originAhead;
|
||||
}
|
||||
const bool tracked = f.t - lastStereo_ <= p_.stereoWithin;
|
||||
const bool showOk = sideIsHand && tracked && tests_.show && !f.near;
|
||||
const bool keepOk = tests_.keep;
|
||||
if (!tests_.fails.empty()) why_ = tests_.fails;
|
||||
else if (!tracked) why_ = "one camera";
|
||||
else if (f.near) why_ = "near";
|
||||
else why_ = "";
|
||||
if (!sideIsHand) why_ = "off";
|
||||
// the timers run on capture times
|
||||
if (!shown_) {
|
||||
if (showOk) {
|
||||
if (showSince_ < 0) showSince_ = f.t;
|
||||
if (f.t - showSince_ >= p_.showFor - 1e-9) Show();
|
||||
} else {
|
||||
showSince_ = -1;
|
||||
}
|
||||
} else {
|
||||
if (!sideIsHand) {
|
||||
Hide("off", f.t);
|
||||
} else if (f.near && !held_) {
|
||||
Hide("near", f.t);
|
||||
} else if (keepOk || held_ || f.t < holdUntil_) {
|
||||
failSince_ = -1;
|
||||
} else {
|
||||
if (failSince_ < 0) failSince_ = f.t;
|
||||
if (f.t - failSince_ >= p_.hideFor - 1e-9) Hide(tests_.fails.empty() ? "pose" : tests_.fails, f.t);
|
||||
}
|
||||
}
|
||||
if (shown_ && wasShown) filter_.Sample(raw_, reach_, f.t, now); // (Show took this frame already)
|
||||
}
|
||||
// Each tick, with the time now on the capture clock: the hand lost (no frame for lostFor) hides it.
|
||||
void Tick(double now, bool sideIsHand) {
|
||||
if (!shown_) return;
|
||||
if (!sideIsHand) Hide("off", now);
|
||||
else if (!held_ && now - lastFrame_ > p_.lostFor && now >= holdUntil_) Hide("lost", now);
|
||||
}
|
||||
// A ray pinch is held: the ray stays up until hideFor after it ends, and its filter takes the
|
||||
// drag's precision (holdMinCutoff) meanwhile.
|
||||
void Hold(bool on, double t) {
|
||||
if (held_ && !on) holdUntil_ = t + p_.hideFor;
|
||||
if (on && !held_) {
|
||||
if (p_.holdShoulder) steady_ = true, steadyAt_ = lastS_;
|
||||
// K pinned where it was in the newest frame, against the axes then (or the next frame's)
|
||||
if (p_.pinchAnchor) {
|
||||
if (axes_.ok) anchor_.Pin(axes_, lastMeasured_);
|
||||
else pinAt_ = true;
|
||||
}
|
||||
}
|
||||
held_ = on;
|
||||
filter_.SetHeld(on);
|
||||
}
|
||||
void Reset() { Hide("reset", -1e9); }
|
||||
// A pinch began on this hand at t: may its ray show for it (see the top)? within: how long ago
|
||||
// it may have hidden.
|
||||
bool CanShowForPinch(double t, double within) const {
|
||||
return shown_ || showSince_ >= 0 || (revivable_ && t - hidAt_ <= within);
|
||||
}
|
||||
// Show it (filter started afresh from the hand now), for a pinch that may (CanShowForPinch).
|
||||
void ShowForPinch() {
|
||||
if (!shown_) Show();
|
||||
}
|
||||
|
||||
bool Shown() const { return shown_; }
|
||||
Ray Current() const { return filter_.Last(); } // filtered, where its timeline stands (the last frame or At)
|
||||
Ray At(double now) { return filter_.At(now); } // filtered, at now (between frames: on its way to the newest)
|
||||
const Ray &Raw() const { return raw_; } // unfiltered, predicted
|
||||
double Reach() const { return filter_.Reach(); } // |K - S|, filtered: arm out or in (push and pull)
|
||||
double RawReach() const { return reach_; }
|
||||
const Tests &LastTests() const { return tests_; }
|
||||
const std::string &Why() const { return why_; } // what fails now ("" if nothing)
|
||||
// Changes since the caller last cleared it: "show" or "hide <reason>".
|
||||
std::string event;
|
||||
|
||||
private:
|
||||
// How much of a held pinch's steadiness is left at t: 1 while it's held and hideFor after, then
|
||||
// down to 0 over shoulderBlend (see "Steady shoulder" at the top).
|
||||
double Steady(double t) const {
|
||||
if (held_) return 1;
|
||||
return 1 - std::clamp((t - holdUntil_) / std::max(1e-3, p_.shoulderBlend), 0.0, 1.0);
|
||||
}
|
||||
// The ray's shoulder (see "Steady shoulder" at the top).
|
||||
V3 Shoulder(const Body &body, double t) {
|
||||
V3 s = body.Shoulder(side_);
|
||||
if (steady_) {
|
||||
const double w = Steady(t);
|
||||
if (w <= 0) steady_ = false;
|
||||
else s = steadyAt_ + (s - steadyAt_) * (1 - w);
|
||||
}
|
||||
lastS_ = s;
|
||||
return s;
|
||||
}
|
||||
void Show() {
|
||||
shown_ = true, failSince_ = -1, showSince_ = -1;
|
||||
filter_.Reset();
|
||||
filter_.Sample(raw_, reach_, lastFrame_, arrived_);
|
||||
event = "show";
|
||||
}
|
||||
void Hide(const std::string &why, double t) {
|
||||
if (!shown_) return;
|
||||
shown_ = false, failSince_ = -1, showSince_ = -1, held_ = false, holdUntil_ = 0, steady_ = false;
|
||||
anchor_.Unpin(), pinAt_ = false;
|
||||
filter_.SetHeld(false);
|
||||
hidAt_ = t, revivable_ = why != "off" && why != "near" && why != "reset";
|
||||
event = "hide " + why;
|
||||
}
|
||||
int side_;
|
||||
Params p_;
|
||||
RayFilter filter_;
|
||||
Ray raw_;
|
||||
Tests tests_;
|
||||
std::string why_;
|
||||
V3 lastK_, lastS_, steadyAt_, lastMeasured_;
|
||||
Axes axes_;
|
||||
Anchor anchor_;
|
||||
bool pinAt_ = false; // pin at the next frame (no axes when the pinch began)
|
||||
double lastT_ = 0, lastFrame_ = -1e9, lastStereo_ = -1e9, reach_ = 0, arrived_ = 0;
|
||||
double showSince_ = -1, failSince_ = -1, holdUntil_ = 0, hidAt_ = -1e9;
|
||||
bool haveLast_ = false, shown_ = false, held_ = false, revivable_ = false, steady_ = false;
|
||||
};
|
||||
|
||||
} // namespace fhray
|
||||
@@ -1,265 +0,0 @@
|
||||
// fh_smooth - smoothing for what a hand moves: One Euro filters (Casiez, Roussel and Vogel, CHI
|
||||
// 2012) run in continuous time, so a hand tracked at 15-30 Hz moves things smoothly at the
|
||||
// display's rate instead of in steps. Shared by the pointer helper (the hand ray, through
|
||||
// hands/include/fh_ray.h) and ft-screens (near grabs, screens/handinput.cpp).
|
||||
// Design: frame-hands/notes/hand-interaction.md, section 13.
|
||||
//
|
||||
// The One Euro filter is a first-order low-pass whose cutoff rises with speed: minCutoff (Hz)
|
||||
// still, plus beta per unit of speed (the speed itself low-passed at dCutoff). Slow and still:
|
||||
// heavy smoothing, so jitter goes; fast: little, so it keeps up. Here it runs in continuous
|
||||
// time: each tracker sample sets the cutoff and where the input goes, and At(now) is the
|
||||
// low-pass's output at now, however often the caller asks. The input slides from the sample
|
||||
// before to the newest over one frame interval (the capture times' gap, at most 0.1 s), so a
|
||||
// steady move comes out as a steady move: held still, the input made the output catch up and
|
||||
// stall each frame, steps again at a high cutoff. The slide costs half a frame of delay
|
||||
// (17 ms at 30 Hz) over holding the sample. The output never passes the newest sample (no
|
||||
// overshoot); the classic discrete filter, read at the display's rate, holds still and then
|
||||
// jumps a whole step each sample. No prediction past the newest sample: the ray and a carried
|
||||
// panel stay behind the hand by the tracking latency plus the filter's lag, never ahead of it.
|
||||
//
|
||||
// Point<N>: N values (1 for a distance, 3 for a point); the speed is the length of the
|
||||
// change, so it smooths the same along every axis. Direction: a unit vector, filtered on the
|
||||
// sphere (it turns along the great circle toward the newest sample; speed in degrees per
|
||||
// second), with no angle to wrap and no trouble pointing straight up.
|
||||
//
|
||||
// Times are seconds on one steady clock. Each sample has its capture time (for the speed) and
|
||||
// the time it arrived (now: where the output's timeline stands). Header-only C++17, no
|
||||
// dependencies.
|
||||
#pragma once
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
|
||||
namespace fhsmooth {
|
||||
|
||||
struct V3 {
|
||||
double x = 0, y = 0, z = 0;
|
||||
};
|
||||
inline V3 operator+(V3 a, V3 b) { return {a.x + b.x, a.y + b.y, a.z + b.z}; }
|
||||
inline V3 operator-(V3 a, V3 b) { return {a.x - b.x, a.y - b.y, a.z - b.z}; }
|
||||
inline V3 operator*(V3 a, double s) { return {a.x * s, a.y * s, a.z * s}; }
|
||||
inline double Dot(V3 a, V3 b) { return a.x * b.x + a.y * b.y + a.z * b.z; }
|
||||
inline V3 Cross(V3 a, V3 b) { return {a.y * b.z - a.z * b.y, a.z * b.x - a.x * b.z, a.x * b.y - a.y * b.x}; }
|
||||
inline double Len(V3 a) { return std::sqrt(Dot(a, a)); }
|
||||
inline V3 Norm(V3 a) {
|
||||
const double l = Len(a);
|
||||
return l > 1e-12 ? a * (1 / l) : V3{0, 0, -1};
|
||||
}
|
||||
// The angle between two unit vectors, radians (accurate near 0, unlike acos).
|
||||
inline double Angle(V3 a, V3 b) { return std::atan2(Len(Cross(a, b)), Dot(a, b)); }
|
||||
// a turned toward b (unit vectors) by the fraction k of the angle between them.
|
||||
inline V3 Slerp(V3 a, V3 b, double k) {
|
||||
const double w = Angle(a, b);
|
||||
if (w < 1e-9) return b;
|
||||
if (w > M_PI - 1e-6) return k < 0.5 ? a : b; // opposite: no one way round
|
||||
const double s = std::sin(w);
|
||||
return Norm(a * (std::sin((1 - k) * w) / s) + b * (std::sin(k * w) / s));
|
||||
}
|
||||
|
||||
struct Params {
|
||||
double minCutoff = 1; // Hz, still
|
||||
double beta = 0; // Hz more per unit of speed (per m/s for points, per deg/s for directions)
|
||||
double dCutoff = 1; // Hz: the speed's own low-pass
|
||||
double maxCutoff = 0; // Hz: at most this (0: no limit), however fast it moves
|
||||
};
|
||||
inline double CutoffAt(const Params &p, double speed) {
|
||||
const double c = p.minCutoff + p.beta * speed;
|
||||
return p.maxCutoff > 0 ? std::min(c, p.maxCutoff) : c;
|
||||
}
|
||||
|
||||
namespace detail {
|
||||
// The discrete low-pass's weight for a step dt at this cutoff (the derivative's, as One Euro's).
|
||||
inline double Alpha(double cutoff, double dt) { return 1 / (1 + 1 / (2 * M_PI * cutoff * dt)); }
|
||||
// How far a continuous first-order low-pass at this cutoff gets toward its input in dt.
|
||||
inline double Toward(double cutoff, double dt) { return 1 - std::exp(-2 * M_PI * cutoff * dt); }
|
||||
} // namespace detail
|
||||
|
||||
// The input's slide from the sample before to the newest (see the top), and the low-pass's steps
|
||||
// along it (at most kStep: the input moves within each).
|
||||
constexpr double kStep = 0.004, kMaxSlide = 0.1;
|
||||
|
||||
template <int N>
|
||||
class Point {
|
||||
public:
|
||||
explicit Point(const Params &p = {}) : p_(p) {}
|
||||
// New settings (a held drag's precision, say): the cutoff changes from now, the output doesn't jump.
|
||||
void Set(const Params &p) {
|
||||
p_ = p;
|
||||
cutoff_ = CutoffAt(p_, speed_);
|
||||
}
|
||||
const Params &Get() const { return p_; }
|
||||
void Reset() { init_ = false; }
|
||||
bool Ready() const { return init_; }
|
||||
// A tracker sample, captured at tc, arrived at now (pass tc for both when there's no clock to
|
||||
// tell them apart). The first one, or one after Reset, is taken as it is.
|
||||
void Sample(const double v[N], double tc, double now) {
|
||||
if (!init_) {
|
||||
std::copy(v, v + N, x_), std::copy(v, v + N, r_), std::copy(v, v + N, from_);
|
||||
std::fill(d_, d_ + N, 0.0);
|
||||
speed_ = 0, tc_ = tc, t_ = start_ = now, slide_ = 0, init_ = true;
|
||||
cutoff_ = p_.minCutoff;
|
||||
return;
|
||||
}
|
||||
Advance(now);
|
||||
const double dt = tc - tc_;
|
||||
if (dt > 1e-4) {
|
||||
// the speed from the raw samples (as the reference implementation), low-passed
|
||||
const double a = detail::Alpha(p_.dCutoff, std::min(dt, 0.5));
|
||||
double s2 = 0;
|
||||
for (int i = 0; i < N; ++i) {
|
||||
d_[i] += a * ((v[i] - r_[i]) / dt - d_[i]);
|
||||
s2 += d_[i] * d_[i];
|
||||
}
|
||||
speed_ = std::sqrt(s2);
|
||||
tc_ = tc;
|
||||
}
|
||||
Input(now, from_); // the slide starts where the input is now
|
||||
std::copy(v, v + N, r_);
|
||||
start_ = now, slide_ = std::clamp(dt, 0.0, kMaxSlide);
|
||||
cutoff_ = CutoffAt(p_, speed_);
|
||||
}
|
||||
// The output at now (moves the filter's timeline there; a time before it reads the output as it is).
|
||||
const double *At(double now) {
|
||||
Advance(now);
|
||||
return x_;
|
||||
}
|
||||
// After Reset and a first Sample: the speed it's moving at (units per second, per axis), so it
|
||||
// follows a moving input at once instead of from rest (a hand back after a gap, at the gap's speed).
|
||||
void Seed(const double rate[N]) {
|
||||
double s2 = 0;
|
||||
for (int i = 0; i < N; ++i) d_[i] = rate[i], s2 += rate[i] * rate[i];
|
||||
speed_ = std::sqrt(s2), cutoff_ = CutoffAt(p_, speed_);
|
||||
}
|
||||
const double *Last() const { return x_; } // the output where the timeline stands
|
||||
const double *Raw() const { return r_; } // the newest sample
|
||||
double Speed() const { return speed_; } // low-passed, units per second
|
||||
double Cutoff() const { return cutoff_; }
|
||||
|
||||
private:
|
||||
void Input(double t, double out[N]) const {
|
||||
const double u = slide_ > 0 ? std::clamp((t - start_) / slide_, 0.0, 1.0) : 1.0;
|
||||
for (int i = 0; i < N; ++i) out[i] = from_[i] + u * (r_[i] - from_[i]);
|
||||
}
|
||||
void Advance(double now) {
|
||||
if (now <= t_) return;
|
||||
if (now - t_ > 1) t_ = now - 1;
|
||||
while (t_ < now) {
|
||||
const double h = std::min(kStep, now - t_);
|
||||
t_ += h;
|
||||
double in[N];
|
||||
Input(t_, in);
|
||||
const double k = detail::Toward(cutoff_, h);
|
||||
for (int i = 0; i < N; ++i) x_[i] += k * (in[i] - x_[i]);
|
||||
}
|
||||
}
|
||||
Params p_;
|
||||
double x_[N] = {}, r_[N] = {}, from_[N] = {}, d_[N] = {};
|
||||
double speed_ = 0, cutoff_ = 1, tc_ = 0, t_ = 0, start_ = 0, slide_ = 0;
|
||||
bool init_ = false;
|
||||
};
|
||||
|
||||
// A point in 3D (V3 in and out).
|
||||
class Point3 {
|
||||
public:
|
||||
explicit Point3(const Params &p = {}) : f_(p) {}
|
||||
void Set(const Params &p) { f_.Set(p); }
|
||||
const Params &Get() const { return f_.Get(); }
|
||||
void Reset() { f_.Reset(); }
|
||||
bool Ready() const { return f_.Ready(); }
|
||||
void Sample(V3 v, double tc, double now) {
|
||||
const double a[3] = {v.x, v.y, v.z};
|
||||
f_.Sample(a, tc, now);
|
||||
}
|
||||
V3 At(double now) {
|
||||
const double *x = f_.At(now);
|
||||
return {x[0], x[1], x[2]};
|
||||
}
|
||||
V3 Last() const { return {f_.Last()[0], f_.Last()[1], f_.Last()[2]}; }
|
||||
double Speed() const { return f_.Speed(); }
|
||||
void Seed(V3 rate) {
|
||||
const double a[3] = {rate.x, rate.y, rate.z};
|
||||
f_.Seed(a);
|
||||
}
|
||||
|
||||
private:
|
||||
Point<3> f_;
|
||||
};
|
||||
|
||||
// One value (a distance).
|
||||
class Scalar {
|
||||
public:
|
||||
explicit Scalar(const Params &p = {}) : f_(p) {}
|
||||
void Set(const Params &p) { f_.Set(p); }
|
||||
void Reset() { f_.Reset(); }
|
||||
bool Ready() const { return f_.Ready(); }
|
||||
void Sample(double v, double tc, double now) { f_.Sample(&v, tc, now); }
|
||||
double At(double now) { return *f_.At(now); }
|
||||
double Last() const { return *f_.Last(); }
|
||||
|
||||
private:
|
||||
Point<1> f_;
|
||||
};
|
||||
|
||||
// A direction (unit vector), on the sphere; beta per deg/s.
|
||||
class Direction {
|
||||
public:
|
||||
explicit Direction(const Params &p = {}) : p_(p) {}
|
||||
void Set(const Params &p) {
|
||||
p_ = p;
|
||||
cutoff_ = CutoffAt(p_, speed_);
|
||||
}
|
||||
const Params &Get() const { return p_; }
|
||||
void Reset() { init_ = false; }
|
||||
bool Ready() const { return init_; }
|
||||
void Sample(V3 v, double tc, double now) {
|
||||
v = Norm(v);
|
||||
if (!init_) {
|
||||
x_ = r_ = from_ = v, w_ = {}, speed_ = 0, tc_ = tc, t_ = start_ = now, slide_ = 0, init_ = true;
|
||||
cutoff_ = p_.minCutoff;
|
||||
return;
|
||||
}
|
||||
Advance(now);
|
||||
const double dt = tc - tc_;
|
||||
if (dt > 1e-4) {
|
||||
// angular velocity (axis times rate, deg/s) from the raw samples, low-passed as a vector
|
||||
const V3 axis = Cross(r_, v);
|
||||
const double s = Len(axis), w = Angle(r_, v) * 180 / M_PI / dt;
|
||||
const V3 now_w = s > 1e-12 ? axis * (w / s) : V3{};
|
||||
w_ = w_ + (now_w - w_) * detail::Alpha(p_.dCutoff, std::min(dt, 0.5));
|
||||
speed_ = Len(w_);
|
||||
tc_ = tc;
|
||||
}
|
||||
from_ = Input(now);
|
||||
r_ = v;
|
||||
start_ = now, slide_ = std::clamp(dt, 0.0, kMaxSlide);
|
||||
cutoff_ = CutoffAt(p_, speed_);
|
||||
}
|
||||
V3 At(double now) {
|
||||
Advance(now);
|
||||
return x_;
|
||||
}
|
||||
V3 Last() const { return x_; }
|
||||
V3 Raw() const { return r_; }
|
||||
double Speed() const { return speed_; } // deg/s, low-passed
|
||||
double Cutoff() const { return cutoff_; }
|
||||
|
||||
private:
|
||||
V3 Input(double t) const {
|
||||
return Slerp(from_, r_, slide_ > 0 ? std::clamp((t - start_) / slide_, 0.0, 1.0) : 1.0);
|
||||
}
|
||||
void Advance(double now) {
|
||||
if (now <= t_) return;
|
||||
if (now - t_ > 1) t_ = now - 1;
|
||||
while (t_ < now) {
|
||||
const double h = std::min(kStep, now - t_);
|
||||
t_ += h;
|
||||
x_ = Slerp(x_, Input(t_), detail::Toward(cutoff_, h));
|
||||
}
|
||||
}
|
||||
Params p_;
|
||||
V3 x_, r_, from_, w_;
|
||||
double speed_ = 0, cutoff_ = 1, tc_ = 0, t_ = 0, start_ = 0, slide_ = 0;
|
||||
bool init_ = false;
|
||||
};
|
||||
|
||||
} // namespace fhsmooth
|
||||
+2
-19
@@ -11,9 +11,7 @@ The plan this belongs to is `~/Desktop/Projects/frame-hands/notes/hands-plan.md`
|
||||
| `hands/rec/panel/ft-handpanel.cpp` (C++, dev container) | The headset panel: a SteamVR overlay fixed to the head that shows the prompts. It also places the "touch the dot" target in the room and logs head and controller poses. Driven over `@ft_handpanel`. |
|
||||
| `hands/rec/session.py` (Python, no Qt) | The session runner. It reads `script.json`, starts and stops the recordings, and drives the panel. It reads the live hands file for feedback and writes each take's files. It also runs from the command line (`--dry-run`) for testing. |
|
||||
| `hands/rec/script.json` | The guided script: sections, prompts, timings. |
|
||||
| `hands/rec/script-v2.json` | The v2 script (`ft-handrec --script script-v2.json`, `session.py --script`): a demo clip per step (below). Being tried; it becomes `script.json` once settled. |
|
||||
| `hands/rec/poses/` | The pose pictures, `poses.json` and its PNGs (below). |
|
||||
| `hands/rec/demos/` | The demo clips, `<id>.gif`, made from `demo/parts.json` (below). |
|
||||
| `hands/rec/takes.py` (Python, no Qt) | Reads sessions and takes from disk: frame sets for review, deleted ranges, export (compress, strip, manifest, checksums). |
|
||||
| `hands/rec/ft_handrec.py` + `main.qml` (PySide6 + Kirigami, dev container) | The desktop window: consent, the before-you-start checklist, the session controls, review, export and upload instructions. |
|
||||
| `hands/rec/ft-handrec` | The host launcher, like `input-settings/ft-input-settings`. |
|
||||
@@ -52,7 +50,7 @@ Test hooks:
|
||||
- `--dry-run` runs no processes and only prints the panel commands, with timing sped up by `--speed X`.
|
||||
- `--next-after S` presses Next by itself after S seconds of waiting (real time), so a step-mode session runs unattended. Lines on stdin steer it too: `n` or an empty line is Next, `p` pause or resume, `r` redo, `s` skip the section, `q` stop.
|
||||
- `--no-headset-button` leaves the headset's button alone (`ft-handrec --no-headset-button` too). `--button-device PATH` reads it from PATH instead, an event device or a FIFO of `input_event` structs, also in a dry run: a simulated button for tests.
|
||||
- `--auto` runs the timed flow; `--poses DIR` takes the pose pictures from DIR, `--demos DIR` the demo clips (`ft-handrec --demos` too); `--plan` prints the sections, their steps and length.
|
||||
- `--auto` runs the timed flow; `--poses DIR` takes the pose pictures from DIR; `--plan` prints the sections, their steps and length.
|
||||
- `--ignore-cameras` (`ft-handrec --ignore-cameras` too) starts even when the camera check fails. A dry run and `--ring` skip the check by themselves.
|
||||
|
||||
## Files
|
||||
@@ -133,7 +131,6 @@ One JSON object per line:
|
||||
{"t": 123, "event": "take", "section": "static-poses", "take": "03-static-poses"}
|
||||
{"t": 123, "event": "prompt", "id": "static-poses/fist/left/near", "text": "...", "hands": "left|right|both|none",
|
||||
"pose": "fist", "distance": "near|mid|far|", "position": "centre|left|right|up|down|", "object": "", "controller": false}
|
||||
{"t": 123, "event": "prompt", "id": "index-taps/2a/left", "...": "...", "demo": "2a"}
|
||||
{"t": 123, "event": "target", "id": "touch/3", "head": [x, y, z], "room": [x, y, z], "state": "show|hold|done|timeout"}
|
||||
{"t": 123, "event": "bar", "target": 0.0}
|
||||
{"t": 123, "event": "feedback", "left": true, "right": false, "palm_m": [0.0, 0.0]}
|
||||
@@ -172,7 +169,7 @@ ft-handpanel writes one line per sample, 250 a second, from `GetDeviceToAbsolute
|
||||
|
||||
## The panel (`ft-handpanel`)
|
||||
|
||||
- **Placement.** A SteamVR overlay fixed to the head, like `gaze/panel/ft-gazepanel.cpp`: key `frametop.handpanel`, sort order 250. It sits 1.2 m ahead, centred 12 degrees above straight ahead, so the hands stay clear below it. It's 36 degrees wide, 4:3, 1024x768 pixels, dim and see-through. It's drawn on the CPU with stb_truetype (and stb_image for the pose pictures and demo clips, PNG and GIF, the same pinned stb commit) into three shared DMA-BUFs SteamVR imports once, as ft-gazepanel does, and is drawn again only when something changes.
|
||||
- **Placement.** A SteamVR overlay fixed to the head, like `gaze/panel/ft-gazepanel.cpp`: key `frametop.handpanel`, sort order 250. It sits 1.2 m ahead, centred 12 degrees above straight ahead, so the hands stay clear below it. It's 36 degrees wide, 4:3, 1024x768 pixels, dim and see-through. It's drawn on the CPU with stb_truetype (and stb_image for the pose pictures, PNG only, the same pinned stb commit) into three shared DMA-BUFs SteamVR imports once, as ft-gazepanel does, and is drawn again only when something changes.
|
||||
- **Layout.** The title and step on top (a red "Rec" by the step while recording), a rule. With a pose picture or a diagram, a 14-degree column on the left holds the picture (or the flipped copy and the picture side by side) and the where-to diagram under it; the text takes the right. The text column holds the instruction, the orange note, the big countdown ("3", "2", "1", then "Hold" or "Go") and the cyan action line ("Ready? Press Space or click Next"), centred together. At the bottom: the near/far bar, the hand chips, the time-left bar and the key hints.
|
||||
- **Socket.** Abstract unix datagram `@ft_handpanel` (`--socket NAME`). A sender with an address gets `ok ...` or `error ...`.
|
||||
- **Options:** `--watch-stdin` (quit when stdin closes), `--socket NAME`, `--distance M`, `--no-vr`. `--no-vr` makes no SteamVR connection and prints each picture's text to stdout: for testing without a headset.
|
||||
@@ -190,7 +187,6 @@ Commands (UTF-8; `|` starts a new line in text):
|
||||
| `hands <left> <right>` | Two chips, "Left hand" and "Right hand", each `seen` (green), `lost` (orange) or `off` (hidden). |
|
||||
| `bar <target 0..1> <current 0..1 or -1> [near label] [far label]` / `bar off` | The near/far bar for the push out and back: a horizontal track with a target marker and the hand's current position. |
|
||||
| `paused on` / `paused off` | A "Paused" overlay over the picture. |
|
||||
| `demo <path> [mirror]` / `demo restart` / `demo off` | A demo clip (below) playing in a wider left column (16 degrees), looped, with the key to its colours under it: a light swatch "Light: palm", a dark one "Dark: back of hand". It takes the picture's place, and the text beside it is left-aligned (a list of motions). A new path starts it from its first frame; the same one again leaves it playing (`mirror` only flips it); `restart` starts it again. It stands still while paused. A file that can't be read: `error can't read ...` and no clip. |
|
||||
| `image <path> [mirror\|both]` / `image off` | The pose picture, a PNG (below), in the left column. `mirror` flips it (a left hand); `both` draws a flipped copy on its left. A file that can't be read: `error ...` and no picture. |
|
||||
| `where <position\|-> <distance\|->` / `where off` | The where-to diagram under the picture: a 3x3 front view with the asked cell lit (`centre`, `left`, `right`, `up`, `down`, and the push sections' `chest`, `desk`, `eye`), and a side view of the head and three marks for `near` ("Close"), `mid` ("Halfway out") and `far` ("Arm out"). `-` leaves that half out. |
|
||||
| `action <text>` | The cyan line under the instruction (empty clears it). |
|
||||
@@ -219,16 +215,6 @@ It quits on SteamVR's quit event, as ft-gazepanel does. `--no-vr --dump DIR` wri
|
||||
|
||||
A pose with no entry, or whose file is missing, shows no picture: the text alone. The session reads `poses.json` when it starts. The window shows the same picture (QML `Image.mirror`), its caption, and the same diagram.
|
||||
|
||||
## Demo clips (`demos/`)
|
||||
|
||||
The v2 script shows a looping clip for each step instead of a pose picture: generated animation of a stick-figure hand in the colours of Steam Frame passthrough, one GIF per part of `demo/parts.json` (`<id>.gif`, 640x360, 12 frames a second, 6-16 s, under 2 MB). They're made outside the Frame (Blender), so their source isn't here. The palm side of the hand is drawn light (#d5d1fe) and the back dark (#6a62d8); the panel and the window say so under each clip, and the welcome screen too.
|
||||
|
||||
- A prompt's `"demo": "<id>"` names its clip; `"demo_mirror": true` flips it (`session.demo_view`): the clips show a right hand, or both hands, so a left-hand step flips a right hand's clip. A missing file shows no clip: the text alone.
|
||||
- The clip starts on the step's ready screen and starts again from its first frame when the hold begins (after the 3-2-1), so it can be followed. The status's `demo_run` counts those starts for the window, which plays the same clip (QML `AnimatedImage`).
|
||||
- The panel decodes a frame at a time as it plays (stb_image's GIF decoder, stepped as `stbi_load_gif_from_memory` steps it), so a clip costs three frames of memory however long it is. A frame redraws only the clip's rectangle and copies only that to a buffer that already has the rest of the picture: about 450x250 pixels 12 times a second.
|
||||
- Two kinds, the contributor's choice on the window's start page ("Guidance", kept in `BASE/settings.json`, written to `session.json` as `"guidance"`): **recorded** clips in `demos-recorded/`, a real person's hands from a recorder session's side cameras (`demo/camgifs.py`: each frame drawn again as a plain picture from between the eyes, 640x360, 10 a second, grey), and **skeleton** clips in `demos/`, the drawn hand. Recorded is the default when there are recorded clips. A part with no clip of the chosen kind shows the other kind. Only a skeleton clip shows the key to its colours (the panel's `demo <path> [mirror] [nokey]`; the status's `demo_key`). Recorded clips show the room around the person who recorded them: check them before they go anywhere public.
|
||||
- In `script-v2.json` a one-hand part runs three steps: the left hand alone (the clip flipped), the right alone, then both at once. A part made for two hands runs once. Each step lasts about 1.2 times its clip. `"name": "v2"` goes into `session.json`'s `script`.
|
||||
|
||||
## The script (`script.json`)
|
||||
|
||||
```json
|
||||
@@ -259,7 +245,6 @@ The v2 script shows a looping clip for each step instead of a pose picture: gene
|
||||
- `bar`: the target marker sweeps near to far and back, `reps` times per height, at `period_s` per sweep. The current marker follows the live palm distance.
|
||||
- Each section is one take. Prompts within it are marked in `prompts.jsonl`.
|
||||
- `cue_names`: the short labels under the strip's pictures (otherwise the pose id).
|
||||
- `name`: the script's name, recorded in `session.json` (`script.name`); `demo`, `demo_mirror`: a prompt's demo clip (above).
|
||||
|
||||
### Sweeps
|
||||
|
||||
@@ -317,8 +302,6 @@ On 2026-10-02 a whole session showed "I can't see your hands": after the headset
|
||||
|
||||
- **Before the session.** The window runs `hands/camcheck.py` when the checklist page opens ("Tracking cameras:", with the evidence under Details and Check again), and again when Start is pressed; `session.py` runs it first thing, before it makes the session's folder or starts anything (`Session._preflight`). If it finds the cameras degraded, the session doesn't start. The window says: "The headset's upper cameras and IR light are off. SteamVR couldn't start the colour camera module (it happens sometimes after the headset sleeps). Restart SteamVR, or restart the headset if that doesn't fix it." (another `degraded:` reason gets a sentence naming it), and Start stays off. `unknown` (SteamVR not running, the cameras asleep) doesn't stop it: the session's own start fails clearly then. From the command line, `session.py` prints the check and exits with status 3.
|
||||
- **Restart SteamVR.** The message has a Restart SteamVR button. After a confirmation it runs `systemctl --user restart --no-block steamvr.service` on the host (through `distrobox-host-exec`), then checks the cameras every 3 s, for up to 2 minutes, until a new XRService has opened its cameras. The confirmation says what really happens: every VR app closes, and so does the Frametop desktop with all its windows, this one included, and it doesn't come back by itself (`hands/README.md`, "What a SteamVR restart does to Frametop"). So the check after the restart mostly happens when the recorder is opened again; it re-runs when the checklist page opens. `--no-block` lets the restart finish after the window is gone.
|
||||
- **Frametop's hands are off during a session.** Frametop's drawn hands (`frametop-handview`) cover what the person watches, and its hand pointer (`frametop-pointer`) turns the steps' pinches and taps into clicks, or opens Steam's menu. When a session starts (after the camera check, before the welcome screen), the ones running are stopped, and listed in `/run/user/UID/frametop-hands/handrec-paused-units` first. They're started again when the session ends in any way. If the recorder is killed meanwhile, the window starts them when it opens again, and so does the next session. The hand tracker itself keeps running: the session's hand notes read it.
|
||||
- **Cameras moved or lost.** Unplugging or plugging in the colour camera module makes XRService move the side cameras to other devices (video9/13 and video0/3). On 2026-10-12 an ft-camd started before the move kept publishing the old devices, ft-hands left the side pair out, and a whole session recorded only the upper cameras. Now: the camera check fails when ft-camd's devices aren't the ones XRService runs (`ring_missing`, "it moved them"), and the window's check and the session's preflight restart that ft-camd (the recorder's own unit, or Frametop's live `frametop-camd` and then `frametop-hands`). ft-camd exits by itself when a camera goes silent while the others run (camd.c, `SILENT_S`), and its service starts it on the new devices. In a session, a part whose first set has no `slam_left`/`slam_right`, a recording that exits, or one that writes no set for `STALL_S` (3 s) loses the step (`_CamerasLost`): in step mode its try gets a `redo` event, the panel says "Waiting for the cameras", the session runs the camera check (with its repair) every 2 s for up to `CAMERAS_BACK_S` (120 s), then shows the step's ready screen again. If they don't come back, or after `MAX_CAMERA_LOSSES` (3) losses, the session stops with an error. Auto mode stops at once. Known gap: if a restarted ft-camd binds the side pair the other way round, the session's single `sides` decision (reversed for the later parts) renames the earlier parts wrongly; `hub_review check` tells.
|
||||
- **No hands in the first step.** The first step of the hand-size section has both hands up, about 40 cm away. During its hold, the session counts the hands file's reads (about 20 a second). If the tracker published in at least half of at least 10 reads and never saw a hand, the step stops: a `wait` and the recording stops as usual, then a `nohands` event and a `redo` over the try. The session runs the camera check and shows "I can't see your hands" with its result (the camera text above when it's the VCINT failure, else "The camera check found nothing wrong"). The state is `nohands`, waiting: Next (the headset button, Space, Try again) or R starts the same step's countdown at once; Stop (Esc) ends the session with `stop_reason` set; S skips the section. One missed step costs a retry, not the session, and every hold of that step is logged ("hands check ...: a hand in N of M reads"). Without a tracker publishing the session can't tell, logs that, and goes on. Only that one step is checked: later steps have their notes ("I can't see your left hand") as before. Auto mode does the same; its recording pauses meanwhile.
|
||||
|
||||
### Controls
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
# Build the hand recorder's headset panel ft-handpanel in the dev container on the Frame
|
||||
# (hands/rec/build/; it also runs there). Like gaze/build.sh: the pinned public OpenVR header
|
||||
# (the DMA-BUF import is newer than the header shipped with SteamVR's samples), stb_truetype
|
||||
# for the text and stb_image (PNG and GIF) for the pose pictures and the demo clips, at the same stb commit.
|
||||
# for the text and stb_image (PNG only) for the pose pictures, at the same stb commit.
|
||||
set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
|
||||
"$root/scripts/sync.sh" >/dev/null
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
# Demo clips for the hand recorder
|
||||
|
||||
The v2 recording script shows a looping GIF for each part of `parts.json` (`script-v2.json` is made
|
||||
from it). There are two kinds (hands/rec/DESIGN.md): skeleton clips in `hands/rec/demos/`, generated
|
||||
animation made outside the Frame, and recorded clips in `hands/rec/demos-recorded/`, made here from a
|
||||
recorder session.
|
||||
|
||||
## Recorded clips from a recorder session
|
||||
|
||||
The window's "Recorded hands" guidance plays clips of real hands from a Hand Recorder session that
|
||||
ran the v2 script, instead of the drawn ones:
|
||||
`frame-job --local -- PYTHON hands/rec/demo/camgifs.py SESSION_DIR` (PYTHON with NumPy, e.g.
|
||||
frame-hands' `.venv/bin/python`) writes `hands/rec/demos-recorded/<id>.gif`. Each part uses its
|
||||
right-hand step (or its both-hands step), redrawn from the two side cameras as a plain picture.
|
||||
`--sheets` writes contact sheets first; `--view ID=PITCH,HFOV` aims a part's view, kept in
|
||||
`views.json` next to the clips. They show the room: check them before they go anywhere public.
|
||||
@@ -1,277 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Recorded-guidance demo GIFs from a hand recorder session: the side cameras' own frames.
|
||||
|
||||
frame-job --local -- PYTHON hands/rec/demo/camgifs.py SESSION_DIR [--out DIR] [--part ID]
|
||||
[--sheets] [--view ID=PITCH,HFOV[,YAW]] [--lossy N]
|
||||
|
||||
PYTHON needs NumPy (frame-hands' .venv/bin/python has it). SESSION_DIR is a recorder session
|
||||
(BASE/sessions/<id>) that ran the v2 script. Per demo clip of script-v2.json it takes the part's
|
||||
right-hand step (the window mirrors it for the left hand, like the skeleton clips), or its
|
||||
both-hands step, from the start of its prompt to the end of its last try; a redone step uses
|
||||
the try that was kept.
|
||||
|
||||
The side cameras are fisheyes looking down and out, so each frame is drawn again as a plain
|
||||
picture from between the eyes: a virtual camera PITCH degrees below straight ahead, HFOV wide,
|
||||
filled from both side cameras (the session's calibration.json) for points 45 cm away, each
|
||||
pixel from the camera that sees it nearer its centre. Clips run at the recording's 10 sets/s,
|
||||
640x360 like the skeleton clips, grey, smoothed (steady), gifsicle's lossy pass when it's there
|
||||
(HANDDEMO_GIFSICLE, or gifsicle on the PATH): about 1.4 MB for 10 s.
|
||||
--sheets writes a contact sheet per clip instead (<id>.png, a frame every 2 s), to pick
|
||||
--view per part before making the GIFs; views.json in the out folder keeps them
|
||||
({"12": [-55, 100]}).
|
||||
|
||||
The GIFs show the room around the recorder: check them before they go anywhere public.
|
||||
Writes OUT/<id>.gif (default hands/rec/demos-recorded, where the window's "Recorded" guidance
|
||||
looks).
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
REC = os.path.dirname(HERE)
|
||||
sys.path.insert(0, REC)
|
||||
import takes # noqa: E402 (hands/rec/takes.py)
|
||||
|
||||
GIFSICLE = os.environ.get("HANDDEMO_GIFSICLE") or shutil.which("gifsicle") or os.path.expanduser(
|
||||
"~/.cache/gifwork/gifsicle-src/src/gifsicle")
|
||||
|
||||
SCRIPT = os.path.join(REC, "script-v2.json")
|
||||
OUT = os.path.join(REC, "demos-recorded")
|
||||
W, H, SS = 640, 360, 2 # the clip, drawn at SS x SS and averaged down
|
||||
PITCH, HFOV = -30.0, 100.0 # the default view: degrees below straight ahead, wide
|
||||
DEPTH = 0.45 # metres: where the two side cameras' pictures meet
|
||||
FPS = 10 # the recorder's RECORD_HZ
|
||||
SIDES = ("slam_left", "slam_right")
|
||||
|
||||
|
||||
def _pose(d, scale=1.0):
|
||||
"""4x4 transform from a {plus_x, plus_z, position} pose (child axes in the parent frame)."""
|
||||
x = np.asarray(d["plus_x"], float)
|
||||
z = np.asarray(d["plus_z"], float)
|
||||
m = np.eye(4)
|
||||
m[:3, 0], m[:3, 1], m[:3, 2] = x, np.cross(z, x), z
|
||||
m[:3, 3] = np.asarray(d["position"], float) * scale
|
||||
return m
|
||||
|
||||
|
||||
class Fisheye:
|
||||
"""A side camera: Kannala-Brandt intrinsics, posed in the head frame (OpenVR: +x right, +y up,
|
||||
-z forward) as hands/track does it, from a session's calibration.json: "cameras" are posed in
|
||||
the first camera's frame, "cad" holds every camera and the head in the headset's CAD frame."""
|
||||
|
||||
def __init__(self, cal, name):
|
||||
i, c = next((i, c) for i, c in enumerate(cal["cameras"]) if c["sourceCamera"] == name)
|
||||
kb = next(k for k in c["intrinsics"] if k["cameraModel"] == "kb")
|
||||
self.fx, self.fy, self.cx, self.cy = kb["fx"], kb["fy"], kb["cx"], kb["cy"]
|
||||
self.k = [kb["k1"], kb["k2"], kb["k3"], kb["k4"]]
|
||||
self.width, self.height = c["width"], c["height"]
|
||||
head_from_cad = np.linalg.inv(_pose(cal["cad"]["__head"]))
|
||||
cad_from_cam0 = _pose(cal["cad"]["cameras"][0], 1e-3)
|
||||
head_from_cam = head_from_cad @ cad_from_cam0 @ _pose(c["extrinsics"], 1e-3)
|
||||
self.R, self.origin = head_from_cam[:3, :3], head_from_cam[:3, 3]
|
||||
|
||||
def project(self, p):
|
||||
"""Head-frame points (N,3) -> pixels (N,2) and the angle off the optical axis (N,), degrees."""
|
||||
q = (p - self.origin) @ self.R
|
||||
r = np.hypot(q[:, 0], q[:, 1])
|
||||
theta = np.arctan2(r, q[:, 2])
|
||||
t2 = theta * theta
|
||||
k1, k2, k3, k4 = self.k
|
||||
td = theta * (1 + t2 * (k1 + t2 * (k2 + t2 * (k3 + t2 * k4))))
|
||||
s = np.where(r > 1e-12, td / np.maximum(r, 1e-12), 0.0)
|
||||
uv = np.stack([self.fx * q[:, 0] * s + self.cx, self.fy * q[:, 1] * s + self.cy], axis=1)
|
||||
return uv, np.degrees(theta)
|
||||
|
||||
|
||||
def view_maps(cams, pitch, hfov, yaw=0.0):
|
||||
"""Per side camera: where each pixel of the virtual view samples it, and its weight."""
|
||||
w, h = W * SS, H * SS
|
||||
f = (w / 2) / np.tan(np.radians(hfov / 2))
|
||||
u, v = np.meshgrid(np.arange(w) - (w - 1) / 2, np.arange(h) - (h - 1) / 2)
|
||||
d = np.stack([u / f, -v / f, -np.ones_like(u)], -1).reshape(-1, 3)
|
||||
d /= np.linalg.norm(d, axis=1, keepdims=True)
|
||||
a, b = np.radians(pitch), np.radians(yaw)
|
||||
rx = np.array([[1, 0, 0], [0, np.cos(a), -np.sin(a)], [0, np.sin(a), np.cos(a)]])
|
||||
ry = np.array([[np.cos(b), 0, -np.sin(b)], [0, 1, 0], [np.sin(b), 0, np.cos(b)]])
|
||||
p = d @ (ry @ rx).T * DEPTH
|
||||
maps, total = {}, 0
|
||||
for name, cam in cams.items():
|
||||
uv, ang = cam.project(p)
|
||||
inside = (uv[:, 0] >= 0) & (uv[:, 0] <= cam.width - 1) & (uv[:, 1] >= 0) & (uv[:, 1] <= cam.height - 1)
|
||||
wt = np.where(inside, np.clip((88 - ang) / 48, 0, 1) ** 3, 0) # nearer the centre wins
|
||||
maps[name] = (uv[:, 0].reshape(h, w), uv[:, 1].reshape(h, w), wt.reshape(h, w))
|
||||
total = total + wt.reshape(h, w)
|
||||
return maps, np.maximum(total, 1e-6)
|
||||
|
||||
|
||||
def sample(img, x, y):
|
||||
"""Bilinear lookup of a grey image at float pixel positions."""
|
||||
x = np.clip(x, 0, img.shape[1] - 1.001)
|
||||
y = np.clip(y, 0, img.shape[0] - 1.001)
|
||||
x0, y0 = x.astype(np.int32), y.astype(np.int32)
|
||||
fx, fy = x - x0, y - y0
|
||||
a = img[y0, x0] * (1 - fx) + img[y0, x0 + 1] * fx
|
||||
b = img[y0 + 1, x0] * (1 - fx) + img[y0 + 1, x0 + 1] * fx
|
||||
return a * (1 - fy) + b * fy
|
||||
|
||||
|
||||
def render(maps, total, frames):
|
||||
acc = 0
|
||||
for name, (x, y, wt) in maps.items():
|
||||
acc = acc + sample(frames[name], x, y) * wt
|
||||
out = acc / total
|
||||
return out.reshape(H, SS, W, SS).mean(axis=(1, 3))
|
||||
|
||||
|
||||
def demo_steps(script):
|
||||
"""{demo id: [(section, hands), ...]} in the script's order, the step to use first: the
|
||||
right-hand one where the clip is a right hand's (mirrored for the left), else both hands."""
|
||||
steps = {}
|
||||
for sec in script["sections"]:
|
||||
for p in sec.get("prompts", []):
|
||||
if p.get("demo"):
|
||||
steps.setdefault(p["demo"], []).append((sec["id"], p.get("hands") or ""))
|
||||
rank = {"right": 0, "both": 1}
|
||||
return {d: sorted(s, key=lambda sh: rank.get(sh[1], 2)) for d, s in steps.items()}
|
||||
|
||||
|
||||
def find_window(session, section, demo, hands):
|
||||
"""(take dir, start ns, end ns) of the step's kept try, or None."""
|
||||
tdir = os.path.join(session, "takes")
|
||||
for name in sorted(os.listdir(tdir)):
|
||||
if not name.endswith("-" + section):
|
||||
continue
|
||||
take = os.path.join(tdir, name)
|
||||
prefix = "%s/%s/%s" % (section, demo, hands)
|
||||
t0 = t1 = None
|
||||
with open(os.path.join(take, "prompts.jsonl")) as f:
|
||||
for line in f:
|
||||
e = json.loads(line)
|
||||
ev = e.get("event")
|
||||
if ev == "prompt" and (e.get("id") == prefix or e.get("id", "").startswith(prefix + "/")):
|
||||
t0, t1 = e["t"], None
|
||||
elif t0 is not None and t1 is None and ev in ("wait", "redo", "end", "pause"):
|
||||
t1 = e["t"]
|
||||
if ev == "redo":
|
||||
t0 = None
|
||||
if t0 is not None:
|
||||
return take, t0, t1
|
||||
return None
|
||||
|
||||
|
||||
def clip_sets(take, t0, t1):
|
||||
ti = takes.TakeIndex(take)
|
||||
deleted = takes.read_json(os.path.join(take, "take.json")).get("deleted") or []
|
||||
keep = [i for i in range(len(ti)) if t0 <= ti.time_ns(i) <= (t1 or 1 << 62)
|
||||
and not any(a <= ti.time_ns(i) <= b for a, b in deleted)]
|
||||
return ti, keep
|
||||
|
||||
|
||||
def frames_of(ti, i):
|
||||
out = {}
|
||||
for c in ti.read_set(i):
|
||||
if c["name"] in SIDES:
|
||||
out[c["name"]] = np.frombuffer(c["pixels"], np.uint8).reshape(c["height"], c["width"]).astype(np.float32)
|
||||
return out
|
||||
|
||||
|
||||
def encode(clip, out, lossy, sheet):
|
||||
"""Grey uint8 frames (N,H,W) -> a looping GIF, or a contact sheet PNG."""
|
||||
raw_in = ["ffmpeg", "-v", "error", "-y", "-f", "rawvideo", "-pix_fmt", "gray", "-s", "%dx%d" % (W, H),
|
||||
"-r", str(FPS), "-i", "-"]
|
||||
if sheet:
|
||||
n = len(clip[::2 * FPS])
|
||||
cols = min(4, n)
|
||||
rows = (n + cols - 1) // cols
|
||||
cmd = raw_in + ["-vf", "select=not(mod(n\\,%d)),scale=320:-2,tile=%dx%d" % (2 * FPS, cols, rows),
|
||||
"-frames:v", "1", out]
|
||||
subprocess.run(cmd, input=clip.tobytes(), check=True)
|
||||
return
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
raw = os.path.join(tmp, "raw.gif")
|
||||
cmd = raw_in + ["-vf", "split[a][b];[a]palettegen=max_colors=48:stats_mode=full[p];"
|
||||
"[b][p]paletteuse=dither=none:diff_mode=rectangle", "-loop", "0", raw]
|
||||
subprocess.run(cmd, input=clip.tobytes(), check=True)
|
||||
if os.access(GIFSICLE, os.X_OK):
|
||||
subprocess.run([GIFSICLE, "-O3", "--lossy=%d" % lossy, raw, "-o", out], check=True)
|
||||
else:
|
||||
os.replace(raw, out)
|
||||
|
||||
|
||||
def steady(clip, hold=12.0):
|
||||
"""Less sensor noise, for a GIF that compresses (10-15 MB a clip without it, about 1.5 with):
|
||||
two [1 2 1] blurs, then a pixel that changed by less than hold grey levels keeps its last
|
||||
value, so the still parts of a frame repeat the one before."""
|
||||
c = clip.astype(np.float32)
|
||||
for _ in range(2):
|
||||
c[:, 1:-1] = (c[:, :-2] + 2 * c[:, 1:-1] + c[:, 2:]) / 4
|
||||
c[:, :, 1:-1] = (c[:, :, :-2] + 2 * c[:, :, 1:-1] + c[:, :, 2:]) / 4
|
||||
for i in range(1, len(c)):
|
||||
same = np.abs(c[i] - c[i - 1]) < hold
|
||||
c[i][same] = c[i - 1][same]
|
||||
return c
|
||||
|
||||
|
||||
def stretch(clip):
|
||||
"""Levels per clip: the IR frames are dim and flat."""
|
||||
lo, hi = np.percentile(clip[::max(1, len(clip) // 20)], (1, 99.7))
|
||||
x = np.clip((clip - lo) / max(hi - lo, 1), 0, 1) ** 0.8
|
||||
return (x * 255 + 0.5).astype(np.uint8)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
ap.add_argument("session")
|
||||
ap.add_argument("--out", default=OUT)
|
||||
ap.add_argument("--part", help="one demo clip id (e.g. 7a)")
|
||||
ap.add_argument("--sheets", action="store_true", help="contact sheets instead of GIFs")
|
||||
ap.add_argument("--view", action="append", default=[], metavar="ID=PITCH,HFOV[,YAW]")
|
||||
ap.add_argument("--lossy", type=int, default=70, help="gifsicle's lossy level")
|
||||
a = ap.parse_args()
|
||||
with open(SCRIPT) as f:
|
||||
script = json.load(f)
|
||||
with open(os.path.join(a.session, "calibration.json")) as f:
|
||||
cal = json.load(f)
|
||||
cams = {n: Fisheye(cal, n) for n in SIDES}
|
||||
os.makedirs(a.out, exist_ok=True)
|
||||
views = takes.read_json(os.path.join(a.out, "views.json")) or {}
|
||||
for v in a.view:
|
||||
part, _, vals = v.partition("=")
|
||||
views[part] = [float(x) for x in vals.split(",")]
|
||||
if a.view:
|
||||
with open(os.path.join(a.out, "views.json"), "w") as f:
|
||||
json.dump(views, f, indent=1, sort_keys=True)
|
||||
cache = {}
|
||||
for demo, steps in demo_steps(script).items():
|
||||
if a.part and demo != a.part:
|
||||
continue
|
||||
found = next(((s, h, w) for s, h in steps for w in [find_window(a.session, s, demo, h)] if w), None)
|
||||
if not found:
|
||||
print("%-4s not in this session" % demo)
|
||||
continue
|
||||
section, hands, (take, t0, t1) = found
|
||||
ti, keep = clip_sets(take, t0, t1)
|
||||
if not keep:
|
||||
print("%-4s %s/%s: no sets" % (demo, section, hands))
|
||||
continue
|
||||
view = tuple(views.get(demo) or (PITCH, HFOV))
|
||||
if view not in cache:
|
||||
cache[view] = view_maps(cams, *view)
|
||||
maps, total = cache[view]
|
||||
clip = stretch(steady(np.stack([render(maps, total, frames_of(ti, i)) for i in keep])))
|
||||
out = os.path.join(a.out, demo + (".png" if a.sheets else ".gif"))
|
||||
encode(clip, out, a.lossy, a.sheets)
|
||||
print("%-4s %-12s %-5s %5.1f s view %s %.2f MB" % (demo, section, hands, len(keep) / FPS,
|
||||
",".join("%g" % x for x in view),
|
||||
os.path.getsize(out) / 1e6), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,216 +0,0 @@
|
||||
{
|
||||
"version": 1,
|
||||
"note": "The demo GIFs for the hand recorder's v2 script: one recording per part, in this order. Motions are what the maintainer performs on camera; the plan is the Claude Doc ee05ba65 (Demo recordings, 2026-10-12).",
|
||||
"parts": [
|
||||
{
|
||||
"id": "1",
|
||||
"section": "Hand size",
|
||||
"motions": [
|
||||
"Both hands about 40 cm out, still",
|
||||
"Palms toward you, then backs toward you",
|
||||
"Fingers spread, then a fist, then point",
|
||||
"About 1 s for each shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "2a",
|
||||
"section": "Index taps",
|
||||
"title": "Wrist roll",
|
||||
"motions": [
|
||||
"One hand low and relaxed, other fingers loosely curled: look down at it",
|
||||
"2 light taps palm down",
|
||||
"2 taps edge-on, 2 palm up, 2 palm toward your face"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "2b",
|
||||
"section": "Index taps",
|
||||
"title": "Through the zones",
|
||||
"motions": [
|
||||
"Keep tapping while you move: centre at chest height",
|
||||
"Far out to your side, then crossed past the middle",
|
||||
"Close to your face, then at arm's length",
|
||||
"Quick taps, a double tap, one slow press"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "2c",
|
||||
"section": "Index taps",
|
||||
"title": "Both hands",
|
||||
"motions": [
|
||||
"Both hands tapping",
|
||||
"Then one hand over the other, still tapping"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "3a",
|
||||
"section": "Middle taps",
|
||||
"title": "Wrist roll and zones",
|
||||
"motions": [
|
||||
"Thumb to the middle finger's pad, index half-curled: look down at the hand",
|
||||
"2 taps palm down, edge-on, palm up, toward your face",
|
||||
"Then a few taps out to the side and close to your face"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "3b",
|
||||
"section": "Middle taps",
|
||||
"title": "Firm presses and a head turn",
|
||||
"motions": [
|
||||
"3 slow, firm presses",
|
||||
"Keep tapping and turn your head until the hand reaches the edge of the view",
|
||||
"Then turn the other way"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "4a",
|
||||
"section": "Look-alikes",
|
||||
"title": "Fingers",
|
||||
"motions": [
|
||||
"Stop 1 cm short of a tap",
|
||||
"Rub the thumb along the side of the index",
|
||||
"Ring finger to thumb",
|
||||
"Let a loosely curled hand rest"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "4b",
|
||||
"section": "Look-alikes",
|
||||
"title": "Fists",
|
||||
"motions": [
|
||||
"Close a fist slowly, open it",
|
||||
"Close it fast, open it",
|
||||
"Once at chest height, once low"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "5a",
|
||||
"section": "Hold and drag",
|
||||
"title": "One hand",
|
||||
"motions": [
|
||||
"Pinch and hold 3 s",
|
||||
"Drag up, down, left, right",
|
||||
"Roll the wrist while holding",
|
||||
"Loosen the other fingers without letting go"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "5b",
|
||||
"section": "Hold and drag",
|
||||
"title": "Two hands",
|
||||
"motions": [
|
||||
"One hand holds a pinch, the other taps",
|
||||
"Hands apart, then close, then crossed"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "6",
|
||||
"section": "Three-finger scroll",
|
||||
"motions": [
|
||||
"Thumb on the index and middle tips",
|
||||
"Move up and down slowly, then fast",
|
||||
"Let the ring finger join once"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "7a",
|
||||
"section": "Palm up",
|
||||
"title": "Fingertips drop",
|
||||
"motions": [
|
||||
"Arms forward, palms up, fingers flat: hands close (elbows tucked)",
|
||||
"Fingertips drop toward the floor and back: 1 slow, 2 fast",
|
||||
"Same with forearms half out, then arms straight",
|
||||
"One drop low at the waist, one at eye height"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "7b",
|
||||
"section": "Palm up",
|
||||
"title": "Fingertips rise",
|
||||
"motions": [
|
||||
"Palms up at the same three distances",
|
||||
"Fingertips curl up toward you and back: 1 slow, 2 fast each"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "7c",
|
||||
"section": "Palm up",
|
||||
"title": "Pinch while bent",
|
||||
"motions": [
|
||||
"Palm up, hold a pinch",
|
||||
"Drop the fingertips and raise them, still pinching",
|
||||
"Flip palm up and down with the wrist bent"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "8",
|
||||
"section": "Strokes and gestures",
|
||||
"motions": [
|
||||
"Lazy susan stroke each way, with the return stroke",
|
||||
"A wave",
|
||||
"5 s of talking with your hands"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "9a",
|
||||
"section": "Reach and pull",
|
||||
"title": "Grab that",
|
||||
"motions": [
|
||||
"Open palm at a far target",
|
||||
"Close to a fist and pull back"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "9b",
|
||||
"section": "Reach and pull",
|
||||
"title": "Push, flick, a real object",
|
||||
"motions": [
|
||||
"Push away, then flick away",
|
||||
"Reach for a cup or your phone"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "10",
|
||||
"section": "Fists",
|
||||
"motions": [
|
||||
"Two fists: hold, pull apart, push together",
|
||||
"Fists with the palms in, down, then up"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "11a",
|
||||
"section": "Edges and exits",
|
||||
"title": "Out and back",
|
||||
"motions": [
|
||||
"Right hand open: slide out past the right edge until gone, wait 1 s, back",
|
||||
"Same up past your face",
|
||||
"Left hand pinching: out past the left edge, then down past your lap"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "11b",
|
||||
"section": "Edges and exits",
|
||||
"title": "Crossing",
|
||||
"motions": [
|
||||
"Both hands cross fully to the opposite sides",
|
||||
"Hold a moment, come back"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "12",
|
||||
"section": "Desk work",
|
||||
"motions": [
|
||||
"Type a few words, use the mouse, pick up the phone",
|
||||
"One tap, then back to the keys"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "13",
|
||||
"section": "Controllers",
|
||||
"motions": [
|
||||
"Pick up the controllers, set them down",
|
||||
"A tap in between"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
+3
-54
@@ -7,9 +7,8 @@ A Kirigami (QML) app with a Python backend. It runs in the dev container:
|
||||
- Before you start: the checklist (objects, controller straps, lighting, sleeves, privacy,
|
||||
free space), the camera check (hands/camcheck.py: Start stays off while the upper cameras
|
||||
are off, with a Restart SteamVR button; --ignore-cameras overrides it), a full session or a
|
||||
quick round (suggested once a full one is done), the demo clips to follow when the script
|
||||
has them (recorded hands or skeleton hands, kept in settings.json) and what will happen.
|
||||
Start hands it to the session runner (session.py).
|
||||
quick round (suggested once a full one is done) and what will happen. Start hands it to
|
||||
the session runner (session.py).
|
||||
- Session: the runner's live status with the step's pose picture and where-to diagram, Next,
|
||||
Pause/Resume, Redo, Skip section and Stop (Space: Next, P, R, S and Esc while the window has
|
||||
focus; the button on the headset's right side is Next, pause and resume). The prompts appear
|
||||
@@ -224,7 +223,6 @@ class Backend(QObject):
|
||||
_uploadFinished = Signal(dict)
|
||||
cameraChanged = Signal()
|
||||
_cameraArrived = Signal(dict)
|
||||
guidanceChanged = Signal()
|
||||
|
||||
def __init__(self, store, session_options=None, hub_dry_run=False, ignore_cameras=False):
|
||||
super().__init__()
|
||||
@@ -353,41 +351,6 @@ class Backend(QObject):
|
||||
self.profileChanged.emit()
|
||||
self.message.emit("Thank you. Your contributor id: " + profile["contributor"], False)
|
||||
|
||||
# --- the demo clips to follow (the v2 script): kept in BASE/settings.json
|
||||
@Property(bool, constant=True)
|
||||
def scriptHasDemos(self):
|
||||
mod = self._runner()
|
||||
try:
|
||||
script = mod.load_script(SCRIPT_PATH) if mod else {}
|
||||
except Exception:
|
||||
return False
|
||||
return any(p.get("demo") for sec in script.get("sections", []) for p in sec.get("prompts", []))
|
||||
|
||||
@Property("QVariantList", constant=True)
|
||||
def guidanceChoices(self):
|
||||
mod = self._runner()
|
||||
return [{"value": k, "text": v} for k, v in (mod.GUIDANCE if mod else ())]
|
||||
|
||||
@Property(str, notify=guidanceChanged)
|
||||
def guidance(self):
|
||||
mod = self._runner()
|
||||
if not mod:
|
||||
return ""
|
||||
chosen = takes.read_json(os.path.join(self.store.base, "settings.json")).get("guidance")
|
||||
return chosen if chosen in dict(mod.GUIDANCE) else mod.default_guidance(
|
||||
self._session_options.get("recorded_demos_dir"))
|
||||
|
||||
@Slot(str)
|
||||
def setGuidance(self, value):
|
||||
mod = self._runner()
|
||||
if not mod or value not in dict(mod.GUIDANCE) or value == self.guidance:
|
||||
return
|
||||
path = os.path.join(self.store.base, "settings.json")
|
||||
settings = takes.read_json(path)
|
||||
settings["guidance"] = value
|
||||
takes.write_json(path, settings)
|
||||
self.guidanceChanged.emit()
|
||||
|
||||
# --- the checklist
|
||||
@Property("QVariantList", constant=True)
|
||||
def objects(self):
|
||||
@@ -617,7 +580,7 @@ class Backend(QObject):
|
||||
try:
|
||||
self._session = mod.Session(self.store.base, self.store.profile(), checklist, lighting, SCRIPT_PATH,
|
||||
on_status=lambda s: self._statusArrived.emit(dict(s)), auto=auto,
|
||||
quick=quick, guidance=self.guidance, **self._session_options)
|
||||
quick=quick, **self._session_options)
|
||||
except Exception as e:
|
||||
self.message.emit(f"Couldn't set up the session: {e}", True)
|
||||
return False
|
||||
@@ -1213,10 +1176,6 @@ def main():
|
||||
help="test: sessions start no processes and print the panel's commands")
|
||||
ap.add_argument("--speed", type=float, default=1.0, help="test, with --dry-run: run sessions this much faster")
|
||||
ap.add_argument("--poses", help="test: the pose pictures' folder (default hands/rec/poses)")
|
||||
ap.add_argument("--script", help="the recording script (default hands/rec/script.json; script-v2.json: "
|
||||
"the v2 script with demo clips)")
|
||||
ap.add_argument("--demos", help="the skeleton demo clips' folder, <id>.gif (default hands/rec/demos)")
|
||||
ap.add_argument("--recorded-demos", help="the recorded demo clips' folder (default hands/rec/demos-recorded)")
|
||||
ap.add_argument("--no-headset-button", action="store_true",
|
||||
help="sessions don't read the headset's button (Next, pause, resume)")
|
||||
ap.add_argument("--hub-dry-run", action="store_true",
|
||||
@@ -1240,22 +1199,12 @@ def main():
|
||||
options = {"dry_run": True, "speed": a.speed} if a.dry_run else {}
|
||||
if a.poses:
|
||||
options["poses_dir"] = a.poses
|
||||
if a.demos:
|
||||
options["demos_dir"] = a.demos
|
||||
if a.recorded_demos:
|
||||
options["recorded_demos_dir"] = a.recorded_demos
|
||||
if a.script:
|
||||
global SCRIPT_PATH
|
||||
SCRIPT_PATH = os.path.abspath(a.script if os.path.exists(a.script) else os.path.join(HERE, a.script))
|
||||
if a.no_headset_button:
|
||||
options["button"] = False
|
||||
if a.ignore_cameras:
|
||||
options["check_cameras"] = False
|
||||
backend = Backend(store, options, hub_dry_run=a.hub_dry_run, ignore_cameras=a.ignore_cameras)
|
||||
app.aboutToQuit.connect(backend.shutdown)
|
||||
mod = None if a.dry_run else backend._runner()
|
||||
if mod: # Frametop's drawn hands and hand pointer, if a session that was killed left them paused
|
||||
backend._thread(lambda: mod.resume_units(os.path.join(mod.run_dir(), mod.PAUSED_UNITS_FILE)))
|
||||
engine.rootContext().setContextProperty("backend", backend)
|
||||
engine.rootContext().setContextProperty("startPage", a.page)
|
||||
engine.load(QUrl.fromLocalFile(os.path.abspath(a.qml)))
|
||||
|
||||
+1
-74
@@ -529,21 +529,6 @@ Kirigami.ApplicationWindow {
|
||||
+ "know the steps already. "
|
||||
: "") + "Length: " + backend.planText(checklist.answers(), autoAdvance.checked, quickRound.checked) + "."
|
||||
}
|
||||
ColumnLayout {
|
||||
Kirigami.FormData.label: "Guidance:"
|
||||
visible: backend.scriptHasDemos
|
||||
Repeater {
|
||||
model: backend.guidanceChoices
|
||||
Controls.RadioButton {
|
||||
required property var modelData
|
||||
text: modelData.text + (modelData.value === "recorded"
|
||||
? ": clips of a real person's hands, from the headset's cameras"
|
||||
: ": a drawn hand, its palm light and its back dark")
|
||||
checked: backend.guidance === modelData.value
|
||||
onToggled: if (checked) backend.setGuidance(modelData.value)
|
||||
}
|
||||
}
|
||||
}
|
||||
Controls.Button {
|
||||
text: "Start"
|
||||
icon.name: "media-record"
|
||||
@@ -651,55 +636,6 @@ Kirigami.ApplicationWindow {
|
||||
}
|
||||
}
|
||||
|
||||
// A demo clip as the panel plays it (DESIGN.md "Demo clips"): looped, a left hand's
|
||||
// flipped, from its start again when a hold starts (run counts the starts), the key
|
||||
// to a skeleton clip's colours under it.
|
||||
component DemoClip: ColumnLayout {
|
||||
id: demoClip
|
||||
property string path
|
||||
property string mode
|
||||
property int run
|
||||
property bool paused
|
||||
property bool key: true
|
||||
spacing: Kirigami.Units.smallSpacing
|
||||
visible: path !== ""
|
||||
onRunChanged: clip.currentFrame = 0
|
||||
AnimatedImage {
|
||||
id: clip
|
||||
Layout.preferredWidth: Kirigami.Units.gridUnit * 20
|
||||
Layout.preferredHeight: Layout.preferredWidth * 9 / 16
|
||||
source: demoClip.path ? "file://" + demoClip.path : ""
|
||||
fillMode: Image.PreserveAspectFit
|
||||
mirror: demoClip.mode === "mirror"
|
||||
playing: demoClip.visible && !demoClip.paused
|
||||
cache: false
|
||||
smooth: true
|
||||
}
|
||||
RowLayout {
|
||||
Layout.alignment: Qt.AlignHCenter
|
||||
visible: demoClip.key
|
||||
spacing: Kirigami.Units.largeSpacing
|
||||
Repeater {
|
||||
model: [{ color: "#d5d1fe", text: "Light: palm" }, { color: "#6a62d8", text: "Dark: back of hand" }]
|
||||
RowLayout {
|
||||
required property var modelData
|
||||
spacing: Kirigami.Units.smallSpacing
|
||||
Rectangle {
|
||||
implicitWidth: Kirigami.Units.gridUnit * 0.8
|
||||
implicitHeight: implicitWidth
|
||||
radius: implicitWidth / 4
|
||||
color: parent.modelData.color
|
||||
}
|
||||
Controls.Label {
|
||||
text: parent.modelData.text
|
||||
font: Kirigami.Theme.smallFont
|
||||
opacity: 0.8
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Where to hold the hands: a front view (left, centre, right, up, down; the push
|
||||
// sections' chest, desk and eye) and how far out (near, mid, far), as on the panel.
|
||||
component WhereDiagram: RowLayout {
|
||||
@@ -844,17 +780,8 @@ Kirigami.ApplicationWindow {
|
||||
spacing: Kirigami.Units.gridUnit * 1.5
|
||||
|
||||
ColumnLayout {
|
||||
visible: !!sessionView.st.image || !!sessionView.st.demo || !!sessionView.st.position
|
||||
|| !!sessionView.st.distance
|
||||
visible: !!sessionView.st.image || !!sessionView.st.position || !!sessionView.st.distance
|
||||
spacing: Kirigami.Units.largeSpacing
|
||||
DemoClip {
|
||||
Layout.alignment: Qt.AlignHCenter
|
||||
path: sessionView.st.demo || ""
|
||||
mode: sessionView.st.demo_mode || ""
|
||||
run: sessionView.st.demo_run || 0
|
||||
paused: sessionView.state === "paused"
|
||||
key: !!sessionView.st.demo_key
|
||||
}
|
||||
PosePicture {
|
||||
Layout.alignment: Qt.AlignHCenter
|
||||
path: sessionView.st.image || ""
|
||||
|
||||
@@ -28,14 +28,6 @@
|
||||
// the pose picture (a PNG, square, RGBA, a right hand as the wearer
|
||||
// sees it) in a column on the left, the text moving right: mirror
|
||||
// flips it (a left hand), both shows a flipped copy on its left
|
||||
// demo <path> [mirror] [nokey] / demo restart / demo off
|
||||
// a demo clip (an animated GIF, hands/rec/demos) playing in a wider
|
||||
// column on the left, looped, with the key to its colours under it
|
||||
// (light: palm, dark: back of hand; nokey: none, for a clip of
|
||||
// real hands from hands/rec/demos-recorded); it takes the place of the
|
||||
// image. A new path starts it from its first frame, the same one
|
||||
// again leaves it playing (mirror only flips it), restart starts
|
||||
// it again. It stands still while paused
|
||||
// where <position|-> <distance|-> / where off
|
||||
// a small diagram under the picture: where to hold the hands, a
|
||||
// front view (centre, left, right, up, down; or the bar's chest,
|
||||
@@ -71,17 +63,14 @@
|
||||
// Each picture goes into the next of three shared buffers (linear DMA-BUFs SteamVR imported
|
||||
// once), and the overlay switches to it, as gaze/panel/ft-gazepanel.cpp does: SetOverlayRaw
|
||||
// flickered and lagged there, so it's only the fallback. The target has its own three small
|
||||
// buffers. Pictures are drawn only when a command changed something; while a demo clip plays,
|
||||
// each of its frames redraws only the clip's rectangle, and copies only that to a buffer that
|
||||
// already has the rest.
|
||||
// buffers. Pictures are drawn only when a command changed something.
|
||||
//
|
||||
// The pose log runs on a thread of its own, so a slow picture or command never delays a
|
||||
// sample. OpenVR's client isn't documented as thread-safe, so both threads take g_vr around
|
||||
// every OpenVR call.
|
||||
//
|
||||
// Options: --watch-stdin (quit when stdin closes: the session runs it), --socket NAME,
|
||||
// --distance METRES, --up DEG and --width DEG (UP_DEG and WIDTH_DEG: raise it out of a
|
||||
// recorded headset view), --no-vr. --no-vr is for testing without a headset: no SteamVR at all; each
|
||||
// --distance METRES, --no-vr. --no-vr is for testing without a headset: no SteamVR at all; each
|
||||
// picture's text goes to stdout instead (and with --dump DIR, the pictures to DIR/panel.pam
|
||||
// and DIR/target.pam), "poses start" writes nothing, "devices" and "head" reply with an error,
|
||||
// and "target" places the point as if the headset were at the room's origin.
|
||||
@@ -92,7 +81,6 @@
|
||||
#include "stb_truetype.h"
|
||||
#define STB_IMAGE_IMPLEMENTATION
|
||||
#define STBI_ONLY_PNG
|
||||
#define STBI_ONLY_GIF
|
||||
#define STBI_NO_HDR
|
||||
#define STBI_NO_LINEAR
|
||||
#pragma GCC diagnostic push
|
||||
@@ -119,7 +107,6 @@
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
@@ -385,100 +372,6 @@ Image &StripImage(const std::string &path) {
|
||||
}
|
||||
|
||||
// The where-to diagram's cells: a front view, 3 x 3, (column, row) from the top left.
|
||||
// A demo clip: an animated GIF decoded a frame at a time as it plays, so a long clip holds
|
||||
// three frames in memory, not all of them. stb_image keeps its GIF decoder to itself, but this
|
||||
// file compiles it, so it steps stbi__gif_load_next as stbi_load_gif_from_memory does.
|
||||
struct Demo {
|
||||
std::string path;
|
||||
bool mirror = false;
|
||||
bool key = true; // the key to the skeleton clips' colours under it
|
||||
std::vector<uint8_t> file;
|
||||
stbi__context ctx{};
|
||||
stbi__gif g{};
|
||||
int w = 0, h = 0, frame = -1; // frame: from 0 at the clip's start
|
||||
int loops = 0; // times it went back to the start by itself
|
||||
std::vector<uint8_t> cur, prev; // this frame and the one before (GIF's "restore to previous")
|
||||
int64_t due = 0; // when the next frame shows (NowNs)
|
||||
Picture scaled{0, 0, {}}; // this frame as drawn
|
||||
bool fresh = false; // scaled is out of date
|
||||
|
||||
Demo() = default;
|
||||
Demo(const Demo &) = delete;
|
||||
Demo &operator=(const Demo &) = delete;
|
||||
~Demo() { Free(); }
|
||||
|
||||
void Free() {
|
||||
STBI_FREE(g.out), STBI_FREE(g.history), STBI_FREE(g.background);
|
||||
std::memset(&g, 0, sizeof g);
|
||||
}
|
||||
|
||||
bool Load(const std::string &p) {
|
||||
FILE *f = std::fopen(p.c_str(), "rb");
|
||||
if (!f) return stbi__err("can't open file", "can't open file"), false;
|
||||
uint8_t chunk[65536];
|
||||
size_t n;
|
||||
while ((n = std::fread(chunk, 1, sizeof chunk, f)) > 0) file.insert(file.end(), chunk, chunk + n);
|
||||
std::fclose(f);
|
||||
path = p;
|
||||
return Start(NowNs());
|
||||
}
|
||||
|
||||
// From the first frame again.
|
||||
bool Start(int64_t now) {
|
||||
Free();
|
||||
stbi__start_mem(&ctx, file.data(), int(file.size()));
|
||||
cur.clear(), prev.clear(), frame = -1;
|
||||
return Next(now, false);
|
||||
}
|
||||
|
||||
// The next frame into cur, the first again after the last. false: the file is no GIF, or broken.
|
||||
bool Next(int64_t now, bool loop = true) {
|
||||
int comp = 0;
|
||||
stbi_uc *u = stbi__gif_load_next(&ctx, &g, &comp, 4, prev.empty() ? nullptr : prev.data());
|
||||
if (u == reinterpret_cast<stbi_uc *>(&ctx)) return loop && frame >= 0 && (++loops, Start(now)); // the end
|
||||
if (!u) return false;
|
||||
w = g.w, h = g.h, ++frame;
|
||||
prev.swap(cur);
|
||||
cur.assign(u, u + size_t(w) * h * 4);
|
||||
// As browsers: a delay under 20 ms is 100 ms.
|
||||
due = now + int64_t(g.delay >= 20 ? g.delay : 100) * 1'000'000;
|
||||
fresh = true;
|
||||
return true;
|
||||
}
|
||||
};
|
||||
|
||||
// The demo's frame drawn opaque at x0, y0 in dw x dh, averaging the source pixels each output
|
||||
// pixel covers (the frame is scaled again only when it or the size changed).
|
||||
void DrawDemo(Picture &pic, Demo &d, int x0, int y0, int dw, int dh) {
|
||||
if (d.cur.empty() || dw <= 0 || dh <= 0) return;
|
||||
if (d.fresh || d.scaled.w != dw || d.scaled.h != dh) {
|
||||
d.scaled = Picture{dw, dh, std::vector<uint8_t>(size_t(dw) * dh * 4)};
|
||||
const double sx = double(d.w) / dw, sy = double(d.h) / dh;
|
||||
for (int y = 0; y < dh; ++y) {
|
||||
const int ya = std::min(d.h - 1, int(y * sy)), yb = std::clamp(int(std::ceil((y + 1) * sy)), ya + 1, d.h);
|
||||
for (int x = 0; x < dw; ++x) {
|
||||
const int xa = std::min(d.w - 1, int(x * sx)), xb = std::clamp(int(std::ceil((x + 1) * sx)), xa + 1, d.w);
|
||||
unsigned r = 0, g = 0, b = 0, n = 0;
|
||||
for (int yy = ya; yy < yb; ++yy)
|
||||
for (int xx = xa; xx < xb; ++xx) {
|
||||
const uint8_t *q = &d.cur[(size_t(yy) * d.w + (d.mirror ? d.w - 1 - xx : xx)) * 4];
|
||||
r += q[0], g += q[1], b += q[2], ++n;
|
||||
}
|
||||
uint8_t *o = &d.scaled.px[(size_t(y) * dw + x) * 4];
|
||||
o[0] = uint8_t(r / n), o[1] = uint8_t(g / n), o[2] = uint8_t(b / n), o[3] = 255;
|
||||
}
|
||||
}
|
||||
d.fresh = false;
|
||||
}
|
||||
for (int y = 0; y < dh; ++y) {
|
||||
if (y0 + y < 0 || y0 + y >= pic.h) continue;
|
||||
const int xa = std::max(0, -x0), xb = std::min(dw, pic.w - x0);
|
||||
if (xb > xa)
|
||||
std::memcpy(&pic.px[(size_t(y0 + y) * pic.w + x0 + xa) * 4], &d.scaled.px[(size_t(y) * dw + xa) * 4],
|
||||
size_t(xb - xa) * 4);
|
||||
}
|
||||
}
|
||||
|
||||
bool WhereCell(const std::string &pos, int &col, int &row) {
|
||||
col = 1, row = 1;
|
||||
if (pos == "left") col = 0;
|
||||
@@ -505,9 +398,6 @@ int DistanceStep(const std::string &dist) { return dist == "near" ? 0 : dist ==
|
||||
const char *DistanceLabel(int k) { return k == 0 ? "Close" : k == 1 ? "Halfway out" : "Arm out"; }
|
||||
|
||||
constexpr double kCyan[3] = {0.3, 0.85, 1.0};
|
||||
// The demo clips' palm and back of the hand (hands/rec/demos): the key's swatches.
|
||||
constexpr double kPalm[3] = {0xd5 / 255.0, 0xd1 / 255.0, 0xfe / 255.0};
|
||||
constexpr double kBack[3] = {0x6a / 255.0, 0x62 / 255.0, 0xd8 / 255.0};
|
||||
|
||||
// The diagram in x0..x0+w, y0..y0+h: the front view on the left, the distance on the right
|
||||
// (or either alone, centred), each with its caption under it.
|
||||
@@ -565,8 +455,6 @@ struct Panel {
|
||||
std::string wherePos, whereDist; // the diagram's, "" for none
|
||||
std::vector<StripItem> strip; // a sweep's pictures, or none
|
||||
int stripCue = -1; // the lit one
|
||||
std::unique_ptr<Demo> demo; // the demo clip, or none
|
||||
int demoX = 0, demoY = 0, demoW = 0, demoH = 0; // where the last Draw put it (demoW 0: nowhere)
|
||||
// What the last Draw laid out, for --no-vr's printout.
|
||||
std::vector<std::string> lines, noteLines;
|
||||
double textDeg = 0;
|
||||
@@ -582,7 +470,6 @@ void Draw(Panel &p) {
|
||||
}
|
||||
Picture &pic = p.pic;
|
||||
pic.px = back.px;
|
||||
p.demoW = 0;
|
||||
const double ppd = kPxPerDeg, faint = 0.75;
|
||||
const int margin = int(ppd * 1.6), width = kW - 2 * margin;
|
||||
|
||||
@@ -646,8 +533,7 @@ void Draw(Panel &p) {
|
||||
// diagram at its right end; the text goes above it.
|
||||
int textX0 = margin, textX1 = kW - margin;
|
||||
int col, row;
|
||||
const bool demoOn = p.demo && p.strip.empty();
|
||||
const bool imageOn = !p.image.path.empty() && p.strip.empty() && !demoOn;
|
||||
const bool imageOn = !p.image.path.empty() && p.strip.empty();
|
||||
const bool whereOn = WhereCell(p.wherePos, col, row) || DistanceStep(p.whereDist) >= 0;
|
||||
if (!p.strip.empty()) {
|
||||
const int n = int(p.strip.size()), gap = int(ppd * 0.6), whereW = whereOn ? int(ppd * 9) : 0;
|
||||
@@ -670,37 +556,6 @@ void Draw(Panel &p) {
|
||||
if (whereOn) DrawWhere(pic, kW - margin - whereW, y0 + (side + labelH - int(ppd * 4.4)) / 2, whereW, int(ppd * 4.4),
|
||||
p.wherePos, p.whereDist);
|
||||
bottom = y0 - frame - int(ppd * 0.8);
|
||||
} else if (demoOn) {
|
||||
// The demo clip in a wider column, the key to its colours under it (on one line if it
|
||||
// fits; none for a clip of real hands), the text to the right.
|
||||
const int colW = int(ppd * 16), gap = int(ppd * 0.6), keySize = int(ppd * 0.75), keyLh = int(keySize * 1.6);
|
||||
const int sw = std::max(1, p.demo->w), sh = std::max(1, p.demo->h), swatch = keySize, sgap = int(ppd * 0.3);
|
||||
const std::string light = "Light: palm", dark = "Dark: back of hand";
|
||||
const int lightW = swatch + sgap + TextWidth(light, keySize), darkW = swatch + sgap + TextWidth(dark, keySize);
|
||||
const bool keyOn = p.demo->key, oneLine = lightW + int(ppd) + darkW <= colW;
|
||||
const int room = bottom - top - (keyOn ? gap + (oneLine ? 1 : 2) * keyLh : 0);
|
||||
int dw = colW, dh = dw * sh / sw;
|
||||
if (dh > room) dh = std::max(0, room), dw = dh * sw / sh;
|
||||
int y = top + std::max(0, (room - dh) / 2);
|
||||
const int x = margin + (colW - dw) / 2;
|
||||
RoundRect(pic, x - gap / 2, y - gap / 2, x + dw + gap / 2, y + dh + gap / 2, ppd * 0.6, 1, 1, 1, 0.05);
|
||||
DrawDemo(pic, *p.demo, x, y, dw, dh);
|
||||
p.demoX = x, p.demoY = y, p.demoW = dw, p.demoH = dh;
|
||||
y += dh + gap;
|
||||
auto key = [&](const std::string &label, const double *c, int kx, int cy) {
|
||||
RoundRect(pic, kx, cy - swatch / 2, kx + swatch, cy + swatch / 2, swatch * 0.25, c[0], c[1], c[2], 1);
|
||||
Text(pic, label, keySize, kx + swatch + sgap, cy, 0.85, kLeft);
|
||||
};
|
||||
if (keyOn && oneLine) {
|
||||
const int kx = margin + (colW - lightW - int(ppd) - darkW) / 2;
|
||||
key(light, kPalm, kx, y + keyLh / 2);
|
||||
key(dark, kBack, kx + lightW + int(ppd), y + keyLh / 2);
|
||||
} else if (keyOn) {
|
||||
const int kx = margin + (colW - std::max(lightW, darkW)) / 2;
|
||||
key(light, kPalm, kx, y + keyLh / 2);
|
||||
key(dark, kBack, kx, y + keyLh + keyLh / 2);
|
||||
}
|
||||
textX0 = margin + colW + int(ppd * 1.2);
|
||||
} else if (imageOn || whereOn) {
|
||||
const int colW = int(ppd * 14), gap = int(ppd * 0.6);
|
||||
const int whereH = whereOn ? int(ppd * 4.4) : 0, between = imageOn && whereOn ? gap : 0;
|
||||
@@ -722,9 +577,6 @@ void Draw(Panel &p) {
|
||||
textX0 = margin + colW + int(ppd * 1.2);
|
||||
}
|
||||
const int textW = textX1 - textX0, textCx = (textX0 + textX1) / 2;
|
||||
// Beside a demo clip the text is its list of motions: from the left, not centred.
|
||||
const int textX = demoOn ? textX0 : textCx;
|
||||
const Align textAlign = demoOn ? kLeft : kCentre;
|
||||
|
||||
// The instruction, the note, the big countdown and the action line, centred together in
|
||||
// the space between. A long instruction gets smaller, down to 1 degree a line, before it runs over.
|
||||
@@ -745,21 +597,20 @@ void Draw(Panel &p) {
|
||||
auto gapBefore = [&] { if (!first) y += gapY; first = false; };
|
||||
if (!p.lines.empty()) {
|
||||
first = false;
|
||||
for (const std::string &line : p.lines) Text(pic, line, size, textX, y + lh / 2, 1.0, textAlign), y += lh;
|
||||
for (const std::string &line : p.lines) Text(pic, line, size, textCx, y + lh / 2, 1.0), y += lh;
|
||||
}
|
||||
if (!p.noteLines.empty()) {
|
||||
gapBefore();
|
||||
for (const std::string &line : p.noteLines)
|
||||
Text(pic, line, noteSize, textX, y + noteLh / 2, 1.0, 0.62, 0.3, textAlign), y += noteLh;
|
||||
for (const std::string &line : p.noteLines) Text(pic, line, noteSize, textCx, y + noteLh / 2, 1.0, 0.62, 0.3), y += noteLh;
|
||||
}
|
||||
if (!p.big.empty()) {
|
||||
gapBefore();
|
||||
Text(pic, p.big, bigSize, textX, y + bigLh / 2, kCyan[0], kCyan[1], kCyan[2], textAlign), y += bigLh;
|
||||
Text(pic, p.big, bigSize, textCx, y + bigLh / 2, kCyan[0], kCyan[1], kCyan[2]), y += bigLh;
|
||||
}
|
||||
if (!actionLines.empty()) {
|
||||
gapBefore();
|
||||
for (const std::string &line : actionLines)
|
||||
Text(pic, line, actionSize, textX, y + actionLh / 2, kCyan[0], kCyan[1], kCyan[2], textAlign), y += actionLh;
|
||||
Text(pic, line, actionSize, textCx, y + actionLh / 2, kCyan[0], kCyan[1], kCyan[2]), y += actionLh;
|
||||
}
|
||||
|
||||
if (p.paused) {
|
||||
@@ -838,10 +689,6 @@ void Print(const Panel &p, const Target &t, bool visible, int number) {
|
||||
std::printf("image: %s %dx%d%s%s\n", p.image.path.c_str(), p.image.src.w, p.image.src.h,
|
||||
p.imageMode.empty() ? "" : " ", p.imageMode.c_str());
|
||||
else std::printf("image: off\n");
|
||||
if (p.demo)
|
||||
std::printf("demo: %s %dx%d frame %d loop %d%s%s, drawn %dx%d\n", p.demo->path.c_str(), p.demo->w, p.demo->h,
|
||||
p.demo->frame, p.demo->loops, p.demo->mirror ? " mirror" : "", p.demo->key ? "" : " nokey", p.demoW, p.demoH);
|
||||
else std::printf("demo: off\n");
|
||||
if (!p.wherePos.empty() || !p.whereDist.empty())
|
||||
std::printf("where: %s %s\n", p.wherePos.empty() ? "-" : p.wherePos.c_str(), p.whereDist.empty() ? "-" : p.whereDist.c_str());
|
||||
else std::printf("where: off\n");
|
||||
@@ -867,11 +714,6 @@ struct Buffer {
|
||||
gbm_bo *bo = nullptr;
|
||||
int fd = -1;
|
||||
vr::SharedTextureHandle_t handle = 0;
|
||||
uint64_t whole = 0; // the whole picture it last had (Buffers::whole), 0: none
|
||||
};
|
||||
|
||||
struct Area {
|
||||
int x, y, w, h;
|
||||
};
|
||||
|
||||
struct Buffers {
|
||||
@@ -881,7 +723,6 @@ struct Buffers {
|
||||
Buffer b[3];
|
||||
int next = 0;
|
||||
bool ok = false;
|
||||
uint64_t whole = 0; // counts the pictures presented whole
|
||||
|
||||
bool Make(int width, int height) {
|
||||
w = width, h = height;
|
||||
@@ -925,9 +766,8 @@ struct Buffers {
|
||||
|
||||
// A picture to the overlay: into the next buffer, premultiplied (the overlay's flag says
|
||||
// so), then the overlay switches to it. The copy runs outside g_vr, so the pose log waits
|
||||
// only for the two calls at the end. With area (a demo clip's frame): only that changed
|
||||
// since the last whole picture, so a buffer that has that picture gets only the area.
|
||||
void Present(vr::IVROverlay *ov, vr::VROverlayHandle_t oh, const Picture &p, const Area *area = nullptr) {
|
||||
// only for the two calls at the end.
|
||||
void Present(vr::IVROverlay *ov, vr::VROverlayHandle_t oh, const Picture &p) {
|
||||
vr::EVROverlayError e;
|
||||
if (!ok) {
|
||||
std::lock_guard<std::mutex> lk(g_vr);
|
||||
@@ -935,11 +775,6 @@ struct Buffers {
|
||||
} else {
|
||||
Buffer &x = b[next];
|
||||
next = (next + 1) % 3;
|
||||
if (!area) ++whole;
|
||||
Area all{0, 0, p.w, p.h}, a = area && x.whole == whole ? *area : all;
|
||||
a.x = std::clamp(a.x, 0, p.w), a.y = std::clamp(a.y, 0, p.h);
|
||||
a.w = std::clamp(a.w, 0, p.w - a.x), a.h = std::clamp(a.h, 0, p.h - a.y);
|
||||
x.whole = whole;
|
||||
uint32_t stride = 0;
|
||||
void *mapping = nullptr;
|
||||
auto *dst = static_cast<uint8_t *>(gbm_bo_map(x.bo, 0, 0, p.w, p.h, GBM_BO_TRANSFER_WRITE, &stride, &mapping));
|
||||
@@ -947,10 +782,10 @@ struct Buffers {
|
||||
std::fprintf(stderr, "ft-handpanel: can't map a buffer\n");
|
||||
return;
|
||||
}
|
||||
for (int y = a.y; y < a.y + a.h; ++y) {
|
||||
const uint8_t *src = &p.px[(size_t(y) * p.w + a.x) * 4];
|
||||
uint8_t *row = dst + size_t(y) * stride + size_t(a.x) * 4;
|
||||
for (int i = 0; i < a.w * 4; i += 4) {
|
||||
for (int y = 0; y < p.h; ++y) {
|
||||
const uint8_t *src = &p.px[size_t(y) * p.w * 4];
|
||||
uint8_t *row = dst + size_t(y) * stride;
|
||||
for (int i = 0; i < p.w * 4; i += 4) {
|
||||
const unsigned a = src[i + 3];
|
||||
row[i] = uint8_t(src[i] * a / 255), row[i + 1] = uint8_t(src[i + 1] * a / 255);
|
||||
row[i + 2] = uint8_t(src[i + 2] * a / 255), row[i + 3] = uint8_t(a);
|
||||
@@ -1098,17 +933,15 @@ bool HandState(const std::string &s) { return s == "seen" || s == "lost" || s ==
|
||||
int main(int argc, char **argv) {
|
||||
bool watchStdin = false, noVr = false;
|
||||
std::string sockName = "ft_handpanel", dumpDir;
|
||||
double distance = 1.2, upDeg = kUpDeg, widthDeg = kWidthDeg;
|
||||
double distance = 1.2;
|
||||
for (int i = 1; i < argc; ++i) {
|
||||
if (!std::strcmp(argv[i], "--watch-stdin")) watchStdin = true;
|
||||
else if (!std::strcmp(argv[i], "--no-vr")) noVr = true;
|
||||
else if (!std::strcmp(argv[i], "--socket") && i + 1 < argc) sockName = argv[++i];
|
||||
else if (!std::strcmp(argv[i], "--distance") && i + 1 < argc) distance = std::clamp(std::atof(argv[++i]), 0.5, 5.0);
|
||||
else if (!std::strcmp(argv[i], "--up") && i + 1 < argc) upDeg = std::clamp(std::atof(argv[++i]), -60.0, 60.0);
|
||||
else if (!std::strcmp(argv[i], "--width") && i + 1 < argc) widthDeg = std::clamp(std::atof(argv[++i]), 5.0, 90.0);
|
||||
else if (!std::strcmp(argv[i], "--dump") && i + 1 < argc) dumpDir = argv[++i];
|
||||
else {
|
||||
std::fprintf(stderr, "usage: %s [--watch-stdin] [--socket NAME] [--distance METRES] [--up DEG] [--width DEG] [--no-vr [--dump DIR]]\n",
|
||||
std::fprintf(stderr, "usage: %s [--watch-stdin] [--socket NAME] [--distance METRES] [--no-vr [--dump DIR]]\n",
|
||||
argv[0]);
|
||||
return 2;
|
||||
}
|
||||
@@ -1159,14 +992,14 @@ int main(int argc, char **argv) {
|
||||
if (targetBuffers.Make(kTargetPx, kTargetPx)) ov->SetOverlayFlag(th, vr::VROverlayFlags_IsPremultiplied, true);
|
||||
// Fixed to the head: distance along a line UP_DEG above straight ahead, tilted back by
|
||||
// as much so it faces the eyes square on.
|
||||
const double up = upDeg * M_PI / 180;
|
||||
const double up = kUpDeg * M_PI / 180;
|
||||
vr::HmdMatrix34_t m{};
|
||||
m.m[0][0] = 1;
|
||||
m.m[1][1] = float(std::cos(up)), m.m[1][2] = float(-std::sin(up));
|
||||
m.m[2][1] = float(std::sin(up)), m.m[2][2] = float(std::cos(up));
|
||||
m.m[1][3] = float(distance * std::sin(up)), m.m[2][3] = float(-distance * std::cos(up));
|
||||
ov->SetOverlayTransformTrackedDeviceRelative(h, vr::k_unTrackedDeviceIndex_Hmd, &m);
|
||||
ov->SetOverlayWidthInMeters(h, float(2 * distance * std::tan(widthDeg * M_PI / 360)));
|
||||
ov->SetOverlayWidthInMeters(h, float(2 * distance * std::tan(kWidthDeg * M_PI / 360)));
|
||||
ov->SetOverlayWidthInMeters(th, float(kTargetM));
|
||||
}
|
||||
LoadFont();
|
||||
@@ -1181,10 +1014,7 @@ int main(int argc, char **argv) {
|
||||
|
||||
while (!g_stop) {
|
||||
pollfd pfd{sock, POLLIN, 0};
|
||||
int wait = 50;
|
||||
if (p.demo && visible && !p.paused) // up to the demo's next frame
|
||||
wait = int(std::clamp<int64_t>((p.demo->due - NowNs()) / 1'000'000 + 1, 0, 50));
|
||||
poll(&pfd, 1, wait);
|
||||
poll(&pfd, 1, 50);
|
||||
char buf[4096];
|
||||
sockaddr_un from{};
|
||||
socklen_t fromLen = sizeof from;
|
||||
@@ -1270,34 +1100,6 @@ int main(int argc, char **argv) {
|
||||
} else {
|
||||
p.imageMode = mode, dirty = true;
|
||||
}
|
||||
} else if (Is(buf, "demo", &rest)) {
|
||||
// demo <path> [mirror] [nokey] / demo restart / demo off: the path may hold spaces
|
||||
std::string path = rest;
|
||||
auto flag = [&path](const char *word) {
|
||||
const size_t n = std::strlen(word) + 1;
|
||||
if (path.size() <= n || path.compare(path.size() - n, n, std::string(" ") + word)) return false;
|
||||
path.resize(path.size() - n);
|
||||
return true;
|
||||
};
|
||||
const bool nokey = flag("nokey");
|
||||
const bool mirror = flag("mirror");
|
||||
if (path.empty()) {
|
||||
reply = "error usage: demo <path> [mirror] [nokey] | demo restart | demo off";
|
||||
} else if (path == "off") {
|
||||
p.demo.reset(), dirty = true;
|
||||
} else if (path == "restart") {
|
||||
if (p.demo) p.demo->loops = 0;
|
||||
if (p.demo && !p.demo->Start(NowNs())) reply = "error can't read " + p.demo->path, p.demo.reset();
|
||||
dirty = true;
|
||||
} else if (!p.demo || path != p.demo->path) {
|
||||
auto d = std::make_unique<Demo>();
|
||||
d->mirror = mirror, d->key = !nokey;
|
||||
if (d->Load(path)) p.demo = std::move(d);
|
||||
else reply = "error can't read " + path + ": " + stbi_failure_reason(), p.demo.reset();
|
||||
dirty = true; // no clip rather than the last one
|
||||
} else if (mirror != p.demo->mirror || nokey == p.demo->key) {
|
||||
p.demo->mirror = mirror, p.demo->key = !nokey, p.demo->fresh = dirty = true;
|
||||
}
|
||||
} else if (Is(buf, "strip", &rest)) {
|
||||
// strip <cue> <path>|<mode>|<label>;...
|
||||
int cue = -1, used = 0;
|
||||
@@ -1443,16 +1245,6 @@ int main(int argc, char **argv) {
|
||||
g_stop = true;
|
||||
}
|
||||
}
|
||||
// The demo's next frame when it's due: drawn into the picture's clip rectangle alone,
|
||||
// unless something else changed too (then the whole picture). Hidden or paused, it waits.
|
||||
bool frameNow = false;
|
||||
if (p.demo && visible && !p.paused && NowNs() >= p.demo->due) {
|
||||
const int64_t now = NowNs();
|
||||
if (now - p.demo->due > 500'000'000) p.demo->due = now; // after a wait: no catching up
|
||||
if (!p.demo->Next(p.demo->due))
|
||||
std::fprintf(stderr, "ft-handpanel: %s: %s\n", p.demo->path.c_str(), stbi_failure_reason()), p.demo.reset(), dirty = true;
|
||||
else frameNow = !dirty && p.demoW > 0;
|
||||
}
|
||||
const bool panelNow = visible && dirty, targetNow = targetDirty;
|
||||
if (panelNow) {
|
||||
Draw(p);
|
||||
@@ -1462,10 +1254,6 @@ int main(int argc, char **argv) {
|
||||
if (!shown) ov->ShowOverlay(h), shown = true;
|
||||
}
|
||||
dirty = false;
|
||||
} else if (frameNow) {
|
||||
DrawDemo(p.pic, *p.demo, p.demoX, p.demoY, p.demoW, p.demoH);
|
||||
const Area area{p.demoX, p.demoY, p.demoW, p.demoH};
|
||||
if (ov && shown) buffers.Present(ov, h, p.pic, &area);
|
||||
}
|
||||
if (targetNow) {
|
||||
if (t.on) {
|
||||
|
||||
@@ -1,115 +0,0 @@
|
||||
{
|
||||
"version": 1,
|
||||
"name": "v2",
|
||||
"intro_s": 4,
|
||||
"between_s": 3,
|
||||
"welcome": {"title": "Hand recorder", "seconds": 8, "text": "Follow the steps on this panel. Each one plays a short clip of what to do: copy it. In the clips the palm is light and the back of the hand is dark.|The button on the right side of the headset goes on when a step waits for you, and pauses while recording."},
|
||||
"done": {"title": "All done", "seconds": 6, "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."},
|
||||
"stopped": {"title": "Stopped", "seconds": 4, "text": "What was recorded so far is saved. Review it in the Hand recorder window."},
|
||||
"objects": {"pencil": "pencil or pen", "phone": "phone", "cup": "cup", "keyboard": "keyboard", "mouse": "mouse", "gamepad": "gamepad", "small": "small thing"},
|
||||
"sections": [
|
||||
{"id": "hand-size", "title": "Hand size", "requires": [], "go": "Hold", "quick": true, "intro": "First, your hand size. Copy the clip: both hands about 40 cm out, still, about 1 s for each shape.",
|
||||
"prompts": [
|
||||
{"text": "Both hands about 40 cm out, still|Palms toward you, then backs toward you|Fingers spread, then a fist, then point|About 1 s for each shape", "seconds": 10, "hands": "both", "demo": "1", "distance": "mid"}
|
||||
]},
|
||||
|
||||
{"id": "index-taps", "title": "Index taps", "requires": [], "quick": true, "intro": "Thumb to index fingertip taps: light and lazy, about 2 a second, the way you'd click.",
|
||||
"prompts": [
|
||||
{"text": "Left hand only, the right one down and out of view.|One hand low and relaxed, other fingers loosely curled: look down at it|2 light taps palm down|2 taps edge-on, 2 palm up, 2 palm toward your face", "seconds": 12, "hands": "left", "demo": "2a", "demo_mirror": true, "quick": false},
|
||||
{"text": "Right hand only, the left one down and out of view.|One hand low and relaxed, other fingers loosely curled: look down at it|2 light taps palm down|2 taps edge-on, 2 palm up, 2 palm toward your face", "seconds": 12, "hands": "right", "demo": "2a"},
|
||||
{"text": "Now both hands at once, each doing the same.|One hand low and relaxed, other fingers loosely curled: look down at it|2 light taps palm down|2 taps edge-on, 2 palm up, 2 palm toward your face", "seconds": 12, "hands": "both", "demo": "2a", "quick": false},
|
||||
{"text": "Left hand only, the right one down and out of view.|Keep tapping while you move: centre at chest height|Far out to your side, then crossed past the middle|Close to your face, then at arm's length|Quick taps, a double tap, one slow press", "seconds": 18, "hands": "left", "demo": "2b", "demo_mirror": true, "quick": false},
|
||||
{"text": "Right hand only, the left one down and out of view.|Keep tapping while you move: centre at chest height|Far out to your side, then crossed past the middle|Close to your face, then at arm's length|Quick taps, a double tap, one slow press", "seconds": 18, "hands": "right", "demo": "2b", "quick": false},
|
||||
{"text": "Now both hands at once, each doing the same.|Keep tapping while you move: centre at chest height|Far out to your side, then crossed past the middle|Close to your face, then at arm's length|Quick taps, a double tap, one slow press", "seconds": 18, "hands": "both", "demo": "2b", "quick": false},
|
||||
{"text": "Both hands tapping|Then one hand over the other, still tapping", "seconds": 10, "hands": "both", "demo": "2c", "quick": false}
|
||||
]},
|
||||
|
||||
{"id": "middle-taps", "title": "Middle taps", "requires": [], "intro": "Now the thumb meets the middle finger's pad. Keep the index half-curled, out of the way.",
|
||||
"prompts": [
|
||||
{"text": "Left hand only, the right one down and out of view.|Thumb to the middle finger's pad, index half-curled: look down at the hand|2 taps palm down, edge-on, palm up, toward your face|Then a few taps out to the side and close to your face", "seconds": 12, "hands": "left", "demo": "3a", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|Thumb to the middle finger's pad, index half-curled: look down at the hand|2 taps palm down, edge-on, palm up, toward your face|Then a few taps out to the side and close to your face", "seconds": 12, "hands": "right", "demo": "3a"},
|
||||
{"text": "Now both hands at once, each doing the same.|Thumb to the middle finger's pad, index half-curled: look down at the hand|2 taps palm down, edge-on, palm up, toward your face|Then a few taps out to the side and close to your face", "seconds": 12, "hands": "both", "demo": "3a"},
|
||||
{"text": "Left hand only, the right one down and out of view.|3 slow, firm presses|Keep tapping and turn your head until the hand reaches the edge of the view|Then turn the other way", "seconds": 13, "hands": "left", "demo": "3b", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|3 slow, firm presses|Keep tapping and turn your head until the hand reaches the edge of the view|Then turn the other way", "seconds": 13, "hands": "right", "demo": "3b"},
|
||||
{"text": "Now both hands at once, each doing the same.|3 slow, firm presses|Keep tapping and turn your head until the hand reaches the edge of the view|Then turn the other way", "seconds": 13, "hands": "both", "demo": "3b"}
|
||||
]},
|
||||
|
||||
{"id": "look-alikes", "title": "Look-alikes", "requires": [], "intro": "Things that look like taps but aren't: near misses, rubbing, other fingers, fists. None of them should touch thumb to index.",
|
||||
"prompts": [
|
||||
{"text": "Left hand only, the right one down and out of view.|Stop 1 cm short of a tap|Rub the thumb along the side of the index|Ring finger to thumb|Let a loosely curled hand rest", "seconds": 12, "hands": "left", "demo": "4a", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|Stop 1 cm short of a tap|Rub the thumb along the side of the index|Ring finger to thumb|Let a loosely curled hand rest", "seconds": 12, "hands": "right", "demo": "4a"},
|
||||
{"text": "Now both hands at once, each doing the same.|Stop 1 cm short of a tap|Rub the thumb along the side of the index|Ring finger to thumb|Let a loosely curled hand rest", "seconds": 12, "hands": "both", "demo": "4a"},
|
||||
{"text": "Left hand only, the right one down and out of view.|Close a fist slowly, open it|Close it fast, open it|Once at chest height, once low", "seconds": 12, "hands": "left", "demo": "4b", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|Close a fist slowly, open it|Close it fast, open it|Once at chest height, once low", "seconds": 12, "hands": "right", "demo": "4b"},
|
||||
{"text": "Now both hands at once, each doing the same.|Close a fist slowly, open it|Close it fast, open it|Once at chest height, once low", "seconds": 12, "hands": "both", "demo": "4b"}
|
||||
]},
|
||||
|
||||
{"id": "hold-drag", "title": "Hold and drag", "requires": [], "intro": "Pinch and hold, then move while holding, as if dragging a window.",
|
||||
"prompts": [
|
||||
{"text": "Left hand only, the right one down and out of view.|Pinch and hold 3 s|Drag up, down, left, right|Roll the wrist while holding|Loosen the other fingers without letting go", "seconds": 13, "hands": "left", "demo": "5a", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|Pinch and hold 3 s|Drag up, down, left, right|Roll the wrist while holding|Loosen the other fingers without letting go", "seconds": 13, "hands": "right", "demo": "5a"},
|
||||
{"text": "Now both hands at once, each doing the same.|Pinch and hold 3 s|Drag up, down, left, right|Roll the wrist while holding|Loosen the other fingers without letting go", "seconds": 13, "hands": "both", "demo": "5a"},
|
||||
{"text": "One hand holds a pinch, the other taps|Hands apart, then close, then crossed", "seconds": 10, "hands": "both", "demo": "5b"}
|
||||
]},
|
||||
|
||||
{"id": "scroll", "title": "Three-finger scroll", "requires": [], "intro": "Thumb on the index and middle fingertips together, held while the hand moves.",
|
||||
"prompts": [
|
||||
{"text": "Left hand only, the right one down and out of view.|Thumb on the index and middle tips|Move up and down slowly, then fast|Let the ring finger join once", "seconds": 11, "hands": "left", "demo": "6", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|Thumb on the index and middle tips|Move up and down slowly, then fast|Let the ring finger join once", "seconds": 11, "hands": "right", "demo": "6"},
|
||||
{"text": "Now both hands at once, each doing the same.|Thumb on the index and middle tips|Move up and down slowly, then fast|Let the ring finger join once", "seconds": 11, "hands": "both", "demo": "6"}
|
||||
]},
|
||||
|
||||
{"id": "palm-up", "title": "Palm up", "requires": [], "quick": true, "intro": "Palms up. This is where tracking gets lost today, so take your time: the fingertips drop toward the floor, then rise toward you.",
|
||||
"prompts": [
|
||||
{"text": "Arms forward, palms up, fingers flat: hands close (elbows tucked)|Fingertips drop toward the floor and back: 1 slow, 2 fast|Same with forearms half out, then arms straight|One drop low at the waist, one at eye height", "seconds": 19, "hands": "both", "demo": "7a"},
|
||||
{"text": "Palms up at the same three distances|Fingertips curl up toward you and back: 1 slow, 2 fast each", "seconds": 15, "hands": "both", "demo": "7b", "quick": false},
|
||||
{"text": "Left hand only, the right one down and out of view.|Palm up, hold a pinch|Drop the fingertips and raise them, still pinching|Flip palm up and down with the wrist bent", "seconds": 12, "hands": "left", "demo": "7c", "demo_mirror": true, "quick": false},
|
||||
{"text": "Right hand only, the left one down and out of view.|Palm up, hold a pinch|Drop the fingertips and raise them, still pinching|Flip palm up and down with the wrist bent", "seconds": 12, "hands": "right", "demo": "7c", "quick": false},
|
||||
{"text": "Now both hands at once, each doing the same.|Palm up, hold a pinch|Drop the fingertips and raise them, still pinching|Flip palm up and down with the wrist bent", "seconds": 12, "hands": "both", "demo": "7c", "quick": false}
|
||||
]},
|
||||
|
||||
{"id": "strokes", "title": "Strokes and gestures", "requires": [], "intro": "Quick strokes, a wave, and talking with your hands.",
|
||||
"prompts": [
|
||||
{"text": "Left hand only, the right one down and out of view.|Lazy susan stroke each way, with the return stroke|A wave|5 s of talking with your hands", "seconds": 14, "hands": "left", "demo": "8", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|Lazy susan stroke each way, with the return stroke|A wave|5 s of talking with your hands", "seconds": 14, "hands": "right", "demo": "8"},
|
||||
{"text": "Now both hands at once, each doing the same.|Lazy susan stroke each way, with the return stroke|A wave|5 s of talking with your hands", "seconds": 14, "hands": "both", "demo": "8"}
|
||||
]},
|
||||
|
||||
{"id": "reach", "title": "Reach and pull", "requires": [], "intro": "Reaching out: grab something far away and pull it in, push and flick away, then a real object. Have a cup or your phone in reach.",
|
||||
"prompts": [
|
||||
{"text": "Left hand only, the right one down and out of view.|Open palm at a far target|Close to a fist and pull back", "seconds": 8, "hands": "left", "demo": "9a", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|Open palm at a far target|Close to a fist and pull back", "seconds": 8, "hands": "right", "demo": "9a"},
|
||||
{"text": "Now both hands at once, each doing the same.|Open palm at a far target|Close to a fist and pull back", "seconds": 8, "hands": "both", "demo": "9a"},
|
||||
{"text": "Left hand only, the right one down and out of view.|Push away, then flick away|Reach for a cup or your phone", "seconds": 10, "hands": "left", "demo": "9b", "demo_mirror": true},
|
||||
{"text": "Right hand only, the left one down and out of view.|Push away, then flick away|Reach for a cup or your phone", "seconds": 10, "hands": "right", "demo": "9b"},
|
||||
{"text": "Now both hands at once, each doing the same.|Push away, then flick away|Reach for a cup or your phone", "seconds": 10, "hands": "both", "demo": "9b"}
|
||||
]},
|
||||
|
||||
{"id": "fists", "title": "Fists", "requires": [], "intro": "Two fists, as if holding a big screen by its edges.",
|
||||
"prompts": [
|
||||
{"text": "Two fists: hold, pull apart, push together|Fists with the palms in, down, then up", "seconds": 9, "hands": "both", "demo": "10"}
|
||||
]},
|
||||
|
||||
{"id": "edges", "title": "Edges and exits", "requires": [], "intro": "Hands leaving the view and coming back. Keep your head still and go slowly near the edges.",
|
||||
"prompts": [
|
||||
{"text": "Right hand open: slide out past the right edge until gone, wait 1 s, back|Same up past your face|Left hand pinching: out past the left edge, then down past your lap", "seconds": 19, "hands": "both", "demo": "11a"},
|
||||
{"text": "Both hands cross fully to the opposite sides|Hold a moment, come back", "seconds": 10, "hands": "both", "demo": "11b"}
|
||||
]},
|
||||
|
||||
{"id": "desk-work", "title": "Desk work", "requires": [], "intro": "Now your desk. Look down at your hands while you work.", "defaults": {"position": "down"},
|
||||
"prompts": [
|
||||
{"text": "Type a few words, use the mouse, pick up the phone|One tap, then back to the keys", "seconds": 15, "hands": "both", "demo": "12", "when": "objects:keyboard"},
|
||||
{"text": "No keyboard: rest your hands on the desk and pretend.|Type a few words, use the mouse, pick up the phone|One tap, then back to the keys", "seconds": 15, "hands": "both", "demo": "12", "when": "!objects:keyboard"}
|
||||
]},
|
||||
|
||||
{"id": "controllers", "title": "Controllers", "requires": ["controllers"], "intro": "Put both controllers on the desk in front of you, in view.",
|
||||
"prompts": [
|
||||
{"text": "Pick up the controllers, set them down|A tap in between", "seconds": 10, "hands": "both", "demo": "13"}
|
||||
]},
|
||||
|
||||
{"id": "no-hands", "title": "No hands", "requires": [], "quick": true, "intro": "Last one: put your hands down, out of view.",
|
||||
"prompts": [
|
||||
{"text": "Keep your hands out of view and look slowly around the room.", "seconds": 10, "hands": "none", "picture": "no-hands"}
|
||||
]}
|
||||
]
|
||||
}
|
||||
+19
-271
@@ -57,28 +57,9 @@ FT_CAMD = os.path.join(HANDS, "build", "ft-camd")
|
||||
PANEL_BIN = os.path.join(HERE, "build", "ft-handpanel")
|
||||
SCRIPT_PATH = os.path.join(HERE, "script.json")
|
||||
POSES_DIR = os.path.join(HERE, "poses") # the pose pictures: poses.json and its PNGs
|
||||
DEMOS_DIR = os.path.join(HERE, "demos") # the demo clips: <id>.gif (DESIGN.md "Demo clips")
|
||||
RECORDED_DEMOS_DIR = os.path.join(HERE, "demos-recorded") # the same parts from real camera frames
|
||||
# The demo clips a contributor follows: recorded (a real person's hands, from a session's side
|
||||
# cameras: demo/camgifs.py) or skeleton (the drawn hand, its palm light and its back dark).
|
||||
GUIDANCE = (("recorded", "Recorded hands"), ("skeleton", "Skeleton hands"))
|
||||
BASE_DIR = os.path.expanduser("~/.local/share/frametop/hands/contrib")
|
||||
PANEL_SOCKET = "ft_handpanel"
|
||||
CAMD_UNIT = "frametop-handrec-camd.service"
|
||||
# Frametop's live hand tracking (hands/run.sh, ft-handsctl): an ft-camd the recorder uses as it is
|
||||
LIVE_CAMD_UNIT, LIVE_HANDS_UNIT = "frametop-camd.service", "frametop-hands.service"
|
||||
# Frametop's drawn hands and hand pointer: off for a session, so no hand overlay is in view and no gesture
|
||||
# clicks or opens Steam's menu while the person performs the steps (Session._quiet_hands)
|
||||
QUIET_UNITS = ("frametop-handview.service", "frametop-pointer.service")
|
||||
PAUSED_UNITS_FILE = "handrec-paused-units" # in run_dir(): the ones to start again, if a session didn't
|
||||
SIDE_CAMERAS = ("slam_left", "slam_right") # a recording needs these (Session._first_set)
|
||||
CAMERAS_BACK_S = 120 # how long a lost recording waits for the cameras (Session._cameras_back)
|
||||
STALL_S = 3.0 # a recording with no new set for this long is lost (Session._watch_recording)
|
||||
MAX_CAMERA_LOSSES = 3 # then the session stops
|
||||
CAMERAS_TITLE = "Waiting for the cameras"
|
||||
CAMERAS_TEXT = ("The headset's cameras restarted (SteamVR moves them when the colour camera module is unplugged "
|
||||
"or plugged in), or they stopped. If you took the headset off, put it back on. This step goes "
|
||||
"again once they're back.")
|
||||
HANDS_UNIT = "frametop-handrec-hands.service"
|
||||
# The standalone Hand Recorder (takes.standalone()): its binaries run on the host, so ft-hands
|
||||
# starts directly rather than in the dev container, and repairs mean its install command.
|
||||
@@ -393,45 +374,6 @@ def stop_unit(unit):
|
||||
subprocess.run(host_command("systemctl", "--user", "stop", unit), capture_output=True, timeout=30)
|
||||
|
||||
|
||||
def pause_units(units, path, log=lambda line: None):
|
||||
"""Stop those of units that are running, listed in path first so resume_units starts them again
|
||||
even after a crash. Returns the ones stopped."""
|
||||
paused = [u for u in units if unit_active(u)]
|
||||
if not paused:
|
||||
return []
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w") as f:
|
||||
json.dump(paused, f)
|
||||
for unit in paused:
|
||||
stop_unit(unit)
|
||||
log("paused %s for the session" % unit)
|
||||
return paused
|
||||
|
||||
|
||||
def resume_units(path, log=lambda line: None):
|
||||
"""Start the units pause_units listed in path, and forget them. Returns them."""
|
||||
try:
|
||||
with open(path) as f:
|
||||
units = [u for u in json.load(f) if isinstance(u, str) and u.endswith(".service")]
|
||||
except (OSError, ValueError, TypeError):
|
||||
return []
|
||||
for unit in units:
|
||||
r = subprocess.run(host_command("systemctl", "--user", "start", unit), capture_output=True, text=True, timeout=60)
|
||||
log("started %s again%s" % (unit, "" if r.returncode == 0 else
|
||||
" FAILED (exit %d): %s" % (r.returncode, (r.stderr or r.stdout).strip())))
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
pass
|
||||
return units
|
||||
|
||||
|
||||
def restart_units(units):
|
||||
"""Restart those that are running, in order."""
|
||||
for unit in units:
|
||||
subprocess.run(host_command("systemctl", "--user", "try-restart", unit), capture_output=True, timeout=60)
|
||||
|
||||
|
||||
def ring_alive(path=None):
|
||||
try:
|
||||
r = Ring(path or os.path.join(run_dir(), "cam-ring"))
|
||||
@@ -551,7 +493,7 @@ class HandsFile:
|
||||
if len(data) < size:
|
||||
return None
|
||||
magic, version, _, seq, _, publish_ns, nhands, _ = HANDS_HDR.unpack_from(data, 0)
|
||||
if magic != b"FHHANDS1" or version < 1: # 2 appends the skeleton; the start is unchanged
|
||||
if magic != b"FHHANDS1" or version != 1:
|
||||
return None
|
||||
if seq % 2 or os.pread(fd, 8, 16) != data[16:24]:
|
||||
continue
|
||||
@@ -710,30 +652,6 @@ class Recorder:
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
def sets_written(self):
|
||||
"""How many sets the part has written so far (0 before the first)."""
|
||||
try:
|
||||
path = os.path.join(self.dir, "sets.bin")
|
||||
with open(path, "rb") as f:
|
||||
head = f.read(16)
|
||||
magic, _, size = struct.unpack("<8sII", head)
|
||||
return os.path.getsize(path) // size if magic.rstrip(b"\0") == b"FHSET01" and size else 0
|
||||
except (OSError, struct.error):
|
||||
return 0
|
||||
|
||||
def cameras(self):
|
||||
"""The cameras in the first set written (their calibration names), [] before there is one."""
|
||||
try:
|
||||
with open(os.path.join(self.dir, "sets.bin"), "rb") as f:
|
||||
data = f.read(16 + 8 * 40)
|
||||
magic, n, _ = struct.unpack_from("<8sII", data, 0)
|
||||
if magic.rstrip(b"\0") != b"FHSET01" or len(data) < 16 + n * 40:
|
||||
return []
|
||||
return [struct.unpack_from("<16s", data, 16 + i * 40)[0].split(b"\0", 1)[0].decode(errors="replace")
|
||||
for i in range(n)]
|
||||
except (OSError, struct.error):
|
||||
return []
|
||||
|
||||
def stop(self):
|
||||
"""End it (SIGTERM: ft-hands writes out its queue) and put the part in place. Afterwards
|
||||
names_swapped is what ft-hands says it applied (its DIR/sides.json, which goes into
|
||||
@@ -1028,47 +946,11 @@ def pose_view(poses, p):
|
||||
return e["path"], mode, e["caption"]
|
||||
|
||||
|
||||
def demo_view(demos_dir, p):
|
||||
"""A prompt's demo clip: (path, mode), mode "" (as made: a right hand, or both hands) or
|
||||
"mirror" (the prompt's "demo_mirror": a right hand's clip for the left hand). ("", "")
|
||||
when it has none, or its file isn't there."""
|
||||
demo = p.get("demo") or ""
|
||||
path = os.path.join(demos_dir, os.path.basename(demo) + ".gif") if demo else ""
|
||||
if not path or not os.path.isfile(path):
|
||||
return "", ""
|
||||
return path, "mirror" if p.get("demo_mirror") else ""
|
||||
|
||||
|
||||
def has_clips(demos_dir):
|
||||
try:
|
||||
return any(n.endswith(".gif") for n in os.listdir(demos_dir))
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def default_guidance(recorded_dir=None):
|
||||
"""Recorded clips when there are any, else the skeleton ones."""
|
||||
return "recorded" if has_clips(recorded_dir or RECORDED_DEMOS_DIR) else "skeleton"
|
||||
|
||||
|
||||
def guided_demo_view(guidance, skeleton_dir, recorded_dir, p):
|
||||
"""demo_view in the guidance's folder, else the other one (a part with no clip of that kind):
|
||||
(path, mode, key), key true for a skeleton clip (the panel and window show its colours' key)."""
|
||||
dirs = [(recorded_dir, False), (skeleton_dir, True)]
|
||||
if guidance != "recorded":
|
||||
dirs.reverse()
|
||||
for d, key in dirs:
|
||||
path, mode = demo_view(d, p)
|
||||
if path:
|
||||
return path, mode, key
|
||||
return "", "", False
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# The script
|
||||
|
||||
HAND_VALUES = ("left", "right", "both", "none", "any", "")
|
||||
PROMPT_KEYS = ("text", "seconds", "hands", "pose", "distance", "position", "object", "controller", "demo")
|
||||
PROMPT_KEYS = ("text", "seconds", "hands", "pose", "distance", "position", "object", "controller")
|
||||
|
||||
|
||||
def _when_ok(cond, ctx):
|
||||
@@ -1107,7 +989,7 @@ def _set_ids(section, prompts):
|
||||
for p in prompts:
|
||||
if not p.get("id"):
|
||||
parts = [section["id"]] + [re.sub(r"[^a-z0-9]+", "-", str(p[k]).lower()).strip("-")
|
||||
for k in ("demo", "pose", "hands", "distance", "position", "object") if p[k]]
|
||||
for k in ("pose", "hands", "distance", "position", "object") if p[k]]
|
||||
p["id"] = "/".join(parts) if len(parts) > 1 else "%s/%d" % (section["id"], len(seen) + 1)
|
||||
n = seen.get(p["id"], 0) + 1
|
||||
seen[p["id"]] = n
|
||||
@@ -1319,12 +1201,6 @@ class _Redo(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class _CamerasLost(_Fail):
|
||||
"""The recording stopped by itself or has no side cameras: usually ft-camd restarting after
|
||||
XRService moved the cameras (camd.c, SILENT_S). Step mode waits for them and does the step
|
||||
again (Session._cameras_back); anywhere else it fails the session like any _Fail."""
|
||||
|
||||
|
||||
def _os_version():
|
||||
path = "/run/host/etc/os-release" if in_container() else "/etc/os-release"
|
||||
try:
|
||||
@@ -1372,21 +1248,10 @@ def camera_check(repair=False):
|
||||
try:
|
||||
r = camcheck.check()
|
||||
pid = (r.get("ring") or {}).get("writer_pid")
|
||||
if repair and camcheck.is_ring_short(r) and pid not in _camd_restarted:
|
||||
if _unit_pid(CAMD_UNIT) == pid:
|
||||
_camd_restarted.add(pid)
|
||||
stop_unit(CAMD_UNIT)
|
||||
start_camd()
|
||||
elif _unit_pid(LIVE_CAMD_UNIT) == pid:
|
||||
# Frametop's live hand tracking: its ft-hands names the cameras when it starts,
|
||||
# so it goes again after ft-camd
|
||||
_camd_restarted.add(pid)
|
||||
restart_units([LIVE_CAMD_UNIT, LIVE_HANDS_UNIT])
|
||||
end = time.monotonic() + 15
|
||||
while not ring_alive() and time.monotonic() < end:
|
||||
time.sleep(0.2)
|
||||
else:
|
||||
return r
|
||||
if repair and camcheck.is_ring_short(r) and pid not in _camd_restarted and _unit_pid(CAMD_UNIT) == pid:
|
||||
_camd_restarted.add(pid)
|
||||
stop_unit(CAMD_UNIT)
|
||||
start_camd()
|
||||
r = camcheck.check()
|
||||
r["evidence"].append("restarted ft-camd (pid %d) because it published only some of the cameras" % pid)
|
||||
return r
|
||||
@@ -1401,10 +1266,6 @@ def camera_text(result):
|
||||
return ""
|
||||
if camcheck.is_vcint_failure(result):
|
||||
return camcheck.USER_TEXT
|
||||
if camcheck.is_ring_short(result) and "moved them" in result.get("reason", ""):
|
||||
return ("SteamVR moved the headset's tracking cameras after the camera broker started (the colour "
|
||||
"camera module was unplugged or plugged in), so the recorder can't read the side cameras. Close "
|
||||
"the Hand Recorder and open it again: it restarts the camera broker.")
|
||||
if camcheck.is_ring_short(result):
|
||||
return ("The headset's tracking cameras are all running, but the recorder can't read some of them (%s). "
|
||||
"Close the Hand Recorder, %s, and open it again. If that doesn't help, ask in the Frametop "
|
||||
@@ -1431,8 +1292,7 @@ class Session:
|
||||
def __init__(self, base_dir, profile, checklist, lighting_choice, script_path, *, ring=None,
|
||||
start_processes=True, dry_run=False, speed=1.0, on_status=None, hands_dir=None, panel_bin=None,
|
||||
auto=False, next_after=None, poses_dir=None, button=True, button_device=None,
|
||||
check_cameras=True, hands_reader=None, quick=False, seed=None, demos_dir=None,
|
||||
recorded_demos_dir=None, guidance=None):
|
||||
check_cameras=True, hands_reader=None, quick=False, seed=None):
|
||||
self.base_dir = os.path.abspath(os.path.expanduser(base_dir or BASE_DIR))
|
||||
self.profile = dict(profile or {})
|
||||
self.checklist = dict(checklist or {})
|
||||
@@ -1453,11 +1313,6 @@ class Session:
|
||||
self.seed = seed # set from the session's id when it starts, unless given
|
||||
# the camera check (a test hook: tests swap it); a dry run looks at no real cameras
|
||||
self.camera_check_fn = (lambda: None) if dry_run else camera_check
|
||||
self.repair_cameras_fn = (lambda: None) if dry_run else (lambda: camera_check(repair=True))
|
||||
self._camera_losses = 0
|
||||
# Frametop's drawn hands and hand pointer, paused for the session (a test hook: tests empty it)
|
||||
self.quiet_units = () if dry_run or ring else QUIET_UNITS
|
||||
self._rec_watch = None # (recorder, sets written, when that changed): _watch_recording
|
||||
self.input_devices = INPUT_DEVICES # where mice are looked for (a test hook)
|
||||
self._button = None # the ButtonReader
|
||||
self.print = print # where dry-run panel commands go (the CLI's stdout)
|
||||
@@ -1465,9 +1320,6 @@ class Session:
|
||||
# the plan in the script's order; the session's own shuffle comes with its id (_make_dir)
|
||||
self.plan, self.skipped = build_plan(self.script, self.checklist, seed=seed, quick=self.quick)
|
||||
self._poses = load_poses(poses_dir or POSES_DIR)
|
||||
self._demos_dir = demos_dir or DEMOS_DIR
|
||||
self._recorded_demos_dir = recorded_demos_dir or RECORDED_DEMOS_DIR
|
||||
self.guidance = guidance if guidance in dict(GUIDANCE) else default_guidance(self._recorded_demos_dir)
|
||||
self.session_dir = ""
|
||||
self._thread = None
|
||||
self._lock = threading.Lock()
|
||||
@@ -1482,7 +1334,7 @@ class Session:
|
||||
"prompt": "", "seconds_left": 0.0, "note": "", "hands": {"left": None, "right": None},
|
||||
"take": None, "error": "", "waiting": False, "countdown": 0, "big": "", "can_redo": False,
|
||||
"image": "", "image_mode": "", "caption": "", "position": "", "distance": "",
|
||||
"demo": "", "demo_mode": "", "demo_key": False, "demo_run": 0, "ready_text": READY_TEXT, "button": False, "mouse": True,
|
||||
"ready_text": READY_TEXT, "button": False, "mouse": True,
|
||||
"camera": None, "nohands": False, "strip": [], "cue": -1, "quick": self.quick}
|
||||
self._last_emit = 0.0
|
||||
self._log_file = None
|
||||
@@ -1617,7 +1469,6 @@ class Session:
|
||||
# --- the run
|
||||
def _run(self):
|
||||
try:
|
||||
self._resume_quiet() # left paused by a session that crashed
|
||||
if self._preflight():
|
||||
return
|
||||
self._make_dir()
|
||||
@@ -1642,8 +1493,6 @@ class Session:
|
||||
if self.dry_run or self.ring or not self.check_cameras:
|
||||
return False
|
||||
r = self.camera_check_fn()
|
||||
if camcheck.is_ring_short(r): # an ft-camd left on moved cameras: nothing records yet, so restart it
|
||||
r = self.repair_cameras_fn() or r
|
||||
self._status["camera"] = r
|
||||
text = camera_text(r)
|
||||
if not text:
|
||||
@@ -1694,7 +1543,6 @@ class Session:
|
||||
self._start_tracker()
|
||||
else:
|
||||
self._log("no tracking ft-hands found: feedback only if one publishes")
|
||||
self._quiet_hands()
|
||||
lighting = ring_lighting(ring_path)
|
||||
cams = []
|
||||
try:
|
||||
@@ -1712,10 +1560,9 @@ class Session:
|
||||
"checklist": self.checklist,
|
||||
"device": {"steamos": _os_version(), "steamvr": _steamvr_version(), "cameras": cams},
|
||||
"calibration_removed": removed,
|
||||
"script": {"version": self.script.get("version"), "name": self.script.get("name", ""),
|
||||
"sections": [s["id"] for s in self.plan],
|
||||
"script": {"version": self.script.get("version"), "sections": [s["id"] for s in self.plan],
|
||||
"skipped": self.skipped},
|
||||
"mode": "auto" if self.auto else "step", "quick": self.quick, "guidance": self.guidance,
|
||||
"mode": "auto" if self.auto else "step", "quick": self.quick,
|
||||
"shuffle": {"seed": self.seed, "sweeps": plan_record(self.plan)},
|
||||
"takes": [], "status": "recording", "sides": {"swapped": None}}
|
||||
self._read_sides(force=True)
|
||||
@@ -1739,7 +1586,7 @@ class Session:
|
||||
self._panel.cmd("paused off")
|
||||
for key, c in (("note", "note "), ("countdown", "countdown off"), ("hands", "hands off off"),
|
||||
("bar", "bar off"), ("target", "target off"), ("image", "image off"), ("where", "where off"),
|
||||
("demo", "demo off"), ("big", "big "), ("action", "action "), ("rec", "rec off"), ("strip", "strip off")):
|
||||
("big", "big "), ("action", "action "), ("rec", "rec off"), ("strip", "strip off")):
|
||||
self._panel.set(key, c)
|
||||
self._hints(action=False)
|
||||
|
||||
@@ -1862,16 +1709,6 @@ class Session:
|
||||
except RuntimeError as e:
|
||||
raise _Fail(str(e))
|
||||
|
||||
def _quiet_hands(self):
|
||||
"""Frametop's drawn hands and hand pointer off until the session ends: the hand overlay
|
||||
covers what the person watches, and gestures would click or open Steam's menu."""
|
||||
if self.quiet_units:
|
||||
pause_units(self.quiet_units, os.path.join(self.hands_dir, PAUSED_UNITS_FILE), self._log)
|
||||
|
||||
def _resume_quiet(self):
|
||||
if self.quiet_units:
|
||||
resume_units(os.path.join(self.hands_dir, PAUSED_UNITS_FILE), self._log)
|
||||
|
||||
def _stop_units(self):
|
||||
for unit in reversed(self._units):
|
||||
stop_unit(unit)
|
||||
@@ -1927,7 +1764,6 @@ class Session:
|
||||
if self._stop_note and state == "stopped":
|
||||
self._log(self._stop_note)
|
||||
self._stop_units()
|
||||
self._resume_quiet()
|
||||
if self._button:
|
||||
self._button.stop()
|
||||
self._button = None
|
||||
@@ -1952,7 +1788,7 @@ class Session:
|
||||
prompt=error or (self._stop_note if state == "stopped" else "") or screen.get("text", "").replace("|", "\n"),
|
||||
hands={"left": None, "right": None}, nohands=False,
|
||||
waiting=False, countdown=0, big="", can_redo=False, image="", image_mode="", caption="",
|
||||
position="", distance="", strip=[], cue=-1, demo="", demo_mode="", demo_key=False)
|
||||
position="", distance="", strip=[], cue=-1)
|
||||
|
||||
# --- the timing loop
|
||||
def _controls(self):
|
||||
@@ -2052,8 +1888,7 @@ class Session:
|
||||
if self._panel_proc and self._panel_proc.poll() is not None:
|
||||
raise _Fail("The headset panel closed (did SteamVR quit?)")
|
||||
if self._recorder and self._recorder.exited():
|
||||
raise _CamerasLost("The recording stopped by itself: see session.log")
|
||||
self._watch_recording(now)
|
||||
raise _Fail("The recording stopped by itself: see session.log")
|
||||
if not self.dry_run and now - fb.get("disk", 0) > 5:
|
||||
fb["disk"] = now
|
||||
if shutil.disk_usage(self.session_dir).free < MIN_FREE:
|
||||
@@ -2157,14 +1992,10 @@ class Session:
|
||||
self._panel.set("text", "text " + clean_text(text))
|
||||
|
||||
def _view(self, p=None):
|
||||
"""The prompt's picture or demo clip and where-to diagram, on the panel and in the status
|
||||
(sent with the next _emit); none without a prompt. A demo clip takes the picture's place."""
|
||||
"""The prompt's picture and where-to diagram, on the panel and in the status (sent with
|
||||
the next _emit); none without a prompt."""
|
||||
p = p or {}
|
||||
demo, demo_mode, demo_key = guided_demo_view(self.guidance, self._demos_dir, self._recorded_demos_dir, p)
|
||||
path, mode, caption = ("", "", "") if p.get("cues") or demo else pose_view(self._poses, p)
|
||||
self._panel.set("demo", "demo %s%s%s" % (demo, " " + demo_mode if demo_mode else "", "" if demo_key else " nokey")
|
||||
if demo else "demo off")
|
||||
self._status.update(demo=demo, demo_mode=demo_mode, demo_key=demo_key)
|
||||
path, mode, caption = ("", "", "") if p.get("cues") else pose_view(self._poses, p)
|
||||
self._strip(p)
|
||||
pos, dist = p.get("position") or "", p.get("distance") or ""
|
||||
self._panel.set("image", "image %s%s" % (path, " " + mode if mode else "") if path else "image off")
|
||||
@@ -2362,8 +2193,6 @@ class Session:
|
||||
|
||||
def _prompt_event(self, p):
|
||||
extra = {"controllers": p["controllers"]} if p["controllers"] else {}
|
||||
if p.get("demo"):
|
||||
extra["demo"] = p["demo"]
|
||||
if p.get("cue"): # a sweep's cue: the pose highlighted from now, within the step
|
||||
extra.update(cue=True, step=p["step"])
|
||||
return self._event("prompt", id=p["id"], text=p["text"], hands=p["hands"], pose=p["pose"],
|
||||
@@ -2375,11 +2204,6 @@ class Session:
|
||||
self._fb.pop("lost_key", None)
|
||||
self._show(s["title"], step, p["text"])
|
||||
self._view(p)
|
||||
if state == "running" and self._status["demo"]:
|
||||
# The hold starts: the clip from its start, so it can be followed (the window's too:
|
||||
# demo_run counts the starts).
|
||||
self._panel.cmd("demo restart")
|
||||
self._status["demo_run"] += 1
|
||||
t = self._prompt_event(p)
|
||||
if self._step_t0 is None:
|
||||
self._step_t0 = t
|
||||
@@ -2461,16 +2285,6 @@ class Session:
|
||||
self._hold_end()
|
||||
again.discard(k)
|
||||
k += 1
|
||||
except _CamerasLost as e:
|
||||
if self.auto:
|
||||
raise
|
||||
from_ns = started if started is not None else self._step_t0
|
||||
self._hold_end()
|
||||
if from_ns is not None:
|
||||
self._event("redo", id=p["id"], **{"from": from_ns, "to": mono_ns()})
|
||||
self._cameras_back(p, str(e), where)
|
||||
again.add(k)
|
||||
retry = False
|
||||
except _Redo:
|
||||
self._redo_ok = False
|
||||
from_ns = started if started is not None else self._step_t0
|
||||
@@ -2491,55 +2305,6 @@ class Session:
|
||||
self._log(" redo %s" % pid)
|
||||
again.add(k)
|
||||
|
||||
def _watch_recording(self, now):
|
||||
"""A recording that has started writing sets but writes no new one for STALL_S is lost
|
||||
(_CamerasLost): the cameras stopped or moved, and the rest of the hold would be empty."""
|
||||
rec = self._recorder
|
||||
if rec is None or self.dry_run or not hasattr(rec, "sets_written"):
|
||||
self._rec_watch = None
|
||||
return
|
||||
n = rec.sets_written()
|
||||
w = self._rec_watch
|
||||
if w is None or w[0] is not rec or n != w[1]:
|
||||
self._rec_watch = (rec, n, now)
|
||||
elif n > 0 and now - w[2] > STALL_S:
|
||||
self._rec_watch = None
|
||||
raise _CamerasLost("The recording got no new frames for %.0f s (after %d sets)" % (now - w[2], n))
|
||||
|
||||
def _cameras_back(self, p, why, where):
|
||||
"""A step's recording was lost (_CamerasLost): its try is marked redone. Wait for the
|
||||
camera check to pass, restarting a camera broker left on moved cameras (camera_check's
|
||||
repair), then the step shows again. _Fail if they don't come back, or too often."""
|
||||
self._camera_losses += 1
|
||||
self._log(" %s: %s (%d this session); waiting for the cameras" % (p["id"], why, self._camera_losses))
|
||||
if self._camera_losses > MAX_CAMERA_LOSSES:
|
||||
raise _Fail("The recording kept stopping (%d times): %s" % (self._camera_losses, why))
|
||||
self._prompt = None
|
||||
self._view()
|
||||
self._show(CAMERAS_TITLE, where, CAMERAS_TEXT)
|
||||
for key, c in (("big", "big "), ("countdown", "countdown off"), ("note", "note ")):
|
||||
self._panel.set(key, c)
|
||||
self._emit(state="between", prompt=CAMERAS_TEXT, big="", countdown=0, seconds_left=0.0, can_redo=False)
|
||||
if self.ring:
|
||||
return
|
||||
end, r = time.monotonic() + CAMERAS_BACK_S / self.speed, None
|
||||
while time.monotonic() < end:
|
||||
if self._want["stop"]:
|
||||
raise _Stop()
|
||||
r = self.repair_cameras_fn()
|
||||
if r is None: # a dry run: nothing to check
|
||||
return
|
||||
if r.get("status") == "ok":
|
||||
self._log(" cameras back: %s" % r.get("summary", "ok"))
|
||||
return
|
||||
self._wake.wait(2.0 / self.speed)
|
||||
self._wake.clear()
|
||||
summary = (r or {}).get("summary", "not run")
|
||||
for line in (r or {}).get("evidence", []):
|
||||
self._log(" " + line)
|
||||
raise _Fail("The recording stopped and the cameras didn't come back (camera check: %s). %s"
|
||||
% (summary, camera_text(r) or why))
|
||||
|
||||
def _no_hands_seen(self, p, w):
|
||||
"""The no-hands check's verdict on a hold (logged either way): True if the tracker
|
||||
published through it and never saw a hand."""
|
||||
@@ -2648,23 +2413,13 @@ class Session:
|
||||
end = time.monotonic() + FIRST_SET_S
|
||||
while not rec.has_data():
|
||||
if rec.exited():
|
||||
raise _CamerasLost("The recording stopped by itself: see session.log")
|
||||
raise _Fail("The recording stopped by itself: see session.log")
|
||||
if self._want["stop"]:
|
||||
raise _Stop()
|
||||
if time.monotonic() > end:
|
||||
self._log("recording part %d: no set yet after the countdown" % rec.part)
|
||||
return
|
||||
time.sleep(0.02)
|
||||
# Without the side pair a recording is no use for training (2026-10-12: XRService moved
|
||||
# the side cameras while ft-camd kept publishing the old devices, and ft-hands left them
|
||||
# out of a whole session). Stop at once rather than record the rest of the session.
|
||||
cams = rec.cameras()
|
||||
missing = [c for c in SIDE_CAMERAS if c not in cams]
|
||||
if cams and missing:
|
||||
self._log("recording part %d has only %s" % (rec.part, " ".join(cams)))
|
||||
raise _CamerasLost("The recording is missing the side cameras (%s), so it can't be used. Close the "
|
||||
"Hand Recorder and open it again: it checks the cameras and restarts the camera "
|
||||
"broker if it needs to." % " ".join(missing))
|
||||
|
||||
def _hold_end(self):
|
||||
"""Step mode: the step is over. A "wait" event ends its labels and the recording stops
|
||||
@@ -2834,10 +2589,6 @@ def main():
|
||||
ap.add_argument("--next-after", type=float, metavar="S",
|
||||
help="test: press Next by itself after S seconds of waiting (real time)")
|
||||
ap.add_argument("--poses", help="the pose pictures' folder, with poses.json (default hands/rec/poses)")
|
||||
ap.add_argument("--demos", help="the skeleton demo clips' folder, <id>.gif (default hands/rec/demos)")
|
||||
ap.add_argument("--recorded-demos", help="the recorded demo clips' folder (default hands/rec/demos-recorded)")
|
||||
ap.add_argument("--guidance", choices=[k for k, _ in GUIDANCE],
|
||||
help="the demo clips to follow (default recorded when there are any, else skeleton)")
|
||||
ap.add_argument("--no-headset-button", action="store_true",
|
||||
help="don't read the headset's button (gpio-keys KEY_SELECT: a press is Next, pause or resume; "
|
||||
"two are redo; a hold is stop)")
|
||||
@@ -2884,8 +2635,6 @@ def main():
|
||||
st["prompt"].replace("\n", " | "))
|
||||
if st["image"]:
|
||||
line += " [%s%s]" % (os.path.basename(st["image"]), " " + st["image_mode"] if st["image_mode"] else "")
|
||||
if st["demo"]:
|
||||
line += " [%s%s]" % (os.path.basename(st["demo"]), " " + st["demo_mode"] if st["demo_mode"] else "")
|
||||
if st["waiting"]:
|
||||
line += " (waiting for Next)"
|
||||
if st["note"]:
|
||||
@@ -2897,8 +2646,7 @@ def main():
|
||||
s = Session(base, profile, checklist, a.lighting, a.script, ring=a.ring, start_processes=not a.no_start,
|
||||
dry_run=a.dry_run, speed=a.speed, on_status=on_status, hands_dir=a.hands_dir, panel_bin=a.panel,
|
||||
auto=a.auto, next_after=a.next_after, poses_dir=a.poses, button=not a.no_headset_button,
|
||||
button_device=a.button_device, check_cameras=not a.ignore_cameras, quick=a.quick, seed=a.seed,
|
||||
demos_dir=a.demos, recorded_demos_dir=a.recorded_demos, guidance=a.guidance)
|
||||
button_device=a.button_device, check_cameras=not a.ignore_cameras, quick=a.quick, seed=a.seed)
|
||||
est = plan_seconds(s.script, s.plan, auto=a.auto)
|
||||
worst = plan_seconds(s.script, s.plan, worst=True, auto=a.auto)
|
||||
print("%d sections%s, %s mode: about %.1f min%s (at most %.1f)%s" % (
|
||||
|
||||
@@ -68,36 +68,5 @@ class QmlBackendTest(unittest.TestCase):
|
||||
self.assertEqual(sorted(read - self.props - self.slots), [], "read in main.qml but not on Backend")
|
||||
|
||||
|
||||
|
||||
class GuidanceTest(unittest.TestCase):
|
||||
"""The demo clips to follow: recorded by default when there are recorded clips, the choice
|
||||
kept in BASE/settings.json and handed to the session."""
|
||||
|
||||
def test_choice(self):
|
||||
import json
|
||||
import shutil
|
||||
import tempfile
|
||||
try:
|
||||
import ft_handrec
|
||||
import takes
|
||||
except ImportError as e:
|
||||
self.skipTest(f"no PySide6: {e}")
|
||||
tmp = tempfile.mkdtemp()
|
||||
self.addCleanup(shutil.rmtree, tmp)
|
||||
rec = os.path.join(tmp, "rec")
|
||||
os.makedirs(rec)
|
||||
backend = ft_handrec.Backend(takes.Store(os.path.join(tmp, "base")), {"recorded_demos_dir": rec})
|
||||
self.assertEqual([c["value"] for c in backend.guidanceChoices], ["recorded", "skeleton"])
|
||||
self.assertEqual(backend.guidance, "skeleton") # no recorded clips
|
||||
with open(os.path.join(rec, "2a.gif"), "wb") as f:
|
||||
f.write(b"GIF89a")
|
||||
self.assertEqual(backend.guidance, "recorded")
|
||||
backend.setGuidance("skeleton")
|
||||
backend.setGuidance("nonsense")
|
||||
with open(os.path.join(tmp, "base", "settings.json")) as f:
|
||||
self.assertEqual(json.load(f), {"guidance": "skeleton"})
|
||||
self.assertEqual(backend.guidance, "skeleton")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -102,31 +102,6 @@ class SessionTest(SessionBase):
|
||||
self.assertIn("panel: where left near", self.panel)
|
||||
self.assertIn("panel: rec on", self.panel)
|
||||
|
||||
def test_guidance(self):
|
||||
"""The chosen guidance's clip on the panel (a recorded one without the colours' key), and
|
||||
the choice in session.json."""
|
||||
script = json.loads(json.dumps(SCRIPT))
|
||||
script["sections"][0]["prompts"][0].update(demo="2a", demo_mirror=True)
|
||||
with open(self.script, "w") as f:
|
||||
json.dump(script, f)
|
||||
skel, rec = os.path.join(self.tmp, "skel"), os.path.join(self.tmp, "rec")
|
||||
for d in (skel, rec):
|
||||
os.makedirs(d)
|
||||
with open(os.path.join(d, "2a.gif"), "wb") as f:
|
||||
f.write(tiny_gif([[0, 1, 2, 3]]))
|
||||
for guidance, want in (("recorded", "panel: demo %s/2a.gif mirror nokey" % rec),
|
||||
("skeleton", "panel: demo %s/2a.gif mirror" % skel)):
|
||||
self.panel.clear()
|
||||
s = self.session(next_after=0.0, demos_dir=skel, recorded_demos_dir=rec, guidance=guidance)
|
||||
s.start()
|
||||
s.join(20)
|
||||
self.assertEqual(s.state, "done")
|
||||
self.assertIn(want, self.panel)
|
||||
with open(os.path.join(s.session_dir, "session.json")) as f:
|
||||
self.assertEqual(json.load(f)["guidance"], guidance)
|
||||
# no choice: recorded, since there are recorded clips
|
||||
self.assertEqual(self.session(demos_dir=skel, recorded_demos_dir=rec).guidance, "recorded")
|
||||
|
||||
def test_redo(self):
|
||||
s = self.session()
|
||||
s.start()
|
||||
@@ -411,148 +386,6 @@ class StripPanelTest(unittest.TestCase):
|
||||
self.assertIn("strip: off", out)
|
||||
|
||||
|
||||
def tiny_gif(frames, delay_cs=5):
|
||||
"""A GIF of 2x2 frames, each 4 palette indices (0-3: black, red, green, blue). Every pixel
|
||||
is coded after a clear code, so the LZW table never grows (a valid, uncompressed GIF)."""
|
||||
def lzw(pixels):
|
||||
bits, out = [], bytearray()
|
||||
for v in [x for p in pixels for x in (4, p)] + [5]: # clear, pixel ... end
|
||||
bits += [(v >> k) & 1 for k in range(3)]
|
||||
for i in range(0, len(bits), 8):
|
||||
out.append(sum(b << k for k, b in enumerate(bits[i:i + 8])))
|
||||
return bytes([2, len(out)]) + bytes(out) + b"\0"
|
||||
head = b"GIF89a" + struct.pack("<HHBBB", 2, 2, 0xF1, 0, 0) + bytes([0, 0, 0, 255, 0, 0, 0, 255, 0, 0, 0, 255])
|
||||
loop = b"\x21\xff\x0bNETSCAPE2.0\x03\x01\0\0\0"
|
||||
body = b"".join(b"\x21\xf9\x04\x04" + struct.pack("<H", delay_cs) + b"\0\0" + b"\x2c" + struct.pack("<HHHHB", 0, 0, 2, 2, 0)
|
||||
+ lzw(f) for f in frames)
|
||||
return head + loop + body + b"\x3b"
|
||||
|
||||
|
||||
class ScriptV2Test(unittest.TestCase):
|
||||
"""script-v2.json: a step per demo clip, one-hand parts left (the clip flipped), right, both."""
|
||||
|
||||
def setUp(self):
|
||||
self.script = session.load_script(os.path.join(session.HERE, "script-v2.json"))
|
||||
with open(os.path.join(session.HERE, "demo", "parts.json")) as f:
|
||||
self.parts = [p["id"] for p in json.load(f)["parts"]]
|
||||
|
||||
def test_plan(self):
|
||||
full, skipped = session.build_plan(self.script, {"controllers": "straps", "objects": ["keyboard"]})
|
||||
self.assertEqual(skipped, [])
|
||||
self.assertEqual(full[0]["id"], session.HANDS_CHECK_SECTION) # the no-hands check runs first
|
||||
demos = [p for s in full for p in s["prompts"] if p.get("demo")]
|
||||
self.assertEqual(sorted(set(p["demo"] for p in demos)), sorted(self.parts)) # every part, and no others
|
||||
ids = [p["id"] for s in full for p in s["prompts"]]
|
||||
self.assertEqual(len(ids), len(set(ids)))
|
||||
for p in demos:
|
||||
self.assertEqual(bool(p.get("demo_mirror")), p["hands"] == "left", p["id"])
|
||||
self.assertGreater(p["seconds"], 0)
|
||||
self.assertEqual([p["hands"] for p in demos if p["demo"] == "2a"], ["left", "right", "both"])
|
||||
self.assertLess(session.plan_seconds(self.script, full, auto=False), 15 * 60)
|
||||
quick, _ = session.build_plan(self.script, {}, quick=True)
|
||||
self.assertEqual([p["id"] for s in quick for p in s["prompts"]],
|
||||
["hand-size/1/both/mid", "index-taps/2a/right", "palm-up/7a/both", "no-hands/none"])
|
||||
|
||||
def test_clips(self):
|
||||
if not os.path.isdir(session.DEMOS_DIR):
|
||||
self.skipTest("no demo clips in %s" % session.DEMOS_DIR)
|
||||
for pid in self.parts:
|
||||
self.assertTrue(os.path.isfile(os.path.join(session.DEMOS_DIR, pid + ".gif")), pid)
|
||||
|
||||
def test_view(self):
|
||||
tmp = tempfile.mkdtemp()
|
||||
self.addCleanup(shutil.rmtree, tmp)
|
||||
with open(os.path.join(tmp, "2a.gif"), "wb") as f:
|
||||
f.write(tiny_gif([[0, 1, 2, 3]]))
|
||||
self.assertEqual(session.demo_view(tmp, {"demo": "2a", "demo_mirror": True}), (os.path.join(tmp, "2a.gif"), "mirror"))
|
||||
self.assertEqual(session.demo_view(tmp, {"demo": "2a"}), (os.path.join(tmp, "2a.gif"), ""))
|
||||
self.assertEqual(session.demo_view(tmp, {"demo": "3a"}), ("", ""))
|
||||
self.assertEqual(session.demo_view(tmp, {}), ("", ""))
|
||||
|
||||
def test_guidance(self):
|
||||
"""Recorded or skeleton clips, each falling back to the other kind for a part it lacks; only
|
||||
a skeleton clip gets the colours' key."""
|
||||
tmp = tempfile.mkdtemp()
|
||||
self.addCleanup(shutil.rmtree, tmp)
|
||||
skel, rec = os.path.join(tmp, "skel"), os.path.join(tmp, "rec")
|
||||
os.makedirs(skel)
|
||||
os.makedirs(rec)
|
||||
self.assertEqual(session.default_guidance(rec), "skeleton") # no recorded clips yet
|
||||
self.assertEqual(session.default_guidance(os.path.join(tmp, "none")), "skeleton")
|
||||
for d, ids in ((skel, ("2a", "3a")), (rec, ("2a",))):
|
||||
for i in ids:
|
||||
with open(os.path.join(d, i + ".gif"), "wb") as f:
|
||||
f.write(tiny_gif([[0, 1, 2, 3]]))
|
||||
self.assertEqual(session.default_guidance(rec), "recorded")
|
||||
left = {"demo": "2a", "demo_mirror": True}
|
||||
self.assertEqual(session.guided_demo_view("recorded", skel, rec, left), (os.path.join(rec, "2a.gif"), "mirror", False))
|
||||
self.assertEqual(session.guided_demo_view("skeleton", skel, rec, left), (os.path.join(skel, "2a.gif"), "mirror", True))
|
||||
self.assertEqual(session.guided_demo_view("recorded", skel, rec, {"demo": "3a"}), (os.path.join(skel, "3a.gif"), "", True))
|
||||
self.assertEqual(session.guided_demo_view("recorded", skel, rec, {"demo": "4a"}), ("", "", False))
|
||||
self.assertEqual(session.guided_demo_view("skeleton", skel, rec, {}), ("", "", False))
|
||||
|
||||
def test_recorded_clips(self):
|
||||
if not session.has_clips(session.RECORDED_DEMOS_DIR):
|
||||
self.skipTest("no recorded demo clips in %s" % session.RECORDED_DEMOS_DIR)
|
||||
for pid in self.parts:
|
||||
self.assertTrue(os.path.isfile(os.path.join(session.RECORDED_DEMOS_DIR, pid + ".gif")), pid)
|
||||
|
||||
|
||||
class DemoPanelTest(unittest.TestCase):
|
||||
"""The panel's demo command on ft-handpanel --no-vr (skipped if it isn't built)."""
|
||||
|
||||
def test_demo(self):
|
||||
import subprocess
|
||||
binary = session.PANEL_BIN
|
||||
if not os.access(binary, os.X_OK):
|
||||
self.skipTest("ft-handpanel isn't built")
|
||||
tmp = tempfile.mkdtemp()
|
||||
self.addCleanup(shutil.rmtree, tmp)
|
||||
clip = os.path.join(tmp, "clip one.gif") # a space: the path may hold them
|
||||
with open(clip, "wb") as f:
|
||||
f.write(tiny_gif([[0, 1, 2, 3], [3, 2, 1, 0], [1, 1, 1, 1]], delay_cs=5))
|
||||
with open(os.path.join(tmp, "bad.gif"), "wb") as f:
|
||||
f.write(b"GIF89a broken")
|
||||
name = "hrtest-demo-%d" % os.getpid()
|
||||
proc = subprocess.Popen([binary, "--no-vr", "--socket", name, "--dump", tmp], stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE, text=True)
|
||||
try:
|
||||
panel = session.Panel(name=name)
|
||||
for _ in range(50):
|
||||
if panel.cmd("ping", reply=True, timeout=0.1):
|
||||
break
|
||||
if proc.poll() is not None:
|
||||
self.skipTest("ft-handpanel doesn't run here: " + proc.stderr.read().strip()[-200:])
|
||||
time.sleep(0.1)
|
||||
else:
|
||||
self.fail("ft-handpanel --no-vr doesn't answer")
|
||||
replies = [panel.cmd("demo %s mirror" % clip, reply=True), panel.cmd("show", reply=True)]
|
||||
time.sleep(0.4) # 3 frames of 50 ms: it loops
|
||||
replies.append(panel.cmd("note looped", reply=True))
|
||||
replies.append(panel.cmd("demo restart", reply=True))
|
||||
replies += [panel.cmd("demo", reply=True), panel.cmd("demo %s/missing.gif" % tmp, reply=True),
|
||||
panel.cmd("demo %s/bad.gif" % tmp, reply=True), panel.cmd("demo off", reply=True)]
|
||||
# a recorded clip: no key; the same clip again with the key back
|
||||
replies += [panel.cmd("demo %s mirror nokey" % clip, reply=True), panel.cmd("show", reply=True),
|
||||
panel.cmd("demo %s nokey" % clip, reply=True), panel.cmd("demo %s" % clip, reply=True),
|
||||
panel.cmd("demo off", reply=True)]
|
||||
panel.close()
|
||||
finally:
|
||||
proc.terminate()
|
||||
out = proc.communicate(timeout=5)[0]
|
||||
self.assertEqual(replies[:4], ["ok"] * 4)
|
||||
self.assertTrue(replies[4].startswith("error usage: demo"))
|
||||
self.assertTrue(replies[5].startswith("error can't read"))
|
||||
self.assertTrue(replies[6].startswith("error can't read"))
|
||||
self.assertEqual(replies[7:], ["ok"] * 6)
|
||||
demos = [l for l in out.splitlines() if l.startswith("demo:")]
|
||||
self.assertTrue(any(l.startswith("demo: %s 2x2 frame" % clip) and " mirror nokey, drawn" in l for l in demos), demos)
|
||||
self.assertIn("demo: %s 2x2 frame 0 loop 0 mirror, drawn" % clip, demos[0])
|
||||
self.assertGreater(int(demos[1].split(" loop ")[1].split()[0]), 0) # past the last frame, from the start again
|
||||
self.assertIn("demo: %s 2x2 frame 0 loop 0 mirror" % clip, demos[2]) # restart
|
||||
self.assertEqual(demos[-1], "demo: off")
|
||||
|
||||
|
||||
# /proc/bus/input/devices as the Frame has it (2026-10-02), trimmed, plus a USB mouse.
|
||||
DEVICES = """I: Bus=0019 Vendor=0001 Product=0001 Version=0100
|
||||
N: Name="gpio-keys"
|
||||
@@ -934,179 +767,5 @@ class StandaloneTest(unittest.TestCase):
|
||||
self.assertNotIn("~/frametop", text)
|
||||
|
||||
|
||||
class SideCameraGuardTest(unittest.TestCase):
|
||||
"""A part whose first set has no side cameras stops the session (2026-10-12: a whole
|
||||
session recorded only the upper pair after XRService moved the side cameras)."""
|
||||
|
||||
def write_set(self, d, names):
|
||||
data = bytearray(struct.pack("<8sII", b"FHSET01", len(names), 16 + 40 * len(names)))
|
||||
for n in names:
|
||||
data += struct.pack("<16sIIQQ", n.encode(), 2, 2, 1, 1)
|
||||
with open(os.path.join(d, "sets.bin"), "wb") as f:
|
||||
f.write(bytes(data))
|
||||
|
||||
def first_set(self, d):
|
||||
rec = mock.Mock(dir=d, part=1, exited=lambda: False)
|
||||
rec.has_data = lambda: True
|
||||
rec.cameras = lambda: session.Recorder.cameras(rec)
|
||||
fake = mock.Mock(_recorder=rec, _want={"stop": False})
|
||||
session.Session._first_set(fake)
|
||||
return fake
|
||||
|
||||
def test_upper_pair_only(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
self.write_set(d, ["upper_left", "upper_right"])
|
||||
with self.assertRaises(session._Fail) as cm:
|
||||
self.first_set(d)
|
||||
self.assertIn("missing the side cameras (slam_left slam_right)", str(cm.exception))
|
||||
|
||||
def test_all_four(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
self.write_set(d, ["slam_left", "slam_right", "upper_left", "upper_right"])
|
||||
self.first_set(d)
|
||||
|
||||
def test_unreadable_first_set_goes_on(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
with open(os.path.join(d, "sets.bin"), "wb") as f:
|
||||
f.write(b"xx")
|
||||
self.first_set(d)
|
||||
|
||||
def test_moved_cameras_text(self):
|
||||
text = session.camera_text({"status": "degraded", "ring_missing": [0, 3],
|
||||
"reason": "ft-camd publishes video9 video13, but XRService runs those cameras "
|
||||
"on video0 video3 now (it moved them after ft-camd started)"})
|
||||
self.assertIn("open it again: it restarts the camera broker", text)
|
||||
|
||||
|
||||
class CamerasLostTest(SessionBase):
|
||||
"""A step whose recording is lost (ft-camd restarting after XRService moved the cameras)
|
||||
waits for the camera check, then goes again; too many losses stop the session."""
|
||||
|
||||
def lose(self, s, times):
|
||||
real = session.Session._run_step
|
||||
lost = {"n": 0}
|
||||
|
||||
def run_step(self_, *a):
|
||||
if lost["n"] < times:
|
||||
lost["n"] += 1
|
||||
raise session._CamerasLost("The recording stopped by itself: see session.log")
|
||||
return real(self_, *a)
|
||||
return mock.patch.object(session.Session, "_run_step", run_step)
|
||||
|
||||
def test_waits_and_redoes_the_step(self):
|
||||
s = self.session(next_after=0.0)
|
||||
checks = iter([{"status": "degraded", "summary": "degraded: moved"}, {"status": "ok", "summary": "ok"}])
|
||||
s.repair_cameras_fn = lambda: next(checks)
|
||||
with self.lose(s, 1):
|
||||
s.start()
|
||||
s.join(20)
|
||||
self.assertEqual(s.state, "done")
|
||||
ev = self.events(s)
|
||||
self.assertEqual([e["event"] for e in ev],
|
||||
["take", "ready", "wait", "redo", "ready", "prompt", "wait", "ready", "prompt", "wait",
|
||||
"end"])
|
||||
self.assertEqual(ev[3]["id"], ev[5]["id"])
|
||||
with open(os.path.join(s.session_dir, "session.log")) as f:
|
||||
log = f.read()
|
||||
self.assertIn("waiting for the cameras", log)
|
||||
self.assertIn("cameras back: ok", log)
|
||||
|
||||
def test_cameras_dont_come_back(self):
|
||||
s = self.session(next_after=0.0, speed=100)
|
||||
s.repair_cameras_fn = lambda: {"status": "degraded", "summary": "degraded: moved", "reason": "moved",
|
||||
"evidence": []}
|
||||
with self.lose(s, 1):
|
||||
s.start()
|
||||
s.join(20)
|
||||
self.assertEqual(s.state, "error")
|
||||
self.assertIn("the cameras didn't come back", s._status["error"])
|
||||
|
||||
def test_too_many_losses(self):
|
||||
s = self.session(next_after=0.0)
|
||||
s.repair_cameras_fn = lambda: None
|
||||
with self.lose(s, session.MAX_CAMERA_LOSSES + 1):
|
||||
s.start()
|
||||
s.join(20)
|
||||
self.assertEqual(s.state, "error")
|
||||
self.assertIn("kept stopping", s._status["error"])
|
||||
|
||||
def test_stalled_recording(self):
|
||||
rec = mock.Mock(sets_written=mock.Mock(return_value=12))
|
||||
fake = mock.Mock(_recorder=rec, dry_run=False, _rec_watch=None)
|
||||
session.Session._watch_recording(fake, 100.0)
|
||||
session.Session._watch_recording(fake, 100.0 + session.STALL_S - 0.5) # not yet
|
||||
with self.assertRaises(session._CamerasLost) as cm:
|
||||
session.Session._watch_recording(fake, 100.0 + session.STALL_S + 0.5)
|
||||
self.assertIn("no new frames", str(cm.exception))
|
||||
rec.sets_written.return_value = 0 # not started writing yet: the first-set wait covers that
|
||||
fake._rec_watch = None
|
||||
session.Session._watch_recording(fake, 200.0)
|
||||
session.Session._watch_recording(fake, 200.0 + 2 * session.STALL_S)
|
||||
rec.sets_written.side_effect = [20, 21] # writing: fine however long
|
||||
session.Session._watch_recording(fake, 300.0)
|
||||
session.Session._watch_recording(fake, 300.0 + 2 * session.STALL_S)
|
||||
|
||||
def test_sets_written(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
rec = mock.Mock(dir=d)
|
||||
self.assertEqual(session.Recorder.sets_written(rec), 0)
|
||||
one = struct.pack("<8sII", b"FHSET01", 0, 16)
|
||||
with open(os.path.join(d, "sets.bin"), "wb") as f:
|
||||
f.write(one * 5)
|
||||
self.assertEqual(session.Recorder.sets_written(rec), 5)
|
||||
|
||||
|
||||
class QuietHandsTest(SessionBase):
|
||||
"""Frametop's drawn hands and hand pointer are paused for a session and started again after,
|
||||
also when an earlier session was killed and left them paused."""
|
||||
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
self.path = os.path.join(self.tmp, "run", session.PAUSED_UNITS_FILE)
|
||||
self.calls = []
|
||||
|
||||
def systemctl(self):
|
||||
calls = self.calls
|
||||
|
||||
def run(cmd, **kw):
|
||||
if "systemctl" in cmd:
|
||||
calls.append(cmd[-2:])
|
||||
return mock.Mock(returncode=0, stdout="", stderr="")
|
||||
return mock.patch.object(session.subprocess, "run", run)
|
||||
|
||||
def test_pauses_only_running_units_and_resumes_them(self):
|
||||
active = {"frametop-handview.service": True, "frametop-pointer.service": False}
|
||||
with mock.patch.object(session, "unit_active", lambda u: active[u]), \
|
||||
mock.patch.object(session, "stop_unit", lambda u: self.calls.append(["stop", u])):
|
||||
paused = session.pause_units(session.QUIET_UNITS, self.path)
|
||||
self.assertEqual(paused, ["frametop-handview.service"])
|
||||
self.assertEqual(self.calls, [["stop", "frametop-handview.service"]])
|
||||
with open(self.path) as f:
|
||||
self.assertEqual(json.load(f), paused)
|
||||
with self.systemctl():
|
||||
self.assertEqual(session.resume_units(self.path), paused)
|
||||
self.assertEqual(self.calls[1:], [["start", "frametop-handview.service"]])
|
||||
self.assertFalse(os.path.exists(self.path))
|
||||
|
||||
def test_nothing_running_nothing_paused(self):
|
||||
with mock.patch.object(session, "unit_active", lambda u: False):
|
||||
self.assertEqual(session.pause_units(session.QUIET_UNITS, self.path), [])
|
||||
self.assertFalse(os.path.exists(self.path))
|
||||
self.assertEqual(session.resume_units(self.path), [])
|
||||
|
||||
def test_a_session_starts_what_a_killed_one_left_paused(self):
|
||||
os.makedirs(os.path.dirname(self.path))
|
||||
with open(self.path, "w") as f:
|
||||
json.dump(["frametop-pointer.service", "rm -rf"], f)
|
||||
s = self.session(next_after=0.0, hands_dir=os.path.dirname(self.path))
|
||||
s.quiet_units = session.QUIET_UNITS
|
||||
with self.systemctl():
|
||||
s.start()
|
||||
s.join(20)
|
||||
self.assertEqual(s.state, "done")
|
||||
self.assertEqual(self.calls, [["start", "frametop-pointer.service"]]) # not the junk entry
|
||||
self.assertFalse(os.path.exists(self.path))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+9
-17
@@ -1,9 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install, start, stop, or inspect hand tracking on the Frame: ft-camd (the camera broker) and
|
||||
# ft-hands (the tracker), user services that stop with SteamVR. HANDS_MODE in frametop.conf says
|
||||
# when they start (hands/handsmode.py: with the Frametop desktop, with SteamVR, or off, the
|
||||
# default); ft-handsctl on|off (on the Frame) or hands/run.sh start|stop do it by hand.
|
||||
# Usage: hands/run.sh install|uninstall # install applies HANDS_MODE
|
||||
# ft-hands (the tracker), user services that stop with SteamVR. They don't start with it:
|
||||
# ft-handsctl on|off (on the Frame) or hands/run.sh start|stop.
|
||||
# Usage: hands/run.sh install|uninstall
|
||||
# hands/run.sh caps # give ft-camd its capabilities again (a rebuild clears them)
|
||||
# hands/run.sh uncaps # take them back, unless the Hand Recorder or the services use them
|
||||
# hands/run.sh start|stop|restart|status|log [lines]
|
||||
@@ -14,10 +13,7 @@ set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
||||
. "$root/scripts/_env.sh"
|
||||
frame="$root/scripts/frame.sh"
|
||||
units="frametop-camd.service frametop-hands.service frametop-handview.service"
|
||||
# ft-handview (the hands drawn with Room View off) is screens/' build (screens/build.sh, which
|
||||
# the desktop's install runs); its unit sits there too.
|
||||
unit_dir() { [ "$1" = frametop-handview.service ] && echo screens || echo hands; }
|
||||
units="frametop-camd.service frametop-hands.service"
|
||||
# pidfd_getfd on XRService (ptrace_scope=1), system-wide tracepoints, and their root-only
|
||||
# format files. ft-camd drops them all once it has set up.
|
||||
caps=cap_sys_ptrace,cap_perfmon,cap_dac_read_search+ep
|
||||
@@ -61,17 +57,13 @@ case ${1:-status} in
|
||||
"$root/hands/build.sh"
|
||||
set_caps
|
||||
for u in $units; do
|
||||
fill_template "$root/$(unit_dir $u)/$u" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$u"
|
||||
fill_template "$root/hands/$u" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$u"
|
||||
done
|
||||
# Started as HANDS_MODE says (off: not at all; Frametop Input Settings, Hands), or by hand
|
||||
# with ft-handsctl on|off (linked into ~/.local/bin).
|
||||
"$frame" --host "set -e; systemctl --user daemon-reload
|
||||
python3 $(printf %q "$FRAME_REPO/hands/handsmode.py") apply
|
||||
# Installed but not started with SteamVR: ft-handsctl on|off (linked into ~/.local/bin).
|
||||
"$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user disable $units 2>/dev/null || true
|
||||
mkdir -p ~/.local/bin && ln -sfn $(printf %q "$FRAME_REPO/hands/ft-handsctl") ~/.local/bin/ft-handsctl
|
||||
$states; python3 $(printf %q "$FRAME_REPO/hands/handsmode.py") status" ;;
|
||||
caps) # then what HANDS_MODE wants starts, if it couldn't without them
|
||||
set_caps
|
||||
on_frame "python3 $(printf %q "$FRAME_REPO/hands/handsmode.py") apply" ;;
|
||||
$states; echo 'start it with: ft-handsctl on'" ;;
|
||||
caps) set_caps ;;
|
||||
uncaps) drop_caps ;;
|
||||
uninstall)
|
||||
"$frame" --host "systemctl --user disable --now $units 2>/dev/null
|
||||
|
||||
@@ -1,294 +0,0 @@
|
||||
// Unit tests for the flag buffer (track/flagbuf.h): the crops' geometry, the rolling window, the
|
||||
// slots running out, a flag's window and files, ghosts, a second flag while one waits, and the
|
||||
// clock completing a flag when no sets come. Made-up images and views; no models.
|
||||
// Build and run: make check (in the dev container; hands/build.sh's environment).
|
||||
#include "../track/flagbuf.h"
|
||||
|
||||
#include <json/json.h>
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <fstream>
|
||||
#include <sstream>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
static int failures = 0;
|
||||
#define CHECK(cond, ...) \
|
||||
do { \
|
||||
if (!(cond)) { \
|
||||
std::printf("FAIL %s:%d: ", __FILE__, __LINE__); \
|
||||
std::printf(__VA_ARGS__); \
|
||||
std::printf("\n"); \
|
||||
++failures; \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int64_t kSec = 1'000'000'000;
|
||||
|
||||
struct Img {
|
||||
int w, h;
|
||||
std::vector<uint8_t> px;
|
||||
Img(int w_, int h_) : w(w_), h(h_), px(size_t(w_) * h_) {
|
||||
for (int y = 0; y < h; ++y)
|
||||
for (int x = 0; x < w; ++x) px[size_t(y) * w + x] = uint8_t((x + 2 * y) & 255);
|
||||
}
|
||||
Image image() const { return {px.data(), w, h, w}; }
|
||||
int at(int x, int y) const { return px[size_t(y) * w + x]; }
|
||||
};
|
||||
|
||||
Seen view(const std::string &cam, int hand, double cx, double cy, double size) {
|
||||
Seen s;
|
||||
s.cam = cam, s.hand = hand;
|
||||
s.roi.center = {cx, cy}, s.roi.size = size, s.roi.rotation = 0.3;
|
||||
s.lm.presence = 0.9, s.lm.right = 0.8;
|
||||
for (int k = 0; k < 21; ++k) s.lm.pts[k] = {cx + k, cy - k};
|
||||
return s;
|
||||
}
|
||||
|
||||
bool parse(const std::string &text, Json::Value &v) {
|
||||
Json::CharReaderBuilder b;
|
||||
std::string e;
|
||||
std::istringstream in(text);
|
||||
return Json::parseFromStream(b, in, &v, &e);
|
||||
}
|
||||
|
||||
std::vector<Json::Value> lines(const std::string &path) {
|
||||
std::vector<Json::Value> out;
|
||||
std::ifstream f(path);
|
||||
std::string line;
|
||||
while (std::getline(f, line)) {
|
||||
Json::Value v;
|
||||
CHECK(parse(line, v), "%s: a line isn't JSON: %.80s", path.c_str(), line.c_str());
|
||||
out.push_back(v);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
Json::Value file_json(const std::string &path) {
|
||||
std::ifstream f(path);
|
||||
std::stringstream ss;
|
||||
ss << f.rdbuf();
|
||||
Json::Value v;
|
||||
CHECK(parse(ss.str(), v), "%s isn't JSON", path.c_str());
|
||||
return v;
|
||||
}
|
||||
|
||||
std::string temp_dir() {
|
||||
char tmpl[] = "/tmp/flagbuf-test-XXXXXX";
|
||||
const char *d = mkdtemp(tmpl);
|
||||
return d ? d : "/tmp";
|
||||
}
|
||||
|
||||
void test_crop_geometry() {
|
||||
const Img img(320, 240);
|
||||
std::vector<uint8_t> out(224 * 224);
|
||||
// native size: crop pixel (u, v) is image pixel (x0 + u, y0 + v) exactly
|
||||
CropGeom g = crop_square(img.image(), {100.5, 80.5}, 60, 224, out.data());
|
||||
CHECK(g.w == 60 && g.h == 60 && std::fabs(g.scale - 1) < 1e-9, "native crop %dx%d scale %f", g.w, g.h, g.scale);
|
||||
bool same = true;
|
||||
for (int v = 0; v < g.h; ++v)
|
||||
for (int u = 0; u < g.w; ++u) {
|
||||
const int x = int(std::lround(g.x0 + (u + 0.5) * g.scale - 0.5)), y = int(std::lround(g.y0 + (v + 0.5) * g.scale - 0.5));
|
||||
same = same && out[size_t(v) * g.w + u] == img.at(x, y);
|
||||
}
|
||||
CHECK(same, "native crop's pixels aren't the image's at the mapped place");
|
||||
// the crop's middle maps to the centre
|
||||
CHECK(std::fabs(g.x0 + (g.w / 2.0) * g.scale - 0.5 - 100.5) < 1e-9, "centre x %f", g.x0 + (g.w / 2.0) * g.scale - 0.5);
|
||||
// downscaled: at most max_px, the mapping still holds (the pattern is smooth: within a level or two)
|
||||
g = crop_square(img.image(), {160, 120}, 200, 64, out.data());
|
||||
CHECK(g.w == 64 && std::fabs(g.scale - 200.0 / 64) < 1e-9, "downscaled %d scale %f", g.w, g.scale);
|
||||
int worst = 0;
|
||||
for (int v = 0; v < g.h; ++v)
|
||||
for (int u = 0; u < g.w; ++u) {
|
||||
const double x = g.x0 + (u + 0.5) * g.scale - 0.5, y = g.y0 + (v + 0.5) * g.scale - 0.5;
|
||||
const int want = int(std::lround(x + 2 * y)) & 255, got = out[size_t(v) * g.w + u];
|
||||
if (want > 8 && want < 248) worst = std::max(worst, std::abs(want - got));
|
||||
}
|
||||
CHECK(worst <= 3, "downscaled crop off the mapping by %d levels", worst);
|
||||
// past the image's corner: 0 there, the image inside
|
||||
g = crop_square(img.image(), {2, 2}, 40, 224, out.data());
|
||||
CHECK(out[0] == 0 && out[size_t(g.h - 1) * g.w + g.w - 1] == img.at(22, 22), "corner crop: %d, %d (want 0, %d)", out[0],
|
||||
out[size_t(g.h - 1) * g.w + g.w - 1], img.at(22, 22));
|
||||
CHECK(g.x0 < 0 && g.y0 < 0, "corner crop's x0 %f y0 %f", g.x0, g.y0);
|
||||
}
|
||||
|
||||
FlagConfig small_config() {
|
||||
FlagConfig c;
|
||||
c.before_s = 1.0, c.after_s = 0.5, c.ghost_s = 0.25, c.crop_px = 32, c.scale = 1.6;
|
||||
c.max_bytes = 32 * 32 * 20; // 20 slots
|
||||
c.writer_cpus.clear();
|
||||
return c;
|
||||
}
|
||||
|
||||
void test_window_and_slots() {
|
||||
const Img img(128, 128);
|
||||
const std::map<std::string, Image> images = {{"a", img.image()}, {"b", img.image()}};
|
||||
FlagBuffer fb(small_config());
|
||||
// 10 sets a second, two views each (20 slots: 10 sets fit)
|
||||
for (int i = 0; i < 30; ++i)
|
||||
fb.add(i * kSec / 10, i * kSec / 10, i, images, {view("a", 1, 60, 60, 20), view("b", 1, 64, 50, 20)}, "\"hands\": []");
|
||||
FlagBuffer::Stats s = fb.stats();
|
||||
CHECK(s.slots == 20 && s.slots_used <= 20, "slots %zu used %zu", s.slots, s.slots_used);
|
||||
CHECK(s.dropped == 0, "nothing held, so nothing dropped: %zu", s.dropped);
|
||||
CHECK(s.crops == 60, "crops %zu", s.crops);
|
||||
// one view a set: the window (1 s), not the slots, keeps it to about 11 sets
|
||||
for (int i = 30; i < 60; ++i)
|
||||
fb.add(i * kSec / 10, i * kSec / 10, i, images, {view("a", 1, 60, 60, 20)}, "\"hands\": []");
|
||||
s = fb.stats();
|
||||
CHECK(s.slots_used >= 10 && s.slots_used <= 12, "one view a set: %zu slots in use", s.slots_used);
|
||||
}
|
||||
|
||||
void test_flag_files() {
|
||||
const Img img(128, 128);
|
||||
const std::map<std::string, Image> images = {{"a", img.image()}, {"b", img.image()}};
|
||||
FlagBuffer fb(small_config());
|
||||
fb.set_static("{\"a\": {\"width\": 128}}", "{\"models\": \"m\"}");
|
||||
const std::string root = temp_dir(), dir = root + "/flag-1";
|
||||
int i = 0;
|
||||
for (; i < 30; ++i) fb.add(i * kSec / 10, i * kSec / 10, i, images, {view("a", 1, 60, 60, 20)}, "\"hands\": []");
|
||||
CHECK(fb.flag({dir, "\"source\": \"test\", \"note\": \"fingers \\\"crossed\\\"\""}, 29 * kSec / 10), "flag refused");
|
||||
CHECK(!fb.flag({dir + "-2", ""}, 30 * kSec / 10, "again"), "a second flag while one waits was taken");
|
||||
CHECK(fb.waiting(), "not waiting");
|
||||
// two views a set now: the held sets fill the slots, so some crops drop
|
||||
for (; i < 34; ++i)
|
||||
fb.add(i * kSec / 10, i * kSec / 10, i, images, {view("a", 1, 60, 60, 20), view("b", 2, 30, 30, 20)}, "\"hands\": []");
|
||||
CHECK(fb.waiting(), "completed early (0.5 s after 2.9 s is 3.4 s; the last set is 3.3 s)");
|
||||
fb.add(i * kSec / 10, i * kSec / 10, i, images, {}, "\"hands\": []");
|
||||
++i;
|
||||
CHECK(!fb.waiting(), "not completed");
|
||||
fb.wait();
|
||||
const FlagBuffer::Stats s = fb.stats();
|
||||
CHECK(s.flags == 1 && s.last_dir == dir && s.last_error.empty(), "flags %zu dir %s error %s", s.flags, s.last_dir.c_str(),
|
||||
s.last_error.c_str());
|
||||
CHECK(s.slots_used == 0, "the written sets' slots weren't given back: %zu in use", s.slots_used);
|
||||
const Json::Value m = file_json(dir + "/manifest.json");
|
||||
CHECK(m["version"].asInt() == 1 && m["source"].asString() == "test", "manifest version/source");
|
||||
CHECK(m["note"].asString() == "fingers \"crossed\"", "note %s", m["note"].asString().c_str());
|
||||
CHECK(m["more_notes"].size() == 1 && m["more_notes"][0].asString() == "again", "more_notes");
|
||||
CHECK(m["flag_t_raw_ns"].asInt64() == 29 * kSec / 10, "flag time");
|
||||
CHECK(m["settings"]["models"].asString() == "m" && m["cameras"]["a"]["width"].asInt() == 128, "static parts");
|
||||
const std::vector<Json::Value> sets = lines(dir + "/sets.jsonl"), crops = lines(dir + "/crops.jsonl");
|
||||
// held: from 1.9 s (1 s before) to the set at 3.4 s
|
||||
CHECK(!sets.empty() && sets.front()["t_raw_ns"].asInt64() >= 19 * kSec / 10 - 1 &&
|
||||
sets.back()["t_raw_ns"].asInt64() == 34 * kSec / 10,
|
||||
"window %lld..%lld", sets.empty() ? 0LL : (long long)sets.front()["t_raw_ns"].asInt64(),
|
||||
sets.empty() ? 0LL : (long long)sets.back()["t_raw_ns"].asInt64());
|
||||
CHECK(m["counts"]["sets"].asUInt() == sets.size() && m["counts"]["crops"].asUInt() == crops.size(), "counts");
|
||||
CHECK(m["counts"]["crops_dropped"].asUInt() > 0, "held sets filling the slots should drop crops");
|
||||
// every view's crop index points at a crop line of that set and camera; the bytes add up
|
||||
size_t bytes = 0;
|
||||
for (const Json::Value &c : crops) {
|
||||
CHECK(c["offset"].asUInt64() == bytes, "crop %u offset", c["k"].asUInt());
|
||||
bytes += c["bytes"].asUInt64();
|
||||
CHECK(c["bytes"].asUInt64() == c["w"].asUInt64() * c["h"].asUInt64(), "crop bytes");
|
||||
}
|
||||
struct stat st;
|
||||
CHECK(stat((dir + "/crops.bin").c_str(), &st) == 0 && size_t(st.st_size) == bytes, "crops.bin size");
|
||||
for (const Json::Value &set : sets)
|
||||
for (const Json::Value &v : set["views"]) {
|
||||
const int k = v["crop"].asInt();
|
||||
if (k < 0) continue;
|
||||
CHECK(k < int(crops.size()) && crops[size_t(k)]["set"].asInt() == set["i"].asInt() &&
|
||||
crops[size_t(k)]["cam"].asString() == v["cam"].asString(),
|
||||
"view crop %d", k);
|
||||
CHECK(v["lm"].size() == 21 && v["roi"].size() == 4, "view landmarks");
|
||||
}
|
||||
// the first crop's pixels are the image's where the mapping says
|
||||
if (!crops.empty()) {
|
||||
std::ifstream f(dir + "/crops.bin", std::ios::binary);
|
||||
std::vector<uint8_t> px(crops[0]["bytes"].asUInt64());
|
||||
f.read(reinterpret_cast<char *>(px.data()), std::streamsize(px.size()));
|
||||
const double x0 = crops[0]["x0"].asDouble(), y0 = crops[0]["y0"].asDouble(), sc = crops[0]["scale"].asDouble();
|
||||
const int w = crops[0]["w"].asInt(), u = w / 2, v = w / 3;
|
||||
const int x = int(std::lround(x0 + (u + 0.5) * sc - 0.5)), y = int(std::lround(y0 + (v + 0.5) * sc - 0.5));
|
||||
CHECK(std::abs(int(px[size_t(v) * w + u]) - img.at(x, y)) <= 3, "crop pixel %d vs image %d", px[size_t(v) * w + u], img.at(x, y));
|
||||
}
|
||||
std::system(("rm -rf " + root).c_str());
|
||||
}
|
||||
|
||||
void test_ghosts() {
|
||||
const Img img(128, 128);
|
||||
const std::map<std::string, Image> images = {{"a", img.image()}};
|
||||
FlagConfig c = small_config();
|
||||
c.after_s = 0.05;
|
||||
FlagBuffer fb(c);
|
||||
const std::string root = temp_dir(), dir = root + "/flag-g";
|
||||
fb.add(0, 0, 0, images, {view("a", 7, 60, 60, 20)}, "");
|
||||
for (int i = 1; i <= 5; ++i) fb.add(i * kSec / 10, i * kSec / 10, i, images, {}, "");
|
||||
CHECK(fb.stats().ghosts == 2, "ghosts within 0.25 s: %zu (want 2: at 0.1 and 0.2 s)", fb.stats().ghosts);
|
||||
fb.flag({dir, ""}, 5 * kSec / 10);
|
||||
fb.add(6 * kSec / 10, 6 * kSec / 10, 6, images, {}, "");
|
||||
fb.wait();
|
||||
const std::vector<Json::Value> sets = lines(dir + "/sets.jsonl"), crops = lines(dir + "/crops.jsonl");
|
||||
int ghost_lines = 0;
|
||||
for (const Json::Value &s : sets) ghost_lines += int(s["ghosts"].size());
|
||||
CHECK(ghost_lines == 2 && crops.size() == 3, "ghost lines %d crops %zu", ghost_lines, crops.size());
|
||||
if (crops.size() == 3) CHECK(!crops[0]["ghost"].asBool() && crops[1]["ghost"].asBool() && crops[1]["hand"].asInt() == 7, "ghost crop");
|
||||
std::system(("rm -rf " + root).c_str());
|
||||
}
|
||||
|
||||
void test_tick() {
|
||||
const Img img(64, 64);
|
||||
FlagBuffer fb(small_config());
|
||||
const std::string root = temp_dir(), dir = root + "/flag-t";
|
||||
fb.add(0, 0, 0, {{"a", img.image()}}, {view("a", 1, 30, 30, 10)}, "");
|
||||
fb.flag({dir, ""}, 0);
|
||||
fb.tick(kSec / 2); // 0.5 s: not yet (after_s + 0.5)
|
||||
CHECK(fb.waiting(), "ticked too early");
|
||||
fb.tick(kSec + 1);
|
||||
CHECK(!fb.waiting(), "tick didn't complete it");
|
||||
fb.wait();
|
||||
CHECK(file_json(dir + "/manifest.json")["counts"]["sets"].asInt() == 1, "tick flag sets");
|
||||
// a folder it can't make: reported, the slots still come back
|
||||
FlagBuffer fb2(small_config());
|
||||
fb2.add(0, 0, 0, {{"a", img.image()}}, {view("a", 1, 30, 30, 10)}, "");
|
||||
fb2.flag({root + "/missing/deeper", ""}, 0);
|
||||
fb2.finish();
|
||||
fb2.wait();
|
||||
CHECK(!fb2.stats().last_error.empty() && fb2.stats().slots_used == 0, "bad folder: error '%s', %zu slots",
|
||||
fb2.stats().last_error.c_str(), fb2.stats().slots_used);
|
||||
std::system(("rm -rf " + root).c_str());
|
||||
}
|
||||
|
||||
// Not a check: what a crop costs here, a hand at arm's length in a 1056 x 1024 camera (a 375 px crop,
|
||||
// so a 600 px square) down to 224 and 288, and a small one copied as it is.
|
||||
void bench() {
|
||||
const Img img(1056, 1024);
|
||||
std::vector<uint8_t> out(512 * 512);
|
||||
for (int px : {224, 288}) {
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
for (int i = 0; i < 300; ++i) crop_square(img.image(), {500.0 + i % 7, 480}, 600, px, out.data());
|
||||
const double us = std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - t0).count() / 300;
|
||||
std::printf("flagbuf_test: a 600 px square to %d px: %.0f us a crop\n", px, us);
|
||||
}
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
for (int i = 0; i < 300; ++i) crop_square(img.image(), {500.0 + i % 7, 480}, 200, 224, out.data());
|
||||
std::printf("flagbuf_test: a 200 px square as it is: %.0f us a crop\n",
|
||||
std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - t0).count() / 300);
|
||||
}
|
||||
|
||||
void test_json_string() {
|
||||
CHECK(json_string("a\"b\\c\nd") == "\"a\\\"b\\\\c\\nd\"", "%s", json_string("a\"b\\c\nd").c_str());
|
||||
CHECK(json_string("", true) == "null" && json_string("") == "\"\"", "empty");
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main() {
|
||||
test_crop_geometry();
|
||||
test_window_and_slots();
|
||||
test_flag_files();
|
||||
test_ghosts();
|
||||
test_tick();
|
||||
test_json_string();
|
||||
bench();
|
||||
std::printf("flagbuf_test: %s (%d failures)\n", failures ? "FAILED" : "ok", failures);
|
||||
return failures ? 1 : 0;
|
||||
}
|
||||
@@ -1,276 +0,0 @@
|
||||
// Unit tests for the pinch and grip detectors' hold on a drag (track/pinch.h: PinchParams' drag_*
|
||||
// and grip_takeover_s), the dip rule (PinchParams::dip) and the fist guard (fist_curl): made-up hands
|
||||
// at 30 Hz, the distances from their points (no views).
|
||||
// Build and run: make check (in the dev container; hands/build.sh's environment).
|
||||
#include "../track/pinch.h"
|
||||
|
||||
#include <cmath>
|
||||
#include <cstdio>
|
||||
#include <vector>
|
||||
|
||||
static int failures = 0;
|
||||
#define CHECK(cond, ...) \
|
||||
do { \
|
||||
if (!(cond)) { \
|
||||
std::printf("FAIL %s:%d: ", __FILE__, __LINE__); \
|
||||
std::printf(__VA_ARGS__); \
|
||||
std::printf("\n"); \
|
||||
++failures; \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int64_t kFrame = 33'333'333;
|
||||
|
||||
// A right hand 0.4 m ahead (moved by shift), the thumb tip d from the index tip, the other fingers
|
||||
// open (curl 2) or closed around (fist: curl 0.85); middle_gap > 0 puts the middle tip that far
|
||||
// from the thumb tip (a hand closing into a grab). curled: the fingers closed (bit 0 index .. 3 little)
|
||||
// when not a fist.
|
||||
Hand MakeHand(double d, bool fist = false, V3 shift = {0, 0, 0}, double middle_gap = 0, int curled = 0) {
|
||||
Hand h;
|
||||
h.id = 1, h.side = 1, h.frames = 30, h.has_pts = true;
|
||||
const V3 wrist = V3{0, -0.1, -0.35} + shift;
|
||||
auto at = [&](double x, double y, double z) { return V3{wrist[0] + x, wrist[1] + y, wrist[2] + z}; };
|
||||
const double knuckle = 0.08;
|
||||
h.pts[0] = at(0, 0, 0);
|
||||
for (int f = 0; f < 4; ++f) {
|
||||
const double x = -0.02 + 0.015 * f, tip = fist || (curled >> f & 1) ? 0.068 : 0.16;
|
||||
h.pts[5 + 4 * f] = at(x, 0, -knuckle);
|
||||
h.pts[6 + 4 * f] = at(x, 0, -knuckle - 0.02);
|
||||
h.pts[7 + 4 * f] = at(x, 0, -knuckle - 0.04);
|
||||
h.pts[8 + 4 * f] = at(x, 0, -tip);
|
||||
}
|
||||
// the thumb: its tip d from the index tip, beside it
|
||||
h.pts[1] = at(-0.03, 0, -0.02), h.pts[2] = at(-0.04, 0, -0.04), h.pts[3] = at(-0.045, 0, -0.06);
|
||||
const V3 index = h.pts[8];
|
||||
h.pts[4] = {index[0] - d, index[1], index[2]};
|
||||
if (middle_gap > 0) h.pts[12] = {h.pts[4][0], h.pts[4][1] + middle_gap, h.pts[4][2]};
|
||||
for (int i = 0; i < 21; ++i) h.smooth[i] = h.pts[i];
|
||||
return h;
|
||||
}
|
||||
|
||||
struct Run {
|
||||
Pinch pinch;
|
||||
Grip grip;
|
||||
int64_t t = 1'000'000'000;
|
||||
int ends = 0, lost = 0;
|
||||
int begins = 0;
|
||||
V3 shift{0, 0, 0}, velocity{0, 0, 0}; // the hand's offset and its motion, m/s
|
||||
double middle_gap = 0;
|
||||
int curled = 0;
|
||||
void Step(double d, bool fist = false) {
|
||||
Hand h = MakeHand(d, fist, shift, middle_gap, curled);
|
||||
shift = shift + velocity * (kFrame / 1e9);
|
||||
const std::vector<const Hand *> hands{&h};
|
||||
grip.update(hands, {}, t, pinch.holding(t));
|
||||
pinch.update(hands, {}, t, grip.gripping());
|
||||
for (const Pinch::Event &e : pinch.events) {
|
||||
begins += std::string(e.what) == "begin";
|
||||
ends += std::string(e.what) != "begin";
|
||||
lost += std::string(e.what) == "lost";
|
||||
}
|
||||
t += kFrame;
|
||||
}
|
||||
bool down() const { return pinch.side(1).flags & FH_PINCH_DOWN; }
|
||||
};
|
||||
|
||||
PinchParams Params() {
|
||||
PinchParams p;
|
||||
p.other_finger = 12; // as ft-hands
|
||||
return p;
|
||||
}
|
||||
|
||||
void TestDragSpike() {
|
||||
// a drag (held 1 s): the tips read 3.8 cm for two frames (a hidden thumb), then close again
|
||||
Run r{Pinch(Params()), Grip()};
|
||||
for (int k = 0; k < 5; ++k) r.Step(0.06);
|
||||
for (int k = 0; k < 30; ++k) r.Step(0.01);
|
||||
CHECK(r.down(), "the pinch is down");
|
||||
r.Step(0.038), r.Step(0.038), r.Step(0.01);
|
||||
CHECK(r.down() && r.ends == 0, "a two-frame spike to 3.8 cm doesn't drop a drag");
|
||||
// loosened a little and staying there: it ends after drag_open_s (0.12 s: four frames)
|
||||
int frames = 0;
|
||||
while (r.down() && frames < 20) r.Step(0.04), ++frames;
|
||||
CHECK(!r.down() && frames >= 4 && frames <= 5 && r.lost == 0, "loosened to 4 cm: ended after %d frames", frames);
|
||||
// fingers hovering 3.3-3.8 cm apart (a relaxed hand) end it too: the dips under 3.5 cm aren't
|
||||
// a close
|
||||
Run h{Pinch(Params()), Grip()};
|
||||
for (int k = 0; k < 5; ++k) h.Step(0.06);
|
||||
for (int k = 0; k < 30; ++k) h.Step(0.01);
|
||||
frames = 0;
|
||||
while (h.down() && frames < 30) h.Step(frames % 2 ? 0.033 : 0.038), ++frames;
|
||||
CHECK(!h.down() && frames <= 6, "hovering 3.3-3.8 cm apart: ended after %d frames", frames);
|
||||
// a real release: the tips 6 cm apart end it after end_frames (2), as before
|
||||
Run q{Pinch(Params()), Grip()};
|
||||
for (int k = 0; k < 5; ++k) q.Step(0.06);
|
||||
for (int k = 0; k < 30; ++k) q.Step(0.01);
|
||||
q.Step(0.06);
|
||||
CHECK(q.down(), "one frame open: still down");
|
||||
q.Step(0.07);
|
||||
CHECK(!q.down() && q.lost == 0, "a release to 6-7 cm ends it on the second frame");
|
||||
}
|
||||
|
||||
void TestTapUnchanged() {
|
||||
// a tap (0.1 s): ends after two frames past 3.5 cm, wherever they are
|
||||
Run r{Pinch(Params()), Grip()};
|
||||
for (int k = 0; k < 5; ++k) r.Step(0.06);
|
||||
for (int k = 0; k < 3; ++k) r.Step(0.01);
|
||||
r.Step(0.038);
|
||||
CHECK(r.down(), "a tap's first frame open");
|
||||
r.Step(0.038);
|
||||
CHECK(!r.down(), "a tap ends on the second frame past 3.5 cm");
|
||||
}
|
||||
|
||||
void TestGripTakeover() {
|
||||
// closing the hand passes through a pinch: a grip within 0.3 s takes it (ends it, as lost)
|
||||
Run a{Pinch(Params()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) a.Step(0.06);
|
||||
a.Step(0.015);
|
||||
CHECK(a.down(), "the pinch began");
|
||||
for (int k = 0; k < 4 && a.down(); ++k) a.Step(0.032, true);
|
||||
CHECK(!a.down() && a.lost == 1, "a grip right after the pinch took it (lost %d)", a.lost);
|
||||
// a drag held 1 s, the other fingers curling around it: no grip, and the pinch stays
|
||||
Run b{Pinch(Params()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) b.Step(0.06);
|
||||
for (int k = 0; k < 30; ++k) b.Step(0.015);
|
||||
for (int k = 0; k < 15; ++k) b.Step(0.032, true);
|
||||
CHECK(b.down() && b.ends == 0, "a held pinch survives the hand closing around it");
|
||||
CHECK(!(b.grip.side(1).flags & FH_PINCH_DOWN), "and no grip begins on it");
|
||||
}
|
||||
|
||||
PinchParams DipParams() {
|
||||
PinchParams p = Params();
|
||||
p.dip = true;
|
||||
return p;
|
||||
}
|
||||
|
||||
// open, a quick dip to `low` (two frames) and back open
|
||||
void QuickDip(Run &r, double low) {
|
||||
for (int k = 0; k < 10; ++k) r.Step(0.06);
|
||||
r.Step(0.045), r.Step(low), r.Step(low + 0.001), r.Step(0.045), r.Step(0.06), r.Step(0.06);
|
||||
}
|
||||
|
||||
void TestDip() {
|
||||
// a quick tap that reads 2.8 cm at contact: no pinch without the dip rule, a click with it
|
||||
Run off{Pinch(Params()), Grip()};
|
||||
QuickDip(off, 0.028);
|
||||
CHECK(off.begins == 0, "without the dip rule a 2.8 cm dip is no pinch (%d)", off.begins);
|
||||
Run on{Pinch(DipParams()), Grip()};
|
||||
QuickDip(on, 0.028);
|
||||
CHECK(on.begins == 1 && on.ends == 1 && on.lost == 0 && !on.down(), "a 2.8 cm dip clicks once (%d begins, %d ends)",
|
||||
on.begins, on.ends);
|
||||
CHECK(on.pinch.dip_begins[1] == 1, "counted as a dip");
|
||||
// too shallow: 6 to 3.5 cm (over dip_max_m)
|
||||
Run shallow{Pinch(DipParams()), Grip()};
|
||||
QuickDip(shallow, 0.035);
|
||||
CHECK(shallow.begins == 0, "a dip to 3.5 cm isn't a tap");
|
||||
// closing slowly (1 mm a frame) to 2.8 cm and staying: not a dip
|
||||
Run slow{Pinch(DipParams()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) slow.Step(0.06);
|
||||
for (double d = 0.06; d > 0.028; d -= 0.001) slow.Step(d);
|
||||
for (int k = 0; k < 10; ++k) slow.Step(0.028);
|
||||
CHECK(slow.begins == 0, "a slow close to 2.8 cm isn't a tap (%d)", slow.begins);
|
||||
// the same quick dip while the hand moves at 0.5 m/s: no tap
|
||||
Run moving{Pinch(DipParams()), Grip()};
|
||||
moving.velocity = {0.5, 0, 0};
|
||||
QuickDip(moving, 0.028);
|
||||
CHECK(moving.begins == 0, "a dip while the hand moves fast isn't a tap (%d)", moving.begins);
|
||||
// the middle fingertip coming in with the thumb (a hand closing into a grab): no tap
|
||||
Run grab{Pinch(DipParams()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) grab.Step(0.06);
|
||||
grab.middle_gap = 0.03;
|
||||
grab.Step(0.045), grab.Step(0.028), grab.Step(0.028);
|
||||
CHECK(grab.begins == 0, "a dip with the middle tip 3 cm from the thumb isn't a tap (%d)", grab.begins);
|
||||
// a dip that goes on to contact is an ordinary pinch: held 1 s, a two-frame spike doesn't end it
|
||||
Run hold{Pinch(DipParams()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) hold.Step(0.06);
|
||||
hold.Step(0.045), hold.Step(0.028);
|
||||
CHECK(hold.down(), "the dip began it");
|
||||
for (int k = 0; k < 30; ++k) hold.Step(0.01);
|
||||
hold.Step(0.038), hold.Step(0.038), hold.Step(0.01);
|
||||
CHECK(hold.down() && hold.ends == 0, "after contact it holds like any drag");
|
||||
// a dip held without contact ends once the tips spring back dip_rebound_m (6 mm) over its low
|
||||
Run spring{Pinch(DipParams()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) spring.Step(0.06);
|
||||
spring.Step(0.045), spring.Step(0.026);
|
||||
for (int k = 0; k < 10; ++k) spring.Step(0.027);
|
||||
CHECK(spring.down(), "held at 2.7 cm: still down");
|
||||
spring.Step(0.033);
|
||||
CHECK(spring.down(), "one frame 7 mm over the low: still down");
|
||||
spring.Step(0.033);
|
||||
CHECK(!spring.down() && spring.lost == 0, "two frames 7 mm over the low: released, under end_m");
|
||||
// two taps in a row click twice; one held low after its release doesn't click again: a new dip
|
||||
// needs the hand open again since the release
|
||||
Run twice{Pinch(DipParams()), Grip()};
|
||||
QuickDip(twice, 0.028);
|
||||
QuickDip(twice, 0.028);
|
||||
CHECK(twice.begins == 2, "two dips, two clicks (%d)", twice.begins);
|
||||
Run settle{Pinch(DipParams()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) settle.Step(0.06);
|
||||
settle.Step(0.045), settle.Step(0.026), settle.Step(0.033), settle.Step(0.033); // released at 3.3 cm
|
||||
for (int k = 0; k < 10; ++k) settle.Step(k % 2 ? 0.033 : 0.028);
|
||||
CHECK(settle.begins == 1 && !settle.down(), "jitter at 2.8-3.3 cm after a dip's release: no second click (%d)",
|
||||
settle.begins);
|
||||
}
|
||||
|
||||
void TestFistGuard() {
|
||||
// a fist with the thumb tip 1.5 cm from the curled index tip: no pinch, counted as held back
|
||||
Run fist{Pinch(Params()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) fist.Step(0.06);
|
||||
for (int k = 0; k < 15; ++k) fist.Step(0.015, true);
|
||||
CHECK(fist.begins == 0 && fist.pinch.fist_held[1] == 1, "a fist isn't a pinch (%d begins)", fist.begins);
|
||||
// its curls flickering open for a frame, the thumb still on the index: still no pinch
|
||||
fist.Step(0.015), fist.Step(0.015, true);
|
||||
CHECK(fist.begins == 0, "a fist's curls flickering open don't begin a pinch (%d)", fist.begins);
|
||||
// the hand opens (the tips past end_m), then pinches: it begins
|
||||
fist.Step(0.06), fist.Step(0.015), fist.Step(0.015);
|
||||
CHECK(fist.begins == 1, "a pinch once the fist opened (%d)", fist.begins);
|
||||
// the index and middle curled, ring and little open: still a fist for the guard
|
||||
Run two{Pinch(Params()), Grip()};
|
||||
two.curled = 0b0011;
|
||||
for (int k = 0; k < 15; ++k) two.Step(0.015);
|
||||
CHECK(two.begins == 0, "index and middle curled: no pinch (%d)", two.begins);
|
||||
// ring and little curled (a pinch with the other fingers tucked in), or the index alone: a pinch
|
||||
Run tucked{Pinch(Params()), Grip()};
|
||||
tucked.curled = 0b1100;
|
||||
for (int k = 0; k < 5; ++k) tucked.Step(0.015);
|
||||
CHECK(tucked.begins == 1, "ring and little curled: a pinch (%d)", tucked.begins);
|
||||
Run index{Pinch(Params()), Grip()};
|
||||
index.curled = 0b0001;
|
||||
for (int k = 0; k < 5; ++k) index.Step(0.015);
|
||||
CHECK(index.begins == 1, "the index alone curled: a pinch (%d)", index.begins);
|
||||
// a pinch already down carries on while the index and middle curl around it
|
||||
Run held{Pinch(Params()), Grip()};
|
||||
for (int k = 0; k < 10; ++k) held.Step(0.015);
|
||||
held.curled = 0b0011;
|
||||
for (int k = 0; k < 15; ++k) held.Step(0.015);
|
||||
CHECK(held.down() && held.ends == 0, "a pinch down survives its fingers curling");
|
||||
// the fist opening with the fingers apart, then a pinch: it begins
|
||||
held.curled = 0;
|
||||
held.Step(0.06), held.Step(0.06);
|
||||
for (int k = 0; k < 3; ++k) held.Step(0.015);
|
||||
CHECK(held.begins == 2, "a pinch after the fist opened (%d)", held.begins);
|
||||
// fist_curl 0: the guard off, a fist with the thumb on the index pinches as before
|
||||
PinchParams off = Params();
|
||||
off.fist_curl = 0;
|
||||
Run old{Pinch(off), Grip()};
|
||||
for (int k = 0; k < 5; ++k) old.Step(0.015, true);
|
||||
CHECK(old.begins == 1, "guard off: the fist pinches (%d)", old.begins);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main() {
|
||||
TestDragSpike();
|
||||
TestTapUnchanged();
|
||||
TestGripTakeover();
|
||||
TestDip();
|
||||
TestFistGuard();
|
||||
if (failures) {
|
||||
std::printf("%d failures\n", failures);
|
||||
return 1;
|
||||
}
|
||||
std::printf("pinch-test: ok\n");
|
||||
return 0;
|
||||
}
|
||||
@@ -1,167 +0,0 @@
|
||||
// Unit tests for the skeleton fit (track/skeleton.h) on made-up hands: poses within the
|
||||
// joint limits are posed with the forward kinematics, noise is added, and the fit has to give
|
||||
// them back. No models, no recordings. make check (in the dev container) builds and runs it.
|
||||
#include "../track/skeleton.h"
|
||||
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <random>
|
||||
|
||||
using namespace skel;
|
||||
|
||||
namespace {
|
||||
|
||||
int failures = 0;
|
||||
#define CHECK(c) \
|
||||
do { \
|
||||
if (!(c)) std::printf("FAIL %s:%d: %s\n", __FILE__, __LINE__, #c), ++failures; \
|
||||
} while (0)
|
||||
|
||||
double lo_hi(std::mt19937 &rng, double lo, double hi) { return std::uniform_real_distribution<double>(lo, hi)(rng); }
|
||||
|
||||
// A random but plausible pose: angles well inside the limits, the palm somewhere in front.
|
||||
void random_pose(std::mt19937 &rng, Rot &R, V3 &t, double q[kParams]) {
|
||||
R = exp_rot({lo_hi(rng, -1.5, 1.5), lo_hi(rng, -1.5, 1.5), lo_hi(rng, -1.5, 1.5)});
|
||||
t = {lo_hi(rng, -0.25, 0.25), lo_hi(rng, -0.4, 0.0), lo_hi(rng, -0.6, -0.25)};
|
||||
const double D = M_PI / 180;
|
||||
q[0] = lo_hi(rng, -30, 30) * D, q[1] = lo_hi(rng, -30, 30) * D, q[2] = lo_hi(rng, -15, 15) * D;
|
||||
q[3] = lo_hi(rng, -10, 10) * D, q[4] = lo_hi(rng, 0, 60) * D, q[5] = lo_hi(rng, 0, 70) * D;
|
||||
for (int f = 0; f < 4; ++f) {
|
||||
double *a = q + 6 + 5 * f;
|
||||
a[0] = lo_hi(rng, -15, 15) * D, a[1] = lo_hi(rng, -10, 80) * D, a[2] = lo_hi(rng, -8, 8) * D;
|
||||
a[3] = lo_hi(rng, 0, 90) * D, a[4] = lo_hi(rng, 0, 60) * D;
|
||||
}
|
||||
}
|
||||
|
||||
V3 cross(V3 a, V3 b) { return {a[1] * b[2] - a[2] * b[1], a[2] * b[0] - a[0] * b[2], a[0] * b[1] - a[1] * b[0]}; }
|
||||
|
||||
} // namespace
|
||||
|
||||
int main() {
|
||||
std::mt19937 rng(7);
|
||||
std::normal_distribution<double> noise(0, 1);
|
||||
const Shape s = Skeleton::canonical(1.2);
|
||||
|
||||
// rotations
|
||||
for (int i = 0; i < 100; ++i) {
|
||||
const V3 w{lo_hi(rng, -2, 2), lo_hi(rng, -2, 2), lo_hi(rng, -2, 2)};
|
||||
if (norm(w) > 3.1) continue;
|
||||
const V3 back = log_rot(exp_rot(w));
|
||||
CHECK(norm(back - w) < 1e-9);
|
||||
double q[4];
|
||||
to_quat(exp_rot(w), q);
|
||||
const double a = 2 * std::atan2(std::sqrt(q[0] * q[0] + q[1] * q[1] + q[2] * q[2]), std::fabs(q[3]));
|
||||
CHECK(std::fabs(a - norm(w)) < 1e-9);
|
||||
}
|
||||
|
||||
// the frames: right-handed, -Z along the bone, +Y the back of the hand
|
||||
{
|
||||
Rot R;
|
||||
V3 t{0, 0, -0.4}, pts[21];
|
||||
Rot fr[21];
|
||||
double q[kParams] = {};
|
||||
for (int side : {1, -1}) {
|
||||
Skeleton::pose(s, side, R, t, q, pts, fr);
|
||||
for (int i = 0; i < 21; ++i) {
|
||||
CHECK(norm(cross(fr[i].col(1), fr[i].col(2)) - fr[i].col(0)) < 1e-9);
|
||||
if (i % 4 != 0 && i < 20 && i != 0) {
|
||||
const int child = i + 1;
|
||||
CHECK(dot(unit(pts[child] - pts[i]), fr[i].col(2) * -1) > 0.999);
|
||||
}
|
||||
}
|
||||
// palm down, fingers forward (-z): the back of the hand is +y; the index knuckle is
|
||||
// on the thumb's side: -x on a right hand, +x on a left one
|
||||
CHECK(fr[0].col(1)[1] > 0.99);
|
||||
CHECK(side * pts[5][0] < 0);
|
||||
// flexing a finger moves its tip toward the palm (-y)
|
||||
double qb[kParams] = {};
|
||||
qb[6 + 3] = 1.0; // the index PIP
|
||||
V3 bent[21];
|
||||
Skeleton::pose(s, side, R, t, qb, bent, fr);
|
||||
CHECK(bent[8][1] < pts[8][1] - 0.01);
|
||||
// abduction moves the index toward the thumb
|
||||
double qa[kParams] = {};
|
||||
qa[6] = 0.3;
|
||||
Skeleton::pose(s, side, R, t, qa, bent, fr);
|
||||
CHECK(side * (bent[8][0] - pts[8][0]) < -0.005);
|
||||
}
|
||||
}
|
||||
|
||||
// fits: noise-free poses come back exactly; noisy ones close to the truth
|
||||
double sl[21], sd[21];
|
||||
for (int i = 0; i < 21; ++i) sl[i] = 0.002, sd[i] = 0.007;
|
||||
for (const double sigma : {0.0, 0.002}) {
|
||||
double worst_rms = 0, sum_err = 0, sum_raw = 0, us = 0;
|
||||
int n = 0;
|
||||
for (int trial = 0; trial < 300; ++trial) {
|
||||
const int side = trial % 2 ? 1 : -1;
|
||||
Rot R0, R;
|
||||
V3 t0, t;
|
||||
double q0[kParams] = {}, q[kParams];
|
||||
random_pose(rng, R0, t0, q0);
|
||||
V3 truth[21], meas[21], fit[21];
|
||||
Rot fr[21];
|
||||
Skeleton::pose(s, side, R0, t0, q0, truth, fr);
|
||||
// noise as stereo has it: 3.5 times as much along the line of sight
|
||||
for (int i = 0; i < 21; ++i) {
|
||||
const V3 d = unit(truth[i]), e = V3{noise(rng), noise(rng), noise(rng)} * sigma;
|
||||
meas[i] = truth[i] + e + d * (2.5 * dot(e, d));
|
||||
}
|
||||
double rms = 0;
|
||||
const auto a = std::chrono::steady_clock::now();
|
||||
const double cost = Skeleton::solve(s, side, meas, sl, sd, {0, 0, 0}, nullptr, 4, R, t, q, &rms);
|
||||
us += std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - a).count();
|
||||
CHECK(cost >= 0);
|
||||
Skeleton::pose(s, side, R, t, q, fit, fr);
|
||||
double err = 0, raw = 0;
|
||||
for (int i = 0; i < 21; ++i) err += norm(fit[i] - truth[i]) / 21, raw += norm(meas[i] - truth[i]) / 21;
|
||||
worst_rms = std::max(worst_rms, rms);
|
||||
sum_err += err, sum_raw += raw, ++n;
|
||||
}
|
||||
std::printf("noise %.1f mm (x3.5 in depth): fit rms worst %.2f mm; mean error to the truth: fitted %.2f mm, measured %.2f mm; "
|
||||
"%.1f us a fit\n", sigma * 1000, worst_rms * 1000, sum_err / n * 1000, sum_raw / n * 1000, us / n);
|
||||
if (sigma == 0) CHECK(worst_rms < 0.002);
|
||||
else CHECK(sum_err < sum_raw); // the bone lengths take noise out
|
||||
}
|
||||
|
||||
// calibration: from noisy samples of one hand, the lengths settle on it
|
||||
{
|
||||
Skeleton sk;
|
||||
const Shape truth = Skeleton::canonical(1.3);
|
||||
Fit f;
|
||||
int64_t ns = 1'000'000'000;
|
||||
for (int k = 0; k < 120; ++k, ns += 66'000'000) {
|
||||
Rot R0;
|
||||
V3 t0, pts[21];
|
||||
Rot fr[21];
|
||||
double q0[kParams] = {};
|
||||
random_pose(rng, R0, t0, q0);
|
||||
for (int a = 6; a < kAngles; ++a) q0[a] *= 0.3; // fairly open, so calibration takes it
|
||||
Skeleton::pose(truth, 1, R0, t0, q0, pts, fr);
|
||||
for (V3 &p : pts) p = p + V3{noise(rng), noise(rng), noise(rng)} * 0.0015;
|
||||
sk.update(f, pts, true, 0.95, 2, 0.0015, nullptr, {0, 0, 0}, 1.0, ns);
|
||||
CHECK(f.valid);
|
||||
}
|
||||
const Shape got = sk.shape(1, 1.0);
|
||||
double worst = 0;
|
||||
for (int b = 0; b < 20; ++b) worst = std::max(worst, std::fabs(got.len[b] - truth.len[b]));
|
||||
std::printf("calibration: settled %.2f, worst bone off %.2f mm\n", sk.calibration(1).settled(), worst * 1000);
|
||||
CHECK(sk.calibration(1).settled() >= 1);
|
||||
CHECK(worst < 0.002);
|
||||
// the left hand borrows the right's lengths until it has its own
|
||||
const Shape left = sk.shape(-1, 1.0);
|
||||
CHECK(std::fabs(left.len[5] - got.len[5]) < 1e-9);
|
||||
}
|
||||
|
||||
// the elbow: behind and below a hand held in front, 2.8 palms away
|
||||
{
|
||||
const V3 wrist{0.15, -0.25, -0.35}, knuckle{0.15, -0.22, -0.43};
|
||||
const V3 e = Skeleton::elbow(wrist, knuckle, 1, 0.23, {0, 1, 0});
|
||||
std::printf("elbow for a right hand in front: %+.3f %+.3f %+.3f\n", e[0], e[1], e[2]);
|
||||
CHECK(std::fabs(norm(e - wrist) - 0.23) < 1e-6);
|
||||
CHECK(e[1] < wrist[1] && e[2] > wrist[2]);
|
||||
}
|
||||
|
||||
std::printf(failures ? "%d FAILED\n" : "skeleton tests passed\n", failures);
|
||||
return failures ? 1 : 0;
|
||||
}
|
||||
@@ -317,29 +317,11 @@ class Ring(unittest.TestCase):
|
||||
self.assertEqual(r["map"]["slam_left"]["node"], 0)
|
||||
|
||||
def test_four_cameras_in_ring(self):
|
||||
make_ring(self.ring, ["a_video9", "b_video13", "c_video6", "d_video7", "a_video9_dk"], nodes=[9, 13, 6, 7, 9])
|
||||
make_ring(self.ring, ["a_video9", "b_video13", "c_video6", "d_video7", "a_video9_dk"])
|
||||
r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
|
||||
self.assertEqual(r["status"], "ok")
|
||||
self.assertEqual(len(r["ring"]["mono"]), 4) # the dark twin doesn't count
|
||||
|
||||
def test_ring_from_before_the_cameras_moved(self):
|
||||
# 2026-10-12: the colour module disconnected at 08:10 and XRService moved the side pair
|
||||
# to video0/3; ft-camd, started the day before, still published video9/13, and a whole
|
||||
# session recorded only the upper pair
|
||||
unplugged = [L("11:00:00", "[buildMediaCtlSetupTasks] ISP enabled for tracking cameras (main VFE available)")] + \
|
||||
[L("11:00:01", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: x" % (i, n))
|
||||
for i, n in enumerate((0, 3, 6, 7))]
|
||||
with open(self.log, "w") as f:
|
||||
f.write("\n".join(START + GOOD_OPEN + unplugged) + "\n")
|
||||
make_ring(self.ring, ["og01a1b_4-0036_video9", "og01a1b_4-0060_video13", "og0ve10_5-0060_video6",
|
||||
"og0ve10_5-0060_video7"], nodes=[9, 13, 6, 7])
|
||||
r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
|
||||
self.assertEqual(r["status"], "degraded")
|
||||
self.assertEqual(r["ring_missing"], [0, 3])
|
||||
self.assertIn("ft-camd publishes video9 video13, but XRService runs those cameras on video0 video3",
|
||||
r["reason"])
|
||||
self.assertTrue(camcheck.is_ring_short(r))
|
||||
|
||||
def test_stale_ring_ignored(self):
|
||||
make_ring(self.ring, ["slam_video9"], alive=False)
|
||||
r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
|
||||
|
||||
@@ -1,145 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for hands/handsflag.py (`ft-handsctl flag`): the request ft-hands reads, the folder, the
|
||||
flash and head poses asked of a made-up ft-handview, and a request nobody takes. Nothing here talks
|
||||
to the real ft-hands or ft-handview, or opens /dev/video99.
|
||||
|
||||
python3 hands/tests/test_handsflag.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, os.path.dirname(HERE))
|
||||
import handsflag as hf # noqa: E402
|
||||
|
||||
|
||||
class FakeView(threading.Thread):
|
||||
"""ft-handview's socket under another name: answers flash and poses."""
|
||||
|
||||
def __init__(self, name):
|
||||
super().__init__(daemon=True)
|
||||
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
self.sock.bind("\0" + name)
|
||||
self.sock.settimeout(0.2)
|
||||
self.got, self.stop = [], False
|
||||
|
||||
def run(self):
|
||||
while not self.stop:
|
||||
try:
|
||||
data, addr = self.sock.recvfrom(256)
|
||||
except socket.timeout:
|
||||
continue
|
||||
cmd = data.decode()
|
||||
self.got.append(cmd)
|
||||
if cmd == "flash":
|
||||
reply = "ok 2"
|
||||
elif cmd.startswith("poses"):
|
||||
m = " ".join("%.5f" % v for v in (1, 0, 0, 0.1, 0, 1, 0, 1.6, 0, 0, 1, -0.2))
|
||||
reply = "ok" + "".join("\n%d %s" % (1000 + 33 * i, m) for i in range(3))
|
||||
else:
|
||||
reply = "error"
|
||||
self.sock.sendto(reply.encode(), addr)
|
||||
|
||||
|
||||
class FakeHands(threading.Thread):
|
||||
"""ft-hands' side: takes run_dir/flag and keeps what it said."""
|
||||
|
||||
def __init__(self, run_dir):
|
||||
super().__init__(daemon=True)
|
||||
self.path, self.seen, self.stop = os.path.join(run_dir, "flag"), None, False
|
||||
|
||||
def run(self):
|
||||
while not self.stop and self.seen is None:
|
||||
if os.path.exists(self.path):
|
||||
with open(self.path) as f:
|
||||
self.seen = dict(line.rstrip("\n").split("=", 1) for line in f if "=" in line)
|
||||
os.unlink(self.path)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
class FlagTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.mkdtemp(prefix="handsflag-test-")
|
||||
self.run_dir = os.path.join(self.tmp, "run")
|
||||
self.env = {k: os.environ.get(k) for k in ("FT_HANDS_FLAG_DIR", "FT_HANDS_FLAG_VIEW", "FT_HANDS_FLAG_TONE",
|
||||
"FT_HANDS_FLAG_FLASH", "FRAME_VOICE_WAKE_UNIX")}
|
||||
os.environ.update(FT_HANDS_FLAG_DIR=os.path.join(self.tmp, "flags"), FT_HANDS_FLAG_VIEW="0", FT_HANDS_FLAG_TONE="0",
|
||||
FT_HANDS_FLAG_FLASH="1")
|
||||
self.view_name = "ft_handview_test_%d" % os.getpid()
|
||||
|
||||
def tearDown(self):
|
||||
for k, v in self.env.items():
|
||||
if v is None:
|
||||
os.environ.pop(k, None)
|
||||
else:
|
||||
os.environ[k] = v
|
||||
shutil.rmtree(self.tmp, ignore_errors=True)
|
||||
|
||||
def test_flag_taken(self):
|
||||
view, hands = FakeView(self.view_name), FakeHands(self.run_dir)
|
||||
view.start(), hands.start()
|
||||
os.environ["FRAME_VOICE_WAKE_UNIX"] = "1760000000.25"
|
||||
try:
|
||||
rc = hf.main(["fingers", "crossed,\nnext", "line"], run_dir=self.run_dir, sock_name=self.view_name, detach=False)
|
||||
finally:
|
||||
view.stop = hands.stop = True
|
||||
view.join(1)
|
||||
view.sock.close()
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertIsNotNone(hands.seen)
|
||||
path = hands.seen["dir"]
|
||||
self.assertTrue(os.path.isdir(path) and os.path.basename(path).startswith("flag-"))
|
||||
self.assertEqual(os.stat(path).st_mode & 0o777, 0o700)
|
||||
self.assertEqual(hands.seen["note"], "fingers crossed, next line") # one line
|
||||
self.assertEqual(float(hands.seen["said_unix"]), 1760000000.25)
|
||||
self.assertIn("flash", view.got)
|
||||
self.assertTrue(any(c.startswith("poses ") for c in view.got))
|
||||
with open(os.path.join(path, "hmd.jsonl")) as f:
|
||||
poses = [json.loads(line) for line in f]
|
||||
self.assertEqual(len(poses), 3)
|
||||
self.assertEqual(len(poses[0]["m"]), 12)
|
||||
with open(os.path.join(path, "ctl.json")) as f:
|
||||
ctl = json.load(f)
|
||||
self.assertTrue(ctl["taken"])
|
||||
self.assertEqual(ctl["flash"], "ok 2")
|
||||
self.assertEqual(ctl["hmd_poses"], 3)
|
||||
self.assertNotIn("view", ctl) # HANDS_FLAG_VIEW=0
|
||||
self.assertFalse(os.path.exists(os.path.join(self.run_dir, "flag")))
|
||||
|
||||
def test_not_taken(self):
|
||||
start = time.monotonic()
|
||||
rc = hf.main([], run_dir=self.run_dir, sock_name=self.view_name, detach=False)
|
||||
self.assertEqual(rc, 1)
|
||||
self.assertLess(time.monotonic() - start, hf.TAKE_WAIT_S + hf.AFTER_S + 2)
|
||||
# the request is gone, so a later ft-hands doesn't take a stale one
|
||||
self.assertFalse(os.path.exists(os.path.join(self.run_dir, "flag")))
|
||||
with open(os.path.join(os.listdir(os.environ["FT_HANDS_FLAG_DIR"]) and os.path.join(
|
||||
os.environ["FT_HANDS_FLAG_DIR"], os.listdir(os.environ["FT_HANDS_FLAG_DIR"])[0]), "ctl.json")) as f:
|
||||
ctl = json.load(f)
|
||||
self.assertFalse(ctl["taken"])
|
||||
self.assertNotIn("flash", ctl)
|
||||
self.assertEqual(ctl["hmd_poses"], 0) # no ft-handview answering
|
||||
|
||||
def test_folders_dont_collide(self):
|
||||
base = os.path.join(self.tmp, "b")
|
||||
a, b = hf.flag_dir(base, 1760000000), hf.flag_dir(base, 1760000000)
|
||||
self.assertNotEqual(a, b)
|
||||
self.assertTrue(b.endswith("-2"))
|
||||
|
||||
def test_poses_lines(self):
|
||||
self.assertEqual(hf.poses_lines(None), [])
|
||||
self.assertEqual(hf.poses_lines("error x"), [])
|
||||
lines = hf.poses_lines("ok\n5 " + " ".join(["0.5"] * 12) + "\nbad line")
|
||||
self.assertEqual(len(lines), 1)
|
||||
self.assertEqual(json.loads(lines[0])["t_ns"], 5)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,275 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for hands/handsmode.py: HANDS_MODE and HANDS_MATTE_MARGIN in a temporary frametop.conf, the
|
||||
Hands page's three input switches (methods_of, method_writes), and which units each mode starts,
|
||||
stops and makes SteamVR want, with a made-up systemd (nothing here runs systemctl, getcap or pgrep).
|
||||
|
||||
python3 hands/tests/test_handsmode.py
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, os.path.dirname(HERE))
|
||||
import handsmode as hm # noqa: E402
|
||||
|
||||
CAMD, HANDS, VIEW = hm.TRACKING + (hm.VIEW,)
|
||||
CUT_CAMD, CUT_HANDS = hm.CUTOUTS
|
||||
BIN = "/home/u/frametop/hands/build/ft-camd"
|
||||
|
||||
|
||||
class FakeHost:
|
||||
"""systemd as the tests set it up: loaded units, active ones, SteamVR's wants."""
|
||||
|
||||
def __init__(self, loaded=(CAMD, HANDS), active=(), steamvr=True, desktop=True, caps=True, built=True,
|
||||
wanted=()):
|
||||
self.loaded, self.active, self.wanted_units = set(loaded), set(active), set(wanted)
|
||||
self.steamvr, self.desktop, self.has_caps, self.built = steamvr, desktop, caps, built
|
||||
self.calls = []
|
||||
|
||||
def units(self, names):
|
||||
out = {}
|
||||
for u in names:
|
||||
on = u in self.active or (u == hm.STEAMVR and self.steamvr)
|
||||
out[u] = {"load": "loaded" if u in self.loaded or u == hm.STEAMVR or (u in hm.CUTOUTS and on)
|
||||
else "not-found", "active": "active" if on else "inactive"}
|
||||
return out
|
||||
|
||||
def exec_path(self, unit):
|
||||
return BIN if unit == CAMD and CAMD in self.loaded else ""
|
||||
|
||||
def wanted(self, unit):
|
||||
return unit in self.wanted_units
|
||||
|
||||
def exists(self, path):
|
||||
return self.built
|
||||
|
||||
def caps(self, path):
|
||||
return self.has_caps
|
||||
|
||||
def desktop_up(self):
|
||||
return self.desktop
|
||||
|
||||
def cameras(self):
|
||||
return "ok"
|
||||
|
||||
def systemctl(self, *args):
|
||||
self.calls.append(list(args))
|
||||
|
||||
|
||||
def plan(mode, host, desktop=None, strict=False):
|
||||
return hm.plan(mode, hm.status(host, {"HANDS_MODE": mode}, cameras=False), desktop, strict)
|
||||
|
||||
|
||||
class ConfTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.dir = tempfile.mkdtemp(prefix="handsmode-test-")
|
||||
self.conf = os.path.join(self.dir, "frametop.conf")
|
||||
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.dir)
|
||||
|
||||
def test_mode_default_and_bad_values(self):
|
||||
self.assertEqual(hm.mode_of({}), "off")
|
||||
self.assertEqual(hm.mode_of({"HANDS_MODE": "sometimes"}), "off")
|
||||
for m in hm.MODES:
|
||||
self.assertEqual(hm.mode_of({"HANDS_MODE": m}), m)
|
||||
|
||||
def test_margin(self):
|
||||
self.assertEqual(hm.margin_of({}), hm.MARGIN_DEFAULT)
|
||||
self.assertEqual(hm.margin_of({"HANDS_MATTE_MARGIN": "0.75"}), 0.75)
|
||||
self.assertEqual(hm.margin_of({"HANDS_MATTE_MARGIN": "big"}), hm.MARGIN_DEFAULT)
|
||||
self.assertEqual(hm.margin_of({"HANDS_MATTE_MARGIN": "12"}), hm.MARGIN_DEFAULT) # ft-screens ignores it too
|
||||
|
||||
def test_write_keeps_comment_and_others(self):
|
||||
with open(self.conf, "w") as f:
|
||||
f.write("SCREENS=2\nHANDS_MODE=off # when hand tracking runs\n# HANDS_MODE=always\n")
|
||||
hm.write_value("HANDS_MODE", "frametop", self.conf)
|
||||
with open(self.conf) as f:
|
||||
text = f.read()
|
||||
self.assertIn("HANDS_MODE=frametop # when hand tracking runs\n", text)
|
||||
self.assertIn("SCREENS=2\n", text)
|
||||
self.assertIn("# HANDS_MODE=always\n", text) # a commented line isn't the setting
|
||||
self.assertEqual(hm.read_conf(self.conf), {"SCREENS": "2", "HANDS_MODE": "frametop"})
|
||||
|
||||
def test_write_adds_missing_key_and_file(self):
|
||||
hm.write_value("HANDS_MODE", "always", self.conf)
|
||||
hm.write_value("HANDS_MATTE_MARGIN", "0.75", self.conf)
|
||||
conf = hm.read_conf(self.conf)
|
||||
self.assertEqual((hm.mode_of(conf), hm.margin_of(conf)), ("always", 0.75))
|
||||
|
||||
def test_quoted_values(self):
|
||||
with open(self.conf, "w") as f:
|
||||
f.write('HANDS_MODE="always"\n')
|
||||
self.assertEqual(hm.mode_of(hm.read_conf(self.conf)), "always")
|
||||
|
||||
|
||||
class PlanTest(unittest.TestCase):
|
||||
def test_always_wants_and_starts(self):
|
||||
h = FakeHost(loaded=(CAMD, HANDS, VIEW))
|
||||
self.assertEqual(plan("always", h), [["add-wants", hm.STEAMVR, CAMD, HANDS, VIEW],
|
||||
["start", "--no-block", CAMD, HANDS, VIEW]])
|
||||
|
||||
def test_always_without_steamvr_only_wants(self):
|
||||
h = FakeHost(steamvr=False, wanted=(CAMD,))
|
||||
self.assertEqual(plan("always", h, strict=True), [["add-wants", hm.STEAMVR, HANDS]])
|
||||
|
||||
def test_hand_view_not_installed_is_left_out(self):
|
||||
h = FakeHost(loaded=(CAMD, HANDS))
|
||||
self.assertFalse(any(VIEW in call for call in plan("always", h)))
|
||||
|
||||
def test_frametop_with_desktop(self):
|
||||
h = FakeHost(wanted=(CAMD, HANDS))
|
||||
self.assertEqual(plan("frametop", h), [["disable", CAMD, HANDS], ["start", "--no-block", CAMD, HANDS]])
|
||||
|
||||
def test_frametop_without_desktop(self):
|
||||
h = FakeHost(desktop=False, active=(CAMD, HANDS))
|
||||
self.assertEqual(plan("frametop", h), []) # an install or caps run leaves ft-handsctl's alone
|
||||
self.assertEqual(plan("frametop", h, strict=True), [["stop", "--no-block", CAMD, HANDS]])
|
||||
|
||||
def test_desktop_hooks(self):
|
||||
h = FakeHost(desktop=False) # the hook's word wins over pgrep (ft-screens still starting)
|
||||
self.assertEqual(plan("frametop", h, desktop=True), [["start", "--no-block", CAMD, HANDS]])
|
||||
h = FakeHost(desktop=True, active=(CAMD, HANDS)) # ft-screens still ending
|
||||
self.assertEqual(plan("frametop", h, desktop=False), [["stop", "--no-block", CAMD, HANDS]])
|
||||
|
||||
def test_off_stops_everything_when_chosen(self):
|
||||
h = FakeHost(loaded=(CAMD, HANDS, VIEW), active=(CAMD, HANDS, VIEW, CUT_CAMD), wanted=(CAMD, HANDS, VIEW))
|
||||
self.assertEqual(plan("off", h, strict=True), [["disable", CAMD, HANDS, VIEW],
|
||||
["stop", "--no-block", CAMD, HANDS, VIEW, CUT_CAMD]])
|
||||
self.assertEqual(plan("off", h), [["disable", CAMD, HANDS, VIEW]])
|
||||
|
||||
def test_off_with_nothing_running(self):
|
||||
self.assertEqual(plan("off", FakeHost(), strict=True), [])
|
||||
|
||||
def test_no_caps_starts_nothing(self):
|
||||
h = FakeHost(caps=False)
|
||||
self.assertEqual(plan("always", h), [["add-wants", hm.STEAMVR, CAMD, HANDS]])
|
||||
self.assertEqual(plan("frametop", h), [])
|
||||
|
||||
def test_not_installed(self):
|
||||
h = FakeHost(loaded=())
|
||||
self.assertEqual(plan("always", h), [])
|
||||
self.assertEqual(plan("frametop", h), [])
|
||||
|
||||
def test_apply_runs_the_plan(self):
|
||||
h = FakeHost()
|
||||
calls, _ = hm.apply(h, {"HANDS_MODE": "frametop"})
|
||||
self.assertEqual(h.calls, calls)
|
||||
self.assertEqual(calls, [["start", "--no-block", CAMD, HANDS]])
|
||||
|
||||
|
||||
class StatusTest(unittest.TestCase):
|
||||
def test_problems(self):
|
||||
st = hm.status(FakeHost(loaded=()), {})
|
||||
self.assertIn("hands/run.sh install", st["problem"])
|
||||
st = hm.status(FakeHost(built=False), {})
|
||||
self.assertIn("isn't built", st["problem"])
|
||||
st = hm.status(FakeHost(caps=False), {"HANDS_MODE": "always"})
|
||||
self.assertIn("/home/u/frametop/hands/run.sh caps", st["problem"])
|
||||
self.assertEqual(st["summary"], "Hand tracking can't start with SteamVR")
|
||||
self.assertEqual(hm.status(FakeHost(), {})["problem"], "")
|
||||
|
||||
def test_cameras(self):
|
||||
h = FakeHost()
|
||||
h.cameras = lambda: "degraded: upper cameras and IR light off (VCINT FPGA failed to load)"
|
||||
self.assertIn("upper cameras", hm.status(h, {})["problem"])
|
||||
h.steamvr = False # nothing to check without SteamVR
|
||||
self.assertEqual(hm.status(h, {})["problem"], "")
|
||||
|
||||
def test_summary(self):
|
||||
self.assertEqual(hm.status(FakeHost(), {})["summary"], "Hand tracking is off")
|
||||
self.assertEqual(hm.status(FakeHost(active=(CAMD, HANDS)), {})["summary"],
|
||||
"Hand tracking is running (started by hand: ft-handsctl)")
|
||||
self.assertEqual(hm.status(FakeHost(active=(CAMD, HANDS)), {"HANDS_MODE": "frametop"})["summary"],
|
||||
"Hand tracking is running")
|
||||
self.assertEqual(hm.status(FakeHost(steamvr=False), {"HANDS_MODE": "frametop"})["summary"],
|
||||
"Hand tracking starts with the Frametop desktop")
|
||||
self.assertEqual(hm.status(FakeHost(active=(CUT_CAMD, CUT_HANDS)), {})["summary"],
|
||||
"The hand cutouts are running (started by hand: ft-cutouts)")
|
||||
|
||||
|
||||
class MethodsTest(unittest.TestCase):
|
||||
"""The Hands page's three switches (gaze clicks, the ray, direct touch): read from old configs as
|
||||
they always worked, and each toggled without moving the others."""
|
||||
|
||||
def apply(self, conf, method, on):
|
||||
new = dict(conf, **hm.method_writes(conf, method, on))
|
||||
return new, hm.methods_of(new)
|
||||
|
||||
def test_old_configs_read_as_before(self):
|
||||
self.assertEqual(hm.methods_of({}), {"gaze_clicks": False, "ray": False, "touch": False})
|
||||
on = {"gaze_clicks": True, "ray": True, "touch": True}
|
||||
self.assertEqual(hm.methods_of({"POINTER_HANDS": "1"}), on)
|
||||
self.assertEqual(hm.methods_of({"POINTER_HANDS": "1", "HANDS_NEAR": "1"}), on) # the example config's
|
||||
self.assertEqual(hm.methods_of({"POINTER_HANDS": "1", "POINTER_HAND_RAY": "0"})["ray"], False)
|
||||
self.assertEqual(hm.methods_of({"POINTER_HANDS": "1", "HANDS_NEAR": "0"})["touch"], False)
|
||||
self.assertTrue(hm.methods_of({"HANDS_NEAR": "force"})["touch"])
|
||||
self.assertFalse(hm.methods_of({"HANDS_NEAR": "1"})["touch"], "1 without POINTER_HANDS: off, as before")
|
||||
self.assertTrue(hm.methods_of({"HANDS_NEAR": "on"})["touch"])
|
||||
self.assertFalse(hm.methods_of({"POINTER_HANDS": "yes"})["ray"], "as the helper reads it: not a number, 0")
|
||||
|
||||
def test_each_switch_alone(self):
|
||||
for method in hm.METHODS:
|
||||
conf, got = self.apply({}, method, True)
|
||||
self.assertEqual(got, {m: m == method for m in hm.METHODS}, f"{method} alone: {conf}")
|
||||
conf, got = self.apply(conf, method, False)
|
||||
self.assertEqual(got, {m: False for m in hm.METHODS}, f"{method} off again: {conf}")
|
||||
|
||||
def test_touch_alone_needs_no_pointer_hands(self):
|
||||
conf, got = self.apply({}, "touch", True)
|
||||
self.assertEqual(conf, {"HANDS_NEAR": "on"})
|
||||
conf, got = self.apply({"POINTER_HANDS": "1"}, "gaze_clicks", False)
|
||||
conf, got = self.apply(conf, "ray", False)
|
||||
self.assertEqual(conf["POINTER_HANDS"], "0")
|
||||
self.assertEqual(conf["HANDS_NEAR"], "on", "touch stays on without POINTER_HANDS")
|
||||
self.assertTrue(got["touch"])
|
||||
|
||||
def test_turning_one_on_pins_the_others(self):
|
||||
# all off, then the ray: POINTER_GAZE_PINCH's default (1) and HANDS_NEAR's (1) would turn the
|
||||
# others on with POINTER_HANDS, so they're written off
|
||||
conf, got = self.apply({}, "ray", True)
|
||||
self.assertEqual(got, {"gaze_clicks": False, "ray": True, "touch": False})
|
||||
self.assertEqual(conf, {"POINTER_HANDS": "1", "POINTER_GAZE_PINCH": "0", "HANDS_NEAR": "0"})
|
||||
conf, got = self.apply(conf, "gaze_clicks", True)
|
||||
self.assertEqual(got, {"gaze_clicks": True, "ray": True, "touch": False})
|
||||
|
||||
def test_minimal_writes(self):
|
||||
conf = {"POINTER_HANDS": "1", "HANDS_NEAR": "1"}
|
||||
self.assertEqual(hm.method_writes(conf, "ray", True), {}, "already on: nothing")
|
||||
self.assertEqual(hm.method_writes(conf, "ray", False), {"POINTER_HAND_RAY": "0"})
|
||||
self.assertEqual(hm.method_writes(conf, "touch", False), {"HANDS_NEAR": "0"})
|
||||
self.assertEqual(hm.method_writes({"HANDS_NEAR": "force"}, "touch", True), {}, "force stays force")
|
||||
|
||||
def test_every_combination(self):
|
||||
start = {"POINTER_HANDS": "1", "HANDS_NEAR": "1"}
|
||||
for bits in range(8):
|
||||
want = {m: bool(bits >> i & 1) for i, m in enumerate(hm.METHODS)}
|
||||
conf = dict(start)
|
||||
for m in hm.METHODS:
|
||||
conf, _ = self.apply(conf, m, want[m])
|
||||
self.assertEqual(hm.methods_of(conf), want, f"{want}: {conf}")
|
||||
|
||||
|
||||
class HostTest(unittest.TestCase):
|
||||
def test_units_parses_systemctl_show(self):
|
||||
h = hm.Host()
|
||||
out = ("Id=steamvr.service\nLoadState=loaded\nActiveState=active\n\n"
|
||||
"Id=frametop-camd.service\nLoadState=not-found\nActiveState=inactive\n")
|
||||
h.run = lambda cmd, timeout=20: hm.subprocess.CompletedProcess(cmd, 0, out, "")
|
||||
got = h.units((hm.STEAMVR, CAMD, VIEW))
|
||||
self.assertEqual(got[hm.STEAMVR], {"load": "loaded", "active": "active"})
|
||||
self.assertEqual(got[CAMD]["load"], "not-found")
|
||||
self.assertEqual(got[VIEW], {"load": "not-found", "active": "inactive"}) # no answer for it
|
||||
|
||||
def test_exec_path(self):
|
||||
h = hm.Host()
|
||||
out = "{ path=/x/hands/build/ft-camd ; argv[]=/x/hands/build/ft-camd --status 60 ; ignore_errors=no }\n"
|
||||
h.run = lambda cmd, timeout=20: hm.subprocess.CompletedProcess(cmd, 0, out, "")
|
||||
self.assertEqual(h.exec_path(CAMD), "/x/hands/build/ft-camd")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,232 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Why did a hand go missing? Reads an ft-handreplay --timeline written with the tracker's trace
|
||||
(and best with --oracle N) and, per side (the tracker's left/right call), finds every
|
||||
gap of 0.3 s or more between processed sets with a hand there, then says what the tracker was doing.
|
||||
|
||||
tools/gap_report.py TIMELINE [--min 0.3] [--list]
|
||||
|
||||
Per gap:
|
||||
- start: what happened to the side's last hand at the first set without it (its views lost, a
|
||||
handoff miss, a split, the misread guard, or the hand crossed to the other side);
|
||||
- time held (the hand kept for 0.3 s, its crops retried where it was), time with the search blocked
|
||||
(two hand ids tracked), time searching, and time waiting between searches;
|
||||
- the oracle (a full search every N sets): in how many of its samples the side's hand was findable,
|
||||
and whether any tile it was found in was searched during the gap; and how late the hand came back:
|
||||
from the oracle's first sample finding it (through to the gap's end) to the end, for gaps where it
|
||||
was out of the full search's reach for a while (back in view) and for the rest;
|
||||
- palms the search found on the way that the landmark model rejected, and how the hand came back.
|
||||
|
||||
Causes, one per gap, by where most of the missing time went:
|
||||
findable-not-searched the oracle found it, but the search never tried a tile it was in
|
||||
searched-missed a tile that holds it was searched, and the palm detector missed it then
|
||||
palm-rejected the search found its palm, but the landmark model said no hand (< 0.5)
|
||||
search-blocked the search didn't run: two hand ids were tracked
|
||||
not-findable the oracle didn't find it either (out of view, or no model sees it)
|
||||
held back within the hold (rare at 0.3 s)
|
||||
label-flip not lost: both hands were published, this one called the other side
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
def parse(path):
|
||||
sets = collections.OrderedDict() # ts -> dict(hands=[...], ev=[...])
|
||||
oracle = collections.defaultdict(list)
|
||||
breaks = [] # a pause in the recording (the hand recorder's takes stop between prompts) ends here
|
||||
for line in open(path):
|
||||
parts = line.split()
|
||||
if len(parts) < 2:
|
||||
continue
|
||||
try:
|
||||
ts = float(parts[0])
|
||||
except ValueError:
|
||||
continue
|
||||
rest = parts[1:]
|
||||
if rest[0] == 'ev':
|
||||
sets.setdefault(ts, {'hands': [], 'ev': [], 'proc': False})['ev'].append(' '.join(rest[1:]))
|
||||
elif rest[0] == 'oracle':
|
||||
# oracle CAM tile N presence P right R wrist X Y Z roi ...
|
||||
oracle[ts].append({'cam': rest[1], 'tile': int(rest[3]), 'presence': float(rest[5]),
|
||||
'x': float(rest[9]), 'right': float(rest[7])})
|
||||
elif rest[0] == 'break':
|
||||
breaks.append(ts)
|
||||
elif rest[0] == 'oracle-done':
|
||||
oracle.setdefault(ts, [])
|
||||
elif rest[0] == '-':
|
||||
sets.setdefault(ts, {'hands': [], 'ev': [], 'proc': False})['proc'] = True
|
||||
elif re.fullmatch(r'\d+', rest[0]) and len(rest) >= 6 and rest[1] in ('L', 'R'):
|
||||
d = sets.setdefault(ts, {'hands': [], 'ev': [], 'proc': False})
|
||||
d['proc'] = True
|
||||
# side: the tracker's left/right call when the line has it (hand L|R), else by the wrist
|
||||
side = rest[7] if len(rest) >= 8 and rest[6] == 'hand' else rest[1]
|
||||
d['hands'].append({'id': int(rest[0]), 'side': side, 'views': int(rest[2]), 'x': float(rest[3])})
|
||||
procs = [(ts, d) for ts, d in sets.items() if d['proc']]
|
||||
return procs, oracle, breaks
|
||||
|
||||
|
||||
def report(path, min_gap=0.3, listing=False):
|
||||
procs, oracle, breaks = parse(path)
|
||||
otimes = sorted(oracle)
|
||||
total_time = procs[-1][0] - procs[0][0] if procs else 0
|
||||
out = {}
|
||||
for side in 'LR':
|
||||
gaps = []
|
||||
last = None # (index, hand id)
|
||||
for i, (ts, d) in enumerate(procs):
|
||||
here = [h for h in d['hands'] if h['side'] == side]
|
||||
if not here:
|
||||
continue
|
||||
if last is not None and any(procs[last[0]][0] < b <= ts for b in breaks):
|
||||
last = None
|
||||
if last is not None and ts - procs[last[0]][0] >= min_gap:
|
||||
gaps.append((last[0], i, last[1], here[0]['id']))
|
||||
last = (i, here[0]['id'])
|
||||
rows = []
|
||||
for a, b, hid, back_id in gaps:
|
||||
t0, t1 = procs[a][0], procs[b][0]
|
||||
nxt = procs[a + 1][1]
|
||||
ev0 = [e for e in nxt['ev'] if re.search(r'\b%d\b' % hid, e)]
|
||||
crossed = any(h['id'] == hid for h in nxt['hands'])
|
||||
start = 'crossed' if crossed else ','.join(sorted(set(e.split()[0] for e in ev0
|
||||
if e.split()[0] in ('lost', 'miss', 'split', 'misread', 'dup', 'over-budget')))) or '?'
|
||||
# state per processed interval in the gap
|
||||
forgotten = crossed
|
||||
t = collections.Counter()
|
||||
searched_tiles = set()
|
||||
rejected = kept = 0
|
||||
blocked_ids = set()
|
||||
for k in range(a + 1, b + 1):
|
||||
ts, d = procs[k]
|
||||
dt = ts - procs[k - 1][0]
|
||||
evs = d['ev']
|
||||
if any(e.startswith('forget %d ' % hid) for e in evs):
|
||||
forgotten = True
|
||||
srch = [e for e in evs if e.startswith('search ')]
|
||||
blk = [e for e in evs if e.startswith('search-blocked')]
|
||||
for e in srch:
|
||||
for m in re.finditer(r'(\S+)#(\d+)', e):
|
||||
searched_tiles.add((m.group(1), int(m.group(2))))
|
||||
for e in evs:
|
||||
if e.startswith('palm '):
|
||||
if e.endswith('rejected'):
|
||||
rejected += 1
|
||||
else:
|
||||
kept += 1
|
||||
for e in blk:
|
||||
blocked_ids.update(e.split()[2:])
|
||||
if len(d['hands']) >= 2 and all(h['side'] != side for h in d['hands']):
|
||||
t['flip'] += dt # two hands published, neither called this side: a labelling error
|
||||
elif not forgotten:
|
||||
t['held'] += dt
|
||||
elif blk:
|
||||
t['blocked'] += dt
|
||||
elif srch:
|
||||
t['search'] += dt
|
||||
else:
|
||||
t['wait'] += dt
|
||||
# oracle samples inside the gap (after the hold)
|
||||
osamp = [o for o in otimes if t0 < o < t1]
|
||||
found = [o for o in osamp if any((h['right'] > 0.5) == (side == 'R') for h in oracle[o])]
|
||||
otiles = set()
|
||||
for o in found:
|
||||
for h in oracle[o]:
|
||||
if (h['right'] > 0.5) == (side == 'R'):
|
||||
otiles.add((h['cam'], h['tile']))
|
||||
frac = len(found) / len(osamp) if osamp else None
|
||||
# how late it came back: from the first of the oracle samples that found it through to the end
|
||||
# of the gap (none of them missing it) to the end; reentry: the gap had samples not finding it
|
||||
late = None
|
||||
fset = set(found)
|
||||
for o in reversed(osamp):
|
||||
if o not in fset:
|
||||
break
|
||||
late = t1 - o
|
||||
end_ev = procs[b][1]['ev']
|
||||
how = 'search' if any(e.startswith('palm ') and e.endswith('kept') for e in end_ev) else \
|
||||
'split-off' if any('split off' in e for e in end_ev) else \
|
||||
'crossed-back' if any(h['id'] == hid for h in procs[b][1]['hands']) else 'handover/other'
|
||||
dur = t1 - t0
|
||||
if t['flip'] >= 0.5 * dur:
|
||||
cause = 'label-flip'
|
||||
elif t['held'] >= 0.8 * dur:
|
||||
cause = 'held'
|
||||
elif t['blocked'] >= 0.5 * dur:
|
||||
cause = 'search-blocked'
|
||||
elif frac is not None and frac < 0.3:
|
||||
cause = 'not-findable'
|
||||
elif rejected >= 2:
|
||||
cause = 'palm-rejected'
|
||||
elif otiles and not (otiles & searched_tiles):
|
||||
cause = 'findable-not-searched'
|
||||
elif otiles & searched_tiles:
|
||||
cause = 'searched-missed'
|
||||
else:
|
||||
cause = 'not-findable' if frac is not None else 'no-oracle'
|
||||
rows.append(dict(t0=t0, t1=t1, dur=dur, hid=hid, back=back_id, start=start, cause=cause, t=t,
|
||||
frac=frac, rejected=rejected, kept=kept, how=how, ntiles=len(searched_tiles),
|
||||
otiles=len(otiles), hit=len(otiles & searched_tiles), blocked=blocked_ids,
|
||||
late=late, reentry=len(found) < len(osamp)))
|
||||
out[side] = rows
|
||||
print(f'{path}: {total_time:.0f} s processed')
|
||||
# label-free: per oracle sample, hands the full search found (distinct left/right calls) against
|
||||
# hands the tracker published at the last processed set
|
||||
import bisect
|
||||
pts = [p[0] for p in procs]
|
||||
oh = om = 0
|
||||
for o in otimes:
|
||||
i = bisect.bisect_right(pts, o + 1e-6) - 1
|
||||
if i < 0:
|
||||
continue
|
||||
n = len(set(h['right'] > 0.5 for h in oracle[o]))
|
||||
oh += n
|
||||
om += max(0, n - min(2, len(procs[i][1]['hands'])))
|
||||
if oh:
|
||||
print(f' oracle: {oh} hands findable, tracker had {om} fewer ({100 * om / oh:.1f}% missed)')
|
||||
for side in 'LR':
|
||||
rows = out[side]
|
||||
tot = sum(r['dur'] for r in rows)
|
||||
longest = max([r['dur'] for r in rows], default=0)
|
||||
print(f' {side}: {len(rows)} gaps >= {min_gap} s, {tot:.1f} s missing, longest {longest:.1f} s')
|
||||
by = collections.Counter()
|
||||
n = collections.Counter()
|
||||
for r in rows:
|
||||
by[r['cause']] += r['dur']
|
||||
n[r['cause']] += 1
|
||||
for c, s in by.most_common():
|
||||
print(f' {c:22s} {n[c]:3d} gaps {s:6.1f} s ({100 * s / max(tot, 1e-9):.0f}%)')
|
||||
for what, sel in (('back in view', True), ('in view all along', False)):
|
||||
lates = sorted(r['late'] for r in rows if r['late'] is not None and r['reentry'] == sel)
|
||||
if lates:
|
||||
q = lambda f: lates[min(len(lates) - 1, int(f * len(lates)))]
|
||||
print(f' {what}: {len(lates)} gaps, found again after it could be: median {q(0.5):.2f} s, '
|
||||
f'90% {q(0.9):.2f} s, total {sum(lates):.1f} s')
|
||||
st = collections.Counter()
|
||||
for r in rows:
|
||||
st[r['start']] += 1
|
||||
tt = collections.Counter()
|
||||
for r in rows:
|
||||
tt.update(r['t'])
|
||||
print(' started by: ' + ', '.join(f'{k} {v}' for k, v in st.most_common()) +
|
||||
'; time labelled as the other side %.1f, held %.1f, blocked %.1f, searching %.1f, waiting %.1f s' %
|
||||
(tt['flip'], tt['held'], tt['blocked'], tt['search'], tt['wait']))
|
||||
if listing:
|
||||
for r in rows:
|
||||
fr = '-' if r['frac'] is None else f"{100 * r['frac']:.0f}%"
|
||||
print(f" {r['t0']:7.2f}-{r['t1']:7.2f} {r['dur']:4.1f}s id {r['hid']}->{r['back']} start {r['start']:12s} "
|
||||
f"{r['cause']:22s} flip {r['t']['flip']:.1f} held {r['t']['held']:.1f} blk {r['t']['blocked']:.1f} srch {r['t']['search']:.1f} "
|
||||
f"wait {r['t']['wait']:.1f} oracle {fr} tiles searched {r['ntiles']} oracle-tiles {r['otiles']} hit {r['hit']} "
|
||||
f"palms rejected {r['rejected']} kept {r['kept']} back by {r['how']}"
|
||||
+ (f" blocked by {sorted(r['blocked'])}" if r['blocked'] else ''))
|
||||
return out
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('timeline', nargs='+')
|
||||
ap.add_argument('--min', type=float, default=0.3)
|
||||
ap.add_argument('--list', action='store_true')
|
||||
a = ap.parse_args()
|
||||
for p in a.timeline:
|
||||
report(p, a.min, a.list)
|
||||
@@ -1,39 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Replays recordings through ft-handreplay with a timeline (trace events + oracle) for
|
||||
# tools/gap_report.py. Runs several at once; meant for the 7i through frame-job:
|
||||
# frame-job -- tools/replay_gaps.sh OUT MODELS "EXTRA ARGS" REC[:FROM:TO]...
|
||||
# REPLAY=BINARY replays with another ft-handreplay build (a baseline).
|
||||
# OUT gets REC.txt (the summary) and REC.tl (the timeline) per recording.
|
||||
set -uo pipefail
|
||||
out=$1 models=$2 extra=$3
|
||||
shift 3
|
||||
here=$(cd "$(dirname "$0")/.." && pwd)
|
||||
mkdir -p "$out"
|
||||
jobs=${JOBS:-4}
|
||||
for spec in "$@"; do
|
||||
IFS=: read -r rec from to <<<"$spec"
|
||||
name=$(basename "$rec")
|
||||
range=()
|
||||
[ -n "${from:-}" ] && range=(--from "$from" --to "$to") && name=$name-$from-$to
|
||||
# a recorder session (calibration.json + device.json beside sets.bin or two levels up): replay
|
||||
# with that headset's calibration
|
||||
devroot=()
|
||||
for c in "$rec" "$rec/../.."; do
|
||||
if [ -f "$c/calibration.json" ] && [ -f "$c/device.json" ]; then
|
||||
d=$out/devroot-$(basename "$(realpath "$c")")
|
||||
mkdir -p "$d/persist"
|
||||
cp "$c/calibration.json" "$d/persist/xrservice.json"
|
||||
cp "$c/device.json" "$d/persist/device_config.json"
|
||||
devroot=(env FRAME_JOB_DEVICE_ROOT="$d")
|
||||
[ "$c" = "$rec" ] || name=$(basename "$(realpath "$c")")-$name
|
||||
break
|
||||
fi
|
||||
done
|
||||
while [ "$(jobs -rp | wc -l)" -ge "$jobs" ]; do wait -n; done
|
||||
# shellcheck disable=SC2086
|
||||
${devroot[@]+"${devroot[@]}"} "${REPLAY:-$here/build/ft-handreplay}" "$rec" --cost --threads ${THREADS:-3} --models "$models" --timeline "$out/$name.tl" \
|
||||
${range[@]+"${range[@]}"} $extra > "$out/$name.txt" 2>&1 &
|
||||
done
|
||||
wait
|
||||
for f in "$out"/*.txt; do echo "== $f"; grep -E "processed|hands per|gaps|within 8|re-acquire|views lost|model calls|oracle,|full search|jitter" "$f"; done
|
||||
python3 "$here/tools/gap_report.py" "$out"/*.tl > "$out/gaps.txt" 2>&1 || true
|
||||
@@ -1,38 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Replays recordings through ft-handreplay with --tips (each hand's thumb, index and middle tips and the
|
||||
# pinch detector's measures per processed set, plus its pinch events), for frame-hands'
|
||||
# train/taps/tapeval.py. Runs several at once; meant for a compute host through frame-job:
|
||||
# frame-job -- tools/replay_tips.sh OUT NAME=MODELS "EXTRA ARGS" REC[:FROM:TO]...
|
||||
# OUT gets NAME/<rec>.tips and NAME/<rec>.txt (the summary) per recording. A recorder take
|
||||
# (takes/NN-name, with calibration.json and device.json two levels up) replays with that headset's
|
||||
# calibration and its side cameras as stored (--sides 0: exports are named right).
|
||||
set -uo pipefail
|
||||
out=$1 nm=$2 extra=$3
|
||||
shift 3
|
||||
name=${nm%%=*} models=${nm#*=}
|
||||
here=$(cd "$(dirname "$0")/.." && pwd)
|
||||
mkdir -p "$out/$name"
|
||||
jobs=${JOBS:-4}
|
||||
for spec in "$@"; do
|
||||
IFS=: read -r rec from to <<<"$spec"
|
||||
base=$(basename "$rec")
|
||||
range=()
|
||||
[ -n "${from:-}" ] && range=(--from "$from" --to "$to") && base=$base-$from-$to
|
||||
devroot=() sides=()
|
||||
if [ -f "$rec/../../calibration.json" ] && [ -f "$rec/../../device.json" ]; then
|
||||
s=$(realpath "$rec/../..")
|
||||
d=$out/devroot-$(basename "$(dirname "$s")" | cut -c1-8)-$(basename "$s")
|
||||
mkdir -p "$d/persist"
|
||||
cp "$s/calibration.json" "$d/persist/xrservice.json"
|
||||
cp "$s/device.json" "$d/persist/device_config.json"
|
||||
devroot=(env FRAME_JOB_DEVICE_ROOT="$d")
|
||||
sides=(--sides 0)
|
||||
base=$(basename "$(dirname "$s")" | cut -c1-8)-$base
|
||||
fi
|
||||
while [ "$(jobs -rp | wc -l)" -ge "$jobs" ]; do wait -n; done
|
||||
# shellcheck disable=SC2086
|
||||
${devroot[@]+"${devroot[@]}"} "$here/build/ft-handreplay" "$rec" --cost --threads ${THREADS:-3} --models "$models" \
|
||||
--tips "$out/$name/$base.tips" ${sides[@]+"${sides[@]}"} ${range[@]+"${range[@]}"} $extra > "$out/$name/$base.txt" 2>&1 &
|
||||
done
|
||||
wait
|
||||
for f in "$out/$name"/*.txt; do echo "== $f"; grep -E "processed|pinches" "$f"; done
|
||||
@@ -1,15 +1,14 @@
|
||||
"""Watch the gestures ft-hands publishes, live: pinches, grips and middle pinches, begins, ends, and drags.
|
||||
"""Watch the gestures ft-hands publishes, live: pinches and grips, begins, ends, and drags.
|
||||
|
||||
usage: python3 tools/watch_gestures.py [--every S] [--distance]
|
||||
|
||||
Prints a line when a pinch, a grip (a closed hand) or a middle pinch (thumb to middle finger)
|
||||
begins or ends on either hand. It goes
|
||||
Prints a line when a pinch or a grip (a closed hand) begins or ends on either hand. It goes
|
||||
by the counters, so a quick tap between two reads still shows. While one is held, every
|
||||
--every seconds (default 0.1) it prints how far its point has moved since it began, in the
|
||||
head frame (turning your head moves it too; a real consumer turns both points into the room
|
||||
first, see include/fh_gestures.h). --distance also prints each hand's thumb-to-index
|
||||
distance and finger curl, to see how close a gesture comes to the thresholds.
|
||||
Version 1 files (pinches only) and version 2 files (no middle pinches) still work.
|
||||
Version 1 files (pinches only) still work.
|
||||
"""
|
||||
import argparse
|
||||
import mmap
|
||||
@@ -21,7 +20,7 @@ HDR = struct.Struct('<8sIIQQQffff8x') # 64 bytes
|
||||
SLOT = struct.Struct('<IIIIQQff3f3f') # 64 bytes
|
||||
TRACKED, DOWN, LOST = 1, 2, 4
|
||||
SIDES = ('left ', 'right')
|
||||
KINDS = ('pinch', 'grip ', 'middle pinch')
|
||||
KINDS = ('pinch', 'grip ')
|
||||
|
||||
|
||||
def path():
|
||||
@@ -29,20 +28,15 @@ def path():
|
||||
|
||||
|
||||
def read(m):
|
||||
"""(header, [[pinch left, right], [grip left, right], [middle pinch left, right]]) under the
|
||||
sequence lock, or None if it's being written. Version 1 has no grips and version 2 no middle
|
||||
pinches: they read as all zero."""
|
||||
"""(header, [[pinch left, right], [grip left, right]]) under the sequence lock, or None
|
||||
if it's being written. Version 1 has no grips: they read as all zero."""
|
||||
for _ in range(10):
|
||||
s1 = struct.unpack_from('<Q', m, 16)[0]
|
||||
if s1 % 2 == 0:
|
||||
h = HDR.unpack_from(m, 0)
|
||||
kinds = 1
|
||||
if h[1] >= 2 and len(m) >= HDR.size + 4 * SLOT.size:
|
||||
kinds = 2
|
||||
if h[1] >= 3 and len(m) >= HDR.size + 6 * SLOT.size:
|
||||
kinds = 3
|
||||
kinds = 2 if h[1] >= 2 and len(m) >= HDR.size + 4 * SLOT.size else 1
|
||||
g = [[SLOT.unpack_from(m, HDR.size + (k * 2 + s) * SLOT.size) for s in range(2)] for k in range(kinds)]
|
||||
while len(g) < 3:
|
||||
if kinds == 1:
|
||||
g.append([(0,) * 14, (0,) * 14])
|
||||
if struct.unpack_from('<Q', m, 16)[0] == s1:
|
||||
return h, g
|
||||
|
||||
@@ -1,371 +0,0 @@
|
||||
// Flagging tracking failures: see flagbuf.h.
|
||||
#include "flagbuf.h"
|
||||
|
||||
#include "pinch.h"
|
||||
|
||||
#include <pthread.h>
|
||||
#include <sched.h>
|
||||
#include <sys/resource.h>
|
||||
#include <sys/stat.h>
|
||||
#include <sys/syscall.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
#include <cstdarg>
|
||||
#include <cstdio>
|
||||
#include <ctime>
|
||||
|
||||
namespace {
|
||||
|
||||
void appendf(std::string &s, const char *fmt, ...) __attribute__((format(printf, 2, 3)));
|
||||
void appendf(std::string &s, const char *fmt, ...) {
|
||||
char buf[512];
|
||||
va_list ap;
|
||||
va_start(ap, fmt);
|
||||
const int n = std::vsnprintf(buf, sizeof buf, fmt, ap);
|
||||
va_end(ap);
|
||||
if (n > 0) s.append(buf, size_t(std::min(n, int(sizeof buf) - 1)));
|
||||
}
|
||||
|
||||
void points_json(std::string &s, const V3 *p, int n) {
|
||||
s += '[';
|
||||
for (int i = 0; i < n; ++i) appendf(s, "%s[%.4f, %.4f, %.4f]", i ? ", " : "", p[i][0], p[i][1], p[i][2]);
|
||||
s += ']';
|
||||
}
|
||||
|
||||
double wall_now() {
|
||||
timespec ts;
|
||||
clock_gettime(CLOCK_REALTIME, &ts);
|
||||
return ts.tv_sec + ts.tv_nsec / 1e9;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string json_string(const std::string &s, bool null_if_empty) {
|
||||
if (null_if_empty && s.empty()) return "null";
|
||||
std::string o = "\"";
|
||||
for (unsigned char c : s) {
|
||||
if (c == '"' || c == '\\') o += '\\', o += char(c);
|
||||
else if (c == '\n') o += "\\n";
|
||||
else if (c == '\t') o += "\\t";
|
||||
else if (c < 0x20) appendf(o, "\\u%04x", c);
|
||||
else o += char(c);
|
||||
}
|
||||
return o + "\"";
|
||||
}
|
||||
|
||||
CropGeom crop_square(const Image &img, V2 center, double side, int max_px, uint8_t *out) {
|
||||
CropGeom g;
|
||||
side = std::max(side, 1.0);
|
||||
const int n = std::clamp(int(std::ceil(side - 1e-6)), 1, std::max(1, max_px));
|
||||
g.w = g.h = n;
|
||||
g.scale = side / n;
|
||||
g.x0 = center[0] - side / 2 + 0.5;
|
||||
g.y0 = center[1] - side / 2 + 0.5;
|
||||
// the taps: per output column and row, the image column and row (or -1 outside)
|
||||
const bool area = g.scale > 1.25;
|
||||
const int taps = area ? 2 : 1;
|
||||
const double off[2] = {area ? -g.scale / 4 : 0, area ? g.scale / 4 : 0};
|
||||
thread_local std::vector<int> xs, ys;
|
||||
xs.resize(size_t(n) * taps), ys.resize(size_t(n) * taps);
|
||||
for (int u = 0; u < n; ++u)
|
||||
for (int t = 0; t < taps; ++t) {
|
||||
const double c = (u + 0.5) * g.scale - 0.5 + off[t];
|
||||
const long x = std::lround(g.x0 + c), y = std::lround(g.y0 + c);
|
||||
xs[size_t(u) * taps + t] = x >= 0 && x < img.width ? int(x) : -1;
|
||||
ys[size_t(u) * taps + t] = y >= 0 && y < img.height ? int(y) : -1;
|
||||
}
|
||||
for (int v = 0; v < n; ++v) {
|
||||
uint8_t *o = out + size_t(v) * n;
|
||||
if (!area) {
|
||||
const int y = ys[size_t(v)];
|
||||
if (y < 0) {
|
||||
std::fill(o, o + n, 0);
|
||||
continue;
|
||||
}
|
||||
const uint8_t *row = img.data + size_t(y) * img.stride;
|
||||
for (int u = 0; u < n; ++u) o[u] = xs[size_t(u)] < 0 ? 0 : row[xs[size_t(u)]];
|
||||
continue;
|
||||
}
|
||||
const int y0 = ys[size_t(v) * 2], y1 = ys[size_t(v) * 2 + 1];
|
||||
const uint8_t *r0 = y0 < 0 ? nullptr : img.data + size_t(y0) * img.stride;
|
||||
const uint8_t *r1 = y1 < 0 ? nullptr : img.data + size_t(y1) * img.stride;
|
||||
for (int u = 0; u < n; ++u) {
|
||||
const int x0 = xs[size_t(u) * 2], x1 = xs[size_t(u) * 2 + 1];
|
||||
unsigned sum = 0;
|
||||
if (r0) sum += (x0 < 0 ? 0 : r0[x0]) + (x1 < 0 ? 0 : r0[x1]);
|
||||
if (r1) sum += (x0 < 0 ? 0 : r1[x0]) + (x1 < 0 ? 0 : r1[x1]);
|
||||
o[u] = uint8_t((sum + 2) / 4);
|
||||
}
|
||||
}
|
||||
return g;
|
||||
}
|
||||
|
||||
FlagBuffer::FlagBuffer(const FlagConfig &cfg) : cfg_(cfg) {
|
||||
cfg_.crop_px = std::clamp(cfg_.crop_px, 16, 1024);
|
||||
slot_bytes_ = size_t(cfg_.crop_px) * cfg_.crop_px;
|
||||
nslots_ = int(std::max<size_t>(1, cfg_.max_bytes / slot_bytes_));
|
||||
stats_.slots = size_t(nslots_);
|
||||
}
|
||||
|
||||
FlagBuffer::~FlagBuffer() { wait(); }
|
||||
|
||||
void FlagBuffer::wait() {
|
||||
if (writer_.joinable()) writer_.join();
|
||||
}
|
||||
|
||||
int FlagBuffer::take_slot() {
|
||||
std::lock_guard<std::mutex> l(mu_);
|
||||
if (!free_.empty()) {
|
||||
const int s = free_.back();
|
||||
free_.pop_back();
|
||||
return s;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
void FlagBuffer::give_back(const std::vector<Crop> &crops) {
|
||||
std::lock_guard<std::mutex> l(mu_);
|
||||
for (const Crop &c : crops)
|
||||
if (c.slot >= 0) free_.push_back(c.slot);
|
||||
}
|
||||
|
||||
void FlagBuffer::evict_front() {
|
||||
give_back(sets_.front().crops);
|
||||
sets_.pop_front();
|
||||
}
|
||||
|
||||
void FlagBuffer::add(int64_t t_raw, int64_t t_mono, int rec_set, const std::map<std::string, Image> &images,
|
||||
const std::vector<Seen> &views, const std::string &hands_json) {
|
||||
if (!arena_) {
|
||||
arena_.reset(new uint8_t[size_t(nslots_) * slot_bytes_]); // not zeroed: untouched pages stay unmapped
|
||||
std::lock_guard<std::mutex> l(mu_);
|
||||
free_.clear();
|
||||
for (int s = nslots_ - 1; s >= 0; --s) free_.push_back(s);
|
||||
}
|
||||
const int64_t keep_ns = int64_t(cfg_.before_s * 1e9);
|
||||
while (!sets_.empty() && !sets_.front().held && sets_.front().t_raw < t_raw - keep_ns) evict_front();
|
||||
|
||||
Set set;
|
||||
set.t_raw = t_raw, set.t_mono = t_mono, set.rec_set = rec_set, set.out = hands_json;
|
||||
set.held = pending_ && t_raw >= flag_t_ - keep_ns;
|
||||
auto crop = [&](const std::string &cam, int hand, const Roi &roi, bool ghost) -> int {
|
||||
const auto im = images.find(cam);
|
||||
if (im == images.end() || !im->second.data) return -1;
|
||||
int slot = take_slot();
|
||||
while (slot < 0 && !sets_.empty() && !sets_.front().held) evict_front(), slot = take_slot();
|
||||
if (slot < 0) return ++stats_.dropped, -1;
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
Crop c;
|
||||
c.cam = cam, c.hand = hand, c.ghost = ghost, c.slot = slot;
|
||||
c.g = crop_square(im->second, roi.center, cfg_.scale * roi.size, cfg_.crop_px, slot_ptr(slot));
|
||||
stats_.crop_us += std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - t0).count();
|
||||
++stats_.crop_calls, ++stats_.crops;
|
||||
stats_.ghosts += ghost;
|
||||
set.crops.push_back(std::move(c));
|
||||
return int(set.crops.size()) - 1;
|
||||
};
|
||||
for (const Seen &s : views) {
|
||||
View v;
|
||||
v.cam = s.cam, v.hand = s.hand;
|
||||
v.roi[0] = s.roi.center[0], v.roi[1] = s.roi.center[1], v.roi[2] = s.roi.size, v.roi[3] = s.roi.rotation;
|
||||
v.presence = s.lm.presence, v.right = s.lm.right;
|
||||
for (int k = 0; k < 21; ++k) v.lm[2 * k] = float(s.lm.pts[k][0]), v.lm[2 * k + 1] = float(s.lm.pts[k][1]);
|
||||
v.crop = crop(s.cam, s.hand, s.roi, false);
|
||||
set.views.push_back(v);
|
||||
last_[{s.cam, s.hand}] = {s.roi, t_raw};
|
||||
}
|
||||
// ghosts: views lost within ghost_s, cropped where they last were
|
||||
const int64_t ghost_ns = int64_t(cfg_.ghost_s * 1e9);
|
||||
int ghosts = 0;
|
||||
for (auto it = last_.begin(); it != last_.end();) {
|
||||
if (t_raw - it->second.t > ghost_ns) {
|
||||
it = last_.erase(it);
|
||||
continue;
|
||||
}
|
||||
if (it->second.t != t_raw && it->first.second != 0 && ghosts < cfg_.max_ghosts &&
|
||||
crop(it->first.first, it->first.second, it->second.roi, true) >= 0)
|
||||
++ghosts;
|
||||
++it;
|
||||
}
|
||||
sets_.push_back(std::move(set));
|
||||
++stats_.sets;
|
||||
if (pending_ && t_raw >= flag_t_ + int64_t(cfg_.after_s * 1e9)) complete();
|
||||
}
|
||||
|
||||
bool FlagBuffer::flag(const FlagRequest &req, int64_t t_raw, const std::string &extra_note) {
|
||||
if (pending_) {
|
||||
if (!extra_note.empty()) extra_notes_ += (extra_notes_.empty() ? "" : ", ") + json_string(extra_note);
|
||||
return false;
|
||||
}
|
||||
if (req.dir.empty()) return false;
|
||||
pending_ = true, req_ = req, flag_t_ = t_raw, extra_notes_.clear();
|
||||
dropped_at_flag_ = stats_.dropped;
|
||||
const int64_t keep_ns = int64_t(cfg_.before_s * 1e9);
|
||||
for (Set &s : sets_)
|
||||
if (s.t_raw >= flag_t_ - keep_ns) s.held = true;
|
||||
return true;
|
||||
}
|
||||
|
||||
void FlagBuffer::tick(int64_t t_raw_now) {
|
||||
if (pending_ && t_raw_now >= flag_t_ + int64_t((cfg_.after_s + 0.5) * 1e9)) complete();
|
||||
}
|
||||
|
||||
void FlagBuffer::finish() {
|
||||
if (pending_) complete();
|
||||
}
|
||||
|
||||
void FlagBuffer::complete() {
|
||||
Dump d;
|
||||
d.req = req_, d.flag_t = flag_t_, d.first_unix = wall_now();
|
||||
d.dropped = stats_.dropped - dropped_at_flag_;
|
||||
if (!extra_notes_.empty()) d.req.fields += std::string(d.req.fields.empty() ? "" : ", ") + "\"more_notes\": [" + extra_notes_ + "]";
|
||||
auto first = std::find_if(sets_.begin(), sets_.end(), [](const Set &s) { return s.held; });
|
||||
for (auto it = first; it != sets_.end(); ++it) d.sets.push_back(std::move(*it));
|
||||
sets_.erase(first, sets_.end());
|
||||
pending_ = false;
|
||||
wait(); // one writer at a time (a flag takes 14 s; writing one well under that)
|
||||
writer_ = std::thread(&FlagBuffer::write, this, std::move(d));
|
||||
}
|
||||
|
||||
void FlagBuffer::write(Dump d) {
|
||||
// off the tracking cores, at the lowest priority: it's only files
|
||||
if (!cfg_.writer_cpus.empty()) {
|
||||
cpu_set_t set;
|
||||
CPU_ZERO(&set);
|
||||
for (int c : cfg_.writer_cpus) CPU_SET(c, &set);
|
||||
pthread_setaffinity_np(pthread_self(), sizeof set, &set);
|
||||
}
|
||||
setpriority(PRIO_PROCESS, pid_t(syscall(SYS_gettid)), 19);
|
||||
std::string err;
|
||||
const std::string &dir = d.req.dir;
|
||||
mkdir(dir.c_str(), 0700);
|
||||
struct stat st;
|
||||
if (stat(dir.c_str(), &st) != 0 || !S_ISDIR(st.st_mode) || st.st_uid != getuid()) err = "can't use " + dir;
|
||||
FILE *bin = err.empty() ? std::fopen((dir + "/crops.bin").c_str(), "wb") : nullptr;
|
||||
FILE *cj = err.empty() ? std::fopen((dir + "/crops.jsonl").c_str(), "w") : nullptr;
|
||||
FILE *sj = err.empty() ? std::fopen((dir + "/sets.jsonl").c_str(), "w") : nullptr;
|
||||
if (err.empty() && (!bin || !cj || !sj)) err = "can't write in " + dir;
|
||||
size_t k = 0, offset = 0, ghosts = 0;
|
||||
std::string line;
|
||||
for (size_t i = 0; i < d.sets.size(); ++i) {
|
||||
Set &s = d.sets[i];
|
||||
if (err.empty()) {
|
||||
std::vector<size_t> kk(s.crops.size());
|
||||
for (size_t j = 0; j < s.crops.size(); ++j) {
|
||||
const Crop &c = s.crops[j];
|
||||
const size_t bytes = size_t(c.g.w) * c.g.h;
|
||||
if (std::fwrite(slot_ptr(c.slot), 1, bytes, bin) != bytes) err = "write failed in " + dir;
|
||||
line.clear();
|
||||
appendf(line, "{\"k\": %zu, \"set\": %zu, \"cam\": %s, \"hand\": %d, \"ghost\": %s, \"x0\": %.3f, \"y0\": %.3f, "
|
||||
"\"scale\": %.5f, \"w\": %d, \"h\": %d, \"offset\": %zu, \"bytes\": %zu}\n",
|
||||
k, i, json_string(c.cam).c_str(), c.hand, c.ghost ? "true" : "false", c.g.x0, c.g.y0, c.g.scale, c.g.w,
|
||||
c.g.h, offset, bytes);
|
||||
std::fputs(line.c_str(), cj);
|
||||
kk[j] = k++, offset += bytes, ghosts += c.ghost;
|
||||
}
|
||||
line.clear();
|
||||
appendf(line, "{\"i\": %zu, \"t_raw_ns\": %lld, \"t_ns\": %lld", i, (long long)s.t_raw, (long long)s.t_mono);
|
||||
if (s.rec_set >= 0) appendf(line, ", \"rec_set\": %d", s.rec_set);
|
||||
if (!s.out.empty()) line += ", " + s.out;
|
||||
line += ", \"views\": [";
|
||||
for (size_t j = 0; j < s.views.size(); ++j) {
|
||||
const View &v = s.views[j];
|
||||
appendf(line, "%s{\"cam\": %s, \"hand\": %d, \"roi\": [%.2f, %.2f, %.2f, %.4f], \"presence\": %.3f, \"right\": %.3f, \"lm\": [",
|
||||
j ? ", " : "", json_string(v.cam).c_str(), v.hand, v.roi[0], v.roi[1], v.roi[2], v.roi[3], v.presence, v.right);
|
||||
for (int p = 0; p < 21; ++p) appendf(line, "%s[%.2f, %.2f]", p ? ", " : "", v.lm[2 * p], v.lm[2 * p + 1]);
|
||||
appendf(line, "], \"crop\": %lld}", v.crop >= 0 ? (long long)kk[size_t(v.crop)] : -1LL);
|
||||
}
|
||||
line += "], \"ghosts\": [";
|
||||
bool first = true;
|
||||
for (size_t j = 0; j < s.crops.size(); ++j)
|
||||
if (s.crops[j].ghost) {
|
||||
appendf(line, "%s{\"cam\": %s, \"hand\": %d, \"crop\": %zu}", first ? "" : ", ", json_string(s.crops[j].cam).c_str(),
|
||||
s.crops[j].hand, kk[j]);
|
||||
first = false;
|
||||
}
|
||||
line += "]}\n";
|
||||
std::fputs(line.c_str(), sj);
|
||||
}
|
||||
give_back(s.crops); // as it goes: the buffer gets its slots back
|
||||
s.crops.clear();
|
||||
}
|
||||
for (FILE *f : {bin, cj, sj})
|
||||
if (f && std::fclose(f) != 0 && err.empty()) err = "write failed in " + dir;
|
||||
if (err.empty()) {
|
||||
std::string m = "{\"version\": 1";
|
||||
if (!d.req.fields.empty()) m += ", " + d.req.fields;
|
||||
appendf(m, ", \"flag_t_raw_ns\": %lld, \"window\": {\"before_s\": %.2f, \"after_s\": %.2f, \"first_t_raw_ns\": %lld, "
|
||||
"\"last_t_raw_ns\": %lld}",
|
||||
(long long)d.flag_t, cfg_.before_s, cfg_.after_s, (long long)(d.sets.empty() ? 0 : d.sets.front().t_raw),
|
||||
(long long)(d.sets.empty() ? 0 : d.sets.back().t_raw));
|
||||
appendf(m, ", \"crop\": {\"scale\": %.3f, \"px\": %d, \"ghost_s\": %.2f}", cfg_.scale, cfg_.crop_px, cfg_.ghost_s);
|
||||
m += ", \"settings\": " + settings_json_ + ", \"cameras\": " + cameras_json_;
|
||||
appendf(m, ", \"counts\": {\"sets\": %zu, \"crops\": %zu, \"ghosts\": %zu, \"crops_dropped\": %zu, \"bytes\": %zu}", d.sets.size(), k,
|
||||
ghosts, d.dropped, offset);
|
||||
appendf(m, ", \"written_unix\": %.3f}\n", wall_now());
|
||||
const std::string tmp = dir + "/manifest.json.tmp";
|
||||
FILE *f = std::fopen(tmp.c_str(), "w");
|
||||
const bool ok = f && std::fputs(m.c_str(), f) >= 0;
|
||||
if (!f || std::fclose(f) != 0 || !ok || std::rename(tmp.c_str(), (dir + "/manifest.json").c_str()) != 0)
|
||||
err = "can't write " + dir + "/manifest.json";
|
||||
}
|
||||
std::lock_guard<std::mutex> l(mu_);
|
||||
if (err.empty()) ++stats_.flags, stats_.last_dir = dir, stats_.last_error.clear();
|
||||
else stats_.last_error = err;
|
||||
}
|
||||
|
||||
FlagBuffer::Stats FlagBuffer::stats() const {
|
||||
std::lock_guard<std::mutex> l(mu_);
|
||||
Stats s = stats_;
|
||||
s.slots_used = arena_ ? size_t(nslots_) - free_.size() : 0;
|
||||
return s;
|
||||
}
|
||||
|
||||
size_t FlagBuffer::bytes() const {
|
||||
size_t b = stats().slots_used * slot_bytes_;
|
||||
for (const Set &s : sets_) b += sizeof(Set) + s.out.capacity() + s.views.capacity() * sizeof(View) + s.crops.capacity() * sizeof(Crop);
|
||||
return b;
|
||||
}
|
||||
|
||||
std::string cameras_json(const std::map<std::string, Camera> &cams) {
|
||||
std::string s = "{";
|
||||
for (auto &[name, c] : cams) {
|
||||
appendf(s, "%s%s: {\"width\": %d, \"height\": %d, \"fx\": %.6f, \"fy\": %.6f, \"cx\": %.6f, \"cy\": %.6f, ", s.size() > 1 ? ", " : "",
|
||||
json_string(name).c_str(), c.width, c.height, c.fx, c.fy, c.cx, c.cy);
|
||||
appendf(s, "\"k\": [%.9g, %.9g, %.9g, %.9g], \"R\": [", c.k[0], c.k[1], c.k[2], c.k[3]);
|
||||
for (int r = 0; r < 3; ++r) appendf(s, "%s[%.9f, %.9f, %.9f]", r ? ", " : "", c.R[r][0], c.R[r][1], c.R[r][2]);
|
||||
appendf(s, "], \"origin\": [%.6f, %.6f, %.6f]}", c.origin[0], c.origin[1], c.origin[2]);
|
||||
}
|
||||
return s + "}";
|
||||
}
|
||||
|
||||
std::string output_json(const std::vector<const Hand *> &hands, const Pinch *pinch, const Pinch *middle, const Grip *grip) {
|
||||
std::string s = "\"hands\": [";
|
||||
for (size_t i = 0; i < hands.size(); ++i) {
|
||||
const Hand &h = *hands[i];
|
||||
appendf(s, "%s{\"id\": %d, \"side\": \"%s\", \"right_score\": %.3f, \"nviews\": %d, \"residual\": %.4f, \"scale\": %.3f, "
|
||||
"\"speed\": %.3f, \"fit\": %s, \"pts\": ",
|
||||
i ? ", " : "", h.id, h.right() ? "R" : "L", h.right_score, h.nviews, h.residual, h.scale, h.speed,
|
||||
h.fit.valid ? "true" : "false");
|
||||
points_json(s, h.pts, 21);
|
||||
s += ", \"pub\": ";
|
||||
points_json(s, h.fit.valid ? h.fit.pts : h.smooth, 21);
|
||||
s += '}';
|
||||
}
|
||||
s += ']';
|
||||
if (!pinch && !middle && !grip) return s;
|
||||
s += ", \"gestures\": {";
|
||||
bool first = true;
|
||||
auto sides = [&](const char *name, const fh_pinch_t &l, const fh_pinch_t &r) {
|
||||
appendf(s, "%s\"%s\": [{\"flags\": %u, \"d\": %.4f}, {\"flags\": %u, \"d\": %.4f}]", first ? "" : ", ", name, l.flags, l.distance,
|
||||
r.flags, r.distance);
|
||||
first = false;
|
||||
};
|
||||
if (pinch) sides("pinch", pinch->side(0), pinch->side(1));
|
||||
if (middle) sides("middle", middle->side(0), middle->side(1));
|
||||
if (grip) sides("grip", grip->side(0), grip->side(1));
|
||||
return s + "}";
|
||||
}
|
||||
@@ -1,179 +0,0 @@
|
||||
// Flagging tracking failures (frame-hands notes/flag-failures.md has the plan and the format): a
|
||||
// rolling buffer, in RAM, of what's needed to relabel and diagnose the last few seconds of
|
||||
// tracking, and on a flag (`ft-handsctl flag [note]`, or a voice command through it), the buffer
|
||||
// and the next couple of seconds written to a folder for the offline tools (frame-hands'
|
||||
// train/flags).
|
||||
//
|
||||
// Per processed set it keeps, per view the tracker used, a mono crop around the tracker's
|
||||
// landmark crop: its centre, `scale` times its size, axis aligned (so it covers the rotated crop
|
||||
// with room to spare), downscaled to at most `crop_px`; with the crop's place in the image, so
|
||||
// labels made on it lift back to the full image and triangulate. A view the tracker lost is still
|
||||
// cropped where it was for `ghost_s` (a "ghost"), so the teacher can say whether the hand was there.
|
||||
// And ft-hands' output for the set (its hands and gestures, as JSON) and each view's crop and
|
||||
// landmarks. Nothing is written until a flag.
|
||||
//
|
||||
// The crops live in fixed slots of one arena (`max_bytes`), taken and given back through a free
|
||||
// list; pages are only touched as slots are used. Sets older than `before_s` are let go; when the
|
||||
// slots run out, the oldest set goes, and while a flag is waiting for its last seconds its sets
|
||||
// are held, so a crop that finds no free slot then is dropped (counted). A flag's sets leave the
|
||||
// buffer when it's complete, and a thread of their own writes them (at low priority, off the
|
||||
// tracking cores), giving the slots back as it goes.
|
||||
#pragma once
|
||||
|
||||
#include "calib.h"
|
||||
#include "tracker.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <deque>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
class Pinch;
|
||||
class Grip;
|
||||
|
||||
struct FlagConfig {
|
||||
double before_s = 12; // kept before a flag
|
||||
double after_s = 2; // and taken after it
|
||||
double scale = 1.6; // the crop's side over the tracker's crop's
|
||||
double ghost_s = 1.0; // a lost view is still cropped where it was this long
|
||||
int crop_px = 224; // a crop's side at most (larger ones are downscaled)
|
||||
int max_ghosts = 4; // ghost crops per set at most
|
||||
size_t max_bytes = size_t(80) << 20; // the crops' arena
|
||||
std::vector<int> writer_cpus = {0, 1, 2, 3, 4}; // where the writer runs (empty: anywhere)
|
||||
};
|
||||
|
||||
// One crop's place: image pixel (centres at integers) of crop pixel (u, v) is
|
||||
// (x0 + (u + 0.5) * scale - 0.5, y0 + (v + 0.5) * scale - 0.5). Outside the image is 0.
|
||||
struct CropGeom {
|
||||
double x0 = 0, y0 = 0, scale = 1;
|
||||
int w = 0, h = 0;
|
||||
};
|
||||
|
||||
// The square of side `side` centred on `center`, axis aligned, downscaled to at most max_px a side,
|
||||
// into out (w * h bytes). Downscaled by more than 1.25, each pixel averages a 2 x 2 grid of taps
|
||||
// across its footprint; else the nearest pixel.
|
||||
CropGeom crop_square(const Image &img, V2 center, double side, int max_px, uint8_t *out);
|
||||
|
||||
// A flag's request: where it goes and what the manifest says about it besides the buffer's own
|
||||
// (`fields`: JSON members, "\"key\": value, ..." without braces, may be empty).
|
||||
struct FlagRequest {
|
||||
std::string dir; // exists, or is made
|
||||
std::string fields;
|
||||
};
|
||||
|
||||
class FlagBuffer {
|
||||
public:
|
||||
explicit FlagBuffer(const FlagConfig &cfg = {});
|
||||
~FlagBuffer(); // waits for the writer
|
||||
const FlagConfig &config() const { return cfg_; }
|
||||
|
||||
// The manifest's static parts, set once: the cameras (cameras_json) and ft-hands' settings
|
||||
// (a JSON object).
|
||||
void set_static(const std::string &cameras_json, const std::string &settings_json) {
|
||||
cameras_json_ = cameras_json, settings_json_ = settings_json;
|
||||
}
|
||||
// After each processed set: its capture time (t_raw: the cameras' clock, CLOCK_MONOTONIC_RAW;
|
||||
// t_mono: CLOCK_MONOTONIC), its index in a recording (-1 live), the images the tracker had, its
|
||||
// views this step (Tracker::views_now) and the set's output (hands_json: JSON members
|
||||
// "\"hands\": [...], \"gestures\": {...}", see output_json). Completes a waiting flag once
|
||||
// a set comes after_s past it.
|
||||
void add(int64_t t_raw, int64_t t_mono, int rec_set, const std::map<std::string, Image> &images,
|
||||
const std::vector<Seen> &views, const std::string &hands_json);
|
||||
// A flag at t_raw (the newest set's time, or now on that clock): keeps from before_s before it
|
||||
// and completes after_s after. False if a flag is already waiting (this one's note joins it:
|
||||
// `extra_note`) or the request is bad.
|
||||
bool flag(const FlagRequest &req, int64_t t_raw, const std::string &extra_note = "");
|
||||
// Completes a waiting flag if its time is up by the clock (no sets come while parked): call
|
||||
// now and then with the cameras' clock's now.
|
||||
void tick(int64_t t_raw_now);
|
||||
// Completes a waiting flag with what it has (stopping).
|
||||
void finish();
|
||||
bool waiting() const { return pending_; }
|
||||
// Waits for the writer.
|
||||
void wait();
|
||||
|
||||
struct Stats {
|
||||
size_t sets = 0, crops = 0, ghosts = 0;
|
||||
size_t slots_used = 0, slots = 0;
|
||||
size_t dropped = 0; // crops with no free slot
|
||||
size_t flags = 0; // written
|
||||
double crop_us = 0; // summed crop time, microseconds
|
||||
size_t crop_calls = 0;
|
||||
std::string last_dir, last_error;
|
||||
};
|
||||
Stats stats() const;
|
||||
// RAM the buffer holds now: the slots in use (their pages) and the sets' records.
|
||||
size_t bytes() const;
|
||||
|
||||
private:
|
||||
struct Crop {
|
||||
std::string cam;
|
||||
int hand = 0;
|
||||
bool ghost = false;
|
||||
CropGeom g;
|
||||
int slot = -1;
|
||||
};
|
||||
struct View {
|
||||
std::string cam;
|
||||
int hand = 0;
|
||||
double roi[4] = {};
|
||||
double presence = 0, right = 0;
|
||||
float lm[42] = {};
|
||||
int crop = -1; // into the set's crops
|
||||
};
|
||||
struct Set {
|
||||
int64_t t_raw = 0, t_mono = 0;
|
||||
int rec_set = -1;
|
||||
std::string out;
|
||||
std::vector<View> views;
|
||||
std::vector<Crop> crops;
|
||||
bool held = false;
|
||||
};
|
||||
struct Dump {
|
||||
FlagRequest req;
|
||||
std::vector<Set> sets;
|
||||
int64_t flag_t = 0;
|
||||
double first_unix = 0;
|
||||
size_t dropped = 0;
|
||||
};
|
||||
int take_slot();
|
||||
void give_back(const std::vector<Crop> &crops);
|
||||
void evict_front();
|
||||
void complete();
|
||||
void write(Dump d);
|
||||
uint8_t *slot_ptr(int slot) { return arena_.get() + size_t(slot) * slot_bytes_; }
|
||||
|
||||
FlagConfig cfg_;
|
||||
std::string cameras_json_ = "{}", settings_json_ = "{}";
|
||||
size_t slot_bytes_ = 0;
|
||||
int nslots_ = 0;
|
||||
std::unique_ptr<uint8_t[]> arena_; // made at the first set; untouched pages cost nothing
|
||||
mutable std::mutex mu_; // free_ and the stats the writer changes
|
||||
std::vector<int> free_;
|
||||
std::deque<Set> sets_;
|
||||
// where each camera's view of each hand last was, for ghosts
|
||||
struct Last {
|
||||
Roi roi;
|
||||
int64_t t = 0;
|
||||
};
|
||||
std::map<std::pair<std::string, int>, Last> last_;
|
||||
bool pending_ = false;
|
||||
FlagRequest req_;
|
||||
int64_t flag_t_ = 0;
|
||||
size_t dropped_at_flag_ = 0;
|
||||
std::string extra_notes_;
|
||||
std::thread writer_;
|
||||
Stats stats_;
|
||||
};
|
||||
|
||||
// The calibrated cameras as the manifest has them (frame-hands notes/flag-failures.md).
|
||||
std::string cameras_json(const std::map<std::string, Camera> &cams);
|
||||
// A set's output for FlagBuffer::add: the hands (id, side, scores, measured and published points)
|
||||
// and, if given, the gestures' state per side.
|
||||
std::string output_json(const std::vector<const Hand *> &hands, const Pinch *pinch, const Pinch *middle, const Grip *grip);
|
||||
// s as a JSON string ("..." with escapes); "null" when null_if_empty and s is empty.
|
||||
std::string json_string(const std::string &s, bool null_if_empty = false);
|
||||
+11
-57
@@ -122,55 +122,16 @@ std::string run_dir() {
|
||||
bool Publisher::open(std::string &err) {
|
||||
const std::string path = run_dir() + "/hands";
|
||||
const int fd = ::open(path.c_str(), O_RDWR | O_CREAT | O_NOFOLLOW | O_CLOEXEC, 0600);
|
||||
if (fd < 0 || ftruncate(fd, sizeof(fh_hands_v2_t)) < 0) return err = path + ": " + std::strerror(errno), false;
|
||||
void *m = mmap(nullptr, sizeof(fh_hands_v2_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
|
||||
if (fd < 0 || ftruncate(fd, sizeof(fh_hands_t)) < 0) return err = path + ": " + std::strerror(errno), false;
|
||||
void *m = mmap(nullptr, sizeof(fh_hands_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
|
||||
close(fd);
|
||||
if (m == MAP_FAILED) return err = path + ": can't map it", false;
|
||||
use(static_cast<fh_hands_v2_t *>(m));
|
||||
return true;
|
||||
}
|
||||
|
||||
void Publisher::use(fh_hands_v2_t *mem) {
|
||||
file_ = mem;
|
||||
out_ = &file_->v1;
|
||||
std::memset(file_, 0, sizeof *file_);
|
||||
out_ = static_cast<fh_hands_t *>(m);
|
||||
std::memset(out_, 0, sizeof *out_);
|
||||
std::memcpy(out_->magic, FH_HANDS_MAGIC, 8);
|
||||
out_->version = FH_HANDS_VERSION;
|
||||
out_->size = sizeof(fh_hands_v2_t);
|
||||
}
|
||||
|
||||
static_assert(sizeof(fh_hand_fit_t::angles) / sizeof(float) == skel::kAngles, "fh_hand_fit_t angles");
|
||||
|
||||
void Publisher::fill_fit(const Hand &h, uint64_t capture_ns, fh_hand_fit_t &o) {
|
||||
std::memset(&o, 0, sizeof o);
|
||||
o.id = uint32_t(h.id);
|
||||
const skel::Fit &f = h.fit;
|
||||
if (!f.valid) return;
|
||||
auto put = [](float *out, V3 v) {
|
||||
for (int k = 0; k < 3; ++k) out[k] = float(v[k]);
|
||||
};
|
||||
o.flags = FH_FIT_VALID | FH_FIT_ELBOW_HEAD_UP | (f.side > 0 ? FH_FIT_RIGHT : 0) | (f.settled >= 1 ? FH_FIT_SETTLED : 0) |
|
||||
(f.flipped ? FH_FIT_FLIPPED : 0);
|
||||
o.tracked_ns = capture_ns;
|
||||
for (int i = 0; i < 21; ++i) {
|
||||
put(o.joints[i].p, f.pts[i]);
|
||||
o.joints[i].radius = float(f.radius[i]);
|
||||
double q[4];
|
||||
skel::to_quat(f.frames[i], q);
|
||||
for (int k = 0; k < 4; ++k) o.joints[i].q[k] = float(q[k]);
|
||||
}
|
||||
for (int b = 0; b < 20; ++b) o.bone_len[b] = float(f.len[b]);
|
||||
for (int a = 0; a < skel::kAngles; ++a) o.angles[a] = float(f.qf[a]);
|
||||
put(o.elbow, f.elbow);
|
||||
o.elbow_conf = 0;
|
||||
put(o.pinch_point, (f.pts[5] + f.pts[9]) * 0.5);
|
||||
put(o.contact, (f.pts[4] + f.pts[8]) * 0.5);
|
||||
o.pinch_d[0] = float(norm(f.pts[4] - f.pts[8]));
|
||||
o.pinch_d[1] = float(norm(f.pts[4] - f.pts[12]));
|
||||
o.fit_rms = float(f.rms);
|
||||
o.settled = float(f.settled);
|
||||
put(o.vel_head, f.vel);
|
||||
put(o.angvel_head, f.angvel);
|
||||
out_->size = sizeof(fh_hands_t);
|
||||
return true;
|
||||
}
|
||||
|
||||
void Publisher::write(const std::vector<const Hand *> &in, uint64_t capture_ns) {
|
||||
@@ -185,29 +146,22 @@ void Publisher::write(const std::vector<const Hand *> &in, uint64_t capture_ns)
|
||||
std::memset(&o, 0, sizeof o);
|
||||
if (k >= hands.size()) continue;
|
||||
const Hand &h = *hands[k];
|
||||
// the fitted skeleton's points when there is one (fingers keep their length), else the
|
||||
// filtered landmarks (ft-hands --skeleton 0)
|
||||
const V3 *pts = h.fit.valid ? h.fit.pts : h.smooth;
|
||||
o.id = uint32_t(h.id);
|
||||
o.flags = (h.right() ? FH_HAND_RIGHT : 0) | (h.nviews >= 2 ? FH_HAND_STEREO : 0);
|
||||
o.confidence = float(std::min(1.0, h.frames / 5.0));
|
||||
for (int i = 0; i < 21; ++i)
|
||||
for (int j = 0; j < 3; ++j) o.pts[i][j] = float(pts[i][j]);
|
||||
for (int j = 0; j < 3; ++j) o.pts[i][j] = float(h.smooth[i][j]);
|
||||
const uint32_t first = nc;
|
||||
for (auto &b : kThumb) put(out_->capsules, nc, pts[b[0]], pts[b[1]], kThumbR, kThumbR);
|
||||
for (auto &b : kFingers) put(out_->capsules, nc, pts[b[0]], pts[b[1]], kFinger, kFinger);
|
||||
for (auto &b : kPalm) put(out_->capsules, nc, pts[b[0]], pts[b[1]], kPalmR, kPalmR);
|
||||
for (auto &b : kThumb) put(out_->capsules, nc, h.smooth[b[0]], h.smooth[b[1]], kThumbR, kThumbR);
|
||||
for (auto &b : kFingers) put(out_->capsules, nc, h.smooth[b[0]], h.smooth[b[1]], kFinger, kFinger);
|
||||
for (auto &b : kPalm) put(out_->capsules, nc, h.smooth[b[0]], h.smooth[b[1]], kPalmR, kPalmR);
|
||||
// the forearm carries on from the hand's own axis (middle knuckle -> wrist); the
|
||||
// wrist bends, but much less than a guess at where the elbow is gets wrong
|
||||
const V3 wrist = pts[0], d = wrist - pts[9];
|
||||
const V3 wrist = h.smooth[0], d = wrist - h.smooth[9];
|
||||
const double n = norm(d);
|
||||
if (n > 0.02) put(out_->capsules, nc, wrist, wrist + d * (kArmLen / n), kArm[0], kArm[1]);
|
||||
o.ncapsules = nc - first;
|
||||
}
|
||||
for (size_t k = 0; k < FH_HANDS_MAX_HANDS; ++k) {
|
||||
if (k < hands.size()) fill_fit(*hands[k], capture_ns, file_->fits[k]);
|
||||
else std::memset(&file_->fits[k], 0, sizeof file_->fits[k]);
|
||||
}
|
||||
for (uint32_t k = nc; k < FH_HANDS_MAX_CAPSULES; ++k) std::memset(&out_->capsules[k], 0, sizeof(fh_capsule_t));
|
||||
out_->capture_ns = capture_ns;
|
||||
out_->publish_ns = mono_ns();
|
||||
|
||||
@@ -35,14 +35,9 @@ private:
|
||||
class Publisher {
|
||||
public:
|
||||
bool open(std::string &err);
|
||||
// Writes into `mem` instead of the hands file (ft-handreplay --publish-dump).
|
||||
void use(fh_hands_v2_t *mem);
|
||||
void write(const std::vector<const Hand *> &hands, uint64_t capture_ns);
|
||||
// Fills one hand's version 2 record (also for ft-handreplay's checks).
|
||||
static void fill_fit(const Hand &h, uint64_t capture_ns, fh_hand_fit_t &o);
|
||||
|
||||
private:
|
||||
fh_hands_v2_t *file_ = nullptr;
|
||||
fh_hands_t *out_ = nullptr;
|
||||
uint64_t seq_ = 0;
|
||||
};
|
||||
|
||||
+11
-241
@@ -6,13 +6,6 @@
|
||||
// [--no-publish] [--no-gestures] [--record DIR] [--sides auto|0|1] [--cams auto|mono|color|all] ...
|
||||
// (--help lists them all)
|
||||
//
|
||||
// Parked (hand interaction in VR games): while run_dir/park exists (/run/user/UID/frametop-hands/park;
|
||||
// the input relay makes it when Frametop pauses for a game and removes it when the dashboard opens or
|
||||
// Frametop resumes), ft-hands processes no frames: the models stay loaded, the hands file says no
|
||||
// hands, and a gesture down ends, as lost. It writes "0" to run_dir/hands-want every second, so
|
||||
// ft-camd stops copying the mono cameras' frames meanwhile (it copies again 3 s after the last
|
||||
// write). Unparked, it tracks again from a palm search (about 200 ms). Not while recording.
|
||||
//
|
||||
// --no-gestures: hands for the cutouts only. No pinch or grip detection, so nothing reaches
|
||||
// the pointer and a closing hand doesn't raise the rate; the gestures file is removed.
|
||||
//
|
||||
@@ -46,18 +39,7 @@
|
||||
// colour camera is passthrough_left: color_video0 or color_video3), HANDS_COLOR_CROP
|
||||
// (subtract or none: tools/check_color.py tells both), HANDS_MISREAD_GUARD (0 or 1: the
|
||||
// tracker's guards for fine-tuned landmark models, Tracker::set_misread_guard), HANDS_MODELS (as
|
||||
// --models: a folder with palm.ncnn.* and hand.ncnn.*, e.g. fine-tuned ones), HANDS_REACQUIRE (seconds
|
||||
// the palm search aims at a lost hand, 0 off: Tracker::set_reacquire), HANDS_PAIR_SIDES (0 or 1: two
|
||||
// hands are one left and one right, Tracker::set_pair_sides), HANDS_PINCH_DIP (0 or 1: pinches also
|
||||
// begin on a sharp dip of the thumb-index distance, PinchParams::dip), HANDS_PINCH_FIST_CURL (1.2; 0 off:
|
||||
// no pinch begins while the index and middle are curled under this, PinchParams::fist_curl).
|
||||
//
|
||||
// Flags (track/flagbuf.h; HANDS_FLAG_BUFFER, default 1): ft-hands keeps the last HANDS_FLAG_SECONDS
|
||||
// (12) of hand crops and its output in RAM (at most HANDS_FLAG_MB, 80, of crops HANDS_FLAG_CROP, 224
|
||||
// px, a side), and `ft-handsctl flag [note]` writes them and the next 2 s to a folder for frame-hands'
|
||||
// offline tools. The request is run_dir/flag (key=value lines: dir, note, said_unix, flag_unix),
|
||||
// checked 4 times a second and removed when taken; the folder's manifest.json is written last.
|
||||
#include "flagbuf.h"
|
||||
// --models: a folder with palm.ncnn.* and hand.ncnn.*, e.g. fine-tuned ones).
|
||||
#include "io.h"
|
||||
#include "pinch.h"
|
||||
#include "record.h"
|
||||
@@ -278,36 +260,10 @@ int main(int argc, char **argv) {
|
||||
Contrast palm_contrast, hand_contrast{Contrast::None};
|
||||
double keep_presence = 0.5; // landmark presence a tracked view needs to stay
|
||||
bool misread_guard = setting("HANDS_MISREAD_GUARD") == "1";
|
||||
double reacquire = 1.5;
|
||||
if (const std::string v = setting("HANDS_REACQUIRE"); !v.empty()) reacquire = std::atof(v.c_str());
|
||||
bool pair_sides = setting("HANDS_PAIR_SIDES") != "0";
|
||||
double hold = 0.6; // s a lost, established hand is kept and retried before it's forgotten
|
||||
// the skeleton fit (track/skeleton.h): published as fh_hands version 2, and as version 1's
|
||||
// points and capsules; 0 publishes the One Euro filtered landmarks as before, and no skeleton
|
||||
bool skeleton = setting("HANDS_SKELETON") != "0";
|
||||
PinchParams pinch_params;
|
||||
pinch_params.dip = setting("HANDS_PINCH_DIP") == "1";
|
||||
if (const std::string v = setting("HANDS_PINCH_FIST_CURL"); !v.empty()) pinch_params.fist_curl = std::atof(v.c_str());
|
||||
// How easily a pinch lets go (user, 2026-10-11, gaze + hands: "taps and pinches release way too easily
|
||||
// even when I haven't actually released them": ends came at 4-6 cm, and grips took pinches 130 ms in):
|
||||
// HANDS_PINCH_END (m), HANDS_PINCH_DRAG_OPEN (m), HANDS_PINCH_DRAG_OPEN_S, HANDS_GRIP_TAKEOVER_S.
|
||||
auto num = [&](const char *key, double &v, double lo, double hi) {
|
||||
if (const std::string t = setting(key); !t.empty()) v = std::clamp(std::atof(t.c_str()), lo, hi);
|
||||
};
|
||||
num("HANDS_PINCH_END", pinch_params.end_m, 0.025, 0.08);
|
||||
num("HANDS_PINCH_DRAG_OPEN", pinch_params.drag_open_m, 0.03, 0.12);
|
||||
num("HANDS_PINCH_DRAG_OPEN_S", pinch_params.drag_open_s, 0.0, 1.0);
|
||||
num("HANDS_GRIP_TAKEOVER_S", pinch_params.grip_takeover_s, 0.0, 1.0);
|
||||
GripParams grip_params;
|
||||
bool gesture_log = false; // what the pinch and grip detectors measure, 10 times a second
|
||||
double record_for = 120, record_hz = 0; // record_hz: at most this many sets a second (0: all)
|
||||
// the flag buffer (see the top)
|
||||
bool flag_on = setting("HANDS_FLAG_BUFFER") != "0";
|
||||
FlagConfig flag_cfg;
|
||||
if (const std::string v = setting("HANDS_FLAG_SECONDS"); !v.empty()) flag_cfg.before_s = std::clamp(std::atof(v.c_str()), 1.0, 60.0);
|
||||
if (const std::string v = setting("HANDS_FLAG_CROP"); !v.empty()) flag_cfg.crop_px = std::clamp(std::atoi(v.c_str()), 64, 512);
|
||||
if (const std::string v = setting("HANDS_FLAG_MB"); !v.empty())
|
||||
flag_cfg.max_bytes = size_t(std::clamp(std::atof(v.c_str()), 8.0, 1024.0) * 1048576);
|
||||
for (int i = 1; i < argc; ++i) {
|
||||
const std::string a = argv[i];
|
||||
const bool more = i + 1 < argc;
|
||||
@@ -323,8 +279,6 @@ int main(int argc, char **argv) {
|
||||
else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
|
||||
else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
|
||||
else if (a == "--pinch-palm-down" && more) pinch_params.palm_down_max = std::atof(argv[++i]);
|
||||
else if (a == "--pinch-dip" && more) pinch_params.dip = std::atoi(argv[++i]) != 0;
|
||||
else if (a == "--pinch-fist-curl" && more) pinch_params.fist_curl = std::atof(argv[++i]);
|
||||
else if (a == "--grip-begin" && more) grip_params.begin = std::atof(argv[++i]);
|
||||
else if (a == "--grip-end" && more) grip_params.end = std::atof(argv[++i]);
|
||||
else if (a == "--gesture-log") gesture_log = true;
|
||||
@@ -337,11 +291,6 @@ int main(int argc, char **argv) {
|
||||
else if (a == "--record-hz" && more) record_hz = std::max(0.0, std::atof(argv[++i]));
|
||||
else if (a == "--keep-presence" && more) keep_presence = std::atof(argv[++i]);
|
||||
else if (a == "--misread-guard" && more) misread_guard = std::string(argv[++i]) == "1";
|
||||
else if (a == "--reacquire" && more) reacquire = std::atof(argv[++i]);
|
||||
else if (a == "--pair-sides" && more) pair_sides = std::string(argv[++i]) == "1";
|
||||
else if (a == "--hold" && more) hold = std::atof(argv[++i]);
|
||||
else if (a == "--skeleton" && more) skeleton = std::string(argv[++i]) != "0";
|
||||
else if (a == "--flag-buffer" && more) flag_on = std::string(argv[++i]) != "0";
|
||||
else if (a == "--cams" && more) cams_arg = argv[++i];
|
||||
else if (a == "--bright" && more) bright_arg = argv[++i];
|
||||
else if (a == "--bright-on" && more) light.on = std::atof(argv[++i]);
|
||||
@@ -363,15 +312,9 @@ int main(int argc, char **argv) {
|
||||
" as --swap-sides; 0: as ft-camd names them)\n"
|
||||
" [--keep-presence P] (0.5) [--ring PATH] (ft-camd's, or ft-ringplay's)\n"
|
||||
" [--misread-guard 0|1] (0; 1 for fine-tuned landmark models: see Tracker::set_misread_guard)\n"
|
||||
" [--reacquire S] (1.5; 0: off: see Tracker::set_reacquire) [--pair-sides 0|1] (1) [--hold S] (0.6)\n"
|
||||
" [--skeleton 0|1] (1: fit a skeleton with the user's bone lengths and publish it; HANDS_SKELETON)\n"
|
||||
" [--flag-buffer 0|1] (1: keep the last seconds for `ft-handsctl flag`; HANDS_FLAG_BUFFER)\n"
|
||||
" [--cams auto|mono|color|all] (auto) [--bright all|color] (all) [--bright-on L] (40) [--bright-off L] (25)\n"
|
||||
" [--color-left color_video0|color_video3] [--color-crop subtract|none]\n"
|
||||
" [--pinch-begin M] (0.020) [--pinch-end M] (0.035) [--pinch-triangulated] [--pinch-palm-down MAX] (1: off)\n"
|
||||
" [--pinch-dip 0|1] (0; 1: also on a sharp dip, see PinchParams::dip; HANDS_PINCH_DIP)\n"
|
||||
" [--pinch-fist-curl R] (1.2; 0: off: no pinch begins on a fist, see PinchParams::fist_curl;\n"
|
||||
" HANDS_PINCH_FIST_CURL)\n"
|
||||
" [--grip-begin R] (1.2) [--grip-end R] (1.45) [--gesture-log]\n"
|
||||
" [--contrast MODE|PALM/HAND] (clahe[:CLIP], none, stretch; default clahe:2/none)\n"
|
||||
"Recording saves every frame set (at most N a second with --record-hz) for S seconds (120) to DIR/sets.bin,\n"
|
||||
@@ -381,8 +324,7 @@ int main(int argc, char **argv) {
|
||||
"camera's newest dark frame, as <name>_dk; with --with-color, the color cameras' as color_video<N>.\n"
|
||||
"auto picks the cameras by the light (see the top of track/main.cpp).\n"
|
||||
"Settings in ~/.config/frametop.conf: HANDS_SWAP_SIDES=auto|0|1, HANDS_CPUS=5,6,7, HANDS_CAMERAS, HANDS_BRIGHT,\n"
|
||||
"HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT, HANDS_COLOR_CROP, HANDS_MISREAD_GUARD=0|1, HANDS_MODELS=DIR,\n"
|
||||
"HANDS_PINCH_DIP=0|1, HANDS_PINCH_FIST_CURL, HANDS_FLAG_BUFFER=0|1, HANDS_FLAG_SECONDS, HANDS_FLAG_CROP, HANDS_FLAG_MB\n"
|
||||
"HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT, HANDS_COLOR_CROP, HANDS_MISREAD_GUARD=0|1, HANDS_MODELS=DIR\n"
|
||||
"(FT_<name> overrides).\n",
|
||||
argv[0]);
|
||||
return a == "--help" ? 0 : 1;
|
||||
@@ -407,17 +349,7 @@ int main(int argc, char **argv) {
|
||||
Nets nets;
|
||||
Publisher pub;
|
||||
GesturePublisher gestures;
|
||||
// The middle pinch (thumb to middle fingertip, fh_gestures v3): the same thresholds, and the two
|
||||
// exclude each other: each begins only with the thumb nearer its own fingertip (the middle one
|
||||
// by 1 cm), and not while the other is down on that hand. A middle pinch also wants the index
|
||||
// extended and the hand tracked 5 frames (see PinchParams).
|
||||
PinchParams middle_params = pinch_params;
|
||||
middle_params.finger = 12, middle_params.other_finger = 8, middle_params.other_margin = 0.01;
|
||||
middle_params.extended_finger = 8, middle_params.min_frames = 5;
|
||||
middle_params.dip = false; // the dip rule is tuned on the index pinch only
|
||||
pinch_params.other_finger = 12;
|
||||
Pinch pinch(pinch_params);
|
||||
Pinch middle(middle_params);
|
||||
Grip grip(grip_params);
|
||||
std::unique_ptr<Recorder> rec;
|
||||
uint64_t rec_start = 0, rec_next_ns = 0; // rec_next_ns: --record-hz's next set, capture clock
|
||||
@@ -566,38 +498,7 @@ int main(int argc, char **argv) {
|
||||
Tracker tracker(used, nets, pool);
|
||||
tracker.set_keep_presence(keep_presence);
|
||||
tracker.set_misread_guard(misread_guard);
|
||||
tracker.set_reacquire(reacquire, 2, true);
|
||||
tracker.set_pair_sides(pair_sides);
|
||||
tracker.set_hold(reacquire > 0 ? hold : 0.3);
|
||||
tracker.set_skeleton(skeleton);
|
||||
std::map<std::string, std::vector<uint8_t>> pixels;
|
||||
// The flag buffer (see the top), and its request file.
|
||||
std::unique_ptr<FlagBuffer> flags;
|
||||
const std::string flag_file = run_dir() + "/flag";
|
||||
if (flag_on && track) {
|
||||
flags = std::make_unique<FlagBuffer>(flag_cfg);
|
||||
char cfg[1024];
|
||||
std::string cpu_list;
|
||||
for (int c : cpus) cpu_list += (cpu_list.empty() ? "" : ",") + std::to_string(c);
|
||||
std::snprintf(cfg, sizeof cfg,
|
||||
"{\"models\": %s, \"int8\": %s, \"threads\": %d, \"cpus\": \"%s\", \"cams\": \"%s\", \"bright\": \"%s\", "
|
||||
"\"keep_presence\": %.2f, \"misread_guard\": %s, \"reacquire\": %.2f, \"pair_sides\": %s, \"hold\": %.2f, "
|
||||
"\"skeleton\": %s, \"gestures\": %s, \"pinch_begin\": %.3f, \"pinch_end\": %.3f, \"pinch_dip\": %s, "
|
||||
"\"pinch_fist_curl\": %.2f, \"grip_begin\": %.2f, "
|
||||
"\"grip_end\": %.2f, \"sides\": \"%s\", \"flag_seconds\": %.1f, \"flag_crop\": %d, \"flag_mb\": %.0f}",
|
||||
json_string(models).c_str(), int8 ? "true" : "false", threads, cpu_list.c_str(), cams_arg.c_str(),
|
||||
bright_arg.c_str(), keep_presence, misread_guard ? "true" : "false", reacquire, pair_sides ? "true" : "false",
|
||||
hold, skeleton ? "true" : "false", gestures_on ? "true" : "false", pinch_params.begin_m, pinch_params.end_m,
|
||||
pinch_params.dip ? "true" : "false", pinch_params.fist_curl, grip_params.begin, grip_params.end,
|
||||
sides_mode.c_str(), flag_cfg.before_s, flag_cfg.crop_px,
|
||||
flag_cfg.max_bytes / 1048576.0);
|
||||
flags->set_static(cameras_json(used), cfg);
|
||||
}
|
||||
struct stat flag_st; // a request left from before this run is stale
|
||||
if (stat(flag_file.c_str(), &flag_st) == 0 && std::time(nullptr) - flag_st.st_mtime > 10) unlink(flag_file.c_str());
|
||||
uint64_t t_flag_check = 0;
|
||||
size_t flags_told = 0;
|
||||
std::string flag_error_told;
|
||||
std::map<std::string, uint64_t> lit_seen; // per colour camera: the frame last counted for the light
|
||||
const uint64_t start = mono_ns();
|
||||
uint64_t t_status = start, next_ns = 0, t_want = 0, t_glog = 0;
|
||||
@@ -609,12 +510,6 @@ int main(int argc, char **argv) {
|
||||
// to place the colour frames, whose capture clock is their own (see the top).
|
||||
double mono_delay_ns = -1;
|
||||
const std::string want_file = run_dir() + "/color-fps";
|
||||
// A reader that needs the full rate writes it to want-hz and keeps the file fresh: ft-screens,
|
||||
// while a hand is near one of its panels (a poke's latency at 15 Hz doubles). Older than 1 s
|
||||
// (by its mtime), it lapses. Checked 5 times a second.
|
||||
const std::string want_hz_file = run_dir() + "/want-hz";
|
||||
double want_hz = 0;
|
||||
uint64_t t_want_hz = 0;
|
||||
// The side cameras' naming, for the hand recorder and other tools (hands/rec/session.py
|
||||
// reads it): written by a tracking, publishing ft-hands at the start, on every change and
|
||||
// every 2 s, and removed when it exits. swapped: ft-camd's naming is backwards (null: not
|
||||
@@ -729,99 +624,13 @@ int main(int argc, char **argv) {
|
||||
return n ? sum / n : -1;
|
||||
};
|
||||
|
||||
// Parking (see "Parked" at the top): run_dir/park says so; ft-camd hears it through hands-want.
|
||||
const std::string park_file = run_dir() + "/park", hands_want_file = run_dir() + "/hands-want";
|
||||
const bool can_park = track && publish;
|
||||
bool parked = false;
|
||||
uint64_t t_park = 0, t_hands_want = 0;
|
||||
int exit_code = 0;
|
||||
// A flag request (see the top): taken, and answered with a folder.
|
||||
auto take_flag = [&] {
|
||||
std::ifstream f(flag_file);
|
||||
if (!f) return;
|
||||
std::map<std::string, std::string> kv;
|
||||
std::string line;
|
||||
while (std::getline(f, line))
|
||||
if (const auto eq = line.find('='); eq != std::string::npos) kv[line.substr(0, eq)] = line.substr(eq + 1);
|
||||
unlink(flag_file.c_str());
|
||||
const std::string dir = kv["dir"];
|
||||
if (dir.empty() || dir[0] != '/') return (void)std::printf("flag: no folder in the request\n");
|
||||
auto num = [&](const char *k) { return kv[k].empty() ? std::string("null") : std::to_string(std::atof(kv[k].c_str())); };
|
||||
char light_json[160];
|
||||
std::snprintf(light_json, sizeof light_json, "{\"cams\": \"%s\", \"level\": %.1f, \"ambient_ir\": %.1f}", cams_name(mode),
|
||||
light.level, ambient());
|
||||
FlagRequest req;
|
||||
req.dir = dir;
|
||||
req.fields = "\"source\": \"ft-hands\", \"flag_unix\": " + num("flag_unix") + ", \"note\": " + json_string(kv["note"], true) +
|
||||
", \"said_unix\": " + num("said_unix") + ", \"models\": " + json_string(models) +
|
||||
", \"sides\": {\"names_swapped\": " + json_bool(names_swapped) + ", \"swapped\": " + json_bool(truth) +
|
||||
"}, \"light\": " + light_json + ", \"parked\": " + (parked ? "true" : "false");
|
||||
if (!flags) { // no buffer: the folder says why
|
||||
mkdir(dir.c_str(), 0700);
|
||||
write_file(dir + "/manifest.json", "{\"version\": 1, " + req.fields +
|
||||
", \"error\": \"no flag buffer (HANDS_FLAG_BUFFER=0, or not tracking)\"}\n");
|
||||
std::printf("flag: %s (no buffer: HANDS_FLAG_BUFFER=0)\n", dir.c_str());
|
||||
} else if (flags->flag(req, int64_t(mono_ns()) + raw_minus_mono_ns(), kv["note"])) {
|
||||
std::printf("flag: %s%s%s, written in %.0f s\n", dir.c_str(), kv["note"].empty() ? "" : ": ", kv["note"].c_str(),
|
||||
flags->config().after_s);
|
||||
} else {
|
||||
std::printf("flag: one is already being taken%s\n", kv["note"].empty() ? "" : " (its note added)");
|
||||
}
|
||||
std::fflush(stdout);
|
||||
};
|
||||
while (!g_stop && (seconds <= 0 || (mono_ns() - start) / 1e9 < seconds)) {
|
||||
if (mono_ns() - t_flag_check > 250'000'000) {
|
||||
t_flag_check = mono_ns();
|
||||
if (stat(flag_file.c_str(), &flag_st) == 0 && flag_st.st_uid == getuid()) take_flag();
|
||||
if (flags) {
|
||||
flags->tick(int64_t(mono_ns()) + raw_minus_mono_ns());
|
||||
const FlagBuffer::Stats fs = flags->stats();
|
||||
if (fs.flags != flags_told || fs.last_error != flag_error_told) {
|
||||
if (fs.flags != flags_told) std::printf("flag written: %s\n", fs.last_dir.c_str());
|
||||
if (!fs.last_error.empty() && fs.last_error != flag_error_told) std::printf("flag failed: %s\n", fs.last_error.c_str());
|
||||
flags_told = fs.flags, flag_error_told = fs.last_error;
|
||||
std::fflush(stdout);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!ring.alive()) {
|
||||
std::fprintf(stderr, "ft-camd stopped\n");
|
||||
exit_code = 2;
|
||||
break;
|
||||
}
|
||||
if (can_park && mono_ns() - t_park > 250'000'000) {
|
||||
t_park = mono_ns();
|
||||
struct stat pst;
|
||||
const bool want = !rec && !g_record && stat(park_file.c_str(), &pst) == 0;
|
||||
if (want != parked) {
|
||||
parked = want;
|
||||
std::printf("%s\n", parked ? "parked: no tracking until run_dir/park goes (models stay loaded)"
|
||||
: "unparked: tracking again");
|
||||
std::fflush(stdout);
|
||||
if (parked) {
|
||||
// the hands are gone for every reader, and a gesture down ends, as lost
|
||||
pub.write({}, mono_ns());
|
||||
if (gestures_on) {
|
||||
pinch.release(int64_t(mono_ns()));
|
||||
middle.release(int64_t(mono_ns()));
|
||||
grip.release(int64_t(mono_ns()));
|
||||
gestures.write(pinch, grip, mono_ns(), &middle);
|
||||
}
|
||||
t_hands_want = 0;
|
||||
} else {
|
||||
unlink(hands_want_file.c_str());
|
||||
next_ns = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (parked) {
|
||||
if (mono_ns() - t_hands_want > 1'000'000'000ull) { // fresh, so ft-camd keeps idling
|
||||
t_hands_want = mono_ns();
|
||||
write_file(hands_want_file, "0\n");
|
||||
}
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(50));
|
||||
continue;
|
||||
}
|
||||
const uint64_t now0 = mono_ns();
|
||||
const int64_t raw_off = raw_minus_mono_ns();
|
||||
|
||||
@@ -847,19 +656,6 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
}
|
||||
|
||||
if (now0 - t_want_hz > 200'000'000) {
|
||||
t_want_hz = now0, want_hz = 0;
|
||||
struct stat st;
|
||||
timespec rt;
|
||||
clock_gettime(CLOCK_REALTIME, &rt);
|
||||
if (stat(want_hz_file.c_str(), &st) == 0 && st.st_uid == getuid() &&
|
||||
(rt.tv_sec - st.st_mtim.tv_sec) * 1e9 + (rt.tv_nsec - st.st_mtim.tv_nsec) < 1e9)
|
||||
if (FILE *f = std::fopen(want_hz_file.c_str(), "r")) {
|
||||
if (std::fscanf(f, "%lf", &want_hz) != 1 || !(want_hz > 0)) want_hz = 0;
|
||||
want_hz = std::min(want_hz, 30.0);
|
||||
std::fclose(f);
|
||||
}
|
||||
}
|
||||
const bool use_mono = mode != Cams::Color, use_color = mode != Cams::Mono;
|
||||
const bool mono_driven = use_mono || rec || g_record || !track;
|
||||
std::map<std::string, Image> images;
|
||||
@@ -992,12 +788,8 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
if (mono_ns() - t_sides > 2'000'000'000ull) write_sides();
|
||||
if (gestures_on) {
|
||||
// no grip begins on a hand holding a pinch (a drag: PinchParams::grip_takeover_s)
|
||||
std::vector<int> holding = pinch.holding(int64_t(capture)), held2 = middle.holding(int64_t(capture));
|
||||
holding.insert(holding.end(), held2.begin(), held2.end());
|
||||
grip.update(hands, views, int64_t(capture), holding);
|
||||
pinch.update(hands, views, int64_t(capture), grip.gripping(), middle.down());
|
||||
middle.update(hands, views, int64_t(capture), grip.gripping(), pinch.down());
|
||||
grip.update(hands, views, int64_t(capture));
|
||||
pinch.update(hands, views, int64_t(capture), grip.gripping());
|
||||
}
|
||||
if (gestures_on && gesture_log && capture - t_glog >= 100'000'000) {
|
||||
t_glog = capture;
|
||||
@@ -1008,25 +800,16 @@ int main(int argc, char **argv) {
|
||||
pinch.side(k).flags & FH_PINCH_DOWN ? " PINCH" : grip.side(k).flags & FH_PINCH_DOWN ? " GRIP" : "");
|
||||
}
|
||||
// a gesture down or closing gets the full rate, even while the palm holds still
|
||||
// and so does a reader that asks for it (want-hz)
|
||||
next_ns = tmin + uint64_t((std::min({tracker.interval(),
|
||||
pinch.engaged() || middle.engaged() || grip.engaged() ? 1 / 30.0 : 1.0,
|
||||
want_hz > 0 ? 1 / want_hz : 1.0}) -
|
||||
next_ns = tmin + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) -
|
||||
0.005) * 1e9);
|
||||
if (publish) pub.write(hands, capture);
|
||||
if (publish && gestures_on) gestures.write(pinch, grip, capture, &middle);
|
||||
if (flags)
|
||||
flags->add(int64_t(tmin), int64_t(capture), -1, images, views,
|
||||
output_json(hands, gestures_on ? &pinch : nullptr, gestures_on ? &middle : nullptr, gestures_on ? &grip : nullptr));
|
||||
if (publish && gestures_on) gestures.write(pinch, grip, capture);
|
||||
for (const Pinch::Event &e : grip.events)
|
||||
std::printf("grip %s %-5s curl %.2f at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
|
||||
e.point[0], e.point[1], e.point[2]);
|
||||
for (const Pinch::Event &e : pinch.events)
|
||||
std::printf("pinch %s %-5s d %.3f m at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what, e.distance,
|
||||
e.point[0], e.point[1], e.point[2]);
|
||||
for (const Pinch::Event &e : middle.events)
|
||||
std::printf("middle pinch %s %-5s d %.3f m at %+.3f %+.3f %+.3f\n", e.side ? "right" : "left ", e.what,
|
||||
e.distance, e.point[0], e.point[1], e.point[2]);
|
||||
lat.push_back((mono_ns() - dq) / 1e6);
|
||||
hands_sum += double(hands.size());
|
||||
bool on_left = false, on_right = false; // by where the wrist is, not the model's label
|
||||
@@ -1042,14 +825,11 @@ int main(int argc, char **argv) {
|
||||
const Stats &s = tracker.stats;
|
||||
std::sort(lat.begin(), lat.end());
|
||||
std::printf("%5.1fs %4.1f sets/s hands %.2f views %zu palm %3d calls %4.1f ms/batch hand %3d calls %4.1f ms/batch "
|
||||
"step %4.1f ms latency %4.1f ms resid %.1f mm fit %.0f us CPU %3.0f%%\n",
|
||||
"step %4.1f ms latency %4.1f ms resid %.1f mm CPU %3.0f%%\n",
|
||||
(now - start) / 1e9, s.sets / dt, s.sets ? hands_sum / s.sets : 0, tracker.views(), s.palm_calls,
|
||||
s.palm_batches ? s.palm_ms / s.palm_batches : 0, s.hand_calls,
|
||||
s.hand_batches ? s.hand_ms / s.hand_batches : 0, s.sets ? s.step_ms / s.sets : 0,
|
||||
lat.empty() ? 0 : lat[lat.size() / 2], resid_n ? resid_sum / resid_n : 0,
|
||||
tracker.skeleton().fits ? tracker.skeleton().us_total / tracker.skeleton().fits : 0.0,
|
||||
100 * (cpu1 - cpu0) / dt);
|
||||
tracker.skeleton().fits = 0, tracker.skeleton().us_total = 0;
|
||||
lat.empty() ? 0 : lat[lat.size() / 2], resid_n ? resid_sum / resid_n : 0, 100 * (cpu1 - cpu0) / dt);
|
||||
if (!color.empty())
|
||||
std::printf(" cameras %s%s: colour frames at %.1f (bright at %.0f, dim under %.0f), ambient IR %.1f, "
|
||||
"%d steps with colour, colour placed %.1f ms after capture\n",
|
||||
@@ -1062,20 +842,13 @@ int main(int argc, char **argv) {
|
||||
s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten,
|
||||
!rec ? "" : (" recorded " + std::to_string(rec->written()) + " dropped " +
|
||||
std::to_string(rec->dropped())).c_str());
|
||||
if (flags) {
|
||||
const FlagBuffer::Stats fs = flags->stats();
|
||||
std::printf(" flag buffer: %zu sets, %zu crops (%zu ghosts) in %.1f MB, %.0f us a crop, %zu dropped, %zu flags written%s%s\n",
|
||||
fs.sets, fs.crops, fs.ghosts, flags->bytes() / 1048576.0, fs.crop_calls ? fs.crop_us / fs.crop_calls : 0.0,
|
||||
fs.dropped, fs.flags, fs.last_error.empty() ? "" : "; last error: ", fs.last_error.c_str());
|
||||
}
|
||||
if (checking)
|
||||
std::printf(" side cameras: %s, %s\n", side_round ? "checking the decision" : "deciding",
|
||||
side_check.summary().c_str());
|
||||
if (gestures_on) {
|
||||
std::printf(" pinches: left %u right %u (held back, palm down: %d %d) middle pinches: left %u right %u "
|
||||
"grips: left %u right %u",
|
||||
std::printf(" pinches: left %u right %u (held back, palm down: %d %d) grips: left %u right %u",
|
||||
pinch.side(0).begins, pinch.side(1).begins, pinch.held_back[0], pinch.held_back[1],
|
||||
middle.side(0).begins, middle.side(1).begins, grip.side(0).begins, grip.side(1).begins);
|
||||
grip.side(0).begins, grip.side(1).begins);
|
||||
for (int k = 0; k < 2; ++k)
|
||||
if (pinch.side(k).flags & FH_PINCH_TRACKED)
|
||||
std::printf(" %s %s d %.3f curl %.2f", k ? "right" : "left",
|
||||
@@ -1096,17 +869,14 @@ int main(int argc, char **argv) {
|
||||
color_steps = 0;
|
||||
}
|
||||
}
|
||||
if (flags) flags->finish(), flags->wait();
|
||||
if (!color.empty()) unlink(want_file.c_str());
|
||||
if (parked) unlink(hands_want_file.c_str());
|
||||
if (write_sides_file) unlink(sides_file.c_str());
|
||||
write_rec_sides();
|
||||
if (publish) pub.write({}, mono_ns());
|
||||
if (publish && gestures_on) {
|
||||
pinch.release(int64_t(mono_ns())); // a drag in progress ends, as lost
|
||||
middle.release(int64_t(mono_ns()));
|
||||
grip.release(int64_t(mono_ns()));
|
||||
gestures.write(pinch, grip, mono_ns(), &middle);
|
||||
gestures.write(pinch, grip, mono_ns());
|
||||
}
|
||||
return exit_code;
|
||||
}
|
||||
+116
-235
@@ -14,10 +14,7 @@
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int kThumbTip = 4;
|
||||
// The dip rule ignores dips from fingers spread wider than this (m): a hand opening wide and
|
||||
// relaxing again isn't a tap.
|
||||
constexpr double kDipFromMax = 0.10;
|
||||
constexpr int kThumbTip = 4, kIndexTip = 8;
|
||||
|
||||
V3 cross(V3 a, V3 b) { return {a[1] * b[2] - a[2] * b[1], a[2] * b[0] - a[0] * b[2], a[0] * b[1] - a[1] * b[0]}; }
|
||||
|
||||
@@ -25,6 +22,112 @@ void put3(float out[3], V3 v) {
|
||||
for (int k = 0; k < 3; ++k) out[k] = float(v[k]);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void Pinch::update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns,
|
||||
const std::vector<int> &gripping) {
|
||||
events.clear();
|
||||
auto grips = [&](int id) { return std::find(gripping.begin(), gripping.end(), id) != gripping.end(); };
|
||||
for (int s = 0; s < 2; ++s) // a grip took this pinch's hand: it's a drag now, not a click
|
||||
if ((side_[s].flags & FH_PINCH_DOWN) && grips(follow_[s])) end(s, t_ns, true);
|
||||
// A hand a pinch is down on belongs to that side until it ends. The left/right call is a
|
||||
// running average of the model's, and when it flips mid-pinch the other side would take
|
||||
// the same hand and pinch too (4 times in the 2026-09-30 lit recording).
|
||||
int taken[2] = {0, 0};
|
||||
for (int s = 0; s < 2; ++s)
|
||||
if (side_[s].flags & FH_PINCH_DOWN) taken[s] = follow_[s];
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
fh_pinch_t &o = side_[s];
|
||||
const bool down = o.flags & FH_PINCH_DOWN;
|
||||
// the hand: while down, the one the pinch began on; else the best tracked hand of this side
|
||||
const Hand *h = nullptr;
|
||||
for (const Hand *c : hands) {
|
||||
if (down ? c->id != follow_[s] : c->right() != (s == 1) || c->id == taken[1 - s]) continue;
|
||||
if (!h || c->frames > h->frames) h = c;
|
||||
}
|
||||
world_d[s] = tri_d[s] = palm_down[s] = -1;
|
||||
if (!h) {
|
||||
o.flags &= ~FH_PINCH_TRACKED;
|
||||
if (down && (t_ns - seen_ns_[s]) / 1e9 > p_.grace_s) end(s, t_ns, true);
|
||||
continue;
|
||||
}
|
||||
seen_ns_[s] = t_ns;
|
||||
tri_d[s] = norm(h->pts[kThumbTip] - h->pts[kIndexTip]);
|
||||
const V3 normal = cross(h->smooth[5] - h->smooth[0], h->smooth[17] - h->smooth[0]);
|
||||
palm_down[s] = norm(normal) > 0 ? std::fabs(normal[1]) / norm(normal) : 0;
|
||||
double sum = 0;
|
||||
int n = 0;
|
||||
for (const Seen &v : views) {
|
||||
if (v.hand != h->id) continue;
|
||||
const V3 a{v.lm.world[kThumbTip][0], v.lm.world[kThumbTip][1], v.lm.world[kThumbTip][2]};
|
||||
const V3 b{v.lm.world[kIndexTip][0], v.lm.world[kIndexTip][1], v.lm.world[kIndexTip][2]};
|
||||
sum += norm(a - b), ++n;
|
||||
}
|
||||
if (n) world_d[s] = sum / n * h->scale;
|
||||
const double d = p_.triangulated || world_d[s] < 0 ? tri_d[s] : world_d[s];
|
||||
// Where the pinch is, for drags: the index and middle knuckles, which hold still while
|
||||
// the fingers open and close. The point between the tips moved 1-2 cm as a pinch
|
||||
// opened, so every release dragged the pointer off what it pressed (headset test,
|
||||
// 2026-09-30).
|
||||
const V3 point = (h->smooth[5] + h->smooth[9]) * 0.5;
|
||||
o.flags |= FH_PINCH_TRACKED;
|
||||
o.hand_id = uint32_t(h->id);
|
||||
o.distance = float(d);
|
||||
o.strength = float(std::clamp((p_.end_m - d) / (p_.end_m - p_.begin_m), 0.0, 1.0));
|
||||
put3(o.point, point);
|
||||
if (!down) {
|
||||
// a close held back (palm down) has to open again before a pinch can begin, so
|
||||
// turning the hand with the fingers still closed doesn't start one
|
||||
if (d > p_.end_m) held_[s] = false;
|
||||
if (grips(h->id)) {
|
||||
// closed: no pinch until it opens
|
||||
} else if (d < p_.begin_m && !held_[s] && palm_down[s] > p_.palm_down_max) {
|
||||
held_[s] = true;
|
||||
++held_back[s];
|
||||
} else if (d < p_.begin_m && !held_[s]) {
|
||||
o.flags = (o.flags | FH_PINCH_DOWN) & ~FH_PINCH_LOST;
|
||||
++o.begins;
|
||||
o.begin_ns = uint64_t(t_ns);
|
||||
put3(o.begin_point, point);
|
||||
follow_[s] = h->id;
|
||||
open_frames_[s] = 0;
|
||||
events.push_back({s, "begin", t_ns, d, point});
|
||||
}
|
||||
} else if (d > p_.end_m) {
|
||||
if (++open_frames_[s] >= p_.end_frames) end(s, t_ns, false);
|
||||
} else {
|
||||
open_frames_[s] = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void Pinch::end(int s, int64_t t_ns, bool lost) {
|
||||
fh_pinch_t &o = side_[s];
|
||||
o.flags = (o.flags & ~FH_PINCH_DOWN) | (lost ? FH_PINCH_LOST : 0);
|
||||
++o.ends;
|
||||
o.end_ns = uint64_t(t_ns);
|
||||
follow_[s] = 0;
|
||||
events.push_back({s, lost ? "lost" : "end", t_ns, o.distance, {o.point[0], o.point[1], o.point[2]}});
|
||||
}
|
||||
|
||||
void Pinch::release(int64_t t_ns) {
|
||||
events.clear();
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
side_[s].flags &= ~FH_PINCH_TRACKED;
|
||||
if (side_[s].flags & FH_PINCH_DOWN) end(s, t_ns, true);
|
||||
}
|
||||
}
|
||||
|
||||
bool Pinch::engaged() const {
|
||||
for (const fh_pinch_t &o : side_)
|
||||
if ((o.flags & FH_PINCH_DOWN) || ((o.flags & FH_PINCH_TRACKED) && o.strength > 0.3f)) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------- grip
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int kWrist = 0, kFingers[4][2] = {{5, 8}, {9, 12}, {13, 16}, {17, 20}}; // knuckle, tip
|
||||
|
||||
// Each finger's curl (see GripParams), and the thumb tip's distance from the index tip (m):
|
||||
@@ -56,230 +159,14 @@ bool curls(const Hand &h, const std::vector<Seen> &views, double out[4], double
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void Pinch::update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns,
|
||||
const std::vector<int> &gripping, const std::vector<int> &blocked) {
|
||||
events.clear();
|
||||
auto grips = [&](int id) { return std::find(gripping.begin(), gripping.end(), id) != gripping.end(); };
|
||||
auto other_down = [&](int id) { return std::find(blocked.begin(), blocked.end(), id) != blocked.end(); };
|
||||
const int tip = p_.finger, other = p_.other_finger;
|
||||
for (int s = 0; s < 2; ++s) // a grip took this pinch's hand on the way to it: a grip's drag, not a click
|
||||
if ((side_[s].flags & FH_PINCH_DOWN) && grips(follow_[s]) &&
|
||||
(t_ns - int64_t(side_[s].begin_ns)) / 1e9 < p_.grip_takeover_s)
|
||||
end(s, t_ns, true);
|
||||
// A hand a pinch is down on belongs to that side until it ends. The left/right call is a
|
||||
// running average of the model's, and when it flips mid-pinch the other side would take
|
||||
// the same hand and pinch too (4 times in the 2026-09-30 lit recording).
|
||||
int taken[2] = {0, 0};
|
||||
for (int s = 0; s < 2; ++s)
|
||||
if (side_[s].flags & FH_PINCH_DOWN) taken[s] = follow_[s];
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
fh_pinch_t &o = side_[s];
|
||||
const bool down = o.flags & FH_PINCH_DOWN;
|
||||
// the hand: while down, the one the pinch began on; else the best tracked hand of this side
|
||||
const Hand *h = nullptr;
|
||||
for (const Hand *c : hands) {
|
||||
if (down ? c->id != follow_[s] : c->right() != (s == 1) || c->id == taken[1 - s]) continue;
|
||||
if (!h || c->frames > h->frames) h = c;
|
||||
}
|
||||
world_d[s] = tri_d[s] = fit_d[s] = palm_down[s] = other_d[s] = -1;
|
||||
for (double &c : curl[s]) c = -1;
|
||||
if (!h) {
|
||||
o.flags &= ~FH_PINCH_TRACKED;
|
||||
if (down && (t_ns - seen_ns_[s]) / 1e9 > p_.grace_s) end(s, t_ns, true);
|
||||
continue;
|
||||
}
|
||||
seen_ns_[s] = t_ns;
|
||||
while (!recent_[s].empty() && (t_ns - recent_[s].front().t_ns) / 1e9 > std::max(p_.dip_win_s, 0.15) + 1e-3)
|
||||
recent_[s].pop_front();
|
||||
tri_d[s] = norm(h->pts[kThumbTip] - h->pts[tip]);
|
||||
if (h->fit.valid) fit_d[s] = norm(h->fit.pts[kThumbTip] - h->fit.pts[tip]);
|
||||
const V3 normal = cross(h->smooth[5] - h->smooth[0], h->smooth[17] - h->smooth[0]);
|
||||
palm_down[s] = norm(normal) > 0 ? std::fabs(normal[1]) / norm(normal) : 0;
|
||||
// thumb tip to fingertip f: the model's world landmarks averaged over the views (at the hand's
|
||||
// scale), the fitted tips, or the triangulated ones, as configured
|
||||
auto measure = [&](int f, double *world, double *tri, double *fit) {
|
||||
double sum = 0;
|
||||
int n = 0;
|
||||
for (const Seen &v : views) {
|
||||
if (v.hand != h->id) continue;
|
||||
const V3 a{v.lm.world[kThumbTip][0], v.lm.world[kThumbTip][1], v.lm.world[kThumbTip][2]};
|
||||
const V3 b{v.lm.world[f][0], v.lm.world[f][1], v.lm.world[f][2]};
|
||||
sum += norm(a - b), ++n;
|
||||
}
|
||||
*world = n ? sum / n * h->scale : -1;
|
||||
*tri = norm(h->pts[kThumbTip] - h->pts[f]);
|
||||
*fit = h->fit.valid ? norm(h->fit.pts[kThumbTip] - h->fit.pts[f]) : -1;
|
||||
return p_.fitted && *fit >= 0 ? *fit : p_.triangulated || *world < 0 ? *tri : *world;
|
||||
};
|
||||
double unused_tri, unused_fit;
|
||||
const double d = measure(tip, &world_d[s], &unused_tri, &unused_fit);
|
||||
double ow = -1, ot = -1, of = -1;
|
||||
if (other) other_d[s] = measure(other, &ow, &ot, &of);
|
||||
// the thumb clearly nearer this fingertip than the other one, the extended finger straight
|
||||
// enough (from the model's world landmarks, else the tracked points), the hand tracked a while
|
||||
bool own = !other || d + p_.other_margin < other_d[s];
|
||||
if (own && p_.extended_finger > 0) {
|
||||
const int f = p_.extended_finger, knuckle = f - 3;
|
||||
double ratio = -1;
|
||||
for (const Seen &v : views) {
|
||||
if (v.hand != h->id) continue;
|
||||
auto w = [&](int i) { return V3{v.lm.world[i][0], v.lm.world[i][1], v.lm.world[i][2]}; };
|
||||
const double k = norm(w(knuckle) - w(0));
|
||||
if (k > 1e-4) ratio = std::max(ratio, norm(w(f) - w(0)) / k);
|
||||
}
|
||||
if (ratio < 0) {
|
||||
const double k = norm(h->smooth[knuckle] - h->smooth[0]);
|
||||
ratio = k > 1e-4 ? norm(h->smooth[f] - h->smooth[0]) / k : 0;
|
||||
}
|
||||
own = ratio >= p_.extended_min;
|
||||
}
|
||||
if (own && h->frames < p_.min_frames) own = false;
|
||||
// the fist guard: the index and middle both curled (see PinchParams::fist_curl)
|
||||
double f[4], unused_thumb;
|
||||
const bool curled = curls(*h, views, f, &unused_thumb);
|
||||
for (int k = 0; k < 4; ++k) curl[s][k] = curled ? f[k] : -1;
|
||||
const bool fist = p_.fist_curl > 0 && curled && f[0] < p_.fist_curl && f[1] < p_.fist_curl;
|
||||
// Where the pinch is, for drags: the index and middle knuckles, which hold still while
|
||||
// the fingers open and close. The point between the tips moved 1-2 cm as a pinch
|
||||
// opened, so every release dragged the pointer off what it pressed (headset test,
|
||||
// 2026-09-30).
|
||||
const V3 point = (h->smooth[5] + h->smooth[9]) * 0.5;
|
||||
const V3 palm = (h->smooth[0] + h->smooth[5] + h->smooth[9] + h->smooth[13] + h->smooth[17]) * 0.2;
|
||||
if (recent_id_[s] != h->id) recent_[s].clear(), recent_id_[s] = h->id;
|
||||
auto begin = [&](bool dip) {
|
||||
o.flags = (o.flags | FH_PINCH_DOWN) & ~FH_PINCH_LOST;
|
||||
++o.begins;
|
||||
o.begin_ns = uint64_t(t_ns);
|
||||
put3(o.begin_point, point);
|
||||
follow_[s] = h->id;
|
||||
open_frames_[s] = 0;
|
||||
dip_[s] = dip, dip_min_[s] = d;
|
||||
if (dip) ++dip_begins[s];
|
||||
events.push_back({s, "begin", t_ns, d, point});
|
||||
};
|
||||
o.flags |= FH_PINCH_TRACKED;
|
||||
o.hand_id = uint32_t(h->id);
|
||||
o.distance = float(d);
|
||||
o.strength = float(std::clamp((p_.end_m - d) / (p_.end_m - p_.begin_m), 0.0, 1.0));
|
||||
put3(o.point, point);
|
||||
if (!down) {
|
||||
// a close held back (palm down) has to open again before a pinch can begin, so
|
||||
// turning the hand with the fingers still closed doesn't start one
|
||||
if (d > p_.end_m) held_[s] = false;
|
||||
if (grips(h->id) || other_down(h->id)) {
|
||||
// closed, or the other pinch is down on it: no pinch until it opens
|
||||
} else if (d < p_.begin_m && !own) {
|
||||
// the thumb is on the other fingertip
|
||||
} else if (fist) {
|
||||
// a fist, not a pinch (nor a dip); like a close held back, the fingers have to open
|
||||
// again before one can begin: a fist's curls flicker over fist_curl for a frame
|
||||
if (d < p_.begin_m && !held_[s]) held_[s] = true, ++fist_held[s];
|
||||
} else if (d < p_.begin_m && !held_[s] && palm_down[s] > p_.palm_down_max) {
|
||||
held_[s] = true;
|
||||
++held_back[s];
|
||||
} else if (d < p_.begin_m && !held_[s]) {
|
||||
begin(false);
|
||||
} else if (p_.dip && own && !held_[s] && palm_down[s] <= p_.palm_down_max &&
|
||||
dip_fires(s, t_ns, d, other_d[s], palm)) {
|
||||
begin(true);
|
||||
}
|
||||
} else {
|
||||
if (dip_[s]) {
|
||||
dip_min_[s] = std::min(dip_min_[s], d);
|
||||
if (d < p_.begin_m) dip_[s] = false; // the fingers met: an ordinary pinch from here (drags)
|
||||
}
|
||||
if (dip_[s]) {
|
||||
// began on a dip and the fingers haven't met: it ends once they spring back
|
||||
if (d > std::min(p_.end_m, dip_min_[s] + p_.dip_rebound_m)) {
|
||||
if (++open_frames_[s] >= p_.end_frames) end(s, t_ns, false);
|
||||
} else {
|
||||
open_frames_[s] = 0;
|
||||
}
|
||||
} else if (d > p_.end_m) {
|
||||
// past end_m for end_frames; a drag also clearly open, or open a while (see PinchParams)
|
||||
if (open_frames_[s]++ == 0) open_since_[s] = t_ns;
|
||||
const bool drag = (t_ns - int64_t(o.begin_ns)) / 1e9 >= p_.drag_s;
|
||||
if (open_frames_[s] >= p_.end_frames &&
|
||||
(!drag || d > p_.drag_open_m || (t_ns - open_since_[s]) / 1e9 >= p_.drag_open_s - 1e-6))
|
||||
end(s, t_ns, false);
|
||||
} else if (d < p_.drag_reclose_m || (t_ns - int64_t(o.begin_ns)) / 1e9 < p_.drag_s) {
|
||||
open_frames_[s] = 0; // closed again (a drag: firmly)
|
||||
}
|
||||
}
|
||||
if (p_.dip) recent_[s].push_back({t_ns, d, palm});
|
||||
}
|
||||
}
|
||||
|
||||
bool Pinch::dip_fires(int s, int64_t t_ns, double d, double other, const V3 &palm) const {
|
||||
if (d >= p_.dip_max_m || (p_.other_finger && other >= 0 && other < p_.dip_other_min_m)) return false;
|
||||
// the highest distance within dip_win_s, and the palm's travel over the last 150 ms
|
||||
double high = 0;
|
||||
const Sample *old = nullptr;
|
||||
for (const Sample &x : recent_[s]) {
|
||||
const double age = (t_ns - x.t_ns) / 1e9;
|
||||
if (age <= p_.dip_win_s + 1e-3) high = std::max(high, x.d);
|
||||
if (!old && age <= 0.15) old = &x;
|
||||
}
|
||||
const double speed = old ? norm(palm - old->palm) / std::max((t_ns - old->t_ns) / 1e9, 1e-3) : 0;
|
||||
return high - d >= p_.dip_drop_m && d <= high * (1 - p_.dip_rel) && high <= kDipFromMax && speed < p_.dip_speed_max;
|
||||
}
|
||||
|
||||
void Pinch::end(int s, int64_t t_ns, bool lost) {
|
||||
fh_pinch_t &o = side_[s];
|
||||
o.flags = (o.flags & ~FH_PINCH_DOWN) | (lost ? FH_PINCH_LOST : 0);
|
||||
++o.ends;
|
||||
o.end_ns = uint64_t(t_ns);
|
||||
follow_[s] = 0;
|
||||
recent_[s].clear(); // a new dip needs a fresh open reference
|
||||
dip_[s] = false;
|
||||
events.push_back({s, lost ? "lost" : "end", t_ns, o.distance, {o.point[0], o.point[1], o.point[2]}});
|
||||
}
|
||||
|
||||
void Pinch::release(int64_t t_ns) {
|
||||
events.clear();
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
side_[s].flags &= ~FH_PINCH_TRACKED;
|
||||
if (side_[s].flags & FH_PINCH_DOWN) end(s, t_ns, true);
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<int> Pinch::down() const {
|
||||
std::vector<int> out;
|
||||
for (int s = 0; s < 2; ++s)
|
||||
if (side_[s].flags & FH_PINCH_DOWN) out.push_back(follow_[s]);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<int> Pinch::holding(int64_t t_ns) const {
|
||||
std::vector<int> out;
|
||||
for (int s = 0; s < 2; ++s)
|
||||
if ((side_[s].flags & FH_PINCH_DOWN) && (t_ns - int64_t(side_[s].begin_ns)) / 1e9 >= p_.grip_takeover_s)
|
||||
out.push_back(follow_[s]);
|
||||
return out;
|
||||
}
|
||||
|
||||
bool Pinch::engaged() const {
|
||||
for (const fh_pinch_t &o : side_)
|
||||
if ((o.flags & FH_PINCH_DOWN) || ((o.flags & FH_PINCH_TRACKED) && o.strength > 0.3f)) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------- grip
|
||||
|
||||
namespace {
|
||||
|
||||
V3 palm_centre(const Hand &h) {
|
||||
return (h.smooth[0] + h.smooth[5] + h.smooth[9] + h.smooth[13] + h.smooth[17]) * 0.2;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void Grip::update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns,
|
||||
const std::vector<int> &blocked) {
|
||||
void Grip::update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns) {
|
||||
events.clear();
|
||||
auto held = [&](int id) { return std::find(blocked.begin(), blocked.end(), id) != blocked.end(); };
|
||||
int taken[2] = {0, 0};
|
||||
for (int s = 0; s < 2; ++s)
|
||||
if (side_[s].flags & FH_PINCH_DOWN) taken[s] = follow_[s];
|
||||
@@ -311,8 +198,7 @@ void Grip::update(const std::vector<const Hand *> &hands, const std::vector<Seen
|
||||
if (mean > p_.end) open_ns_[s] = t_ns;
|
||||
const bool ahead = -point[2] >= p_.min_ahead_m &&
|
||||
std::atan2(-point[1], -point[2]) * 180 / M_PI <= p_.max_down_deg;
|
||||
if (most < p_.begin && ahead && thumb >= p_.thumb_off_m && !held(h->id) && open_ns_[s] &&
|
||||
(t_ns - open_ns_[s]) / 1e9 <= p_.armed_s) {
|
||||
if (most < p_.begin && ahead && thumb >= p_.thumb_off_m && open_ns_[s] && (t_ns - open_ns_[s]) / 1e9 <= p_.armed_s) {
|
||||
o.flags = (o.flags | FH_PINCH_DOWN) & ~FH_PINCH_LOST;
|
||||
++o.begins;
|
||||
o.begin_ns = uint64_t(t_ns);
|
||||
@@ -371,12 +257,7 @@ bool GesturePublisher::open(const Pinch &pinch, const Grip &grip, std::string &e
|
||||
void *m = mmap(nullptr, sizeof(fh_gestures_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
|
||||
close(fd);
|
||||
if (m == MAP_FAILED) return err = path + ": can't map it", false;
|
||||
use(static_cast<fh_gestures_t *>(m), pinch, grip);
|
||||
return true;
|
||||
}
|
||||
|
||||
void GesturePublisher::use(fh_gestures_t *mem, const Pinch &pinch, const Grip &grip) {
|
||||
out_ = mem;
|
||||
out_ = static_cast<fh_gestures_t *>(m);
|
||||
// keep the counters a previous tracker left, so a reader doesn't see them jump back
|
||||
const bool ours = !std::memcmp(out_->magic, FH_GESTURES_MAGIC, 8) && out_->version == FH_GESTURES_VERSION;
|
||||
if (!ours) {
|
||||
@@ -389,7 +270,7 @@ void GesturePublisher::use(fh_gestures_t *mem, const Pinch &pinch, const Grip &g
|
||||
// a gesture the last tracker left down (it crashed) is over: count its end, as lost
|
||||
__atomic_store_n(&out_->seq, 2 * ++seq_ - 1, __ATOMIC_RELAXED);
|
||||
__atomic_thread_fence(__ATOMIC_RELEASE);
|
||||
for (fh_pinch_t *slots : {out_->pinch, out_->grip, out_->pinch2})
|
||||
for (fh_pinch_t *slots : {out_->pinch, out_->grip})
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
fh_pinch_t &o = slots[s];
|
||||
if (o.begins == o.ends) continue;
|
||||
@@ -402,17 +283,17 @@ void GesturePublisher::use(fh_gestures_t *mem, const Pinch &pinch, const Grip &g
|
||||
out_->end_m = float(pinch.params().end_m);
|
||||
out_->grip_begin = float(grip.params().begin);
|
||||
out_->grip_end = float(grip.params().end);
|
||||
return true;
|
||||
}
|
||||
|
||||
void GesturePublisher::write(const Pinch &pinch, const Grip &grip, uint64_t capture_ns, const Pinch *middle) {
|
||||
void GesturePublisher::write(const Pinch &pinch, const Grip &grip, uint64_t capture_ns) {
|
||||
__atomic_store_n(&out_->seq, 2 * ++seq_ - 1, __ATOMIC_RELAXED);
|
||||
__atomic_thread_fence(__ATOMIC_RELEASE);
|
||||
static const fh_pinch_t none{};
|
||||
for (int g = 0; g < 3; ++g)
|
||||
for (int g = 0; g < 2; ++g)
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
// counters carry on from what's in the file (a restarted tracker starts its own at 0)
|
||||
const fh_pinch_t &in = g == 2 ? (middle ? middle->side(s) : none) : g ? grip.side(s) : pinch.side(s);
|
||||
fh_pinch_t &o = g == 2 ? out_->pinch2[s] : g ? out_->grip[s] : out_->pinch[s];
|
||||
const fh_pinch_t &in = g ? grip.side(s) : pinch.side(s);
|
||||
fh_pinch_t &o = g ? out_->grip[s] : out_->pinch[s];
|
||||
const uint32_t base_b = o.begins - last_begins_[g][s], base_e = o.ends - last_ends_[g][s];
|
||||
o = in;
|
||||
o.begins = base_b + in.begins;
|
||||
|
||||
+7
-94
@@ -5,7 +5,6 @@
|
||||
|
||||
#include "tracker.h"
|
||||
|
||||
#include <deque>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
@@ -19,27 +18,11 @@ struct PinchParams {
|
||||
int end_frames = 2; // processed frames in a row past end_m before it ends, so one
|
||||
// noisy frame doesn't drop a drag
|
||||
double grace_s = 0.25; // a pinching hand lost this long ends its pinch (FH_PINCH_LOST)
|
||||
// A drag: a pinch held drag_s or longer ends only once the tips are drag_open_m apart (for
|
||||
// end_frames, as above), or drag_open_s after they first went past end_m, unless they closed
|
||||
// again within drag_reclose_m in between. A thumb hidden for a frame or two, or fingers loosened
|
||||
// a little mid-drag, read 3.5-4 cm and dropped what was dragged (2026-10-09 headset test: drags
|
||||
// ending at 3.5-4.0 cm). Fingers hovering 3-5 cm apart still end it: on the lit recording's
|
||||
// replay (pinches held 0.3 s or more, typing among them) the ends came 33-134 ms later, at most
|
||||
// 200, where any dip under end_m starting the wait over kept one held 6 s longer. A tap ends as
|
||||
// before: its release is the click.
|
||||
double drag_s = 0.3, drag_open_m = 0.05, drag_open_s = 0.12, drag_reclose_m = 0.03;
|
||||
// A grip takes this pinch's hand (ends it, as lost) only within grip_takeover_s of its begin:
|
||||
// closing the hand into a fist passes through a pinch. Later the pinch is being held (a drag),
|
||||
// and the other fingers curling around it don't end it (3 of 28 pinches ended so, the same test).
|
||||
double grip_takeover_s = 0.3;
|
||||
// Where the distance comes from: MediaPipe's world landmarks (the model's own 3D hand
|
||||
// pose, averaged over the hand's views, at the user's hand size), or the tracker's
|
||||
// triangulated tips. The model's pose should hold up better when the fingers hide each
|
||||
// other; tomorrow's recordings will tell.
|
||||
bool triangulated = false;
|
||||
// Or the fitted skeleton's tips (track/skeleton.h), where the hand has one: for comparing
|
||||
// in ft-handreplay (--pinch-fitted); not used live yet.
|
||||
bool fitted = false;
|
||||
// No pinch begins while the palm faces down more than this (|palm normal . up| in the
|
||||
// head frame; 1 turns it off). Typing curls the thumb onto the index: in the 2026-09-30
|
||||
// lit recording, pinches that began while typing had 0.69-1.00, deliberate ones 0.00-0.50.
|
||||
@@ -48,48 +31,6 @@ struct PinchParams {
|
||||
// pinches, hand raised in front, read 0.90-0.99 too. The pointer helper now leaves out
|
||||
// gestures that begin low (hands on a desk), which it can tell with the head's pose.
|
||||
double palm_down_max = 1.0;
|
||||
// Which fingertip meets the thumb: 8 the index (the pinch), 12 the middle finger (the middle
|
||||
// pinch, fh_gestures v3: a right click or scrolling on the pointer helper's hand ray).
|
||||
int finger = 8;
|
||||
// The other fingertip the thumb could be on (0: none). A pinch begins only with the thumb
|
||||
// nearer its own fingertip than this one by at least other_margin (m): the middle pinch's
|
||||
// index tip is often within the begin distance too.
|
||||
int other_finger = 0;
|
||||
double other_margin = 0.0;
|
||||
// And only with this finger extended (0: any): its tip at least extended_min times as far from
|
||||
// the wrist as its knuckle (about 1.8-2.0 straight, under 1.2 curled). The middle pinch wants the
|
||||
// index extended: closing the hand into a grab brings the middle tip to the thumb with the index
|
||||
// curled too (6 false middle pinches in the recorder's gestures take, PR #8, all at curls 1.2-1.4).
|
||||
int extended_finger = 0;
|
||||
double extended_min = 1.5;
|
||||
// The fist guard (HANDS_PINCH_FIST_CURL; 0 turns it off): no pinch begins while the index and the
|
||||
// middle finger are both curled, each tip nearer the wrist than fist_curl times its knuckle (Grip's
|
||||
// curl measure). A fist puts the thumb tip on the curled index: in the held-out flags a median 28.5 mm
|
||||
// from it and under 20 mm on 22% of fist frames, and live ft-hands read 96% of fist frames as pinched
|
||||
// (frame-hands notes/taps.md, "Fists are partly a gesture-rule problem"). A close held back so has to
|
||||
// open past end_m before a pinch can begin: a held fist's curls flicker over the line for a frame now
|
||||
// and then. A pinch already down carries on: a drag's other fingers curling around it don't end it.
|
||||
// 1.2, not the flag analysis' 1.3 (FIST_CURL, on the teacher's 3D): the model's world landmarks read
|
||||
// pinching fingers more curled (index 1.06-1.34 at quick taps). Replays with student v11c against the
|
||||
// tap labels (2026-10-11): clear non-fist pinches lost 2 of 204 at 1.2 (4 at 1.3), the labelled fists
|
||||
// read as pinches 8 -> 5 at either; typing pinches on the desk recordings 29 -> 22.
|
||||
double fist_curl = 1.2;
|
||||
// And only on a hand tracked at least this many frames (a hand just found is often misread).
|
||||
int min_frames = 0;
|
||||
// The dip rule (HANDS_PINCH_DIP=1, off by default): a pinch also begins on a sharp dip that
|
||||
// doesn't reach begin_m. Quick taps with the hand raised often read 25-30 mm at contact and
|
||||
// spring back (the model doesn't read the touch; frame-hands notes/taps.md, "Dip detector"). It
|
||||
// begins when, within dip_win_s, the distance fell by dip_rel of its highest and by dip_drop_m,
|
||||
// to under dip_max_m, with the palm moving under dip_speed_max (m/s, over the last 150 ms) and
|
||||
// the thumb still dip_other_min_m from the other fingertip (other_finger: a hand closing into a
|
||||
// grab brings both in). It ends once the distance is dip_rebound_m over the lowest it read (or
|
||||
// past end_m) for end_frames; if it gets under begin_m it is an ordinary pinch from there on
|
||||
// (drags and all). Tuned on the probe-moves and lit recordings (frame-hands train/taps/dipsim.py);
|
||||
// in the guided tap capture's replay, quick taps at a 2 Hz click 10 -> 25 of 30 caught. The cost
|
||||
// is near misses: fingers stopped ~1 cm apart click too (5 -> 16 pinches in that segment).
|
||||
bool dip = false;
|
||||
double dip_win_s = 0.15, dip_rel = 0.4, dip_drop_m = 0.004, dip_max_m = 0.032;
|
||||
double dip_speed_max = 0.3, dip_other_min_m = 0.04, dip_rebound_m = 0.006;
|
||||
};
|
||||
|
||||
class Pinch {
|
||||
@@ -99,20 +40,13 @@ public:
|
||||
// After each processed set: the hands out of Tracker::step, the tracker's views (for
|
||||
// the world landmarks) and the capture time. Hands in `gripping` (their ids) are closed:
|
||||
// no pinch begins on them, and one that's down on them ends, as lost.
|
||||
// Hands in `blocked` (their ids: another pinch is down on them) begin no pinch, and one that's
|
||||
// down on them carries on.
|
||||
void update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns,
|
||||
const std::vector<int> &gripping = {}, const std::vector<int> &blocked = {});
|
||||
// The hands a pinch is down on now (for another pinch's `blocked`).
|
||||
std::vector<int> down() const;
|
||||
const std::vector<int> &gripping = {});
|
||||
// Ends any pinch that's down (as lost), e.g. when the tracker stops.
|
||||
void release(int64_t t_ns);
|
||||
const fh_pinch_t &side(int s) const { return side_[s]; } // 0 left, 1 right
|
||||
// A pinch is down or closing: worth tracking at the full rate.
|
||||
bool engaged() const;
|
||||
// The hands a pinch has been down on for grip_takeover_s or longer at t_ns (for Grip::update's
|
||||
// `blocked`: no grip begins on them).
|
||||
std::vector<int> holding(int64_t t_ns) const;
|
||||
|
||||
// What changed in the last update, for logs.
|
||||
struct Event {
|
||||
@@ -124,34 +58,18 @@ public:
|
||||
};
|
||||
std::vector<Event> events;
|
||||
// Both distance measures for the last update, per side (-1: no hand), for logs.
|
||||
double world_d[2] = {-1, -1}, tri_d[2] = {-1, -1}, fit_d[2] = {-1, -1};
|
||||
double other_d[2] = {-1, -1}; // the thumb to the other fingertip (other_finger), same measure
|
||||
double world_d[2] = {-1, -1}, tri_d[2] = {-1, -1};
|
||||
double palm_down[2] = {-1, -1}; // |palm normal . up| of each side's hand
|
||||
int held_back[2] = {0, 0}; // pinches that didn't begin because the palm faced down
|
||||
int fist_held[2] = {0, 0}; // closes the fist guard held back
|
||||
double curl[2][4] = {{-1, -1, -1, -1}, {-1, -1, -1, -1}}; // each side's hand: the fingers' curls, for logs
|
||||
int dip_begins[2] = {0, 0}; // pinches the dip rule began (PinchParams::dip)
|
||||
|
||||
private:
|
||||
void end(int s, int64_t t_ns, bool lost);
|
||||
// the dip rule's recent distances on each side's hand (PinchParams::dip)
|
||||
struct Sample {
|
||||
int64_t t_ns;
|
||||
double d;
|
||||
V3 palm;
|
||||
};
|
||||
bool dip_fires(int s, int64_t t_ns, double d, double other, const V3 &palm) const;
|
||||
std::deque<Sample> recent_[2];
|
||||
int recent_id_[2] = {0, 0};
|
||||
bool dip_[2] = {false, false}; // the pinch down began on a dip and hasn't reached begin_m
|
||||
double dip_min_[2] = {1, 1}; // its lowest distance so far
|
||||
PinchParams p_;
|
||||
fh_pinch_t side_[2]{};
|
||||
int follow_[2] = {0, 0}; // the hand id a pinch follows while down
|
||||
int open_frames_[2] = {0, 0};
|
||||
int64_t open_since_[2] = {0, 0}; // the first frame of this run past end_m
|
||||
int64_t seen_ns_[2] = {0, 0};
|
||||
bool held_[2] = {false, false}; // a close held back (palm down, a fist) that hasn't opened yet
|
||||
bool held_[2] = {false, false}; // a close held back (palm down) that hasn't opened yet
|
||||
};
|
||||
|
||||
struct GripParams {
|
||||
@@ -183,9 +101,7 @@ class Grip {
|
||||
public:
|
||||
explicit Grip(const GripParams &p = {}) : p_(p) {}
|
||||
const GripParams ¶ms() const { return p_; }
|
||||
// blocked: hands no grip begins on (a pinch held on them: Pinch::holding).
|
||||
void update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns,
|
||||
const std::vector<int> &blocked = {});
|
||||
void update(const std::vector<const Hand *> &hands, const std::vector<Seen> &views, int64_t t_ns);
|
||||
void release(int64_t t_ns);
|
||||
const fh_pinch_t &side(int s) const { return side_[s]; }
|
||||
// The hands gripping now (for Pinch::update).
|
||||
@@ -208,15 +124,12 @@ private:
|
||||
// Writes /run/user/UID/frametop-hands/gestures.
|
||||
class GesturePublisher {
|
||||
public:
|
||||
// middle: the middle pinches (fh_gestures v3), or none (written as zero).
|
||||
bool open(const Pinch &pinch, const Grip &grip, std::string &err);
|
||||
// Writes into `mem` instead of the gestures file (ft-handreplay --publish-dump).
|
||||
void use(fh_gestures_t *mem, const Pinch &pinch, const Grip &grip);
|
||||
void write(const Pinch &pinch, const Grip &grip, uint64_t capture_ns, const Pinch *middle = nullptr);
|
||||
void write(const Pinch &pinch, const Grip &grip, uint64_t capture_ns);
|
||||
|
||||
private:
|
||||
fh_gestures_t *out_ = nullptr;
|
||||
uint64_t seq_ = 0;
|
||||
// the counters last written, per gesture (0 pinch, 1 grip, 2 middle pinch) and side
|
||||
uint32_t last_begins_[3][2] = {}, last_ends_[3][2] = {};
|
||||
// the counters last written, per gesture (0 pinch, 1 grip) and side
|
||||
uint32_t last_begins_[2][2] = {}, last_ends_[2][2] = {};
|
||||
};
|
||||
+2
-10
@@ -82,22 +82,14 @@ SetReader::~SetReader() {
|
||||
|
||||
bool SetReader::open(const std::string &dir, std::string &err) {
|
||||
const std::string path = dir + "/sets.bin";
|
||||
dir_ = dir, part_ = 1;
|
||||
f_ = std::fopen(path.c_str(), "rb");
|
||||
return f_ ? true : (err = path + ": " + std::strerror(errno), false);
|
||||
}
|
||||
|
||||
bool SetReader::next(std::vector<fh_set_cam_t> &cams, std::vector<std::vector<uint8_t>> &pixels) {
|
||||
fh_set_hdr_t h;
|
||||
if (std::fread(&h, sizeof h, 1, f_) != 1) { // the next part, if any
|
||||
if (!std::feof(f_)) return false;
|
||||
FILE *next = std::fopen((dir_ + "/sets-" + std::to_string(part_ + 1) + ".bin").c_str(), "rb");
|
||||
if (!next) return false;
|
||||
std::fclose(f_);
|
||||
f_ = next, ++part_;
|
||||
if (std::fread(&h, sizeof h, 1, f_) != 1) return false;
|
||||
}
|
||||
if (std::memcmp(h.magic, FH_SET_MAGIC, 8) || h.ncams == 0 || h.ncams > 16) return false;
|
||||
if (std::fread(&h, sizeof h, 1, f_) != 1 || std::memcmp(h.magic, FH_SET_MAGIC, 8) || h.ncams == 0 || h.ncams > 16)
|
||||
return false;
|
||||
cams.resize(h.ncams);
|
||||
if (std::fread(cams.data(), sizeof(fh_set_cam_t), h.ncams, f_) != h.ncams) return false;
|
||||
pixels.resize(h.ncams);
|
||||
|
||||
@@ -59,8 +59,7 @@ private:
|
||||
size_t written_ = 0, dropped_ = 0, added_ = 0;
|
||||
};
|
||||
|
||||
// Reads a recording back one set at a time: DIR/sets.bin, then the hand recorder's later
|
||||
// parts (sets-2.bin, sets-3.bin, ...) if there are any.
|
||||
// Reads a recording back one set at a time.
|
||||
class SetReader {
|
||||
public:
|
||||
bool open(const std::string &dir, std::string &err);
|
||||
@@ -70,6 +69,4 @@ public:
|
||||
|
||||
private:
|
||||
FILE *f_ = nullptr;
|
||||
std::string dir_;
|
||||
int part_ = 1;
|
||||
};
|
||||
+12
-446
@@ -11,25 +11,12 @@
|
||||
// each step's time here times F as the busy time (the headset is busier live).
|
||||
// --cost: instead of timing the steps, charge each round of model calls what it
|
||||
// typically costs live (10 ms landmarks, 18 ms palms): repeatable results.
|
||||
// --cost-hand MS, --cost-palm MS: those per-round costs (with --cost). The 2026-10-08 live probe
|
||||
// (CPUs 0,1,4 beside SteamVR, 3 threads) measured about 14 and 52.
|
||||
// --timeline: per processed set, a line per hand (time, id, side, views, wrist) and per view
|
||||
// (hand, camera, presence, next crop, set index).
|
||||
// --keep-presence P: landmark presence a tracked view needs to stay (default 0.5, as new ones).
|
||||
// --reacquire S, --reacquire-tiles N, --reacquire-fast: aim the palm search at hands lost within S s
|
||||
// (Tracker::set_reacquire; default off, tiles 2).
|
||||
// --near F: Tracker::set_near (0.5). --hold S: Tracker::set_hold (0.3). --search-budget N: palm tiles per round with one hand tracked (2).
|
||||
// --drop-every S: the re-acquire test. Every S s, while two hands are tracked, one of them (left and
|
||||
// right in turn) is made invisible for 0.35 s (Tracker::blind: just past the 0.3 s the tracker
|
||||
// holds a lost hand), and the report says how long it then took to have two hands again:
|
||||
// the time to find a hand that never left the view.
|
||||
// --pair-sides: two hands seen at once get one left and one right (Tracker::set_pair_sides).
|
||||
// --misread-guard: the tracker's guards for fine-tuned landmark models (Tracker::set_misread_guard).
|
||||
// --pinch-begin M, --pinch-end M, --pinch-triangulated, --pinch-palm-down MAX, --pinch-dip 0|1,
|
||||
// --pinch-fist-curl R: the pinch detector (track/pinch.h; --pinch-dip 1: PinchParams::dip, as
|
||||
// HANDS_PINCH_DIP=1; --pinch-fist-curl: PinchParams::fist_curl, 1.2 as ft-hands, 0 off, as
|
||||
// HANDS_PINCH_FIST_CURL); the timeline gets its begin/end/lost events, both distance measures
|
||||
// and each side's finger curls (index, middle, ring, little) per set.
|
||||
// --pinch-begin M, --pinch-end M, --pinch-triangulated, --pinch-palm-down MAX: the pinch detector (track/pinch.h);
|
||||
// the timeline gets its begin/end/lost events and both distance measures per set.
|
||||
// --grip-begin R, --grip-end R: the grip detector (a closed hand; track/pinch.h); the timeline
|
||||
// gets its events and each side's finger curl per set.
|
||||
// --cams mono|color|all: which cameras to track with (default mono). color and all need a
|
||||
@@ -49,66 +36,16 @@
|
||||
// the tracker's views moving with their images (Tracker::exchange). The side check runs in every mode, and the report
|
||||
// says what it found and when: seconds into the recording, and after how long of
|
||||
// hands (from the first set with a hand in view).
|
||||
// --no-skeleton: the tracker without the skeleton fit (track/skeleton.h). With it (the
|
||||
// default) the report ends with the skeleton's measures (track/skeval.h): fit time
|
||||
// and residual, the bone-length calibration, bone-length spread, jitter, fingertip
|
||||
// lag and the elbow, each for the measured, One Euro filtered and fitted landmarks.
|
||||
// --skel NAME=VALUE: a skeleton setting (skel::Params: pos_min, pos_beta, rot_min, rot_beta,
|
||||
// ang_min, ang_beta, d_cutoff, glitch, depth_ratio, limits, iterations), repeatable.
|
||||
// --labels FILE: frame-hands' auto-labels (labels.jsonl) for this recording: the landmarks'
|
||||
// distance to them, measured, One Euro and fitted.
|
||||
// --hmd-poses FILE: a hand-recorder take's poses.jsonl: the arm model's elbow with gravity's up
|
||||
// against the head's up.
|
||||
// --publish-dump FILE: what ft-hands would have published after each processed set, for
|
||||
// Frametop's offline tests (screens/tests): a header (FHDUMP02, then the sizes of
|
||||
// fh_hands_v2_t and fh_gestures_t as two uint32), then per set the hands file's and
|
||||
// the gestures file's bytes and the set's dequeue time (int64, CLOCK_MONOTONIC: the
|
||||
// clock of a hand-recorder take's poses.jsonl; capture_ns is the cameras' clock).
|
||||
// publish_ns is the set's time plus the step's busy time.
|
||||
// --skeleton-dump FILE: with --labels, a line per hand matched to a label: time, id, stereo, then
|
||||
// the measured, fitted (unfiltered) and labelled landmarks.
|
||||
// --pinch-fitted: also run a second pinch detector on the fitted tips, and report its count.
|
||||
// The middle pinch (thumb to middle fingertip) runs beside the pinch as in ft-hands; the timeline
|
||||
// gets its events as "pinch2".
|
||||
// --ray FILE: per processed set, a line per hand with what the pointer helper's hand ray reads
|
||||
// (hands/include/fh_ray.h; pointer/tools/rayeval): the set's capture time (ns,
|
||||
// CLOCK_MONOTONIC_RAW as recorded), id, side (L/R), stereo, fitted, its side's pinch
|
||||
// down, grip down, and pinch begun in this set (0/1 each), then the 21 published points
|
||||
// (the fit's when it has one; head frame), the wrist's frame as a quaternion (x y z w;
|
||||
// 0 0 0 1 without a fit) and the elbow.
|
||||
// --tips FILE: per processed set, a line per hand for frame-hands' train/taps/tapeval.py (the pinch
|
||||
// distance against auto-labels): "H", the set's capture time (ns, the labels' t_ns), id, side
|
||||
// (L/R), views, scale, the pinch detector's thumb-index measures (world landmarks as Pinch
|
||||
// averages them, triangulated, fitted; -1 none), the same for thumb-middle, the palm normal
|
||||
// (smoothed points, pointing out of the palm) and centre, then thumb tip, index tip and
|
||||
// middle tip as measured, smoothed and fitted (head frame), then per view: camera, presence,
|
||||
// and the 2D thumb, index and middle tips (px). Pinch events go in as "E time side
|
||||
// begin|end|lost distance" (side 0 left, 1 right; "E2" for the middle pinch), and each
|
||||
// processed set's time as "S time".
|
||||
// --rate HZ: process sets at a fixed rate (the live tracker adapts: 5 Hz with no hands, 15 still, 30
|
||||
// fast or while a pinch is down or closing). --want-hz HZ: as a reader's want-hz (ft-screens
|
||||
// near a panel): the live rule, but never slower than HZ.
|
||||
// --flag-every S, --flag-at S[,S...], --flag-out DIR: flags as ft-hands makes them on `ft-handsctl flag`
|
||||
// (track/flagbuf.h; frame-hands notes/flag-failures.md has the format), with the same buffer,
|
||||
// at those times into the recording (seconds from its first set; every S: S, 2S, ...), each
|
||||
// into DIR/flag-<recording>-<seconds>. With --flag-every 14 (the 12 s before and 2 after) they
|
||||
// cover the recording end to end, for checking the offline tools against full-set labels.
|
||||
// --flag-crop PX (224), --flag-scale F (1.6), --flag-mb MB (400: unlike live, nothing dropped),
|
||||
// --flag-seconds S (12): the buffer's settings.
|
||||
// --depth FILE: per processed set, a line per hand for tools/depth_report.py: its views'
|
||||
// cameras, triangulation residual, hand scale, measured and published palm, and
|
||||
// each view's one-view palm (Tracker::single_view at the hand's scale). The
|
||||
// header has each camera's centre and focal length.
|
||||
#include "flagbuf.h"
|
||||
#include "io.h"
|
||||
#include "pinch.h"
|
||||
#include "record.h"
|
||||
#include "sides.h"
|
||||
#include "skeval.h"
|
||||
#include "tracker.h"
|
||||
|
||||
#include <json/json.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
@@ -116,7 +53,6 @@
|
||||
#include <cstring>
|
||||
#include <fstream>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <set>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
@@ -132,8 +68,6 @@ struct Track {
|
||||
int line = 0; // updates on the current unbroken run
|
||||
};
|
||||
|
||||
V3 cross3(V3 a, V3 b) { return {a[1] * b[2] - a[2] * b[1], a[2] * b[0] - a[0] * b[2], a[0] * b[1] - a[1] * b[0]}; }
|
||||
|
||||
double median(std::vector<double> v) {
|
||||
if (v.empty()) return 0;
|
||||
std::nth_element(v.begin(), v.begin() + v.size() / 2, v.end());
|
||||
@@ -151,30 +85,12 @@ int main(int argc, char **argv) {
|
||||
int oracle = 0, threads = 2;
|
||||
double slow = 1.0, from = 0, to = 1e9;
|
||||
bool cost = false;
|
||||
double cost_hand = 10, cost_palm = 18;
|
||||
Contrast palm_contrast, hand_contrast{Contrast::None}; // as ft-hands's
|
||||
double keep_presence = 0.5; // landmark presence a tracked view needs to stay
|
||||
bool misread_guard = false;
|
||||
double reacquire = 0;
|
||||
int reacquire_tiles = 2;
|
||||
bool reacquire_fast = false, pair_sides = false;
|
||||
double drop_every = 0; // seconds; 0: off
|
||||
double reacquire_clear = 0.12;
|
||||
double near_frac = 0.5, hold = 0.3;
|
||||
int search_budget1 = 2;
|
||||
bool skeleton = true, pinch_fitted = false;
|
||||
skel::Params skel_params;
|
||||
std::string labels, hmd_poses, skeleton_dump, publish_dump;
|
||||
PinchParams pinch_params;
|
||||
GripParams grip_params;
|
||||
std::string use = "mono", color_left = "color_video0", color_crop = "subtract", sides = "file";
|
||||
std::string ray_out, tips_out;
|
||||
double fixed_rate = 0, want_hz = 0;
|
||||
double flag_every = 0;
|
||||
std::vector<double> flag_at;
|
||||
std::string flag_out;
|
||||
FlagConfig flag_cfg;
|
||||
flag_cfg.max_bytes = size_t(400) << 20, flag_cfg.writer_cpus.clear();
|
||||
std::string timeline, depth, poses, models = std::string(argv[0]).substr(0, std::string(argv[0]).rfind('/') + 1) + "../models/ncnn";
|
||||
for (int i = 2; i < argc; ++i) {
|
||||
const std::string a = argv[i];
|
||||
@@ -184,56 +100,17 @@ int main(int argc, char **argv) {
|
||||
else if (a == "--timeline" && more) timeline = argv[++i];
|
||||
else if (a == "--depth" && more) depth = argv[++i];
|
||||
else if (a == "--poses" && more) poses = argv[++i];
|
||||
else if (a == "--ray" && more) ray_out = argv[++i];
|
||||
else if (a == "--tips" && more) tips_out = argv[++i];
|
||||
else if (a == "--rate" && more) fixed_rate = std::atof(argv[++i]);
|
||||
else if (a == "--want-hz" && more) want_hz = std::min(30.0, std::atof(argv[++i]));
|
||||
else if (a == "--threads" && more) threads = std::atoi(argv[++i]);
|
||||
else if (a == "--models" && more) models = argv[++i];
|
||||
else if (a == "--cost") cost = true;
|
||||
else if (a == "--cost-hand" && more) cost_hand = std::atof(argv[++i]);
|
||||
else if (a == "--cost-palm" && more) cost_palm = std::atof(argv[++i]);
|
||||
else if (a == "--keep-presence" && more) keep_presence = std::atof(argv[++i]);
|
||||
else if (a == "--misread-guard") misread_guard = true;
|
||||
else if (a == "--reacquire" && more) reacquire = std::atof(argv[++i]);
|
||||
else if (a == "--reacquire-tiles" && more) reacquire_tiles = std::atoi(argv[++i]);
|
||||
else if (a == "--reacquire-fast") reacquire_fast = true;
|
||||
else if (a == "--reacquire-clear" && more) reacquire_clear = std::atof(argv[++i]);
|
||||
else if (a == "--pair-sides") pair_sides = true;
|
||||
else if (a == "--drop-every" && more) drop_every = std::atof(argv[++i]);
|
||||
else if (a == "--near" && more) near_frac = std::atof(argv[++i]);
|
||||
else if (a == "--hold" && more) hold = std::atof(argv[++i]);
|
||||
else if (a == "--search-budget" && more) search_budget1 = std::atoi(argv[++i]);
|
||||
else if (a == "--no-skeleton") skeleton = false;
|
||||
else if (a == "--pinch-fitted") pinch_fitted = true;
|
||||
else if (a == "--labels" && more) labels = argv[++i];
|
||||
else if (a == "--hmd-poses" && more) hmd_poses = argv[++i];
|
||||
else if (a == "--skeleton-dump" && more) skeleton_dump = argv[++i];
|
||||
else if (a == "--publish-dump" && more) publish_dump = argv[++i];
|
||||
else if (a == "--skel" && more) {
|
||||
const std::string kv = argv[++i];
|
||||
const size_t eq = kv.find('=');
|
||||
const std::string k = kv.substr(0, eq);
|
||||
const double v = eq == std::string::npos ? NAN : std::atof(kv.c_str() + eq + 1);
|
||||
double *f = k == "pos_min" ? &skel_params.pos_min : k == "pos_beta" ? &skel_params.pos_beta
|
||||
: k == "rot_min" ? &skel_params.rot_min : k == "rot_beta" ? &skel_params.rot_beta
|
||||
: k == "ang_min" ? &skel_params.ang_min : k == "ang_beta" ? &skel_params.ang_beta
|
||||
: k == "d_cutoff" ? &skel_params.d_cutoff : k == "limits" ? &skel_params.limits
|
||||
: k == "depth_ratio" ? &skel_params.depth_ratio
|
||||
: k == "glitch" ? &skel_params.glitch
|
||||
: k == "warm" ? &skel_params.warm : nullptr;
|
||||
if (k == "iterations") skel_params.iterations = int(v);
|
||||
else if (f && v == v) *f = v;
|
||||
else return std::fprintf(stderr, "--skel NAME=VALUE: %s?\n", kv.c_str()), 1;
|
||||
}
|
||||
else if (a == "--cams" && more) use = argv[++i];
|
||||
else if (a == "--sides" && more) sides = argv[++i];
|
||||
else if (a == "--pinch-begin" && more) pinch_params.begin_m = std::atof(argv[++i]);
|
||||
else if (a == "--pinch-end" && more) pinch_params.end_m = std::atof(argv[++i]);
|
||||
else if (a == "--pinch-triangulated") pinch_params.triangulated = true;
|
||||
else if (a == "--pinch-palm-down" && more) pinch_params.palm_down_max = std::atof(argv[++i]);
|
||||
else if (a == "--pinch-dip" && more) pinch_params.dip = std::atoi(argv[++i]) != 0;
|
||||
else if (a == "--pinch-fist-curl" && more) pinch_params.fist_curl = std::atof(argv[++i]);
|
||||
else if (a == "--grip-begin" && more) grip_params.begin = std::atof(argv[++i]);
|
||||
else if (a == "--grip-end" && more) grip_params.end = std::atof(argv[++i]);
|
||||
else if (a == "--color-left" && more) color_left = argv[++i];
|
||||
@@ -242,21 +119,6 @@ int main(int argc, char **argv) {
|
||||
if (!Contrast::parse_pair(argv[++i], palm_contrast, hand_contrast))
|
||||
return std::fprintf(stderr, "--contrast MODE or PALM/HAND, each clahe[:CLIP]|none|stretch\n"), 1;
|
||||
}
|
||||
else if (a == "--flag-every" && more) flag_every = std::atof(argv[++i]);
|
||||
else if (a == "--flag-at" && more) {
|
||||
for (const char *p = argv[++i]; *p;) {
|
||||
char *end;
|
||||
const double v = std::strtod(p, &end);
|
||||
if (end == p) break;
|
||||
flag_at.push_back(v);
|
||||
p = *end == ',' ? end + 1 : end;
|
||||
}
|
||||
}
|
||||
else if (a == "--flag-out" && more) flag_out = argv[++i];
|
||||
else if (a == "--flag-crop" && more) flag_cfg.crop_px = std::atoi(argv[++i]);
|
||||
else if (a == "--flag-scale" && more) flag_cfg.scale = std::atof(argv[++i]);
|
||||
else if (a == "--flag-mb" && more) flag_cfg.max_bytes = size_t(std::atof(argv[++i]) * 1048576);
|
||||
else if (a == "--flag-seconds" && more) flag_cfg.before_s = std::atof(argv[++i]);
|
||||
else if (a == "--from" && more) from = std::atof(argv[++i]);
|
||||
else if (a == "--to" && more) to = std::atof(argv[++i]);
|
||||
else return std::fprintf(stderr, "unknown option %s\n", a.c_str()), 1;
|
||||
@@ -325,43 +187,8 @@ int main(int argc, char **argv) {
|
||||
};
|
||||
tracker.set_keep_presence(keep_presence);
|
||||
tracker.set_misread_guard(misread_guard);
|
||||
tracker.set_reacquire(reacquire, reacquire_tiles, reacquire_fast, reacquire_clear);
|
||||
tracker.set_pair_sides(pair_sides);
|
||||
tracker.set_near(near_frac);
|
||||
tracker.set_hold(hold);
|
||||
tracker.set_search_budget(3, search_budget1);
|
||||
std::vector<std::string> trace;
|
||||
if (tl) {
|
||||
tracker.set_trace(&trace);
|
||||
for (const std::string &line : tracker.describe_tiles()) std::fprintf(tl, "# %s\n", line.c_str());
|
||||
}
|
||||
tracker.set_skeleton(skeleton);
|
||||
tracker.skeleton().set_params(skel_params);
|
||||
skeval::Eval eval;
|
||||
eval.dump = skeleton_dump;
|
||||
if (!labels.empty() && !eval.load_labels(labels)) return std::fprintf(stderr, "%s: can't read it\n", labels.c_str()), 1;
|
||||
std::vector<std::pair<int64_t, int>> label_hits; // processed sets' times
|
||||
std::vector<std::pair<int64_t, V3>> ups; // the head's up (head frame) by CLOCK_MONOTONIC
|
||||
if (!hmd_poses.empty()) {
|
||||
std::ifstream f(hmd_poses);
|
||||
std::string line;
|
||||
Json::CharReaderBuilder b;
|
||||
while (std::getline(f, line)) {
|
||||
Json::Value v;
|
||||
std::string e;
|
||||
std::istringstream in(line);
|
||||
if (!Json::parseFromStream(b, in, &v, &e) || !v["hmd"]["ok"].asBool() || v["hmd"]["m"].size() != 12) continue;
|
||||
const Json::Value &m = v["hmd"]["m"]; // 3x4, head -> room: up in the head frame is its second row
|
||||
ups.push_back({v["t"].asInt64(), unit(V3{m[4].asDouble(), m[5].asDouble(), m[6].asDouble()})});
|
||||
}
|
||||
if (ups.empty()) return std::fprintf(stderr, "%s: no HMD poses\n", hmd_poses.c_str()), 1;
|
||||
}
|
||||
FILE *dp = depth.empty() ? nullptr : std::fopen(depth.c_str(), "w");
|
||||
FILE *pp = poses.empty() ? nullptr : std::fopen(poses.c_str(), "w");
|
||||
FILE *rp = ray_out.empty() ? nullptr : std::fopen(ray_out.c_str(), "w");
|
||||
FILE *tp = tips_out.empty() ? nullptr : std::fopen(tips_out.c_str(), "w");
|
||||
if (tp) std::fprintf(tp, "# tips v1: H t id side views scale world tri fit world2 tri2 fit2 normal3 palm3 "
|
||||
"thumb/index/middle x (measured, smoothed, fitted) x xyz, then cam presence 6 px\n");
|
||||
if (dp)
|
||||
for (auto &[name, c] : used)
|
||||
std::fprintf(dp, "# cam %s %.4f %.4f %.4f %.1f\n", name.c_str(), c.origin[0], c.origin[1], c.origin[2], c.fx);
|
||||
@@ -374,71 +201,18 @@ int main(int argc, char **argv) {
|
||||
// oracle: sets where a side's hand was findable, and where the tracker had it then
|
||||
int o_sets = 0, o_left = 0, o_right = 0, o_left_hit = 0, o_right_hit = 0, o_left_extra = 0, o_right_extra = 0;
|
||||
std::map<std::string, int> o_by_cam;
|
||||
// label-free: per oracle set, hands the full search found (distinct left/right calls, at most 2)
|
||||
// against hands the tracker published
|
||||
int o_hands = 0, o_missed = 0;
|
||||
// the re-acquire test
|
||||
double drop_next = drop_every, drop_end = -1;
|
||||
int drop_turn = 0, drops = 0;
|
||||
std::vector<double> reacq;
|
||||
double busy_ms = 0;
|
||||
// gaps: per side (the tracker's left/right call, as the live probe counts them), the time from
|
||||
// the last processed set with a hand of that side to the next one, when it's 0.3 s or more; and
|
||||
// sets with two published palms within 8 cm (one hand counted twice)
|
||||
double side_last[2] = {-1, -1};
|
||||
std::vector<double> gaps[2];
|
||||
int overlap_sets = 0;
|
||||
// jitter: how far each update's palm is from a straight line through the last two,
|
||||
// mm (steady motion cancels out; what's left is noise and real acceleration)
|
||||
std::vector<double> jit_raw, jit_sm;
|
||||
int near_face = 0, hand_updates = 0; // published palms within 20 cm of the eyes
|
||||
PinchParams middle_params = pinch_params; // as ft-hands: the middle pinch, and the two exclude each other
|
||||
middle_params.finger = 12, middle_params.other_finger = 8, middle_params.other_margin = 0.01;
|
||||
middle_params.extended_finger = 8, middle_params.min_frames = 5;
|
||||
middle_params.dip = false;
|
||||
pinch_params.other_finger = 12;
|
||||
Pinch pinch(pinch_params);
|
||||
Pinch middle(middle_params);
|
||||
PinchParams fitted_params = pinch_params;
|
||||
fitted_params.fitted = true;
|
||||
Pinch pinch_fit(fitted_params);
|
||||
int pinch_fit_begins[2] = {0, 0};
|
||||
double pinch_begin_ts[2] = {0, 0};
|
||||
std::vector<double> pinch_len[2]; // seconds, per side
|
||||
int pinch_lost = 0;
|
||||
Grip grip(grip_params);
|
||||
// --publish-dump: the files ft-hands would write, in memory, appended per set
|
||||
FILE *pubf = publish_dump.empty() ? nullptr : std::fopen(publish_dump.c_str(), "wb");
|
||||
if (!publish_dump.empty() && !pubf) return std::fprintf(stderr, "%s: can't write it\n", publish_dump.c_str()), 1;
|
||||
static fh_hands_v2_t pub_hands;
|
||||
static fh_gestures_t pub_gestures;
|
||||
Publisher pub;
|
||||
GesturePublisher pub_g;
|
||||
if (pubf) {
|
||||
pub.use(&pub_hands);
|
||||
pub_g.use(&pub_gestures, pinch, grip);
|
||||
const uint32_t sizes[2] = {uint32_t(sizeof pub_hands), uint32_t(sizeof pub_gestures)};
|
||||
std::fwrite("FHDUMP02", 1, 8, pubf);
|
||||
std::fwrite(sizes, sizeof sizes, 1, pubf);
|
||||
}
|
||||
double grip_begin_ts[2] = {0, 0};
|
||||
std::vector<double> grip_len[2];
|
||||
// --flag-*: the flag buffer, fed as ft-hands feeds it
|
||||
std::unique_ptr<FlagBuffer> flags;
|
||||
if ((flag_every > 0 || !flag_at.empty()) && flag_out.empty()) return std::fprintf(stderr, "--flag-every/--flag-at need --flag-out DIR\n"), 1;
|
||||
std::string rec_name = dir;
|
||||
while (rec_name.size() > 1 && rec_name.back() == '/') rec_name.pop_back();
|
||||
rec_name = rec_name.substr(rec_name.rfind('/') + 1);
|
||||
if (!flag_out.empty()) {
|
||||
mkdir(flag_out.c_str(), 0755);
|
||||
flags = std::make_unique<FlagBuffer>(flag_cfg);
|
||||
flags->set_static(cameras_json(used), "{\"models\": " + json_string(models) + ", \"replay\": true, \"sides\": \"" + sides +
|
||||
"\", \"cams\": \"" + use + "\", \"misread_guard\": " + (misread_guard ? "true" : "false") +
|
||||
", \"skeleton\": " + (skeleton ? "true" : "false") + "}");
|
||||
std::sort(flag_at.begin(), flag_at.end());
|
||||
}
|
||||
double flag_next_every = flag_every;
|
||||
size_t flag_next_at = 0;
|
||||
do {
|
||||
if (sides == "file")
|
||||
for (auto &[first, rename] : file_runs)
|
||||
@@ -458,8 +232,6 @@ int main(int argc, char **argv) {
|
||||
++index;
|
||||
continue;
|
||||
}
|
||||
// a pause in the recording (the hand recorder's takes pause between prompts): gaps don't span it
|
||||
const bool pause = t_prev && t - t_prev > 500'000'000;
|
||||
t_prev = t;
|
||||
if (!t0) t0 = t;
|
||||
const double ts = (t - t0) / 1e9;
|
||||
@@ -467,42 +239,19 @@ int main(int argc, char **argv) {
|
||||
if (ts < from) continue;
|
||||
if (ts > to) break;
|
||||
++nsets;
|
||||
if (pause) {
|
||||
side_last[0] = side_last[1] = -1;
|
||||
if (tl) std::fprintf(tl, "%.3f break\n", ts);
|
||||
}
|
||||
|
||||
if (t >= busy_until && t >= next_ns) {
|
||||
const auto w0 = std::chrono::steady_clock::now();
|
||||
const Stats before = tracker.stats;
|
||||
if (drop_every > 0 && drop_end < 0 && ts >= drop_next && last_out.size() == 2) {
|
||||
const Hand *h = last_out[0]->right() == bool(drop_turn) ? last_out[0] : last_out[1];
|
||||
if (tracker.blind(h->id, int64_t(t) + 350'000'000)) {
|
||||
drop_end = ts + 0.35, ++drops, drop_turn ^= 1;
|
||||
if (tl) std::fprintf(tl, "%.3f drop %d\n", ts, h->id);
|
||||
}
|
||||
drop_next = ts + drop_every;
|
||||
}
|
||||
const auto out = tracker.step(images, int64_t(t));
|
||||
if (drop_end >= 0 && ts >= drop_end && out.size() >= 2) {
|
||||
reacq.push_back(ts - drop_end);
|
||||
if (tl) std::fprintf(tl, "%.3f reacquired after %.2f s\n", ts, ts - drop_end);
|
||||
drop_end = -1, drop_next = std::max(drop_next, ts + 1.0);
|
||||
} else if (drop_end >= 0 && ts - drop_end > 10) { // gone: give up, counts as 10 s
|
||||
reacq.push_back(10);
|
||||
drop_end = -1;
|
||||
}
|
||||
double ms = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - w0).count();
|
||||
if (cost) { // repeatable: rounds of model calls at typical live costs, per thread
|
||||
const int hands = tracker.stats.hand_calls - before.hand_calls, palms = tracker.stats.palm_calls - before.palm_calls;
|
||||
ms = (2 + cost_hand * ((hands + threads - 1) / threads) + cost_palm * ((palms + threads - 1) / threads)) / slow;
|
||||
ms = (2 + 10.0 * ((hands + threads - 1) / threads) + 18.0 * ((palms + threads - 1) / threads)) / slow;
|
||||
}
|
||||
busy_ms += ms;
|
||||
busy_until = t + uint64_t(ms * slow * 1e6) + 3'000'000; // + the ring hand-off
|
||||
const std::vector<Seen> seen = tracker.views_now();
|
||||
if (tl)
|
||||
for (const std::string &e : trace) std::fprintf(tl, "%.3f ev %s\n", ts, e.c_str());
|
||||
trace.clear();
|
||||
if (first_hand_ts < 0 && !seen.empty()) first_hand_ts = ts;
|
||||
std::vector<Seen> side_views = seen;
|
||||
if (checking)
|
||||
@@ -527,120 +276,9 @@ int main(int argc, char **argv) {
|
||||
checking = false;
|
||||
}
|
||||
}
|
||||
std::vector<int> holding = pinch.holding(int64_t(t)), held2 = middle.holding(int64_t(t)); // as ft-hands
|
||||
holding.insert(holding.end(), held2.begin(), held2.end());
|
||||
grip.update(out, seen, int64_t(t), holding);
|
||||
pinch.update(out, seen, int64_t(t), grip.gripping(), middle.down());
|
||||
middle.update(out, seen, int64_t(t), grip.gripping(), pinch.down());
|
||||
if (flags) {
|
||||
flags->add(int64_t(t), int64_t(t), index, images, seen, output_json(out, &pinch, &middle, &grip));
|
||||
const bool due_at = flag_next_at < flag_at.size() && ts >= flag_at[flag_next_at];
|
||||
const bool due_every = flag_every > 0 && ts >= flag_next_every;
|
||||
if ((due_at || due_every) && !flags->waiting()) {
|
||||
char name[256], fields[1024];
|
||||
std::snprintf(name, sizeof name, "%s/flag-%s-%07.2f", flag_out.c_str(), rec_name.c_str(), ts);
|
||||
std::snprintf(fields, sizeof fields,
|
||||
"\"source\": \"ft-handreplay\", \"flag_unix\": null, \"note\": \"replay at %.2f s\", \"said_unix\": null, "
|
||||
"\"models\": %s, \"sides\": {\"names_swapped\": %s, \"swapped\": null}, \"recording\": {\"dir\": %s, "
|
||||
"\"at_s\": %.3f, \"from_s\": %.3f, \"to_s\": %.3f}",
|
||||
ts, json_string(models).c_str(), names_swapped ? "true" : "false", json_string(dir).c_str(), ts,
|
||||
ts - flag_cfg.before_s, ts + flag_cfg.after_s);
|
||||
flags->flag({name, fields}, int64_t(t));
|
||||
if (due_at) ++flag_next_at;
|
||||
if (due_every) flag_next_every += flag_every;
|
||||
}
|
||||
}
|
||||
if (tl)
|
||||
for (const Pinch::Event &e : middle.events)
|
||||
std::fprintf(tl, "%.3f pinch2 %s %s d %.3f point %+.3f %+.3f %+.3f hand %u\n", ts, e.side ? "R" : "L",
|
||||
e.what, e.distance, e.point[0], e.point[1], e.point[2], middle.side(e.side).hand_id);
|
||||
if (pinch_fitted) {
|
||||
pinch_fit.update(out, seen, int64_t(t), grip.gripping());
|
||||
for (const Pinch::Event &e : pinch_fit.events) {
|
||||
pinch_fit_begins[e.side] += std::string(e.what) == "begin";
|
||||
if (tl) std::fprintf(tl, "%.3f pinch-fitted %s %s d %.3f\n", ts, e.side ? "R" : "L", e.what, e.distance);
|
||||
}
|
||||
}
|
||||
if (skeleton) {
|
||||
V3 up{0, 0, 0};
|
||||
if (!ups.empty()) { // at the set's dequeue time, as frame-hands' armmodel.py
|
||||
uint64_t dq = UINT64_MAX;
|
||||
for (auto &c : cams)
|
||||
if (used.count(c.name) && std::string(c.name).rfind("color_", 0) != 0) dq = std::min(dq, c.dqbuf_ns);
|
||||
const auto it = std::lower_bound(ups.begin(), ups.end(), std::make_pair(int64_t(dq), V3{}),
|
||||
[](auto &a, auto &b) { return a.first < b.first; });
|
||||
if (it != ups.end() && std::llabs(it->first - int64_t(dq)) < 50'000'000) up = it->second;
|
||||
}
|
||||
for (const Hand *h : out) eval.add(*h, ts, up);
|
||||
eval.calibration(tracker.skeleton(), ts);
|
||||
label_hits.push_back({int64_t(t), int(out.size())});
|
||||
}
|
||||
if (rp)
|
||||
for (const Hand *h : out) {
|
||||
const int sd = h->right() ? 1 : 0;
|
||||
const bool pdown = (pinch.side(sd).flags & FH_PINCH_DOWN) && pinch.side(sd).hand_id == uint32_t(h->id);
|
||||
const bool gdown = (grip.side(sd).flags & FH_PINCH_DOWN) && grip.side(sd).hand_id == uint32_t(h->id);
|
||||
bool began = false;
|
||||
for (const Pinch::Event &e : pinch.events) began = began || (e.side == sd && std::string(e.what) == "begin");
|
||||
const bool fitted = h->fit.valid;
|
||||
const V3 *pts = fitted ? h->fit.pts : h->smooth;
|
||||
double q[4] = {0, 0, 0, 1};
|
||||
if (fitted) skel::to_quat(h->fit.frames[0], q);
|
||||
std::fprintf(rp, "%llu %d %s %d %d %d %d %d", (unsigned long long)t, h->id, sd ? "R" : "L",
|
||||
h->nviews >= 2, fitted, pdown, gdown, began);
|
||||
for (int k = 0; k < 21; ++k)
|
||||
for (int j = 0; j < 3; ++j) std::fprintf(rp, " %.5f", pts[k][j]);
|
||||
std::fprintf(rp, " %.5f %.5f %.5f %.5f", q[0], q[1], q[2], q[3]);
|
||||
for (int j = 0; j < 3; ++j) std::fprintf(rp, " %.4f", fitted ? h->fit.elbow[j] : 0.0);
|
||||
std::fputc('\n', rp);
|
||||
}
|
||||
// as ft-hands (track/main.cpp), or a fixed rate (--rate)
|
||||
next_ns = t + uint64_t(((fixed_rate > 0 ? 1 / fixed_rate
|
||||
: std::min({tracker.interval(),
|
||||
pinch.engaged() || middle.engaged() || grip.engaged() ? 1 / 30.0 : 1.0,
|
||||
want_hz > 0 ? 1 / want_hz : 1.0})) -
|
||||
0.005) * 1e9);
|
||||
if (tp) {
|
||||
std::fprintf(tp, "S %llu\n", (unsigned long long)t);
|
||||
for (const Pinch::Event &e : pinch.events)
|
||||
std::fprintf(tp, "E %llu %d %s %.4f\n", (unsigned long long)t, e.side, e.what, e.distance);
|
||||
for (const Pinch::Event &e : middle.events)
|
||||
std::fprintf(tp, "E2 %llu %d %s %.4f\n", (unsigned long long)t, e.side, e.what, e.distance);
|
||||
for (const Hand *h : out) {
|
||||
// the pinch detector's measures (Pinch::update's measure) for this hand
|
||||
auto measure = [&](int f, double m[3]) {
|
||||
double sum = 0;
|
||||
int n = 0;
|
||||
for (const Seen &v : seen) {
|
||||
if (v.hand != h->id) continue;
|
||||
const V3 a{v.lm.world[4][0], v.lm.world[4][1], v.lm.world[4][2]};
|
||||
const V3 b{v.lm.world[f][0], v.lm.world[f][1], v.lm.world[f][2]};
|
||||
sum += norm(a - b), ++n;
|
||||
}
|
||||
m[0] = n ? sum / n * h->scale : -1;
|
||||
m[1] = norm(h->pts[4] - h->pts[f]);
|
||||
m[2] = h->fit.valid ? norm(h->fit.pts[4] - h->fit.pts[f]) : -1;
|
||||
};
|
||||
double m8[3], m12[3];
|
||||
measure(8, m8), measure(12, m12);
|
||||
const V3 *p = h->smooth;
|
||||
V3 nrm = cross3(p[5] - p[0], p[17] - p[0]);
|
||||
nrm = norm(nrm) > 0 ? nrm * ((h->right() ? 1.0 : -1.0) / norm(nrm)) : V3{0, 0, 0};
|
||||
const V3 palm = (p[0] + p[5] + p[9] + p[13] + p[17]) * 0.2;
|
||||
std::fprintf(tp, "H %llu %d %s %d %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f",
|
||||
(unsigned long long)t, h->id, h->right() ? "R" : "L", h->nviews, h->scale, m8[0], m8[1], m8[2],
|
||||
m12[0], m12[1], m12[2], nrm[0], nrm[1], nrm[2], palm[0], palm[1], palm[2]);
|
||||
for (int k : {4, 8, 12})
|
||||
for (const V3 *src : {h->pts, h->smooth, h->fit.valid ? h->fit.pts : nullptr})
|
||||
for (int j = 0; j < 3; ++j) std::fprintf(tp, " %.4f", src ? src[k][j] : NAN);
|
||||
for (const Seen &v : seen) {
|
||||
if (v.hand != h->id) continue;
|
||||
std::fprintf(tp, " %s %.3f", v.cam.c_str(), v.lm.presence);
|
||||
for (int k : {4, 8, 12}) std::fprintf(tp, " %.1f %.1f", v.lm.pts[k][0], v.lm.pts[k][1]);
|
||||
}
|
||||
std::fputc('\n', tp);
|
||||
}
|
||||
}
|
||||
grip.update(out, seen, int64_t(t));
|
||||
pinch.update(out, seen, int64_t(t), grip.gripping());
|
||||
next_ns = t + uint64_t((std::min(tracker.interval(), pinch.engaged() || grip.engaged() ? 1 / 30.0 : 1.0) - 0.005) * 1e9);
|
||||
for (const Pinch::Event &e : grip.events) {
|
||||
if (std::string(e.what) == "begin") grip_begin_ts[e.side] = ts;
|
||||
else grip_len[e.side].push_back(ts - grip_begin_ts[e.side]);
|
||||
@@ -656,21 +294,9 @@ int main(int argc, char **argv) {
|
||||
e.what, e.distance, e.point[0], e.point[1], e.point[2], pinch.side(e.side).hand_id);
|
||||
}
|
||||
if (tl && (pinch.world_d[0] >= 0 || pinch.world_d[1] >= 0)) // both measures, for choosing one
|
||||
std::fprintf(tl, "%.3f pinchd L world %.3f tri %.3f R world %.3f tri %.3f palm %.2f %.2f curl L %.2f %.2f %.2f %.2f R %.2f %.2f %.2f %.2f\n",
|
||||
ts, pinch.world_d[0], pinch.tri_d[0], pinch.world_d[1], pinch.tri_d[1], pinch.palm_down[0],
|
||||
pinch.palm_down[1], pinch.curl[0][0], pinch.curl[0][1], pinch.curl[0][2], pinch.curl[0][3],
|
||||
pinch.curl[1][0], pinch.curl[1][1], pinch.curl[1][2], pinch.curl[1][3]);
|
||||
if (pubf) {
|
||||
pub.write(out, t);
|
||||
pub_g.write(pinch, grip, t);
|
||||
pub_hands.v1.publish_ns = pub_gestures.publish_ns = busy_until;
|
||||
std::fwrite(&pub_hands, sizeof pub_hands, 1, pubf);
|
||||
std::fwrite(&pub_gestures, sizeof pub_gestures, 1, pubf);
|
||||
int64_t dq = INT64_MAX;
|
||||
for (auto &c : cams)
|
||||
if (used.count(c.name) && std::string(c.name).rfind("color_", 0) != 0) dq = std::min(dq, int64_t(c.dqbuf_ns));
|
||||
std::fwrite(&dq, sizeof dq, 1, pubf);
|
||||
}
|
||||
std::fprintf(tl, "%.3f pinchd L world %.3f tri %.3f R world %.3f tri %.3f palm %.2f %.2f\n", ts,
|
||||
pinch.world_d[0], pinch.tri_d[0], pinch.world_d[1], pinch.tri_d[1], pinch.palm_down[0],
|
||||
pinch.palm_down[1]);
|
||||
++processed;
|
||||
last_out = out;
|
||||
bool l = false, r = false;
|
||||
@@ -692,8 +318,8 @@ int main(int argc, char **argv) {
|
||||
if (!tr.sets) tr.first = ts;
|
||||
tr.last = ts, ++tr.sets, tr.left += h->pts[0][0] < 0;
|
||||
if (tl)
|
||||
std::fprintf(tl, "%.3f %d %s %d %+.3f %+.3f %+.3f hand %s\n", ts, h->id, h->pts[0][0] < 0 ? "L" : "R", h->nviews,
|
||||
h->pts[0][0], h->pts[0][1], h->pts[0][2], h->right() ? "R" : "L");
|
||||
std::fprintf(tl, "%.3f %d %s %d %+.3f %+.3f %+.3f\n", ts, h->id, h->pts[0][0] < 0 ? "L" : "R", h->nviews,
|
||||
h->pts[0][0], h->pts[0][1], h->pts[0][2]);
|
||||
if (pp) {
|
||||
double world[21][3] = {};
|
||||
int n = 0;
|
||||
@@ -735,20 +361,6 @@ int main(int argc, char **argv) {
|
||||
std::fprintf(tl, "%.3f view %d %s presence %.2f roi %.0f %.0f %.0f %.3f set %d\n", ts, v.hand, v.cam.c_str(),
|
||||
v.lm.presence, v.roi.center[0], v.roi.center[1], v.roi.size, v.roi.rotation, index);
|
||||
left += l, right += r, both += l && r;
|
||||
bool hl = false, hr = false;
|
||||
for (const Hand *h : out) (h->right() ? hr : hl) = true;
|
||||
for (int k = 0; k < 2; ++k)
|
||||
if (k ? hr : hl) {
|
||||
if (side_last[k] >= 0 && ts - side_last[k] >= 0.3) gaps[k].push_back(ts - side_last[k]);
|
||||
side_last[k] = ts;
|
||||
}
|
||||
{
|
||||
bool close = false;
|
||||
auto palm = [](const V3 *p) { return (p[0] + p[5] + p[9] + p[13] + p[17]) * 0.2; };
|
||||
for (size_t a = 0; a < out.size(); ++a)
|
||||
for (size_t b = a + 1; b < out.size(); ++b) close = close || norm(palm(out[a]->pts) - palm(out[b]->pts)) < 0.08;
|
||||
overlap_sets += close;
|
||||
}
|
||||
++hist[std::min<size_t>(out.size(), 2)];
|
||||
}
|
||||
|
||||
@@ -760,17 +372,6 @@ int main(int argc, char **argv) {
|
||||
for (const Seen &s : seen) {
|
||||
(s.wrist[0] < 0 ? l : r) = true;
|
||||
++o_by_cam[s.cam + (s.wrist[0] < 0 ? " L" : " R")];
|
||||
if (tl)
|
||||
std::fprintf(tl, "%.3f oracle %s tile %d presence %.2f right %.2f wrist %+.3f %+.3f %+.3f roi %.0f %.0f %.0f\n", ts,
|
||||
s.cam.c_str(), s.tile, s.lm.presence, s.lm.right, s.wrist[0], s.wrist[1], s.wrist[2],
|
||||
s.roi.center[0], s.roi.center[1], s.roi.size);
|
||||
}
|
||||
if (tl) std::fprintf(tl, "%.3f oracle-done %zu\n", ts, seen.size());
|
||||
{
|
||||
bool ol = false, orr = false;
|
||||
for (const Seen &s : seen) (s.lm.right > 0.5 ? orr : ol) = true;
|
||||
const int n = int(ol) + int(orr);
|
||||
o_hands += n, o_missed += std::max(0, n - int(std::min<size_t>(last_out.size(), 2)));
|
||||
}
|
||||
bool tl_ = false, tr_ = false;
|
||||
for (const Hand *h : last_out) (h->pts[0][0] < 0 ? tl_ : tr_) = true;
|
||||
@@ -784,9 +385,6 @@ int main(int argc, char **argv) {
|
||||
if (tl) std::fclose(tl);
|
||||
if (dp) std::fclose(dp);
|
||||
if (pp) std::fclose(pp);
|
||||
if (pubf) std::fclose(pubf);
|
||||
if (rp) std::fclose(rp);
|
||||
if (tp) std::fclose(tp);
|
||||
|
||||
const double secs = nsets > 1 ? nsets / 30.0 : 0;
|
||||
const Stats &s = tracker.stats;
|
||||
@@ -802,21 +400,6 @@ int main(int argc, char **argv) {
|
||||
for (double d : lens[k]) total += d;
|
||||
std::printf("%s tracks: %zu, median %.1f s, total %.0f s\n", k ? "right" : "left ", lens[k].size(), median(lens[k]), total);
|
||||
}
|
||||
for (int k = 0; k < 2; ++k) {
|
||||
double total = 0, longest = 0;
|
||||
for (double g : gaps[k]) total += g, longest = std::max(longest, g);
|
||||
std::printf("%s hand gaps >= 0.3 s: %zu, total %.1f s, longest %.1f s\n", k ? "right" : "left ", gaps[k].size(), total, longest);
|
||||
}
|
||||
std::printf("sets with two hands within 8 cm: %d\n", overlap_sets);
|
||||
if (drop_every > 0) {
|
||||
std::sort(reacq.begin(), reacq.end());
|
||||
double sum = 0;
|
||||
for (double r : reacq) sum += r;
|
||||
auto q = [&](double f) { return reacq.empty() ? 0.0 : reacq[std::min(reacq.size() - 1, size_t(f * reacq.size()))]; };
|
||||
std::printf("re-acquire test: %d drops, %zu back; after the blind 0.35 s: median %.2f s, 75%% %.2f s, 90%% %.2f s, max %.2f s, "
|
||||
"mean %.2f s\n", drops, reacq.size(), q(0.5), q(0.75), q(0.9), reacq.empty() ? 0 : reacq.back(),
|
||||
reacq.empty() ? 0 : sum / reacq.size());
|
||||
}
|
||||
std::printf("views lost %d, handoff misses %d, dups %d, splits %d; hands new %d, merged %d, forgotten %d\n", s.lost,
|
||||
s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten);
|
||||
std::printf("model calls: palm %d (%.1f/s), hand %d (%.1f/s)\n", s.palm_calls, s.palm_calls / std::max(secs, 1e-9),
|
||||
@@ -844,10 +427,6 @@ int main(int argc, char **argv) {
|
||||
pinch_len[0].size(), pinch_len[0].empty() ? 0 : pinch_len[0][pinch_len[0].size() / 2], pinch_len[1].size(),
|
||||
pinch_len[1].empty() ? 0 : pinch_len[1][pinch_len[1].size() / 2], pinch_lost, pinch.held_back[0],
|
||||
pinch.held_back[1]);
|
||||
std::printf("fist guard (index and middle curled under %.2f; 0 off): held back %d closes left, %d right\n",
|
||||
pinch_params.fist_curl, pinch.fist_held[0], pinch.fist_held[1]);
|
||||
if (pinch_params.dip)
|
||||
std::printf("pinches begun on a dip (--pinch-dip): left %d, right %d\n", pinch.dip_begins[0], pinch.dip_begins[1]);
|
||||
for (int k = 0; k < 2; ++k) std::sort(grip_len[k].begin(), grip_len[k].end());
|
||||
std::printf("grips (curl under %.2f, open over %.2f): left %zu (median %.2f s), right %zu (median %.2f s)\n",
|
||||
grip_params.begin, grip_params.end, grip_len[0].size(),
|
||||
@@ -857,7 +436,7 @@ int main(int argc, char **argv) {
|
||||
std::sort(jit_sm.begin(), jit_sm.end());
|
||||
if (!jit_raw.empty())
|
||||
std::printf("palm jitter (off a straight line through the last two updates): measured median %.1f mm, 90%% %.1f mm; "
|
||||
"One Euro median %.1f mm, 90%% %.1f mm\n", jit_raw[jit_raw.size() / 2], jit_raw[jit_raw.size() * 9 / 10],
|
||||
"published median %.1f mm, 90%% %.1f mm\n", jit_raw[jit_raw.size() / 2], jit_raw[jit_raw.size() * 9 / 10],
|
||||
jit_sm[jit_sm.size() / 2], jit_sm[jit_sm.size() * 9 / 10]);
|
||||
if (!side_check.usable())
|
||||
std::printf("side cameras: not both in this recording\n");
|
||||
@@ -865,27 +444,14 @@ int main(int argc, char **argv) {
|
||||
std::printf("%sside cameras %s: %s\n", side_report.c_str(), side_report.empty() ? "undecided" : "at the end",
|
||||
checking || side_report.empty() ? side_check.summary().c_str()
|
||||
: names_swapped ? "exchanged from the recorded names" : "as recorded");
|
||||
if (pinch_fitted)
|
||||
std::printf("pinches on the fitted tips (same thresholds): left %d, right %d\n", pinch_fit_begins[0], pinch_fit_begins[1]);
|
||||
if (o_sets) {
|
||||
std::printf("oracle, %d sets: a left hand findable in %d, the tracker had it in %d (%.0f%%); right %d, had %d (%.0f%%)\n",
|
||||
o_sets, o_left, o_left_hit, 100.0 * o_left_hit / std::max(o_left, 1), o_right, o_right_hit,
|
||||
100.0 * o_right_hit / std::max(o_right, 1));
|
||||
std::printf(" tracker had a hand the full search didn't find: left %d, right %d\n", o_left_extra, o_right_extra);
|
||||
std::printf(" hands the full search found: %d, the tracker had %d fewer (%.1f%% missed)\n", o_hands, o_missed,
|
||||
100.0 * o_missed / std::max(o_hands, 1));
|
||||
std::printf(" found by camera:");
|
||||
for (auto &[k, n] : o_by_cam) std::printf(" %s %d", k.c_str(), n);
|
||||
std::printf("\n");
|
||||
}
|
||||
if (skeleton) eval.report(tracker.skeleton(), int64_t(t0), label_hits);
|
||||
if (flags) {
|
||||
flags->finish();
|
||||
flags->wait();
|
||||
const FlagBuffer::Stats fs = flags->stats();
|
||||
std::printf("flags: %zu written to %s, %zu crops (%zu ghosts), %.0f us a crop, %zu dropped%s%s\n", fs.flags, flag_out.c_str(),
|
||||
fs.crops, fs.ghosts, fs.crop_calls ? fs.crop_us / fs.crop_calls : 0.0, fs.dropped,
|
||||
fs.last_error.empty() ? "" : "; error: ", fs.last_error.c_str());
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@@ -1,716 +0,0 @@
|
||||
#include "skeleton.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
|
||||
namespace skel {
|
||||
|
||||
// ------------------------------------------------------------------------------ rotations
|
||||
|
||||
V3 Rot::operator*(V3 v) const {
|
||||
return {m[0][0] * v[0] + m[0][1] * v[1] + m[0][2] * v[2], m[1][0] * v[0] + m[1][1] * v[1] + m[1][2] * v[2],
|
||||
m[2][0] * v[0] + m[2][1] * v[1] + m[2][2] * v[2]};
|
||||
}
|
||||
|
||||
Rot Rot::operator*(const Rot &o) const {
|
||||
Rot r;
|
||||
for (int i = 0; i < 3; ++i)
|
||||
for (int j = 0; j < 3; ++j) r.m[i][j] = m[i][0] * o.m[0][j] + m[i][1] * o.m[1][j] + m[i][2] * o.m[2][j];
|
||||
return r;
|
||||
}
|
||||
|
||||
Rot Rot::t() const {
|
||||
Rot r;
|
||||
for (int i = 0; i < 3; ++i)
|
||||
for (int j = 0; j < 3; ++j) r.m[i][j] = m[j][i];
|
||||
return r;
|
||||
}
|
||||
|
||||
Rot exp_rot(V3 w) {
|
||||
const double a = norm(w);
|
||||
Rot r;
|
||||
if (a < 1e-12) return r;
|
||||
const V3 k = w * (1 / a);
|
||||
const double c = std::cos(a), s = std::sin(a), v = 1 - c;
|
||||
r.m[0][0] = c + k[0] * k[0] * v, r.m[0][1] = k[0] * k[1] * v - k[2] * s, r.m[0][2] = k[0] * k[2] * v + k[1] * s;
|
||||
r.m[1][0] = k[1] * k[0] * v + k[2] * s, r.m[1][1] = c + k[1] * k[1] * v, r.m[1][2] = k[1] * k[2] * v - k[0] * s;
|
||||
r.m[2][0] = k[2] * k[0] * v - k[1] * s, r.m[2][1] = k[2] * k[1] * v + k[0] * s, r.m[2][2] = c + k[2] * k[2] * v;
|
||||
return r;
|
||||
}
|
||||
|
||||
V3 log_rot(const Rot &r) {
|
||||
const double c = std::clamp((r.m[0][0] + r.m[1][1] + r.m[2][2] - 1) / 2, -1.0, 1.0), a = std::acos(c);
|
||||
const V3 v{r.m[2][1] - r.m[1][2], r.m[0][2] - r.m[2][0], r.m[1][0] - r.m[0][1]};
|
||||
if (a < 1e-6) return v * 0.5;
|
||||
if (a > M_PI - 1e-4) { // near a half turn: the axis from the diagonal
|
||||
int i = 0;
|
||||
for (int k = 1; k < 3; ++k)
|
||||
if (r.m[k][k] > r.m[i][i]) i = k;
|
||||
V3 axis{};
|
||||
axis[i] = std::sqrt(std::max(0.0, (r.m[i][i] + 1) / 2));
|
||||
for (int k = 0; k < 3; ++k)
|
||||
if (k != i) axis[k] = (r.m[k][i] + r.m[i][k]) / (4 * axis[i]);
|
||||
return unit(axis) * a;
|
||||
}
|
||||
return v * (a / (2 * std::sin(a)));
|
||||
}
|
||||
|
||||
void to_quat(const Rot &r, double q[4]) {
|
||||
const auto &m = r.m;
|
||||
const double tr = m[0][0] + m[1][1] + m[2][2];
|
||||
if (tr > 0) {
|
||||
const double s = std::sqrt(tr + 1) * 2;
|
||||
q[3] = s / 4, q[0] = (m[2][1] - m[1][2]) / s, q[1] = (m[0][2] - m[2][0]) / s, q[2] = (m[1][0] - m[0][1]) / s;
|
||||
} else if (m[0][0] > m[1][1] && m[0][0] > m[2][2]) {
|
||||
const double s = std::sqrt(1 + m[0][0] - m[1][1] - m[2][2]) * 2;
|
||||
q[3] = (m[2][1] - m[1][2]) / s, q[0] = s / 4, q[1] = (m[0][1] + m[1][0]) / s, q[2] = (m[0][2] + m[2][0]) / s;
|
||||
} else if (m[1][1] > m[2][2]) {
|
||||
const double s = std::sqrt(1 + m[1][1] - m[0][0] - m[2][2]) * 2;
|
||||
q[3] = (m[0][2] - m[2][0]) / s, q[0] = (m[0][1] + m[1][0]) / s, q[1] = s / 4, q[2] = (m[1][2] + m[2][1]) / s;
|
||||
} else {
|
||||
const double s = std::sqrt(1 + m[2][2] - m[0][0] - m[1][1]) * 2;
|
||||
q[3] = (m[1][0] - m[0][1]) / s, q[0] = (m[0][2] + m[2][0]) / s, q[1] = (m[1][2] + m[2][1]) / s, q[2] = s / 4;
|
||||
}
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
V3 cross(V3 a, V3 b) { return {a[1] * b[2] - a[2] * b[1], a[2] * b[0] - a[0] * b[2], a[0] * b[1] - a[1] * b[0]}; }
|
||||
V3 mirror(V3 v, int side) { return {side * v[0], v[1], v[2]}; }
|
||||
|
||||
Rot from_cols(V3 x, V3 y, V3 z) {
|
||||
Rot r;
|
||||
for (int i = 0; i < 3; ++i) r.m[i][0] = x[i], r.m[i][1] = y[i], r.m[i][2] = z[i];
|
||||
return r;
|
||||
}
|
||||
|
||||
// Rotation about one of the frame's own axes.
|
||||
Rot axis_rot(int axis, double a) {
|
||||
Rot r;
|
||||
const double c = std::cos(a), s = std::sin(a);
|
||||
const int i = (axis + 1) % 3, j = (axis + 2) % 3;
|
||||
r.m[i][i] = c, r.m[i][j] = -s, r.m[j][i] = s, r.m[j][j] = c;
|
||||
return r;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- the hand model's tables
|
||||
|
||||
// The palm's landmarks (rigid), and each chain: the thumb from its CMC (1), the fingers from
|
||||
// their knuckles.
|
||||
const int kPalmIds[6] = {0, 1, 5, 9, 13, 17};
|
||||
const int kChainBase[5] = {1, 5, 9, 13, 17};
|
||||
const int kChainQ[5] = {0, 6, 11, 16, 21}; // first angle in Fit::q
|
||||
|
||||
struct Dof {
|
||||
int joint; // 0, 1, 2 along the chain
|
||||
int axis; // 0 x: flexion (positive toward the palm); 1 y: abduction (positive toward the
|
||||
// thumb's side); 2 z: twist about the bone
|
||||
double lo, hi;
|
||||
};
|
||||
// Angles in radians. Abduction is applied before flexion at a joint, so a bent finger keeps
|
||||
// its spread.
|
||||
constexpr double D = M_PI / 180;
|
||||
// How far a chain's base (a knuckle, the thumb's CMC) may sit off the rigid palm: a prior's
|
||||
// sigma and a hard limit, m.
|
||||
constexpr double kBaseSigma = 0.004, kBaseMax = 0.015;
|
||||
const Dof kThumb[6] = {{0, 1, -50 * D, 50 * D}, {0, 0, -50 * D, 50 * D}, {0, 2, -35 * D, 35 * D}, // CMC
|
||||
{1, 1, -20 * D, 20 * D}, {1, 0, -30 * D, 80 * D}, // MCP
|
||||
{2, 0, -30 * D, 90 * D}}; // IP
|
||||
// Fingers get a little twist at the knuckle too: their flexion planes don't quite follow the
|
||||
// palm's normal (in the 2026-10-03 gestures take, without it the fitted index tip sat a median
|
||||
// 8 mm across the finger from the measured one).
|
||||
const Dof kFinger[4][5] = {
|
||||
{{0, 1, -25 * D, 25 * D}, {0, 0, -35 * D, 95 * D}, {0, 2, -25 * D, 25 * D}, {1, 0, -10 * D, 110 * D}, {2, 0, -20 * D, 90 * D}},
|
||||
{{0, 1, -25 * D, 25 * D}, {0, 0, -35 * D, 95 * D}, {0, 2, -25 * D, 25 * D}, {1, 0, -10 * D, 110 * D}, {2, 0, -20 * D, 90 * D}},
|
||||
{{0, 1, -25 * D, 25 * D}, {0, 0, -35 * D, 95 * D}, {0, 2, -25 * D, 25 * D}, {1, 0, -10 * D, 110 * D}, {2, 0, -20 * D, 90 * D}},
|
||||
{{0, 1, -30 * D, 30 * D}, {0, 0, -35 * D, 95 * D}, {0, 2, -25 * D, 25 * D}, {1, 0, -10 * D, 110 * D}, {2, 0, -20 * D, 90 * D}},
|
||||
};
|
||||
|
||||
const Dof *chain_dofs(int c, int *n) {
|
||||
*n = c == 0 ? 6 : 5;
|
||||
return c == 0 ? kThumb : kFinger[c - 1];
|
||||
}
|
||||
|
||||
// The rotation a dof's angle makes about its axis: flexion turns the bone from -Z toward -Y,
|
||||
// and abduction and twist are mirrored on a left hand so the same angles mean the same motion.
|
||||
double dof_angle(const Dof &d, double q, int side) { return d.axis == 0 ? -q : side * q; }
|
||||
double dof_sign(const Dof &d, int side) { return d.axis == 0 ? -1.0 : double(side); }
|
||||
|
||||
// The model's average hand (MediaPipe's world landmarks, medians over the 2026-09-30 lit
|
||||
// recording, at Hand::scale 1), in the right hand's palm frame (x to the little finger,
|
||||
// y out of the back, z toward the wrist), metres.
|
||||
const V3 kCanonPalm[6] = {{0, 0, 0},
|
||||
{-0.0149, -0.0142, -0.0158},
|
||||
{-0.0172, -0.0043, -0.0657},
|
||||
{0, 0, -0.0706},
|
||||
{0.0162, 0, -0.0646},
|
||||
{0.0286, -0.0043, -0.0559}};
|
||||
const double kCanonPhalanx[15] = {0.0266, 0.0216, 0.0201, 0.0237, 0.0175, 0.0165, 0.0235, 0.0166,
|
||||
0.0176, 0.0214, 0.0163, 0.0134, 0.0168, 0.0109, 0.0108};
|
||||
// The thumb's rest direction from its CMC, and the way its MCP bends (same data), right hand.
|
||||
const V3 kThumbRest{-0.50, -0.48, -0.70};
|
||||
const V3 kThumbFlex{0.80, -0.40, -0.37};
|
||||
|
||||
int phalanx_bone(int k) { return kChainBase[k / 3] + k % 3; } // phalanx k -> bone index
|
||||
|
||||
// A chain's rest frame in the palm frame (mirrored for side): -Z along the resting bone.
|
||||
Rot rest_frame(const Shape &s, int c, int side) {
|
||||
V3 z, y;
|
||||
if (c == 0) {
|
||||
const V3 r = unit(kThumbRest), f = kThumbFlex;
|
||||
z = r * -1;
|
||||
y = unit(f - r * dot(f, r)) * -1;
|
||||
} else {
|
||||
// halfway between the metacarpal and the palm's axis, in the palm's plane
|
||||
V3 m = s.palm[kChainBase[c]];
|
||||
m[1] = 0;
|
||||
z = unit(unit(m) + V3{0, 0, -1}) * -1;
|
||||
y = {0, 1, 0};
|
||||
}
|
||||
z = mirror(z, side), y = mirror(y, side);
|
||||
return from_cols(cross(y, z), y, z);
|
||||
}
|
||||
|
||||
struct ChainState {
|
||||
V3 base; // the chain's first landmark (on the palm, plus its offset)
|
||||
V3 pts[3]; // the chain's landmarks
|
||||
Rot frames[3]; // each joint's frame
|
||||
V3 axis[6], pivot[6];
|
||||
};
|
||||
|
||||
// rest: rest_frame(s, c, side), which the caller keeps for the chain's iterations
|
||||
void chain_fk(const Shape &s, int c, int side, const Rot &rest, const Rot &R, V3 t, const double *q, V3 off, ChainState &st) {
|
||||
int n;
|
||||
const Dof *dofs = chain_dofs(c, &n);
|
||||
const int base = kChainBase[c];
|
||||
Rot F = R * rest;
|
||||
V3 p = R * mirror(s.palm[base], side) + t + off;
|
||||
st.base = p;
|
||||
int d = 0;
|
||||
for (int j = 0; j < 3; ++j) {
|
||||
for (; d < n && dofs[d].joint == j; ++d) {
|
||||
st.axis[d] = F.col(dofs[d].axis);
|
||||
st.pivot[d] = p;
|
||||
F = F * axis_rot(dofs[d].axis, dof_angle(dofs[d], q[d], side));
|
||||
}
|
||||
st.frames[j] = F;
|
||||
p = p - F.col(2) * s.len[base + j];
|
||||
st.pts[j] = p;
|
||||
}
|
||||
}
|
||||
|
||||
// Point weights: W = a I + b d d^T, d the line of sight (depth is the noisy direction).
|
||||
struct Weight {
|
||||
double a, b;
|
||||
V3 d;
|
||||
V3 apply(V3 r) const { return r * a + d * (b * dot(d, r)); }
|
||||
};
|
||||
|
||||
Weight weight(V3 y, V3 eye, double sl, double sd) {
|
||||
return {1 / (sl * sl), 1 / (sd * sd) - 1 / (sl * sl), unit(y - eye)};
|
||||
}
|
||||
|
||||
// Solves H x = -g for small symmetric positive definite H (Cholesky). False if it isn't.
|
||||
constexpr int kMaxP = 9; // a chain's parameters: up to 6 angles and the base's offset
|
||||
bool solve_spd(double H[kMaxP][kMaxP], const double *g, double *x, int n) {
|
||||
double L[kMaxP][kMaxP] = {};
|
||||
for (int i = 0; i < n; ++i)
|
||||
for (int j = 0; j <= i; ++j) {
|
||||
double s = H[i][j];
|
||||
for (int k = 0; k < j; ++k) s -= L[i][k] * L[j][k];
|
||||
if (i == j) {
|
||||
if (s <= 1e-18) return false;
|
||||
L[i][i] = std::sqrt(s);
|
||||
} else {
|
||||
L[i][j] = s / L[j][j];
|
||||
}
|
||||
}
|
||||
double y[kMaxP];
|
||||
for (int i = 0; i < n; ++i) {
|
||||
double s = -g[i];
|
||||
for (int k = 0; k < i; ++k) s -= L[i][k] * y[k];
|
||||
y[i] = s / L[i][i];
|
||||
}
|
||||
for (int i = n - 1; i >= 0; --i) {
|
||||
double s = y[i];
|
||||
for (int k = i + 1; k < n; ++k) s -= L[k][i] * x[k];
|
||||
x[i] = s / L[i][i];
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// The palm frame straight from measured landmarks (wrist, index, middle and little knuckles).
|
||||
bool measured_frame(const V3 p[21], int side, Rot &R) {
|
||||
const V3 f0 = p[9] - p[0];
|
||||
if (norm(f0) < 0.02) return false;
|
||||
const V3 f = unit(f0);
|
||||
V3 a = p[5] - p[17];
|
||||
a = a - f * dot(a, f);
|
||||
if (norm(a) < 0.008) return false;
|
||||
a = unit(a);
|
||||
const V3 y = cross(f, a) * double(side), z = f * -1;
|
||||
R = from_cols(cross(y, z), y, z);
|
||||
return true;
|
||||
}
|
||||
|
||||
double angle(V3 a, V3 b) { return std::acos(std::clamp(dot(unit(a), unit(b)), -1.0, 1.0)); }
|
||||
|
||||
double median(const std::vector<float> &v) {
|
||||
if (v.empty()) return 0;
|
||||
float tmp[Calibration::kSettle];
|
||||
const size_t n = std::min(v.size(), size_t(Calibration::kSettle));
|
||||
std::copy(v.begin(), v.begin() + n, tmp);
|
||||
std::nth_element(tmp, tmp + n / 2, tmp + n);
|
||||
return tmp[n / 2];
|
||||
}
|
||||
|
||||
void to_quantities(const Shape &s, double *q) {
|
||||
for (int k = 0; k < 5; ++k)
|
||||
for (int j = 0; j < 3; ++j) q[3 * k + j] = s.palm[kPalmIds[k + 1]][j];
|
||||
for (int k = 0; k < 15; ++k) q[15 + k] = s.len[phalanx_bone(k)];
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void Shape::from_quantities(const double *q) {
|
||||
for (int k = 0; k < 5; ++k)
|
||||
for (int j = 0; j < 3; ++j) palm[kPalmIds[k + 1]][j] = q[3 * k + j];
|
||||
for (int k = 0; k < 15; ++k) len[phalanx_bone(k)] = q[15 + k];
|
||||
for (int c = 0; c < 5; ++c) len[kChainBase[c] - 1] = norm(palm[kChainBase[c]]); // bones 0, 4, 8, 12, 16
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------- calibration
|
||||
|
||||
void Calibration::add(const V3 local[21], const bool ok[21], const bool finger_ok[5]) {
|
||||
double x[kQuantities];
|
||||
Shape s;
|
||||
for (int k : {1, 5, 9, 13, 17}) s.palm[k] = local[k];
|
||||
for (int b = 0; b < 20; ++b) s.len[b] = norm(local[b + 1] - local[b % 4 == 0 ? 0 : b]);
|
||||
to_quantities(s, x);
|
||||
// a phalanx far off the model's average hand's proportions is a misplaced joint (the dim
|
||||
// typing recording measured a middle phalanx 1.9 times as long as that), not this hand
|
||||
const double palm_len = norm(local[9] - local[0]);
|
||||
for (int k = 0; k < kQuantities; ++k) {
|
||||
if (k >= 15) {
|
||||
const double r = x[k] / (palm_len * kCanonPhalanx[k - 15] / -kCanonPalm[3][2]);
|
||||
if (!(r > 0.7 && r < 1.5)) continue;
|
||||
}
|
||||
if (k < 15 ? !ok[kPalmIds[k / 3 + 1]] || !ok[0] || !ok[5] || !ok[9] || !ok[17] : !finger_ok[(k - 15) / 3]) continue;
|
||||
if (k >= 15) {
|
||||
const int b = phalanx_bone(k - 15);
|
||||
if (!ok[b] || !ok[b + 1]) continue;
|
||||
}
|
||||
if (n_[k] < kSettle) {
|
||||
buf_[k].push_back(float(x[k]));
|
||||
est_[k] = median(buf_[k]); // so far
|
||||
if (++n_[k] == kSettle) {
|
||||
buf_[k].clear();
|
||||
buf_[k].shrink_to_fit();
|
||||
}
|
||||
} else if (std::fabs(x[k] - est_[k]) < 0.3 * std::fabs(est_[k]) + 0.003) {
|
||||
est_[k] += std::clamp(x[k] - est_[k], -2e-5, 2e-5); // a running median, at most 0.02 mm a sample
|
||||
++n_[k];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Shape Calibration::shape(const Shape &prior) const {
|
||||
double p[kQuantities], q[kQuantities];
|
||||
to_quantities(prior, p);
|
||||
constexpr double kPriorWeight = 8;
|
||||
for (int k = 0; k < kQuantities; ++k) {
|
||||
if (n_[k] >= kSettle) q[k] = est_[k];
|
||||
else if (n_[k] == 0) q[k] = p[k];
|
||||
else q[k] = (n_[k] * est_[k] + kPriorWeight * p[k]) / (n_[k] + kPriorWeight);
|
||||
}
|
||||
Shape s;
|
||||
s.from_quantities(q);
|
||||
return s;
|
||||
}
|
||||
|
||||
double Calibration::settled() const {
|
||||
int least = kSettle;
|
||||
for (int k = 0; k < kQuantities; ++k) least = std::min(least, n_[k]);
|
||||
return double(least) / kSettle;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------- skeleton
|
||||
|
||||
Shape Skeleton::canonical(double scale) {
|
||||
Shape s;
|
||||
double q[kQuantities];
|
||||
for (int k = 0; k < 5; ++k)
|
||||
for (int j = 0; j < 3; ++j) q[3 * k + j] = kCanonPalm[k + 1][j] * scale;
|
||||
for (int k = 0; k < 15; ++k) q[15 + k] = kCanonPhalanx[k] * scale;
|
||||
s.from_quantities(q);
|
||||
return s;
|
||||
}
|
||||
|
||||
Shape Skeleton::shape(int side, double scale) const {
|
||||
const Calibration &me = cal_[side > 0], &other = cal_[side <= 0];
|
||||
// the other hand, when it knows more: left and right hands are mirror images
|
||||
const Shape prior = other.settled() > me.settled() ? other.shape(canonical(scale)) : canonical(scale);
|
||||
return me.shape(prior);
|
||||
}
|
||||
|
||||
void Skeleton::pose(const Shape &s, int side, const Rot &R, V3 t, const double *q, V3 out[21], Rot frames[21]) {
|
||||
out[0] = t;
|
||||
frames[0] = R;
|
||||
for (int c = 0; c < 5; ++c) {
|
||||
ChainState st;
|
||||
const double *o = q + kAngles + 3 * c;
|
||||
chain_fk(s, c, side, rest_frame(s, c, side), R, t, q + kChainQ[c], {o[0], o[1], o[2]}, st);
|
||||
const int base = kChainBase[c];
|
||||
out[base] = st.base;
|
||||
for (int j = 0; j < 3; ++j) out[base + 1 + j] = st.pts[j], frames[base + j] = st.frames[j];
|
||||
frames[base + 3] = st.frames[2]; // the tip: its bone's frame
|
||||
}
|
||||
}
|
||||
|
||||
double Skeleton::solve(const Shape &s, int side, const V3 pts[21], const double sigma_l[21], const double sigma_d[21],
|
||||
V3 eye, const double *prev, int iterations, Rot &R, V3 &t, double *q, double *rms, double limits,
|
||||
double chain_rms[5], bool warm) {
|
||||
Weight w[21];
|
||||
for (int i = 0; i < 21; ++i) w[i] = weight(pts[i], eye, sigma_l[i], sigma_d[i]);
|
||||
|
||||
// 1. the palm, rigid: from the measured frame, then Gauss-Newton on rotation and position
|
||||
if (!measured_frame(pts, side, R)) return -1;
|
||||
V3 T[6];
|
||||
for (int k = 0; k < 6; ++k) T[k] = mirror(s.palm[kPalmIds[k]], side);
|
||||
{
|
||||
V3 c{};
|
||||
double ws = 0;
|
||||
for (int k = 0; k < 6; ++k) {
|
||||
const double wk = w[kPalmIds[k]].a;
|
||||
c = c + (pts[kPalmIds[k]] - R * T[k]) * wk, ws += wk;
|
||||
}
|
||||
t = c * (1 / ws);
|
||||
}
|
||||
for (int it = 0; it < 3; ++it) {
|
||||
double H[kMaxP][kMaxP] = {}, g[6] = {}, x[6];
|
||||
for (int k = 0; k < 6; ++k) {
|
||||
const int i = kPalmIds[k];
|
||||
const V3 v = R * T[k], r = v + t - pts[i];
|
||||
V3 J[6];
|
||||
for (int a = 0; a < 3; ++a) {
|
||||
V3 e{};
|
||||
e[a] = 1;
|
||||
J[a] = cross(e, v);
|
||||
J[3 + a] = e;
|
||||
}
|
||||
V3 WJ[6];
|
||||
for (int a = 0; a < 6; ++a) WJ[a] = w[i].apply(J[a]);
|
||||
for (int a = 0; a < 6; ++a) {
|
||||
g[a] += dot(WJ[a], r);
|
||||
for (int b = 0; b <= a; ++b) H[a][b] += dot(J[b], WJ[a]);
|
||||
}
|
||||
}
|
||||
for (int a = 0; a < 6; ++a) {
|
||||
for (int b = 0; b < a; ++b) H[b][a] = H[a][b];
|
||||
H[a][a] *= 1 + 1e-6;
|
||||
}
|
||||
if (!solve_spd(H, g, x, 6)) break;
|
||||
R = exp_rot({x[0], x[1], x[2]}) * R;
|
||||
t = t + V3{x[3], x[4], x[5]};
|
||||
if (norm(V3{x[0], x[1], x[2]}) < 1e-4 && norm(V3{x[3], x[4], x[5]}) < 1e-5) break;
|
||||
}
|
||||
|
||||
// 2. each chain from its base on the fitted palm. The base may move off the rigid palm a
|
||||
// little (kBaseSigma): the measured knuckles wander far more than a palm bends (the little
|
||||
// finger's by 3 cm), so a rigid palm alone puts whole fingers beside their measurements.
|
||||
// Angles start straight from the measured bone directions, then Gauss-Newton with the
|
||||
// weights, the joint limits and those priors.
|
||||
double cost = 0;
|
||||
{
|
||||
const V3 r = t - pts[0];
|
||||
cost += dot(r, w[0].apply(r));
|
||||
}
|
||||
for (int c = 0; c < 5; ++c) {
|
||||
int n;
|
||||
const Dof *dofs = chain_dofs(c, &n);
|
||||
const int base = kChainBase[c], np = n + 3; // angles, then the base's offset
|
||||
const Rot rest = rest_frame(s, c, side);
|
||||
double *qc = q + kChainQ[c], *oc = q + kAngles + 3 * c;
|
||||
const double *pc = prev ? prev + kChainQ[c] : nullptr;
|
||||
// initial offset: most of the way to the measured knuckle
|
||||
{
|
||||
const V3 d = pts[base] - (R * mirror(s.palm[base], side) + t);
|
||||
const double k = kBaseSigma * kBaseSigma / (kBaseSigma * kBaseSigma + sigma_l[base] * sigma_l[base]);
|
||||
V3 o = d * k;
|
||||
if (norm(o) > kBaseMax) o = o * (kBaseMax / norm(o));
|
||||
for (int j = 0; j < 3; ++j) oc[j] = o[j];
|
||||
}
|
||||
// initial angles
|
||||
{
|
||||
Rot F = R * rest;
|
||||
V3 p = R * mirror(s.palm[base], side) + t + V3{oc[0], oc[1], oc[2]};
|
||||
int d = 0;
|
||||
for (int j = 0; j < 3; ++j) {
|
||||
const V3 l = F.t() * unit(pts[base + 1 + j] - p);
|
||||
for (; d < n && dofs[d].joint == j; ++d) {
|
||||
double a = 0;
|
||||
if (dofs[d].axis == 1) a = std::atan2(-l[0], -l[2]) * side; // abduction (before flexion)
|
||||
else if (dofs[d].axis == 0) a = dofs[d].joint == 0 || (c == 0 && j == 1) ? std::asin(std::clamp(-l[1], -1.0, 1.0))
|
||||
: std::atan2(-l[1], -l[2]);
|
||||
else a = pc ? pc[d] : 0; // twist: unseen in one bone's direction
|
||||
qc[d] = std::clamp(a, dofs[d].lo * limits, dofs[d].hi * limits);
|
||||
F = F * axis_rot(dofs[d].axis, dof_angle(dofs[d], qc[d], side));
|
||||
}
|
||||
p = p - F.col(2) * s.len[base + j];
|
||||
}
|
||||
}
|
||||
// priors: the base's offset toward none; a weak pull toward last frame's angles (and the
|
||||
// twist toward none), for what one frame doesn't show: the twist of a straight thumb, a
|
||||
// finger seen end-on
|
||||
double lambda[kMaxP], target[kMaxP], x[kMaxP];
|
||||
for (int d = 0; d < n; ++d) {
|
||||
const double sigma = dofs[d].axis == 2 ? 0.35 : 0.6;
|
||||
lambda[d] = 1e-6 / (0.002 * 0.002) / (sigma * sigma); // 1 rad off weighs like 1 mm off at 2 mm sigma, over sigma^2
|
||||
target[d] = dofs[d].axis == 2 ? 0 : pc ? pc[d] : qc[d];
|
||||
}
|
||||
for (int j = 0; j < 3; ++j) lambda[n + j] = 1 / (kBaseSigma * kBaseSigma), target[n + j] = 0;
|
||||
// Gauss-Newton from (qa, oa), in place; returns the chain's cost, and its plain rms in *crms
|
||||
auto run = [&](double *qa, double *oa, double *crms) {
|
||||
auto params = [&](int k) -> double & { return k < n ? qa[k] : oa[k - n]; };
|
||||
ChainState st;
|
||||
for (int it = 0; it < iterations; ++it) {
|
||||
chain_fk(s, c, side, rest, R, t, qa, {oa[0], oa[1], oa[2]}, st);
|
||||
double H[kMaxP][kMaxP] = {}, g[kMaxP] = {};
|
||||
for (int m = 0; m < 4; ++m) { // the base, then the chain's three landmarks
|
||||
const int lm = base + m;
|
||||
const Weight &wm = w[lm];
|
||||
const V3 at = m ? st.pts[m - 1] : st.base, r = at - pts[lm];
|
||||
V3 J[kMaxP], WJ[kMaxP];
|
||||
for (int d = 0; d < n; ++d)
|
||||
J[d] = m && dofs[d].joint <= m - 1 ? cross(st.axis[d], at - st.pivot[d]) * dof_sign(dofs[d], side) : V3{0, 0, 0};
|
||||
for (int j = 0; j < 3; ++j) J[n + j] = {0, 0, 0}, J[n + j][j] = 1;
|
||||
for (int d = 0; d < np; ++d) WJ[d] = wm.apply(J[d]);
|
||||
for (int a2 = 0; a2 < np; ++a2) {
|
||||
g[a2] += dot(WJ[a2], r);
|
||||
for (int b2 = 0; b2 <= a2; ++b2) H[a2][b2] += dot(J[b2], WJ[a2]);
|
||||
}
|
||||
}
|
||||
for (int a2 = 0; a2 < np; ++a2) {
|
||||
for (int b2 = 0; b2 < a2; ++b2) H[b2][a2] = H[a2][b2];
|
||||
H[a2][a2] = H[a2][a2] * (1 + 1e-3) + lambda[a2];
|
||||
g[a2] += lambda[a2] * (params(a2) - target[a2]);
|
||||
}
|
||||
if (!solve_spd(H, g, x, np)) break;
|
||||
double step = 0;
|
||||
for (int d = 0; d < n; ++d) {
|
||||
const double nq = std::clamp(qa[d] + x[d], dofs[d].lo * limits, dofs[d].hi * limits);
|
||||
step = std::max(step, std::fabs(nq - qa[d]));
|
||||
qa[d] = nq;
|
||||
}
|
||||
V3 o{oa[0] + x[n], oa[1] + x[n + 1], oa[2] + x[n + 2]};
|
||||
if (norm(o) > kBaseMax) o = o * (kBaseMax / norm(o));
|
||||
for (int j = 0; j < 3; ++j) step = std::max(step, std::fabs(o[j] - oa[j]) * 10), oa[j] = o[j]; // 0.1 mm ~ 1e-3 rad
|
||||
if (step < 1e-3) break;
|
||||
}
|
||||
chain_fk(s, c, side, rest, R, t, qa, {oa[0], oa[1], oa[2]}, st);
|
||||
double cc = 0, sq = 0;
|
||||
for (int m = 0; m < 4; ++m) {
|
||||
const V3 r = (m ? st.pts[m - 1] : st.base) - pts[base + m];
|
||||
cc += dot(r, w[base + m].apply(r));
|
||||
sq += dot(r, r);
|
||||
}
|
||||
for (int d = 0; d < np; ++d) cc += lambda[d] * (params(d) - target[d]) * (params(d) - target[d]);
|
||||
*crms = std::sqrt(sq / 4);
|
||||
return cc;
|
||||
};
|
||||
double crms = 0, cc = run(qc, oc, &crms);
|
||||
// and from last frame's pose: the one that fits better wins, so the fit doesn't hop
|
||||
// between two poses that fit about as well from one frame's analytic start to the next
|
||||
if (pc && warm) {
|
||||
double q2[kMaxP], o2[3], crms2 = 0;
|
||||
const double *po = prev + kAngles + 3 * c;
|
||||
for (int d = 0; d < n; ++d) q2[d] = std::clamp(pc[d], dofs[d].lo * limits, dofs[d].hi * limits);
|
||||
for (int j = 0; j < 3; ++j) o2[j] = po[j];
|
||||
const double cc2 = run(q2, o2, &crms2);
|
||||
if (cc2 < cc) {
|
||||
cc = cc2, crms = crms2;
|
||||
std::copy(q2, q2 + n, qc);
|
||||
std::copy(o2, o2 + 3, oc);
|
||||
}
|
||||
}
|
||||
cost += cc;
|
||||
if (chain_rms) chain_rms[c] = crms;
|
||||
}
|
||||
if (rms) {
|
||||
V3 out[21];
|
||||
Rot frames[21];
|
||||
pose(s, side, R, t, q, out, frames);
|
||||
double sum = 0;
|
||||
for (int i = 0; i < 21; ++i) sum += dot(out[i] - pts[i], out[i] - pts[i]);
|
||||
*rms = std::sqrt(sum / 21);
|
||||
}
|
||||
return cost;
|
||||
}
|
||||
|
||||
V3 Skeleton::elbow(V3 wrist, V3 mid_knuckle, int side, double forearm, V3 up) {
|
||||
// frame-hands' train/forearm/armmodel.py: a neck pivot fixed in the head frame, the
|
||||
// shoulders off it in a torso that faces where the head does (level), upper arm 1.5 x the
|
||||
// forearm, the elbow on the circle the two bones allow, turned -3 degrees from straight down
|
||||
const V3 neck{0, -0.12, 0.08};
|
||||
constexpr double kShoulderW = 0.18, kShoulderD = 0.06, kUpper = 1.5, kSwivel = -3 * D, kMaxBend = 75 * D;
|
||||
V3 fwd{0, 0, -1};
|
||||
fwd = fwd - up * dot(fwd, up);
|
||||
const V3 torso = norm(fwd) > 1e-3 ? unit(fwd) : V3{0, 0, -1};
|
||||
const V3 right = unit(cross(torso, up));
|
||||
const V3 S = neck + right * (side * kShoulderW) - up * kShoulderD;
|
||||
const double F = forearm, U = kUpper * forearm;
|
||||
const V3 v = wrist - S;
|
||||
const double d = norm(v);
|
||||
const V3 stub = unit(wrist - mid_knuckle);
|
||||
V3 dir = stub;
|
||||
if (d > 1e-3) {
|
||||
const V3 n = v * (1 / d);
|
||||
const double dd = std::clamp(d, std::fabs(U - F) + 1e-4, U + F - 1e-4);
|
||||
const double a = (U * U - F * F + dd * dd) / (2 * dd), rho = std::sqrt(std::max(U * U - a * a, 0.0));
|
||||
const V3 C = S + n * a;
|
||||
V3 down = up * -1;
|
||||
down = down - n * dot(n, down);
|
||||
if (norm(down) > 1e-6) {
|
||||
const V3 v0 = unit(down);
|
||||
V3 v1 = cross(n, v0) * double(side);
|
||||
if (dot(v1, right) * side < 0) v1 = v1 * -1;
|
||||
const V3 E = C + (v0 * std::cos(kSwivel) + v1 * std::sin(kSwivel)) * rho;
|
||||
if (norm(E - wrist) > 1e-6) dir = unit(E - wrist);
|
||||
}
|
||||
}
|
||||
// a wrist bends at most about this far from the forearm's line
|
||||
const double bend = angle(dir, stub);
|
||||
if (bend > kMaxBend) {
|
||||
const V3 perp = unit(dir - stub * dot(dir, stub));
|
||||
dir = stub * std::cos(kMaxBend) + perp * std::sin(kMaxBend);
|
||||
}
|
||||
return wrist + dir * F;
|
||||
}
|
||||
|
||||
void Skeleton::update(Fit &f, const V3 pts[21], bool right, double right_score, int nviews, double residual,
|
||||
const double *res_pt, V3 eye, double scale, int64_t t_ns) {
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
const bool continuing = f.valid && t_ns > f.ns && t_ns - f.ns < 300'000'000;
|
||||
f.valid = false;
|
||||
if (nviews <= 0) return;
|
||||
const int model_side = right ? 1 : -1;
|
||||
double sl[21], sd[21];
|
||||
for (int i = 0; i < 21; ++i) {
|
||||
// stereo: a landmark whose rays miss each other is less sure
|
||||
const double r = res_pt && res_pt[i] >= 0 ? res_pt[i] : residual;
|
||||
sl[i] = nviews >= 2 ? std::hypot(0.002, std::max(0.0, r)) : 0.003;
|
||||
sd[i] = nviews >= 2 ? p_.depth_ratio * sl[i] : 0.015;
|
||||
}
|
||||
// the side: the model's, unless the mirror image fits far better (a finger can't bend
|
||||
// backward, so a hand fitted as the wrong side fits badly once its fingers bend) for
|
||||
// kFlipFrames in a row. A side chosen that way sticks while the hand is tracked, as the
|
||||
// model's call does; switching resets the filter, so it mustn't flicker.
|
||||
constexpr int kFlipFrames = 3;
|
||||
// The model's call is a running average (Hand::right_score); while it hovers near 0.5 a
|
||||
// tracked hand keeps the side it has, so the skeleton doesn't turn inside out.
|
||||
int side = model_side;
|
||||
if (continuing && f.side != model_side && (f.flipped || std::fabs(right_score - 0.5) < 0.25)) side = f.side;
|
||||
if (!continuing) f.votes = 0;
|
||||
Shape s = shape(side, scale);
|
||||
Rot R;
|
||||
V3 t;
|
||||
double q[kParams] = {}, rms = 0;
|
||||
const double *prev = continuing && f.side == side ? f.q : nullptr;
|
||||
double crms[5] = {};
|
||||
double cost = solve(s, side, pts, sl, sd, eye, prev, p_.iterations, R, t, q, &rms, p_.limits, crms, p_.warm > 0);
|
||||
if (cost < 0) return;
|
||||
bool better = false;
|
||||
if (rms > 0.015) {
|
||||
Rot R2;
|
||||
V3 t2;
|
||||
double q2[kParams] = {}, rms2 = 0;
|
||||
const Shape s2 = shape(-side, scale);
|
||||
double crms2[5] = {};
|
||||
const double cost2 = solve(s2, -side, pts, sl, sd, eye, continuing && f.side == -side ? f.q : nullptr,
|
||||
p_.iterations, R2, t2, q2, &rms2, p_.limits, crms2, p_.warm > 0);
|
||||
better = cost2 >= 0 && cost2 < 0.5 * cost;
|
||||
if (better && ++f.votes >= kFlipFrames) {
|
||||
side = -side, s = s2, R = R2, t = t2, rms = rms2, cost = cost2;
|
||||
std::copy(q2, q2 + kParams, q);
|
||||
std::copy(crms2, crms2 + 5, crms);
|
||||
f.votes = 0;
|
||||
++flips;
|
||||
}
|
||||
}
|
||||
if (!better) f.votes = 0;
|
||||
f.flipped = side != model_side;
|
||||
|
||||
// learn the bone lengths from clean stereo frames, in the hand's own palm frame
|
||||
if (nviews >= 2 && residual >= 0 && residual <= 0.004 && side == model_side && std::fabs(right_score - 0.5) > 0.3) {
|
||||
bool pt_ok[21];
|
||||
for (int i = 0; i < 21; ++i) pt_ok[i] = !res_pt || (res_pt[i] >= 0 && res_pt[i] <= 0.004);
|
||||
Rot M;
|
||||
if (measured_frame(pts, side, M)) {
|
||||
V3 local[21];
|
||||
const Rot Mt = M.t();
|
||||
for (int i = 0; i < 21; ++i) local[i] = mirror(Mt * (pts[i] - pts[0]), side);
|
||||
bool ok[5];
|
||||
for (int c = 0; c < 5; ++c) {
|
||||
const int b = kChainBase[c];
|
||||
// bent fingers hide their own joints from the cameras: only fairly straight ones
|
||||
const double bend = angle(local[b + 1] - local[b], local[b + 2] - local[b + 1]) +
|
||||
angle(local[b + 2] - local[b + 1], local[b + 3] - local[b + 2]);
|
||||
ok[c] = bend < 90 * D;
|
||||
}
|
||||
cal_[side > 0].add(local, pt_ok, ok);
|
||||
}
|
||||
}
|
||||
|
||||
// filter the pose: the palm centre, the palm's rotation, each chain's angles
|
||||
V3 Tc{};
|
||||
for (int k : {0, 5, 9, 13, 17}) Tc = Tc + mirror(s.palm[k], side) * 0.2;
|
||||
const V3 c = R * Tc + t;
|
||||
const double dt = continuing ? (t_ns - f.ns) / 1e9 : 0;
|
||||
if (!continuing || f.side != side || dt <= 0) {
|
||||
f.Rf = R, f.cf = c, f.vel = f.angvel = {0, 0, 0};
|
||||
std::copy(q, q + kParams, f.qf);
|
||||
std::fill(f.dq, f.dq + kParams, 0.0);
|
||||
} else {
|
||||
auto alpha = [dt](double cutoff) { return 1 / (1 + 1 / (2 * M_PI * cutoff * dt)); };
|
||||
const double ad = alpha(p_.d_cutoff);
|
||||
f.vel = f.vel + ((c - f.cf) * (1 / dt) - f.vel) * ad;
|
||||
f.cf = f.cf + (c - f.cf) * alpha(p_.pos_min + p_.pos_beta * norm(f.vel));
|
||||
const V3 delta = log_rot(f.Rf.t() * R); // in the filtered palm's frame
|
||||
f.angvel = f.angvel + (f.Rf * delta * (1 / dt) - f.angvel) * ad;
|
||||
f.Rf = f.Rf * exp_rot(delta * alpha(p_.rot_min + p_.rot_beta * norm(f.angvel)));
|
||||
for (int ch = 0; ch < 5; ++ch) {
|
||||
const int a = kChainQ[ch], b = ch + 1 < 5 ? kChainQ[ch + 1] : kAngles;
|
||||
// a chain that fits its landmarks badly this frame is mostly a misplaced landmark (a
|
||||
// finger hidden behind another): it moves the filter, and its speed, less
|
||||
const double trust = std::min(1.0, std::pow(p_.glitch / std::max(crms[ch], 1e-6), 2));
|
||||
double speed = 0;
|
||||
for (int d = a; d < b; ++d) {
|
||||
f.dq[d] += trust * ad * ((q[d] - f.qf[d]) / dt - f.dq[d]);
|
||||
speed = std::max(speed, std::fabs(f.dq[d]));
|
||||
}
|
||||
const double ac = trust * alpha(p_.ang_min + p_.ang_beta * speed);
|
||||
for (int d = a; d < b; ++d) f.qf[d] += ac * (q[d] - f.qf[d]);
|
||||
for (int d = kAngles + 3 * ch; d < kAngles + 3 * ch + 3; ++d) f.qf[d] += ac * (q[d] - f.qf[d]); // its base
|
||||
}
|
||||
}
|
||||
std::copy(crms, crms + 5, f.chain_rms);
|
||||
f.R = R, f.t = t, f.rms = rms, f.side = side, f.ns = t_ns, f.valid = true;
|
||||
std::copy(q, q + kParams, f.q);
|
||||
{
|
||||
Rot fr[21];
|
||||
pose(s, side, R, t, q, f.raw, fr);
|
||||
}
|
||||
const V3 tf = f.cf - f.Rf * Tc;
|
||||
pose(s, side, f.Rf, tf, f.qf, f.pts, f.frames);
|
||||
std::copy(s.len, s.len + 20, f.len);
|
||||
// rough radii, from the capsules' (io.cpp) at this hand's size
|
||||
const double k = norm(s.palm[9]) / 0.0815;
|
||||
for (int i = 0; i < 21; ++i) f.radius[i] = k * (i == 0 ? 0.020 : i == 1 || i % 4 == 1 ? 0.012 : i <= 4 ? 0.0095 : 0.0085);
|
||||
f.elbow = elbow(tf, f.pts[9], side, p_.forearm_ratio * norm(s.palm[9]), up_);
|
||||
f.settled = cal_[side > 0].settled();
|
||||
f.us = std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - t0).count();
|
||||
++fits, us_total += f.us;
|
||||
}
|
||||
|
||||
} // namespace skel
|
||||
@@ -1,144 +0,0 @@
|
||||
// A per-user hand skeleton fitted to the tracker's 21 landmarks (hands/README.md, "Skeleton").
|
||||
//
|
||||
// Each side (left, right) learns its bone lengths and palm shape from clean stereo frames.
|
||||
// Every processed frame then fits a skeleton with those lengths to the measured points: the
|
||||
// palm as a rigid body, each finger as a chain of hinges with joint limits. The fitted pose
|
||||
// (palm pose and joint angles) is filtered, not the points, so fingers keep their length.
|
||||
// An arm model puts an elbow behind the wrist. All 3D is metres in the head frame.
|
||||
//
|
||||
// Frames follow OpenXR's hand joints: a joint's -Z points along the bone that starts there
|
||||
// (toward the fingertip), +Y out of the back of the hand, +X = Y x Z. The wrist's frame is
|
||||
// the palm's: -Z toward the middle knuckle. Both hands use the same axes, so +X points to
|
||||
// the little finger on a right hand and to the thumb on a left one.
|
||||
#pragma once
|
||||
|
||||
#include "geom.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <vector>
|
||||
|
||||
namespace skel {
|
||||
|
||||
// 3x3 rotation; columns are the frame's axes in the head frame.
|
||||
struct Rot {
|
||||
double m[3][3] = {{1, 0, 0}, {0, 1, 0}, {0, 0, 1}};
|
||||
V3 col(int c) const { return {m[0][c], m[1][c], m[2][c]}; }
|
||||
V3 operator*(V3 v) const;
|
||||
Rot operator*(const Rot &o) const;
|
||||
Rot t() const;
|
||||
};
|
||||
Rot exp_rot(V3 w); // rotation vector -> rotation
|
||||
V3 log_rot(const Rot &r); // rotation -> rotation vector
|
||||
void to_quat(const Rot &r, double q[4]); // x, y, z, w
|
||||
|
||||
constexpr int kAngles = 26; // thumb 6, each finger 5
|
||||
constexpr int kParams = kAngles + 15; // and each chain's base offset from the rigid palm (m)
|
||||
constexpr int kQuantities = 30; // what calibration learns: 5 palm points x 3, 15 phalanges
|
||||
|
||||
// One user's hand, in the right hand's palm frame (a left hand is its mirror image in x).
|
||||
struct Shape {
|
||||
V3 palm[21]{}; // landmarks 1, 5, 9, 13 and 17 from the wrist (others unused)
|
||||
double len[20]{}; // bone b ends at landmark b + 1; the 5 from the wrist are |palm[k]|
|
||||
void from_quantities(const double *q);
|
||||
};
|
||||
|
||||
// Learns one side's Shape from clean stereo frames: medians of the first kSettle samples of
|
||||
// each quantity, then a running median that moves each by at most 0.02 mm a sample.
|
||||
class Calibration {
|
||||
public:
|
||||
static constexpr int kSettle = 45; // samples: 3 s of a clean hand at 15 Hz
|
||||
// A frame's raw landmarks (2+ views, small residual), in this side's palm frame already
|
||||
// mirrored to a right hand (Skeleton::update does that). Fingers bent a lot are skipped.
|
||||
// ok: landmarks triangulated cleanly (both ends of a bone must be).
|
||||
void add(const V3 local[21], const bool ok[21], const bool finger_ok[5]);
|
||||
// Estimates, blended with prior (a Shape) while few samples are in.
|
||||
Shape shape(const Shape &prior) const;
|
||||
double settled() const; // 0..1: the least-sampled quantity's share of kSettle
|
||||
int samples(int k) const { return n_[k]; }
|
||||
|
||||
private:
|
||||
std::vector<float> buf_[kQuantities];
|
||||
double est_[kQuantities]{};
|
||||
int n_[kQuantities]{};
|
||||
};
|
||||
|
||||
// Per hand, kept with the tracker's Hand.
|
||||
struct Fit {
|
||||
bool valid = false; // fitted this frame
|
||||
int side = 1; // +1 right, -1 left: the mirror the fit used
|
||||
int64_t ns = 0; // last fit
|
||||
// this frame's fit, before filtering
|
||||
Rot R; // palm frame -> head
|
||||
V3 t{}; // wrist
|
||||
double q[kParams]{}; // angles (rad), then each chain base's offset (m)
|
||||
double rms = 0; // fitted landmarks against measured, m (all 21)
|
||||
double chain_rms[5]{}; // each chain's (thumb, index, ...: its base and 3 landmarks)
|
||||
double us = 0; // fit time, microseconds
|
||||
bool flipped = false; // fitted as the other side than the model's call (it fitted much better)
|
||||
int votes = 0; // frames in a row the other side fitted much better
|
||||
// filtered
|
||||
Rot Rf;
|
||||
V3 cf{}, vel{}, angvel{}; // palm centre, its velocity (m/s), the palm's angular velocity (rad/s)
|
||||
double qf[kParams]{}, dq[kParams]{};
|
||||
// results, from the filtered pose
|
||||
V3 raw[21]{}; // this frame's fit, unfiltered
|
||||
V3 pts[21]{}; // fitted landmarks
|
||||
Rot frames[21]; // each joint's frame (see the top)
|
||||
double radius[21]{}; // rough joint radii, m, for colliders
|
||||
double len[20]{}; // bone lengths used
|
||||
V3 elbow{};
|
||||
double settled = 0; // the side's calibration, 0..1
|
||||
};
|
||||
|
||||
struct Params {
|
||||
// the filter on the fitted pose: like the One Euro on the points (tracker.cpp), cutoffs in Hz
|
||||
double pos_min = 2.0, pos_beta = 30.0; // palm centre: + beta per m/s
|
||||
double rot_min = 2.0, rot_beta = 10.0; // palm rotation: + beta per rad/s
|
||||
double ang_min = 2.0, ang_beta = 3.0; // a finger's joint angles: + beta per rad/s of that finger
|
||||
double d_cutoff = 1.5; // for the speeds themselves
|
||||
double warm = 1; // 1: also start each chain from last frame's pose
|
||||
double glitch = 0.015; // m: a chain fitting worse than this moves the filter less
|
||||
int iterations = 4; // Gauss-Newton steps per chain
|
||||
double depth_ratio = 3.5; // stereo: how much less sure depth is than across the view
|
||||
double limits = 1; // scales the joint limits (for tests: 10 is about none)
|
||||
double forearm_ratio = 2.8; // forearm / palm length (notes/forearm.md)
|
||||
};
|
||||
|
||||
class Skeleton {
|
||||
public:
|
||||
explicit Skeleton(const Params &p = {}) : p_(p) {}
|
||||
// After the tracker's 3D: pts (raw, this frame), the model's side, views (1: one camera),
|
||||
// residual (m, stereo: the landmarks' median) and res_pt (each landmark's, or null), eye (the views' camera centre, for which way depth is), the
|
||||
// hand's size against the model's (Hand::scale).
|
||||
void update(Fit &f, const V3 pts[21], bool right, double right_score, int nviews, double residual,
|
||||
const double *res_pt, V3 eye, double scale, int64_t t_ns);
|
||||
// The head's up direction, for the arm model (default: the head frame's +y).
|
||||
void set_up(V3 up) { up_ = unit(up); }
|
||||
const Calibration &calibration(int side) const { return cal_[side > 0]; }
|
||||
Shape shape(int side, double scale) const; // the side's current estimate
|
||||
static Shape canonical(double scale); // the model's average hand, times scale
|
||||
const Params ¶ms() const { return p_; }
|
||||
void set_params(const Params &p) { p_ = p; }
|
||||
|
||||
// diagnostics
|
||||
int fits = 0, flips = 0;
|
||||
double us_total = 0;
|
||||
|
||||
// Forward kinematics: landmarks and frames for palm pose (R, t), parameters q (kParams), a
|
||||
// shape and side.
|
||||
static void pose(const Shape &s, int side, const Rot &R, V3 t, const double *q, V3 out[21], Rot frames[21]);
|
||||
// Fits (R, t, q) to pts; weights from views and residual (see skeleton.cpp). Returns the
|
||||
// weighted cost; rms gets the plain rms distance. prev: last frame's angles for the prior, or null.
|
||||
static double solve(const Shape &s, int side, const V3 pts[21], const double sigma_l[21], const double sigma_d[21],
|
||||
V3 eye, const double *prev, int iterations, Rot &R, V3 &t, double *q, double *rms,
|
||||
double limits = 1, double chain_rms[5] = nullptr, bool warm = true);
|
||||
// The arm model: an elbow for a wrist pose, side, palm length (m) and the up direction.
|
||||
static V3 elbow(V3 wrist, V3 mid_knuckle, int side, double forearm, V3 up);
|
||||
|
||||
private:
|
||||
Params p_;
|
||||
Calibration cal_[2]; // left, right
|
||||
V3 up_{0, 1, 0};
|
||||
};
|
||||
|
||||
} // namespace skel
|
||||
@@ -1,412 +0,0 @@
|
||||
// ft-handreplay's measures of the skeleton fit (track/skeleton.h): every published hand's
|
||||
// measured landmarks, the One Euro filtered ones (what version 1 published) and the fitted
|
||||
// skeleton's, compared after the replay. See hands/README.md, "Skeleton".
|
||||
#pragma once
|
||||
|
||||
#include "skeleton.h"
|
||||
#include "tracker.h"
|
||||
|
||||
#include <json/json.h>
|
||||
#include <pthread.h>
|
||||
#include <sched.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <fstream>
|
||||
#include <map>
|
||||
#include <sstream>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace skeval {
|
||||
|
||||
inline double quantile(std::vector<double> v, double q) {
|
||||
if (v.empty()) return NAN;
|
||||
const size_t k = std::min(v.size() - 1, size_t(q * v.size()));
|
||||
std::nth_element(v.begin(), v.begin() + k, v.end());
|
||||
return v[k];
|
||||
}
|
||||
|
||||
struct Rec {
|
||||
int id;
|
||||
double ts;
|
||||
bool stereo, fit_ok, right;
|
||||
double palm_speed; // the tracker's filtered palm speed, m/s
|
||||
V3 pts[4][21]; // measured, One Euro, skeleton (filtered, published), skeleton before filtering
|
||||
double rms, us;
|
||||
V3 elbow;
|
||||
V3 normal[2]; // the back of the hand's normal: from the One Euro landmarks, the skeleton's palm frame
|
||||
V3 up; // the head's up from the HMD pose, or 0
|
||||
// the fit's inputs, to time it again on its own (bench)
|
||||
double right_score, residual, scale, res_pt[21];
|
||||
int nviews;
|
||||
V3 eye;
|
||||
int64_t t_ns;
|
||||
};
|
||||
|
||||
struct Label {
|
||||
bool right;
|
||||
double conf;
|
||||
V3 kp[21];
|
||||
};
|
||||
|
||||
constexpr int kBones[20][2] = {{0, 1}, {1, 2}, {2, 3}, {3, 4}, {0, 5}, {5, 6}, {6, 7},
|
||||
{7, 8}, {0, 9}, {9, 10}, {10, 11}, {11, 12}, {0, 13}, {13, 14},
|
||||
{14, 15}, {15, 16}, {0, 17}, {17, 18}, {18, 19}, {19, 20}};
|
||||
|
||||
|
||||
inline V3 palm(const V3 *p) { return (p[0] + p[5] + p[9] + p[13] + p[17]) * 0.2; }
|
||||
|
||||
class Eval {
|
||||
public:
|
||||
std::vector<Rec> recs;
|
||||
std::map<int64_t, std::vector<Label>> labels; // by set capture time
|
||||
// per side: when the calibration settled (s into the replay, -1 never) and the shape then
|
||||
double settle_ts[2] = {-1, -1};
|
||||
skel::Shape settled_shape[2];
|
||||
double first_ts[2] = {-1, -1};
|
||||
|
||||
bool load_labels(const std::string &path) {
|
||||
std::ifstream f(path);
|
||||
if (!f) return false;
|
||||
std::string line;
|
||||
Json::CharReaderBuilder b;
|
||||
while (std::getline(f, line)) {
|
||||
Json::Value v;
|
||||
std::string e;
|
||||
std::istringstream in(line);
|
||||
if (!Json::parseFromStream(b, in, &v, &e)) continue;
|
||||
std::vector<Label> hs;
|
||||
for (const Json::Value &h : v["hands"]) {
|
||||
Label l{h["side"].asString() == "R", h["conf"].asDouble(), {}};
|
||||
for (int i = 0; i < 21; ++i)
|
||||
for (int k = 0; k < 3; ++k) l.kp[i][k] = h["kp3d"][i][k].asDouble();
|
||||
hs.push_back(l);
|
||||
}
|
||||
labels[v["t_ns"].asInt64()] = hs;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void add(const Hand &h, double ts, V3 up) {
|
||||
Rec r;
|
||||
r.id = h.id, r.ts = ts, r.stereo = h.nviews >= 2, r.fit_ok = h.fit.valid, r.right = h.right();
|
||||
r.palm_speed = norm(h.dpalm);
|
||||
for (int i = 0; i < 21; ++i) r.pts[0][i] = h.pts[i], r.pts[1][i] = h.smooth[i], r.pts[2][i] = h.fit.valid ? h.fit.pts[i] : h.smooth[i];
|
||||
for (int i = 0; i < 21; ++i) r.pts[3][i] = h.fit.valid ? h.fit.raw[i] : h.smooth[i];
|
||||
{
|
||||
const V3 *p = h.smooth;
|
||||
const V3 a = p[5] - p[0], b = p[17] - p[0];
|
||||
const V3 c{a[1] * b[2] - a[2] * b[1], a[2] * b[0] - a[0] * b[2], a[0] * b[1] - a[1] * b[0]};
|
||||
r.normal[0] = unit(c) * (h.right() ? 1.0 : -1.0);
|
||||
r.normal[1] = h.fit.valid ? h.fit.frames[0].col(1) : r.normal[0];
|
||||
}
|
||||
r.right_score = h.right_score, r.residual = h.residual, r.scale = h.scale, r.nviews = h.nviews, r.eye = h.eye;
|
||||
r.t_ns = int64_t(ts * 1e9);
|
||||
std::copy(h.res_pt, h.res_pt + 21, r.res_pt);
|
||||
r.rms = h.fit.rms, r.us = h.fit.us, r.elbow = h.fit.elbow, r.up = up;
|
||||
recs.push_back(r);
|
||||
}
|
||||
|
||||
void calibration(const skel::Skeleton &sk, double ts) {
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
const skel::Calibration &c = sk.calibration(s ? 1 : -1);
|
||||
if (first_ts[s] < 0 && c.samples(0) > 0) first_ts[s] = ts;
|
||||
if (settle_ts[s] < 0 && c.settled() >= 1) settle_ts[s] = ts, settled_shape[s] = sk.shape(s ? 1 : -1, 1.0);
|
||||
}
|
||||
}
|
||||
|
||||
std::string dump; // ft-handreplay --skeleton-dump: a line per hand matched to a label
|
||||
|
||||
// The fit's cost on its own: every hand update again, in order, through a fresh Skeleton on
|
||||
// this thread, without the models' threads competing (the replay's own times include them),
|
||||
// pinned to a little core (CPU 0, a Cortex-A520 on the Frame) and a big one (CPU 3, an A720).
|
||||
void bench(const skel::Params &p) const {
|
||||
cpu_set_t was;
|
||||
pthread_getaffinity_np(pthread_self(), sizeof was, &was);
|
||||
for (int cpu : {0, 3}) {
|
||||
cpu_set_t one;
|
||||
CPU_ZERO(&one);
|
||||
CPU_SET(cpu, &one);
|
||||
if (pthread_setaffinity_np(pthread_self(), sizeof one, &one) != 0) continue;
|
||||
for (int round = 0; round < 2; ++round) { // the first warms the caches
|
||||
skel::Skeleton sk(p);
|
||||
std::map<int, skel::Fit> fits;
|
||||
std::vector<double> us;
|
||||
for (const Rec &r : recs) {
|
||||
skel::Fit &f = fits[r.id];
|
||||
const auto a = std::chrono::steady_clock::now();
|
||||
sk.update(f, r.pts[0], r.right, r.right_score, r.nviews, r.residual, r.res_pt, r.eye, r.scale, r.t_ns);
|
||||
us.push_back(std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - a).count());
|
||||
}
|
||||
if (round)
|
||||
std::printf("fit time on its own, CPU %d (%zu hand updates again): median %.0f us, 90%% %.0f us, 99%% %.0f us, max %.0f us\n",
|
||||
cpu, us.size(), quantile(us, 0.5), quantile(us, 0.9), quantile(us, 0.99),
|
||||
us.empty() ? 0.0 : *std::max_element(us.begin(), us.end()));
|
||||
}
|
||||
}
|
||||
pthread_setaffinity_np(pthread_self(), sizeof was, &was);
|
||||
}
|
||||
|
||||
void report(const skel::Skeleton &sk, int64_t t0, const std::vector<std::pair<int64_t, int>> &label_hits) const {
|
||||
FILE *dp = dump.empty() ? nullptr : std::fopen(dump.c_str(), "w");
|
||||
std::printf("\n-- skeleton fit (track/skeleton.h) --\n");
|
||||
// 1. cost
|
||||
std::vector<double> us, rms[2];
|
||||
int fits = 0;
|
||||
for (const Rec &r : recs)
|
||||
if (r.fit_ok) us.push_back(r.us), rms[r.stereo].push_back(r.rms * 1000), ++fits;
|
||||
std::printf("fits %d of %zu hand updates, %d fitted as the other side; time per hand: median %.0f us, 90%% %.0f us, max %.0f us\n",
|
||||
fits, recs.size(), sk.flips, quantile(us, 0.5), quantile(us, 0.9), us.empty() ? 0.0 : *std::max_element(us.begin(), us.end()));
|
||||
bench(sk.params());
|
||||
std::printf("fit rms against the measured landmarks: stereo median %.1f mm, 90%% %.1f mm; one view median %.1f mm, 90%% %.1f mm\n",
|
||||
quantile(rms[1], 0.5), quantile(rms[1], 0.9), quantile(rms[0], 0.5), quantile(rms[0], 0.9));
|
||||
{
|
||||
std::vector<double> e[21];
|
||||
for (const Rec &r : recs)
|
||||
if (r.fit_ok && r.stereo)
|
||||
for (int i = 0; i < 21; ++i) e[i].push_back(norm(r.pts[3][i] - r.pts[0][i]) * 1000);
|
||||
std::printf(" stereo fit error per landmark, median mm:");
|
||||
for (int i = 0; i < 21; ++i) std::printf(" %.1f", quantile(e[i], 0.5));
|
||||
std::printf("\n");
|
||||
}
|
||||
// 2. calibration
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
const skel::Calibration &c = sk.calibration(s ? 1 : -1);
|
||||
const skel::Shape now = sk.shape(s ? 1 : -1, 1.0);
|
||||
std::printf("%s calibration: settled %.2f", s ? "right" : "left ", c.settled());
|
||||
if (settle_ts[s] >= 0) {
|
||||
double drift = 0;
|
||||
for (int b = 0; b < 20; ++b) drift = std::max(drift, std::fabs(now.len[b] - settled_shape[s].len[b]));
|
||||
std::printf(" at %.1f s (%.1f s after its first sample); largest bone change since %.1f mm", settle_ts[s],
|
||||
settle_ts[s] - first_ts[s], drift * 1000);
|
||||
}
|
||||
std::printf("\n bones mm:");
|
||||
for (int b = 0; b < 20; ++b) std::printf(" %.1f", now.len[b] * 1000);
|
||||
std::printf("\n");
|
||||
}
|
||||
// 3. bone-length stability: coefficient of variation per hand track and bone
|
||||
{
|
||||
std::map<int, std::vector<const Rec *>> by;
|
||||
for (const Rec &r : recs) by[r.id].push_back(&r);
|
||||
std::vector<double> cv[3], cv_stereo;
|
||||
for (auto &[id, rs] : by) {
|
||||
if (rs.size() < 30) continue;
|
||||
for (int b = 0; b < 20; ++b) {
|
||||
for (int v = 0; v < 3; ++v) {
|
||||
double s = 0, s2 = 0;
|
||||
for (const Rec *r : rs) {
|
||||
const double l = norm(r->pts[v][kBones[b][1]] - r->pts[v][kBones[b][0]]);
|
||||
s += l, s2 += l * l;
|
||||
}
|
||||
const double m = s / rs.size(), var = std::max(0.0, s2 / rs.size() - m * m);
|
||||
cv[v].push_back(m > 0 ? std::sqrt(var) / m : 0);
|
||||
}
|
||||
double s = 0, s2 = 0;
|
||||
int n = 0;
|
||||
for (const Rec *r : rs)
|
||||
if (r->stereo) {
|
||||
const double l = norm(r->pts[0][kBones[b][1]] - r->pts[0][kBones[b][0]]);
|
||||
s += l, s2 += l * l, ++n;
|
||||
}
|
||||
if (n > 10) cv_stereo.push_back(std::sqrt(std::max(0.0, s2 / n - s * s / n / n)) / (s / n));
|
||||
}
|
||||
}
|
||||
std::printf("bone length CV (per track and bone, tracks of 30+ updates): measured median %.3f (stereo only %.3f), "
|
||||
"One Euro %.3f, skeleton %.3f; 90%%: %.3f, %.3f, %.3f\n",
|
||||
quantile(cv[0], 0.5), quantile(cv_stereo, 0.5), quantile(cv[1], 0.5), quantile(cv[2], 0.5),
|
||||
quantile(cv[0], 0.9), quantile(cv[1], 0.9), quantile(cv[2], 0.9));
|
||||
}
|
||||
// 4. jitter: off a straight line through the last two updates (as the replay's palm
|
||||
// jitter), for the palm and the thumb and index tips, at rest and moving
|
||||
{
|
||||
std::map<int, std::vector<const Rec *>> by;
|
||||
for (const Rec &r : recs) by[r.id].push_back(&r);
|
||||
const char *what[3] = {"palm", "index tip", "thumb tip"};
|
||||
for (int w = 0; w < 3; ++w) {
|
||||
std::vector<double> j[2][4]; // rest/moving, variant
|
||||
for (auto &[id, rs] : by)
|
||||
for (size_t k = 2; k < rs.size(); ++k) {
|
||||
const Rec &a = *rs[k - 2], &b = *rs[k - 1], &c = *rs[k];
|
||||
if (c.ts - b.ts >= 0.1 || b.ts - a.ts >= 0.1 || b.ts - a.ts < 1e-3) continue;
|
||||
// at rest: the palm still, and the fingers too (typing moves them on a still
|
||||
// palm: that's motion, not jitter); by a centred average of the measured tips
|
||||
bool fingers_still = false;
|
||||
if (k >= 4 && k + 2 < rs.size() && rs[k + 2]->ts - rs[k - 4]->ts < 0.35) {
|
||||
auto rel = [&](size_t i) {
|
||||
V3 s{};
|
||||
for (size_t j = i - 2; j <= i + 2; ++j)
|
||||
s = s + (rs[j]->pts[0][8] - palm(rs[j]->pts[0])) * 0.1 + (rs[j]->pts[0][4] - palm(rs[j]->pts[0])) * 0.1;
|
||||
return s;
|
||||
};
|
||||
fingers_still = norm(rel(k) - rel(k - 2)) / (rs[k]->ts - rs[k - 2]->ts) < 0.1;
|
||||
}
|
||||
const int moving = c.palm_speed > 0.3 ? 1 : c.palm_speed < 0.1 && fingers_still ? 0 : -1;
|
||||
if (moving < 0) continue;
|
||||
const double f = (c.ts - b.ts) / (b.ts - a.ts);
|
||||
for (int v = 0; v < 4; ++v) {
|
||||
auto P = [&](const Rec &r) { return w == 0 ? palm(r.pts[v]) : r.pts[v][w == 1 ? 8 : 4]; };
|
||||
j[moving][v].push_back(norm(P(c) - P(b) - (P(b) - P(a)) * f) * 1000);
|
||||
}
|
||||
}
|
||||
for (int m = 0; m < 2; ++m)
|
||||
std::printf("%-9s jitter %s (%zu): median measured %.1f, One Euro %.1f, skeleton %.1f (unfiltered %.1f) mm; "
|
||||
"90%% %.1f, %.1f, %.1f (%.1f)\n",
|
||||
what[w], m ? "moving" : "at rest (fingers too)", j[m][0].size(), quantile(j[m][0], 0.5), quantile(j[m][1], 0.5),
|
||||
quantile(j[m][2], 0.5), quantile(j[m][3], 0.5), quantile(j[m][0], 0.9), quantile(j[m][1], 0.9),
|
||||
quantile(j[m][2], 0.9), quantile(j[m][3], 0.9));
|
||||
}
|
||||
}
|
||||
// the palm's normal, update to update: still, and with the fingers moving on a still palm
|
||||
// (a pinch: ray and grab orientation want it steady)
|
||||
{
|
||||
std::map<int, std::vector<const Rec *>> by;
|
||||
for (const Rec &r : recs) by[r.id].push_back(&r);
|
||||
std::vector<double> d[2][2]; // still / fingers moving, One Euro / skeleton
|
||||
for (auto &[id, rs] : by)
|
||||
for (size_t k = 1; k < rs.size(); ++k) {
|
||||
const Rec &a = *rs[k - 1], &b = *rs[k];
|
||||
if (b.ts - a.ts >= 0.1 || !a.fit_ok || !b.fit_ok || b.palm_speed > 0.1) continue;
|
||||
const double tip = norm((b.pts[0][8] - palm(b.pts[0])) - (a.pts[0][8] - palm(a.pts[0]))) / (b.ts - a.ts);
|
||||
const int fingers = tip > 0.3 ? 1 : tip < 0.1 ? 0 : -1;
|
||||
if (fingers < 0) continue;
|
||||
for (int v = 0; v < 2; ++v)
|
||||
d[fingers][v].push_back(std::acos(std::clamp(dot(a.normal[v], b.normal[v]), -1.0, 1.0)) * 180 / M_PI);
|
||||
}
|
||||
for (int f = 0; f < 2; ++f)
|
||||
std::printf("palm normal change per update, palm still, fingers %s (%zu): median One Euro %.2f, skeleton %.2f deg; "
|
||||
"90%% %.2f, %.2f\n", f ? "moving" : "still", d[f][0].size(), quantile(d[f][0], 0.5),
|
||||
quantile(d[f][1], 0.5), quantile(d[f][0], 0.9), quantile(d[f][1], 0.9));
|
||||
}
|
||||
// 5. lag: the published tip against a centred average of the measured one (+-2 updates,
|
||||
// so it doesn't lag), along the tip's motion, where the tip moves fast
|
||||
{
|
||||
std::map<int, std::vector<const Rec *>> by;
|
||||
for (const Rec &r : recs) by[r.id].push_back(&r);
|
||||
for (int rel = 0; rel < 2; ++rel) {
|
||||
std::vector<double> lag[4], err[4];
|
||||
for (auto &[id, rs] : by)
|
||||
for (size_t k = 3; k + 3 < rs.size(); ++k) {
|
||||
bool ok = true;
|
||||
for (size_t i = k - 3; i < k + 3; ++i) ok = ok && rs[i + 1]->ts - rs[i]->ts < 0.1;
|
||||
if (!ok) continue;
|
||||
for (int tip : {4, 8}) {
|
||||
// relative to the palm (fingers moving on a still hand) or in the head frame
|
||||
auto P = [&](const Rec &r, int v) { return rel ? r.pts[v][tip] - palm(r.pts[v]) : r.pts[v][tip]; };
|
||||
auto ref = [&](size_t i) {
|
||||
V3 s{};
|
||||
for (size_t j = i - 2; j <= i + 2; ++j) s = s + P(*rs[j], 0) * 0.2;
|
||||
return s;
|
||||
};
|
||||
const V3 vel = (ref(k + 1) - ref(k - 1)) * (1 / (rs[k + 1]->ts - rs[k - 1]->ts));
|
||||
if (norm(vel) < (rel ? 0.3 : 0.5)) continue;
|
||||
const V3 truth = ref(k), dir = unit(vel);
|
||||
for (int v = 0; v < 4; ++v) {
|
||||
lag[v].push_back(dot(truth - P(*rs[k], v), dir) * 1000);
|
||||
err[v].push_back(norm(truth - P(*rs[k], v)) * 1000);
|
||||
}
|
||||
}
|
||||
}
|
||||
std::printf("tip lag, %s (%zu): median measured %.1f, One Euro %.1f, skeleton %.1f (unfiltered %.1f) mm behind; "
|
||||
"distance median %.1f, %.1f, %.1f (%.1f) mm\n",
|
||||
rel ? "moving against the palm faster than 0.3 m/s" : "moving faster than 0.5 m/s", lag[0].size(),
|
||||
quantile(lag[0], 0.5), quantile(lag[1], 0.5), quantile(lag[2], 0.5), quantile(lag[3], 0.5),
|
||||
quantile(err[0], 0.5), quantile(err[1], 0.5), quantile(err[2], 0.5), quantile(err[3], 0.5));
|
||||
}
|
||||
}
|
||||
// 6. the elbow: how far it turns from the stub (the hand's axis carried on), and from the
|
||||
// same arm model with gravity's up (when the HMD poses are known)
|
||||
{
|
||||
std::vector<double> stub, grav, len;
|
||||
for (const Rec &r : recs) {
|
||||
if (!r.fit_ok) continue;
|
||||
const V3 w = r.pts[2][0], d = unit(r.elbow - w), s = unit(w - r.pts[2][9]);
|
||||
stub.push_back(std::acos(std::clamp(dot(d, s), -1.0, 1.0)) * 180 / M_PI);
|
||||
len.push_back(norm(r.elbow - w) * 1000);
|
||||
if (norm(r.up) > 0.5) {
|
||||
const V3 e = skel::Skeleton::elbow(w, r.pts[2][9], r.right ? 1 : -1, norm(r.elbow - w), r.up);
|
||||
grav.push_back(std::acos(std::clamp(dot(d, unit(e - w)), -1.0, 1.0)) * 180 / M_PI);
|
||||
}
|
||||
}
|
||||
std::printf("elbow: forearm median %.0f mm; angle from the stub median %.0f deg, 90%% %.0f", quantile(len, 0.5),
|
||||
quantile(stub, 0.5), quantile(stub, 0.9));
|
||||
if (!grav.empty())
|
||||
std::printf("; head-up against gravity-up arm model median %.1f deg, 90%% %.1f (%zu)", quantile(grav, 0.5),
|
||||
quantile(grav, 0.9), grav.size());
|
||||
std::printf("\n");
|
||||
}
|
||||
// 7. the labels (frame-hands' auto-labels: one teacher, triangulated, smoothed and fitted;
|
||||
// not ground truth)
|
||||
if (!labels.empty()) {
|
||||
std::vector<double> e[4], e_tip[4], per[2][21], by_views[2][2][2]; // [stereo][measured/skeleton][all/tip]
|
||||
int matched = 0, unmatched = 0;
|
||||
std::map<int, std::vector<double>> label_len[2]; // per side, bone -> label lengths
|
||||
for (auto &[t, n] : label_hits) {
|
||||
(void)n;
|
||||
const auto it = labels.find(t);
|
||||
if (it == labels.end()) continue;
|
||||
const double ts = (t - t0) / 1e9;
|
||||
for (const Rec &r : recs) {
|
||||
if (std::fabs(r.ts - ts) > 1e-6) continue;
|
||||
const Label *best = nullptr;
|
||||
double bd = 0.06;
|
||||
for (const Label &l : it->second) {
|
||||
const double d = norm(l.kp[0] - r.pts[0][0]);
|
||||
if (l.conf >= 0.3 && d < bd) bd = d, best = &l;
|
||||
}
|
||||
if (!best) {
|
||||
++unmatched;
|
||||
continue;
|
||||
}
|
||||
++matched;
|
||||
if (dp) { // time id stereo, then measured, unfiltered fit and label landmarks
|
||||
std::fprintf(dp, "%.4f %d %d", r.ts, r.id, int(r.stereo));
|
||||
for (const V3 *P : {r.pts[0], r.pts[3], best->kp})
|
||||
for (int i = 0; i < 21; ++i) std::fprintf(dp, " %.4f %.4f %.4f", P[i][0], P[i][1], P[i][2]);
|
||||
std::fputc('\n', dp);
|
||||
}
|
||||
for (int v = 0; v < 4; ++v) {
|
||||
double s = 0;
|
||||
for (int i = 0; i < 21; ++i) s += norm(r.pts[v][i] - best->kp[i]) / 21;
|
||||
e[v].push_back(s * 1000);
|
||||
e_tip[v].push_back(norm(r.pts[v][8] - best->kp[8]) * 1000);
|
||||
if (v == 0 || v == 3) {
|
||||
by_views[r.stereo][v ? 1 : 0][0].push_back(s * 1000);
|
||||
by_views[r.stereo][v ? 1 : 0][1].push_back(norm(r.pts[v][8] - best->kp[8]) * 1000);
|
||||
}
|
||||
if (v == 0 || v == 3)
|
||||
for (int i = 0; i < 21; ++i) per[v ? 1 : 0][i].push_back(norm(r.pts[v][i] - best->kp[i]) * 1000);
|
||||
}
|
||||
for (int b = 0; b < 20; ++b)
|
||||
label_len[r.right][b].push_back(norm(best->kp[kBones[b][1]] - best->kp[kBones[b][0]]));
|
||||
}
|
||||
}
|
||||
std::printf("against the labels (%d hands matched within 6 cm of the wrist, %d not): mean landmark error median "
|
||||
"measured %.1f, One Euro %.1f, skeleton %.1f (unfiltered %.1f) mm (90%% %.1f, %.1f, %.1f, %.1f); "
|
||||
"index tip median %.1f, %.1f, %.1f (%.1f) mm\n",
|
||||
matched, unmatched, quantile(e[0], 0.5), quantile(e[1], 0.5), quantile(e[2], 0.5), quantile(e[3], 0.5),
|
||||
quantile(e[0], 0.9), quantile(e[1], 0.9), quantile(e[2], 0.9), quantile(e[3], 0.9), quantile(e_tip[0], 0.5),
|
||||
quantile(e_tip[1], 0.5), quantile(e_tip[2], 0.5), quantile(e_tip[3], 0.5));
|
||||
for (int st = 1; st >= 0; --st)
|
||||
std::printf(" %s (%zu): mean landmark error median measured %.1f, skeleton unfiltered %.1f mm; index tip %.1f, %.1f mm\n",
|
||||
st ? "stereo" : "one view", by_views[st][0][0].size(), quantile(by_views[st][0][0], 0.5),
|
||||
quantile(by_views[st][1][0], 0.5), quantile(by_views[st][0][1], 0.5), quantile(by_views[st][1][1], 0.5));
|
||||
for (int v = 0; v < 2; ++v) {
|
||||
std::printf(" per landmark, %s, median mm:", v ? "skeleton unfiltered" : "measured");
|
||||
for (int i = 0; i < 21; ++i) std::printf(" %.1f", quantile(per[v][i], 0.5));
|
||||
std::printf("\n");
|
||||
}
|
||||
if (dp) std::fclose(dp);
|
||||
for (int s = 0; s < 2; ++s) {
|
||||
if (label_len[s].empty()) continue;
|
||||
const skel::Shape now = sk.shape(s ? 1 : -1, 1.0);
|
||||
std::printf(" %s bones, ours - the labels' median, mm:", s ? "right" : "left ");
|
||||
for (int b = 0; b < 20; ++b) std::printf(" %+.1f", (now.len[b] - quantile(label_len[s].at(b), 0.5)) * 1000);
|
||||
std::printf("\n");
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace skeval
|
||||
+42
-232
@@ -5,8 +5,6 @@
|
||||
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <cstdarg>
|
||||
#include <cstdio>
|
||||
#include <set>
|
||||
|
||||
namespace {
|
||||
@@ -101,16 +99,6 @@ void Pool::run(std::vector<std::function<void()>> &jobs) {
|
||||
|
||||
// ------------------------------------------------------------------------- tracker
|
||||
|
||||
void Tracker::note(const char *fmt, ...) const {
|
||||
if (!trace_) return;
|
||||
char buf[512];
|
||||
va_list ap;
|
||||
va_start(ap, fmt);
|
||||
std::vsnprintf(buf, sizeof buf, fmt, ap);
|
||||
va_end(ap);
|
||||
trace_->push_back(buf);
|
||||
}
|
||||
|
||||
Tracker::Tracker(const std::map<std::string, Camera> &cams, const Nets &nets, Pool &pool, int max_views)
|
||||
: nets_(nets), pool_(pool), max_views_(max_views) {
|
||||
for (const auto &[name, cam] : cams) {
|
||||
@@ -126,13 +114,6 @@ Tracker::Tracker(const std::map<std::string, Camera> &cams, const Nets &nets, Po
|
||||
}
|
||||
}
|
||||
|
||||
// turn a crop so the expected shoulder-to-hand direction at p points up
|
||||
double Tracker::tile_rotation(const Camera &cam, V3 p) const {
|
||||
const V3 d = unit(p - kShoulders[p[0] > 0 ? 0 : 1]);
|
||||
const V2 a = cam.project(p, nullptr), b = cam.project(p + d * 0.05, nullptr);
|
||||
return std::atan2(b[0] - a[0], -(b[1] - a[1]));
|
||||
}
|
||||
|
||||
void Tracker::add_tiles(const Camera &cam, double frac, int gx, int gy) {
|
||||
const double s = frac * std::max(cam.width, cam.height);
|
||||
for (int i = 0; i < gx; ++i)
|
||||
@@ -150,26 +131,6 @@ void Tracker::add_tiles(const Camera &cam, double frac, int gx, int gy) {
|
||||
}
|
||||
}
|
||||
|
||||
bool Tracker::blind(int id, int64_t until_ns) {
|
||||
const auto h = hands_.find(id);
|
||||
if (h == hands_.end() || !h->second.has_pts) return false;
|
||||
blind_id_ = id, blind_until_ = until_ns, blind_at_ = h->second.pts[9];
|
||||
note("blind %d", id);
|
||||
return true;
|
||||
}
|
||||
|
||||
std::vector<std::string> Tracker::describe_tiles() const {
|
||||
std::vector<std::string> out;
|
||||
for (size_t i = 0; i < tiles_.size(); ++i) {
|
||||
const Tile &t = tiles_[i];
|
||||
char buf[200];
|
||||
std::snprintf(buf, sizeof buf, "tile %zu %s %.0f %.0f size %.0f rot %.2f weight %.2f", i, t.cam->name.c_str(), t.center[0],
|
||||
t.center[1], t.size, t.rotation, t.weight);
|
||||
out.push_back(buf);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
double Tracker::interval() const {
|
||||
double fastest = -1;
|
||||
for (const auto &[id, h] : hands_)
|
||||
@@ -255,8 +216,6 @@ bool Tracker::hand_3d(Hand &hand, std::vector<View *> views, int64_t t_ns) {
|
||||
}
|
||||
pts[k] = triangulate(origins.data(), dirs.data(), w.data(), n, &res[k]);
|
||||
}
|
||||
double res_pt[21];
|
||||
std::copy(res.begin(), res.end(), res_pt);
|
||||
std::nth_element(res.begin(), res.begin() + 10, res.end());
|
||||
const double residual = res[10];
|
||||
// the views disagree: two different hands; keep the stronger. Rays to two different
|
||||
@@ -275,7 +234,6 @@ bool Tracker::hand_3d(Hand &hand, std::vector<View *> views, int64_t t_ns) {
|
||||
if (z > 0 && norm(at - palm_centre(v->lm)) < 0.5 * hand_size(v->lm)) v->hand = -2;
|
||||
}
|
||||
++stats.splits;
|
||||
note("split %d residual %.3f keep %s", hand.id, residual, best->cam->name.c_str());
|
||||
return hand_3d(hand, {best}, t_ns);
|
||||
}
|
||||
// learn how big this user's hand is compared to the model's average hand
|
||||
@@ -292,9 +250,6 @@ bool Tracker::hand_3d(Hand &hand, std::vector<View *> views, int64_t t_ns) {
|
||||
hand.scale += 0.1 * (std::clamp(t[t.size() / 2] / m[m.size() / 2], 0.8, 1.6) - hand.scale);
|
||||
std::copy(pts, pts + 21, hand.pts);
|
||||
hand.residual = residual;
|
||||
std::copy(res_pt, res_pt + 21, hand.res_pt);
|
||||
hand.eye = {0, 0, 0};
|
||||
for (View *v : views) hand.eye = hand.eye + v->cam->origin * (1.0 / views.size());
|
||||
for (View *v : views) {
|
||||
V3 mono[21];
|
||||
if (!single_view(*v->cam, v->lm, hand.scale, mono)) continue;
|
||||
@@ -312,16 +267,10 @@ bool Tracker::hand_3d(Hand &hand, std::vector<View *> views, int64_t t_ns) {
|
||||
}
|
||||
std::copy(pts, pts + 21, hand.pts);
|
||||
hand.residual = -1;
|
||||
std::fill(hand.res_pt, hand.res_pt + 21, -1.0);
|
||||
hand.eye = cam.origin;
|
||||
}
|
||||
hand.has_pts = true;
|
||||
hand.nviews = int(views.size());
|
||||
for (View *v : views) {
|
||||
hand.right_score += 0.2 * (v->lm.right - hand.right_score);
|
||||
const double r = std::clamp(v->lm.right, 0.05, 0.95);
|
||||
hand.side_evidence = 0.95 * hand.side_evidence + std::log(r / (1 - r));
|
||||
}
|
||||
for (View *v : views) hand.right_score += 0.2 * (v->lm.right - hand.right_score);
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -419,7 +368,6 @@ void Tracker::associate() {
|
||||
++stats.created;
|
||||
}
|
||||
++stats.merged;
|
||||
note("repair %s %s", cams[i]->name.c_str(), cams[j]->name.c_str());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -443,16 +391,13 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
if (v.hand == id) have.insert(v.cam->name);
|
||||
if (int(have.size()) >= max_views_) continue;
|
||||
std::vector<std::pair<double, View>> options;
|
||||
// a hand missed last step (held): look where it was heading, not where it was
|
||||
const V3 ahead = reacquire_s_ > 0 && hand.seen_ns < last_ns_ ? hand.dpalm * std::min((t_ns - hand.seen_ns) / 1e9, 0.15)
|
||||
: V3{0, 0, 0};
|
||||
for (auto &[name, cam] : cams_) {
|
||||
if (have.count(name) || !images.count(name)) continue;
|
||||
V2 uv[21];
|
||||
bool front = true;
|
||||
for (int k = 0; k < 21; ++k) {
|
||||
double z;
|
||||
uv[k] = cam->project(hand.pts[k] + ahead, &z);
|
||||
uv[k] = cam->project(hand.pts[k], &z);
|
||||
front = front && z > 0;
|
||||
}
|
||||
const V2 centre = (uv[0] + uv[5] + uv[9] + uv[13] + uv[17]) * 0.2;
|
||||
@@ -476,29 +421,14 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
for (int k = 0; k < int(vs.size()) && k < max_views_; ++k) chosen.push_back(vs[k]);
|
||||
}
|
||||
std::stable_sort(chosen.begin(), chosen.end(), [](View *a, View *b) { return a->has_lm > b->has_lm; });
|
||||
if (int(chosen.size()) > hand_budget_) {
|
||||
for (size_t k = hand_budget_; k < chosen.size(); ++k) note("over-budget %d %s", chosen[k]->hand, chosen[k]->cam->name.c_str());
|
||||
chosen.resize(hand_budget_);
|
||||
}
|
||||
if (int(chosen.size()) > hand_budget_) chosen.resize(hand_budget_);
|
||||
run_landmarks(images, chosen);
|
||||
const bool blinding = blind_until_ > t_ns;
|
||||
auto blinded = [&](const View &v) { // a view near where the blinded hand was
|
||||
if (!blinding || !v.has_lm) return false;
|
||||
if (v.hand == blind_id_) return true;
|
||||
const auto h = hands_.find(v.hand);
|
||||
if (h != hands_.end() && h->second.frames >= 2) return false; // another hand, established
|
||||
V3 g[21];
|
||||
return single_view(*v.cam, v.lm, 1.0, g) && norm(g[9] - blind_at_) < 0.15;
|
||||
};
|
||||
for (View *v : chosen)
|
||||
if (blinded(*v)) v->lm.presence = 0;
|
||||
std::vector<View *> kept;
|
||||
for (View *v : chosen) {
|
||||
if (misread_guard_ && v->has_lm) { // see set_misread_guard
|
||||
const auto h = hands_.find(v->hand);
|
||||
if (h != hands_.end() && h->second.frames >= 5 && std::fabs(v->lm.right - h->second.right_score) > 0.7) {
|
||||
++stats.lost;
|
||||
note("misread %d %s right %.2f hand %.2f", v->hand, v->cam->name.c_str(), v->lm.right, h->second.right_score);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
@@ -508,7 +438,6 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
kept.push_back(v);
|
||||
} else {
|
||||
++(v->frames > 0 ? stats.lost : stats.handoff_miss);
|
||||
note("%s %d %s presence %.2f", v->frames > 0 ? "lost" : "miss", v->hand, v->cam->name.c_str(), v->lm.presence);
|
||||
}
|
||||
}
|
||||
// the same hand twice in one camera: keep the more confident
|
||||
@@ -519,7 +448,7 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
bool dup = false;
|
||||
for (View &o : next) dup = dup || (o.cam == v->cam && same_hand(o.lm, v->lm, size));
|
||||
if (!dup) next.push_back(*v);
|
||||
else ++stats.dups, note("dup %d %s", v->hand, v->cam->name.c_str());
|
||||
else ++stats.dups;
|
||||
}
|
||||
views_ = next;
|
||||
views_.insert(views_.end(), waiting.begin(), waiting.end());
|
||||
@@ -527,74 +456,13 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
// 3. search for missing hands
|
||||
std::set<int> tracked;
|
||||
for (View &v : views_) tracked.insert(v.hand);
|
||||
// hands to re-acquire: held ones (not seen now, not yet forgotten) and recently forgotten ones,
|
||||
// unless a tracked hand is already there
|
||||
std::vector<Lost> targets;
|
||||
if (reacquire_s_ > 0) {
|
||||
std::vector<V3> at;
|
||||
for (auto &[id, h] : hands_)
|
||||
if (tracked.count(id) && h.has_pts) at.push_back(h.pts[9]);
|
||||
auto taken = [&](V3 p) {
|
||||
for (const V3 &q : at)
|
||||
if (norm(p - q) < reacquire_clear_) return true;
|
||||
return false;
|
||||
};
|
||||
lost_.erase(std::remove_if(lost_.begin(), lost_.end(),
|
||||
[&](const Lost &l) { return (t_ns - l.t_ns) / 1e9 > reacquire_s_ || taken(l.palm); }),
|
||||
lost_.end());
|
||||
for (auto &[id, h] : hands_)
|
||||
if (!tracked.count(id) && h.has_pts && h.frames >= 3 && !taken(h.pts[9]))
|
||||
targets.push_back({id, (h.pts[0] + h.pts[5] + h.pts[9] + h.pts[13] + h.pts[17]) * 0.2, h.dpalm, h.seen_ns});
|
||||
for (const Lost &l : lost_) targets.push_back(l);
|
||||
std::sort(targets.begin(), targets.end(), [](const Lost &a, const Lost &b) { return a.t_ns > b.t_ns; });
|
||||
}
|
||||
const bool due = (t_ns - search_ns_) / 1e9 >= kSearchInterval - 0.01 || (reacquire_fast_ && !targets.empty());
|
||||
if (tracked.size() >= 2 && due && trace_) {
|
||||
std::string ids;
|
||||
for (int id : tracked) ids += " " + std::to_string(id);
|
||||
note("search-blocked tracked%s", ids.c_str());
|
||||
}
|
||||
if (tracked.size() < 2 && due) {
|
||||
const bool rotation_due = (t_ns - search_ns_) / 1e9 >= kSearchInterval - 0.01;
|
||||
search_ns_ = rotation_due ? t_ns : search_ns_;
|
||||
const int budget = tracked.empty() ? search_budget_ : search_budget1_;
|
||||
if (tracked.size() < 2 && (t_ns - search_ns_) / 1e9 >= kSearchInterval - 0.01) {
|
||||
search_ns_ = t_ns;
|
||||
const int budget = tracked.empty() ? search_budget_ : std::max(1, search_budget_ - 1);
|
||||
for (Tile &t : tiles_)
|
||||
if (images.count(t.cam->name)) t.credit += t.weight;
|
||||
std::vector<Tile *> picked;
|
||||
// first, tiles on where lost hands should be now
|
||||
std::vector<Tile> aimed;
|
||||
aimed.reserve(size_t(reacquire_tiles_));
|
||||
for (const Lost &l : targets) {
|
||||
const double age = (t_ns - l.t_ns) / 1e9;
|
||||
const V3 p = l.palm + l.vel * std::min(age, 0.25);
|
||||
std::vector<std::pair<double, const Camera *>> cams;
|
||||
for (auto &[name, cam] : cams_) {
|
||||
if (!images.count(name)) continue;
|
||||
double z;
|
||||
const V2 uv = cam->project(p, &z);
|
||||
const double m = 0.03;
|
||||
if (z <= 0 || uv[0] < m * cam->width || uv[0] > (1 - m) * cam->width || uv[1] < m * cam->height ||
|
||||
uv[1] > (1 - m) * cam->height)
|
||||
continue;
|
||||
cams.push_back({cam->off_axis(uv), cam});
|
||||
}
|
||||
std::sort(cams.begin(), cams.end());
|
||||
for (auto &[off, cam] : cams) {
|
||||
if (int(aimed.size()) >= reacquire_tiles_) break;
|
||||
const V2 uv = cam->project(p, nullptr);
|
||||
const double dist = std::max(0.1, norm(p - cam->origin)), full = std::max(cam->width, cam->height);
|
||||
const double hand_px = cam->fx * 0.2 / dist; // a hand's length, roughly, near the centre
|
||||
const double size = std::clamp(hand_px * 2.5 * (1 + 1.5 * age), 0.25 * full, 0.65 * full);
|
||||
aimed.push_back(Tile{cam, uv, size, tile_rotation(*cam, p), 0, 0});
|
||||
}
|
||||
if (int(aimed.size()) >= reacquire_tiles_) break;
|
||||
}
|
||||
for (Tile &t : aimed) picked.push_back(&t);
|
||||
// aimed tiles share the budget, but the rotation keeps at least one tile (unless it's not due:
|
||||
// with every_step, aimed tiles go up between the rotation's rounds)
|
||||
const int rotation = rotation_due ? std::max(1, budget - int(aimed.size())) : 0;
|
||||
if (rotation_due)
|
||||
for (Tile &t : tiles_)
|
||||
if (images.count(t.cam->name)) t.credit += t.weight;
|
||||
for (int b = 0; b < rotation; ++b) {
|
||||
for (int b = 0; b < budget; ++b) {
|
||||
Tile *best = nullptr;
|
||||
for (Tile &t : tiles_)
|
||||
if (images.count(t.cam->name) && std::find(picked.begin(), picked.end(), &t) == picked.end() &&
|
||||
@@ -604,66 +472,36 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
best->credit = 0;
|
||||
picked.push_back(best);
|
||||
}
|
||||
if (!picked.empty()) {
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
std::vector<std::vector<Palm>> found(picked.size());
|
||||
std::vector<std::function<void()>> jobs;
|
||||
for (size_t i = 0; i < picked.size(); ++i) {
|
||||
Tile *t = picked[i];
|
||||
const Image &img = images.at(t->cam->name);
|
||||
jobs.push_back([this, t, &img, &found, i] { found[i] = nets_.palms(img, t->center, t->size, t->rotation); });
|
||||
}
|
||||
pool_.run(jobs);
|
||||
stats.palm_calls += int(picked.size());
|
||||
stats.palm_ms += ms_since(t0);
|
||||
++stats.palm_batches;
|
||||
std::vector<View> fresh;
|
||||
std::string tl;
|
||||
if (trace_)
|
||||
for (Tile *t : picked)
|
||||
tl += " " + t->cam->name + "#" +
|
||||
(t >= tiles_.data() && t < tiles_.data() + tiles_.size() ? std::to_string(t - tiles_.data()) : std::string("aim"));
|
||||
int nfound = 0, nnear = 0;
|
||||
for (size_t i = 0; i < picked.size(); ++i)
|
||||
for (const Palm &p : found[i]) {
|
||||
const Roi roi = p.roi();
|
||||
bool near = false;
|
||||
for (auto *list : {&views_, &fresh})
|
||||
for (View &v : *list)
|
||||
near = near || (v.cam == picked[i]->cam && norm(v.roi.center - roi.center) < near_frac_ * roi.size);
|
||||
++nfound, nnear += near;
|
||||
if (!near) fresh.push_back({picked[i]->cam, roi, 0, {}, false, 0});
|
||||
}
|
||||
note("search tracked %zu tiles%s palms %d near-tracked %d", tracked.size(), tl.c_str(), nfound, nnear);
|
||||
std::vector<View *> ptrs;
|
||||
for (View &v : fresh) ptrs.push_back(&v);
|
||||
run_landmarks(images, ptrs);
|
||||
for (View &v : fresh)
|
||||
if (blinded(v)) v.lm.presence = 0;
|
||||
for (View &v : fresh) {
|
||||
// a palm crop on a hand already tracked in this camera (with set_near under 0.5, palms
|
||||
// right beside a tracked hand get tried). Stricter than same_hand: a hand beside it has its
|
||||
// palm about half a hand length away.
|
||||
bool dup = false;
|
||||
for (const View &o : views_) {
|
||||
if (o.cam != v.cam || !o.has_lm) continue;
|
||||
double d = 0;
|
||||
for (int k = 0; k < 21; ++k) d += norm(o.lm.pts[k] - v.lm.pts[k]) / 21;
|
||||
dup = dup || d < 0.25 * hand_size(v.lm) || norm(palm_centre(o.lm) - palm_centre(v.lm)) < 0.25 * hand_size(v.lm);
|
||||
}
|
||||
if (dup && v.lm.presence >= min_presence_) {
|
||||
note("palm %s at %.0f %.0f on a tracked hand", v.cam->name.c_str(), v.roi.center[0], v.roi.center[1]);
|
||||
continue;
|
||||
}
|
||||
note("palm %s at %.0f %.0f size %.0f presence %.2f right %.2f %s", v.cam->name.c_str(), v.roi.center[0],
|
||||
v.roi.center[1], v.roi.size, v.lm.presence, v.lm.right, v.lm.presence >= min_presence_ ? "kept" : "rejected");
|
||||
if (v.lm.presence >= min_presence_) {
|
||||
v.roi = v.lm.next_roi();
|
||||
v.frames = 1;
|
||||
views_.push_back(v);
|
||||
}
|
||||
}
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
std::vector<std::vector<Palm>> found(picked.size());
|
||||
std::vector<std::function<void()>> jobs;
|
||||
for (size_t i = 0; i < picked.size(); ++i) {
|
||||
Tile *t = picked[i];
|
||||
const Image &img = images.at(t->cam->name);
|
||||
jobs.push_back([this, t, &img, &found, i] { found[i] = nets_.palms(img, t->center, t->size, t->rotation); });
|
||||
}
|
||||
pool_.run(jobs);
|
||||
stats.palm_calls += int(picked.size());
|
||||
stats.palm_ms += ms_since(t0);
|
||||
++stats.palm_batches;
|
||||
std::vector<View> fresh;
|
||||
for (size_t i = 0; i < picked.size(); ++i)
|
||||
for (const Palm &p : found[i]) {
|
||||
const Roi roi = p.roi();
|
||||
bool near = false;
|
||||
for (auto *list : {&views_, &fresh})
|
||||
for (View &v : *list) near = near || (v.cam == picked[i]->cam && norm(v.roi.center - roi.center) < 0.5 * roi.size);
|
||||
if (!near) fresh.push_back({picked[i]->cam, roi, 0, {}, false, 0});
|
||||
}
|
||||
std::vector<View *> ptrs;
|
||||
for (View &v : fresh) ptrs.push_back(&v);
|
||||
run_landmarks(images, ptrs);
|
||||
for (View &v : fresh)
|
||||
if (v.lm.presence >= min_presence_) {
|
||||
v.roi = v.lm.next_roi();
|
||||
v.frames = 1;
|
||||
views_.push_back(v);
|
||||
}
|
||||
}
|
||||
|
||||
// 4. give new views a hand: the nearest existing hand in 3D, else a new one
|
||||
@@ -685,9 +523,6 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
best = next_id_++;
|
||||
hands_[best].id = best;
|
||||
++stats.created;
|
||||
note("new %d %s", best, v.cam->name.c_str());
|
||||
} else {
|
||||
note("joins %d %s at %.3f m", best, v.cam->name.c_str(), dist);
|
||||
}
|
||||
v.hand = best;
|
||||
}
|
||||
@@ -711,32 +546,13 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
smooth(h, t_ns);
|
||||
out.push_back(&h);
|
||||
++it;
|
||||
} else if (t_ns - h.seen_ns > (h.frames >= 30 ? int64_t(hold_s_ * 1e9) : 300'000'000)) {
|
||||
note("forget %d after %d frames", h.id, h.frames);
|
||||
if (reacquire_s_ > 0 && h.has_pts && h.frames >= 3)
|
||||
lost_.push_back({h.id, (h.pts[0] + h.pts[5] + h.pts[9] + h.pts[13] + h.pts[17]) * 0.2, h.dpalm, h.seen_ns});
|
||||
} else if (t_ns - h.seen_ns > 300'000'000) {
|
||||
it = hands_.erase(it);
|
||||
++stats.forgotten;
|
||||
} else {
|
||||
++it;
|
||||
}
|
||||
}
|
||||
for (auto &[id, h] : hands_) h.side = -1;
|
||||
if (pair_sides_ && out.size() == 2) {
|
||||
Hand &a = hands_[out[0]->id], &b = hands_[out[1]->id];
|
||||
// > 0: a is the right hand. Evidence first; which wrist is further right only tips a close call.
|
||||
const double e = a.side_evidence - b.side_evidence + 3.0 * std::clamp((a.pts[0][0] - b.pts[0][0]) / 0.15, -1.0, 1.0);
|
||||
if (a.right() == b.right()) note("pair-sides %d %d both %s -> %d is right", a.id, b.id, a.right() ? "R" : "L", e > 0 ? a.id : b.id);
|
||||
a.side = e > 0, b.side = e <= 0;
|
||||
}
|
||||
// the skeleton fit, after the pairing so it fits the side that's published
|
||||
for (const Hand *p : out) {
|
||||
Hand &h = hands_[p->id];
|
||||
if (skeleton_on_)
|
||||
skel_.update(h.fit, h.pts, h.right(), h.right_score, h.nviews, h.residual, h.res_pt, h.eye, h.scale, t_ns);
|
||||
else
|
||||
h.fit.valid = false;
|
||||
}
|
||||
// views split off by a failed triangulation start over as new hands next frame, unless they
|
||||
// were this hand misread (-2, the misread guard: dropped)
|
||||
const size_t before = views_.size();
|
||||
@@ -747,7 +563,6 @@ std::vector<const Hand *> Tracker::step(const std::map<std::string, Image> &imag
|
||||
v.hand = next_id_++;
|
||||
hands_[v.hand].id = v.hand;
|
||||
++stats.created;
|
||||
note("new %d %s (split off)", v.hand, v.cam->name.c_str());
|
||||
}
|
||||
last_ns_ = t_ns;
|
||||
stats.step_ms += ms_since(t_step);
|
||||
@@ -765,7 +580,7 @@ void Tracker::exchange(const std::string &a, const std::string &b) {
|
||||
if (!cams_.count(a) || !cams_.count(b)) return;
|
||||
const Camera *ca = cams_[a], *cb = cams_[b];
|
||||
for (View &v : views_) v.cam = v.cam == ca ? cb : v.cam == cb ? ca : v.cam;
|
||||
for (auto &[id, h] : hands_) h.has_pts = false, h.frames = 0, h.residual = -1, h.fit.valid = false;
|
||||
for (auto &[id, h] : hands_) h.has_pts = false, h.frames = 0, h.residual = -1;
|
||||
}
|
||||
|
||||
void Tracker::drop_camera(const std::string &name) {
|
||||
@@ -788,11 +603,7 @@ std::vector<Seen> Tracker::exhaustive(const std::map<std::string, Image> &images
|
||||
// one crop per palm: tiles overlap, so the same palm turns up several times
|
||||
std::vector<std::pair<double, View>> palms;
|
||||
for (size_t i = 0; i < tiles.size(); ++i)
|
||||
for (const Palm &p : found[i]) {
|
||||
View v{tiles[i]->cam, p.roi(), 0, {}, false, 0};
|
||||
v.frames = int(tiles[i] - tiles_.data()); // the tile, carried in frames (unused here)
|
||||
palms.push_back({p.score, v});
|
||||
}
|
||||
for (const Palm &p : found[i]) palms.push_back({p.score, View{tiles[i]->cam, p.roi(), 0, {}, false, 0}});
|
||||
std::sort(palms.begin(), palms.end(), [](auto &a, auto &b) { return a.first > b.first; });
|
||||
std::vector<View> crops;
|
||||
for (auto &[score, v] : palms) {
|
||||
@@ -813,7 +624,6 @@ std::vector<Seen> Tracker::exhaustive(const std::map<std::string, Image> &images
|
||||
if (dup) continue;
|
||||
V3 pts[21];
|
||||
Seen s{v.cam->name, 0, v.roi, v.lm, {}};
|
||||
s.tile = v.frames;
|
||||
if (single_view(*v.cam, v.lm, 1.0, pts)) s.wrist = pts[0];
|
||||
out.push_back(s);
|
||||
}
|
||||
|
||||
+3
-66
@@ -4,7 +4,6 @@
|
||||
|
||||
#include "calib.h"
|
||||
#include "nets.h"
|
||||
#include "skeleton.h"
|
||||
|
||||
#include <condition_variable>
|
||||
#include <functional>
|
||||
@@ -37,8 +36,7 @@ struct Hand {
|
||||
V3 pts[21]{}; // as measured this frame; the tracker steers crops by these
|
||||
V3 smooth[21]{}; // filtered (One Euro, see Tracker::smooth): publish these
|
||||
bool has_pts = false;
|
||||
double residual = -1; // rms ray distance of the triangulation (m), the landmarks' median; -1: one view
|
||||
double res_pt[21]{}; // each landmark's (-1: one view)
|
||||
double residual = -1; // rms ray distance of the triangulation (m); -1: one view
|
||||
int nviews = 0;
|
||||
double right_score = 0.5; // the model's right-hand score (these images aren't mirrored)
|
||||
double scale = 1.0; // this user's hand size / the model's world landmarks
|
||||
@@ -49,11 +47,7 @@ struct Hand {
|
||||
V3 last_palm{};
|
||||
V3 dpalm{}; // the filter's palm velocity, m/s
|
||||
int64_t smooth_ns = 0;
|
||||
double side_evidence = 0; // log-odds of a right hand, from every reading, fading (Tracker::set_pair_sides)
|
||||
int side = -1; // the side the pairing settled this step (0 left, 1 right), or -1: by right_score
|
||||
V3 eye{}; // the centre of the cameras it was seen with this frame (depth is along eye -> point)
|
||||
skel::Fit fit; // the skeleton fitted to pts (track/skeleton.h); fit.valid when it fitted this frame
|
||||
bool right() const { return side >= 0 ? side == 1 : right_score > 0.5; }
|
||||
bool right() const { return right_score > 0.5; }
|
||||
};
|
||||
|
||||
struct Stats {
|
||||
@@ -77,7 +71,6 @@ struct Seen {
|
||||
Roi roi;
|
||||
Landmarks lm;
|
||||
V3 wrist{}; // exhaustive(): single-view 3D guess at the model's hand size
|
||||
int tile = -1; // exhaustive(): the search tile whose palm gave this crop
|
||||
};
|
||||
|
||||
class Tracker {
|
||||
@@ -107,41 +100,6 @@ public:
|
||||
// view split off where its hand already is in that camera is dropped instead of starting a new
|
||||
// hand. Off by default: the stock model's side is noisier and the first guard costs it tracking.
|
||||
void set_misread_guard(bool on) { misread_guard_ = on; }
|
||||
// Diagnostics (ft-handreplay --timeline): when set, every step appends one line per event that
|
||||
// makes views and hands come and go (lost, handoff miss, search, palm rejected, new, forgotten...),
|
||||
// so a gap in a hand's track can be put down to its cause. Off (null) live.
|
||||
void set_trace(std::vector<std::string> *trace) { trace_ = trace; }
|
||||
// Re-acquiring lost hands: for window_s after a hand (seen 3+ frames) stops being seen, the palm
|
||||
// search puts up to `tiles` of its tiles on where that hand should be, in the cameras that see that
|
||||
// spot best, before its usual rotation; with every_step, it searches on every processed set while
|
||||
// such a hand is missing, not only 5 times a second. window_s 0: off (as before).
|
||||
void set_reacquire(double window_s, int tiles, bool every_step, double clear_m = 0.12) {
|
||||
reacquire_s_ = window_s, reacquire_tiles_ = tiles, reacquire_fast_ = every_step, reacquire_clear_ = clear_m;
|
||||
}
|
||||
// Two hands seen at once are one left and one right: when on, the pair's sides are settled together
|
||||
// each step from each hand's accumulated left/right evidence (plus, weakly, which is further left),
|
||||
// instead of each hand by its own running average, which let both hands be called the same side.
|
||||
void set_pair_sides(bool on) { pair_sides_ = on; }
|
||||
// A palm the search finds is skipped as a hand already tracked when its crop's centre is within
|
||||
// this share of the crop's size of a tracked view's crop in that camera (0.5 before 2026-10-08).
|
||||
// Smaller lets a hand right beside a tracked one be found; its landmarks then decide (same_hand).
|
||||
void set_near(double frac) { near_frac_ = frac; }
|
||||
// How long a lost hand tracked for 30+ frames is kept (not published) while its crops are retried
|
||||
// where it was heading, before it's forgotten. Newer hands get 0.3 s (as all did before 2026-10-08).
|
||||
void set_hold(double s) { hold_s_ = std::max(0.3, s); }
|
||||
// Palm search tiles per round with no hand tracked, and with one (3 and 2 before 2026-10-08). A
|
||||
// round runs its tiles in parallel, one per model thread, so up to --threads tiles cost one
|
||||
// tile's wall time (but each tile's CPU).
|
||||
void set_search_budget(int none, int one) { search_budget_ = std::max(1, none), search_budget1_ = std::max(1, one); }
|
||||
// Testing re-acquisition (ft-handreplay --drop-every): until until_ns, the landmark model "misses"
|
||||
// this hand: its views read presence 0, and new views within 15 cm of where it was are dropped.
|
||||
// Returns false if there's no such hand with 3D.
|
||||
bool blind(int id, int64_t until_ns);
|
||||
// The palm search's tiles, a line each: index, camera, centre, size, rotation, weight.
|
||||
std::vector<std::string> describe_tiles() const;
|
||||
// The skeleton fit (track/skeleton.h), after each hand's 3D. On by default.
|
||||
void set_skeleton(bool on) { skeleton_on_ = on; }
|
||||
skel::Skeleton &skeleton() { return skel_; }
|
||||
// One view's 3D hand: each landmark along its ray, as far as how big the palm looks says
|
||||
// for a hand `scale` times the model's (Hand::scale). False if the palm is degenerate.
|
||||
static bool single_view(const Camera &cam, const Landmarks &lm, double scale, V3 out[21]);
|
||||
@@ -173,34 +131,13 @@ private:
|
||||
static void smooth(Hand &h, int64_t t_ns);
|
||||
bool inside(const Camera &cam, V2 uv) const;
|
||||
void add_tiles(const Camera &cam, double frac, int gx, int gy);
|
||||
double tile_rotation(const Camera &cam, V3 p) const;
|
||||
// a hand that stopped being seen: where and how fast it was
|
||||
struct Lost {
|
||||
int id;
|
||||
V3 palm, vel;
|
||||
int64_t t_ns;
|
||||
};
|
||||
std::vector<Lost> lost_;
|
||||
bool pair_sides_ = false;
|
||||
int blind_id_ = 0;
|
||||
int64_t blind_until_ = 0;
|
||||
V3 blind_at_{};
|
||||
double near_frac_ = 0.5;
|
||||
double hold_s_ = 0.3;
|
||||
double reacquire_s_ = 0, reacquire_clear_ = 0.12; // a lost hand this near a tracked one isn't aimed at
|
||||
int reacquire_tiles_ = 2;
|
||||
bool reacquire_fast_ = false;
|
||||
|
||||
std::map<std::string, const Camera *> cams_;
|
||||
const Nets &nets_;
|
||||
Pool &pool_;
|
||||
int max_views_, hand_budget_ = 4, search_budget_ = 3, search_budget1_ = 2;
|
||||
int max_views_, hand_budget_ = 4, search_budget_ = 3;
|
||||
double min_presence_ = 0.5, keep_presence_ = 0.5;
|
||||
bool misread_guard_ = false;
|
||||
std::vector<std::string> *trace_ = nullptr;
|
||||
void note(const char *fmt, ...) const __attribute__((format(printf, 2, 3)));
|
||||
bool skeleton_on_ = true;
|
||||
skel::Skeleton skel_;
|
||||
std::vector<View> views_;
|
||||
std::map<int, Hand> hands_;
|
||||
std::vector<Tile> tiles_;
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
# Frametop host for Linux
|
||||
|
||||
A Linux PC can be a host for Frametop's remote displays with Frametop's build of Vibepollo, from [Frametop/frametop-vibepollo](https://github.com/Frametop/frametop-vibepollo). Vibepollo's Linux version is in beta, and Frametop hasn't been tested on a Linux host yet. If you try it, please report what works and what doesn't, on [GitHub](https://github.com/Frametop/frametop/issues) or in the [Frametop Discord](https://discord.gg/W3X9f7z3Bc).
|
||||
|
||||
There's no setup script for Linux. These steps use Vibepollo's own installer with Frametop's package.
|
||||
|
||||
## Requirements
|
||||
|
||||
These are Vibepollo's:
|
||||
|
||||
- Arch Linux or CachyOS, x86_64. Other distributions have no package.
|
||||
- KDE Plasma 6 on Wayland, started by SDDM or Plasma Login Manager. GNOME, X11 sessions and other compositors aren't streamed.
|
||||
- Linux 6.16 or newer, with the headers for the running kernel. The installer installs them if they're missing.
|
||||
- A graphics card with a hardware H.264 encoder: NVENC on NVIDIA, VAAPI on AMD and Intel.
|
||||
- The PC and the Steam Frame on the same network.
|
||||
|
||||
## Set up
|
||||
|
||||
1. Download Frametop's package and Vibepollo's installer, and check the package:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/frametop-host && cd ~/frametop-host
|
||||
release=https://github.com/Frametop/frametop-vibepollo/releases/download/frametop-2.0.0-2
|
||||
curl -fLO "$release/vibepollo-2.0.0-1-x86_64.pkg.tar.zst"
|
||||
curl -fLO "$release/vibepollo-2.0.0-1-x86_64.pkg.tar.zst.sha256"
|
||||
sha256sum -c vibepollo-2.0.0-1-x86_64.pkg.tar.zst.sha256
|
||||
curl -fLO https://raw.githubusercontent.com/Frametop/frametop-vibepollo/frametop-2.0.0-2/scripts/linux_install.sh
|
||||
```
|
||||
|
||||
`sha256sum -c` should print `OK`.
|
||||
2. Install it:
|
||||
|
||||
```bash
|
||||
sudo bash linux_install.sh --package ./vibepollo-2.0.0-1-x86_64.pkg.tar.zst
|
||||
```
|
||||
|
||||
The installer checks the requirements, installs the kernel headers if needed, installs the package, and opens the firewall (firewalld or ufw). Given a package, it doesn't add Vibepollo's own package repository. Don't add that repository later: an update from it would replace Frametop's build.
|
||||
3. Restart the PC after the first install, and log in to Plasma (Wayland).
|
||||
4. Open https://localhost:47990 on the PC and create the Web UI login. The Steam Frame signs in with it once.
|
||||
5. If you'll use virtual displays, turn on "Remove Remote Monitor when its client disconnects" in the Web UI's settings. Then a virtual display goes away when Frametop disconnects it, as on Windows.
|
||||
6. On the Steam Frame, open Frametop Remote Displays and choose Add computer. Pick this PC, sign in with the Web UI login, and choose the displays to show.
|
||||
|
||||
If the installer couldn't open the firewall, open it yourself:
|
||||
|
||||
```bash
|
||||
sudo firewall-cmd --permanent --add-service=vibepollo && sudo firewall-cmd --reload # firewalld
|
||||
sudo ufw allow Vibepollo # ufw
|
||||
```
|
||||
|
||||
## Good to know
|
||||
|
||||
- The resolution switch and the HDR guard of Frametop's build are Windows-only for now. On Linux, a monitor keeps its own resolution and the stream scales it.
|
||||
- Vibepollo runs as system services. `sudo systemctl status vibepollo-session-controller.service vibepollo.service` shows them, and `sudo journalctl -u vibepollo-session-controller.service -u vibepollo.service --since '-10 minutes'` shows their log. Don't start `vibepollo.service` yourself: the session controller starts it when you log in.
|
||||
- Settings, the Web UI login and pairings are in `/var/lib/vibepollo`.
|
||||
- Vibepollo's [Linux install guide](https://github.com/Frametop/frametop-vibepollo/blob/frametop-2.0.0-2/docs/linux/install.md) has the details and troubleshooting.
|
||||
|
||||
## Remove
|
||||
|
||||
```bash
|
||||
sudo vibepollo reset # only to delete its settings, login and pairings too
|
||||
sudo pacman -R vibepollo
|
||||
```
|
||||
|
||||
## Source
|
||||
|
||||
Vibepollo is [Nonary's](https://github.com/Nonary/Vibepollo), and Frametop's build is [Frametop/frametop-vibepollo](https://github.com/Frametop/frametop-vibepollo), both under GPL-3.0.
|
||||
@@ -1,139 +0,0 @@
|
||||
-- Frametop Streams: a menu bar icon for this Mac's Frametop displays (its Sunshine instances, one
|
||||
-- per display). Click a display to turn it on or off, or turn them all on or off. It runs
|
||||
-- ft-sunshine. frametop-host-setup.sh compiles it into ~/Applications as a stay-open app with no
|
||||
-- Dock icon and starts it at login (the LaunchAgent frametop.streams).
|
||||
use AppleScript version "2.4"
|
||||
use framework "Foundation"
|
||||
use framework "AppKit"
|
||||
use scripting additions
|
||||
|
||||
property statusItem : missing value
|
||||
property theMenu : missing value
|
||||
property theTimer : missing value
|
||||
property tool : ""
|
||||
property logFile : ""
|
||||
property busyPid : 0 -- the ft-sunshine started from the menu, while it runs
|
||||
property busyText : ""
|
||||
property ticks : 0
|
||||
|
||||
on run
|
||||
set tool to quoted form of ((POSIX path of (path to application support folder from user domain)) & "Frametop/sunshine/bin/ft-sunshine")
|
||||
set logFile to (POSIX path of (path to library folder from user domain)) & "Logs/Frametop/streams.log"
|
||||
set statusItem to current application's NSStatusBar's systemStatusBar()'s statusItemWithLength:(current application's NSVariableStatusItemLength)
|
||||
set icon to current application's NSImage's imageWithSystemSymbolName:"display.2" accessibilityDescription:"Frametop Streams"
|
||||
icon's setTemplate:true
|
||||
statusItem's button()'s setImage:icon
|
||||
statusItem's button()'s setToolTip:"Frametop Streams"
|
||||
set theMenu to current application's NSMenu's alloc()'s initWithTitle:"Frametop Streams"
|
||||
theMenu's setAutoenablesItems:false
|
||||
theMenu's setDelegate:me
|
||||
statusItem's setMenu:theMenu
|
||||
refresh()
|
||||
set theTimer to current application's NSTimer's scheduledTimerWithTimeInterval:2 target:me selector:"tick:" userInfo:(missing value) repeats:true
|
||||
end run
|
||||
|
||||
-- Every 2 s while a change runs, otherwise every 10 s: the icon dims while every display is
|
||||
-- off, whatever turned them off.
|
||||
on tick:aTimer
|
||||
set ticks to ticks + 1
|
||||
if busyPid is not 0 or ticks mod 5 = 0 then refresh()
|
||||
end tick:
|
||||
|
||||
on menuNeedsUpdate:aMenu
|
||||
refresh()
|
||||
end menuNeedsUpdate:
|
||||
|
||||
on refresh()
|
||||
if busyPid is not 0 then
|
||||
try
|
||||
do shell script "kill -0 " & busyPid
|
||||
on error
|
||||
set busyPid to 0
|
||||
end try
|
||||
end if
|
||||
theMenu's removeAllItems()
|
||||
try
|
||||
set rows to paragraphs of (do shell script tool & " status --tsv")
|
||||
on error msg
|
||||
addItem(msg, missing value, missing value, false)
|
||||
addItem("Run frametop-host-setup.sh to set this Mac up.", missing value, missing value, false)
|
||||
theMenu's addItem:(current application's NSMenuItem's separatorItem())
|
||||
addItem("Quit Frametop Streams", "quitApp:", missing value, true)
|
||||
statusItem's button()'s setAppearsDisabled:true
|
||||
return
|
||||
end try
|
||||
set ready to busyPid is 0
|
||||
set anyOn to false
|
||||
set saved to AppleScript's text item delimiters
|
||||
set AppleScript's text item delimiters to tab
|
||||
repeat with r in rows
|
||||
set f to text items of r
|
||||
set n to (item 1 of f) as integer
|
||||
set instState to item 2 of f
|
||||
set label to item 3 of f
|
||||
if instState is "waiting" then
|
||||
set label to label & " (waiting for it to be connected)"
|
||||
else if instState is "starting" then
|
||||
set label to label & " (starting)"
|
||||
else if instState is "stopped" then
|
||||
set label to label & " (stopped)"
|
||||
end if
|
||||
set mi to addItem(label, "toggle:", n, ready)
|
||||
-- A check while it's on; a dash while it's on but not streaming yet.
|
||||
if instState is "on" then
|
||||
mi's setState:(current application's NSControlStateValueOn)
|
||||
set anyOn to true
|
||||
else if instState is "waiting" or instState is "starting" then
|
||||
mi's setState:(current application's NSControlStateValueMixed)
|
||||
set anyOn to true
|
||||
end if
|
||||
end repeat
|
||||
set AppleScript's text item delimiters to saved
|
||||
theMenu's addItem:(current application's NSMenuItem's separatorItem())
|
||||
if not ready then addItem(busyText, missing value, missing value, false)
|
||||
addItem("Turn All On", "allOn:", missing value, ready)
|
||||
addItem("Turn All Off", "allOff:", missing value, ready)
|
||||
theMenu's addItem:(current application's NSMenuItem's separatorItem())
|
||||
addItem("Quit Frametop Streams", "quitApp:", missing value, true)
|
||||
statusItem's button()'s setAppearsDisabled:(not anyOn)
|
||||
end refresh
|
||||
|
||||
on addItem(itemTitle, itemAction, itemTag, itemEnabled)
|
||||
set mi to current application's NSMenuItem's alloc()'s initWithTitle:itemTitle action:itemAction keyEquivalent:""
|
||||
if itemAction is not missing value then mi's setTarget:me
|
||||
if itemTag is not missing value then mi's setTag:itemTag
|
||||
mi's setEnabled:itemEnabled
|
||||
theMenu's addItem:mi
|
||||
return mi
|
||||
end addItem
|
||||
|
||||
-- A display that's off or stopped turns on; one that's on, waiting or starting turns off.
|
||||
on toggle:sender
|
||||
set n to (sender's tag()) as integer
|
||||
if ((sender's state()) as integer) = 0 then
|
||||
change("on " & n, "Turning a display on…")
|
||||
else
|
||||
change("off " & n, "Turning a display off…")
|
||||
end if
|
||||
end toggle:
|
||||
|
||||
on allOn:sender
|
||||
change("on", "Turning every display on…")
|
||||
end allOn:
|
||||
|
||||
on allOff:sender
|
||||
change("off", "Turning every display off…")
|
||||
end allOff:
|
||||
|
||||
-- In the background: on waits up to 20 s for the instances to answer. Its output goes to the log.
|
||||
on change(args, what)
|
||||
set busyText to what
|
||||
set logDir to do shell script "dirname " & quoted form of logFile
|
||||
set busyPid to (do shell script "mkdir -p " & quoted form of logDir & "; { date; " & tool & " " & args & "; } >> " & quoted form of logFile & " 2>&1 & echo $!") as integer
|
||||
refresh()
|
||||
end change
|
||||
|
||||
on quitApp:sender
|
||||
theTimer's invalidate()
|
||||
tell me to quit
|
||||
end quitApp:
|
||||
@@ -1,132 +0,0 @@
|
||||
# Frametop host for macOS
|
||||
|
||||
This kit makes a Mac a host for Frametop's remote displays: the Mac's monitors appear as screens on a Steam Frame. Vibepollo has no macOS build, so the Mac uses LizardByte's Sunshine, one copy (an instance) per display. Frametop Streams, a menu bar icon, turns them on and off.
|
||||
|
||||
## Requirements
|
||||
|
||||
- A Mac with Apple silicon, on macOS 14.2 or later.
|
||||
- An admin account, to install Sunshine in /Applications.
|
||||
- The Mac and the Steam Frame on the same network.
|
||||
- Frametop on the Steam Frame.
|
||||
|
||||
## Set up
|
||||
|
||||
1. Download `frametop-macos-host-1.zip` and open it. You get a folder named `frametop-macos-host`.
|
||||
2. Open Terminal and run the setup:
|
||||
|
||||
```
|
||||
bash ~/Downloads/frametop-macos-host/frametop-host-setup.sh
|
||||
```
|
||||
|
||||
You can also double-click `Setup Frametop host.command`. The kit isn't signed, so macOS blocks it the first time. To allow it, open System Settings > Privacy & Security, click Open Anyway next to the message about it, then open it again.
|
||||
3. The setup does the following:
|
||||
- It installs Sunshine 2026.1010.141104 in /Applications after checking the download's SHA-256. This is a pre-release, and it's the first build where the mouse reaches displays other than the main one.
|
||||
- It creates one Sunshine instance per display, on ports 47989, 48089, 48189 and so on, each started at login.
|
||||
- It asks for the Web UI user name and password that the Steam Frame signs in with. The password is typed in Terminal and is never shown or saved. If an earlier run of this setup or an earlier Sunshine install already has a login, the setup uses that one instead.
|
||||
- It adds Frametop Streams to the menu bar.
|
||||
4. When macOS asks, allow Sunshine both Screen Recording and Accessibility in System Settings > Privacy & Security. Then go back to Terminal and press Enter. Without Accessibility, the pointer moves but clicks, scrolling and typing don't reach the Mac. The setup checks both permissions and offers to ask again if Accessibility is missing.
|
||||
5. On the Steam Frame, open Frametop Remote Displays and choose Add computer. Pick this Mac, sign in with the Web UI login, and choose the displays to show.
|
||||
|
||||
### Monitors that come and go
|
||||
|
||||
Each instance belongs to one monitor, found by its maker, model and serial number, not by its place or the number macOS gives it. When that monitor is unplugged or the lid closes, its instance stops and waits, and starts again by itself once the monitor is back. Frametop Streams and `ft-sunshine status` show it as waiting. A monitor the Mac hasn't had before gets no instance on its own: run the setup again after you connect one (or set its instance up [by hand](#set-up-the-streams-by-hand)), and then add it in Remote Displays on the Steam Frame. Nothing makes virtual monitors on a Mac: Sunshine can't on macOS.
|
||||
|
||||
## Turn the streams on and off
|
||||
|
||||
Click the display icon in the menu bar to open Frametop Streams. It lists the Mac's displays and puts a check next to each one that's on. Click a display to turn it on or off, or choose Turn All On or Turn All Off. The icon is dimmed while every display is off. A display you turn off stays off after a restart, until you turn it on again.
|
||||
|
||||
You can do the same in Terminal with `ft-sunshine`:
|
||||
|
||||
```
|
||||
ft-sunshine status # each display and whether it's on
|
||||
ft-sunshine off # turn every display off
|
||||
ft-sunshine on # turn every display on
|
||||
ft-sunshine on 1 # turn on one display, by the number status shows
|
||||
```
|
||||
|
||||
If you have a `~/.local/bin` folder, the setup puts `ft-sunshine` there. Otherwise, run it as `~/Library/Application\ Support/Frametop/sunshine/bin/ft-sunshine`.
|
||||
|
||||
While a display is off, the Steam Frame keeps trying to reach it, and it reconnects once the display is on again.
|
||||
|
||||
If you quit Frametop Streams, it starts again at your next login. You can also open it from Spotlight.
|
||||
|
||||
## Good to know
|
||||
|
||||
- The Steam Frame can't reach a Mac that's asleep. To keep the Mac awake on power, turn on System Settings > Battery > Options > Prevent automatic sleeping on power adapter when the display is off. The setup tells you if the Mac sleeps when idle.
|
||||
- Sunshine's own menu bar icon belongs to the main display's instance. Quitting Sunshine from that icon stops only the main display. Frametop Streams or `ft-sunshine on` starts it again.
|
||||
- Logs are in `~/Library/Logs/Frametop`.
|
||||
|
||||
## Check and remove
|
||||
|
||||
- `bash frametop-host-setup.sh --check` shows what the setup would change, without changing anything.
|
||||
- `bash frametop-host-setup.sh --undo` removes the instances' LaunchAgents, Frametop Streams and `ft-sunshine`. Sunshine and its settings and pairings stay, and the setup says how to remove those too.
|
||||
|
||||
## Set up the streams by hand
|
||||
|
||||
The setup does all of this for you. These steps are for setting up an instance yourself, for example to see what the setup does or to fix one display. Each display needs its own Sunshine instance, with its own settings file, ports and display. The Steam Frame looks for instances on ports 47989, 48089 and so on, up to 48689, so a Mac can stream up to eight displays.
|
||||
|
||||
1. Install [Sunshine 2026.1010.141104](https://github.com/LizardByte/Sunshine/releases/tag/v2026.1010.141104) (`Sunshine-macOS-arm64.dmg`) in /Applications. Older releases move the mouse only on the main display.
|
||||
2. List the displays, from the kit's folder:
|
||||
|
||||
```
|
||||
osascript -l JavaScript displays.js
|
||||
```
|
||||
|
||||
Each line is one display: its id, its key (maker, model and serial number), its name, its width and height in pixels, and its place. Sunshine wants the id. The id can change when a monitor is plugged in again; the key doesn't.
|
||||
3. Make one certificate for every instance, so the Steam Frame sees one Mac rather than one per display:
|
||||
|
||||
```
|
||||
dir="$HOME/Library/Application Support/Frametop/sunshine"
|
||||
mkdir -p "$dir/credentials" ~/Library/Logs/Frametop
|
||||
(umask 077; openssl req -x509 -newkey rsa:2048 -nodes -days 7300 -subj "/CN=Sunshine Gamestream Host" \
|
||||
-keyout "$dir/credentials/cakey.pem" -out "$dir/credentials/cacert.pem")
|
||||
```
|
||||
|
||||
4. For each display, pick a number N from 0 to 7. Make a folder `instance-N` in that folder, with this `sunshine.conf` in it. Replace N, the port (47989 + 100 x N), the names, the size and the display's id:
|
||||
|
||||
```
|
||||
sunshine_name = My Mac: DELL U2723QE (3840x2160)
|
||||
port = 48089
|
||||
output_name = 2
|
||||
file_state = /Users/you/Library/Application Support/Frametop/sunshine/instance-1/sunshine_state.json
|
||||
credentials_file = /Users/you/Library/Application Support/Frametop/sunshine/credentials.json
|
||||
file_apps = /Users/you/Library/Application Support/Frametop/sunshine/instance-1/apps.json
|
||||
log_path = /Users/you/Library/Logs/Frametop/sunshine-1.log
|
||||
pkey = /Users/you/Library/Application Support/Frametop/sunshine/credentials/cakey.pem
|
||||
cert = /Users/you/Library/Application Support/Frametop/sunshine/credentials/cacert.pem
|
||||
minimum_fps_target = 1
|
||||
system_tray = disabled
|
||||
```
|
||||
|
||||
- Keep the name's form, `COMPUTER: DISPLAY (WIDTHxHEIGHT)`. Remote Displays shows the part after the colon, and reads the display's size from the end.
|
||||
- Every instance shares the one `credentials_file`, so one Web UI login works for all of them.
|
||||
- `minimum_fps_target = 1` lets a still desktop drop to 1 frame a second.
|
||||
- Turn `system_tray` on for one instance only, or you get one Sunshine icon per display.
|
||||
|
||||
Next to it, put an `apps.json` with one app named Desktop, which the Steam Frame opens:
|
||||
|
||||
```
|
||||
{"env": {}, "apps": [{"name": "Desktop", "image-path": "desktop.png"}]}
|
||||
```
|
||||
|
||||
5. Set the Web UI login once. It's saved in the shared `credentials_file`:
|
||||
|
||||
```
|
||||
/Applications/Sunshine.app/Contents/MacOS/Sunshine "$dir/instance-0/sunshine.conf" --creds USER PASSWORD
|
||||
```
|
||||
|
||||
6. Start each instance:
|
||||
|
||||
```
|
||||
/Applications/Sunshine.app/Contents/MacOS/Sunshine "$dir/instance-0/sunshine.conf"
|
||||
```
|
||||
|
||||
Its Web UI is on the next port up from its `port`, for example https://localhost:47990 for instance 0. To start it at login, make a LaunchAgent that runs the same command. The setup's agents run `ft-sunshine-instance N` instead, which looks the display's id up by its key at every start and waits while the display is unplugged. A hand-written `output_name` is fixed, so check it after you plug monitors in again: when it names no display, Sunshine streams the main one.
|
||||
7. Allow Sunshine Screen Recording and Accessibility in System Settings > Privacy & Security, as in the setup's step 4.
|
||||
8. On the Steam Frame, choose Add computer in Frametop Remote Displays, pick the Mac and sign in. It lists every instance that answers.
|
||||
|
||||
To add a monitor later, give it the next free number and repeat steps 4 and 6. Don't mix this with the setup on one Mac: the setup keeps track of its instances by a `display` file it writes in each folder, so it can give a hand-made instance's number to another display and rewrite that instance's settings.
|
||||
|
||||
## Source
|
||||
|
||||
This kit is `host/macos` in [Frametop](https://github.com/Frametop/frametop), under the MIT license. Sunshine is [LizardByte's](https://github.com/LizardByte/Sunshine), under GPL-3.0. The setup downloads it from Sunshine's GitHub release.
|
||||
@@ -1,4 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Double-click to run the Frametop host setup in Terminal.
|
||||
cd "$(dirname "$0")" && ./frametop-host-setup.sh "$@"
|
||||
read -r -p "Press Enter to close: " _
|
||||
@@ -1,38 +0,0 @@
|
||||
// Lists the Mac's displays for Frametop's host setup, one per line, tab-separated:
|
||||
// id key name width height x y builtin main
|
||||
// id is the CGDirectDisplayID, Sunshine's output_name. It can change when a display is plugged in
|
||||
// again, so instances are tied to key (vendor-model-serial, -2 and up for identical monitors) and
|
||||
// ft-sunshine-instance looks the id up at every start. width and height are capture pixels.
|
||||
// Run with: osascript -l JavaScript displays.js
|
||||
ObjC.import('AppKit');
|
||||
ObjC.import('CoreGraphics');
|
||||
|
||||
function run() {
|
||||
const screens = $.NSScreen.screens;
|
||||
const rows = [];
|
||||
for (let i = 0; i < screens.count; i++) {
|
||||
const screen = screens.objectAtIndex(i);
|
||||
const id = ObjC.unwrap(screen.deviceDescription.objectForKey('NSScreenNumber'));
|
||||
const frame = screen.frame;
|
||||
const scale = screen.backingScaleFactor;
|
||||
rows.push({
|
||||
id,
|
||||
base: [$.CGDisplayVendorNumber(id), $.CGDisplayModelNumber(id), $.CGDisplaySerialNumber(id)].join('-'),
|
||||
name: ObjC.unwrap(screen.localizedName).replace(/[\t\n]/g, ' '),
|
||||
width: Math.round(frame.size.width * scale),
|
||||
height: Math.round(frame.size.height * scale),
|
||||
x: frame.origin.x,
|
||||
y: frame.origin.y,
|
||||
builtin: $.CGDisplayIsBuiltin(id) ? 1 : 0,
|
||||
main: $.CGDisplayIsMain(id) ? 1 : 0,
|
||||
});
|
||||
}
|
||||
// Identical monitors without serial numbers share a base; number them left to right.
|
||||
rows.sort((a, b) => a.x - b.x || a.y - b.y);
|
||||
const seen = {};
|
||||
for (const r of rows) {
|
||||
seen[r.base] = (seen[r.base] || 0) + 1;
|
||||
r.key = seen[r.base] === 1 ? r.base : `${r.base}-${seen[r.base]}`;
|
||||
}
|
||||
return rows.map(r => [r.id, r.key, r.name, r.width, r.height, r.x, r.y, r.builtin, r.main].join('\t')).join('\n');
|
||||
}
|
||||
@@ -1,522 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Frametop host setup for macOS: makes this Mac a host for Frametop's remote displays, its
|
||||
# monitors shown as screens on a Steam Frame (docs/remote-displays.md). Run it in Terminal, or
|
||||
# open "Setup Frametop host.command".
|
||||
#
|
||||
# Sunshine streams one display per running copy, so this runs one copy (an instance) per display,
|
||||
# each with its own settings, pairings and ports. Only real displays: Sunshine can't make virtual
|
||||
# ones on macOS.
|
||||
#
|
||||
# 1. Sunshine (github.com/LizardByte/Sunshine), the pinned build below, into /Applications
|
||||
# (checked against its SHA-256 first). It's a pre-release: the first build where the mouse
|
||||
# reaches displays other than the main one (Sunshine #5733).
|
||||
# 2. One instance per display: settings in ~/Library/Application Support/Frametop/sunshine,
|
||||
# instance N on ports 47989 + 100*N (its Web UI on the next port), started at login by the
|
||||
# LaunchAgent frametop.sunshine.N and started again if it stops. An instance is tied to its
|
||||
# monitor, not to a port: plug the monitor in elsewhere and it keeps its instance, and while
|
||||
# it's unplugged its instance waits for it. Run this again after connecting a new monitor.
|
||||
# Frametop Streams, a menu bar icon started at login, turns each one on or off with a
|
||||
# click, or all of them; ft-sunshine off|on|status does the same in Terminal. Off lasts
|
||||
# through logins until on, or until this runs again.
|
||||
# 3. The Web UI login you sign in with once from the Steam Frame, the same for every instance:
|
||||
# keep the one there is, take the one an earlier Sunshine install had, or set one. The
|
||||
# password is typed here and never shown or saved.
|
||||
# 4. macOS's permissions, Screen Recording and Accessibility, which only you can grant. One
|
||||
# grant covers every instance.
|
||||
# 5. Checks: every instance answers, the permissions, the firewall, and whether the Mac sleeps
|
||||
# while idle.
|
||||
#
|
||||
# Options:
|
||||
# --set-login set a new Web UI login even if there is one
|
||||
# --skip-login leave the Web UI login alone
|
||||
# --check only say what would change
|
||||
# --undo remove the instances' LaunchAgents, Frametop Streams and ft-sunshine
|
||||
# (settings, pairings and Sunshine stay)
|
||||
# A log goes to ~/Library/Logs/Frametop/host-setup.log.
|
||||
set -euo pipefail
|
||||
|
||||
SUNSHINE_VERSION=2026.1010.141104
|
||||
DMG_URL="https://github.com/LizardByte/Sunshine/releases/download/v$SUNSHINE_VERSION/Sunshine-macOS-arm64.dmg"
|
||||
DMG_SHA=c40d7d860a77d8e37ce0105b211b453f7c59ff1f1bbbf4970cdd3b66d4f26a58
|
||||
APP=/Applications/Sunshine.app
|
||||
SUNSHINE_ID=dev.lizardbyte.app.Sunshine
|
||||
SUNSHINE="$APP/Contents/MacOS/Sunshine"
|
||||
DATA="$HOME/Library/Application Support/Frametop/sunshine"
|
||||
LOGS="$HOME/Library/Logs/Frametop"
|
||||
AGENTS="$HOME/Library/LaunchAgents"
|
||||
LABEL=frametop.sunshine # instance N's agent is frametop.sunshine.N
|
||||
BASE_PORT=47989
|
||||
GUI="gui/$(id -u)"
|
||||
OLD_STATE="$HOME/.config/sunshine/sunshine_state.json" # a stock Sunshine install's login
|
||||
BREW_LABEL=homebrew.mxcl.sunshine
|
||||
STREAMS_APP="$HOME/Applications/Frametop Streams.app"
|
||||
STREAMS_LABEL=frametop.streams # its LaunchAgent and bundle id
|
||||
LINK="$HOME/.local/bin/ft-sunshine" # only when there's a ~/.local/bin
|
||||
HERE=$(cd "$(dirname "$0")" && pwd)
|
||||
|
||||
check=0 undo=0 login=auto
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
--check) check=1 ;;
|
||||
--undo) undo=1 ;;
|
||||
--set-login) login=set ;;
|
||||
--skip-login) login=skip ;;
|
||||
-h|--help) sed -n '2,/^set -euo/p' "$0" | sed '$d; s/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "Unknown option: $arg (see --help)" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
mkdir -p "$LOGS"
|
||||
LOG="$LOGS/host-setup.log"
|
||||
say() { echo "$*"; printf '%s %s\n' "$(date '+%F %T')" "$*" >> "$LOG"; }
|
||||
fail() { say "Failed: $*"; exit 1; }
|
||||
|
||||
agent_loaded() { launchctl print "$GUI/$1" >/dev/null 2>&1; }
|
||||
our_agents() { ls "$AGENTS" 2>/dev/null | sed -n "s/^\($LABEL\.[0-9][0-9]*\)\.plist$/\1/p"; }
|
||||
# bootout returns before the job is gone, and bootstrap fails until it is.
|
||||
start_agent() {
|
||||
local i
|
||||
# ft-sunshine off disables the agents, and bootstrap fails while they are.
|
||||
launchctl enable "$GUI/$1"
|
||||
for i in 1 2 3 4 5; do
|
||||
launchctl bootstrap "$GUI" "$AGENTS/$1.plist" 2>/dev/null && return 0
|
||||
agent_loaded "$1" && return 0
|
||||
sleep 1
|
||||
done
|
||||
say "Couldn't start the LaunchAgent $1."
|
||||
return 1
|
||||
}
|
||||
stop_agents() {
|
||||
local label
|
||||
for label in $(our_agents); do
|
||||
launchctl bootout "$GUI/$label" 2>/dev/null || true
|
||||
done
|
||||
}
|
||||
|
||||
# Frametop Streams: a menu bar icon that turns each instance on or off through ft-sunshine,
|
||||
# started at login. It's a stay-open AppleScript app, so nothing needs compiling. LSUIElement
|
||||
# keeps it out of the Dock, and changing Info.plist means signing it again (ad hoc, as osacompile
|
||||
# does).
|
||||
install_streams_app() {
|
||||
local where="$DATA/bin/ft-sunshine" info="$STREAMS_APP/Contents/Info.plist"
|
||||
if [ -d "$HOME/.local/bin" ] && { [ ! -e "$LINK" ] || [ -L "$LINK" ]; }; then
|
||||
ln -sf "$DATA/bin/ft-sunshine" "$LINK"
|
||||
where=ft-sunshine
|
||||
fi
|
||||
launchctl bootout "$GUI/$STREAMS_LABEL" 2>/dev/null || true
|
||||
pkill -f "$STREAMS_APP/Contents/MacOS/applet" 2>/dev/null || true
|
||||
mkdir -p "$(dirname "$STREAMS_APP")"
|
||||
rm -rf "$STREAMS_APP"
|
||||
if ! osacompile -s -o "$STREAMS_APP" "$HERE/Frametop Streams.applescript" 2>/dev/null ||
|
||||
! plutil -replace LSUIElement -bool true "$info" ||
|
||||
! plutil -replace CFBundleIdentifier -string "$STREAMS_LABEL" "$info" ||
|
||||
! codesign -f -s - "$STREAMS_APP" 2>/dev/null; then
|
||||
say "Couldn't make $STREAMS_APP. To turn the instances off or on: $where off|on."
|
||||
return 0
|
||||
fi
|
||||
cat > "$AGENTS/$STREAMS_LABEL.plist" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>$STREAMS_LABEL</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>$STREAMS_APP/Contents/MacOS/applet</string>
|
||||
</array>
|
||||
<key>AssociatedBundleIdentifiers</key><string>$STREAMS_LABEL</string>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<!-- Quit from its menu, it stays quit until the next login; after a crash it starts again. -->
|
||||
<key>KeepAlive</key><dict><key>SuccessfulExit</key><false/></dict>
|
||||
<key>ProcessType</key><string>Interactive</string>
|
||||
<key>LimitLoadToSessionType</key><string>Aqua</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
start_agent "$STREAMS_LABEL" || true
|
||||
say "Frametop Streams is in the menu bar: click a display there to turn it on or off. In Terminal: $where off|on|status."
|
||||
}
|
||||
|
||||
# Answers when the instance on this port is up (its HTTP serverinfo, port N).
|
||||
answers() { curl -s -m 2 -o /dev/null "http://127.0.0.1:$1/serverinfo"; }
|
||||
wait_answers() {
|
||||
local i
|
||||
for i in $(seq 1 "${2:-20}"); do
|
||||
answers "$1" && return 0
|
||||
sleep 1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Restart the main display's instance: macOS applies its permissions at the next start, and it
|
||||
# notes them (and asks for missing ones) a moment after it answers.
|
||||
restart_first() {
|
||||
launchctl kickstart -k "$GUI/$LABEL.$first"
|
||||
wait_answers $((BASE_PORT + 100 * first)) || true
|
||||
sleep 2
|
||||
}
|
||||
|
||||
# The instance number for a display's key: the one it had, else the lowest free one.
|
||||
instance_for() {
|
||||
local key=$1 d n=0
|
||||
for d in "$DATA"/instance-*/display; do
|
||||
[ -e "$d" ] || continue
|
||||
if [ "$(cut -f1 "$d")" = "$key" ]; then
|
||||
basename "$(dirname "$d")" | sed 's/^instance-//'
|
||||
return
|
||||
fi
|
||||
done
|
||||
while [ -e "$DATA/instance-$n/display" ] || printf '%s\n' "${taken:-}" | grep -qx "$n"; do
|
||||
n=$((n + 1))
|
||||
done
|
||||
echo "$n"
|
||||
}
|
||||
|
||||
# Sets Frametop's keys in a sunshine.conf and keeps everything else in it (the Web UI's changes).
|
||||
# The name ends in the display's size: Frametop Remote Displays reads it from the serverinfo.
|
||||
write_conf() {
|
||||
local conf=$1 n=$2 name=$3 w=$4 h=$5 keys tmp
|
||||
keys='sunshine_name|port|file_state|credentials_file|file_apps|log_path|pkey|cert|minimum_fps_target|system_tray'
|
||||
tmp=$(mktemp)
|
||||
if [ -e "$conf" ]; then
|
||||
grep -v -E "^[[:space:]]*($keys)[[:space:]]*=|^# (Frametop|Display):" "$conf" > "$tmp" || true
|
||||
fi
|
||||
{
|
||||
echo "# Frametop: instance $n, written by frametop-host-setup.sh. output_name comes from ft-sunshine-instance."
|
||||
echo "# Display: $name"
|
||||
echo "sunshine_name = $(scutil --get ComputerName 2>/dev/null || hostname -s): $name (${w}x$h)"
|
||||
echo "port = $((BASE_PORT + 100 * n))"
|
||||
echo "file_state = $DATA/instance-$n/sunshine_state.json"
|
||||
echo "credentials_file = $DATA/credentials.json"
|
||||
echo "file_apps = $DATA/instance-$n/apps.json"
|
||||
echo "log_path = $LOGS/sunshine-$n.log"
|
||||
echo "pkey = $DATA/credentials/cakey.pem"
|
||||
echo "cert = $DATA/credentials/cacert.pem"
|
||||
echo "minimum_fps_target = 1"
|
||||
# One menu bar icon is enough; every instance's Web UI is listed at the end.
|
||||
echo "system_tray = $([ "$n" = "$first" ] && echo enabled || echo disabled)"
|
||||
cat "$tmp"
|
||||
} > "$conf"
|
||||
rm -f "$tmp"
|
||||
}
|
||||
|
||||
write_agent() {
|
||||
local n=$1 plist="$AGENTS/$LABEL.$1.plist"
|
||||
cat > "$plist" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>$LABEL.$n</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/bin/bash</string>
|
||||
<string>$DATA/bin/ft-sunshine-instance</string>
|
||||
<string>$n</string>
|
||||
</array>
|
||||
<!-- Login Items shows it as Sunshine rather than as the bash that starts it. -->
|
||||
<key>AssociatedBundleIdentifiers</key><string>$SUNSHINE_ID</string>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><dict><key>SuccessfulExit</key><false/></dict>
|
||||
<key>ThrottleInterval</key><integer>10</integer>
|
||||
<key>ProcessType</key><string>Interactive</string>
|
||||
<key>LimitLoadToSessionType</key><string>Aqua</string>
|
||||
<key>StandardOutPath</key><string>/dev/null</string>
|
||||
<key>StandardErrorPath</key><string>$LOGS/sunshine-$n.launch.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
}
|
||||
|
||||
# A JSON string's value from a file, or nothing (plutil reads JSON).
|
||||
json_get() { plutil -extract "$2" raw -o - "$1" 2>/dev/null || true; }
|
||||
|
||||
install_sunshine() {
|
||||
local tmp mnt
|
||||
tmp=$(mktemp -d)
|
||||
say "Downloading Sunshine $SUNSHINE_VERSION..."
|
||||
curl -fsSL -o "$tmp/Sunshine.dmg" "$DMG_URL" || fail "couldn't download $DMG_URL"
|
||||
[ "$(shasum -a 256 "$tmp/Sunshine.dmg" | cut -d' ' -f1)" = "$DMG_SHA" ] ||
|
||||
fail "the download's SHA-256 doesn't match; nothing was installed"
|
||||
mnt="$tmp/mnt"
|
||||
hdiutil attach -nobrowse -readonly -mountpoint "$mnt" "$tmp/Sunshine.dmg" >/dev/null
|
||||
stop_agents
|
||||
pkill -x Sunshine 2>/dev/null || true
|
||||
rm -rf "$APP"
|
||||
ditto "$mnt/Sunshine.app" "$APP"
|
||||
hdiutil detach -quiet "$mnt" || true
|
||||
rm -rf "$tmp"
|
||||
say "Installed Sunshine $SUNSHINE_VERSION in /Applications."
|
||||
}
|
||||
|
||||
undo() {
|
||||
local label
|
||||
for label in $(our_agents); do
|
||||
if [ $check = 1 ]; then say "Would remove the LaunchAgent $label."; continue; fi
|
||||
launchctl bootout "$GUI/$label" 2>/dev/null || true
|
||||
rm -f "$AGENTS/$label.plist"
|
||||
say "Removed the LaunchAgent $label."
|
||||
done
|
||||
if [ -e "$AGENTS/$STREAMS_LABEL.plist" ] || [ -e "$STREAMS_APP" ] || [ -L "$LINK" ]; then
|
||||
if [ $check = 1 ]; then
|
||||
say "Would remove Frametop Streams and ft-sunshine."
|
||||
else
|
||||
launchctl bootout "$GUI/$STREAMS_LABEL" 2>/dev/null || true
|
||||
pkill -f "$STREAMS_APP/Contents/MacOS/applet" 2>/dev/null || true
|
||||
rm -f "$AGENTS/$STREAMS_LABEL.plist"
|
||||
rm -rf "$STREAMS_APP"
|
||||
[ "$(readlink "$LINK" 2>/dev/null)" = "$DATA/bin/ft-sunshine" ] && rm -f "$LINK"
|
||||
say "Removed Frametop Streams and ft-sunshine."
|
||||
fi
|
||||
fi
|
||||
if [ -e "$DATA/.disabled-$BREW_LABEL" ]; then
|
||||
if [ $check = 1 ]; then
|
||||
say "Would turn Homebrew's Sunshine LaunchAgent back on."
|
||||
else
|
||||
launchctl enable "$GUI/$BREW_LABEL"
|
||||
rm -f "$DATA/.disabled-$BREW_LABEL"
|
||||
say "Turned Homebrew's Sunshine LaunchAgent back on (it starts at the next login)."
|
||||
fi
|
||||
fi
|
||||
say "Done. Kept: Sunshine in /Applications and the instances' settings and pairings in $DATA."
|
||||
say "To remove those too: rm -rf \"$DATA\", then move Sunshine to the Trash."
|
||||
}
|
||||
|
||||
main() {
|
||||
say "== Frametop host setup$([ $check = 1 ] && echo ' (check only)')$([ $undo = 1 ] && echo ' (undo)')"
|
||||
[ "$(uname -s)" = Darwin ] || fail "this is the macOS host setup"
|
||||
if [ $undo = 1 ]; then undo; return; fi
|
||||
[ "$(uname -m)" = arm64 ] || fail "only Apple silicon Macs are covered (the pinned build is arm64)"
|
||||
|
||||
# 1. Sunshine
|
||||
local have
|
||||
have=$(defaults read "$APP/Contents/Info.plist" CFBundleShortVersionString 2>/dev/null || true)
|
||||
if [ "$have" = "$SUNSHINE_VERSION" ]; then
|
||||
say "Sunshine $SUNSHINE_VERSION is installed."
|
||||
elif [ $check = 1 ]; then
|
||||
say "Would install Sunshine $SUNSHINE_VERSION$([ -n "$have" ] && echo " over $have")."
|
||||
else
|
||||
[ -w /Applications ] || fail "can't write to /Applications; run this from an admin account"
|
||||
install_sunshine
|
||||
fi
|
||||
|
||||
# An earlier Homebrew Sunshine starts at login on the first instance's ports.
|
||||
if [ -e "$AGENTS/$BREW_LABEL.plist" ] && ! launchctl print-disabled "$GUI" | grep -q "\"$BREW_LABEL\" => disabled"; then
|
||||
if [ $check = 1 ]; then
|
||||
say "Would turn off Homebrew's Sunshine LaunchAgent ($BREW_LABEL), which takes the same ports."
|
||||
else
|
||||
launchctl bootout "$GUI/$BREW_LABEL" 2>/dev/null || true
|
||||
launchctl disable "$GUI/$BREW_LABEL"
|
||||
mkdir -p "$DATA" && touch "$DATA/.disabled-$BREW_LABEL"
|
||||
say "Turned off Homebrew's Sunshine LaunchAgent ($BREW_LABEL), which takes the same ports. --undo turns it back on."
|
||||
fi
|
||||
fi
|
||||
|
||||
# 2. One instance per display, the main display's first.
|
||||
local displays line id key name w h n taken="" first="" lines=""
|
||||
displays=$(osascript -l JavaScript "$HERE/displays.js") || fail "couldn't list the displays"
|
||||
displays=$(printf '%s\n' "$displays" | sort -t $'\t' -k9,9nr -k6,6n -k7,7n)
|
||||
while IFS=$'\t' read -r id key name w h _; do
|
||||
n=$(instance_for "$key")
|
||||
taken="$taken
|
||||
$n"
|
||||
[ -n "$first" ] || first=$n
|
||||
lines="$lines$n $key $name $w $h
|
||||
"
|
||||
done <<< "$displays"
|
||||
# Instances whose display isn't connected now keep their place and wait for it.
|
||||
local d
|
||||
for d in "$DATA"/instance-*/display; do
|
||||
[ -e "$d" ] || continue
|
||||
n=$(basename "$(dirname "$d")" | sed 's/^instance-//')
|
||||
printf '%s' "$lines" | cut -f1 | grep -qx "$n" && continue
|
||||
say "Instance $n: $(cut -f2 "$d") isn't connected; its instance waits for it (port $((BASE_PORT + 100 * n)))."
|
||||
done
|
||||
if [ $check = 1 ]; then
|
||||
while IFS=$'\t' read -r n key name w h; do
|
||||
[ -n "$n" ] || continue
|
||||
say "Would $([ -e "$DATA/instance-$n/display" ] && echo keep || echo add) instance $n: $name ($w x $h), port $((BASE_PORT + 100 * n))."
|
||||
done <<< "$lines"
|
||||
say "Would install Frametop Streams ($STREAMS_APP), a menu bar icon that turns each instance on or off, and ft-sunshine, the same in Terminal."
|
||||
else
|
||||
stop_agents
|
||||
mkdir -p "$DATA/bin" "$DATA/credentials" "$AGENTS"
|
||||
cp "$HERE/ft-sunshine-instance" "$HERE/ft-sunshine" "$HERE/displays.js" "$DATA/bin/"
|
||||
chmod 755 "$DATA/bin/ft-sunshine-instance" "$DATA/bin/ft-sunshine"
|
||||
install_streams_app
|
||||
# One certificate for every instance, so Frametop pins one key for the Mac.
|
||||
if [ ! -s "$DATA/credentials/cacert.pem" ] || [ ! -s "$DATA/credentials/cakey.pem" ]; then
|
||||
(umask 077 && openssl req -x509 -newkey rsa:2048 -nodes -days 7300 -subj "/CN=Sunshine Gamestream Host" \
|
||||
-keyout "$DATA/credentials/cakey.pem" -out "$DATA/credentials/cacert.pem" 2>/dev/null) ||
|
||||
fail "couldn't make the instances' certificate"
|
||||
fi
|
||||
while IFS=$'\t' read -r n key name w h; do
|
||||
[ -n "$n" ] || continue
|
||||
mkdir -p "$DATA/instance-$n"
|
||||
printf '%s\t%s\t%s\t%s\n' "$key" "$name" "$w" "$h" > "$DATA/instance-$n/display"
|
||||
write_conf "$DATA/instance-$n/sunshine.conf" "$n" "$name" "$w" "$h"
|
||||
[ -e "$DATA/instance-$n/apps.json" ] ||
|
||||
printf '{\n "env": {},\n "apps": [{"name": "Desktop", "image-path": "desktop.png"}]\n}\n' > "$DATA/instance-$n/apps.json"
|
||||
write_agent "$n"
|
||||
say "Instance $n: $name ($w x $h), port $((BASE_PORT + 100 * n))."
|
||||
done <<< "$lines"
|
||||
fi
|
||||
|
||||
# 3. The Web UI login, one file for every instance.
|
||||
local user="" pass pass2 old_user
|
||||
user=$(json_get "$DATA/credentials.json" username)
|
||||
old_user=$(json_get "$OLD_STATE" username)
|
||||
if [ $login = skip ]; then
|
||||
say "Web UI login: left as it is$([ -n "$user" ] && echo " ('$user')")."
|
||||
elif [ -n "$user" ] && [ $login = auto ]; then
|
||||
say "Web UI login: keeping '$user'."
|
||||
elif [ -n "$old_user" ] && [ $login = auto ]; then
|
||||
if [ $check = 1 ]; then
|
||||
say "Would use the earlier Sunshine install's Web UI login ('$old_user')."
|
||||
else
|
||||
(umask 077 && osascript -l JavaScript -e 'function run(a) {
|
||||
const o = JSON.parse($.NSString.stringWithContentsOfFileEncodingError(a[0], $.NSUTF8StringEncoding, null).js);
|
||||
const out = JSON.stringify({username: o.username, salt: o.salt, password: o.password}, null, 2);
|
||||
if (!$(out).writeToFileAtomicallyEncodingError(a[1], true, $.NSUTF8StringEncoding, null)) throw new Error("write failed");
|
||||
}' "$OLD_STATE" "$DATA/credentials.json") || fail "couldn't copy the earlier login"
|
||||
user=$old_user
|
||||
say "Web UI login: using the earlier Sunshine install's ('$user', from $OLD_STATE)."
|
||||
fi
|
||||
elif [ $check = 1 ]; then
|
||||
say "Would ask for a Web UI login."
|
||||
else
|
||||
[ -t 0 ] || fail "no Web UI login yet; run this in Terminal to set one"
|
||||
read -r -p "Web UI user name for Frametop to sign in with: " user
|
||||
[ -n "$user" ] || fail "no user name given"
|
||||
while :; do
|
||||
read -r -s -p "Password: " pass; echo
|
||||
read -r -s -p "Password again: " pass2; echo
|
||||
[ -n "$pass" ] && [ "$pass" = "$pass2" ] && break
|
||||
echo "They don't match (or it's empty); try again."
|
||||
done
|
||||
"$SUNSHINE" "$DATA/instance-$first/sunshine.conf" --creds "$user" "$pass" >/dev/null 2>&1 ||
|
||||
fail "Sunshine couldn't save the login"
|
||||
unset pass pass2
|
||||
chmod 600 "$DATA/credentials.json"
|
||||
say "Web UI login set: '$user'."
|
||||
fi
|
||||
|
||||
if [ $check = 1 ]; then
|
||||
say "Would start the instances, the main display's first, and wait for macOS's permissions."
|
||||
checks
|
||||
return
|
||||
fi
|
||||
|
||||
# 4. Start the main display's instance alone: it asks for the permissions, then the rest
|
||||
# start with them (Sunshine remembers its requests per app, not per instance).
|
||||
start_agent "$LABEL.$first"
|
||||
wait_answers $((BASE_PORT + 100 * first)) || say "Instance $first didn't answer within 20 s; see $LOGS/sunshine-$first.log."
|
||||
if [ -t 0 ]; then
|
||||
echo
|
||||
echo "macOS asks whether Sunshine may record the screen and control the computer (Accessibility)."
|
||||
echo "Allow both in System Settings > Privacy & Security, then come back here. If Sunshine is"
|
||||
echo "already allowed, nothing appears."
|
||||
read -r -p "Press Enter when done: " _
|
||||
restart_first
|
||||
# A grant macOS lost track of, or one given to another Sunshine (Homebrew's), leaves Sunshine
|
||||
# without Accessibility and not asking again: clear its records so it asks once more.
|
||||
local yn
|
||||
while ! permitted inputPostEventPermissionState; do
|
||||
read -r -p "Sunshine still doesn't have Accessibility. Ask again? [Y/n] " yn
|
||||
case $yn in [Nn]*) break ;; esac
|
||||
tccutil reset Accessibility "$SUNSHINE_ID" >/dev/null 2>&1 || true
|
||||
tccutil reset PostEvent "$SUNSHINE_ID" >/dev/null 2>&1 || true
|
||||
defaults delete "$SUNSHINE_ID" inputPostEventPermissionState 2>/dev/null || true
|
||||
restart_first
|
||||
echo "Allow Sunshine in macOS's dialog, or in System Settings > Privacy & Security > Accessibility."
|
||||
read -r -p "Press Enter when done: " _
|
||||
restart_first
|
||||
done
|
||||
else
|
||||
say "Not in a terminal: if macOS asks, allow Screen Recording and Accessibility for Sunshine, then run this again."
|
||||
fi
|
||||
while IFS=$'\t' read -r n _; do
|
||||
[ -n "$n" ] && [ "$n" != "$first" ] || continue
|
||||
start_agent "$LABEL.$n" || true
|
||||
done <<< "$lines"
|
||||
# Instances whose display isn't connected start too, and wait for it.
|
||||
for d in "$DATA"/instance-*/display; do
|
||||
[ -e "$d" ] || continue
|
||||
n=$(basename "$(dirname "$d")" | sed 's/^instance-//')
|
||||
[ -e "$AGENTS/$LABEL.$n.plist" ] || write_agent "$n"
|
||||
agent_loaded "$LABEL.$n" || start_agent "$LABEL.$n" || true
|
||||
done
|
||||
|
||||
checks
|
||||
echo
|
||||
say "Done. On the Steam Frame: open Frametop Remote Displays, Add computer, pick $(scutil --get ComputerName 2>/dev/null || hostname -s) ($(addresses)), and sign in as '$user'."
|
||||
}
|
||||
|
||||
addresses() {
|
||||
local ifc ip out=""
|
||||
for ifc in $(ifconfig -l); do
|
||||
case $ifc in lo*|utun*|bridge*|awdl*|llw*|anpi*|ap*|gif*|stf*) continue ;; esac
|
||||
ip=$(ipconfig getifaddr "$ifc" 2>/dev/null) || continue
|
||||
out="$out${out:+, }$ip"
|
||||
done
|
||||
echo "${out:-no network}"
|
||||
}
|
||||
|
||||
# Whether Sunshine had a macOS permission when it last started: it notes 2 then, 1 when it
|
||||
# asked and didn't get it. Without Accessibility the pointer still moves (Sunshine warps it) but
|
||||
# clicks, scrolling and typing are dropped.
|
||||
permitted() { [ "$(defaults read "$SUNSHINE_ID" "$1" 2>/dev/null)" = 2 ]; }
|
||||
permission() {
|
||||
if permitted "$1"; then
|
||||
say "$2: allowed."
|
||||
else
|
||||
say "$2: not allowed for Sunshine, so $3."
|
||||
say " Allow it in System Settings > Privacy & Security > $2, then run this again."
|
||||
fi
|
||||
}
|
||||
|
||||
# 5. Checks
|
||||
checks() {
|
||||
local n key name w h port
|
||||
if [ $check = 0 ]; then
|
||||
for d in "$DATA"/instance-*/display; do
|
||||
[ -e "$d" ] || continue
|
||||
n=$(basename "$(dirname "$d")" | sed 's/^instance-//')
|
||||
IFS=$'\t' read -r key name w h < "$d"
|
||||
port=$((BASE_PORT + 100 * n))
|
||||
if wait_answers "$port" 15; then
|
||||
say "Instance $n answers: $name, Web UI https://localhost:$((port + 1))"
|
||||
elif [ -e "$DATA/instance-$n/.waiting" ]; then
|
||||
say "Instance $n waits for $name to be connected."
|
||||
else
|
||||
say "Instance $n doesn't answer yet: see $LOGS/sunshine-$n.log and $LOGS/sunshine-$n.launch.log."
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
permission screenRecordingPermissionState "Screen Recording" "the screens it sends stay black"
|
||||
permission inputPostEventPermissionState "Accessibility" "the pointer moves but clicks, scrolling and typing don't reach the Mac"
|
||||
|
||||
local fw=/usr/libexec/ApplicationFirewall/socketfilterfw
|
||||
if "$fw" --getglobalstate 2>/dev/null | grep -q enabled; then
|
||||
if "$fw" --getappblocked "$SUNSHINE" 2>/dev/null | grep -q 'is blocked'; then
|
||||
say "Firewall: Sunshine is blocked. Allow it in System Settings > Network > Firewall > Options."
|
||||
elif "$fw" --getallowsigned 2>/dev/null | grep -q 'ENABLED'; then
|
||||
say "Firewall: on; Sunshine is signed, so macOS lets it accept connections."
|
||||
else
|
||||
say "Firewall: on. If macOS asks whether Sunshine may accept incoming connections, allow it."
|
||||
fi
|
||||
else
|
||||
say "Firewall: off."
|
||||
fi
|
||||
|
||||
# The Steam Frame can't reach a sleeping Mac, and Sunshine keeps it awake only while streaming.
|
||||
local sleep_min
|
||||
sleep_min=$(pmset -g custom 2>/dev/null | awk '/^AC Power/ { ac = 1 } ac && $1 == "sleep" { print $2; exit }')
|
||||
if [ -n "$sleep_min" ] && [ "$sleep_min" != 0 ]; then
|
||||
say "Sleep: on power, the Mac sleeps after $sleep_min idle minutes, and Frametop can't connect while it sleeps."
|
||||
say " To keep it reachable: System Settings > Battery > Options > Prevent automatic sleeping on power adapter when the display is off."
|
||||
fi
|
||||
}
|
||||
|
||||
main
|
||||
@@ -1,155 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Turns this Mac's Frametop displays (its Sunshine instances, one per display) on or off. The
|
||||
# setup installs it, and the Frametop Streams menu bar app runs it.
|
||||
# Usage: ft-sunshine on|off|status [--tsv] [N...]
|
||||
#
|
||||
# on start the instances, and start them at login again
|
||||
# off stop them, and leave them stopped at login until "on"
|
||||
# status each instance's display, port and state (the default). With --tsv, a line per
|
||||
# instance: N, state (on, off, waiting, starting, stopped), display, port, tab-separated
|
||||
#
|
||||
# Without N it's every instance. Settings and pairings stay as they are. The Frame's streams of
|
||||
# the Mac keep trying while it's off and come back by themselves after "on".
|
||||
set -u
|
||||
|
||||
DATA="$HOME/Library/Application Support/Frametop/sunshine"
|
||||
AGENTS="$HOME/Library/LaunchAgents"
|
||||
LABEL=frametop.sunshine # instance N's agent is frametop.sunshine.N
|
||||
BASE_PORT=47989
|
||||
GUI="gui/$(id -u)"
|
||||
|
||||
usage() { sed -n '2,/^set -u/p' "$0" | sed '$d; s/^# \{0,1\}//'; }
|
||||
|
||||
cmd=${1:-status}
|
||||
[ $# -gt 0 ] && shift
|
||||
case $cmd in
|
||||
on|off|status) ;;
|
||||
-h|--help|help) usage; exit 0 ;;
|
||||
*) usage >&2; exit 2 ;;
|
||||
esac
|
||||
tsv=0
|
||||
nums=()
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
--tsv) tsv=1 ;;
|
||||
*) nums+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
set -- ${nums[@]+"${nums[@]}"}
|
||||
|
||||
loaded() { launchctl print "$GUI/$LABEL.$1" >/dev/null 2>&1; }
|
||||
answers() { curl -s -m 2 -o /dev/null "http://127.0.0.1:$((BASE_PORT + 100 * $1))/serverinfo"; }
|
||||
|
||||
state() {
|
||||
if ! loaded "$1"; then
|
||||
echo off
|
||||
elif answers "$1"; then
|
||||
echo on
|
||||
elif [ -e "$DATA/instance-$1/.waiting" ]; then
|
||||
echo waiting
|
||||
elif launchctl print "$GUI/$LABEL.$1" | grep -q 'state = running'; then
|
||||
echo starting
|
||||
else
|
||||
# Quit from Sunshine's menu bar icon: launchd doesn't start it again after a clean exit.
|
||||
echo stopped
|
||||
fi
|
||||
}
|
||||
|
||||
status() {
|
||||
local n st name port
|
||||
for n in "${instances[@]}"; do
|
||||
st=$(state "$n")
|
||||
name=$(cut -f2 "$DATA/instance-$n/display")
|
||||
port=$((BASE_PORT + 100 * n))
|
||||
if [ $tsv = 1 ]; then
|
||||
printf '%s\t%s\t%s\t%s\n' "$n" "$st" "$name" "$port"
|
||||
continue
|
||||
fi
|
||||
case $st in
|
||||
waiting) st="on, waiting for the display to be connected" ;;
|
||||
stopped) st="stopped (quit from Sunshine's menu bar icon)" ;;
|
||||
esac
|
||||
echo "Instance $n, $name (port $port): $st"
|
||||
done
|
||||
}
|
||||
|
||||
start() {
|
||||
local i
|
||||
# off disables the agent so it stays off at login; bootstrap fails while it's disabled.
|
||||
launchctl enable "$GUI/$LABEL.$1"
|
||||
if loaded "$1"; then
|
||||
# Loaded but stopped (quit from the menu bar): start it. One that's running is left alone.
|
||||
launchctl kickstart "$GUI/$LABEL.$1" >/dev/null 2>&1
|
||||
return 0
|
||||
fi
|
||||
# bootout returns before the job is gone, and bootstrap fails until it is.
|
||||
for i in 1 2 3 4 5; do
|
||||
launchctl bootstrap "$GUI" "$AGENTS/$LABEL.$1.plist" 2>/dev/null && return 0
|
||||
loaded "$1" && return 0
|
||||
sleep 1
|
||||
done
|
||||
echo "Couldn't start instance $1's LaunchAgent $LABEL.$1." >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
stop() {
|
||||
launchctl disable "$GUI/$LABEL.$1"
|
||||
launchctl bootout "$GUI/$LABEL.$1" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Every instance the setup made, or the ones named.
|
||||
instances=()
|
||||
if [ $# -gt 0 ]; then
|
||||
for n in "$@"; do
|
||||
if ! [[ $n =~ ^[0-9]+$ ]] || [ ! -e "$DATA/instance-$n/display" ] || [ ! -e "$AGENTS/$LABEL.$n.plist" ]; then
|
||||
echo "There's no instance $n; ft-sunshine status lists them." >&2
|
||||
exit 2
|
||||
fi
|
||||
instances+=("$n")
|
||||
done
|
||||
else
|
||||
for d in "$DATA"/instance-*/display; do
|
||||
[ -e "$d" ] || continue
|
||||
n=$(basename "$(dirname "$d")" | sed 's/^instance-//')
|
||||
[ -e "$AGENTS/$LABEL.$n.plist" ] && instances+=("$n")
|
||||
done
|
||||
if [ ${#instances[@]} -eq 0 ]; then
|
||||
echo "No Frametop Sunshine instances here; run frametop-host-setup.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
IFS=$'\n' read -r -d '' -a instances < <(printf '%s\n' "${instances[@]}" | sort -n) || true
|
||||
fi
|
||||
|
||||
failed=0
|
||||
case $cmd in
|
||||
on)
|
||||
for n in "${instances[@]}"; do
|
||||
start "$n" || failed=1
|
||||
done
|
||||
# Wait for each to answer (or to wait for its display), so the status means something.
|
||||
for i in $(seq 1 20); do
|
||||
pending=0
|
||||
for n in "${instances[@]}"; do
|
||||
loaded "$n" || continue
|
||||
answers "$n" || [ -e "$DATA/instance-$n/.waiting" ] || pending=1
|
||||
done
|
||||
[ $pending = 0 ] && break
|
||||
sleep 1
|
||||
done
|
||||
;;
|
||||
off)
|
||||
for n in "${instances[@]}"; do
|
||||
stop "$n"
|
||||
done
|
||||
for i in $(seq 1 10); do
|
||||
pending=0
|
||||
for n in "${instances[@]}"; do
|
||||
loaded "$n" && pending=1
|
||||
done
|
||||
[ $pending = 0 ] && break
|
||||
sleep 0.5
|
||||
done
|
||||
;;
|
||||
esac
|
||||
status
|
||||
exit $failed
|
||||
@@ -1,36 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Starts Frametop's Sunshine instance N on its display; its LaunchAgent runs this at login.
|
||||
# Usage: ft-sunshine-instance N
|
||||
#
|
||||
# A display's id can change when it's plugged in again, so this looks it up by the key the setup
|
||||
# saved (instance-N/display). Sunshine streams the main display when output_name names no display,
|
||||
# so while this one is missing (unplugged, lid closed) Sunshine isn't started; launchd runs this
|
||||
# again after it exits.
|
||||
set -u
|
||||
|
||||
n=${1:?usage: ft-sunshine-instance N}
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
data="$here/.."
|
||||
inst="$data/instance-$n"
|
||||
sunshine=/Applications/Sunshine.app/Contents/MacOS/Sunshine
|
||||
|
||||
key=$(cut -f1 "$inst/display" 2>/dev/null)
|
||||
if [ -z "$key" ]; then
|
||||
echo "$(date '+%F %T') instance $n: no $inst/display; run frametop-host-setup.sh again" >&2
|
||||
sleep 60
|
||||
exit 78
|
||||
fi
|
||||
|
||||
id=$(osascript -l JavaScript "$here/displays.js" | awk -F'\t' -v k="$key" '$2 == k { print $1 }')
|
||||
if [ -z "$id" ]; then
|
||||
# Logged once, not on every retry.
|
||||
if [ ! -e "$inst/.waiting" ]; then
|
||||
echo "$(date '+%F %T') instance $n: display $key isn't connected; waiting for it" >&2
|
||||
touch "$inst/.waiting"
|
||||
fi
|
||||
sleep 30
|
||||
exit 75
|
||||
fi
|
||||
rm -f "$inst/.waiting"
|
||||
|
||||
exec "$sunshine" "$inst/sunshine.conf" "output_name=$id"
|
||||
@@ -1,62 +0,0 @@
|
||||
# Frametop host for Windows
|
||||
|
||||
This kit makes a Windows PC a host for Frametop's remote displays: the PC's monitors appear as screens on a Steam Frame, and it can also make virtual monitors at any size. The PC streams with Vibepollo, a version of Sunshine, and Frametop's build of Vibepollo's `sunshine.exe`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Windows 10 or 11, 64-bit.
|
||||
- A graphics card with a hardware video encoder. It's tested on an NVIDIA card.
|
||||
- An admin account. The setup asks Windows for admin, so you confirm one prompt.
|
||||
- The PC and the Steam Frame on the same network, or a Steam Link dongle plugged into the PC (see [Stream over a Steam Link dongle](#stream-over-a-steam-link-dongle)).
|
||||
- Frametop on the Steam Frame.
|
||||
|
||||
If Sunshine runs on this PC, uninstall it first: Vibepollo uses the same ports. If Apollo is installed, Vibepollo's installer replaces it and keeps its settings and pairings.
|
||||
|
||||
## Set up
|
||||
|
||||
1. Download `frametop-windows-host-1.zip` from the [`windows-host-1`](https://github.com/Frametop/frametop/releases/tag/windows-host-1) release and extract it. You get a folder named `frametop-windows-host`.
|
||||
2. Double-click `Setup Frametop host.cmd` in that folder. Windows may warn that the file came from the internet: choose Run. Then allow the admin prompt.
|
||||
3. The setup does the following:
|
||||
- If Vibepollo isn't installed, it downloads Vibepollo 2.0.0, checks the download's SHA-256 and runs Vibepollo's installer. Follow the installer's steps, then go back to the setup window.
|
||||
- It replaces Vibepollo's `sunshine.exe` with Frametop's build, after checking its SHA-256. The original is kept next to it as `sunshine.exe.2.0.0-original`.
|
||||
- It changes three Vibepollo settings: remote displays carry the PC's sound, and a virtual display goes away when Frametop disconnects it but stays through a dropped stream. Your other settings stay as they are.
|
||||
- It asks for the Web UI user name and password that the Steam Frame signs in with. If Vibepollo already has a login, you can keep it. The password is typed in the setup window and is never shown or saved.
|
||||
- It lets Vibepollo through the firewall on every network type, says whether a Steam Link dongle is plugged in, and checks that the Web UI answers.
|
||||
4. When it says Done, it prints this PC's name and addresses. On the Steam Frame, open Frametop Remote Displays and choose Add computer. Pick this PC, sign in with the Web UI login, and choose the displays to show: its monitors, or a virtual display at the size you pick.
|
||||
|
||||
When a newer kit comes out, run its setup the same way. It replaces an older Frametop build.
|
||||
|
||||
Monitors you plug in later need nothing on the PC: Add displays asks Vibepollo for the PC's monitors each time it opens. A virtual monitor is made when its stream starts and goes away when you disconnect or remove it in Remote Displays. It stays through a dropped stream, so its windows keep their places. At most four virtual monitors exist at once.
|
||||
|
||||
## What Frametop's build changes
|
||||
|
||||
Frametop's build is Vibepollo 2.0.0 with a few changes, from [Frametop/frametop-vibepollo](https://github.com/Frametop/frametop-vibepollo):
|
||||
|
||||
- It can stream any of the PC's monitors, not only the main one, and several at once.
|
||||
- While a monitor streams, it switches to the stream's resolution (or the closest mode it has with the same shape), and back when the stream ends. Set the stream's resolution in Remote Displays on the Frame.
|
||||
- An HDR monitor's HDR is off while it streams, and back on after. An HDR desktop in the stream looked washed out.
|
||||
- A virtual display that goes away leaves your monitor layout as it was.
|
||||
- The Web UI doesn't stall after a client disconnects.
|
||||
|
||||
## Stream over a Steam Link dongle
|
||||
|
||||
A Steam Link dongle in the PC connects straight to the Frame's hotspot, so the streams skip your router. On a busy home network that loses far fewer frames. Set the dongle up in Steam as you would for streaming VR games from the PC. In Remote Displays, the computer's Connection setting picks auto (the dongle when it's up), network only, or dongle only, and Find looks for the PC on the hotspot.
|
||||
|
||||
## Good to know
|
||||
|
||||
- The Frame signs in once. From Vibepollo it keeps a token that can only pair its displays, set their permissions and list the monitors, not the password. You can revoke the token in the Web UI under API Tokens.
|
||||
- The Web UI is at https://localhost:47990 on the PC.
|
||||
- The Steam Frame can't reach a PC that's asleep. Set the PC not to sleep while you use it.
|
||||
- The setup's log is `%ProgramData%\Frametop\host-setup.log`, and its backups of Vibepollo's settings are in `%ProgramData%\Frametop\backup-*`.
|
||||
|
||||
## Check and remove
|
||||
|
||||
Run these in a Command Prompt in the kit's folder:
|
||||
|
||||
- `"Setup Frametop host.cmd" -Check` shows what the setup would change, without changing anything.
|
||||
- `"Setup Frametop host.cmd" -Undo` puts back the original `sunshine.exe` and the settings from before the setup. The Web UI login stays, and so does Vibepollo. Uninstall Vibepollo from Windows Settings > Apps if you want it gone.
|
||||
- `"Setup Frametop host.cmd" -SkipLogin` sets up everything except the Web UI login.
|
||||
|
||||
## Source
|
||||
|
||||
This kit is `host/windows` in [Frametop](https://github.com/Frametop/frametop), under the MIT license. Vibepollo is [Nonary's](https://github.com/Nonary/Vibepollo), and Frametop's build is [Frametop/frametop-vibepollo](https://github.com/Frametop/frametop-vibepollo), both under GPL-3.0. The setup downloads Vibepollo from its GitHub release and Frametop's build from the fork's release.
|
||||
@@ -34,14 +34,13 @@ $VibepolloVersion = "2.0.0"
|
||||
$SetupUrl = "https://github.com/Nonary/Vibepollo/releases/download/2.0.0/VibepolloSetup-v2.0.0.exe"
|
||||
$SetupSha = "7B3500EC0C774644CE5A435A48F61C046C48494D0F18B67AFA0B3561931794B7"
|
||||
$OriginalSha = "2CC018FD92DDB4D3748D91D8DA25316909ED45DB3710D1FF278BD73D51EB00C1" # its sunshine.exe
|
||||
# Frametop's build: Frametop/frametop-vibepollo release frametop-2.0.0-2 (its CI, from the tag).
|
||||
$BuildSha = "2BF7D1DDBE4350947D58A459A47FF5AE85ED0EFCA957ED7D91F9B582FC37599D"
|
||||
$BuildUrl = "https://github.com/Frametop/frametop-vibepollo/releases/download/frametop-2.0.0-2/sunshine.exe"
|
||||
# Frametop's build: Frametop/frametop-vibepollo release frametop-2.0.0-1 (its CI, from the tag).
|
||||
$BuildSha = "57EECCF9CA6ECEC4F0C5AEBBB27AAFE711A700C9E0986601344A52682856E94E"
|
||||
$BuildUrl = "https://github.com/Frametop/frametop-vibepollo/releases/download/frametop-2.0.0-1/sunshine.exe"
|
||||
# Earlier Frametop builds, replaced by this one like the original is.
|
||||
$OlderBuildShas = @(
|
||||
"B5B7D2E7353454AEA6D895D0B68DE235E4CC581684C9DD44F8EFAF2E872F517F" # 2f032252, built by hand without WebRTC
|
||||
"6AF4F34503C7F6F84D1F6967D9FCEFAF2BD8145B34044C992B5E26A614E37A91" # a84b6cfc, CI, before the HDR-off guard
|
||||
"57EECCF9CA6ECEC4F0C5AEBBB27AAFE711A700C9E0986601344A52682856E94E" # frametop-2.0.0-1, before the resolution switch
|
||||
)
|
||||
$Settings = [ordered]@{
|
||||
"remote_monitor_mute_audio" = "disabled"
|
||||
|
||||
@@ -18,19 +18,9 @@ to the input relay over its control socket (@frametop_relay):
|
||||
- Game optimization: pausing Frametop while a VR game runs, so it leaves the CPU and GPU to the game
|
||||
(input/game_pause.py, in the relay: "pause on|off|?"): pause now, pause by itself during VR
|
||||
games, the controller gesture that pauses and resumes, what happens to the desktop, a sound.
|
||||
- Voice: Frametop Voice's dictation (voice/ft-voiced): on or off, its keys (key combinations
|
||||
with the dictation actions), controller buttons, typing, sounds, and the wake word. Settings
|
||||
go to ~/.config/frametop-voice.json (voice/voice_settings.py); ft-voiced reads them again
|
||||
within 2 s, and answers "status" on its control socket.
|
||||
- Gaze: the pointer's gaze mode (@ft_pointer_helper "gaze") and the gaze service
|
||||
(gaze/ft-gazed, @ft_gazed: status, forget, reload), with its eye tracker (SteamVR's or
|
||||
our own) and eye bias (GAZE_TRACKER, GAZE_EYE in frametop.conf).
|
||||
- Hands: when hand tracking runs (HANDS_MODE: with the Frametop desktop, always with SteamVR,
|
||||
or off; hands/handsmode.py on the host starts and stops its services), the hand cutouts'
|
||||
margin (HANDS_MATTE_MARGIN, sent live to ft-screens: "cutouts margin DEG"), and three hand
|
||||
inputs, each on its own (handsmode.METHODS): clicks at the gaze (POINTER_HANDS,
|
||||
POINTER_GAZE_PINCH), the hand ray with gaze off (POINTER_HAND_RAY), and direct touch on
|
||||
Frametop's panels (HANDS_NEAR, sent live to ft-screens: "handinput on|off").
|
||||
- Bluetooth: paired devices, and re-applying the Bluetooth LE fixes after pairing.
|
||||
- A warning on every page when SteamVR won't load the ft_pointer driver (blocked after a
|
||||
crash, disabled, or SteamVR in safe mode) or hasn't loaded it (@ft_pointer doesn't answer
|
||||
@@ -48,7 +38,6 @@ import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
|
||||
from PySide6.QtCore import Property, QObject, QSocketNotifier, QTimer, QUrl, Signal, Slot
|
||||
@@ -67,112 +56,6 @@ VRSETTINGS_PATHS = [os.path.expanduser("~/.config/openvr/config/steamvr.vrsettin
|
||||
os.path.expanduser("~/.steam/steam/config/steamvr.vrsettings")]
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
GAZE_PROBE = os.path.join(REPO, "gaze", "probe", "ft-gazeprobe")
|
||||
# Hand tracking's mode and the cutouts' margin (standard library only, so it imports here too).
|
||||
sys.path.insert(0, os.path.join(REPO, "hands"))
|
||||
import handsmode # noqa: E402
|
||||
HANDSMODE = os.path.join(REPO, "hands", "handsmode.py")
|
||||
SCREENS = "\0ft_screens"
|
||||
HANDS_MODES = {"frametop": "With Frametop: while the Frametop desktop runs",
|
||||
"always": "Always: whenever SteamVR runs, in games and the SteamVR dashboard too",
|
||||
"off": "Off"}
|
||||
# The Hands page's input switches, each on its own (handsmode.METHODS): POINTER_HANDS with
|
||||
# POINTER_GAZE_PINCH or POINTER_HAND_RAY for the pointer, HANDS_NEAR for ft-screens.
|
||||
HAND_METHODS = {"gaze_clicks": "Hand clicks in gaze mode", "ray": "Hand ray with gaze off",
|
||||
"touch": "Direct touch"}
|
||||
# Every hand interaction on Frametop's panels, each switched on and off on its own (Hands page): the
|
||||
# frametop.conf key ft-screens reads at start, sent live too ("handsetting KEY 0|1"), its default, the
|
||||
# value written for on, and the page's group, label and tooltip.
|
||||
HAND_INTERACTIONS = [
|
||||
# (key, default on, value written for on, group, label, tooltip)
|
||||
("HANDS_TOUCH_DOT", True, "1", "Tapping", "Fingertip dot",
|
||||
"A soft white dot on the screen where a tap would land, frozen as the fingertip comes close."),
|
||||
("HANDS_TAP_LOCK", True, "1", "Tapping", "Lock the aim on the way in",
|
||||
"The tap lands where the dot froze 1.5 cm out, not where the fingertip's jitter takes it."),
|
||||
("HANDS_MOTION_PRESS", True, "1", "Tapping", "Taps that stop just short",
|
||||
"A quick tap that stops or bounces back just before the surface still clicks."),
|
||||
("HANDS_LOCK_PUSH", True, "0.01", "Tapping", "Push through to tap",
|
||||
"Pushing the finger 1 cm through the screen taps at the frozen dot, however slowly."),
|
||||
("HANDS_PINCH_CLICK", True, "1", "Tapping", "Pinch on a screen clicks",
|
||||
"Up close, a pinch on a screen's content clicks there, and drags while held."),
|
||||
("HANDS_TWO_FINGER_SCROLL", True, "1", "Tapping", "Two-finger scroll",
|
||||
"Index and middle finger together on a screen: swipe to scroll, lift without moving to click."),
|
||||
("HANDS_PINCH_BUTTONS", True, "1", "Tapping", "Pinch the bar's buttons",
|
||||
"A pinch at a round button under a screen (curve, reset, dock, close) presses it."),
|
||||
("HANDS_GRAB_ROTATE", True, "1", "Moving screens", "Turn with the wrist",
|
||||
"A screen held by its bar or edge turns as the wrist turns."),
|
||||
("HANDS_CARD_PINCH", True, "1", "Moving screens", "Card grab",
|
||||
"Index and middle finger (or all fingertips) pinching an edge hold the screen like a card."),
|
||||
("HANDS_FIST_GRAB", True, "1", "Moving screens", "Fists",
|
||||
"A fist at a screen holds it; two fists resize it."),
|
||||
("HANDS_FIST_PAIR", True, "1", "Moving screens", "Two fists near a screen resize it",
|
||||
"Two closed hands anywhere near one screen start a resize, without a grip at the screen first."),
|
||||
("HANDS_FIST_RESIZE_ONLY", True, "1", "Moving screens", "Fists only resize",
|
||||
"Fists don't move or turn screens (the tracking's too rough for placing): one holds still, two resize."),
|
||||
("HANDS_STRETCH_TURN", True, "1", "Moving screens", "Two hands turn a screen",
|
||||
"Holding a screen with both hands turns it on every axis with them."),
|
||||
("HANDS_RELEASE_FREEZE", True, "1", "Moving screens", "Let go without a nudge",
|
||||
"A held screen stops following as the fingers start to open, so letting go doesn't move it."),
|
||||
("HANDS_GRAB_THAT", True, "1", "Gestures", "Grab that",
|
||||
"Reach toward a far screen with an open hand, then pull back into a fist: it comes to arm's reach."),
|
||||
("HANDS_PUSH", True, "1", "Gestures", "Push away",
|
||||
"An open palm pushed into a near screen sends it 3 m out."),
|
||||
("HANDS_FLICK", True, "1", "Gestures", "Flick away",
|
||||
"A quick outward flick of an open hand at a screen sends it 3 m out."),
|
||||
("HANDS_THROW", True, "1", "Gestures", "Throw",
|
||||
"Letting go of a held screen while moving it fast throws it 3 m out along the throw."),
|
||||
("HANDS_SWEEP", True, "1", "Gestures", "Sweep for the lazy susan",
|
||||
"An open hand on its edge, palm leading, swept 25 cm sideways, quickly: the next screen on that side "
|
||||
"(Meta+Alt+Tab too). The return stroke does nothing: the other hand palms the other way."),
|
||||
("SCREEN_FRONT_AWAY", False, "1", "Gestures", "A brought screen goes home when you look away",
|
||||
"Off: it stays in front of you until you move it yourself or the next stroke sends it home."),
|
||||
("HANDS_SWEEP_BACK", False, "1", "Gestures", "Lazy susan on the return stroke too",
|
||||
"A stroke with the back of the hand leading counts too (one hand going back and forth)."),
|
||||
("SCREEN_SPIN_BRING", True, "1", "Gestures", "Lazy susan brings screens to you",
|
||||
"The next screen comes 1 m in front of you, big enough to read, and takes the typing; the next stroke sends "
|
||||
"it home. Grab that pulls it to arm's reach; a push steps it back. Off: the screens turn around you instead."),
|
||||
("HANDS_PALM_MENU", True, "1", "Gestures", "Palm menu",
|
||||
"Palm toward your face, away from screens, and a plain pinch: the Steam menu."),
|
||||
("HANDS_TARGET_GLOW", True, "1", "Feedback", "Grab that's glow",
|
||||
"A faint white glow round the screen \"grab that\" would take."),
|
||||
("HANDS_EDGE_GLOW", True, "1", "Feedback", "Resize and edge glows",
|
||||
"A glow at the corner or edge a pinch there would resize or move."),
|
||||
("POINTER_GAZE_TWO_HAND", True, "1", "Gaze + hands", "Two-hand drag (like Meta+J, then Meta+K)",
|
||||
"With gaze on: hold a pinch on one hand (nudge it into place), then pinch with the other: the left button "
|
||||
"goes down where the dot is; the drag lasts while either pinch is held, and the hand still pinching drags."),
|
||||
("POINTER_GAZE_RIGHT_DRAG", False, "1", "Gaze + hands", "Right-drag: tap one hand, then pinch the other",
|
||||
"With gaze on: an index tap on one hand, then an index pinch on the other within a second, holds the right "
|
||||
"button where you look; that hand drags until you let go."),
|
||||
("POINTER_TYPING_HIDE", True, "1.5", "Gaze + hands", "Hide the gaze pointer while typing",
|
||||
"The gaze dot and its laser hide for 1.5 s after each key."),
|
||||
("POINTER_PINCH_MAX_SPEED", True, "1.0", "Gaze + hands", "No clicks from moving hands",
|
||||
"A pinch that begins while the hand moves fast (a lazy susan stroke, a wave) doesn't click."),
|
||||
("POINTER_GAZE_SCROLL", True, "1", "Gaze + hands", "Three-finger pinch scrolls",
|
||||
"With gaze on: thumb on both the index and middle fingertips, then move the hand up, down or across: it "
|
||||
"scrolls where you looked."),
|
||||
("POINTER_GAZE_CHORD", False, "1", "Gaze + hands", "Hold both buttons (chord)",
|
||||
"With gaze on: while one hand's pinch is held, the other hand's pinch holds the right button too "
|
||||
"(replaced by the right-drag)."),
|
||||
("POINTER_RAY_BEAM", False, "1", "Laser", "Show the hand laser's beam",
|
||||
"Off: a hand's laser shows only its dot. On: the whole beam from the hand."),
|
||||
("POINTER_RAY_READY", True, "6", "Laser", "Laser only when about to pinch",
|
||||
"A hand's laser shows once its thumb and index are within 6 cm, and goes when they open past 10 cm. "
|
||||
"Off: whenever the hand is raised and pointing."),
|
||||
("POINTER_RAY_FIST_CURL", True, "1.1", "Laser", "No laser or clicks in a fist",
|
||||
"Closing the hand drops its laser, and a pinch made while the hand closes into a fist doesn't click."),
|
||||
("POINTER_PINCH_NO_RAY", False, "1", "Laser", "Pinch with no laser clicks at the pointer",
|
||||
"With no laser up, a pinch or a fist presses where the pointer is and drags it (an air mouse). "
|
||||
"Off: a pinch with no laser does nothing."),
|
||||
("HANDS_PUSH_GLOW", False, "1", "Feedback", "Push hint glow",
|
||||
"A dim glow on a near screen an open palm faces (a push or flick would send it away)."),
|
||||
("HANDS_TRACK_GATE", True, "1", "Feedback", "Steady through tracking glitches",
|
||||
"A held screen ignores impossible jumps of the hand and holds still when the hand leaves the cameras' view."),
|
||||
]
|
||||
sys.path.insert(0, os.path.join(REPO, "voice"))
|
||||
import voice_commands # noqa: E402 Frametop Voice's commands and settings (standard library)
|
||||
import voice_settings # noqa: E402
|
||||
|
||||
VOICE_UNIT = "frametop-voice.service"
|
||||
VOICE_SOCKET = f"/run/user/{os.getuid()}/frametop-voice/control.sock" # ft-voiced's, owner-only
|
||||
BTN_MISC = 0x100
|
||||
# Header names that mark the start of a range, not a real key (BTN_MOUSE == BTN_LEFT).
|
||||
RANGE_ALIASES = {"BTN_MISC", "BTN_MOUSE", "BTN_JOYSTICK", "BTN_GAMEPAD", "BTN_DIGI", "BTN_WHEEL",
|
||||
@@ -194,11 +77,8 @@ ACTION_LABELS = {
|
||||
"screens_toggle": "Hide/show desktop screens", "keyboard_toggle": "Open/close keyboard",
|
||||
"float_toggle": "Float window in VR / put it back", "dock_all": "Put all floating windows back",
|
||||
"pause_toggle": "Pause/resume Frametop (for VR games)",
|
||||
"dictate": "Dictation: start or stop",
|
||||
"dictate_hold": "Dictation: while held",
|
||||
"spin_next": "Spin the panels: next one on the right to the front",
|
||||
"spin_prev": "Spin the panels: next one on the left to the front",
|
||||
"spotwatch": "Spotwatch: search apps and remote displays",
|
||||
"key": "Pass through as key",
|
||||
"none": "Do nothing",
|
||||
}
|
||||
@@ -230,10 +110,7 @@ PAUSE_REASONS = {"game": "a VR game started", "gesture": "controller gesture", "
|
||||
"settings": "from here"}
|
||||
# Gaze mode is a mouse and keyboard feature (docs/gaze-controllers.md; the relay's GAZE_ACTIONS).
|
||||
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
|
||||
# Dictation's controller buttons are on the Voice page: read from SteamVR's web socket, they stay
|
||||
# the games' too (a Controllers mapping would take them).
|
||||
VOICE_ACTIONS = voice_settings.DICTATION_ACTIONS
|
||||
CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none") + GAZE_ACTIONS + VOICE_ACTIONS]
|
||||
CONTROLLER_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none") + GAZE_ACTIONS]
|
||||
# Key combinations take any action but key and none; the keyboard clicks only work there.
|
||||
SHORTCUT_ACTIONS = [a for a in ACTION_LABELS if a not in ("key", "none")]
|
||||
KEYBOARD_ONLY = ("gaze_left", "gaze_right")
|
||||
@@ -280,9 +157,6 @@ GAZE_SETTINGS = [
|
||||
("POINTER_GAZE_NUDGE_MAX", "Largest correction to learn", 55, 1, 110, 1, "°"),
|
||||
("POINTER_GAZE_HOLD", "Hold still to drag", 0.5, 0.1, 2.0, 0.05, "s"),
|
||||
("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"),
|
||||
# with hand tracking (POINTER_HANDS): how far a held pinch moves before it nudges, and how fast
|
||||
("POINTER_GAZE_PINCH_DEADZONE", "Pinch nudge starts after", 4.0, 1.0, 10.0, 0.5, "°"),
|
||||
("POINTER_GAZE_PINCH_GAIN", "Pinch nudge speed", 0.3, 0.1, 1.0, 0.05, "×"),
|
||||
]
|
||||
# In gaze mode, what the mouse's left button does (POINTER_GAZE_MOUSE).
|
||||
GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, release to click",
|
||||
@@ -290,27 +164,18 @@ GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, releas
|
||||
# Key combinations ("key_bindings" in the rules): modifiers, either side folded into the left code.
|
||||
MODIFIER_CODES = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
|
||||
MODIFIER_NAMES = {29: "Ctrl", 42: "Shift", 56: "Alt", 125: "Meta"}
|
||||
# A tap of a right-hand modifier is bound by its own code: that key alone (the relay's modifier_tap).
|
||||
RIGHT_MODIFIER_NAMES = {97: "Right Ctrl", 54: "Right Shift", 100: "Right Alt", 126: "Right Meta"}
|
||||
# What a rules file without "key_bindings" gets (the relay's DEFAULT_KEY_BINDINGS): a Meta tap
|
||||
# opens Steam's menu, Meta+J and Meta+K click at the gaze, Meta+Shift+F floats a window,
|
||||
# Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels, Meta+Space opens spotwatch.
|
||||
# Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels.
|
||||
DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
|
||||
"42+125+33": "float_toggle", "56+125+15": "spin_next", "42+56+125+15": "spin_prev",
|
||||
"125+57": "spotwatch"}
|
||||
|
||||
|
||||
def voice_installed():
|
||||
return os.path.exists(os.path.expanduser(f"~/.config/systemd/user/{VOICE_UNIT}"))
|
||||
"42+125+33": "float_toggle", "56+125+15": "spin_next", "42+56+125+15": "spin_prev"}
|
||||
|
||||
|
||||
def key_bindings(rules):
|
||||
"""The rules' key combinations, or the defaults if it has none of its own (an empty list
|
||||
counts as its own): with Frametop Voice installed, a Right Ctrl tap dictates too."""
|
||||
counts as its own)."""
|
||||
bound = rules.get("key_bindings")
|
||||
if isinstance(bound, dict):
|
||||
return dict(bound)
|
||||
return {**DEFAULT_KEY_BINDINGS, **(voice_settings.VOICE_KEY_BINDINGS if voice_installed() else {})}
|
||||
return dict(bound) if isinstance(bound, dict) else dict(DEFAULT_KEY_BINDINGS)
|
||||
# The gaze service's settings (gaze/ft-gazed): whose eye tracking, and the eye bias. GAZE_TRACKER
|
||||
# can also be auto (the default): ours when it's installed, else SteamVR's.
|
||||
GAZE_TRACKERS = {"steam": "SteamVR's eye tracker", "own": "our own eye tracker"}
|
||||
@@ -443,8 +308,6 @@ class Backend(QObject):
|
||||
driverChanged = Signal()
|
||||
panelsChanged = Signal()
|
||||
pauseChanged = Signal()
|
||||
handsChanged = Signal()
|
||||
voiceChanged = Signal()
|
||||
activity = Signal(str) # device id
|
||||
captured = Signal(int, str) # code, name
|
||||
capturedController = Signal(str, str) # button, label
|
||||
@@ -463,9 +326,6 @@ class Backend(QObject):
|
||||
self._capture_combo = "" # the action a key combination is being captured for
|
||||
self._combo_mods = set()
|
||||
self._combo_tap = None # a modifier pressed alone, nothing since: released, it's a tap
|
||||
self._combo_tap_code = None # that modifier's key as it is (Right Ctrl: 97)
|
||||
self._voice = {} # ft-voiced's status, {} when it isn't running
|
||||
self._voice_enabled = False # its service starts at login (systemctl, refreshVoice)
|
||||
self._vr = {} # the helper's vrstatus, {} when it doesn't answer
|
||||
self._vr_at = 0.0
|
||||
self._gaze = {} # ft-gazed's status, {} when it isn't running
|
||||
@@ -477,9 +337,6 @@ class Backend(QObject):
|
||||
self._panels = None # SteamVR's overlays, from the helper; None until it answers
|
||||
self._pause = {} # the relay's pause state ("pause ?"), {} when it doesn't answer
|
||||
self._pause_at = 0.0
|
||||
self._hands = {} # handsmode.py's status, {} until it answers
|
||||
self._hands_lock = threading.Lock() # one handsmode.py at a time
|
||||
self._hands_refreshing = False
|
||||
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
self.sock.bind("") # autobind an abstract address the relay can reply to
|
||||
self.sock.setblocking(False)
|
||||
@@ -529,10 +386,6 @@ class Backend(QObject):
|
||||
if self._pause and now - self._pause_at > 5:
|
||||
self._pause = {}
|
||||
self.pauseChanged.emit()
|
||||
voice = self._voice_status() if voice_installed() else {}
|
||||
if voice != self._voice:
|
||||
self._voice = voice
|
||||
self.voiceChanged.emit()
|
||||
|
||||
def _check_driver(self):
|
||||
block = driver_block()
|
||||
@@ -602,12 +455,9 @@ class Backend(QObject):
|
||||
mod = MODIFIER_CODES[code]
|
||||
if value == 1:
|
||||
self._combo_tap = None if self._combo_mods else mod
|
||||
self._combo_tap_code = code
|
||||
(self._combo_mods.add if value else self._combo_mods.discard)(mod)
|
||||
if value == 0 and self._combo_tap == mod:
|
||||
# The relay's modifier tap: a right-hand key by its own code, that key alone.
|
||||
tapped = self._combo_tap_code
|
||||
self._end_shortcut_capture(str(tapped if tapped in RIGHT_MODIFIER_NAMES else mod))
|
||||
self._end_shortcut_capture(str(mod)) # the relay's modifier tap
|
||||
elif value == 1 and code < BTN_MISC:
|
||||
self._end_shortcut_capture("+".join(str(c) for c in sorted(self._combo_mods) + [code]))
|
||||
elif value == 1:
|
||||
@@ -1191,8 +1041,6 @@ class Backend(QObject):
|
||||
parts = [int(c) for c in combo.split("+") if c.isdigit()]
|
||||
if len(parts) == 1 and parts[0] in MODIFIER_NAMES:
|
||||
return MODIFIER_NAMES[parts[0]] + " tap"
|
||||
if len(parts) == 1 and parts[0] in RIGHT_MODIFIER_NAMES:
|
||||
return RIGHT_MODIFIER_NAMES[parts[0]] + " tap"
|
||||
return "+".join(MODIFIER_NAMES.get(c) or self.codeName(c).removeprefix("KEY_").title() for c in parts)
|
||||
|
||||
@Property("QVariantList", notify=mappingsChanged)
|
||||
@@ -1245,113 +1093,6 @@ class Backend(QObject):
|
||||
self._save_rules(rules)
|
||||
self.message.emit(f"{self.comboName(combo)} removed", False)
|
||||
|
||||
# --- voice dictation (Frametop Voice) ---
|
||||
def _voice_status(self):
|
||||
"""ft-voiced's "status" ({} when it isn't running)."""
|
||||
try:
|
||||
with socket.socket(socket.AF_UNIX) as s:
|
||||
s.settimeout(0.3)
|
||||
s.connect(VOICE_SOCKET)
|
||||
s.sendall(b"status\n")
|
||||
return json.loads(s.makefile().readline())
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
@Slot()
|
||||
def refreshVoice(self):
|
||||
"""Whether the service starts at login: asked of the host's systemd, so only when the
|
||||
Voice page opens and after a change, not every 2 s."""
|
||||
self._voice_enabled = host("systemctl", "--user", "is-enabled", VOICE_UNIT).stdout.strip() == "enabled"
|
||||
self.voiceChanged.emit()
|
||||
|
||||
@Property(bool, notify=voiceChanged)
|
||||
def voiceInstalled(self):
|
||||
return voice_installed()
|
||||
|
||||
@Property(bool, notify=voiceChanged)
|
||||
def voiceEnabled(self):
|
||||
return self._voice_enabled
|
||||
|
||||
@Property("QVariantMap", notify=voiceChanged)
|
||||
def voiceStatus(self):
|
||||
v = self._voice
|
||||
return {"running": bool(v), "dictating": bool(v.get("dictating")), "wake": bool(v.get("wake_enabled")),
|
||||
"mic": bool(v.get("mic_open"))}
|
||||
|
||||
@Slot(bool)
|
||||
def setVoiceOn(self, on):
|
||||
result = host("systemctl", "--user", "enable" if on else "disable", "--now", VOICE_UNIT)
|
||||
if result.returncode == 0:
|
||||
self.message.emit("Voice dictation " + ("on" if on else "off, and it won't start at login"), False)
|
||||
else:
|
||||
self.message.emit("Couldn't turn voice dictation " + ("on: " if on else "off: ")
|
||||
+ (result.stderr.strip() or "systemctl failed")[:160], True)
|
||||
self.refreshVoice()
|
||||
|
||||
@Property("QVariantMap", notify=voiceChanged)
|
||||
def voiceSettings(self):
|
||||
s = voice_settings.load()
|
||||
return {k: s[k] for k in voice_settings.DEFAULT}
|
||||
|
||||
def _set_voice(self, key, value, text):
|
||||
s = voice_settings.load()
|
||||
s[key] = value
|
||||
voice_settings.save(s)
|
||||
self.message.emit(text, False)
|
||||
self.voiceChanged.emit()
|
||||
|
||||
@Slot(str, bool)
|
||||
def setVoiceOption(self, key, on):
|
||||
texts = {"live_typing": ("Typing as you talk", "Typing when you stop talking"),
|
||||
"sounds": ("A tone when dictation starts and stops", "No dictation sounds"),
|
||||
"wake_word": ("Listening for the wake word", "Wake word off")}
|
||||
if key in texts:
|
||||
self._set_voice(key, bool(on), texts[key][0 if on else 1])
|
||||
|
||||
@Property("QVariantList", constant=True)
|
||||
def voiceModes(self):
|
||||
return [{"value": "hold", "text": "Hold to talk"},
|
||||
{"value": "toggle", "text": "Press to start, press again to stop"}]
|
||||
|
||||
@Slot(str)
|
||||
def setVoiceControllerMode(self, mode):
|
||||
if mode in voice_settings.MODES:
|
||||
self._set_voice("controller_mode", mode, "Controller: " + ("hold to talk" if mode == "hold"
|
||||
else "press to start and stop"))
|
||||
|
||||
@Property("QVariantList", notify=voiceChanged)
|
||||
def voiceButtons(self):
|
||||
return [{"button": b, "label": CONTROLLER_BUTTONS.get(b, b)} for b in voice_settings.load()["controller"]]
|
||||
|
||||
@Slot(str)
|
||||
def addVoiceButton(self, button):
|
||||
buttons = voice_settings.load()["controller"]
|
||||
if button in CONTROLLER_BUTTONS and button not in buttons:
|
||||
self._set_voice("controller", buttons + [button], f"Dictation: {CONTROLLER_BUTTONS[button]}")
|
||||
|
||||
@Slot(str)
|
||||
def removeVoiceButton(self, button):
|
||||
buttons = voice_settings.load()["controller"]
|
||||
if button in buttons:
|
||||
self._set_voice("controller", [b for b in buttons if b != button],
|
||||
f"{CONTROLLER_BUTTONS.get(button, button)}: no longer dictation")
|
||||
|
||||
@Slot(str)
|
||||
def setWakePhrases(self, text):
|
||||
phrases = list(dict.fromkeys(p for p in (voice_settings.clean_phrase(t) for t in re.split(r"[,\n]", text)) if p))
|
||||
if phrases and phrases != voice_settings.load()["wake_phrases"]:
|
||||
self._set_voice("wake_phrases", phrases, "Wake word: " + ", ".join(p.title() for p in phrases))
|
||||
|
||||
@Property("QVariantList", notify=mappingsChanged)
|
||||
def dictationKeys(self):
|
||||
return [{"combo": c, "label": self.comboName(c), "actionLabel": action_label(a)}
|
||||
for c, a in sorted(key_bindings(read_json(RULES_PATH)).items()) if a in VOICE_ACTIONS]
|
||||
|
||||
@Property("QVariantList", notify=voiceChanged)
|
||||
def voiceCommands(self):
|
||||
commands, _ = voice_commands.parse(voice_settings.load()["commands"])
|
||||
return [{"say": " / ".join(p for p, _ in c.phrases), "run": " ".join(c.run)} for c in commands]
|
||||
|
||||
@Property(str, notify=gazeChanged)
|
||||
def gazeTracker(self):
|
||||
"""Whose eye tracking the gaze service uses: "steam" or "own" (GAZE_TRACKER; auto is
|
||||
@@ -1470,125 +1211,6 @@ class Backend(QObject):
|
||||
except OSError as e:
|
||||
self.message.emit(f"Couldn't open the gaze probe: {e}", True)
|
||||
|
||||
# --- hands ---
|
||||
@Property("QVariantList", constant=True)
|
||||
def handsModes(self):
|
||||
return [{"value": k, "text": v} for k, v in HANDS_MODES.items()]
|
||||
|
||||
@Property(str, notify=handsChanged)
|
||||
def handsMode(self):
|
||||
return handsmode.mode_of(read_conf())
|
||||
|
||||
@Property("QVariantMap", notify=handsChanged)
|
||||
def handsStatus(self):
|
||||
"""handsmode.py's status: "summary", "problem" (why it can't run, or ""), and the rest."""
|
||||
return self._hands
|
||||
|
||||
def _hands_run(self, *args):
|
||||
"""handsmode.py on the host (systemctl, getcap and the cameras are there), off the UI
|
||||
thread: distrobox-host-exec takes a moment. Then its status."""
|
||||
with self._hands_lock:
|
||||
result = host("python3", HANDSMODE, *args) if args else None
|
||||
st = host("python3", HANDSMODE, "status", "--json")
|
||||
try:
|
||||
self._hands = json.loads(st.stdout)
|
||||
except ValueError:
|
||||
self._hands = {"summary": "", "problem": "Couldn't ask the host about hand tracking: "
|
||||
+ (st.stderr.strip() or st.stdout.strip() or "no answer")[:160]}
|
||||
self._hands_refreshing = False
|
||||
self.handsChanged.emit()
|
||||
return result
|
||||
|
||||
@Slot()
|
||||
def refreshHands(self):
|
||||
if not self._hands_refreshing:
|
||||
self._hands_refreshing = True
|
||||
threading.Thread(target=self._hands_run, daemon=True).start()
|
||||
|
||||
@Slot(str)
|
||||
def setHandsMode(self, mode):
|
||||
"""Written here so the page follows at once; handsmode.py set writes the same and applies it
|
||||
(starts and stops the services), and the input relay rereads it (games keep the hands with
|
||||
HANDS_MODE=always)."""
|
||||
if mode not in HANDS_MODES:
|
||||
return
|
||||
write_conf_value("HANDS_MODE", mode)
|
||||
self.handsChanged.emit()
|
||||
self.reload_timer.start()
|
||||
|
||||
def job():
|
||||
result = self._hands_run("set", mode)
|
||||
if result.returncode != 0:
|
||||
self.message.emit("Couldn't apply it: " + (result.stderr.strip() or "no answer")[:160], True)
|
||||
else:
|
||||
self.message.emit("Hand tracking: " + HANDS_MODES[mode].split(":")[0], False)
|
||||
threading.Thread(target=job, daemon=True).start()
|
||||
|
||||
@Property("QVariantMap", notify=handsChanged)
|
||||
def handMethods(self):
|
||||
"""{method: on} for the Hands page's input switches (handsmode.methods_of)."""
|
||||
return handsmode.methods_of(read_conf())
|
||||
|
||||
@Slot(str, bool)
|
||||
def setHandMethod(self, method, on):
|
||||
"""One hand input on or off, the other two left as they are (handsmode.method_writes). The
|
||||
helper and the relay reread frametop.conf; ft-screens hears direct touch now."""
|
||||
if method not in HAND_METHODS:
|
||||
return
|
||||
writes = handsmode.method_writes(read_conf(), method, on)
|
||||
for key, value in writes.items():
|
||||
write_conf_value(key, value)
|
||||
if "HANDS_NEAR" in writes:
|
||||
self._send("handinput " + ("off" if writes["HANDS_NEAR"] == "0" else "on"), SCREENS)
|
||||
self.reload_timer.start()
|
||||
self.handsChanged.emit()
|
||||
self.message.emit(f"{HAND_METHODS[method]}: {'on' if on else 'off'}", False)
|
||||
|
||||
@Property("QVariantList", notify=handsChanged)
|
||||
def handInteractions(self):
|
||||
"""The Hands page's interaction switches, in order: key, label, tip, group (the first of each
|
||||
group carries it, for the section's label), on."""
|
||||
conf, out, last = read_conf(), [], None
|
||||
for key, default, _, group, label, tip in HAND_INTERACTIONS:
|
||||
raw = conf.get(key, "").strip()
|
||||
on = default if raw == "" else raw not in ("0", "0.0", "off", "no")
|
||||
out.append({"key": key, "label": label, "tip": tip, "group": group if group != last else "", "on": on})
|
||||
last = group
|
||||
return out
|
||||
|
||||
@Slot(str, bool)
|
||||
def setHandInteraction(self, key, on):
|
||||
"""One interaction on or off: frametop.conf (ft-screens reads it at start) and ft-screens now
|
||||
(the pointer helper's POINTER_ ones: it reloads its settings)."""
|
||||
entry = next((e for e in HAND_INTERACTIONS if e[0] == key), None)
|
||||
if entry is None:
|
||||
return
|
||||
write_conf_value(key, entry[2] if on else "0")
|
||||
if key.startswith("POINTER_"): # the pointer helper's: it re-reads frametop.conf
|
||||
self._send("reload", HELPER)
|
||||
else:
|
||||
self._send(f"handsetting {key} {1 if on else 0}", SCREENS)
|
||||
self.handsChanged.emit()
|
||||
self.message.emit(f"{entry[4]}: {'on' if on else 'off'}", False)
|
||||
|
||||
@Property("QVariantMap", constant=True)
|
||||
def handsMarginRange(self):
|
||||
return {"min": handsmode.MARGIN_MIN, "max": handsmode.MARGIN_MAX, "step": handsmode.MARGIN_STEP,
|
||||
"default": handsmode.MARGIN_DEFAULT}
|
||||
|
||||
@Property(float, notify=handsChanged)
|
||||
def handsMargin(self):
|
||||
return handsmode.margin_of(read_conf())
|
||||
|
||||
@Slot(float)
|
||||
def setHandsMargin(self, value):
|
||||
"""HANDS_MATTE_MARGIN, and to ft-screens now (only the still margin: its motion part stays).
|
||||
Without the desktop running it takes it when it starts."""
|
||||
value = format(round(min(max(value, 0.0), 10.0), 2), "g")
|
||||
write_conf_value("HANDS_MATTE_MARGIN", value)
|
||||
self._send(f"cutouts margin {value}", SCREENS)
|
||||
self.handsChanged.emit()
|
||||
|
||||
# --- bluetooth ---
|
||||
@Property("QVariantList", notify=bluetoothChanged)
|
||||
def bluetooth(self):
|
||||
|
||||
+7
-332
@@ -21,11 +21,9 @@ Kirigami.ApplicationWindow {
|
||||
Kirigami.Action { text: "Controllers"; icon.name: "input-gamepad"; onTriggered: root.show(controllersPage) },
|
||||
Kirigami.Action { text: "Game optimization"; icon.name: "applications-games"; onTriggered: root.show(gamesPage) },
|
||||
Kirigami.Action { text: "Keyboard"; icon.name: "input-keyboard-virtual"; onTriggered: root.show(keyboardPage) },
|
||||
Kirigami.Action { text: "Voice"; icon.name: "audio-input-microphone"; onTriggered: root.show(voicePage) },
|
||||
Kirigami.Action { text: "Pointer"; icon.name: "transform-move"; onTriggered: root.show(pointerPage) },
|
||||
Kirigami.Action { text: "Ignored panels"; icon.name: "view-hidden"; onTriggered: root.show(ignorePage) },
|
||||
Kirigami.Action { text: "Gaze"; icon.name: "view-visible"; onTriggered: root.show(gazePage) },
|
||||
Kirigami.Action { text: "Hands"; icon.name: "transform-browse"; onTriggered: root.show(handsPage) },
|
||||
Kirigami.Action { text: "Bluetooth"; icon.name: "preferences-system-bluetooth"; onTriggered: root.show(bluetoothPage) }
|
||||
]
|
||||
}
|
||||
@@ -59,9 +57,9 @@ Kirigami.ApplicationWindow {
|
||||
pageStack.push(page)
|
||||
}
|
||||
|
||||
// FT_INPUT_PAGE=buttons|controllers|games (Game optimization)|keyboard|voice|pointer|ignore|gaze|hands|bluetooth opens the app on that page.
|
||||
// FT_INPUT_PAGE=buttons|controllers|games (Game optimization)|keyboard|pointer|ignore|gaze|bluetooth opens the app on that page.
|
||||
pageStack.initialPage: ({ buttons: buttonsPage, controllers: controllersPage, games: gamesPage, keyboard: keyboardPage,
|
||||
voice: voicePage, pointer: pointerPage, ignore: ignorePage, gaze: gazePage, hands: handsPage,
|
||||
pointer: pointerPage, ignore: ignorePage, gaze: gazePage,
|
||||
bluetooth: bluetoothPage })[startPage] || devicesPage
|
||||
|
||||
Connections {
|
||||
@@ -517,7 +515,7 @@ Kirigami.ApplicationWindow {
|
||||
id: kpage
|
||||
title: "Keyboard"
|
||||
// Pass-through keyboards connected now: with "no_keyboard", the keyboard waits for none.
|
||||
// A program's uinput keyboard (a remote desktop's, say) doesn't count.
|
||||
// A program's uinput keyboard (frame-voice's, say) doesn't count.
|
||||
property var keyboards: backend.devices.filter(d => d.connected && d.role === "passthrough"
|
||||
&& d.kinds.indexOf("keyboard") >= 0 && !d.uinput)
|
||||
|
||||
@@ -595,8 +593,7 @@ Kirigami.ApplicationWindow {
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
|
||||
wrapMode: Text.WordWrap
|
||||
text: "Hold the modifiers (Ctrl, Alt, Shift, Meta), then press the key, on any keyboard. Or tap "
|
||||
+ "one modifier on its own: a left one taps either side, a right one only itself (Right Ctrl). "
|
||||
+ "The combination's last key isn't typed; the modifiers still reach "
|
||||
+ "one modifier on its own. The combination's last key isn't typed; the modifiers still reach "
|
||||
+ "the app. While you're in Steam or a game, they see the keys too. Until you remove or change "
|
||||
+ "them, a Meta tap opens the Steam menu (or closes the dashboard) instead of the desktop's "
|
||||
+ "launcher, and Meta+Shift+F floats the desktop window under the pointer in VR, or puts it "
|
||||
@@ -617,183 +614,8 @@ Kirigami.ApplicationWindow {
|
||||
+ "menu or Steam's own keyboard is up. Type on it with a laser or the 3D mouse. It types into any "
|
||||
+ "app, but only Qt, GTK and Firefox apps say when a text field is selected; for the rest "
|
||||
+ "(Chromium, Electron and X11 apps), map Open/close keyboard to a button on the Buttons or "
|
||||
+ "Controllers page. Keyboards set to Ignore, and keyboards other programs make, don't "
|
||||
+ "count as connected."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Voice
|
||||
Component {
|
||||
id: voicePage
|
||||
Kirigami.ScrollablePage {
|
||||
id: vpage
|
||||
title: "Voice"
|
||||
property var st: backend.voiceStatus
|
||||
property var vs: backend.voiceSettings
|
||||
property bool capturing: false
|
||||
Component.onCompleted: backend.refreshVoice()
|
||||
Component.onDestruction: if (capturing) backend.cancelControllerCapture()
|
||||
|
||||
Connections {
|
||||
target: backend
|
||||
function onCapturedController(button, label) {
|
||||
if (vpage.capturing) { vpage.capturing = false; backend.addVoiceButton(button) }
|
||||
}
|
||||
}
|
||||
Timer {
|
||||
// The relay takes every button for 30 s while capturing.
|
||||
running: vpage.capturing
|
||||
interval: 30000
|
||||
onTriggered: { backend.cancelControllerCapture(); vpage.capturing = false }
|
||||
}
|
||||
|
||||
ColumnLayout {
|
||||
spacing: Kirigami.Units.largeSpacing
|
||||
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: !backend.voiceInstalled
|
||||
type: Kirigami.MessageType.Information
|
||||
text: "Voice dictation isn't installed. Run voice/run.sh install in Frametop's folder: it "
|
||||
+ "downloads about 320 MB of speech models, which stay on the headset."
|
||||
}
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
visible: backend.voiceInstalled && backend.voiceEnabled && !vpage.st.running
|
||||
type: Kirigami.MessageType.Warning
|
||||
text: "Voice dictation is on but not running. See journalctl --user -u frametop-voice."
|
||||
}
|
||||
|
||||
Kirigami.FormLayout {
|
||||
Layout.fillWidth: true
|
||||
enabled: backend.voiceInstalled
|
||||
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "Voice dictation:"
|
||||
text: !vpage.st.running ? "Off" : vpage.st.dictating ? "On, listening now" : "On"
|
||||
checked: backend.voiceEnabled
|
||||
onToggled: backend.setVoiceOn(checked)
|
||||
}
|
||||
|
||||
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Push to talk" }
|
||||
Controls.Label {
|
||||
Kirigami.FormData.label: "Keys:"
|
||||
text: backend.dictationKeys.length
|
||||
? backend.dictationKeys.map(k => k.label + " (" + k.actionLabel.toLowerCase() + ")").join(", ")
|
||||
: "none"
|
||||
wrapMode: Text.Wrap
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 24
|
||||
}
|
||||
Controls.Button {
|
||||
text: "Change on the Keyboard page"
|
||||
icon.name: "input-keyboard"
|
||||
onClicked: root.show(keyboardPage)
|
||||
}
|
||||
Repeater {
|
||||
model: backend.voiceButtons
|
||||
delegate: RowLayout {
|
||||
required property var modelData
|
||||
Kirigami.FormData.label: "Controller:"
|
||||
Controls.Label { text: modelData.label }
|
||||
Controls.ToolButton {
|
||||
icon.name: "edit-delete"
|
||||
display: Controls.AbstractButton.IconOnly
|
||||
text: "Remove"
|
||||
Controls.ToolTip.text: text
|
||||
Controls.ToolTip.visible: hovered
|
||||
onClicked: backend.removeVoiceButton(modelData.button)
|
||||
}
|
||||
}
|
||||
}
|
||||
RowLayout {
|
||||
Kirigami.FormData.label: backend.voiceButtons.length ? "Another:" : "Controller:"
|
||||
Controls.Button {
|
||||
text: vpage.capturing ? "Press a button on a controller…" : "Capture a button"
|
||||
icon.name: "input-gamepad"
|
||||
highlighted: vpage.capturing
|
||||
enabled: backend.controllerStatus.helper
|
||||
onClicked: {
|
||||
if (vpage.capturing) { backend.cancelControllerCapture(); vpage.capturing = false }
|
||||
else { vpage.capturing = true; backend.startControllerCapture() }
|
||||
}
|
||||
}
|
||||
Controls.Label { text: "or"; opacity: 0.7 }
|
||||
Controls.ComboBox {
|
||||
id: voiceButtonBox
|
||||
model: backend.controllerButtons
|
||||
textRole: "text"
|
||||
valueRole: "value"
|
||||
Layout.preferredWidth: Kirigami.Units.gridUnit * 10
|
||||
}
|
||||
Controls.Button {
|
||||
text: "Add"
|
||||
icon.name: "list-add"
|
||||
onClicked: backend.addVoiceButton(voiceButtonBox.currentValue)
|
||||
}
|
||||
}
|
||||
Controls.ComboBox {
|
||||
Kirigami.FormData.label: "Controller buttons:"
|
||||
model: backend.voiceModes
|
||||
textRole: "text"
|
||||
valueRole: "value"
|
||||
Component.onCompleted: currentIndex = indexOfValue(vpage.vs.controller_mode)
|
||||
onActivated: backend.setVoiceControllerMode(currentValue)
|
||||
}
|
||||
|
||||
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Typing" }
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "As you talk:"
|
||||
text: "Type words while you're still talking (off: all at once when you stop)"
|
||||
checked: vpage.vs.live_typing
|
||||
onToggled: backend.setVoiceOption("live_typing", checked)
|
||||
}
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "Sounds:"
|
||||
text: "A short tone when dictation starts and stops"
|
||||
checked: vpage.vs.sounds
|
||||
onToggled: backend.setVoiceOption("sounds", checked)
|
||||
}
|
||||
|
||||
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Wake word" }
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "Wake word:"
|
||||
text: "Listen for it, then for a command (the microphone stays open)"
|
||||
checked: vpage.vs.wake_word
|
||||
onToggled: backend.setVoiceOption("wake_word", checked)
|
||||
}
|
||||
Controls.TextField {
|
||||
Kirigami.FormData.label: "Say:"
|
||||
text: vpage.vs.wake_phrases.map(p => p.toLowerCase().replace(/\b\w/g, c => c.toUpperCase())).join(", ")
|
||||
placeholderText: "Hey Computer"
|
||||
enabled: vpage.vs.wake_word
|
||||
Layout.preferredWidth: Kirigami.Units.gridUnit * 16
|
||||
onEditingFinished: backend.setWakePhrases(text)
|
||||
}
|
||||
Repeater {
|
||||
model: backend.voiceCommands
|
||||
delegate: Controls.Label {
|
||||
required property var modelData
|
||||
Kirigami.FormData.label: "Command:"
|
||||
text: "\u201c" + modelData.say + "\u201d runs " + modelData.run
|
||||
elide: Text.ElideRight
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 24
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Controls.Label {
|
||||
Layout.fillWidth: true
|
||||
wrapMode: Text.Wrap
|
||||
opacity: 0.7
|
||||
text: "Speech is recognised on the headset; nothing leaves it. Text is typed where you type: the "
|
||||
+ "desktop window you clicked last, or Steam and games. It waits while you hold Ctrl, Alt, "
|
||||
+ "Shift or Meta, so letters don't turn into shortcuts. Recordings under 2 seconds are "
|
||||
+ "dropped. A key works only on its own: with another modifier held it's a shortcut, not "
|
||||
+ "dictation. Controller buttons still do what they do in games. After the wake word, a "
|
||||
+ "command goes to voice/hooks/on-command, or runs a program you set in "
|
||||
+ "~/.config/frametop-voice.json (\"commands\"; see voice/README.md)."
|
||||
}
|
||||
+ "Controllers page. Keyboards set to Ignore, and keyboards other programs make (like "
|
||||
+ "frame-voice's), don't count as connected."
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1450,10 +1272,7 @@ Kirigami.ApplicationWindow {
|
||||
+ "with the button held and let go there. Hold still to drag: hold a press this long without moving "
|
||||
+ "to drag something instead. Look away to hand back: how far from the pointer you look before the "
|
||||
+ "gaze takes it back from the mouse. Largest correction to learn: a click corrected further than "
|
||||
+ "this isn't learned; the tracker is that far off, so the quick check runs instead. With hand tracking, "
|
||||
+ "a pinch clicks where you look; pinch, hold and move the hand to nudge the pointer first. Pinch nudge "
|
||||
+ "starts after: how far the hand moves before the pointer does. Pinch nudge speed: how far the pointer "
|
||||
+ "moves for each degree of hand movement after that. Eye tracker: SteamVR's, or our own "
|
||||
+ "this isn't learned; the tracker is that far off, so the quick check runs instead. Eye tracker: SteamVR's, or our own "
|
||||
+ "(gaze/tracker), which keeps its own calibration. Eye bias: the gaze "
|
||||
+ "combines both eyes, since their errors partly cancel; Left or Right counts that eye twice as "
|
||||
+ "much, and Auto weights each by how far off it was at your recent nudges."
|
||||
@@ -1461,150 +1280,6 @@ Kirigami.ApplicationWindow {
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Hands
|
||||
Component {
|
||||
id: handsPage
|
||||
Kirigami.ScrollablePage {
|
||||
id: hpage
|
||||
title: "Hands"
|
||||
header: DriverWarning {}
|
||||
property var st: backend.handsStatus
|
||||
readonly property var range: backend.handsMarginRange
|
||||
// What runs, and why it can't: asked on the host, while this page is open.
|
||||
Timer {
|
||||
running: true
|
||||
repeat: true
|
||||
interval: 5000
|
||||
triggeredOnStart: true
|
||||
onTriggered: backend.refreshHands()
|
||||
}
|
||||
|
||||
Kirigami.FormLayout {
|
||||
Kirigami.InlineMessage {
|
||||
Layout.fillWidth: true
|
||||
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
|
||||
visible: !!hpage.st.problem
|
||||
type: backend.handsMode === "off" ? Kirigami.MessageType.Information : Kirigami.MessageType.Warning
|
||||
text: hpage.st.problem || ""
|
||||
}
|
||||
ColumnLayout {
|
||||
Kirigami.FormData.label: "Hand tracking:"
|
||||
Repeater {
|
||||
model: backend.handsModes
|
||||
delegate: Controls.RadioButton {
|
||||
required property var modelData
|
||||
text: modelData.text
|
||||
checked: backend.handsMode === modelData.value
|
||||
onToggled: if (checked) backend.setHandsMode(modelData.value)
|
||||
}
|
||||
}
|
||||
}
|
||||
Controls.Label {
|
||||
text: hpage.st.summary === undefined ? "…" : hpage.st.summary
|
||||
opacity: 0.7
|
||||
font: Kirigami.Theme.smallFont
|
||||
}
|
||||
RowLayout {
|
||||
Kirigami.FormData.label: "Hand cutout margin:"
|
||||
Controls.Label { text: "smaller"; opacity: 0.7 }
|
||||
Controls.Slider {
|
||||
id: marginSlider
|
||||
from: hpage.range.min
|
||||
to: hpage.range.max
|
||||
stepSize: hpage.range.step
|
||||
snapMode: Controls.Slider.SnapAlways
|
||||
value: backend.handsMargin
|
||||
Layout.preferredWidth: Kirigami.Units.gridUnit * 12
|
||||
onMoved: backend.setHandsMargin(value)
|
||||
}
|
||||
Controls.Label { text: "bigger"; opacity: 0.7 }
|
||||
Controls.Label {
|
||||
text: marginSlider.value.toFixed(2) + "°"
|
||||
Layout.preferredWidth: Kirigami.Units.gridUnit * 3
|
||||
}
|
||||
Controls.ToolButton {
|
||||
icon.name: "edit-undo"
|
||||
display: Controls.AbstractButton.IconOnly
|
||||
text: "Default (" + hpage.range.default + "°)"
|
||||
Controls.ToolTip.text: text
|
||||
Controls.ToolTip.visible: hovered
|
||||
onClicked: { marginSlider.value = hpage.range.default; backend.setHandsMargin(hpage.range.default) }
|
||||
}
|
||||
}
|
||||
|
||||
// Three ways the hands work the desktop, each on its own (handsmode.METHODS).
|
||||
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Hand input" }
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "With gaze on:"
|
||||
text: "Pinch to click where you look"
|
||||
checked: !!backend.handMethods.gaze_clicks
|
||||
onToggled: backend.setHandMethod("gaze_clicks", checked)
|
||||
Controls.ToolTip.visible: hovered
|
||||
Controls.ToolTip.delay: Kirigami.Units.toolTipDelay
|
||||
Controls.ToolTip.text: "Your eyes aim and your hands are the mouse's buttons. Index pinch: left click. "
|
||||
+ "Middle finger to thumb: right click. Pinch and hold still to press and drag. Hold a pinch "
|
||||
+ "with one hand and pinch with the other: both buttons held, each until its pinch opens."
|
||||
}
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "With gaze off:"
|
||||
text: "Point a raised hand for a ray"
|
||||
checked: !!backend.handMethods.ray
|
||||
onToggled: backend.setHandMethod("ray", checked)
|
||||
Controls.ToolTip.visible: hovered
|
||||
Controls.ToolTip.delay: Kirigami.Units.toolTipDelay
|
||||
Controls.ToolTip.text: "A hand raised in front and pointing shows a ray that works like a controller's "
|
||||
+ "laser: pinch to click, middle pinch to right-click or scroll. A pinch with the hand held low "
|
||||
+ "presses where the pointer is."
|
||||
}
|
||||
Controls.Switch {
|
||||
Kirigami.FormData.label: "Direct touch:"
|
||||
text: "Poke and grab Frametop's screens, gaze on or off"
|
||||
checked: !!backend.handMethods.touch
|
||||
onToggled: backend.setHandMethod("touch", checked)
|
||||
Controls.ToolTip.visible: hovered
|
||||
Controls.ToolTip.delay: Kirigami.Units.toolTipDelay
|
||||
Controls.ToolTip.text: "Touch a screen with your index finger to click it. Pinch its bar, an edge or "
|
||||
+ "a corner to move or resize it."
|
||||
}
|
||||
Controls.Label {
|
||||
visible: backend.handsMode === "off"
|
||||
text: "These need hand tracking on (above)."
|
||||
opacity: 0.7
|
||||
font: Kirigami.Theme.smallFont
|
||||
}
|
||||
|
||||
// Every interaction on Frametop's screens, each on its own (ft_input_settings.HAND_INTERACTIONS);
|
||||
// they apply at once. Direct touch above switches all of them.
|
||||
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Hand interactions" }
|
||||
Repeater {
|
||||
model: backend.handInteractions
|
||||
delegate: Controls.Switch {
|
||||
required property var modelData
|
||||
Kirigami.FormData.label: modelData.group ? modelData.group + ":" : ""
|
||||
text: modelData.label
|
||||
checked: modelData.on
|
||||
enabled: !!backend.handMethods.touch
|
||||
onToggled: backend.setHandInteraction(modelData.key, checked)
|
||||
Controls.ToolTip.visible: hovered
|
||||
Controls.ToolTip.delay: Kirigami.Units.toolTipDelay
|
||||
Controls.ToolTip.text: modelData.tip
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
footer: Controls.Label {
|
||||
padding: Kirigami.Units.largeSpacing
|
||||
wrapMode: Text.Wrap
|
||||
opacity: 0.7
|
||||
text: "Your hands show through Frametop's screens where they're in front of them. Hand tracking "
|
||||
+ "uses a lot of the headset's CPU. The margin is how far past your hand the screen opens up; "
|
||||
+ "it applies right away. The hand inputs are separate: any of them can be on alone, and they "
|
||||
+ "apply within a second. A hand holding a controller, or whose controller you just used, "
|
||||
+ "does none of them."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Bluetooth
|
||||
Component {
|
||||
id: bluetoothPage
|
||||
|
||||
+14
-192
@@ -5,11 +5,7 @@ Paused:
|
||||
- The gaze service stops (frametop-gaze: ft-gazed, ft-gaze, our own eye tracker ft-eyes, the
|
||||
gaze panel), and with it every read of SteamVR's eye tracking. Our frame grabber, the root
|
||||
service ft-eyegrab, goes idle by itself 3 seconds after ft-eyes stops asking for frames.
|
||||
- Hand tracking stops if it runs (frametop-camd, frametop-hands). With hand interaction on
|
||||
(POINTER_HANDS=1) it's parked instead (PARK_PATH: ft-hands stops processing and ft-camd stops
|
||||
copying, the models stay loaded), and it unparks while the dashboard is open, so the hands work
|
||||
there during the game. With HANDS_MODE=always it keeps running, unparked: the hands show in
|
||||
the game (hands/handsmode.py).
|
||||
- Hand tracking stops if it runs (frametop-camd, frametop-hands).
|
||||
- The desktop, by "pause_desktop": "hide" (the default) hides every screen and floating window
|
||||
and slows the desktop down ("pause on" to ft-screens: KWin gets one frame callback a second
|
||||
instead of 90, so it and the apps on it hardly draw), and remote desktop stops if it runs
|
||||
@@ -56,19 +52,11 @@ DESKTOPS = os.path.join(REPO, "desktops.sh")
|
||||
REMOTE_CTL = os.path.join(REPO, "session", "remote-ctl.sh")
|
||||
RELAY = "\0frametop_relay"
|
||||
SCREENS = "\0ft_screens"
|
||||
HELPER = "\0ft_pointer_helper"
|
||||
STATE_PATH = f"/run/user/{os.getuid()}/frametop-pause.json"
|
||||
# User services that pausing stops, if they run.
|
||||
UNITS = ("frametop-gaze.service", "frametop-hands.service", "frametop-camd.service")
|
||||
# With hand interaction on (POINTER_HANDS=1), pausing parks hand tracking instead of stopping it, so
|
||||
# the hands are there when the dashboard opens during the game: ft-hands stops processing while
|
||||
# PARK_PATH exists (and ft-camd stops copying frames for it), and starts again within a palm search
|
||||
# (about 200 ms) once it goes. The pointer helper says when the dashboard opens and closes ("vrdash").
|
||||
HAND_UNITS = ("frametop-hands.service", "frametop-camd.service")
|
||||
PARK_PATH = f"/run/user/{os.getuid()}/frametop-hands/park"
|
||||
# What still works while paused, besides commands ("command:CMD"): the toggle itself, Steam's menu,
|
||||
# and dictation (Frametop Voice types into games too).
|
||||
PAUSED_ACTIONS = ("pause_toggle", "steam_menu", "dictate", "dictate_hold")
|
||||
# What still works while paused, besides commands ("command:CMD"): the toggle itself, and Steam's menu.
|
||||
PAUSED_ACTIONS = ("pause_toggle", "steam_menu")
|
||||
DEFAULT_GESTURE = {"buttons": ["left/thumbstick", "right/thumbstick"], "presses": 2}
|
||||
DESKTOP_MODES = ("hide", "close")
|
||||
SOUNDS = {True: "/usr/share/sounds/ocean/stereo/device-removed.oga",
|
||||
@@ -148,113 +136,21 @@ class Gesture:
|
||||
return True
|
||||
|
||||
|
||||
class TouchTracker:
|
||||
"""Who holds a controller, for the pointer helper's handoff arbiter (pointer/helper/handoff.h):
|
||||
per side, whether any touch sensor of that side's controller is touched, a button of it went
|
||||
down, or it's gone. Fed the web socket's messages; returns the datagrams for the helper, only
|
||||
on a change:
|
||||
ctl <side> touch 0|1 any touch component of that side's controller(s) is true
|
||||
ctl <side> press a click component went down
|
||||
ctl <side> gone no controller on that side any more
|
||||
ctl ws 0|1 the web socket is up or down (ControllerWatch sends these)
|
||||
A side can have more than one controller (the testbench's synthetic ones beside a real one):
|
||||
any of them counts. vrserver's first message for a subscription has every component, so the
|
||||
state is known from the start; its clicks aren't presses."""
|
||||
|
||||
def __init__(self):
|
||||
self.sides = {} # root path -> side
|
||||
self.touch = {} # root path -> {component: bool}
|
||||
self.clicks = {} # root path -> {component: bool}
|
||||
self.sent = {} # side -> the touch value last sent
|
||||
|
||||
def set_sides(self, sides):
|
||||
"""The controllers there are now ({root path: side}); "gone" for a side that has none left."""
|
||||
out = []
|
||||
for path in list(self.sides):
|
||||
if path not in sides:
|
||||
self.touch.pop(path, None)
|
||||
self.clicks.pop(path, None)
|
||||
old = set(self.sides.values())
|
||||
self.sides = dict(sides)
|
||||
for side in sorted(old - set(self.sides.values())):
|
||||
self.sent.pop(side, None)
|
||||
out.append(f"ctl {side} gone")
|
||||
for side in sorted(set(self.sides.values())): # a touched controller of a side went
|
||||
if side in self.sent and self.sent[side] != self.side_touched(side):
|
||||
self.sent[side] = self.side_touched(side)
|
||||
out.append(f"ctl {side} touch {int(self.sent[side])}")
|
||||
return out
|
||||
|
||||
def side_touched(self, side):
|
||||
return any(v for p, comps in self.touch.items() if self.sides.get(p) == side for v in comps.values())
|
||||
|
||||
def feed(self, path, components):
|
||||
side = self.sides.get(path)
|
||||
if side is None:
|
||||
return []
|
||||
out = []
|
||||
first = path not in self.clicks
|
||||
clicks = self.clicks.setdefault(path, {})
|
||||
touch = self.touch.setdefault(path, {})
|
||||
pressed = False
|
||||
for name, value in components.items():
|
||||
if name.endswith("/touch"):
|
||||
touch[name] = bool(value)
|
||||
elif name.endswith("/click"):
|
||||
if value and not clicks.get(name, False) and not first:
|
||||
pressed = True
|
||||
clicks[name] = bool(value)
|
||||
if pressed:
|
||||
out.append(f"ctl {side} press")
|
||||
now = self.side_touched(side)
|
||||
if self.sent.get(side) != now:
|
||||
self.sent[side] = now
|
||||
out.append(f"ctl {side} touch {int(now)}")
|
||||
return out
|
||||
|
||||
def snapshot(self):
|
||||
"""Every side's touch state, as datagrams (for a helper that just started)."""
|
||||
return [f"ctl {side} touch {int(self.side_touched(side))}" for side in sorted(set(self.sides.values()))]
|
||||
|
||||
|
||||
class ControllerWatch(threading.Thread):
|
||||
"""Follows the Frame controllers on vrserver's web socket for the pause gesture, and sends
|
||||
the relay "pause toggle gesture" when it's made. With the touch feed on (hand interaction,
|
||||
POINTER_HANDS=1), it also sends the pointer helper who holds a controller (TouchTracker).
|
||||
Reconnects every 5 seconds while SteamVR is away. With no gesture set and no touch feed, it
|
||||
doesn't connect."""
|
||||
the relay "pause toggle gesture" when it's made. Reconnects every 5 seconds while SteamVR
|
||||
is away. With no gesture set, it doesn't connect."""
|
||||
|
||||
def __init__(self, log):
|
||||
super().__init__(name="pause-gesture", daemon=True)
|
||||
self.log = log
|
||||
self.lock = threading.Lock()
|
||||
self.gesture = None
|
||||
self.touch_feed = False
|
||||
self.tracker = TouchTracker()
|
||||
self.resend = False # the helper (re)started: send it the touch state again
|
||||
self.changed = threading.Event()
|
||||
self.connected = False
|
||||
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
self.lookup_at = None # look the controllers up again then (see recheck)
|
||||
|
||||
def set_touch_feed(self, on):
|
||||
if on != self.touch_feed:
|
||||
self.touch_feed = on
|
||||
self.log(f"controller touch feed for the pointer helper {'on' if on else 'off'}")
|
||||
if not on:
|
||||
self.to_helper(["ctl ws 0"])
|
||||
self.changed.set()
|
||||
|
||||
def helper_started(self):
|
||||
self.resend = True
|
||||
|
||||
def to_helper(self, msgs):
|
||||
for m in msgs:
|
||||
try:
|
||||
self.sock.sendto(m.encode(), HELPER)
|
||||
except OSError:
|
||||
pass # the helper isn't running; it says vrhello when it starts
|
||||
|
||||
def recheck(self, delay=1.5):
|
||||
"""The controllers' root paths may change soon: the 3D mouse's virtual controller is taking
|
||||
or giving back its hand role. SteamVR takes a moment to hand the role over."""
|
||||
@@ -276,7 +172,7 @@ class ControllerWatch(threading.Thread):
|
||||
|
||||
def run(self):
|
||||
while True:
|
||||
if self.current() is None and not self.touch_feed:
|
||||
if self.current() is None:
|
||||
if self.connected:
|
||||
self.log("pause gesture: none set, stopped reading the controllers")
|
||||
self.connected = False
|
||||
@@ -289,8 +185,6 @@ class ControllerWatch(threading.Thread):
|
||||
if self.connected:
|
||||
self.log(f"pause gesture: SteamVR's web socket went away ({e}); retrying")
|
||||
self.connected = False
|
||||
if self.touch_feed:
|
||||
self.to_helper(["ctl ws 0"])
|
||||
self.changed.wait(5)
|
||||
self.changed.clear()
|
||||
|
||||
@@ -301,18 +195,10 @@ class ControllerWatch(threading.Thread):
|
||||
sides = {} # root path -> side
|
||||
next_poll = 0.0
|
||||
last_poll = 0.0
|
||||
self.tracker = TouchTracker()
|
||||
touch_was = False
|
||||
while True:
|
||||
g = self.current()
|
||||
touch = self.touch_feed
|
||||
if g is None and not touch:
|
||||
if g is None:
|
||||
return
|
||||
if touch and (not touch_was or self.resend):
|
||||
# the feed just started, or the helper restarted: where things stand
|
||||
self.resend = False
|
||||
self.to_helper(["ctl ws 1"] + self.tracker.snapshot())
|
||||
touch_was = touch
|
||||
now = time.monotonic()
|
||||
lookup_at = self.lookup_at
|
||||
if lookup_at is not None and now >= lookup_at:
|
||||
@@ -331,27 +217,19 @@ class ControllerWatch(threading.Thread):
|
||||
for path in found.keys() - sides.keys():
|
||||
ws.subscribe(path)
|
||||
sides = found
|
||||
if g is not None:
|
||||
g.reset()
|
||||
msgs = self.tracker.set_sides(sides)
|
||||
if touch:
|
||||
self.to_helper(msgs)
|
||||
g.reset()
|
||||
if not self.connected:
|
||||
self.connected = True
|
||||
self.log(f"{'pause gesture' if g is not None else 'touch feed'}: following "
|
||||
f"{' and '.join(sorted(sides.values())) or 'no'} "
|
||||
self.log(f"pause gesture: following {' and '.join(sorted(sides.values())) or 'no'} "
|
||||
f"controller{'s' if len(sides) != 1 else ''} on SteamVR's web socket")
|
||||
wanted = {} if g is None else \
|
||||
{b: (b.split("/", 1)[0], f"/input/{b.split('/', 1)[1]}/click") for b in g.buttons}
|
||||
wanted = {b: (b.split("/", 1)[0], f"/input/{b.split('/', 1)[1]}/click") for b in g.buttons}
|
||||
for text in ws.messages(timeout=1.0):
|
||||
device = device_of(text)
|
||||
if device and device not in sides:
|
||||
next_poll = min(next_poll, last_poll + LOOKUP_GAP) # its root path changed, maybe
|
||||
# About 160 messages a second, nearly all capacitive sensing: parse only the
|
||||
# few that carry one of the gesture's buttons, or with the touch feed on, the
|
||||
# touch and click ones.
|
||||
feed = touch and device in sides and ("/touch" in text or "/click" in text)
|
||||
if not feed and not any(c in text for _, c in wanted.values()):
|
||||
# few that carry one of the gesture's buttons.
|
||||
if not any(c in text for _, c in wanted.values()):
|
||||
continue
|
||||
try:
|
||||
msg = json.loads(text)
|
||||
@@ -359,8 +237,6 @@ class ControllerWatch(threading.Thread):
|
||||
continue
|
||||
side = sides.get(msg.get("device"))
|
||||
components = msg.get("components") or {}
|
||||
if feed:
|
||||
self.to_helper(self.tracker.feed(msg.get("device"), components))
|
||||
for button, (s, component) in wanted.items():
|
||||
if s == side and component in components:
|
||||
if g.feed(button, bool(components[component]), time.monotonic()):
|
||||
@@ -410,7 +286,6 @@ class Worker(threading.Thread):
|
||||
self.lock = threading.Lock()
|
||||
self.stopped = {"units": [], "remote": False, "desktop": None}
|
||||
self.busy = False
|
||||
self.keep = () # units pausing leaves running (hand tracking, parked instead)
|
||||
|
||||
def submit(self, job, *args):
|
||||
self.busy = True
|
||||
@@ -439,7 +314,7 @@ class Worker(threading.Thread):
|
||||
|
||||
def stop_units(self):
|
||||
"""Stop the UNITS that run, adding them to what the resume starts again."""
|
||||
units = [u for u in UNITS if u not in self.keep and active(u)]
|
||||
units = [u for u in UNITS if active(u)]
|
||||
if units:
|
||||
run(["systemctl", "--user", "stop", *units], timeout=30)
|
||||
with self.lock:
|
||||
@@ -504,10 +379,6 @@ class GamePause:
|
||||
self.game_seen = 0.0 # last "vrgame" from the helper
|
||||
self.resume_at = None
|
||||
self.settings = settings_of({}, buttons)
|
||||
self.hands = False # hand interaction is on (set_hands)
|
||||
self.hands_always = False # HANDS_MODE=always: hand tracking runs through games (set_hands)
|
||||
self.dashboard = False # the SteamVR dashboard is open (the helper's "vrdash")
|
||||
self.parked = None # hand tracking parked, as last applied (None: not yet)
|
||||
self.screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
|
||||
self.worker = Worker(log, self.save)
|
||||
self.watch = ControllerWatch(log)
|
||||
@@ -568,49 +439,6 @@ class GamePause:
|
||||
except OSError:
|
||||
pass # the desktop isn't running
|
||||
|
||||
def tell_helper(self):
|
||||
"""The pointer helper's hand interaction runs in a VR game only while Frametop isn't paused
|
||||
(toggled back on during the game) or the dashboard is open: "paused 1|0", on a change and
|
||||
when the helper starts."""
|
||||
try:
|
||||
self.screens.sendto(b"paused 1" if self.paused else b"paused 0", HELPER)
|
||||
except OSError:
|
||||
pass # the helper isn't running; it says vrhello when it starts
|
||||
|
||||
def set_hands(self, on, always=False):
|
||||
"""Hand interaction (POINTER_HANDS=1): feed the helper who holds a controller, and park hand
|
||||
tracking for games instead of stopping it. always (HANDS_MODE=always): neither."""
|
||||
self.hands, self.hands_always = on, always
|
||||
self.worker.keep = HAND_UNITS if on or always else ()
|
||||
self.watch.set_touch_feed(on)
|
||||
self.update_park()
|
||||
|
||||
def dashboard_state(self, on):
|
||||
"""From the pointer helper: the dashboard opened or closed ("vrdash 1|0")."""
|
||||
self.dashboard = on
|
||||
self.update_park()
|
||||
|
||||
def update_park(self):
|
||||
"""Hand tracking is parked while Frametop is paused with hand interaction on, unless the
|
||||
dashboard is open (hands work there during a game) or HANDS_MODE=always."""
|
||||
want = self.paused and self.hands and not self.hands_always and not self.dashboard
|
||||
if want == self.parked:
|
||||
return
|
||||
try:
|
||||
if want:
|
||||
if os.path.isdir(os.path.dirname(PARK_PATH)): # hand tracking has run: it'll read it
|
||||
with open(PARK_PATH, "w") as f:
|
||||
f.write("parked by the input relay (Frametop paused for a VR game)\n")
|
||||
elif os.path.exists(PARK_PATH):
|
||||
os.remove(PARK_PATH)
|
||||
except OSError as e:
|
||||
self.log(f"hand tracking: can't {'park' if want else 'unpark'} it: {e}")
|
||||
return
|
||||
if self.parked is not None or want:
|
||||
self.log(f"hand tracking {'parked' if want else 'unparked'}"
|
||||
f"{' (the dashboard is open)' if not want and self.paused else ''}")
|
||||
self.parked = want
|
||||
|
||||
def pause(self, reason, now):
|
||||
if self.paused:
|
||||
return
|
||||
@@ -619,9 +447,7 @@ class GamePause:
|
||||
self.resume_at = None
|
||||
self.log(f"Frametop paused ({reason}{', until the game ends' if self.ends_with_game else ''})")
|
||||
self.sound(True)
|
||||
self.tell_screens(True)
|
||||
self.tell_helper()
|
||||
self.update_park() # at once, even when the desktop is closing: that takes a while
|
||||
self.tell_screens(True) # at once, even when the desktop is closing: that takes a while
|
||||
self.on_change(True)
|
||||
self.worker.submit(self.worker.pause, self.settings["desktop"])
|
||||
self.save()
|
||||
@@ -633,8 +459,6 @@ class GamePause:
|
||||
self.log(f"Frametop resumed ({reason})")
|
||||
self.sound(False)
|
||||
self.tell_screens(False)
|
||||
self.tell_helper()
|
||||
self.update_park()
|
||||
self.on_change(False)
|
||||
self.worker.submit(self.worker.resume)
|
||||
self.save()
|
||||
@@ -680,8 +504,6 @@ class GamePause:
|
||||
|
||||
def helper_started(self):
|
||||
"""The pointer helper (re)started, so SteamVR did too, maybe with the gaze service."""
|
||||
self.tell_helper()
|
||||
self.watch.helper_started()
|
||||
if self.paused:
|
||||
self.worker.submit(self.worker.enforce)
|
||||
|
||||
|
||||
+58
-168
@@ -35,26 +35,21 @@ room about your head so the next one to the right or left comes to the front (ft
|
||||
Meta+Alt+Tab and Meta+Alt+Shift+Tab by default), profile:NAME = switch to that profile (ft-layout
|
||||
use NAME: its screens and apps; docs/profiles.md), steam_menu = open the SteamVR dashboard on
|
||||
Steam's menu, or close the dashboard (steam/ft-steam menu, through Steam's UI), pause_toggle =
|
||||
pause Frametop for a VR game, or resume it (game_pause.py), dictate = start or stop dictation, and
|
||||
dictate_hold = dictation while held (Frametop Voice, voice/ft-voiced: "toggle|start|stop <source>"
|
||||
to @frametop_voice), spotwatch = open or close spotwatch's search for apps and remote displays
|
||||
("toggle <source>" to @ft_spotwatch; Meta+Space by default), command:CMD = run CMD with sh -c (on the host, as this service: its
|
||||
environment, output to its log, and COMMAND_PATH, so ft-layout, ft-float, ft-steam and ft-voice
|
||||
need no path), key = pass through as a key, none).
|
||||
pause Frametop for a VR game, or resume it (game_pause.py), command:CMD = run
|
||||
CMD with sh -c (on the host, as this service: its environment, output to its log, and
|
||||
COMMAND_PATH, so ft-layout, ft-float and ft-steam need no path), key = pass through as a key, none).
|
||||
|
||||
Frame controller buttons can be mapped too ("controller_buttons": {"right/a": action} in the
|
||||
rules file; any action but key and the gaze ones, GAZE_ACTIONS: gaze mode is a mouse feature,
|
||||
docs/gaze-controllers.md). So can key combinations on any keyboard ("key_bindings":
|
||||
{"29+56+34": action}, evdev codes joined by "+", modifiers first and left-hand codes for
|
||||
either side, here Ctrl+Alt+G): the combination does the action, and its last key isn't typed.
|
||||
A modifier on its own ("125": Meta) is a tap: pressed with no other modifier held, and released
|
||||
with no other key, mouse button, or scroll in between; the desktop gets an F24 press before its
|
||||
release, so Plasma's launcher doesn't open on a Meta tap that's bound. A tap's left-hand code is
|
||||
either side's; a right-hand code ("97": Right Ctrl) is that key's alone, and comes first. A rules
|
||||
file without "key_bindings" gets DEFAULT_KEY_BINDINGS (Meta tap: steam_menu, Meta+J: gaze_left,
|
||||
Meta+K: gaze_right, Meta+Shift+F: float_toggle), plus a Right Ctrl tap for dictate once Frametop
|
||||
Voice is installed (VOICE_KEY_BINDINGS); one with its own, even an empty one, doesn't. The float,
|
||||
profile, Steam menu, pause, dictation and command actions work without pointer mode too. A combination with Meta also sends the desktop an
|
||||
A modifier on its own ("125": Meta) is a tap: pressed and released with no other key, mouse
|
||||
button, or scroll in between; the desktop gets an F24 press before its release, so Plasma's
|
||||
launcher doesn't open on a Meta tap that's bound. A rules file without "key_bindings" gets
|
||||
DEFAULT_KEY_BINDINGS (Meta tap: steam_menu, Meta+J: gaze_left, Meta+K: gaze_right, Meta+Shift+F:
|
||||
float_toggle); one with its own, even an empty one, doesn't. The float, profile, Steam menu,
|
||||
pause and command actions work without pointer mode too. A combination with Meta also sends the desktop an
|
||||
F24 press and Meta's release right away: so letting go of Meta doesn't open Plasma's launcher, and
|
||||
a gaze click isn't Meta+click (KWin's window move and resize). Another key while Meta is still
|
||||
held gives the desktop Meta back. While typing goes to Steam, keyboards aren't grabbed, so Steam
|
||||
@@ -82,13 +77,11 @@ Typing goes to the panel clicked last, and ft-screens says which ("keyboard
|
||||
desktop|steam" on the control socket, every second). While it's the desktop,
|
||||
pass-through keyboards are grabbed, so gamescope, which reads every keyboard itself,
|
||||
doesn't type them into its focused app too. Without word from ft-screens for 3 seconds
|
||||
they're released.
|
||||
|
||||
Dictation (Frametop Voice) types here: its text comes as "type TEXT" on the control socket, only
|
||||
from this user, and goes out key by key, TYPE_STEP apart, where typing goes: to ft-screens for the
|
||||
desktop, else on the virtual keyboard, which SteamVR passes on to Steam and games (US layout,
|
||||
ASCII; the voice service turns the rest into ASCII or drops it). A character waits while a
|
||||
modifier is held on a keyboard, which would make it a shortcut (Ctrl+letter).
|
||||
they're released. With SHARE_KEYS=1 in ~/.config/frametop.conf,
|
||||
a grabbed keyboard's keys also go out as "key <code> <value> <device name>" datagrams on
|
||||
@frametop_keys, for programs that watch every keyboard for a hotkey and lose it to the grab.
|
||||
It's off by default: any local process that binds that name first gets every key typed
|
||||
into the desktop.
|
||||
|
||||
Frametop's keyboard (ft-screens' key panel): the desktop's input method (input/ft-textinput)
|
||||
says "textfield 1|0" when a text field on the desktop gains or loses keyboard focus, and
|
||||
@@ -108,13 +101,6 @@ keyboards) get their volume entries remapped to unused stand-in codes, so their
|
||||
other keys keep working for SteamVR; a device without a keymap that has only volume
|
||||
keys (the headset's pmic_resin) is grabbed. The keymaps go back when the relay exits.
|
||||
|
||||
Hand interaction (POINTER_HANDS=1, or HANDS_NEAR=on for ft-screens' direct touch alone): the
|
||||
pointer helper decides per side whether a hand holds a controller or is bare
|
||||
(pointer/helper/handoff.h). The relay's web socket reader (game_pause.py, TouchTracker) sends it
|
||||
"ctl <side> touch 0|1", "ctl <side> press", "ctl <side> gone" and "ctl ws 0|1", and the pause
|
||||
state as "paused 1|0" (hands work in a VR game only with the dashboard open or Frametop toggled
|
||||
back on).
|
||||
|
||||
Frametop can pause for VR games (game_pause.py: by hand, with a controller gesture, or by itself
|
||||
while a game runs). Paused, it stops the services that cost the game CPU and GPU, and the relay
|
||||
plays a plain one: the pointer devices feed the virtual mouse and keyboard as without pointer
|
||||
@@ -134,13 +120,9 @@ Control socket (abstract datagram @frametop_relay, JSON replies to the sender):
|
||||
app can capture one; watchers see them as events with id frame_controller
|
||||
vrbtn, vrhello, gazeawake from the pointer helper (above)
|
||||
vrgame 1|0 from the pointer helper: a VR game runs (on a change and every 5 s), for pausing
|
||||
vrdash 1|0 from the pointer helper (hand interaction): the dashboard is open, so hand
|
||||
tracking parked for a game wakes (game_pause.py, PARK_PATH)
|
||||
textfield 1|0 from the desktop's input method (above)
|
||||
pause on|off|toggle [reason] pause Frametop or resume it (input/ft-pause; the gesture reader)
|
||||
pause ? the pause state, as {"t": "pause", ...}
|
||||
type TEXT from Frametop Voice: type TEXT where typing goes (above); ignored from
|
||||
another user
|
||||
|
||||
Runs on the Frame host as a user service (frametop-input-relay.service). The
|
||||
virtual devices are parked in systemd's file descriptor store, so a relay
|
||||
@@ -236,37 +218,19 @@ RULES_PATH = os.path.expanduser("~/.config/frametop-input.json")
|
||||
ACTIONS = ("left", "right", "middle", "back", "scroll_up", "scroll_down", "dashboard", "recenter",
|
||||
"pointer_toggle", "follow_toggle", "gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right",
|
||||
"gaze_quickcal", "sens_up", "sens_down", "layout_reset", "screens_toggle", "keyboard_toggle", "float_toggle",
|
||||
"dock_all", "spin_next", "spin_prev", "steam_menu", "pause_toggle", "dictate", "dictate_hold",
|
||||
"spotwatch", "key", "none")
|
||||
"dock_all", "spin_next", "spin_prev", "steam_menu", "pause_toggle", "key", "none")
|
||||
# Gaze mode is a mouse feature: these never come from a controller button (docs/gaze-controllers.md).
|
||||
GAZE_ACTIONS = ("gaze_toggle", "gaze_precision", "gaze_drag", "gaze_left", "gaze_right", "gaze_quickcal")
|
||||
# Key combinations a rules file without "key_bindings" gets: a Meta tap opens Steam's menu, Meta+J
|
||||
# and Meta+K click at the gaze (free on the Frametop desktop, and apps don't use Meta),
|
||||
# Meta+Shift+F floats a window, and Meta+Alt+Tab and Meta+Alt+Shift+Tab spin the panels around
|
||||
# you (ft-screens' lazy susan); not Meta+Tab, which is Cmd+Tab on a Mac reached through a remote
|
||||
# desktop like RustDesk. Meta+Space opens spotwatch's search (Plasma's KRunner is Alt+Space).
|
||||
# desktop like RustDesk.
|
||||
DEFAULT_KEY_BINDINGS = {"125": "steam_menu", "125+36": "gaze_left", "125+37": "gaze_right",
|
||||
"42+125+33": "float_toggle", "56+125+15": "spin_next", "42+56+125+15": "spin_prev",
|
||||
"125+57": "spotwatch"}
|
||||
# Frametop Voice (voice/): dictation starts and stops on its socket. With it installed (its unit
|
||||
# file), a rules file without "key_bindings" gets these too: a Right Ctrl tap, Right Ctrl alone,
|
||||
# so a Left Ctrl tap (crouch in many games) doesn't start it.
|
||||
VOICE = "\0frametop_voice"
|
||||
VOICE_ACTIONS = ("dictate", "dictate_hold")
|
||||
VOICE_UNIT = os.path.expanduser("~/.config/systemd/user/frametop-voice.service")
|
||||
VOICE_KEY_BINDINGS = {"97": "dictate"}
|
||||
"42+125+33": "float_toggle", "56+125+15": "spin_next", "42+56+125+15": "spin_prev"}
|
||||
KEY_F24 = 194 # sent to the desktop with a Meta combination (see the top)
|
||||
# Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
|
||||
MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
|
||||
KEY_LEFTSHIFT = 42
|
||||
# Dictation's text, typed on a US layout (see the top): character -> (code, with Shift).
|
||||
US_KEYS = {" ": (57, False)}
|
||||
for _row, _shifted, _first in (("1234567890-=", "!@#$%^&*()_+", 2), ("qwertyuiop[]", "QWERTYUIOP{}", 16),
|
||||
("asdfghjkl;'`", 'ASDFGHJKL:"~', 30), ("\\zxcvbnm,./", "|ZXCVBNM<>?", 43)):
|
||||
for _i, (_plain, _shift) in enumerate(zip(_row, _shifted)):
|
||||
US_KEYS[_plain], US_KEYS[_shift] = (_first + _i, False), (_first + _i, True)
|
||||
TYPE_STEP = 0.004 # seconds from a typed key's press to its release, and on to the next key
|
||||
TYPE_MAX = 20000 # characters waiting to be typed, at most
|
||||
VR_KEYBOARD_MODES = ("always", "no_keyboard", "button", "never") # when Frametop's keyboard opens
|
||||
HELPER = "\0ft_pointer_helper"
|
||||
# Frame controller buttons the pointer helper can read (pointer/helper/vrbuttons.h).
|
||||
@@ -282,10 +246,6 @@ FLOAT_ACTIONS = {"float_toggle": b"float pointer", "dock_all": b"dock all"}
|
||||
# Actions for ft-screens: spin_next and spin_prev turn every panel in the room about your head,
|
||||
# so the next one to the right or left comes to the front. They don't need pointer mode either.
|
||||
SCREENS_ACTIONS = {"spin_next": b"spin next", "spin_prev": b"spin prev"}
|
||||
# spotwatch (spotwatch/ft-spotwatch, in the Frametop desktop) opens or closes its search; it's told
|
||||
# what pressed it, since one from a mouse or controller opens Frametop's keyboard too. It doesn't
|
||||
# need pointer mode, and does nothing while Frametop is paused.
|
||||
SPOTWATCH = "\0ft_spotwatch"
|
||||
PROFILE = "profile:" # "profile:NAME": switch to that profile (doesn't need pointer mode either)
|
||||
COMMAND = "command:" # "command:CMD": run CMD (nor does this)
|
||||
|
||||
@@ -296,8 +256,7 @@ def known_action(a):
|
||||
|
||||
|
||||
def needs_pointer(a):
|
||||
return (a not in FLOAT_ACTIONS and a not in SCREENS_ACTIONS and a not in VOICE_ACTIONS
|
||||
and a not in ("steam_menu", "pause_toggle", "spotwatch") and not a.startswith((PROFILE, COMMAND)))
|
||||
return a not in FLOAT_ACTIONS and a not in SCREENS_ACTIONS and a not in ("steam_menu", "pause_toggle") and not a.startswith((PROFILE, COMMAND))
|
||||
|
||||
|
||||
def works_paused(a):
|
||||
@@ -308,12 +267,12 @@ def works_paused(a):
|
||||
# What pressed pause_toggle, for the log (do_action's source).
|
||||
PAUSE_SOURCES = {"mouse": "mouse button", "keyboard": "key combination", "left": "controller button",
|
||||
"right": "controller button"}
|
||||
KEYS = "\0frametop_keys" # keys of keyboards grabbed for the desktop, for other readers
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
FT_LAYOUT = os.path.join(REPO, "layout", "ft-layout")
|
||||
FT_STEAM = os.path.join(REPO, "steam", "ft-steam")
|
||||
# Commands ("command:CMD") find Frametop's own tools (ft-layout, ft-float, ft-steam, ft-voice) on
|
||||
# their PATH.
|
||||
COMMAND_PATH = ":".join([os.path.join(REPO, d) for d in ("layout", "float", "steam", "voice")]
|
||||
# Commands ("command:CMD") find Frametop's own tools (ft-layout, ft-float, ft-steam) on their PATH.
|
||||
COMMAND_PATH = ":".join([os.path.join(REPO, d) for d in ("layout", "float", "steam")]
|
||||
+ [os.environ.get("PATH", "/usr/local/bin:/usr/bin")])
|
||||
DEFAULT_BUTTONS = {BTN_LEFT: "left", BTN_RIGHT: "right", BTN_MIDDLE: "middle",
|
||||
BTN_SIDE: "back", BTN_EXTRA: "back"}
|
||||
@@ -407,11 +366,6 @@ def read_config(path=os.path.expanduser("~/.config/frametop.conf")):
|
||||
return conf
|
||||
|
||||
|
||||
def default_key_bindings():
|
||||
"""The key combinations a rules file without its own gets."""
|
||||
return {**DEFAULT_KEY_BINDINGS, **(VOICE_KEY_BINDINGS if os.path.exists(VOICE_UNIT) else {})}
|
||||
|
||||
|
||||
def read_rules(path=RULES_PATH):
|
||||
"""{"devices": {id: {"role", "name"}}, "buttons": {id: {"<code>": action}},
|
||||
"controller_buttons": {"<hand>/<button>": action}, "controller_in_games": bool,
|
||||
@@ -426,7 +380,7 @@ def read_rules(path=RULES_PATH):
|
||||
rules.setdefault("buttons", {})
|
||||
rules.setdefault("controller_buttons", {})
|
||||
if not isinstance(rules.get("key_bindings"), dict):
|
||||
rules["key_bindings"] = default_key_bindings()
|
||||
rules["key_bindings"] = dict(DEFAULT_KEY_BINDINGS)
|
||||
return rules
|
||||
|
||||
|
||||
@@ -811,7 +765,7 @@ class Node:
|
||||
def __init__(self, path, fd, name, bus, vendor, product, uniq, is_mouse, is_keyboard,
|
||||
candidate=True, volume_keys=False, only_volume=False, uinput=False):
|
||||
self.path, self.fd, self.name = path, fd, name
|
||||
self.uinput = uinput # made by a program (a remote desktop's keyboard, say), not a real device
|
||||
self.uinput = uinput # made by a program (frame-voice's keyboard, say), not a real device
|
||||
self.bus, self.vendor, self.product, self.uniq = bus, vendor, product, uniq
|
||||
self.is_mouse, self.is_keyboard = is_mouse, is_keyboard
|
||||
self.candidate, self.volume_keys, self.only_volume = candidate, volume_keys, only_volume
|
||||
@@ -881,10 +835,6 @@ def main():
|
||||
control = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
control.bind("\0frametop_relay")
|
||||
control.setblocking(False)
|
||||
# The kernel adds each sender's user ("type" is taken only from ours: an abstract socket
|
||||
# takes datagrams from anyone).
|
||||
control.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1)
|
||||
creds = struct.Struct("3i") # struct ucred: pid, uid, gid
|
||||
watchers = {} # address -> watch end time
|
||||
|
||||
# desktop_until: typing goes to the Frametop desktop until then (ft-screens says so
|
||||
@@ -892,7 +842,7 @@ def main():
|
||||
# vr_capture_until: every controller button is taken until then (the settings app capturing one).
|
||||
# pointer: the 3D mouse while it's in use, pointer_conf: the one the config asks for (they
|
||||
# differ while Frametop is paused).
|
||||
state = {"pointer": None, "pointer_conf": None, "rules": {},
|
||||
state = {"pointer": None, "pointer_conf": None, "rules": {}, "share_keys": False,
|
||||
"desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0, "vr_bind_retry": False,
|
||||
"pointer_holding": False}
|
||||
|
||||
@@ -914,6 +864,7 @@ def main():
|
||||
def load_config():
|
||||
conf = read_config()
|
||||
state["rules"] = read_rules()
|
||||
state["share_keys"] = conf.get("SHARE_KEYS", "0") == "1"
|
||||
if conf.get("POINTER", "0") == "1":
|
||||
p = state["pointer_conf"] or Pointer(0.03, 30)
|
||||
p.sensitivity = float(conf.get("POINTER_SENSITIVITY", "0.03"))
|
||||
@@ -926,11 +877,6 @@ def main():
|
||||
log("pointer mode off: pointer devices feed the virtual mouse and keyboard")
|
||||
state["pointer"] = None if pause.paused else state["pointer_conf"]
|
||||
pause.configure(state["rules"])
|
||||
# Hand interaction: the pointer helper's handoff arbiter needs to know who holds a controller.
|
||||
# HANDS_MODE=always keeps hand tracking running through games.
|
||||
# Direct touch alone (HANDS_NEAR=on) needs the same: the helper's handoff runs for it.
|
||||
pause.set_hands(conf.get("POINTER_HANDS", "0") == "1" or conf.get("HANDS_NEAR") == "on",
|
||||
conf.get("HANDS_MODE") == "always")
|
||||
|
||||
load_config()
|
||||
tap = None # the modifier (folded, MODIFIERS) pressed alone, with nothing since: its release is a tap
|
||||
@@ -1046,27 +992,10 @@ def main():
|
||||
except OSError:
|
||||
pass # the Frametop desktop isn't running
|
||||
log(action)
|
||||
elif action == "spotwatch":
|
||||
if value == 1:
|
||||
try:
|
||||
screens_sock.sendto(f"toggle {source}".encode(), SPOTWATCH)
|
||||
except OSError:
|
||||
log("spotwatch: not running (the Frametop desktop starts it)")
|
||||
log(action)
|
||||
elif action in VOICE_ACTIONS:
|
||||
# dictate starts or stops on a press; dictate_hold starts on the press and stops on
|
||||
# the release. The voice service logs it.
|
||||
command = {1: "toggle"}.get(value) if action == "dictate" else {1: "start", 0: "stop"}.get(value)
|
||||
if command:
|
||||
try:
|
||||
screens_sock.sendto(f"{command} {source}".encode(), VOICE)
|
||||
except OSError:
|
||||
log(f"{action}: Frametop Voice isn't running (frametop-voice.service)")
|
||||
elif state["pointer"]:
|
||||
state["pointer"].action(action, value, now, source)
|
||||
|
||||
held_modifiers = set() # on any keyboard, folded (MODIFIERS)
|
||||
tap_code = None # the tap's key as it is (a right-hand code isn't folded), see modifier_tap
|
||||
held_meta = set() # the Meta keys held, as they are (KEY_LEFTMETA, KEY_RIGHTMETA)
|
||||
meta_hidden = set() # held Meta keys the desktop was told came up (a combination; key_binding)
|
||||
combos_down = {} # key code -> the action its combination started (released with it)
|
||||
@@ -1074,12 +1003,11 @@ def main():
|
||||
def key_binding(code, value, now):
|
||||
"""A key from a keyboard: does it complete a key combination ("key_bindings")? True if
|
||||
it was taken for one (then it isn't typed)."""
|
||||
nonlocal tap, tap_code
|
||||
nonlocal tap
|
||||
if code in MODIFIERS:
|
||||
mod = MODIFIERS[code]
|
||||
if value == 1:
|
||||
tap = mod if not held_modifiers and not combos_down else None
|
||||
tap_code = code
|
||||
(held_modifiers.add if value else held_modifiers.discard)(mod)
|
||||
if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
|
||||
(held_meta.add if value else held_meta.discard)(code)
|
||||
@@ -1088,7 +1016,7 @@ def main():
|
||||
return True # the desktop already had it come up (below)
|
||||
if value == 0 and tap == mod:
|
||||
tap = None
|
||||
modifier_tap(mod, tap_code, now)
|
||||
modifier_tap(mod, now)
|
||||
return False
|
||||
if value == 1 and code not in combos_down and meta_hidden:
|
||||
# Another key while Meta is still held after a combination: the desktop gets Meta
|
||||
@@ -1123,16 +1051,11 @@ def main():
|
||||
log(f"key combination {combo}: {action}")
|
||||
return True
|
||||
|
||||
def modifier_tap(mod, code, now):
|
||||
"""A modifier pressed and released alone: its binding, if it has one ("125": Meta tap).
|
||||
A right-hand key's own binding ("97": Right Ctrl) comes before the folded one."""
|
||||
bindings = state["rules"]["key_bindings"]
|
||||
combo = str(code) if code != mod and str(code) in bindings else str(mod)
|
||||
action = bindings.get(combo)
|
||||
def modifier_tap(mod, now):
|
||||
"""A modifier pressed and released alone: its binding, if it has one ("125": Meta tap)."""
|
||||
action = state["rules"]["key_bindings"].get(str(mod))
|
||||
if not known_action(action) or action in ("key", "none") or (pause.paused and not works_paused(action)):
|
||||
return
|
||||
if action == "dictate_hold":
|
||||
action = "dictate" # a tap is over before it's held: it starts, and the next one stops
|
||||
# The desktop gets a key in between before the release goes there, so the tap isn't one
|
||||
# there too: a Meta tap would open Plasma's launcher.
|
||||
to_screens(KEY_F24, 1)
|
||||
@@ -1140,7 +1063,7 @@ def main():
|
||||
if state["pointer"] or not needs_pointer(action):
|
||||
do_action(action, 1, now, "keyboard")
|
||||
do_action(action, 0, now, "keyboard")
|
||||
log(f"key combination {combo}: {action}")
|
||||
log(f"key combination {mod}: {action}")
|
||||
|
||||
|
||||
screens_down = set() # keys the desktop was told went down and not yet up (see reconcile_desktop_keys)
|
||||
@@ -1166,51 +1089,19 @@ def main():
|
||||
next_scan = 0.0
|
||||
|
||||
volume = Volume()
|
||||
typing = collections.deque() # (code, shift): dictation's characters still to type
|
||||
typing_down = [] # the character pressed last, (code, shift, to the desktop): its release is next
|
||||
typing_at = 0.0 # the next press or release goes then
|
||||
keys_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
|
||||
|
||||
def type_text(text):
|
||||
"""Queue dictation's text (see the top)."""
|
||||
keys = [US_KEYS[c] for c in text if c in US_KEYS]
|
||||
room = max(0, TYPE_MAX - len(typing))
|
||||
typing.extend(keys[:room])
|
||||
log(f"typing {min(len(keys), room)} characters for Frametop Voice"
|
||||
+ (f" ({len(keys) - room} dropped: too many waiting)" if len(keys) > room else ""))
|
||||
|
||||
def type_key(code, value, desktop):
|
||||
if desktop:
|
||||
to_screens(code, value)
|
||||
else:
|
||||
keyboard.emit(EV_KEY, code, value)
|
||||
keyboard.sync()
|
||||
|
||||
def typing_tick(now):
|
||||
"""Release the character pressed last, or press the next, when it's time. A new one
|
||||
waits while a modifier is held."""
|
||||
nonlocal typing_at
|
||||
if now < typing_at:
|
||||
def share_key(node, code, value):
|
||||
"""With SHARE_KEYS=1, a key from a keyboard grabbed for the desktop, for programs
|
||||
that watch every keyboard for a hotkey and lose it to the grab. It can't go on
|
||||
another input device: gamescope reads every keyboard itself and would type it
|
||||
into its focused app."""
|
||||
if not state["share_keys"]:
|
||||
return
|
||||
if typing_down:
|
||||
code, shift, desktop = typing_down.pop()
|
||||
type_key(code, 0, desktop)
|
||||
if shift:
|
||||
type_key(KEY_LEFTSHIFT, 0, desktop)
|
||||
elif typing and not held_modifiers:
|
||||
code, shift = typing.popleft()
|
||||
desktop = now < state["desktop_until"]
|
||||
if shift:
|
||||
type_key(KEY_LEFTSHIFT, 1, desktop)
|
||||
type_key(code, 1, desktop)
|
||||
typing_down.append((code, shift, desktop))
|
||||
else:
|
||||
return
|
||||
typing_at = now + TYPE_STEP
|
||||
|
||||
def typing_timeout(now, default):
|
||||
if typing_down or (typing and not held_modifiers):
|
||||
return max(0.0, min(default, typing_at - now))
|
||||
return default
|
||||
try:
|
||||
keys_sock.sendto(f"key {code} {value} {node.name}".encode(), KEYS)
|
||||
except OSError:
|
||||
pass # nobody listening
|
||||
|
||||
def restore_keymaps():
|
||||
"""Give remapped devices their volume keys back, so they work without the relay."""
|
||||
@@ -1263,7 +1154,10 @@ def main():
|
||||
|
||||
def release_held(node):
|
||||
for code in node.held:
|
||||
(mouse if code >= BTN_MISC else keyboard).emit(EV_KEY, code, 0)
|
||||
if node.role == "passthrough":
|
||||
share_key(node, code, 0)
|
||||
else:
|
||||
(mouse if code >= BTN_MISC else keyboard).emit(EV_KEY, code, 0)
|
||||
node.held.clear()
|
||||
mouse.sync()
|
||||
keyboard.sync()
|
||||
@@ -1355,11 +1249,9 @@ def main():
|
||||
def handle_control(now):
|
||||
while True:
|
||||
try:
|
||||
data, anc, _, addr = control.recvmsg(65536, socket.CMSG_SPACE(creds.size))
|
||||
data, addr = control.recvfrom(4096)
|
||||
except BlockingIOError:
|
||||
return
|
||||
uid = next((creds.unpack(d[:creds.size])[1] for level, kind, d in anc
|
||||
if level == socket.SOL_SOCKET and kind == socket.SCM_CREDENTIALS), None)
|
||||
words = data.decode(errors="replace").split()
|
||||
cmd = words[0] if words else ""
|
||||
# One malformed datagram must not end the relay: it would drop every grab,
|
||||
@@ -1381,9 +1273,6 @@ def main():
|
||||
if cmd == "vrgame" and len(words) == 2:
|
||||
pause.game_state(words[1] == "1", now)
|
||||
continue
|
||||
if cmd == "vrdash" and len(words) == 2:
|
||||
pause.dashboard_state(words[1] == "1")
|
||||
continue
|
||||
if cmd == "pause" and len(words) >= 2 and words[1] in ("on", "off", "toggle", "?"):
|
||||
# From input/ft-pause, the settings app, or the gesture reader (unbound, no reply).
|
||||
reason = words[2] if len(words) > 2 else "command"
|
||||
@@ -1397,13 +1286,6 @@ def main():
|
||||
if cmd == "textfield" and len(words) == 2:
|
||||
text_field(words[1] == "1")
|
||||
continue
|
||||
if cmd == "type":
|
||||
# From Frametop Voice (unbound, no reply). It types keys, so only from this user.
|
||||
if uid == os.getuid():
|
||||
type_text(data[len(b"type "):].decode(errors="replace"))
|
||||
else:
|
||||
log(f"type from another user ({uid}): ignored")
|
||||
continue
|
||||
if cmd == "gazeawake" and len(words) == 2:
|
||||
if state["pointer"]:
|
||||
state["pointer"].gaze_awake_until = now + 12.0 if words[1] == "1" else 0.0
|
||||
@@ -1504,8 +1386,7 @@ def main():
|
||||
state["pointer_holding"] = holding
|
||||
pause.controllers_changed()
|
||||
ready, _, _ = select.select(list(nodes) + [control], [], [],
|
||||
typing_timeout(now, min(volume.timeout(now, pointer.timeout() if pointer else 0.5),
|
||||
pause.timeout(now))))
|
||||
min(volume.timeout(now, pointer.timeout() if pointer else 0.5), pause.timeout(now)))
|
||||
now = time.monotonic()
|
||||
if pointer:
|
||||
pointer.tick(now)
|
||||
@@ -1514,7 +1395,6 @@ def main():
|
||||
if state["vr_bind_retry"]:
|
||||
vr_bind(now)
|
||||
volume.tick(now)
|
||||
typing_tick(now)
|
||||
pause.tick(now)
|
||||
if state["vr_capture_until"] and now >= state["vr_capture_until"]:
|
||||
vr_bind(now) # capture over: back to the mapped buttons
|
||||
@@ -1559,6 +1439,16 @@ def main():
|
||||
if node.role != "pointer":
|
||||
# Observed only, unless typing goes to the desktop. Key combinations work on
|
||||
# any pass-through keyboard.
|
||||
if (node.role == "passthrough" and etype == EV_KEY and node.grabbed
|
||||
and code < BTN_MISC and value in (0, 1)):
|
||||
# Shared as pressed, key combinations included: the Meta release a
|
||||
# combination keeps from the desktop must still reach frame-voice, or
|
||||
# it waits for that release before typing anything.
|
||||
share_key(node, code, value)
|
||||
if value:
|
||||
node.held.add(code)
|
||||
else:
|
||||
node.held.discard(code)
|
||||
if node.role == "passthrough" and etype == EV_KEY and code < BTN_MISC and key_binding(code, value, now):
|
||||
continue
|
||||
if node.role == "passthrough" and etype == EV_KEY:
|
||||
|
||||
+8
-130
@@ -1,7 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline test of the input relay's key combinations and modifier taps, of which keys and
|
||||
mouse buttons reach the desktop (ft-screens), and of dictation (Frametop Voice): the keys that
|
||||
start and stop it, and the text it has the relay type.
|
||||
"""Offline test of the input relay's key combinations and modifier taps, and of which keys and
|
||||
mouse buttons reach the desktop (ft-screens).
|
||||
|
||||
Runs the relay's main() against a fake pass-through keyboard, a fake mouse, and two fake nodes
|
||||
with volume keys, a keyboard's Consumer Control node and the headset's gpio-keys (pipes), with
|
||||
@@ -59,13 +58,13 @@ class StubPause:
|
||||
def configure(self, *args):
|
||||
pass
|
||||
|
||||
game_state = tick = helper_started = controllers_changed = set_hands = dashboard_state = configure
|
||||
game_state = tick = helper_started = controllers_changed = configure
|
||||
|
||||
|
||||
stub = {}
|
||||
relay.game_pause.GamePause = StubPause
|
||||
|
||||
for name in ("SCREENS", "HELPER", "FLOAT", "GAZED", "VOICE", "SPOTWATCH"):
|
||||
for name in ("SCREENS", "HELPER", "FLOAT", "GAZED", "KEYS"):
|
||||
setattr(relay, name, f"\0{tag}_{name.lower()}")
|
||||
OUT = tempfile.mkdtemp(prefix="ft-keys-test-")
|
||||
steam_log, cmd_log = f"{OUT}/steam.log", f"{OUT}/cmd.log"
|
||||
@@ -177,12 +176,6 @@ relay.probe = probe
|
||||
screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
screens.bind(relay.SCREENS)
|
||||
screens.settimeout(0.05)
|
||||
voice = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) # Frametop Voice's socket
|
||||
voice.bind(relay.VOICE)
|
||||
voice.settimeout(0.05)
|
||||
spotwatch = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) # spotwatch's socket (spotwatch/ft-spotwatch)
|
||||
spotwatch.bind(relay.SPOTWATCH)
|
||||
spotwatch.settimeout(0.05)
|
||||
EVENT = struct.Struct("llHHi")
|
||||
|
||||
|
||||
@@ -207,41 +200,6 @@ def typed():
|
||||
return got
|
||||
|
||||
|
||||
def spotwatched():
|
||||
"""What spotwatch was sent since the last call."""
|
||||
got = []
|
||||
while True:
|
||||
try:
|
||||
got.append(spotwatch.recv(256).decode())
|
||||
except socket.timeout:
|
||||
return got
|
||||
|
||||
|
||||
def dictation():
|
||||
"""What Frametop Voice was sent since the last call."""
|
||||
got = []
|
||||
while True:
|
||||
try:
|
||||
got.append(voice.recv(256).decode())
|
||||
except socket.timeout:
|
||||
return got
|
||||
|
||||
|
||||
def to_relay(text):
|
||||
c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
c.sendto(text.encode(), f"\0{tag}_relay")
|
||||
c.close()
|
||||
|
||||
|
||||
def keys_for(text):
|
||||
"""The presses and releases that type text, as (code, value), Shift included."""
|
||||
out = []
|
||||
for ch in text:
|
||||
code, shift = relay.US_KEYS[ch]
|
||||
out += ([(42, 1)] if shift else []) + [(code, 1), (code, 0)] + ([(42, 0)] if shift else [])
|
||||
return out
|
||||
|
||||
|
||||
def emitted():
|
||||
"""What the virtual mouse and keyboard were sent since the last call."""
|
||||
got = []
|
||||
@@ -289,7 +247,7 @@ def check(label, got, want):
|
||||
failures.append(label)
|
||||
|
||||
|
||||
META, RMETA, SHIFT, CTRL, RCTRL, ALT, A, C, F, J, P, TAB = 125, 126, 42, 29, 97, 56, 30, 46, 33, 36, 25, 15
|
||||
META, RMETA, SHIFT, CTRL, RCTRL, ALT, A, F, J, P, TAB = 125, 126, 42, 29, 97, 56, 30, 33, 36, 25, 15
|
||||
F24 = ["key 194 1", "key 194 0"]
|
||||
|
||||
|
||||
@@ -360,9 +318,9 @@ def tests():
|
||||
check("Meta tap bound to none: nothing", lines(steam_log), ["menu", "menu"])
|
||||
check("Meta tap bound to none: no F24", typed(), ["key 125 1", "key 125 0"])
|
||||
|
||||
use({"29": f"command:command -v ft-steam ft-layout ft-float ft-voice | wc -l >> {cmd_log}"})
|
||||
use({"29": f"command:command -v ft-steam ft-layout ft-float | wc -l >> {cmd_log}"})
|
||||
key(RCTRL, 1); key(RCTRL, 0)
|
||||
check("right Ctrl tap runs its command, which finds Frametop's tools", lines(cmd_log), ["combo", "4"])
|
||||
check("right Ctrl tap runs its command, which finds Frametop's tools", lines(cmd_log), ["combo", "3"])
|
||||
check("Ctrl tap: F24 before Ctrl's release", typed(), ["key 97 1"] + F24 + ["key 97 0"])
|
||||
|
||||
use({"125+36": "gaze_left", "29": f"command:echo paused >> {cmd_log}", "42+125+25": "pause_toggle"})
|
||||
@@ -372,7 +330,7 @@ def tests():
|
||||
check("paused: Meta+J (gaze click) does nothing and is typed as is", typed(),
|
||||
["key 125 1", "key 36 1", "key 36 0", "key 125 0"])
|
||||
key(RCTRL, 1); key(RCTRL, 0)
|
||||
check("paused: a command still runs", lines(cmd_log), ["combo", "4", "paused"])
|
||||
check("paused: a command still runs", lines(cmd_log), ["combo", "3", "paused"])
|
||||
typed()
|
||||
key(SHIFT, 1); key(META, 1); key(P, 1); key(P, 0); key(META, 0); key(SHIFT, 0)
|
||||
check("paused: Meta+Shift+P (pause_toggle) resumes", stub["pause"].paused, False)
|
||||
@@ -455,86 +413,6 @@ def tests():
|
||||
check("spinning works without pointer mode", (relay.needs_pointer("spin_next"), relay.needs_pointer("spin_prev")),
|
||||
(False, False))
|
||||
|
||||
# Meta+Space (a default): spotwatch's search opens or closes, told it came from a keyboard.
|
||||
SPACE = 57
|
||||
key(META, 1); key(SPACE, 1); key(SPACE, 0); key(META, 0)
|
||||
check("Meta+Space (default): spotwatch toggles", spotwatched(), ["toggle keyboard"])
|
||||
check("...Space isn't typed, and the desktop gets F24 and Meta's release first", typed(),
|
||||
["key 125 1"] + F24 + ["key 125 0"])
|
||||
pause("on")
|
||||
key(META, 1); key(SPACE, 1); key(SPACE, 0); key(META, 0)
|
||||
check("paused: Meta+Space doesn't open spotwatch", spotwatched(), [])
|
||||
pause("off")
|
||||
typed()
|
||||
check("spotwatch works without pointer mode", relay.needs_pointer("spotwatch"), False)
|
||||
|
||||
# Dictation: Right Ctrl alone (the default with Frametop Voice installed) starts and stops it.
|
||||
use({"97": "dictate"})
|
||||
dictation()
|
||||
key(RCTRL, 1); key(RCTRL, 0)
|
||||
check("Right Ctrl tap (dictate): Frametop Voice toggles", dictation(), ["toggle keyboard"])
|
||||
check("...and the desktop gets F24 before Ctrl's release", typed(), ["key 97 1"] + F24 + ["key 97 0"])
|
||||
key(CTRL, 1); key(CTRL, 0)
|
||||
check("a Left Ctrl tap isn't Right Ctrl's", dictation(), [])
|
||||
key(SHIFT, 1); key(RCTRL, 1); key(RCTRL, 0); key(SHIFT, 0)
|
||||
check("Shift held, then a Right Ctrl tap: not dictation", dictation(), [])
|
||||
key(RCTRL, 1); key(SHIFT, 1); key(SHIFT, 0); key(RCTRL, 0)
|
||||
check("Right Ctrl held, Shift pressed and released: not dictation", dictation(), [])
|
||||
key(RCTRL, 1); key(C, 1); key(C, 0); key(RCTRL, 0)
|
||||
check("Right Ctrl+C: not dictation", dictation(), [])
|
||||
key(RCTRL, 1); key(relay.BTN_LEFT, 1, ms_w); key(relay.BTN_LEFT, 0, ms_w); key(RCTRL, 0)
|
||||
check("Right Ctrl+click: not dictation", dictation(), [])
|
||||
use({"29": "dictate"})
|
||||
key(CTRL, 1); key(CTRL, 0); key(RCTRL, 1); key(RCTRL, 0)
|
||||
check("a Ctrl tap bound by its left code: either side", dictation(), ["toggle keyboard", "toggle keyboard"])
|
||||
use({"97": "dictate_hold"})
|
||||
key(RCTRL, 1); key(RCTRL, 0)
|
||||
check("a tap bound to dictate_hold toggles (a tap is over before it's held)", dictation(), ["toggle keyboard"])
|
||||
use({"29+42+33": "dictate_hold"})
|
||||
key(CTRL, 1); key(SHIFT, 1); key(F, 1)
|
||||
check("Ctrl+Shift+F held (dictate_hold): starts", dictation(), ["start keyboard"])
|
||||
key(F, 0); key(SHIFT, 0); key(CTRL, 0)
|
||||
check("...and its release stops", dictation(), ["stop keyboard"])
|
||||
use({"97": "dictate"})
|
||||
pause("on")
|
||||
key(RCTRL, 1); key(RCTRL, 0)
|
||||
check("paused: dictation still toggles", dictation(), ["toggle keyboard"])
|
||||
pause("off")
|
||||
check("dictation works without pointer mode",
|
||||
(relay.needs_pointer("dictate"), relay.needs_pointer("dictate_hold")), (False, False))
|
||||
unit = os.path.join(OUT, "frametop-voice.service")
|
||||
relay.VOICE_UNIT = unit
|
||||
check("defaults without Frametop Voice: no dictation key", "97" in relay.default_key_bindings(), False)
|
||||
open(unit, "w").close()
|
||||
check("defaults with Frametop Voice installed: Right Ctrl tap", relay.default_key_bindings().get("97"), "dictate")
|
||||
|
||||
# Typing dictation's text: on the virtual keyboard while typing goes to Steam, to the desktop
|
||||
# while it goes there; held back while a modifier is held.
|
||||
use({})
|
||||
typed(), emitted()
|
||||
to_relay("type Hi, you!")
|
||||
time.sleep(0.3)
|
||||
check("type: to the virtual keyboard while typing goes to Steam", emitted(),
|
||||
[("keyboard", c, v) for c, v in keys_for("Hi, you!")])
|
||||
check("...and not to the desktop", typed(), [])
|
||||
to_relay("type caf\u00e9\tok")
|
||||
time.sleep(0.3)
|
||||
check("type: what a US layout can't type is dropped", emitted(), [("keyboard", c, v) for c, v in keys_for("cafok")])
|
||||
key(CTRL, 1)
|
||||
to_relay("type ab")
|
||||
time.sleep(0.3)
|
||||
check("type: waits while Ctrl is held", emitted(), [])
|
||||
key(CTRL, 0)
|
||||
time.sleep(0.3)
|
||||
check("...and types once it's released", emitted(), [("keyboard", c, v) for c, v in keys_for("ab")])
|
||||
typed()
|
||||
to_relay("keyboard desktop")
|
||||
to_relay("type Ok.")
|
||||
time.sleep(0.3)
|
||||
check("type: to the desktop while typing goes there", typed(), [f"key {c} {v}" for c, v in keys_for("Ok.")])
|
||||
check("...and not to the virtual keyboard", emitted(), [])
|
||||
to_relay("keyboard steam")
|
||||
|
||||
check("an empty command isn't an action", relay.known_action("command: "), False)
|
||||
check("steam_menu, pause_toggle and commands work without pointer mode",
|
||||
(relay.needs_pointer("steam_menu"), relay.needs_pointer("pause_toggle"), relay.needs_pointer("command:ls")),
|
||||
|
||||
@@ -8,7 +8,6 @@ renamed, so nothing reaches the live Frametop, SteamVR, or the running relay.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import shutil
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
||||
@@ -116,19 +115,10 @@ class StubWatch:
|
||||
def set_gesture(self, spec):
|
||||
self.spec = spec
|
||||
|
||||
def set_touch_feed(self, on):
|
||||
self.touch = on
|
||||
|
||||
def helper_started(self):
|
||||
pass
|
||||
|
||||
|
||||
tmp = tempfile.mkdtemp(prefix="ft-pause-test-")
|
||||
gp.STATE_PATH = os.path.join(tmp, "pause.json")
|
||||
gp.SCREENS = f"\0ft_pause_test_{os.getpid()}_screens"
|
||||
gp.HELPER = f"\0ft_pause_test_{os.getpid()}_helper" # "paused 1|0" mustn't reach the live pointer helper
|
||||
os.makedirs(os.path.join(tmp, "frametop-hands"))
|
||||
gp.PARK_PATH = os.path.join(tmp, "frametop-hands", "park")
|
||||
gp.Worker, gp.ControllerWatch = StubWorker, StubWatch
|
||||
changes = []
|
||||
|
||||
@@ -208,34 +198,6 @@ p.toggle("gesture", 0)
|
||||
q = gp.GamePause(lambda *a: None, changes.append, VR)
|
||||
check("a relay restarted while paused stays paused", (q.paused, q.reason), (True, "gesture"))
|
||||
check("and stops the services again", q.worker.jobs, ["enforce"])
|
||||
|
||||
# ---------------------------------------------------------------- hand tracking parked for games
|
||||
parked = lambda: os.path.exists(gp.PARK_PATH) # noqa: E731
|
||||
p = new()
|
||||
p.set_hands(True)
|
||||
check("hand interaction on: the worker leaves hand tracking running", p.worker.keep, gp.HAND_UNITS)
|
||||
check("not paused: not parked", parked(), False)
|
||||
p.game_state(True, 0)
|
||||
check("a game pauses Frametop: hand tracking parked", parked(), True)
|
||||
p.dashboard_state(True)
|
||||
check("the dashboard opens: unparked", parked(), False)
|
||||
p.dashboard_state(False)
|
||||
check("it closes: parked again", parked(), True)
|
||||
p.toggle("gesture", 1)
|
||||
check("Frametop toggled back on during the game: unparked", parked(), False)
|
||||
p.game_state(False, 10)
|
||||
p.game_state(True, 20)
|
||||
check("the next game: parked", parked(), True)
|
||||
p.set_hands(False)
|
||||
check("hand interaction off: unparked (the worker stops it as before)", (parked(), p.worker.keep), (False, ()))
|
||||
p.set_hands(True, always=True)
|
||||
check("HANDS_MODE=always: the worker leaves hand tracking running", p.worker.keep, gp.HAND_UNITS)
|
||||
check("and it isn't parked in the game", parked(), False)
|
||||
p.set_hands(False, always=True)
|
||||
check("always without hand interaction: still left running", (parked(), p.worker.keep), (False, gp.HAND_UNITS))
|
||||
p.set_hands(True)
|
||||
check("back to the other modes: parked again", parked(), True)
|
||||
p.set_hands(False)
|
||||
q.toggle("gesture", 1)
|
||||
r = gp.GamePause(lambda *a: None, changes.append, VR)
|
||||
check("resumed: the next relay starts running", r.paused, False)
|
||||
@@ -246,5 +208,7 @@ check("status", sorted(p.status()), sorted(["t", "paused", "reason", "since", "e
|
||||
"stopped", "gesture_reader", "auto", "desktop"]))
|
||||
|
||||
print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
|
||||
shutil.rmtree(tmp)
|
||||
for f in os.listdir(tmp):
|
||||
os.remove(os.path.join(tmp, f))
|
||||
os.rmdir(tmp)
|
||||
sys.exit(1 if failures else 0)
|
||||
@@ -1,60 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline test of the controller touch feed for the pointer helper's handoff arbiter
|
||||
(input/game_pause.py TouchTracker): which "ctl ..." datagrams go out for made-up web socket
|
||||
messages. Nothing connects to SteamVR or sends anything.
|
||||
|
||||
input/test/touch-test.py
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
||||
import game_pause as gp # noqa: E402
|
||||
|
||||
failures = []
|
||||
|
||||
|
||||
def check(label, got, want):
|
||||
ok = got == want
|
||||
print(("ok " if ok else "FAIL ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
|
||||
if not ok:
|
||||
failures.append(label)
|
||||
|
||||
|
||||
LEFT, RIGHT, BENCH = "/user/hand/left", "/user/hand/right", "/devices/cv/frame_testbench_right"
|
||||
FULL = {"/input/trigger/click": False, "/input/trigger/touch": False, "/input/thumbrest/touch": False,
|
||||
"/input/handgrip1/touch": False, "/input/grip/value": 0}
|
||||
|
||||
t = gp.TouchTracker()
|
||||
check("first lookup: nothing yet", t.set_sides({LEFT: "left", RIGHT: "right"}), [])
|
||||
check("the first (full) message: the touch state", t.feed(LEFT, FULL), ["ctl left touch 0"])
|
||||
check("a held click in the first message isn't a press",
|
||||
t.feed(RIGHT, dict(FULL, **{"/input/trigger/click": True})), ["ctl right touch 0"])
|
||||
check("thumbrest touched", t.feed(LEFT, {"/input/thumbrest/touch": True}), ["ctl left touch 1"])
|
||||
check("a second sensor: no change to send", t.feed(LEFT, {"/input/handgrip1/touch": True}), [])
|
||||
check("one let go, one still touched: nothing", t.feed(LEFT, {"/input/thumbrest/touch": False}), [])
|
||||
check("both let go", t.feed(LEFT, {"/input/handgrip1/touch": False}), ["ctl left touch 0"])
|
||||
check("a value change alone: nothing", t.feed(LEFT, {"/input/grip/value": 0.3}), [])
|
||||
check("a click going down: press (and its touch)",
|
||||
t.feed(LEFT, {"/input/trigger/click": True, "/input/trigger/touch": True}), ["ctl left press", "ctl left touch 1"])
|
||||
check("still down: no second press", t.feed(LEFT, {"/input/trigger/click": True}), [])
|
||||
check("up, then down again: press", [t.feed(LEFT, {"/input/trigger/click": False}),
|
||||
t.feed(LEFT, {"/input/trigger/click": True})], [[], ["ctl left press"]])
|
||||
check("an unknown device: nothing", t.feed("/devices/cv/other", {"/input/a/touch": True}), [])
|
||||
# a second controller on the right (the testbench's), touched while the real one isn't
|
||||
check("another right controller appears", t.set_sides({LEFT: "left", RIGHT: "right", BENCH: "right"}), [])
|
||||
check("its first message", t.feed(BENCH, FULL), [])
|
||||
check("it's touched: right touch 1", t.feed(BENCH, {"/input/thumbrest/touch": True}), ["ctl right touch 1"])
|
||||
check("snapshot", t.snapshot(), ["ctl left touch 1", "ctl right touch 1"])
|
||||
check("it goes: right still has one, nothing gone; its touch went with it",
|
||||
t.set_sides({LEFT: "left", RIGHT: "right"}), ["ctl right touch 0"])
|
||||
check("the next right message: nothing new", t.feed(RIGHT, {"/input/grip/value": 0.1}), [])
|
||||
check("the left controller goes: gone", t.set_sides({RIGHT: "right"}), ["ctl left gone"])
|
||||
check("it comes back: its first message is a baseline",
|
||||
[t.set_sides({LEFT: "left", RIGHT: "right"}), t.feed(LEFT, dict(FULL, **{"/input/trigger/click": True}))],
|
||||
[[], ["ctl left touch 0"]])
|
||||
|
||||
if failures:
|
||||
print(f"{len(failures)} failed")
|
||||
sys.exit(1)
|
||||
print("all passed")
|
||||
+7
-18
@@ -6,10 +6,9 @@ follows the controllers' raw input through a web socket there (mailbox input_ser
|
||||
request_input_state_updates). Reading it takes nothing from anyone: whatever the buttons are
|
||||
bound to still happens, and it works whatever app has input focus, VR games included. SteamVR
|
||||
input only reaches the focused app, or an overlay with global input, which then takes the button
|
||||
from the game (pointer/helper/vrbuttons.h). Frametop Voice (voice/ft-voiced) reads its
|
||||
push-to-talk button the same way. It's undocumented, so a SteamVR update could change it
|
||||
(scripts/update-check.py checks it). The host's Python has no `websockets` module, so this is a
|
||||
small RFC 6455 client of its own.
|
||||
from the game (pointer/helper/vrbuttons.h). frame-voice reads its push-to-talk button the same
|
||||
way. It's undocumented, so a SteamVR update could change it (scripts/update-check.py checks it).
|
||||
The host's Python has no `websockets` module, so this is a small RFC 6455 client of its own.
|
||||
|
||||
controllers() the Frame controllers SteamVR has now: {root path: "left" | "right"}. A
|
||||
controller's root path is /user/hand/<side>, or /devices/cv/<serial> while the
|
||||
@@ -19,11 +18,9 @@ small RFC 6455 client of its own.
|
||||
"components": {"/input/thumbstick/click": true, ...}} with the components
|
||||
that changed (mostly capacitive sensing, about 160 a second)
|
||||
|
||||
Run as a program, it prints the controllers' button changes for a while (default 10 s), or with
|
||||
--touch, every touch and value component too (probe P3 of the hand interaction design: which
|
||||
components say a controller is held, and how soon they let go):
|
||||
Run as a program, it prints the controllers' button changes for a while (default 10 s):
|
||||
|
||||
input/vrws.py [--touch] [seconds]
|
||||
input/vrws.py [seconds]
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
@@ -169,10 +166,7 @@ class VrSocket:
|
||||
|
||||
|
||||
def main():
|
||||
args = sys.argv[1:]
|
||||
touch = "--touch" in args
|
||||
args = [a for a in args if a != "--touch"]
|
||||
seconds = float(args[0]) if args else 10.0
|
||||
seconds = float(sys.argv[1]) if len(sys.argv) > 1 else 10.0
|
||||
ws = VrSocket()
|
||||
ws.open(f"frametop_vrws_{os.getpid()}")
|
||||
sides = controllers()
|
||||
@@ -187,7 +181,7 @@ def main():
|
||||
while time.monotonic() - start < seconds:
|
||||
for text in ws.messages(timeout=0.5):
|
||||
count += 1
|
||||
if "/click" not in text and not (touch and ("/touch" in text or "/value" in text)):
|
||||
if "/click" not in text:
|
||||
continue
|
||||
msg = json.loads(text)
|
||||
side = sides.get(msg.get("device"), msg.get("device"))
|
||||
@@ -195,11 +189,6 @@ def main():
|
||||
if name.endswith("/click"):
|
||||
print(f"{time.monotonic() - start:7.3f} {side} {name[len('/input/'):-len('/click')]} "
|
||||
f"{'down' if value else 'up'}", flush=True)
|
||||
elif touch and name.endswith("/touch"):
|
||||
print(f"{time.monotonic() - start:7.3f} {side} {name[len('/input/'):-len('/touch')]} "
|
||||
f"{'touched' if value else 'let go'}", flush=True)
|
||||
elif touch and name.endswith("/value"):
|
||||
print(f"{time.monotonic() - start:7.3f} {side} {name} {value}", flush=True)
|
||||
finally:
|
||||
ws.close()
|
||||
print(f"{count} messages in {seconds:.0f} s")
|
||||
|
||||
+18
-32
@@ -1,16 +1,15 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install everything on the Steam Frame: the build container, Frametop (multi-screen
|
||||
# desktop, input relay, universal 3D mouse, settings app), and optionally voice dictation,
|
||||
# gaze mode, our own eye tracker for it, and the Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
|
||||
# desktop, input relay, universal 3D mouse, settings app), and optionally gaze mode, our own
|
||||
# eye tracker for it, and the Bluetooth fixes. Run it on the headset in a terminal, from this repo. It's safe to re-run,
|
||||
# for example after `git pull`. (Hand tracking, hands/, is deferred: it isn't offered here.)
|
||||
# (It also works from a PC over SSH; see "Developing from a PC" in the README.)
|
||||
# In a release (pack/install-release.sh), the programs come built from its image: this installs the
|
||||
# distrobox the release brings, makes the release's container from its image, and builds nothing.
|
||||
#
|
||||
# Usage: ./install.sh [--yes] [--no-voice] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
|
||||
# --yes don't ask; installs voice dictation, gaze mode, and our eye tracker if sudo can
|
||||
# run without a password prompt; skips the Bluetooth fixes and the SteamVR restart
|
||||
# --no-voice don't offer voice dictation (Frametop Voice: it downloads about 320 MB)
|
||||
# Usage: ./install.sh [--yes] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
|
||||
# --yes don't ask; installs gaze mode, and our eye tracker if sudo can run without
|
||||
# a password prompt; skips the Bluetooth fixes and the SteamVR restart
|
||||
# --no-eye-tracker don't offer our own eye tracker
|
||||
# --no-bluetooth don't offer the Bluetooth fixes
|
||||
# --bluetooth install the Bluetooth fixes without asking (with --yes: if sudo can run
|
||||
@@ -20,15 +19,14 @@ set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$root/scripts/_env.sh"
|
||||
|
||||
assume_yes=0 bluetooth=1 eye_tracker=1 voice=1
|
||||
assume_yes=0 bluetooth=1 eye_tracker=1
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
--yes) assume_yes=1 ;;
|
||||
--no-bluetooth) bluetooth=0 ;;
|
||||
--bluetooth) bluetooth=2 ;;
|
||||
--no-eye-tracker) eye_tracker=0 ;;
|
||||
--no-voice) voice=0 ;;
|
||||
-h|--help) sed -n '2,18p' "$0"; exit 0 ;;
|
||||
-h|--help) sed -n '2,17p' "$0"; exit 0 ;;
|
||||
*) echo "unknown option: $arg" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
@@ -70,7 +68,7 @@ if [ "$FRAME_LOCAL" = 0 ] || [ -n "${SSH_CONNECTION:-}" ]; then
|
||||
echo "that need SteamVR start with it if it isn't running now."
|
||||
fi
|
||||
|
||||
step "1/11 distrobox (container tool, installed in your home folder)"
|
||||
step "1/10 distrobox (container tool, installed in your home folder)"
|
||||
if on_frame 'test -x ~/.local/bin/distrobox'; then
|
||||
echo "already installed: $(on_frame '~/.local/bin/distrobox version | head -1')"
|
||||
elif [ "$FRAME_RELEASE" = 1 ]; then
|
||||
@@ -84,30 +82,29 @@ cd ~/dev/src/distrobox && ./install --prefix ~/.local'
|
||||
fi
|
||||
|
||||
if [ "$FRAME_RELEASE" = 1 ]; then
|
||||
step "2/11 Frametop's container (from this release's image)"
|
||||
step "2/10 Frametop's container (from this release's image)"
|
||||
"$root/pack/release-box.sh"
|
||||
else
|
||||
step "2/11 build container (Fedora 44 'dev', about 1-2 GB the first time)"
|
||||
step "2/10 build container (Fedora 44 'dev', about 1-2 GB the first time)"
|
||||
"$root/setup/dev-container.sh"
|
||||
fi
|
||||
|
||||
step "3/11 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
|
||||
step "3/10 input relay (keeps Bluetooth mice working in SteamVR, device roles, button maps)"
|
||||
"$root/desktops.sh" relay install
|
||||
|
||||
step "4/11 3D mouse: SteamVR driver"
|
||||
step "4/10 3D mouse: SteamVR driver"
|
||||
build "$root/pointer/driver/build.sh"
|
||||
"$root/pointer/driver/install.sh" install
|
||||
|
||||
step "5/11 3D mouse: pointer helper service, and the dashboard patch that lets it move SteamVR's windows"
|
||||
step "5/10 3D mouse: pointer helper service"
|
||||
build "$root/pointer/helper/build.sh"
|
||||
"$root/pointer/helper/run.sh" install
|
||||
"$root/pointer/dashpatch/run.sh" install
|
||||
|
||||
step "6/11 power service (turns the displays off while the headset isn't used, even on a stand)"
|
||||
step "6/10 power service (turns the displays off while the headset isn't used, even on a stand)"
|
||||
build "$root/power/build.sh"
|
||||
"$root/power/run.sh" install
|
||||
|
||||
step "7/11 multi-screen desktop (ft-screens), Frametop Input Settings, Display Settings, and Remote Displays"
|
||||
step "7/10 multi-screen desktop (ft-screens), Frametop Input Settings, Display Settings, and Remote Displays"
|
||||
build "$root/screens/build.sh"
|
||||
build "$root/stream/build.sh" # ft-stream: remote displays (Frametop Remote Displays)
|
||||
"$root/desktops.sh" install >/dev/null
|
||||
@@ -119,18 +116,7 @@ on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^PO
|
||||
echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
|
||||
"$root/scripts/conf-migrate.sh"
|
||||
|
||||
step "8/11 voice dictation (optional: speak, and it's typed where you type; nothing leaves the headset)"
|
||||
if [ "$voice" = 0 ]; then
|
||||
echo "skipped (--no-voice). Install later with: voice/run.sh install"
|
||||
elif ask "Install voice dictation? A Right Ctrl tap or holding the left d-pad down starts it; it downloads about 320 MB of speech models." y; then
|
||||
build "$root/voice/build.sh"
|
||||
"$root/voice/run.sh" install ||
|
||||
echo "voice dictation: not installed (above). Try again later with: voice/run.sh install"
|
||||
else
|
||||
echo "skipped. Install later with: voice/run.sh install"
|
||||
fi
|
||||
|
||||
step "9/11 gaze mode (optional, experimental: the pointer goes where you look)"
|
||||
step "8/10 gaze mode (optional, experimental: the pointer goes where you look)"
|
||||
gaze=0
|
||||
if ask "Install gaze mode? You turn it on and calibrate it in Frametop Input Settings, on the Gaze page." y; then
|
||||
"$root/gaze/run.sh" install
|
||||
@@ -141,7 +127,7 @@ fi
|
||||
|
||||
# Ours is gaze mode's default once it's installed (GAZE_TRACKER=auto). Without it, gaze mode
|
||||
# uses SteamVR's eye tracker.
|
||||
step "10/11 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
|
||||
step "9/10 our own eye tracker for gaze mode (recommended: more accurate than SteamVR's)"
|
||||
if [ "$gaze" = 0 ]; then
|
||||
echo "skipped: gaze mode isn't installed. Install it later with: gaze/tracker/install.sh"
|
||||
elif [ "$eye_tracker" = 0 ]; then
|
||||
@@ -169,7 +155,7 @@ else
|
||||
echo "skipped: gaze mode uses SteamVR's eye tracker. Install ours later with: gaze/tracker/install.sh"
|
||||
fi
|
||||
|
||||
step "11/11 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
|
||||
step "10/10 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
|
||||
if [ "$bluetooth" = 2 ] && [ "$assume_yes" = 1 ] && ! sudo_quiet; then
|
||||
echo "skipped: they need your password (sudo), and --yes doesn't ask. Install later with: setup/bluetooth/install.sh install"
|
||||
elif [ "$bluetooth" = 2 ] || { [ "$bluetooth" = 1 ] && ask "Install the Bluetooth fixes? They need your password (sudo)." n; }; then
|
||||
|
||||
@@ -13,14 +13,13 @@ test: test-python test-c test-bash
|
||||
# the Python suites as a strict gate: every failure fails the run. The image's
|
||||
# python3 has to import the dnf Qt stack and the locked packages together;
|
||||
# without that check the Qt tests would only skip. session/test is not here:
|
||||
# test_accessibility.py needs the host's AT-SPI and PyGObject, test_notifyfwd.py
|
||||
# the host's dbus-daemon and PyGObject.
|
||||
# test_accessibility.py needs the host's AT-SPI and PyGObject.
|
||||
test-python:
|
||||
#!/usr/bin/env bash
|
||||
set -uo pipefail
|
||||
python3 -c 'import PySide6, numpy, cv2, sherpa_onnx' || { echo "python3 can't import PySide6, numpy, cv2, and sherpa_onnx"; exit 1; }
|
||||
python3 -c 'import PySide6, numpy, cv2' || { echo "python3 can't import PySide6, numpy, and cv2"; exit 1; }
|
||||
status=0
|
||||
for t in input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py gaze/test/*.py pack/test/*.py voice/test/*.py spotwatch/test/*.py; do
|
||||
for t in input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py gaze/test/*.py pack/test/*.py; do
|
||||
[ -f "$t" ] || continue
|
||||
if python3 "$t" >/dev/null; then echo "$t: ok"; else echo "$t: FAILED"; status=1; fi
|
||||
done
|
||||
@@ -28,7 +27,7 @@ test-python:
|
||||
exit $status
|
||||
|
||||
# the C/C++ unit tests that need no model runtime: header-only logic with
|
||||
# made-up events (pointer/tests: the helper's hand interaction logic). sides_test.cpp is NOT here: it pulls in ncnn (hands'
|
||||
# made-up events. sides_test.cpp is NOT here: it pulls in ncnn (hands'
|
||||
# deferred heavy build) and stays with `make check` in the hands build
|
||||
# (hands/Makefile).
|
||||
test-c:
|
||||
@@ -41,18 +40,6 @@ test-c:
|
||||
gcc -std=c11 -Wall -Wextra -Werror screens/tests/relay-buttons-test.c \
|
||||
-o build/tests/relay-buttons-test
|
||||
build/tests/relay-buttons-test && echo "screens/tests/relay-buttons-test.c: ok"
|
||||
g++ -std=c++17 -O2 -Wall -Wextra -Wno-unused-parameter -Werror screens/tests/handinput_test.cpp \
|
||||
screens/handcut.cpp screens/handinput.cpp screens/palmmenu.cpp screens/grabthat.cpp screens/steampanels.cpp -o build/tests/handinput-test
|
||||
build/tests/handinput-test && echo "screens/tests/handinput_test.cpp: ok"
|
||||
g++ -std=c++17 -O2 -Wall -Wextra -Wno-unused-parameter -Werror screens/tests/handview_test.cpp \
|
||||
screens/handmesh.cpp screens/handskin.cpp screens/handrobot.cpp screens/handcut.cpp -o build/tests/handview-test
|
||||
build/tests/handview-test >/dev/null && echo "screens/tests/handview_test.cpp: ok"
|
||||
python3 -I screens/tools/hand-asset.py --check
|
||||
for t in pointer/tests/*-test.cpp; do
|
||||
n=$(basename "$t" .cpp)
|
||||
g++ -std=c++17 -Wall -Wextra -Werror -I. "$t" -o "build/tests/$n"
|
||||
"build/tests/$n" >/dev/null && echo "$t: ok"
|
||||
done
|
||||
|
||||
# shell scripts: syntax-gate everything that ships. No git: the image has
|
||||
# none, and a mounted checkout can trip "dubious ownership" — find lists
|
||||
|
||||
+25
-72
@@ -44,10 +44,7 @@ you face (yaw only), like a recenter. It lives in ~/.config/frametop-layout.json
|
||||
"displays": [{"id": "desk-pc-oled", (docs/remote-displays.md), each a panel
|
||||
"client": "frametop-1", streamed by ft-stream: its paired client,
|
||||
"app": "display:{GUID}", what it streams (display:DEVICE, monitor:
|
||||
a Vibepollo Remote Monitor, primary, or
|
||||
an app's name: Desktop for Sunshine),
|
||||
"port": 48089, its Sunshine instance's HTTP port, when
|
||||
not 47989 (a Mac runs one per display)
|
||||
a Vibepollo Remote Monitor, primary),
|
||||
"label": "OLED", "size": [5120, 1440], "fps": 60, "bitrate": 50000,
|
||||
"metres": 1.8, "curve": 0, "pin": ..., "hidden": true,
|
||||
"off": true, disconnected: no stream, no panel
|
||||
@@ -96,10 +93,8 @@ Usage (on the Frame host; Frametop Display Settings calls it too):
|
||||
ft-layout remote list the remote displays, their numbers and their streams' state
|
||||
ft-layout remote monitors ADDRESS the host's displays (Vibepollo's /api/display-devices, JSON)
|
||||
ft-layout remote add HOST ADDRESS APP [--size WxH] [--fps F] [--bitrate KBPS] [--label TEXT]
|
||||
[--port PORT] [--login]
|
||||
a display of a host (pairs its client with the host's token
|
||||
first, or with --login, with the Web UI login read from stdin
|
||||
as two lines: a Sunshine instance on PORT) -> its id
|
||||
first) -> its id
|
||||
ft-layout remote set ID size=WxH|fps=F|bitrate=KBPS|label=TEXT|app=APP|metres=M ...
|
||||
ft-layout remote host NAME route=auto|network|dongle direct=ADDRESS[,ADDRESS]|none
|
||||
how a host's streams reach it; its running streams start over
|
||||
@@ -141,7 +136,6 @@ DEFAULTS = {"auto": True, "mode": "preset",
|
||||
REMOTE_FIRST = 101 # ft-screens' remote screens (screens/remote.h)
|
||||
REMOTE_MAX = 16
|
||||
REMOTE_PIXELS_PER_METRE = 2400 # a new remote display's width in VR (5120 px: 2.1 m), at least 1 m
|
||||
GAMESTREAM_PORT = 47989 # a host's HTTP port, where ft-stream reaches it
|
||||
WEB_UI_PORT = 47990 # Vibepollo's Web UI: a host answers there when it's up
|
||||
STREAM = os.path.join(REPO, "stream", "build", "ft-stream")
|
||||
|
||||
@@ -734,12 +728,6 @@ def find_display(layout, display_id):
|
||||
raise RuntimeError(f"no remote display {display_id!r}")
|
||||
|
||||
|
||||
def stream_size(w, h):
|
||||
"""A stream's size in pixels, rounded down to even numbers: hosts' encoders (NVENC's 4:2:0)
|
||||
send no video at all for an odd width or height, and the stream reconnects forever."""
|
||||
return max(2, int(w) // 2 * 2), max(2, int(h) // 2 * 2)
|
||||
|
||||
|
||||
def remote_size(d):
|
||||
"""A remote display's size in VR (width, height) in metres."""
|
||||
w, h = d.get("size", [2560, 1440])
|
||||
@@ -784,19 +772,12 @@ def remotes_running(sock):
|
||||
return {int(e.split(":")[0]): e.split(":")[2] for e in fields}
|
||||
|
||||
|
||||
def display_target(host, d):
|
||||
"""Where ft-stream reaches a display: the host's address, with its instance's port when it
|
||||
has one of its own (HOST:PORT)."""
|
||||
port = d.get("port")
|
||||
return f"{host['address']}:{int(port)}" if port and int(port) != GAMESTREAM_PORT else host["address"]
|
||||
|
||||
|
||||
def start_remote(sock, n, host, d):
|
||||
"""Its stream (ft-screens starts it over only if the settings changed). ft-screens'
|
||||
reply: "ok restored" when its panel is back where it was before a disconnect."""
|
||||
w, h = stream_size(*d.get("size", [2560, 1440]))
|
||||
w, h = d.get("size", [2560, 1440])
|
||||
label = f"{host.get('name') or host['address']}: {d.get('label') or d['id']}"
|
||||
return sock.ask(f"remote {n} start {d['client']} {display_target(host, d)} {d['app']} {int(w)}x{int(h)} "
|
||||
return sock.ask(f"remote {n} start {d['client']} {host['address']} {d['app']} {int(w)}x{int(h)} "
|
||||
f"{int(d.get('fps', 60))} {int(d.get('bitrate', 0))} {remote_size(d)[0]:.3f} {label}")
|
||||
|
||||
|
||||
@@ -815,19 +796,17 @@ def place_remote(sock, n, d, t, eye, heading):
|
||||
|
||||
|
||||
def host_answers(host, timeout=1.5):
|
||||
"""Whether a host answers, at its address or its Steam Link dongle's (as its route allows):
|
||||
Vibepollo on its Web UI port, Sunshine instances on any of their displays' ports."""
|
||||
"""Whether a host's Vibepollo answers, at its address or its Steam Link dongle's (as its
|
||||
route allows)."""
|
||||
route = host.get("route", "auto")
|
||||
addresses = ([] if route == "dongle" else [host.get("address")]) + \
|
||||
([] if route == "network" else list(host.get("direct", [])))
|
||||
ports = sorted({int(d["port"]) for d in host.get("displays", []) if d.get("port")}) or [WEB_UI_PORT]
|
||||
for address in filter(None, addresses):
|
||||
for port in ports:
|
||||
try:
|
||||
with socket.create_connection((address, port), timeout=timeout):
|
||||
return True
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
with socket.create_connection((address, WEB_UI_PORT), timeout=timeout):
|
||||
return True
|
||||
except OSError:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
@@ -919,14 +898,14 @@ def in_container():
|
||||
return os.path.exists("/run/.containerenv")
|
||||
|
||||
|
||||
def run_stream(*args, timeout=60, stdin=None):
|
||||
def run_stream(*args, timeout=60):
|
||||
"""ft-stream, in the container ft-screens runs in: "dev" for a clone, the release's own for a
|
||||
release (scripts/in-box). A missing "dev" made distrobox ask whether to create it, and the
|
||||
question hung until the timeout. `stdin`: text for it to read (a --login)."""
|
||||
question hung until the timeout."""
|
||||
cmd = [STREAM, *args]
|
||||
if not in_container():
|
||||
cmd = [os.path.join(REPO, "scripts", "in-box"), *cmd]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, input=stdin)
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
|
||||
return r.returncode, r.stdout, r.stderr
|
||||
|
||||
|
||||
@@ -975,7 +954,7 @@ def remote_command(args):
|
||||
w, h = d.get("size", [2560, 1440])
|
||||
state = "disconnected" if d.get("off") else running.get(n, "off")
|
||||
state += f" route {host.get('route', 'auto')}"
|
||||
print(f"{n} {d['id']} {host.get('name')} {display_target(host, d)} {d['app']} {w}x{h}@{d.get('fps', 60)} "
|
||||
print(f"{n} {d['id']} {host.get('name')} {host['address']} {d['app']} {w}x{h}@{d.get('fps', 60)} "
|
||||
f"{state}{' hidden' if d.get('hidden') else ''} {d.get('label', '')}")
|
||||
return
|
||||
if what == "host" and len(args) >= 3:
|
||||
@@ -1048,20 +1027,12 @@ def remote_command(args):
|
||||
return
|
||||
if what == "add" and len(args) >= 4:
|
||||
host_name, address, app = args[1], args[2], args[3]
|
||||
rest = args[4:]
|
||||
login = "--login" in rest # the Web UI login on stdin: a Sunshine instance's pairing
|
||||
rest = [a for a in rest if a != "--login"]
|
||||
opts = dict(zip(rest[0::2], rest[1::2]))
|
||||
unknown = set(opts) - {"--size", "--fps", "--bitrate", "--label", "--client", "--port"}
|
||||
if unknown or len(rest) % 2:
|
||||
raise RuntimeError("remote add: options are --size WxH --fps F --bitrate KBPS --label TEXT --client NAME "
|
||||
"--port PORT --login")
|
||||
opts = dict(zip(args[4::2], args[5::2]))
|
||||
unknown = set(opts) - {"--size", "--fps", "--bitrate", "--label", "--client"}
|
||||
if unknown or len(args[4:]) % 2:
|
||||
raise RuntimeError("remote add: options are --size WxH --fps F --bitrate KBPS --label TEXT --client NAME")
|
||||
if not re.fullmatch(r"[A-Za-z0-9.:-]+", address) or not re.fullmatch(r"[A-Za-z0-9{}._:-]+", app):
|
||||
raise RuntimeError("remote add: the address or app has other characters")
|
||||
port = opts.get("--port", "")
|
||||
if port and not (port.isdigit() and 1024 < int(port) < 65535):
|
||||
raise RuntimeError("remote add: --port is a port number")
|
||||
port = int(port) if port else 0
|
||||
layout = load_layout()
|
||||
displays = remote_displays(layout)
|
||||
if len(displays) >= REMOTE_MAX:
|
||||
@@ -1078,7 +1049,7 @@ def remote_command(args):
|
||||
raise RuntimeError("remote add: the client name has other characters")
|
||||
size = [2560, 1440]
|
||||
if "--size" in opts:
|
||||
size = list(stream_size(*(int(v) for v in opts["--size"].lower().split("x"))))
|
||||
size = [int(v) for v in opts["--size"].lower().split("x")]
|
||||
elif app.startswith("display:"): # the monitor's own resolution
|
||||
code, out, _ = run_stream("monitors", address)
|
||||
try:
|
||||
@@ -1088,11 +1059,7 @@ def remote_command(args):
|
||||
size = [int(res["width"]), int(res["height"])]
|
||||
except (ValueError, KeyError, TypeError):
|
||||
pass
|
||||
target = f"{address}:{port}" if port else address
|
||||
if login:
|
||||
code, out, err = run_stream("pair", target, "--id", client, "--login", timeout=30, stdin=sys.stdin.read())
|
||||
else:
|
||||
code, out, err = run_stream("pair", target, "--id", client, timeout=30)
|
||||
code, out, err = run_stream("pair", address, "--id", client, timeout=30)
|
||||
if code:
|
||||
raise RuntimeError(f"pairing {client} with {address} failed: {(err or out).strip()}")
|
||||
log(out.strip().splitlines()[-1] if out.strip() else f"paired {client}")
|
||||
@@ -1104,8 +1071,6 @@ def remote_command(args):
|
||||
host["displays"].append({"id": display_id, "client": client, "app": app, "label": label, "size": size,
|
||||
"fps": int(opts.get("--fps", 60)), "bitrate": int(opts.get("--bitrate", 0)),
|
||||
"metres": round(max(1.0, size[0] / REMOTE_PIXELS_PER_METRE), 3)})
|
||||
if port:
|
||||
host["displays"][-1]["port"] = port
|
||||
remote_displays(layout) # gives it a screen number
|
||||
save_layout(layout)
|
||||
print(display_id)
|
||||
@@ -1117,7 +1082,7 @@ def remote_command(args):
|
||||
for kv in args[2:]:
|
||||
k, _, v = kv.partition("=")
|
||||
if k == "size" and re.fullmatch(r"\d+x\d+", v.lower()):
|
||||
d["size"] = list(stream_size(*(int(x) for x in v.lower().split("x"))))
|
||||
d["size"] = [int(x) for x in v.lower().split("x")]
|
||||
elif k in ("fps", "bitrate") and v.isdigit():
|
||||
d[k] = int(v)
|
||||
elif k == "label" and v:
|
||||
@@ -1148,7 +1113,7 @@ def remote_command(args):
|
||||
for profile in layout.get("profiles", {}).values():
|
||||
profile.get("remote", {}).pop(d["id"], None)
|
||||
save_layout(layout)
|
||||
code, out, err = run_stream("unpair", display_target(host, d), "--id", d["client"], timeout=30)
|
||||
code, out, err = run_stream("unpair", host["address"], "--id", d["client"], timeout=30)
|
||||
said = (out or err).strip().splitlines()
|
||||
log(f"removed {d['id']}; {said[-1] if said else f'unpair exited {code}'}")
|
||||
return
|
||||
@@ -1623,10 +1588,7 @@ def main(argv):
|
||||
wait = float(argv[argv.index("--wait") + 1]) if "--wait" in argv else 60
|
||||
with open(LOCK_PATH, "w") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
try:
|
||||
apply(wait)
|
||||
except RuntimeError as e: # (no head pose with the headset off): the scales still go
|
||||
log(f"not arranged now: {e}")
|
||||
apply(wait)
|
||||
for _ in range(30): # Plasma may still be starting
|
||||
try:
|
||||
log("kwin: " + (" ".join(apply_scales()) or "unchanged"))
|
||||
@@ -1678,16 +1640,7 @@ def main(argv):
|
||||
profile_remotes(layout, name)
|
||||
open_remotes(layout, name)
|
||||
save_layout(layout)
|
||||
try:
|
||||
apply(wait)
|
||||
except RuntimeError as e:
|
||||
if not wait:
|
||||
raise
|
||||
# Desktop start with the headset off (no head pose): KWin's scales must
|
||||
# still reach ft-screens, or every click on a scaled screen lands at
|
||||
# scale x its distance from the top-left (2026-10-11: 1.25, hundreds of
|
||||
# pixels off on the native screen).
|
||||
log(f"not arranged now: {e}")
|
||||
apply(wait)
|
||||
if not wait:
|
||||
kwin_follow()
|
||||
if wait:
|
||||
|
||||
+5
-13
@@ -21,9 +21,6 @@ RUN dnf -y install --setopt=install_weak_deps=False \
|
||||
# moonlight-common-c, and the host's sound (Opus, PulseAudio)
|
||||
openssl-devel libcurl-devel expat-devel libuuid-devel json-devel \
|
||||
opus-devel pulseaudio-libs-devel \
|
||||
# Frametop Voice (voice/): glslc and the SPIR-V headers for whisper.cpp's
|
||||
# Vulkan backend; ft-voiced records with pw-record and plays its sounds with pw-play
|
||||
glslc spirv-headers-devel pipewire-utils \
|
||||
# Frametop Input Settings (Kirigami, PySide6) and remote desktop (krdp):
|
||||
# the KDE/Qt stack has no PyPI wheels, so it stays a dnf package
|
||||
python3-pyside6 kf6-kirigami kf6-qqc2-desktop-style qt6-qtwayland \
|
||||
@@ -56,15 +53,14 @@ RUN install -d /opt/frametop/bin /opt/frametop/lib \
|
||||
# because PySide6 and Kirigami come from dnf for this python3: one interpreter
|
||||
# imports both, as in the dev container on the Frame. Rebuilt only when the
|
||||
# lockfile changes, so source edits below reuse this layer.
|
||||
# uv sync --frozen --group tracker --group voice: the default dev group (pytest/ruff
|
||||
# for CI's in-image tests) plus the tracker stack and Frametop Voice's (sherpa-onnx);
|
||||
# the types group (mypy) stays out.
|
||||
# uv sync --frozen --group tracker: the default dev group (pytest/ruff for CI's
|
||||
# in-image tests) plus the tracker stack; the types group (mypy) stays out.
|
||||
# --no-cache keeps uv's download cache out of the image.
|
||||
ENV UV_PROJECT_ENVIRONMENT=/opt/frametop/venv UV_PYTHON=/usr/bin/python3 UV_PYTHON_DOWNLOADS=never
|
||||
COPY pyproject.toml uv.lock .python-version /src/frametop/
|
||||
RUN uv venv --system-site-packages /opt/frametop/venv \
|
||||
&& cd /src/frametop && uv sync --frozen --group tracker --group voice --no-cache \
|
||||
&& /opt/frametop/venv/bin/python -c 'import PySide6, numpy, cv2, sherpa_onnx, sentencepiece'
|
||||
&& cd /src/frametop && uv sync --frozen --group tracker --no-cache \
|
||||
&& /opt/frametop/venv/bin/python -c 'import PySide6, numpy, cv2'
|
||||
|
||||
# The sources, then the binaries, built by the same build scripts as on the
|
||||
# Frame (FRAME_IN_BOX=1: they run here instead of entering the dev container).
|
||||
@@ -75,7 +71,7 @@ COPY . /src/frametop
|
||||
WORKDIR /src/frametop
|
||||
ENV FRAME_IN_BOX=1 OPENVR_LIB=/opt/frametop/lib
|
||||
RUN screens/build.sh \
|
||||
&& install -t /opt/frametop/bin screens/build/ft-screens screens/build/ft-handtest screens/build/ft-spotpanel
|
||||
&& install -t /opt/frametop/bin screens/build/ft-screens screens/build/ft-handtest
|
||||
RUN pointer/helper/build.sh && pointer/driver/build.sh \
|
||||
&& install -t /opt/frametop/bin pointer/helper/build/ft-pointer \
|
||||
&& install -t /opt/frametop/lib pointer/driver/build/driver_ft_pointer.so
|
||||
@@ -89,10 +85,6 @@ RUN stream/build.sh \
|
||||
# build/venv, which here only points at the venv above.
|
||||
RUN gaze/tracker/build.sh \
|
||||
&& install -t /opt/frametop/bin gaze/tracker/build/ft-eyegrab
|
||||
# Frametop Voice: whisper.cpp's server (Vulkan), and build/venv pointing at the venv
|
||||
# above. Its speech models aren't in the image: voice/models.sh fetches them on install.
|
||||
RUN voice/build.sh \
|
||||
&& install -t /opt/frametop/bin voice/build/bin/whisper-server
|
||||
|
||||
# A release installs from this image (pack/install-release.sh): it copies
|
||||
# /src/frametop, built, to the Frame for its host side (services, the driver,
|
||||
|
||||
@@ -3,8 +3,7 @@
|
||||
# FrameDrop (into ~/devkit-game/Frametop), by hand, or by get.sh --release. Next to this script
|
||||
# are frametop-image.tar (Frametop, built, as a container image) and frametop-release.json (its
|
||||
# version, commit, the image file's sha256 and the image's ID, and the SteamOS table). Nothing
|
||||
# builds on the headset, and nothing else is downloaded, but voice dictation's speech models
|
||||
# (about 320 MB, fixed URLs checked by sha256: voice/models.sh) if you install it.
|
||||
# builds on the headset, and nothing else is downloaded.
|
||||
#
|
||||
# 1. It checks this SteamOS build against the SteamOS table: Frametop's newest from GitHub
|
||||
# (pack/steamos.json on main) when it can get it, else the one in the release. Tested: on.
|
||||
@@ -17,7 +16,7 @@
|
||||
# are removed, with their containers and images.
|
||||
#
|
||||
# Usage: install-release.sh [--yes] [--any-steamos] [--unpack-only] [--dir DIR]
|
||||
# [--no-voice] [--no-eye-tracker] [--no-bluetooth | --bluetooth]
|
||||
# [--no-eye-tracker] [--no-bluetooth | --bluetooth]
|
||||
# --unpack-only stop after step 2 and the copy: don't run install.sh
|
||||
# --dir DIR where the releases go (default ~/.local/share/frametop/releases)
|
||||
# The rest go to install.sh. FRAMETOP_STEAMOS_TABLE names another table to fetch (a URL), or
|
||||
@@ -110,8 +109,8 @@ main() {
|
||||
--any-steamos) any=1 ;;
|
||||
--unpack-only) unpack_only=1 ;;
|
||||
--dir) base=${2:?--dir needs a folder}; shift ;;
|
||||
--no-voice|--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
|
||||
-h|--help) sed -n '2,24p' "$0"; return 0 ;;
|
||||
--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
|
||||
-h|--help) sed -n '2,23p' "$0"; return 0 ;;
|
||||
*) echo "unknown option: $1" >&2; return 2 ;;
|
||||
esac
|
||||
shift
|
||||
@@ -167,7 +166,7 @@ main() {
|
||||
return 1
|
||||
fi
|
||||
|
||||
printf '\n\033[1m== 0/11 loading Frametop %s (a minute or two)\033[0m\n' "$version"
|
||||
printf '\n\033[1m== 0/10 loading Frametop %s (a minute or two)\033[0m\n' "$version"
|
||||
if [ "$(podman image inspect -f '{{.Id}}' "$image" 2>/dev/null)" = "$id" ]; then
|
||||
echo "already loaded"
|
||||
else
|
||||
|
||||
@@ -9,5 +9,3 @@ On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1
|
||||
A window asks what to install, and your SteamOS password for the two optional parts that need it (our own eye tracker and the Bluetooth fixes); the password is only used for this install. Nothing is compiled on the headset. Restart SteamVR once afterwards.
|
||||
|
||||
Before installing, the installer checks your SteamOS build against the builds Frametop was tested on, and stops if this release is known not to work there.
|
||||
|
||||
To show another computer's monitors as screens (remote displays), set that computer up once with its host kit: [Windows](https://github.com/Frametop/frametop/releases/tag/windows-host-1), [Mac](https://github.com/Frametop/frametop/releases/tag/macos-host-1), or [Linux](https://github.com/Frametop/frametop/blob/experimental/host/linux/README.md). The README's [Remote displays](https://github.com/Frametop/frametop/blob/experimental/README.md#remote-displays) section has the steps.
|
||||
@@ -1,16 +0,0 @@
|
||||
# Template: the installer replaces @REPO@ with the repo path on the Frame.
|
||||
[Unit]
|
||||
Description=Frametop dashboard patch: the 3D mouse's treadmill device moves SteamVR's floating windows
|
||||
Documentation=file://@REPO@/pointer/dashpatch/ft-dashpatch.py
|
||||
# The dashboard runs in SteamVR's vrwebhelper; it starts and stops with SteamVR.
|
||||
After=steamvr.service
|
||||
PartOf=steamvr.service
|
||||
Requisite=steamvr.service
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/bin/python3 -I @REPO@/pointer/dashpatch/ft-dashpatch.py run
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=steamvr.service
|
||||
@@ -1,206 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Let the 3D mouse's treadmill device move SteamVR's floating windows.
|
||||
|
||||
SteamVR's dashboard (vrwebhelper's systemui page) starts a floating window's move only for a
|
||||
press from the head or a hand: UndockedOverlay.startFloatingWindowMove reads the pressing
|
||||
device from the mouse event's CapsLock/NumLock bits and gives up on a treadmill. So the 3D
|
||||
mouse, a treadmill since 2026-10-09 (docs/design.md, "Treadmill role"), clicked and resized
|
||||
those windows but couldn't move them. treadmill-move.js wraps the grab bar's handler so a press
|
||||
with no device moves the window with /user/treadmill. This puts it into the dashboard over the
|
||||
Chrome DevTools Protocol, again whenever the dashboard's page is new (SteamVR restarted).
|
||||
|
||||
The dashboard's debugger is off by default. SteamVR turns it on with VRWebHelper.DebuggerEnabled
|
||||
and listens on VRWebHelper.DebuggerPort, on 127.0.0.1 only. Not 8087: SteamOS's
|
||||
steamvr-web-debug-portforward.service forwards 0.0.0.0:8088 to 127.0.0.1:8087, which would put
|
||||
the dashboard's debugger on the network. Any local program can use the port, as with Steam's
|
||||
own on 8080 (steam/steamui.py).
|
||||
|
||||
pointer/dashpatch/ft-dashpatch.py on # turn the debugger on (from SteamVR's next start)
|
||||
pointer/dashpatch/ft-dashpatch.py off # turn it off again, if this turned it on
|
||||
pointer/dashpatch/ft-dashpatch.py status # the settings, and whether the patch is in
|
||||
pointer/dashpatch/ft-dashpatch.py run # keep the patch in (frametop-dashpatch.service)
|
||||
|
||||
With SteamVR running the settings go through vrcmd, since SteamVR keeps its settings in memory
|
||||
and writes over the file; without it, they go into SteamVR's settings file.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE.parent.parent / "steam"))
|
||||
from steamui import _WebSocket # noqa: E402 (Frametop's small CDP client)
|
||||
|
||||
HOME = Path.home()
|
||||
SECTION = "VRWebHelper"
|
||||
PORT = 8089
|
||||
PATCH = HERE / "treadmill-move.js"
|
||||
STEAMVR_BIN = "/opt/steamvr/bin/linuxarm64"
|
||||
STEAMVR_SETTINGS = (HOME / ".config/openvr/config/steamvr.vrsettings",
|
||||
HOME / ".steam/steam/config/steamvr.vrsettings")
|
||||
# There when this turned the debugger on, so `off` leaves a debugger someone else turned on alone.
|
||||
MARK = HOME / ".local/state/frametop/pointer/dashboard-debugger"
|
||||
# A terminal in the desktop has the desktop's own config folder; vrcmd needs SteamVR's.
|
||||
ENV = dict(os.environ, XDG_CONFIG_HOME=str(HOME / ".config"), LD_LIBRARY_PATH=STEAMVR_BIN)
|
||||
|
||||
|
||||
def steamvr_up():
|
||||
return subprocess.run(["pgrep", "-x", "vrserver"], capture_output=True).returncode == 0
|
||||
|
||||
|
||||
def vrcmd(*args):
|
||||
return subprocess.run([f"{STEAMVR_BIN}/vrcmd", *args], capture_output=True, text=True, timeout=15, env=ENV).stdout
|
||||
|
||||
|
||||
def settings_file():
|
||||
return next((p for p in STEAMVR_SETTINGS if p.exists()), STEAMVR_SETTINGS[0])
|
||||
|
||||
|
||||
def get():
|
||||
"""(DebuggerEnabled, DebuggerPort): a bool or None, an int or None."""
|
||||
if steamvr_up():
|
||||
on = re.search(rf"^{SECTION}\.DebuggerEnabled=(\w+)", vrcmd("--settings-bool", f"{SECTION}.DebuggerEnabled"), re.M)
|
||||
port = re.search(rf"^{SECTION}\.DebuggerPort=(-?\d+)", vrcmd("--settings-int", f"{SECTION}.DebuggerPort"), re.M)
|
||||
return (on.group(1) in ("true", "1") if on else None), (int(port.group(1)) if port else None)
|
||||
try:
|
||||
s = json.loads(settings_file().read_text()).get(SECTION, {})
|
||||
except (OSError, ValueError, AttributeError):
|
||||
return None, None
|
||||
on, port = s.get("DebuggerEnabled"), s.get("DebuggerPort")
|
||||
return (on if isinstance(on, bool) else None), (port if isinstance(port, int) else None)
|
||||
|
||||
|
||||
def put(on):
|
||||
if steamvr_up():
|
||||
# 1 or 0: vrcmd reads "true" as 0.
|
||||
vrcmd("--set-settings-bool", f"{SECTION}.DebuggerEnabled", "1" if on else "0")
|
||||
if on:
|
||||
vrcmd("--set-settings-int", f"{SECTION}.DebuggerPort", str(PORT))
|
||||
got_on, got_port = get()
|
||||
return got_on is on and (not on or got_port == PORT)
|
||||
path = settings_file()
|
||||
try:
|
||||
d = json.loads(path.read_text())
|
||||
except FileNotFoundError:
|
||||
d = {}
|
||||
except (OSError, ValueError) as e:
|
||||
print(f"can't read {path}: {e}", file=sys.stderr)
|
||||
return False
|
||||
s = d.setdefault(SECTION, {})
|
||||
s["DebuggerEnabled"] = on
|
||||
if on:
|
||||
s["DebuggerPort"] = PORT
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".frametop-tmp")
|
||||
tmp.write_text(json.dumps(d, indent=3) + "\n")
|
||||
tmp.replace(path)
|
||||
return True
|
||||
|
||||
|
||||
def dashboard():
|
||||
"""The systemui page's debugger URL, or None (SteamVR not up, or its debugger off)."""
|
||||
try:
|
||||
with urllib.request.urlopen(f"http://127.0.0.1:{PORT}/json", timeout=3) as r:
|
||||
targets = json.load(r)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return next((t.get("webSocketDebuggerUrl") for t in targets if t.get("title") == "systemui"), None)
|
||||
|
||||
|
||||
def evaluate(url, expression):
|
||||
ws = _WebSocket(url)
|
||||
try:
|
||||
ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": expression, "returnByValue": True}}))
|
||||
while True:
|
||||
reply = json.loads(ws.recv())
|
||||
if reply.get("id") == 1:
|
||||
break
|
||||
finally:
|
||||
ws.close()
|
||||
result = reply.get("result", {})
|
||||
if "exceptionDetails" in result:
|
||||
d = result["exceptionDetails"]
|
||||
return "error: " + (d.get("exception", {}).get("description") or d.get("text", "?"))
|
||||
return result.get("result", {}).get("value")
|
||||
|
||||
|
||||
STATE = "JSON.stringify(window.__ftTreadmillMove || null)"
|
||||
|
||||
|
||||
def run():
|
||||
"""Puts the patch in each new dashboard page, and checks it's still in every 30 s (the page
|
||||
can reload without a new URL)."""
|
||||
patch = PATCH.read_text()
|
||||
version = int(re.search(r"const VERSION = (\d+);", patch).group(1))
|
||||
last_url, last_check = None, 0.0
|
||||
while True:
|
||||
url = dashboard()
|
||||
now = time.monotonic()
|
||||
if url and (url != last_url or now - last_check > 30):
|
||||
try:
|
||||
state = json.loads(evaluate(url, STATE) or "null")
|
||||
if not isinstance(state, dict) or state.get("version", 0) < version:
|
||||
print(f"SteamVR's dashboard: {evaluate(url, patch)}", flush=True)
|
||||
last_url, last_check = url, now
|
||||
except (OSError, ValueError) as e:
|
||||
print(f"SteamVR's dashboard: can't reach it yet ({e})", flush=True)
|
||||
last_url = None
|
||||
time.sleep(3)
|
||||
|
||||
|
||||
def main():
|
||||
what = sys.argv[1] if len(sys.argv) == 2 else ""
|
||||
if what == "on":
|
||||
on, port = get()
|
||||
if on and port == PORT:
|
||||
print(f"SteamVR's dashboard debugger: on already, on port {PORT}")
|
||||
return 0
|
||||
if not put(True):
|
||||
print("couldn't turn on SteamVR's dashboard debugger: the 3D mouse can't move SteamVR's windows. "
|
||||
"Try again with: pointer/dashpatch/ft-dashpatch.py on", file=sys.stderr)
|
||||
return 1
|
||||
if not on:
|
||||
MARK.parent.mkdir(parents=True, exist_ok=True)
|
||||
MARK.write_text(f"{SECTION}.DebuggerEnabled was off; Frametop turned it on (port {PORT})\n")
|
||||
print(f"SteamVR's dashboard debugger: on, on 127.0.0.1:{PORT}, once SteamVR restarts "
|
||||
"(so the 3D mouse can move SteamVR's windows)")
|
||||
return 0
|
||||
if what == "off":
|
||||
if not MARK.exists():
|
||||
print("SteamVR's dashboard debugger: left as it is (Frametop didn't turn it on)")
|
||||
return 0
|
||||
if not put(False):
|
||||
print("couldn't turn off SteamVR's dashboard debugger. Try again with: "
|
||||
"pointer/dashpatch/ft-dashpatch.py off", file=sys.stderr)
|
||||
return 1
|
||||
MARK.unlink()
|
||||
print("SteamVR's dashboard debugger: off again, once SteamVR restarts")
|
||||
return 0
|
||||
if what == "status":
|
||||
on, port = get()
|
||||
print(f"{SECTION}.DebuggerEnabled: {'on' if on else 'off'}" + (" (set by Frametop)" if MARK.exists() else "")
|
||||
+ f", DebuggerPort: {port if port is not None else 'unset'}")
|
||||
url = dashboard()
|
||||
if not url:
|
||||
print(f"dashboard: not reachable on 127.0.0.1:{PORT}")
|
||||
else:
|
||||
try:
|
||||
print(f"dashboard patch: {evaluate(url, STATE)}")
|
||||
except (OSError, ValueError) as e:
|
||||
print(f"dashboard patch: unknown ({e})")
|
||||
return 0
|
||||
if what == "run":
|
||||
run()
|
||||
return 0
|
||||
print("usage: pointer/dashpatch/ft-dashpatch.py on|off|status|run", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,25 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install, start, stop, or inspect the dashboard patch (ft-dashpatch.py) on the Frame.
|
||||
# Usage: pointer/dashpatch/run.sh install|uninstall # user service, starts with SteamVR; install also
|
||||
# # turns on SteamVR's dashboard debugger (127.0.0.1:8089)
|
||||
# pointer/dashpatch/run.sh start|stop|restart|status|log [lines]
|
||||
set -euo pipefail
|
||||
root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
|
||||
. "$root/scripts/_env.sh"
|
||||
frame="$root/scripts/frame.sh"
|
||||
unit=frametop-dashpatch.service
|
||||
case ${1:-status} in
|
||||
install)
|
||||
fill_template "$root/pointer/dashpatch/$unit" | on_frame "mkdir -p ~/.config/systemd/user && cat > ~/.config/systemd/user/$unit"
|
||||
on_frame "chmod +x pointer/dashpatch/ft-dashpatch.py"
|
||||
on_frame "/usr/bin/python3 -I pointer/dashpatch/ft-dashpatch.py on" || true
|
||||
"$frame" --host "set -e; systemctl --user daemon-reload; systemctl --user enable $unit
|
||||
$(start_with_steamvr $unit)" ;;
|
||||
uninstall)
|
||||
"$frame" --host "systemctl --user disable --now $unit 2>/dev/null; rm -f ~/.config/systemd/user/$unit; systemctl --user daemon-reload; echo removed"
|
||||
on_frame "/usr/bin/python3 -I pointer/dashpatch/ft-dashpatch.py off" || true ;;
|
||||
start|stop|restart) "$frame" --host "systemctl --user $1 $unit; systemctl --user is-active $unit" ;;
|
||||
status) "$frame" --host "systemctl --user is-active $unit" || true; on_frame "/usr/bin/python3 -I pointer/dashpatch/ft-dashpatch.py status" || true ;;
|
||||
log) "$frame" --host "journalctl --user -u $unit --no-pager -o cat -n ${2:-30}" ;;
|
||||
*) echo "usage: $0 install|uninstall|start|stop|restart|status|log" >&2; exit 2 ;;
|
||||
esac
|
||||
@@ -1,76 +0,0 @@
|
||||
// Frametop: SteamVR's floating windows move for the 3D mouse's treadmill device (pointer/dashpatch).
|
||||
// The dashboard's UndockedOverlay.startFloatingWindowMove reads the pressing device from the mouse
|
||||
// event's CapsLock/NumLock modifier bits (head, left hand, right hand) and gives up when there are
|
||||
// none, as for a treadmill. This wraps the grab bar's onMouseDown: a press with no source runs the
|
||||
// original as the right hand, with that one pose query answered by /user/treadmill, and the move
|
||||
// then follows /user/treadmill.
|
||||
(() => {
|
||||
const VERSION = 1;
|
||||
if (window.__ftTreadmillMove && window.__ftTreadmillMove.version >= VERSION) return 'already ' + window.__ftTreadmillMove.version;
|
||||
let req = null;
|
||||
self.webpackChunkvrwebui.push([[Symbol('frametop-treadmill-move')], {}, r => { req = r; }]);
|
||||
if (!req) return 'no webpack';
|
||||
// React: module 6540 in SteamVR 2.18.2's dashboard; else the module that defines React's element
|
||||
// symbol (loaded already, so requiring it runs nothing new)
|
||||
const isReact = m => m && typeof m.createElement === 'function' && typeof m.Component === 'function' && /^18\./.test(m.version || '');
|
||||
let React = req.m['6540'] ? req('6540') : null;
|
||||
if (!isReact(React)) {
|
||||
React = null;
|
||||
for (const id of Object.keys(req.m)) {
|
||||
const src = req.m[id].toString();
|
||||
if (src.length < 40000 && src.includes('Symbol.for("react.element")') && src.includes('createElement')) {
|
||||
const m = req(id);
|
||||
if (isReact(m)) { React = m; break; }
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!React) return 'no React';
|
||||
const internals = React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED;
|
||||
const TREADMILL = '/user/treadmill', RIGHT = '/user/hand/right';
|
||||
const state = window.__ftTreadmillMove = {version: VERSION, moves: 0, last: ''};
|
||||
const wrapped = new WeakMap(); // the original bound handler -> ours
|
||||
|
||||
const wrap = (orig, inst) => function (e) {
|
||||
let src = 0;
|
||||
try { src = (e.getModifierState('CapsLock') ? 1 : 0) | (e.getModifierState('NumLock') ? 2 : 0); } catch (err) { src = -1; }
|
||||
if (src !== 0 || !inst) return orig(e);
|
||||
const [pose] = VRHTML.GetPose(TREADMILL, 0);
|
||||
if (!pose || !pose.bPoseIsValid) { state.last = 'treadmill pose invalid'; return orig(e); }
|
||||
const fake = new Proxy(e, {get(t, k) {
|
||||
if (k === 'getModifierState') return m => m === 'CapsLock' || m === 'NumLock';
|
||||
const v = t[k];
|
||||
return typeof v === 'function' ? v.bind(t) : v;
|
||||
}});
|
||||
const getPose = VRHTML.GetPose;
|
||||
let first = true; // only the moving device's query (the first one); a hand-docked window's own stays
|
||||
VRHTML.GetPose = function (path, origin) {
|
||||
if (first && path === RIGHT) { first = false; path = TREADMILL; }
|
||||
return getPose.call(this, path, origin);
|
||||
};
|
||||
let r;
|
||||
try { r = orig(fake); } finally { VRHTML.GetPose = getPose; }
|
||||
if (inst.m_sMoveDevicePath === RIGHT) {
|
||||
inst.m_sMoveDevicePath = TREADMILL;
|
||||
inst.setState({sParentDevice: TREADMILL});
|
||||
state.moves++;
|
||||
state.last = 'moved by the treadmill';
|
||||
}
|
||||
return r;
|
||||
};
|
||||
|
||||
const createElement = React.createElement;
|
||||
React.createElement = function (type, props) {
|
||||
if (props && typeof props.onMouseDown === 'function' && props.explicitSGID !== undefined &&
|
||||
props.onMouseDown.name === 'bound startFloatingWindowMove') {
|
||||
const owner = internals && internals.ReactCurrentOwner && internals.ReactCurrentOwner.current;
|
||||
const inst = owner && owner.stateNode;
|
||||
let ours = wrapped.get(props.onMouseDown);
|
||||
if (!ours) { ours = wrap(props.onMouseDown, inst); wrapped.set(props.onMouseDown, ours); }
|
||||
const args = Array.prototype.slice.call(arguments);
|
||||
args[1] = Object.assign({}, props, {onMouseDown: ours});
|
||||
return createElement.apply(this, args);
|
||||
}
|
||||
return createElement.apply(this, arguments);
|
||||
};
|
||||
return 'installed ' + VERSION;
|
||||
})()
|
||||
Loaded 100 of 217 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user