diff --git a/README.md b/README.md index 6dd32be..69edd2c 100644 --- a/README.md +++ b/README.md @@ -204,6 +204,7 @@ Frame's software fits together, all checked against a real headset and labelled | [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes | | [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing | | [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset | +| [Eye tracking and heart rate](docs/tracking.md) | Our OpenXR → OSC bridge, BlueZ heart-rate panel and optional local session log; SlimeVR feasibility notes | | [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, and the headset smoke test | | [Open questions](docs/open-questions.md) | What's still unchecked | diff --git a/docs/how-the-frame-works.md b/docs/how-the-frame-works.md index 5c392d1..385c00f 100644 --- a/docs/how-the-frame-works.md +++ b/docs/how-the-frame-works.md @@ -20,6 +20,15 @@ SteamVR (vrserver, vrcompositor, vrdashboard) ← renders the room + pane Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 3056000 ``` +## Tracking additions (verified 2026-09-28) + +On SteamOS 0.4.1, build `20260925.6191901`, SteamVR exposes combined gaze +through `XR_EXT_eye_gaze_interaction` in a headless OpenXR 1.0 session. Our +reader obtained valid tracked samples and sent OSC to a configured loopback +receiver. BlueZ LE discovery works; GTK4/GI can render our heart-rate panel. +No BLE strap or SlimeVR trackers were attached. See [tracking](tracking.md) +for the evidence, privacy defaults and untested integration boundaries. + ## Facts worth knowing | Fact | Where it matters | diff --git a/docs/img/heart-rate-panel.png b/docs/img/heart-rate-panel.png new file mode 100644 index 0000000..eab48f7 Binary files /dev/null and b/docs/img/heart-rate-panel.png differ diff --git a/docs/testing.md b/docs/testing.md index b06577d..6dc3bdf 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -148,3 +148,10 @@ For example, on 2026-09-27 the smoke test found that Steam's `create-shortcut` refuses ids with a hyphen (`missing/invalid arguments`), which the fake had accepted. The fake now refuses them the same way, and Frame Control makes ids Steam accepts. + +## Tracking protocols and fake BlueZ + +`tests/test_tracking.py` exercises our gaze conversion, OSC sender, HRS parser +and BlueZ lifecycle with an in-memory fake object tree. It runs in the normal +unit suite without Bluetooth, GTK or OpenXR. Real Frame results and the absent +strap/tracker boundaries are recorded in [tracking](tracking.md). diff --git a/docs/tracking.md b/docs/tracking.md new file mode 100644 index 0000000..9347949 --- /dev/null +++ b/docs/tracking.md @@ -0,0 +1,188 @@ +# Eye tracking and heart rate + +Frame Control's own tools run on the Frame, using OpenXR and BlueZ. No +VRCFaceTracking, LunaHR, Pulsoid or other tracking app is required. This is a +command-line first version; it does not add a desktop app tab. + +## What was checked + +**Verified 2026-09-28**, on a real aarch64 Frame running SteamOS 0.4.1, +BUILD_ID `20260925.6191901`: + +| Check | Result | +|---|---| +| OpenXR gaze | SteamVR advertises `XR_EXT_eye_gaze_interaction`, `XR_MND_headless` and `XR_KHR_convert_timespec_time`. `supportsEyeGazeInteraction=1`. A headless session reached FOCUSED and produced 269 valid, tracked orientations in the first ten-second probe. | +| Our gaze → OSC bridge | A separate ten-second run produced 280 valid samples and 280 correctly padded 44-byte `/tracking/eye/CenterPitchYaw` messages at an explicitly configured loopback receiver. Only counters and packet-layout checks were retained. | +| Bluetooth stack | BlueZ active, adapter powered, central/peripheral roles available. LE discovery started and stopped successfully. No pairing or adapter power settings changed. | +| Our heart-rate panel | GTK4/GI runs on the stock image. A **synthetic 72 BPM** notification displayed in our X11 window, tagged `STEAM_GAME=2000000027`. Window capture checked; no real heart-rate measurement was taken. | +| SlimeVR, separate feasibility check | Native aarch64 server v21.1.0 ran with an isolated Temurin 21 JRE, created its driver sockets and accepted a local TCP connection on port 21110. Driver v6.0.0 loaded with all shared libraries resolved; `HmdDriverFactory("IServerTrackedDeviceProvider_004")` returned a non-null provider and error 0. | + +![Our heart-rate panel on the Frame, showing synthetic 72 BPM](img/heart-rate-panel.png) + +The image is a capture of our own Frame window using a fake notification, +not a real sensor reading. + +**Untested:** a real BLE strap's notifications, physical fit/contact behaviour, +end-to-end heart-rate display/OSC/log with a strap, avatar response in VRChat, +gaze accuracy/calibration, coexistence with every immersive app, in-headset +panel placement, SlimeVR tracker/calibration data and the SlimeVR driver running +inside SteamVR. No trackers or strap are attached. The driver was loaded in a +separate process; it was **not registered or activated in SteamVR**. Steam and +SteamVR were not stopped or restarted. + +**Verified blocker resolved:** importing `tkinter` fails because `libtk8.6.so` +is absent. The panel uses the installed GTK4/GI bindings instead. The Frame's +OpenXR headers advertise a newer API version than the runtime accepts; our +reader requests OpenXR 1.0 explicitly. + +## Install our tools + +From this checkout on your computer, while the Frame is awake: + +```sh +python3 scripts/tracking-on-frame.py install +``` + +This copies our Python code and compiles our small C OpenXR reader into +`~/.local/share/frame-control/tracking/` on the Frame. It uses the Frame's +existing compiler, OpenXR headers/loader, Python, dbus-python, GI and GTK4. +Nothing is downloaded, and no sudo, driver registration, system setting, +service or autostart is added. `FRAME_ALIAS` can select another SSH alias. +The desktop app/server keeps its existing stdlib-only dependency set. + +## Eye tracking → OSC + +Start with a ten-second capability/data-availability check: + +```sh +python3 scripts/tracking-on-frame.py gaze --seconds 10 +``` + +This prints support, session-state numbers and sample counters. It opens no +OSC socket and prints no gaze coordinates. Exit 0 means at least one valid +sample, 3 means no valid sample was observed, and 1 means an API/runtime error. +If there are no valid samples, wake/wear the headset and check its tracking +setup; a successful capability check alone does not prove usable gaze. + +To send to VRChat running **on the Frame**, explicitly enable OSC in VRChat +and choose its local UDP endpoint: + +```sh +python3 scripts/tracking-on-frame.py gaze --seconds 3600 --osc 127.0.0.1 9000 +``` + +For a receiver on another computer, replace `127.0.0.1` with that computer's +IP address and choose its listening port. Addresses are IP literals (IPv4 or +IPv6); there is no discovery or default destination. Loopback here always +means **the Frame**, not the computer running the SSH command. OSC uses +unencrypted UDP: configure only a receiver you intend to receive this data. + +**Documented:** [VRChat's eye OSC interface](https://docs.vrchat.com/docs/osc-eye-tracking) +accepts `/tracking/eye/CenterPitchYaw` with two floats in degrees, positive down +and right. We locate OpenXR's combined gaze pose relative to VIEW (the head), +rotate its -Z forward vector and convert that direction to these angles. +Only active, orientation-valid **and tracked** samples are sent, at up to +30 Hz. No eyelid/blink, individual-eye or face values are invented. We do not +send neutral gaze on tracking loss; VRChat's documented timeout restores its +automatic eye behaviour after input stops. + +**Privacy:** gaze is personal data. It stays in process memory and a private +pipe between our reader and bridge. There is no gaze log option, telemetry, +OSC receiver or raw gaze on stdout/stderr. Only an explicit `--osc IP PORT` +opens an output socket. Runtime diagnostics and validity counters are not +measurements. Stop with Ctrl-C or let `--seconds` expire (maximum 24 hours). +A lost headless session ends the run; it does not silently reconnect. + +## BLE heart rate → our panel, OSC and optional log + +First discover/pair your strap in SteamOS's Bluetooth settings. Select that +strap's Bluetooth address explicitly; our tool does not scan for or connect +to arbitrary nearby devices. + +```sh +python3 scripts/tracking-on-frame.py heart \ + --device AA:BB:CC:DD:EE:FF --panel --seconds 3600 +``` + +This uses BlueZ's standard Heart Rate Service (`180d`) and Heart Rate +Measurement (`2a37`) notifications. It finds the characteristic only beneath +the selected device's HRS service. The reader handles 8- and 16-bit BPM, +contact flags and optional energy/RR fields; energy and RR intervals are +validated for length but discarded. Zero BPM, reported loss of skin contact, +malformed packets and readings older than five seconds are not shown as a +current measurement. A disconnect stops the run; reconnect and start again. +This is a social/fitness readout, not a medical monitor. + +The panel is our GTK4 window on gamescope's X display. Use SteamVR's panel +controls to float/dock it (see [panels](panels.md)). **Stop**, closing the panel, +Ctrl-C, SSH hangup or the duration limit ends our subscription. A connection +that was already open when we started is preserved; a connection we opened +is disconnected on exit. No Bluetooth power or pairing state is changed. + +Add either output explicitly: + +```sh +python3 scripts/tracking-on-frame.py heart \ + --device AA:BB:CC:DD:EE:FF --panel --seconds 3600 \ + --osc 127.0.0.1 9000 --address /avatar/parameters/HeartRate \ + --log /home/steamos/heart-session.csv +``` + +The OSC value is integer BPM. `HeartRate` is a chosen avatar parameter, **not a +built-in VRChat heart-rate feature**; your avatar/receiver must define the +matching parameter. `--address` can select another literal OSC path. The local +panel works without OSC, a log or an avatar integration. + +The optional CSV contains only `unix_seconds,bpm`. It is created privately +(mode 0600), refuses existing files/symlinks, and lives **on the Frame** at the +path you specify. Nothing is logged by default, and heart-rate values are not +printed to the terminal. Delete your session file when you no longer need it. + +## SlimeVR: feasibility only + +SlimeVR is an independent application stack. Neither of our features installs, +launches or depends on it. Users who want it can follow +[SlimeVR's setup documentation](https://docs.slimevr.dev/server/index.html). +The consented upstream releases tested were +[server v21.1.0](https://github.com/SlimeVR/SlimeVR-Server/releases/tag/v21.1.0) +and [driver v6.0.0](https://github.com/SlimeVR/SlimeVR-OpenVR-Driver/releases/tag/v6.0.0), +under SlimeVR's MIT/Apache-2.0 licensing. + +**Verified layout, read-only:** the Frame's registered runtime is `/opt/steamvr`; +its native driver is `drivers/cv/bin/linuxarm64/driver_cv.so`, with a +`drivers/cv/driver.vrdrivermanifest`. Frame controller manifests/resources are +under `drivers/frame_controller/`. Configuration is under +`~/.config/openvr/config/`, not the Steam client's config directory. The +SlimeVR release also uses `slimevr/bin/linuxarm64/driver_slimevr.so` plus its +manifest. Nothing in those installed SteamVR directories was changed. + +**Inferred:** the matching ABI/layout and standalone factory success make +SteamVR integration plausible. They do not prove successful driver `Init`, +server/driver IPC, tracking, or calibration. That needs a separate integration +check with hardware and an agreed SteamVR restart. No Java executable was on +PATH for this check, so an isolated JRE was used. SlimeVR's server opens LAN +listeners; our temporary server was stopped and the temporary downloads, +configuration and logs were removed. It is not left installed or running. + +## Tests and remaining checks + +```sh +python3 -m unittest discover -s tests +``` + +`tests/test_tracking.py` covers HRS packet parsing, contact/staleness, OSC +padding/types and a real loopback socket, quaternion signs, opt-in networking, +private/exclusive logging and a fake BlueZ object tree. The fake checks service +ownership, notification routing, delayed GATT discovery and connection cleanup. +It does not pretend to be a physical strap or a real OpenXR runtime. + +Before calling hardware support complete, attach a strap and check BPM against +its own display/reference, loss of contact, disconnect/reconnect, Stop, OSC and +CSV together. Check avatar eyes while looking up/down/left/right in a supported +VRChat session. No third-party tracking app is needed for either test. + +Independent review attempt: `devin -p --model swe-2-max` with the frozen diff, +contribution standards and read-only instructions returned no output for ten +minutes. It was terminated with exit 143. No completed review or actual model +identity was returned; hardware checks and independent review remain follow-up +work before making the draft ready. diff --git a/frame/tracking/gaze.c b/frame/tracking/gaze.c new file mode 100644 index 0000000..95a5789 --- /dev/null +++ b/frame/tracking/gaze.c @@ -0,0 +1,135 @@ +/* Frame Control's OpenXR gaze source. No values on stdout/stderr or disk. + * The Python bridge supplies a private pipe with --fd; standalone probes only + * report counters. Uses a headless session, never submits frames or takes focus. */ +#define XR_USE_TIMESPEC +#include +#include +#include +#include +#include +#include +#include +#include + +static volatile sig_atomic_t stopped; +static void stop(int sig) { (void)sig; stopped = 1; } +#define CHECK(call) do { result = (call); if (XR_FAILED(result)) { \ + fprintf(stderr, "%s failed (%d)\n", #call, result); goto cleanup; } } while (0) + +int main(int argc, char **argv) { + int seconds = 10, fd = -1, running = 0, rc = 1; + unsigned active = 0, valid = 0, samples = 0; + for (int i = 1; i < argc; i++) { + if (!strcmp(argv[i], "--seconds") && i+1 < argc) seconds = atoi(argv[++i]); + else if (!strcmp(argv[i], "--fd") && i+1 < argc) fd = atoi(argv[++i]); + else { fprintf(stderr, "usage: gaze [--seconds 1..86400] [--fd private-pipe]\n"); return 2; } + } + if (seconds < 1 || seconds > 86400 || (fd != -1 && fd < 3)) return 2; + FILE *out = fd == -1 ? NULL : fdopen(fd, "w"); + if (fd != -1 && !out) return 2; + signal(SIGINT, stop); signal(SIGTERM, stop); signal(SIGHUP, stop); signal(SIGPIPE, SIG_IGN); + XrResult result; + XrInstance instance = XR_NULL_HANDLE; + XrSession session = XR_NULL_HANDLE; + XrActionSet set = XR_NULL_HANDLE; + XrSpace gaze = XR_NULL_HANDLE, view = XR_NULL_HANDLE; + const char *extensions[] = {"XR_EXT_eye_gaze_interaction", "XR_MND_headless", "XR_KHR_convert_timespec_time"}; + XrInstanceCreateInfo create = {.type = XR_TYPE_INSTANCE_CREATE_INFO}; + strcpy(create.applicationInfo.applicationName, "Frame Control gaze"); + create.applicationInfo.apiVersion = XR_MAKE_VERSION(1, 0, 0); + create.enabledExtensionCount = 3; create.enabledExtensionNames = extensions; + CHECK(xrCreateInstance(&create, &instance)); + XrSystemGetInfo get = {.type = XR_TYPE_SYSTEM_GET_INFO, .formFactor = XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY}; + XrSystemId system; + CHECK(xrGetSystem(instance, &get, &system)); + XrSystemEyeGazeInteractionPropertiesEXT eye = {.type = XR_TYPE_SYSTEM_EYE_GAZE_INTERACTION_PROPERTIES_EXT}; + XrSystemProperties props = {.type = XR_TYPE_SYSTEM_PROPERTIES, .next = &eye}; + CHECK(xrGetSystemProperties(instance, system, &props)); + printf("supportsEyeGazeInteraction=%u\n", eye.supportsEyeGazeInteraction); + if (!eye.supportsEyeGazeInteraction) goto cleanup; + XrSessionCreateInfo sc = {.type = XR_TYPE_SESSION_CREATE_INFO, .systemId = system}; + CHECK(xrCreateSession(instance, &sc, &session)); + XrActionSetCreateInfo asc = {.type = XR_TYPE_ACTION_SET_CREATE_INFO}; + strcpy(asc.actionSetName, "gaze"); strcpy(asc.localizedActionSetName, "Gaze"); + CHECK(xrCreateActionSet(instance, &asc, &set)); + XrActionCreateInfo ac = {.type = XR_TYPE_ACTION_CREATE_INFO, .actionType = XR_ACTION_TYPE_POSE_INPUT}; + strcpy(ac.actionName, "gaze_pose"); strcpy(ac.localizedActionName, "Gaze pose"); + XrAction action; + CHECK(xrCreateAction(set, &ac, &action)); + XrPath profile, input; + CHECK(xrStringToPath(instance, "/interaction_profiles/ext/eye_gaze_interaction", &profile)); + CHECK(xrStringToPath(instance, "/user/eyes_ext/input/gaze_ext/pose", &input)); + XrActionSuggestedBinding binding = {action, input}; + XrInteractionProfileSuggestedBinding suggested = {.type = XR_TYPE_INTERACTION_PROFILE_SUGGESTED_BINDING, + .interactionProfile = profile, .countSuggestedBindings = 1, .suggestedBindings = &binding}; + CHECK(xrSuggestInteractionProfileBindings(instance, &suggested)); + XrSessionActionSetsAttachInfo attach = {.type = XR_TYPE_SESSION_ACTION_SETS_ATTACH_INFO, .countActionSets = 1, .actionSets = &set}; + CHECK(xrAttachSessionActionSets(session, &attach)); + XrActionSpaceCreateInfo space = {.type = XR_TYPE_ACTION_SPACE_CREATE_INFO, .action = action, .poseInActionSpace.orientation.w = 1}; + CHECK(xrCreateActionSpace(session, &space, &gaze)); + XrReferenceSpaceCreateInfo ref = {.type = XR_TYPE_REFERENCE_SPACE_CREATE_INFO, .referenceSpaceType = XR_REFERENCE_SPACE_TYPE_VIEW, + .poseInReferenceSpace.orientation.w = 1}; + CHECK(xrCreateReferenceSpace(session, &ref, &view)); + PFN_xrConvertTimespecTimeToTimeKHR convert; + CHECK(xrGetInstanceProcAddr(instance, "xrConvertTimespecTimeToTimeKHR", (PFN_xrVoidFunction *)&convert)); + struct timespec start, now; + clock_gettime(CLOCK_MONOTONIC, &start); + while (!stopped) { + clock_gettime(CLOCK_MONOTONIC, &now); + if (now.tv_sec - start.tv_sec >= seconds) break; + XrEventDataBuffer event = {.type = XR_TYPE_EVENT_DATA_BUFFER}; + while ((result = xrPollEvent(instance, &event)) == XR_SUCCESS) { + if (event.type == XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED) { + XrSessionState state = ((XrEventDataSessionStateChanged *)&event)->state; + printf("sessionState=%d\n", state); fflush(stdout); + if (state == XR_SESSION_STATE_READY && !running) { + XrSessionBeginInfo begin = {.type = XR_TYPE_SESSION_BEGIN_INFO, .primaryViewConfigurationType = XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO}; + CHECK(xrBeginSession(session, &begin)); running = 1; + } else if (state == XR_SESSION_STATE_STOPPING) { + CHECK(xrEndSession(session)); running = 0; stopped = 1; + } else if (state == XR_SESSION_STATE_EXITING || state == XR_SESSION_STATE_LOSS_PENDING) stopped = 1; + } else if (event.type == XR_TYPE_EVENT_DATA_INSTANCE_LOSS_PENDING) stopped = 1; + event.type = XR_TYPE_EVENT_DATA_BUFFER; + } + if (XR_FAILED(result)) goto cleanup; + if (running && !stopped) { + XrActiveActionSet activeSet = {set, XR_NULL_PATH}; + XrActionsSyncInfo sync = {.type = XR_TYPE_ACTIONS_SYNC_INFO, .countActiveActionSets = 1, .activeActionSets = &activeSet}; + CHECK(xrSyncActions(session, &sync)); + if (result != XR_SUCCESS) { + struct timespec delay = {.tv_nsec = 33333333}; + nanosleep(&delay, NULL); + continue; /* No stale gaze when the runtime denies focus. */ + } + XrActionStateGetInfo ag = {.type = XR_TYPE_ACTION_STATE_GET_INFO, .action = action}; + XrActionStatePose pose = {.type = XR_TYPE_ACTION_STATE_POSE}; + CHECK(xrGetActionStatePose(session, &ag, &pose)); + samples++; + if (pose.isActive) { + active++; + XrTime time; CHECK(convert(instance, &now, &time)); + XrSpaceLocation location = {.type = XR_TYPE_SPACE_LOCATION}; + CHECK(xrLocateSpace(gaze, view, time, &location)); + XrSpaceLocationFlags needed = XR_SPACE_LOCATION_ORIENTATION_VALID_BIT | XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT; + if ((location.locationFlags & needed) == needed) { + valid++; + if (out) { + XrQuaternionf q = location.pose.orientation; + if (fprintf(out, "%g %g %g %g\n", q.x, q.y, q.z, q.w) < 0 || fflush(out)) goto cleanup; + } + } + } + } + struct timespec delay = {.tv_nsec = 33333333}; nanosleep(&delay, NULL); + } + printf("samples=%u active=%u valid=%u\n", samples, active, valid); + rc = valid ? 0 : 3; /* Distinguish a working session from observed gaze. */ +cleanup: + if (view) xrDestroySpace(view); + if (gaze) xrDestroySpace(gaze); + if (session) xrDestroySession(session); + if (set) xrDestroyActionSet(set); + if (instance) xrDestroyInstance(instance); + if (out) fclose(out); + return rc; +} diff --git a/frame/tracking/tracking.py b/frame/tracking/tracking.py new file mode 100644 index 0000000..10b8612 --- /dev/null +++ b/frame/tracking/tracking.py @@ -0,0 +1,387 @@ +#!/usr/bin/env python3 +"""Frame-local tracking tools. No network destination or data log by default.""" +import argparse +import math +import os +from pathlib import Path +import signal +import socket +import struct +import subprocess +import time + + +class TrackingError(RuntimeError): + """A safe, actionable status message containing no sensor data.""" + + +HRS = "0000180d-0000-1000-8000-00805f9b34fb" +MEASUREMENT = "00002a37-0000-1000-8000-00805f9b34fb" +DEVICE = "org.bluez.Device1" +SERVICE = "org.bluez.GattService1" +CHARACTERISTIC = "org.bluez.GattCharacteristic1" + + +def heart_rate(data): + """Validate the Bluetooth HRS measurement, returning BPM/contact only. + + Energy and RR intervals are checked for length but never retained. + None means contact is supported and the strap reports no skin contact. + """ + data = bytes(data) + if len(data) < 2 or data[0] & 0xe0: + raise ValueError("invalid HRS measurement") + flags = data[0] + size = 2 if flags & 1 else 1 + end = 1 + size + (2 if flags & 8 else 0) + if len(data) < end: + raise ValueError("truncated HRS measurement") + extra = len(data) - end + if (flags & 16 and (extra < 2 or extra % 2)) or (not flags & 16 and extra): + raise ValueError("invalid HRS optional fields") + if flags & 4 and not flags & 2: + return None + bpm = int.from_bytes(data[1:1 + size], "little") + return bpm if bpm else None + + +def gaze_angles(quaternion): + """OpenXR head-relative -Z forward → VRChat degrees, down/right positive.""" + if len(quaternion) != 4 or not all(math.isfinite(v) for v in quaternion): + raise ValueError("invalid gaze orientation") + norm = math.sqrt(sum(v * v for v in quaternion)) + if not 0.9 < norm < 1.1: + raise ValueError("invalid gaze orientation") + x, y, z, w = (v / norm for v in quaternion) + # Rotate OpenXR's forward vector (0, 0, -1) into VIEW space. + dx, dy, dz = -2 * (x*z + w*y), 2 * (w*x - y*z), 2 * (x*x + y*y) - 1 + return math.degrees(math.atan2(-dy, math.hypot(dx, dz))), math.degrees(math.atan2(dx, -dz)) + + +def osc_message(address, values): + if not address.startswith("/") or any(c.isspace() or c in '\0#*,?[]{}' for c in address): + raise ValueError("OSC address must be a literal path") + def string(value): + encoded = value.encode("utf-8") + b"\0" + return encoded + b"\0" * (-len(encoded) % 4) + tags, payload = ",", b"" + for value in values: + if type(value) is int: + tags += "i" + payload += struct.pack(">i", value) + else: + if not math.isfinite(value): + raise ValueError("OSC value must be finite") + tags += "f" + payload += struct.pack(">f", value) + return string(address) + string(tags) + payload + + +class Osc: + def __init__(self, endpoint=None): + self.sock = None + self.target = None + if endpoint: + import ipaddress + address = ipaddress.ip_address(endpoint[0]) + port = int(endpoint[1]) + if address.is_unspecified or address.is_multicast or not 1 <= port <= 65535: + raise ValueError("OSC needs a unicast IP address and port 1..65535") + self.sock = socket.socket(socket.AF_INET6 if address.version == 6 else socket.AF_INET, socket.SOCK_DGRAM) + self.target = (str(address), port) + + def send(self, address, values): + if self.sock: + self.sock.sendto(osc_message(address, values), self.target) + + def close(self): + if self.sock: + self.sock.close() + + +class HeartSession: + def __init__(self, osc, address, log=None, clock=time.monotonic): + self.osc, self.address, self.clock = osc, address, clock + self.bpm, self.updated = None, None + self.log = None + if log: + # Exclusive creation refuses existing files and symlinks; mode is + # private even with a permissive process umask. + fd = os.open(log, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + self.log = os.fdopen(fd, "w") + self.log.write("unix_seconds,bpm\n") + + def notification(self, data): + self.bpm = heart_rate(data) + self.updated = self.clock() + if self.bpm is not None: + self.osc.send(self.address, [self.bpm]) + if self.log: + self.log.write(f"{time.time():.3f},{self.bpm}\n") + self.log.flush() + + def current(self): + if self.updated is None or self.clock() - self.updated > 5: + return None + return self.bpm + + def close(self): + if self.log: + self.log.close() + + +class BluezHeart: + """One explicitly selected, already discovered strap; no ambient scan.""" + def __init__(self, bus, interface, address, on_value): + self.bus, self.interface, self.on_value = bus, interface, on_value + self.device = self.characteristic = None + self.connected_here = False + self.notifying = False + self.match = None + objects = self.objects() + matches = [path for path, interfaces in objects.items() + if str(interfaces.get(DEVICE, {}).get("Address", "")).upper() == address.upper()] + if len(matches) != 1: + raise TrackingError("Strap not found uniquely in BlueZ; pair/discover it in SteamOS Bluetooth settings first") + self.device = matches[0] + self.match = bus.add_signal_receiver(self.changed, signal_name="PropertiesChanged", + dbus_interface="org.freedesktop.DBus.Properties", + bus_name="org.bluez", path_keyword="path") + try: + if not objects[self.device][DEVICE].get("Connected"): + self.call(self.device, DEVICE).Connect(timeout=20) + self.connected_here = True + except Exception: + self.close() + raise + + def objects(self): + return self.call("/", "org.freedesktop.DBus.ObjectManager").GetManagedObjects() + + def call(self, path, kind): + return self.interface(self.bus.get_object("org.bluez", path), kind) + + def subscribe(self): + objects = self.objects() + if not objects.get(self.device, {}).get(DEVICE, {}).get("ServicesResolved"): + return False + services = {p for p, obj in objects.items() if str(obj.get(SERVICE, {}).get("UUID", "")).lower() == HRS + and obj[SERVICE].get("Device") == self.device} + for path, obj in objects.items(): + props = obj.get(CHARACTERISTIC, {}) + if props.get("Service") in services and str(props.get("UUID", "")).lower() == MEASUREMENT: + if "notify" not in props.get("Flags", []): + raise TrackingError("Heart-rate characteristic does not support notifications") + self.characteristic = path + self.call(path, CHARACTERISTIC).StartNotify() + self.notifying = True + return True + raise TrackingError("Selected device has no standard Heart Rate Service measurement") + + def changed(self, kind, changes, invalidated, path=None): + if kind == CHARACTERISTIC and path == self.characteristic and "Value" in changes: + self.on_value(changes["Value"]) + elif kind == DEVICE and path == self.device and "Connected" in changes and not changes["Connected"]: + self.on_value(None) + + def close(self): + try: + if self.notifying: + self.call(self.characteristic, CHARACTERISTIC).StopNotify() + finally: + if self.match: + self.match.remove() + if self.connected_here: + self.call(self.device, DEVICE).Disconnect() + + +def run_gaze(args, osc): + binary = Path(__file__).with_name("gaze") + command = [str(binary), "--seconds", str(args.seconds)] + if not args.osc: + return subprocess.call(command) + read_fd, write_fd = os.pipe() + process = None + try: + process = subprocess.Popen(command + ["--fd", str(write_fd)], pass_fds=(write_fd,)) + os.close(write_fd) + write_fd = None + with os.fdopen(read_fd) as source: + read_fd = None + for line in source: + try: + angles = gaze_angles([float(v) for v in line.split()]) + except ValueError: + continue + osc.send("/tracking/eye/CenterPitchYaw", angles) + return process.wait() + finally: + if read_fd is not None: + os.close(read_fd) + if write_fd is not None: + os.close(write_fd) + if process and process.poll() is None: + process.terminate() + try: + process.wait(timeout=5) + except subprocess.TimeoutExpired: + process.kill() + process.wait() + + +class HeartPanel: + """Our GTK panel, using the Frame's existing GTK4/GI platform libraries.""" + def __init__(self): + os.environ["GDK_BACKEND"] = "x11" + import gi + gi.require_version("Gtk", "4.0") + gi.require_version("GdkX11", "4.0") + from gi.repository import Gtk, Gdk, GdkX11, GLib + Gtk.init() + self.running = True + self.window = Gtk.Window(title="Frame Control · Heart rate") + self.window.set_default_size(480, 320) + self.window.connect("close-request", self.stop) + Gtk.Settings.get_default().set_property("gtk-application-prefer-dark-theme", True) + box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=16) + box.set_valign(Gtk.Align.CENTER) + box.set_halign(Gtk.Align.CENTER) + box.set_size_request(440, -1) + for side in ("top", "bottom", "start", "end"): + getattr(box, "set_margin_" + side)(24) + self.window.set_child(box) + title = Gtk.Label(label="Heart rate") + title.add_css_class("title-2") + box.append(title) + self.reading = Gtk.Label(label="—") + self.reading.add_css_class("reading") + box.append(self.reading) + self.status = Gtk.Label(label="Waiting for strap") + box.append(self.status) + button = Gtk.Button(label="Stop") + button.connect("clicked", self.stop) + box.append(button) + css = Gtk.CssProvider() + css.load_from_data(b".reading { font-size: 144px; font-weight: 700; }") + Gtk.StyleContext.add_provider_for_display(Gdk.Display.get_default(), css, Gtk.STYLE_PROVIDER_PRIORITY_APPLICATION) + self.window.present() + context = GLib.MainContext.default() + while context.pending(): + context.iteration(False) + try: + xid = GdkX11.X11Surface.get_xid(self.window.get_surface()) + subprocess.run(["xprop", "-id", str(xid), "-f", "STEAM_GAME", "32c", "-set", "STEAM_GAME", "2000000027"], + check=True, stdout=subprocess.DEVNULL) + except Exception: + self.window.destroy() + raise + + def stop(self, *args): + self.running = False + return True + + def update(self, bpm): + self.reading.set_label(str(bpm) if bpm is not None else "—") + self.status.set_label("beats per minute" if bpm is not None else "Waiting for strap") + + def close(self): + self.window.destroy() + + +def run_heart(args, osc): + import dbus + from dbus.mainloop.glib import DBusGMainLoop + from gi.repository import GLib + DBusGMainLoop(set_as_default=True) + session = HeartSession(osc, args.address, args.log) + reader, root = None, None + failure = [] + def value(data): + if data is None: + session.bpm = None + failure.append("Strap disconnected; reconnect and start again") + return + try: + session.notification(data) + except ValueError: + session.bpm = None + except OSError: + failure.append("OSC or session log write failed") + try: + reader = BluezHeart(dbus.SystemBus(), dbus.Interface, args.device, value) + context = GLib.MainContext.default() + deadline = time.monotonic() + 20 + while not reader.subscribe(): + if time.monotonic() > deadline: + raise TrackingError("Timed out waiting for the strap's GATT services") + while context.pending(): + context.iteration(False) + time.sleep(0.1) + end = time.monotonic() + args.seconds + print("Heart-rate notifications started; readings stay local unless OSC or a log was selected.") + if args.panel: + root = HeartPanel() + running = lambda: root.running if root else True + while running() and time.monotonic() < end and not failure: + while context.pending(): + context.iteration(False) + if root: + root.update(session.current()) + time.sleep(0.05) + if failure: + raise TrackingError(failure[0]) + return 0 + finally: + if root: + root.close() + try: + if reader: + reader.close() + finally: + session.close() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + gaze = commands.add_parser("gaze", help="headless OpenXR; prints counters only without --osc") + heart = commands.add_parser("heart", help="standard BLE HRS from an explicitly selected strap") + for command in (gaze, heart): + command.add_argument("--osc", nargs=2, metavar=("IP", "PORT"), help="explicit UDP destination; no default") + command.add_argument("--seconds", type=int, default=10, help="bounded run, 1..86400 seconds (default: 10)") + heart.add_argument("--device", required=True, help="strap Bluetooth address already discovered by BlueZ") + heart.add_argument("--panel", action="store_true", help="show our panel on gamescope DISPLAY=:0") + heart.add_argument("--address", default="/avatar/parameters/HeartRate", help="integer BPM OSC parameter") + heart.add_argument("--log", help="new private CSV file; disabled by default") + args = parser.parse_args() + if not 1 <= args.seconds <= 86400: + parser.error("--seconds must be 1..86400") + def interrupted(signum, frame): + raise KeyboardInterrupt + signal.signal(signal.SIGTERM, interrupted) + if hasattr(signal, "SIGHUP"): + signal.signal(signal.SIGHUP, interrupted) + osc = None + try: + if args.command == "heart": + osc_message(args.address, [0]) + if args.panel: + os.environ["DISPLAY"] = ":0" + osc = Osc(args.osc) + return run_gaze(args, osc) if args.command == "gaze" else run_heart(args, osc) + except TrackingError as error: + print(str(error)) + return 1 + except KeyboardInterrupt: + return 130 + except Exception as error: + # Never dump notifications, gaze, BLE addresses or exception payloads. + print(f"Tracking stopped ({type(error).__name__}). Check the device, runtime and selected output.") + return 1 + finally: + if osc: + osc.close() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/tracking-on-frame.py b/scripts/tracking-on-frame.py new file mode 100644 index 0000000..2aa39e0 --- /dev/null +++ b/scripts/tracking-on-frame.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +"""Install or run our local-only tracking tools on the Frame. + + python3 scripts/tracking-on-frame.py install + python3 scripts/tracking-on-frame.py gaze --seconds 10 + python3 scripts/tracking-on-frame.py gaze --seconds 3600 --osc 127.0.0.1 9000 + python3 scripts/tracking-on-frame.py heart --device AA:BB:CC:DD:EE:FF --panel + +FRAME_ALIAS overrides the SSH alias (default: frame). Runs in the foreground; +Ctrl-C stops the reader. No service, autostart, sudo or SteamVR settings changes. +""" +import os +from pathlib import Path +import shlex +import subprocess +import sys +import tarfile + +REMOTE = '"$HOME/.local/share/frame-control/tracking"' +INSTALL = '''set -eu +base="$HOME/.local/share/frame-control/tracking" +mkdir -p "$base" +stage=$(mktemp -d "$base/.install.XXXXXX") +trap 'rm -rf "$stage"' EXIT + tar -xf - -C "$stage" +cc -O2 -Wall -Wextra -Werror "$stage/gaze.c" \\ + -L/opt/steamvr/bin/linuxarm64 -Wl,-rpath,/opt/steamvr/bin/linuxarm64 \\ + -lopenxr_loader -o "$stage/gaze" +chmod 700 "$stage/gaze" "$stage/tracking.py" +mv "$stage/gaze" "$stage/tracking.py" "$base/" +echo 'Installed Frame Control tracking tools (no service started).' +''' + + +def main(): + if len(sys.argv) < 2 or sys.argv[1] not in ("install", "gaze", "heart"): + print(__doc__) + return 2 + host = os.environ.get("FRAME_ALIAS", "frame") + if not host or host.startswith("-"): + raise ValueError("invalid SSH alias") + ssh = ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", host] + if sys.argv[1] == "install": + import tempfile + source = Path(__file__).resolve().parents[1] / "frame" / "tracking" + with tempfile.TemporaryFile() as archive: + with tarfile.open(fileobj=archive, mode="w") as tar: + for name in ("gaze.c", "tracking.py"): + tar.add(source / name, arcname=name) + archive.seek(0) + return subprocess.call(ssh + ["bash -c " + shlex.quote(INSTALL)], stdin=archive) + # Allocate a tty so SSH forwards Ctrl-C and hangup to the foreground process. + command = 'exec python3 ' + REMOTE + '/tracking.py ' + shlex.join(sys.argv[1:]) + return subprocess.call(ssh[:-1] + ["-tt", host, command]) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_tracking.py b/tests/test_tracking.py new file mode 100644 index 0000000..1cef2e6 --- /dev/null +++ b/tests/test_tracking.py @@ -0,0 +1,188 @@ +"""Tracking protocols and fake-Frame BlueZ lifecycle; no headset or strap needed.""" +import importlib.util +import math +import os +from pathlib import Path +import socket +import stat +import struct +import tempfile +import unittest +from unittest.mock import Mock, patch + +SPEC = importlib.util.spec_from_file_location("tracking", Path(__file__).resolve().parents[1] / "frame/tracking/tracking.py") +t = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(t) + + +class Protocols(unittest.TestCase): + def test_hrs_formats(self): + self.assertEqual(t.heart_rate(b"\x00\x48"), 72) + self.assertEqual(t.heart_rate(b"\x01\x2c\x01"), 300) + self.assertEqual(t.heart_rate(b"\x1e\x48\x01\x00\x00\x04\x00\x04"), 72) + self.assertEqual(t.heart_rate(b"\x02\x48"), 72) # contact not supported + self.assertIsNone(t.heart_rate(b"\x04\x48")) # no contact + self.assertIsNone(t.heart_rate(b"\x00\x00")) + + def test_hrs_malformed(self): + for packet in (b"", b"\x00", b"\x01\x48", b"\x08\x48\x00", b"\x10\x48", + b"\x10\x48\x00", b"\x00\x48\x01", b"\xe0\x48"): + with self.subTest(packet=packet), self.assertRaises(ValueError): + t.heart_rate(packet) + + def test_gaze_coordinates(self): + self.assertEqual(t.gaze_angles([0, 0, 0, 1]), (0, 0)) + angle = math.radians(15) + pitch, yaw = t.gaze_angles([math.sin(angle), 0, 0, math.cos(angle)]) + self.assertAlmostEqual(pitch, -30) # OpenXR +X rotation looks up + self.assertAlmostEqual(yaw, 0) + pitch, yaw = t.gaze_angles([0, -math.sin(angle), 0, math.cos(angle)]) + self.assertAlmostEqual(pitch, 0) + self.assertAlmostEqual(yaw, 30) # right + + def test_invalid_gaze(self): + for pose in ([0, 0, 0, 0], [math.nan, 0, 0, 1], [0, 0, 0], [0, 0, 0, math.inf]): + with self.assertRaises(ValueError): + t.gaze_angles(pose) + + def test_osc_wire(self): + self.assertEqual(t.osc_message('/x', [72]), b'/x\0\0,i\0\0' + struct.pack('>i', 72)) + self.assertEqual(t.osc_message('/x', [1.0, -2.0]), b'/x\0\0,ff\0' + struct.pack('>ff', 1, -2)) + for address in ('x', '/x\0y', '/x y', '/x*'): + with self.assertRaises(ValueError): + t.osc_message(address, [1]) + + def test_default_never_opens_socket(self): + with patch.object(t.socket, 'socket') as create: + osc = t.Osc() + osc.send('/x', [72]) + osc.close() + create.assert_not_called() + + def test_only_configured_endpoint(self): + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as receiver: + receiver.bind(('127.0.0.1', 0)) + receiver.settimeout(1) + osc = t.Osc(receiver.getsockname()) + try: + osc.send('/tracking/eye/CenterPitchYaw', [0.0, 30.0]) + self.assertEqual(receiver.recv(1024), t.osc_message('/tracking/eye/CenterPitchYaw', [0.0, 30.0])) + finally: + osc.close() + + def test_endpoint_validation(self): + for endpoint in [('example.org', 9000), ('0.0.0.0', 9000), ('224.0.0.1', 9000), ('127.0.0.1', 0), ('::1', 65536)]: + with self.assertRaises(ValueError): + t.Osc(endpoint) + + def test_heart_staleness_contact_and_log(self): + now = [0] + osc = Mock() + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / 'session.csv' + session = t.HeartSession(osc, '/hr', path, lambda: now[0]) + self.assertIsNone(session.current()) + session.notification(b'\x00\x48') + self.assertEqual(session.current(), 72) + osc.send.assert_called_once_with('/hr', [72]) + now[0] = 6 + self.assertIsNone(session.current()) + session.notification(b'\x04\x48') + self.assertIsNone(session.current()) + self.assertEqual(osc.send.call_count, 1) + session.close() + self.assertEqual(path.read_text().splitlines()[0], 'unix_seconds,bpm') + self.assertEqual(len(path.read_text().splitlines()), 2) + if os.name != 'nt': + self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o600) + with self.assertRaises(FileExistsError): + t.HeartSession(osc, '/hr', path) + + def test_no_log_by_default(self): + with patch.object(t.os, 'open') as create: + session = t.HeartSession(Mock(), '/hr') + session.notification(b'\x00\x48') + session.close() + create.assert_not_called() + + +class FakeBluez(unittest.TestCase): + def setUp(self): + self.device = '/org/bluez/hci0/dev_TEST' + self.service = self.device + '/service1' + self.char = self.service + '/char1' + self.objects = { + self.device: {t.DEVICE: {'Address': 'AA:BB:CC:DD:EE:FF', 'Connected': False, 'ServicesResolved': True}}, + self.service: {t.SERVICE: {'UUID': t.HRS, 'Device': self.device}}, + self.char: {t.CHARACTERISTIC: {'UUID': t.MEASUREMENT, 'Service': self.service, 'Flags': ['notify']}}, + } + self.api = Mock() + self.api.GetManagedObjects.side_effect = lambda: self.objects + self.bus = Mock() + self.interface = Mock(return_value=self.api) + self.values = Mock() + + def reader(self): + return t.BluezHeart(self.bus, self.interface, 'AA:BB:CC:DD:EE:FF', self.values) + + def test_subscribe_receive_and_cleanup(self): + reader = self.reader() + self.api.Connect.assert_called_once() + self.assertTrue(reader.subscribe()) + self.api.StartNotify.assert_called_once() + reader.changed(t.CHARACTERISTIC, {'Value': [0, 72]}, [], self.char) + self.values.assert_called_once_with([0, 72]) + reader.changed(t.CHARACTERISTIC, {'Value': [0, 73]}, [], '/other/strap') + self.assertEqual(self.values.call_count, 1) + reader.close() + self.api.StopNotify.assert_called_once() + self.api.Disconnect.assert_called_once() + self.bus.add_signal_receiver.return_value.remove.assert_called_once() + + def test_preserve_existing_connection(self): + self.objects[self.device][t.DEVICE]['Connected'] = True + reader = self.reader() + reader.subscribe() + reader.close() + self.api.Connect.assert_not_called() + self.api.Disconnect.assert_not_called() + + def test_only_selected_device_service(self): + self.objects[self.service][t.SERVICE]['Device'] = '/other/device' + reader = self.reader() + try: + with self.assertRaises(RuntimeError): + reader.subscribe() + finally: + reader.close() + self.api.StartNotify.assert_not_called() + + def test_wait_for_services(self): + self.objects[self.device][t.DEVICE]['ServicesResolved'] = False + reader = self.reader() + self.assertFalse(reader.subscribe()) + reader.close() + self.api.StartNotify.assert_not_called() + self.api.StopNotify.assert_not_called() + + def test_disconnect_notification(self): + reader = self.reader() + reader.changed(t.DEVICE, {'Connected': 0}, [], self.device) # dbus.Boolean behaves as int + self.values.assert_called_once_with(None) + reader.close() + + def test_failed_notify_cleans_connection(self): + reader = self.reader() + self.api.StartNotify.side_effect = RuntimeError('failure') + with self.assertRaises(RuntimeError): + reader.subscribe() + reader.close() + self.api.StopNotify.assert_not_called() + self.api.Disconnect.assert_called_once() + + def test_unknown_device_does_not_connect_or_scan(self): + self.objects.clear() + with self.assertRaises(RuntimeError): + self.reader() + self.api.Connect.assert_not_called() + self.api.StartDiscovery.assert_not_called()