mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-06 01:00:18 +02:00
Compare commits
8
Commits
v0.4.0
...
tracking-osc
| Author | SHA1 | Date | |
|---|---|---|---|
 saphidandClaude Sonnet 5.5
|
8fb00b263c | ||
|
saphidandClaude Sonnet 5.5
|
4937caf310 | ||
|
saphidandClaude Sonnet 5.5
|
128e7a4c94 | ||
|
saphidandClaude Opus 5.5
|
1a439f05c2 | ||
|
saphidandClaude Opus 5.5
|
016d2b8c2d | ||
|
saphidandClaude Opus 5.5
|
42ec68ed51 | ||
|
saphidandClaude Opus 5.5
|
fa49fab5bf | ||
|
saphid
|
ce9e476ec7 |
No files matched your search
@@ -204,6 +204,7 @@ Frame's software fits together, all checked against a real headset and labelled
|
||||
| [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes |
|
||||
| [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing |
|
||||
| [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset |
|
||||
| [Eye tracking and heart rate](docs/tracking.md) | Our OpenXR → OSC bridge, BlueZ heart-rate panel and optional local session log; SlimeVR feasibility notes |
|
||||
| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, and the headset smoke test |
|
||||
| [Open questions](docs/open-questions.md) | What's still unchecked |
|
||||
|
||||
|
||||
@@ -20,6 +20,15 @@ SteamVR (vrserver, vrcompositor, vrdashboard) ← renders the room + pane
|
||||
Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 3056000
|
||||
```
|
||||
|
||||
## Tracking additions (verified 2026-09-28)
|
||||
|
||||
On SteamOS 0.4.1, build `20260925.6191901`, SteamVR exposes combined gaze
|
||||
through `XR_EXT_eye_gaze_interaction` in a headless OpenXR 1.0 session. Our
|
||||
reader obtained valid tracked samples and sent OSC to a configured loopback
|
||||
receiver. BlueZ LE discovery works; GTK4/GI can render our heart-rate panel.
|
||||
No BLE strap or SlimeVR trackers were attached. See [tracking](tracking.md)
|
||||
for the evidence, privacy defaults and untested integration boundaries.
|
||||
|
||||
## Facts worth knowing
|
||||
|
||||
| Fact | Where it matters |
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 19 KiB |
@@ -148,3 +148,10 @@ For example, on 2026-09-27 the smoke test found that Steam's `create-shortcut`
|
||||
refuses ids with a hyphen (`missing/invalid arguments`), which the fake had
|
||||
accepted. The fake now refuses them the same way, and Frame Control makes ids
|
||||
Steam accepts.
|
||||
|
||||
## Tracking protocols and fake BlueZ
|
||||
|
||||
`tests/test_tracking.py` exercises our gaze conversion, OSC sender, HRS parser
|
||||
and BlueZ lifecycle with an in-memory fake object tree. It runs in the normal
|
||||
unit suite without Bluetooth, GTK or OpenXR. Real Frame results and the absent
|
||||
strap/tracker boundaries are recorded in [tracking](tracking.md).
|
||||
@@ -0,0 +1,329 @@
|
||||
# Eye tracking and heart rate
|
||||
|
||||
Frame Control's own tools run on the Frame, using OpenXR and BlueZ. No
|
||||
VRCFaceTracking, LunaHR, Pulsoid or other tracking app is required. This is a
|
||||
command-line first version; it does not add a desktop app tab.
|
||||
|
||||
## What was checked
|
||||
|
||||
**Verified 2026-09-28**, on a real aarch64 Frame running SteamOS 0.4.1,
|
||||
BUILD_ID `20260925.6191901`:
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| OpenXR gaze | SteamVR advertises `XR_EXT_eye_gaze_interaction`, `XR_MND_headless` and `XR_KHR_convert_timespec_time`. `supportsEyeGazeInteraction=1`. A headless session reached FOCUSED and produced 269 valid, tracked orientations in the first ten-second probe. |
|
||||
| Our gaze → OSC bridge | A separate ten-second run produced 280 valid samples and 280 correctly padded 44-byte `/tracking/eye/CenterPitchYaw` messages at an explicitly configured loopback receiver. Only counters and packet-layout checks were retained. |
|
||||
| Bluetooth stack | BlueZ active, adapter powered, central/peripheral roles available. LE discovery started and stopped successfully. No pairing or adapter power settings changed. |
|
||||
| Our heart-rate panel | GTK4/GI runs on the stock image. A **synthetic 72 BPM** notification displayed in our X11 window, tagged `STEAM_GAME=2000000027`. Window capture checked; no real heart-rate measurement was taken. |
|
||||
| SlimeVR, separate feasibility check | Native aarch64 server v21.1.0 ran with an isolated Temurin 21 JRE, created its driver sockets and accepted a local TCP connection on port 21110. Driver v6.0.0 loaded with all shared libraries resolved; `HmdDriverFactory("IServerTrackedDeviceProvider_004")` returned a non-null provider and error 0. |
|
||||
|
||||
|
||||
|
||||
The image is a capture of our own Frame window using a fake notification,
|
||||
not a real sensor reading.
|
||||
|
||||
**Untested:** a real BLE strap's notifications, physical fit/contact behaviour,
|
||||
end-to-end heart-rate display/OSC/log with a strap, avatar response in VRChat,
|
||||
gaze accuracy/calibration, coexistence with every immersive app, in-headset
|
||||
panel placement, SlimeVR tracker/calibration data and the SlimeVR driver running
|
||||
inside SteamVR. No trackers or strap are attached. The driver was loaded in a
|
||||
separate process; it was **not registered or activated in SteamVR**. Steam and
|
||||
SteamVR were not stopped or restarted.
|
||||
|
||||
**Verified blocker resolved:** importing `tkinter` fails because `libtk8.6.so`
|
||||
is absent. The panel uses the installed GTK4/GI bindings instead. The Frame's
|
||||
OpenXR headers advertise a newer API version than the runtime accepts; our
|
||||
reader requests OpenXR 1.0 explicitly.
|
||||
|
||||
## Install our tools
|
||||
|
||||
From this checkout on your computer, while the Frame is awake:
|
||||
|
||||
```sh
|
||||
python3 scripts/tracking-on-frame.py install
|
||||
```
|
||||
|
||||
This copies our Python code and compiles our small C OpenXR reader into
|
||||
`~/.local/share/frame-control/tracking/` on the Frame. It uses the Frame's
|
||||
existing compiler, OpenXR headers/loader, Python, dbus-python, GI and GTK4.
|
||||
Nothing is downloaded, and no sudo, driver registration, system setting,
|
||||
service or autostart is added. `FRAME_ALIAS` can select another SSH alias.
|
||||
The desktop app/server keeps its existing stdlib-only dependency set.
|
||||
|
||||
## Eye tracking → OSC
|
||||
|
||||
Start with a ten-second capability/data-availability check:
|
||||
|
||||
```sh
|
||||
python3 scripts/tracking-on-frame.py gaze --seconds 10
|
||||
```
|
||||
|
||||
This prints support, session-state numbers and sample counters. It opens no
|
||||
OSC socket and prints no gaze coordinates. Exit 0 means at least one valid
|
||||
sample, 3 means no valid sample was observed, and 1 means an API/runtime error.
|
||||
If there are no valid samples, wake/wear the headset and check its tracking
|
||||
setup; a successful capability check alone does not prove usable gaze.
|
||||
|
||||
To send to VRChat running **on the Frame**, explicitly enable OSC in VRChat
|
||||
and choose its local UDP endpoint:
|
||||
|
||||
```sh
|
||||
python3 scripts/tracking-on-frame.py gaze --seconds 3600 --osc 127.0.0.1 9000
|
||||
```
|
||||
|
||||
For a receiver on another computer, replace `127.0.0.1` with that computer's
|
||||
IP address and choose its listening port. Addresses are IP literals (IPv4 or
|
||||
IPv6); there is no discovery or default destination. Loopback here always
|
||||
means **the Frame**, not the computer running the SSH command. OSC uses
|
||||
unencrypted UDP: configure only a receiver you intend to receive this data.
|
||||
|
||||
**Documented:** [VRChat's eye OSC interface](https://docs.vrchat.com/docs/osc-eye-tracking)
|
||||
accepts `/tracking/eye/CenterPitchYaw` with two floats in degrees, positive down
|
||||
and right. We locate OpenXR's combined gaze pose relative to VIEW (the head),
|
||||
rotate its -Z forward vector and convert that direction to these angles.
|
||||
Only active, orientation-valid **and tracked** samples are sent, at up to
|
||||
30 Hz. No eyelid/blink, individual-eye or face values are invented. We do not
|
||||
send neutral gaze on tracking loss; VRChat's documented timeout restores its
|
||||
automatic eye behaviour after input stops.
|
||||
|
||||
**Privacy:** gaze is personal data. It stays in process memory and a private
|
||||
pipe between our reader and bridge. There is no gaze log option, telemetry,
|
||||
OSC receiver or raw gaze on stdout/stderr. Only an explicit `--osc IP PORT`
|
||||
opens an output socket. Runtime diagnostics and validity counters are not
|
||||
measurements. Stop with Ctrl-C or let `--seconds` expire (maximum 24 hours).
|
||||
A lost headless session ends the run; it does not silently reconnect.
|
||||
|
||||
## BLE heart rate → our panel, OSC and optional log
|
||||
|
||||
First discover/pair your strap in SteamOS's Bluetooth settings. Select that
|
||||
strap's Bluetooth address explicitly; our tool does not scan for or connect
|
||||
to arbitrary nearby devices.
|
||||
|
||||
```sh
|
||||
python3 scripts/tracking-on-frame.py heart \
|
||||
--device AA:BB:CC:DD:EE:FF --panel --seconds 3600
|
||||
```
|
||||
|
||||
This uses BlueZ's standard Heart Rate Service (`180d`) and Heart Rate
|
||||
Measurement (`2a37`) notifications. It finds the characteristic only beneath
|
||||
the selected device's HRS service. The reader handles 8- and 16-bit BPM,
|
||||
contact flags and optional energy/RR fields; energy and RR intervals are
|
||||
validated for length but discarded. Zero BPM, reported loss of skin contact,
|
||||
malformed packets and readings older than five seconds are not shown as a
|
||||
current measurement. A disconnect stops the run; reconnect and start again.
|
||||
This is a social/fitness readout, not a medical monitor.
|
||||
|
||||
The panel is our GTK4 window on gamescope's X display. Use SteamVR's panel
|
||||
controls to float/dock it (see [panels](panels.md)). **Stop**, closing the panel,
|
||||
Ctrl-C, SSH hangup or the duration limit ends our subscription. A connection
|
||||
that was already open when we started is preserved; a connection we opened
|
||||
is disconnected on exit. No Bluetooth power or pairing state is changed.
|
||||
|
||||
Add either output explicitly:
|
||||
|
||||
```sh
|
||||
python3 scripts/tracking-on-frame.py heart \
|
||||
--device AA:BB:CC:DD:EE:FF --panel --seconds 3600 \
|
||||
--osc 127.0.0.1 9000 --address /avatar/parameters/HeartRate \
|
||||
--log /home/steamos/heart-session.csv
|
||||
```
|
||||
|
||||
The OSC value is integer BPM. `HeartRate` is a chosen avatar parameter, **not a
|
||||
built-in VRChat heart-rate feature**; your avatar/receiver must define the
|
||||
matching parameter. `--address` can select another literal OSC path. The local
|
||||
panel works without OSC, a log or an avatar integration.
|
||||
|
||||
The optional CSV contains only `unix_seconds,bpm`. It is created privately
|
||||
(mode 0600), refuses existing files/symlinks, and lives **on the Frame** at the
|
||||
path you specify. Nothing is logged by default, and heart-rate values are not
|
||||
printed to the terminal. Delete your session file when you no longer need it.
|
||||
|
||||
### Checking heart rate against a reference
|
||||
|
||||
`scripts/heart-check.py` runs on your computer. `listen` shows our OSC
|
||||
readings live as they arrive, so you can watch them next to another device:
|
||||
|
||||
```sh
|
||||
python3 scripts/heart-check.py listen --port 9000 --out ours.csv
|
||||
```
|
||||
|
||||
Point the Frame at it with `--osc <your computer's IP> 9000`. `compare` lines
|
||||
up two recordings by time and reports the mean difference, bias, the share
|
||||
within ±5 BPM and the delay between them. It passes when the mean difference
|
||||
is at most 5 BPM, at least 80% of reference readings are matched and nothing
|
||||
was shown while the sensor reported lost skin contact:
|
||||
|
||||
```sh
|
||||
python3 scripts/heart-check.py compare ours.csv reference.csv
|
||||
python3 scripts/heart-check.py compare ours.csv ~/Downloads/export.zip
|
||||
```
|
||||
|
||||
The reference can be a CSV (`time,bpm[,flags]`, time in unix seconds or ISO
|
||||
8601) or an Apple Health export (`export.zip` or `export.xml`). Only heart-rate
|
||||
records within the recording's time range are read. Everything stays on your
|
||||
computer.
|
||||
|
||||
`scripts/heart-test-strap.swift` turns a Mac into a synthetic strap. It
|
||||
advertises the standard Heart Rate Service and sends a fixed, known sequence
|
||||
(8-bit and 16-bit values and a skin-contact loss), printing each sent value, so
|
||||
`compare` can check that the Frame shows exactly what was sent. It needs
|
||||
Bluetooth permission for the process that runs it. **Untested on 2026-09-29:**
|
||||
it compiled, but on this Mac, launched from an agent session, macOS never
|
||||
delivered a Bluetooth state and no permission prompt appeared, so it never
|
||||
advertised.
|
||||
|
||||
## Pulse from the eye cameras (experimental)
|
||||
|
||||
The Frame has no heart-rate sensor. **Verified 2026-09-29** (SteamOS 0.4.1,
|
||||
build `20260925.6191901`): its sensors are an ambient light/proximity sensor
|
||||
(`vcnl4000`), a hall sensor (`als31300`), two passthrough cameras
|
||||
(`arcimx616`), two tracking cameras (`og01a1b`) and two IR eye cameras
|
||||
(`og0ve10`). There is no optical heart-rate (PPG) sensor.
|
||||
|
||||
The experiment asks whether the eye cameras can see a pulse anyway. With each
|
||||
heartbeat, the blood volume in the skin around the eye changes slightly and
|
||||
its IR reflectance changes with it. This is camera-based photoplethysmography;
|
||||
near-IR works, though the signal is weaker than in green light.
|
||||
|
||||
```sh
|
||||
python3 scripts/tracking-on-frame.py pulse --seconds 60 --show
|
||||
```
|
||||
|
||||
How it works:
|
||||
|
||||
- **Capture (verified).** SteamVR ships `eyetracking --calib N`, which saves
|
||||
both eye cameras for N seconds as 400×400 8-bit IR PNGs with a monotonic
|
||||
timestamp per frame, at about 90 fps per eye. SteamVR's live eye tracker,
|
||||
part of `steamvr.service`, gets its frames from the DSP and stops its
|
||||
cameras when the headset is off. Unworn captures ran alongside it: its PID
|
||||
and log were unchanged and our OpenXR gaze session still started
|
||||
afterwards. **Untested:** whether the capture and the live tracker coexist
|
||||
while the headset is worn and tracking.
|
||||
- **Privacy.** Each image is reduced to a 16×16 grid of patch averages as
|
||||
soon as it is complete, then deleted. Three worker processes do this beside
|
||||
the capture. If more than 900 images (about five seconds) ever wait, we stop
|
||||
reading them and delete them undecoded until the capture ends, and report
|
||||
an error. The capture directory is removed on exit, even after errors. No image is kept or leaves the Frame. The estimate
|
||||
is printed only with `--show`, and sent or saved only with `--osc` or
|
||||
`--log`, as for the strap.
|
||||
- **Estimate.** Patch traces are averaged down to 15 Hz and turned into
|
||||
relative change. A 2-second moving median removes drift and blinks.
|
||||
Patches with frequent spikes (the eyeball and eyelid) are dropped, as are
|
||||
dark or saturated ones. The 20% of patches with the clearest rhythm between
|
||||
42 and 180 BPM are combined in the frequency domain. Output is an overall
|
||||
estimate plus one estimate per second over 15-second windows. A result
|
||||
counts as **clear** only when the top patches agree and the combined signal
|
||||
stands out from the noise. Otherwise the command exits 3 and sends no OSC.
|
||||
`--log` still records the per-second estimates, so a comparison shows how
|
||||
far off an unclear result was.
|
||||
The thresholds are provisional until checked on real wearers.
|
||||
|
||||
**Verified on the Frame, unworn, 2026-09-29:** captures of 3,600-5,400 eye
|
||||
frames never had more than 8 images on disk, finished a few seconds after the
|
||||
capture ended and left no capture directory. **The estimator alone gave a
|
||||
false "clear" pulse.** With nobody wearing the headset, five runs reported a
|
||||
steady, self-consistent rhythm (90, 90, 93, 94 and 96 BPM; patch agreement
|
||||
100%, signal/noise 0.63-0.70), and earlier runs reported 127-129 BPM at lower
|
||||
signal/noise. It is a periodic camera or illumination artifact, and its
|
||||
frequency drifts between runs. A wearer-less scene cannot contain a pulse, so
|
||||
the signal/noise gate cannot tell this artifact from one. Because of that,
|
||||
`pulse` reads the Frame's proximity sensor (`vcnl4000`) before and after the
|
||||
capture. It reads about 3 unworn (**verified**). If either reading is below 20
|
||||
the result is never called clear, nothing is sent over OSC, and the command
|
||||
exits 3. **Inferred, unmeasured:** that a worn reading is well above 20; the
|
||||
cut-off is provisional until someone wears the headset. If the sensor can't be
|
||||
read, the guard is skipped. Confirming a real pulse also needs a reference
|
||||
(below).
|
||||
|
||||
**Do not interrupt the capture. Verified on the Frame, 2026-09-29:** sending
|
||||
SIGTERM to `eyetracking --calib` left the DSP service's eye camera (OV6211)
|
||||
stuck "streaming": its log had no "Stopping streaming" line, and every later
|
||||
request failed with "Failed to start streaming". Head tracking kept working.
|
||||
Clearing it needs the DSP service restarted or the Frame rebooted, so `pulse`
|
||||
never signals the tool. After Ctrl-C or a failure it keeps deleting images
|
||||
until the tool ends by itself (at most the `--seconds` plus a few seconds),
|
||||
and only kills a tool that overruns by 30 s, with a warning that the eye
|
||||
cameras may need a reboot. So an interrupted run can take a while to return.
|
||||
|
||||
**Verified on synthetic data** (unit tests): a 0.3% brightness pulse in a
|
||||
third of the patches, with noise, drift, blinks and eye movement, is
|
||||
recovered within 1.5 BPM at 58, 72 and 115 BPM; noise and blinks alone are
|
||||
not reported as a pulse. **Not yet verified:** whether a real wearer's eye
|
||||
images contain a usable pulse, and how accurate it is. That needs someone
|
||||
wearing the headset and a reference, as below.
|
||||
|
||||
### Comparing with an Apple Watch
|
||||
|
||||
1. On the watch, start a workout (for example **Other**) so it measures heart
|
||||
rate every few seconds rather than occasionally.
|
||||
2. Put the Frame on, sit still and look ahead. Run:
|
||||
|
||||
```sh
|
||||
python3 scripts/tracking-on-frame.py pulse --seconds 120 --show \
|
||||
--log /home/steamos/pulse.csv
|
||||
```
|
||||
|
||||
The per-second estimates print at the end. Compare them with what the
|
||||
watch showed.
|
||||
3. End the workout. On the iPhone, open Health → your picture → **Export All
|
||||
Health Data**, and AirDrop `export.zip` to the Mac.
|
||||
4. On the Mac:
|
||||
|
||||
```sh
|
||||
scp frame:pulse.csv . && ssh frame rm pulse.csv
|
||||
python3 scripts/heart-check.py compare pulse.csv ~/Downloads/export.zip
|
||||
```
|
||||
|
||||
This first version analyses after the capture ends, because the method must
|
||||
prove itself before a live panel is worth building. The Apple Watch is a
|
||||
reference, not ground truth: in workouts it is typically within a few BPM of
|
||||
a chest strap when you are still.
|
||||
|
||||
## SlimeVR: feasibility only
|
||||
|
||||
SlimeVR is an independent application stack. Neither of our features installs,
|
||||
launches or depends on it. Users who want it can follow
|
||||
[SlimeVR's setup documentation](https://docs.slimevr.dev/server/index.html).
|
||||
The consented upstream releases tested were
|
||||
[server v21.1.0](https://github.com/SlimeVR/SlimeVR-Server/releases/tag/v21.1.0)
|
||||
and [driver v6.0.0](https://github.com/SlimeVR/SlimeVR-OpenVR-Driver/releases/tag/v6.0.0),
|
||||
under SlimeVR's MIT/Apache-2.0 licensing.
|
||||
|
||||
**Verified layout, read-only:** the Frame's registered runtime is `/opt/steamvr`;
|
||||
its native driver is `drivers/cv/bin/linuxarm64/driver_cv.so`, with a
|
||||
`drivers/cv/driver.vrdrivermanifest`. Frame controller manifests/resources are
|
||||
under `drivers/frame_controller/`. Configuration is under
|
||||
`~/.config/openvr/config/`, not the Steam client's config directory. The
|
||||
SlimeVR release also uses `slimevr/bin/linuxarm64/driver_slimevr.so` plus its
|
||||
manifest. Nothing in those installed SteamVR directories was changed.
|
||||
|
||||
**Inferred:** the matching ABI/layout and standalone factory success make
|
||||
SteamVR integration plausible. They do not prove successful driver `Init`,
|
||||
server/driver IPC, tracking, or calibration. That needs a separate integration
|
||||
check with hardware and an agreed SteamVR restart. No Java executable was on
|
||||
PATH for this check, so an isolated JRE was used. SlimeVR's server opens LAN
|
||||
listeners; our temporary server was stopped and the temporary downloads,
|
||||
configuration and logs were removed. It is not left installed or running.
|
||||
|
||||
## Tests and remaining checks
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -s tests
|
||||
```
|
||||
|
||||
`tests/test_tracking.py` covers HRS packet parsing, contact/staleness, OSC
|
||||
padding/types and a real loopback socket, quaternion signs, opt-in networking,
|
||||
private/exclusive logging and a fake BlueZ object tree. The fake checks service
|
||||
ownership, notification routing, delayed GATT discovery and connection cleanup.
|
||||
It does not pretend to be a physical strap or a real OpenXR runtime.
|
||||
|
||||
Before calling hardware support complete, attach a strap and check BPM against
|
||||
its own display/reference, loss of contact, disconnect/reconnect, Stop, OSC and
|
||||
CSV together. Check avatar eyes while looking up/down/left/right in a supported
|
||||
VRChat session. No third-party tracking app is needed for either test.
|
||||
|
||||
Independent review attempt: `devin -p --model swe-2-max` with the frozen diff,
|
||||
contribution standards and read-only instructions returned no output for ten
|
||||
minutes. It was terminated with exit 143. No completed review or actual model
|
||||
identity was returned; hardware checks and independent review remain follow-up
|
||||
work before making the draft ready.
|
||||
@@ -0,0 +1,135 @@
|
||||
/* Frame Control's OpenXR gaze source. No values on stdout/stderr or disk.
|
||||
* The Python bridge supplies a private pipe with --fd; standalone probes only
|
||||
* report counters. Uses a headless session, never submits frames or takes focus. */
|
||||
#define XR_USE_TIMESPEC
|
||||
#include <time.h>
|
||||
#include <openxr/openxr.h>
|
||||
#include <openxr/openxr_platform.h>
|
||||
#include <signal.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <unistd.h>
|
||||
|
||||
static volatile sig_atomic_t stopped;
|
||||
static void stop(int sig) { (void)sig; stopped = 1; }
|
||||
#define CHECK(call) do { result = (call); if (XR_FAILED(result)) { \
|
||||
fprintf(stderr, "%s failed (%d)\n", #call, result); goto cleanup; } } while (0)
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
int seconds = 10, fd = -1, running = 0, rc = 1;
|
||||
unsigned active = 0, valid = 0, samples = 0;
|
||||
for (int i = 1; i < argc; i++) {
|
||||
if (!strcmp(argv[i], "--seconds") && i+1 < argc) seconds = atoi(argv[++i]);
|
||||
else if (!strcmp(argv[i], "--fd") && i+1 < argc) fd = atoi(argv[++i]);
|
||||
else { fprintf(stderr, "usage: gaze [--seconds 1..86400] [--fd private-pipe]\n"); return 2; }
|
||||
}
|
||||
if (seconds < 1 || seconds > 86400 || (fd != -1 && fd < 3)) return 2;
|
||||
FILE *out = fd == -1 ? NULL : fdopen(fd, "w");
|
||||
if (fd != -1 && !out) return 2;
|
||||
signal(SIGINT, stop); signal(SIGTERM, stop); signal(SIGHUP, stop); signal(SIGPIPE, SIG_IGN);
|
||||
XrResult result;
|
||||
XrInstance instance = XR_NULL_HANDLE;
|
||||
XrSession session = XR_NULL_HANDLE;
|
||||
XrActionSet set = XR_NULL_HANDLE;
|
||||
XrSpace gaze = XR_NULL_HANDLE, view = XR_NULL_HANDLE;
|
||||
const char *extensions[] = {"XR_EXT_eye_gaze_interaction", "XR_MND_headless", "XR_KHR_convert_timespec_time"};
|
||||
XrInstanceCreateInfo create = {.type = XR_TYPE_INSTANCE_CREATE_INFO};
|
||||
strcpy(create.applicationInfo.applicationName, "Frame Control gaze");
|
||||
create.applicationInfo.apiVersion = XR_MAKE_VERSION(1, 0, 0);
|
||||
create.enabledExtensionCount = 3; create.enabledExtensionNames = extensions;
|
||||
CHECK(xrCreateInstance(&create, &instance));
|
||||
XrSystemGetInfo get = {.type = XR_TYPE_SYSTEM_GET_INFO, .formFactor = XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY};
|
||||
XrSystemId system;
|
||||
CHECK(xrGetSystem(instance, &get, &system));
|
||||
XrSystemEyeGazeInteractionPropertiesEXT eye = {.type = XR_TYPE_SYSTEM_EYE_GAZE_INTERACTION_PROPERTIES_EXT};
|
||||
XrSystemProperties props = {.type = XR_TYPE_SYSTEM_PROPERTIES, .next = &eye};
|
||||
CHECK(xrGetSystemProperties(instance, system, &props));
|
||||
printf("supportsEyeGazeInteraction=%u\n", eye.supportsEyeGazeInteraction);
|
||||
if (!eye.supportsEyeGazeInteraction) goto cleanup;
|
||||
XrSessionCreateInfo sc = {.type = XR_TYPE_SESSION_CREATE_INFO, .systemId = system};
|
||||
CHECK(xrCreateSession(instance, &sc, &session));
|
||||
XrActionSetCreateInfo asc = {.type = XR_TYPE_ACTION_SET_CREATE_INFO};
|
||||
strcpy(asc.actionSetName, "gaze"); strcpy(asc.localizedActionSetName, "Gaze");
|
||||
CHECK(xrCreateActionSet(instance, &asc, &set));
|
||||
XrActionCreateInfo ac = {.type = XR_TYPE_ACTION_CREATE_INFO, .actionType = XR_ACTION_TYPE_POSE_INPUT};
|
||||
strcpy(ac.actionName, "gaze_pose"); strcpy(ac.localizedActionName, "Gaze pose");
|
||||
XrAction action;
|
||||
CHECK(xrCreateAction(set, &ac, &action));
|
||||
XrPath profile, input;
|
||||
CHECK(xrStringToPath(instance, "/interaction_profiles/ext/eye_gaze_interaction", &profile));
|
||||
CHECK(xrStringToPath(instance, "/user/eyes_ext/input/gaze_ext/pose", &input));
|
||||
XrActionSuggestedBinding binding = {action, input};
|
||||
XrInteractionProfileSuggestedBinding suggested = {.type = XR_TYPE_INTERACTION_PROFILE_SUGGESTED_BINDING,
|
||||
.interactionProfile = profile, .countSuggestedBindings = 1, .suggestedBindings = &binding};
|
||||
CHECK(xrSuggestInteractionProfileBindings(instance, &suggested));
|
||||
XrSessionActionSetsAttachInfo attach = {.type = XR_TYPE_SESSION_ACTION_SETS_ATTACH_INFO, .countActionSets = 1, .actionSets = &set};
|
||||
CHECK(xrAttachSessionActionSets(session, &attach));
|
||||
XrActionSpaceCreateInfo space = {.type = XR_TYPE_ACTION_SPACE_CREATE_INFO, .action = action, .poseInActionSpace.orientation.w = 1};
|
||||
CHECK(xrCreateActionSpace(session, &space, &gaze));
|
||||
XrReferenceSpaceCreateInfo ref = {.type = XR_TYPE_REFERENCE_SPACE_CREATE_INFO, .referenceSpaceType = XR_REFERENCE_SPACE_TYPE_VIEW,
|
||||
.poseInReferenceSpace.orientation.w = 1};
|
||||
CHECK(xrCreateReferenceSpace(session, &ref, &view));
|
||||
PFN_xrConvertTimespecTimeToTimeKHR convert;
|
||||
CHECK(xrGetInstanceProcAddr(instance, "xrConvertTimespecTimeToTimeKHR", (PFN_xrVoidFunction *)&convert));
|
||||
struct timespec start, now;
|
||||
clock_gettime(CLOCK_MONOTONIC, &start);
|
||||
while (!stopped) {
|
||||
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||
if (now.tv_sec - start.tv_sec >= seconds) break;
|
||||
XrEventDataBuffer event = {.type = XR_TYPE_EVENT_DATA_BUFFER};
|
||||
while ((result = xrPollEvent(instance, &event)) == XR_SUCCESS) {
|
||||
if (event.type == XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED) {
|
||||
XrSessionState state = ((XrEventDataSessionStateChanged *)&event)->state;
|
||||
printf("sessionState=%d\n", state); fflush(stdout);
|
||||
if (state == XR_SESSION_STATE_READY && !running) {
|
||||
XrSessionBeginInfo begin = {.type = XR_TYPE_SESSION_BEGIN_INFO, .primaryViewConfigurationType = XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO};
|
||||
CHECK(xrBeginSession(session, &begin)); running = 1;
|
||||
} else if (state == XR_SESSION_STATE_STOPPING) {
|
||||
CHECK(xrEndSession(session)); running = 0; stopped = 1;
|
||||
} else if (state == XR_SESSION_STATE_EXITING || state == XR_SESSION_STATE_LOSS_PENDING) stopped = 1;
|
||||
} else if (event.type == XR_TYPE_EVENT_DATA_INSTANCE_LOSS_PENDING) stopped = 1;
|
||||
event.type = XR_TYPE_EVENT_DATA_BUFFER;
|
||||
}
|
||||
if (XR_FAILED(result)) goto cleanup;
|
||||
if (running && !stopped) {
|
||||
XrActiveActionSet activeSet = {set, XR_NULL_PATH};
|
||||
XrActionsSyncInfo sync = {.type = XR_TYPE_ACTIONS_SYNC_INFO, .countActiveActionSets = 1, .activeActionSets = &activeSet};
|
||||
CHECK(xrSyncActions(session, &sync));
|
||||
if (result != XR_SUCCESS) {
|
||||
struct timespec delay = {.tv_nsec = 33333333};
|
||||
nanosleep(&delay, NULL);
|
||||
continue; /* No stale gaze when the runtime denies focus. */
|
||||
}
|
||||
XrActionStateGetInfo ag = {.type = XR_TYPE_ACTION_STATE_GET_INFO, .action = action};
|
||||
XrActionStatePose pose = {.type = XR_TYPE_ACTION_STATE_POSE};
|
||||
CHECK(xrGetActionStatePose(session, &ag, &pose));
|
||||
samples++;
|
||||
if (pose.isActive) {
|
||||
active++;
|
||||
XrTime time; CHECK(convert(instance, &now, &time));
|
||||
XrSpaceLocation location = {.type = XR_TYPE_SPACE_LOCATION};
|
||||
CHECK(xrLocateSpace(gaze, view, time, &location));
|
||||
XrSpaceLocationFlags needed = XR_SPACE_LOCATION_ORIENTATION_VALID_BIT | XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT;
|
||||
if ((location.locationFlags & needed) == needed) {
|
||||
valid++;
|
||||
if (out) {
|
||||
XrQuaternionf q = location.pose.orientation;
|
||||
if (fprintf(out, "%g %g %g %g\n", q.x, q.y, q.z, q.w) < 0 || fflush(out)) goto cleanup;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
struct timespec delay = {.tv_nsec = 33333333}; nanosleep(&delay, NULL);
|
||||
}
|
||||
printf("samples=%u active=%u valid=%u\n", samples, active, valid);
|
||||
rc = valid ? 0 : 3; /* Distinguish a working session from observed gaze. */
|
||||
cleanup:
|
||||
if (view) xrDestroySpace(view);
|
||||
if (gaze) xrDestroySpace(gaze);
|
||||
if (session) xrDestroySession(session);
|
||||
if (set) xrDestroyActionSet(set);
|
||||
if (instance) xrDestroyInstance(instance);
|
||||
if (out) fclose(out);
|
||||
return rc;
|
||||
}
|
||||
@@ -0,0 +1,481 @@
|
||||
"""Experimental pulse estimate from the Frame's IR eye-tracking cameras.
|
||||
|
||||
Skin brightens and darkens very slightly with each heartbeat as blood volume
|
||||
changes (photoplethysmography). The eye cameras film the skin around each eye
|
||||
under steady IR light at about 90 frames per second, so that rhythm may be
|
||||
visible in the average brightness of small patches of skin.
|
||||
|
||||
Capture uses SteamVR's own `eyetracking --calib` mode, which writes PNG pairs
|
||||
to /tmp. Each image is reduced to a grid of patch averages the moment it is
|
||||
complete, then deleted; the capture directory is removed on exit. No image
|
||||
leaves the Frame or outlives the run. This is an experiment, not a medical
|
||||
measurement.
|
||||
"""
|
||||
from array import array
|
||||
import bisect
|
||||
import cmath
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
ET_DIR = Path("/opt/steamvr/tools/eyetracking")
|
||||
ET_BIN = ET_DIR / "bin/linuxarm64/eyetracking"
|
||||
ET_WEIGHTS = ET_DIR / "resources/et_dsp_20250610_03136.weights"
|
||||
GRID = 16 # 16 × 16 patches of 25 × 25 pixels on the 400 × 400 image
|
||||
RATE = 15.0 # analysis sample rate, Hz; the band of interest ends at 3 Hz
|
||||
LOW, HIGH = 42.0, 180.0 # BPM search band
|
||||
# The Frame's proximity sensor (vcnl4000) reads about 3 with nobody wearing it.
|
||||
# A worn reading has not been measured yet, so the cut-off is provisional.
|
||||
IIO = Path("/sys/bus/iio/devices")
|
||||
WORN_MIN = 20.0
|
||||
WINDOW = 15.0 # seconds per windowed estimate
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- capture ---
|
||||
|
||||
def grid_means(pixels, width, height, stride, channels, grid=GRID):
|
||||
"""Average of channel 0 in each of grid × grid equal patches."""
|
||||
bw, bh = width // grid, height // grid
|
||||
sums = [0] * (grid * grid)
|
||||
for y in range(bh * grid):
|
||||
start = y * stride
|
||||
row = pixels[start:start + width * channels:channels]
|
||||
base = (y // bh) * grid
|
||||
for bx in range(grid):
|
||||
sums[base + bx] += sum(row[bx * bw:(bx + 1) * bw])
|
||||
area = bw * bh
|
||||
return [s / area for s in sums]
|
||||
|
||||
|
||||
def load_grid(path):
|
||||
import gi
|
||||
gi.require_version("GdkPixbuf", "2.0")
|
||||
from gi.repository import GdkPixbuf
|
||||
image = GdkPixbuf.Pixbuf.new_from_file(str(path))
|
||||
return grid_means(image.read_pixel_bytes().get_data(), image.get_width(), image.get_height(),
|
||||
image.get_rowstride(), image.get_n_channels())
|
||||
|
||||
|
||||
def reduce_file(loader, path):
|
||||
"""Reduce one image to its patch grid and delete it, whatever happens."""
|
||||
try:
|
||||
return loader(path)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
class Capture:
|
||||
"""Run SteamVR's eye-camera capture and reduce frames as they arrive."""
|
||||
NAME = re.compile(r"^(left|right)_(\d+)\.png$")
|
||||
# Only SteamVR's own capture directories are read and removed.
|
||||
PREFIX = "/tmp/etcalib_"
|
||||
# Printed by the capture tool; its output is only flushed when it exits.
|
||||
WRITING = re.compile(r"Writing capture to: (\S+)")
|
||||
# About five seconds of frames (~65 MB in RAM-backed /tmp). If reduction
|
||||
# falls further behind than this, the capture stops rather than letting
|
||||
# eye images pile up.
|
||||
MAX_BACKLOG = 900
|
||||
|
||||
def __init__(self, seconds, runner=subprocess.Popen, loader=load_grid, workers=3):
|
||||
self.seconds, self.runner, self.loader, self.workers = seconds, runner, loader, workers
|
||||
self.grids = {"left": {}, "right": {}}
|
||||
self.directory = None
|
||||
self.pool = None
|
||||
self.submitted = set()
|
||||
self.futures = {}
|
||||
self.new = set() # every capture directory that appeared during our run
|
||||
self.before = None # capture directories that existed before the run
|
||||
self.log = None
|
||||
|
||||
def candidates(self):
|
||||
parent, stem = os.path.split(self.PREFIX)
|
||||
return {Path(entry.path) for entry in os.scandir(parent)
|
||||
if entry.name.startswith(stem) and entry.is_dir(follow_symlinks=False)}
|
||||
|
||||
def run(self):
|
||||
# The capture tool's output goes to a private temporary file, read for
|
||||
# failure messages. It is block-buffered, so the capture directory is
|
||||
# found by watching for a new one rather than waiting for its name.
|
||||
import tempfile
|
||||
self.before = before = self.candidates()
|
||||
process = None
|
||||
with tempfile.TemporaryFile("w+") as log:
|
||||
self.log = log
|
||||
try:
|
||||
if self.workers:
|
||||
# SteamVR pins its eye tracker to cores 0-1; decoding runs beside it.
|
||||
import concurrent.futures
|
||||
import multiprocessing
|
||||
self.pool = concurrent.futures.ProcessPoolExecutor(
|
||||
self.workers, mp_context=multiprocessing.get_context("fork"))
|
||||
process = self.runner([str(ET_BIN), "-b", "CDSP", "-w", str(ET_WEIGHTS), "--calib", str(self.seconds)],
|
||||
cwd=str(ET_BIN.parent), stdout=log, stderr=subprocess.STDOUT)
|
||||
deadline = time.monotonic() + self.seconds + 30
|
||||
while True:
|
||||
# Checked on every poll: a second capture directory means
|
||||
# we can't tell which images are ours, so stop.
|
||||
self.new |= self.candidates() - before
|
||||
if len(self.new) > 1:
|
||||
raise RuntimeError("another eye-camera capture is running")
|
||||
if not self.directory and self.new:
|
||||
self.directory = next(iter(self.new))
|
||||
finished = process.poll() is not None
|
||||
self.reduce(final=finished)
|
||||
if finished:
|
||||
break
|
||||
if time.monotonic() > deadline:
|
||||
raise RuntimeError("eye-camera capture did not finish")
|
||||
time.sleep(0.02)
|
||||
log.seek(0)
|
||||
text = log.read()
|
||||
if process.returncode or not self.directory:
|
||||
reason = "cameras unavailable" if "Failed to" in text else f"exit {process.returncode}"
|
||||
raise RuntimeError(f"eye-camera capture failed ({reason})")
|
||||
named = self.written(log)
|
||||
if named and named != self.directory:
|
||||
raise RuntimeError("the capture wrote somewhere else; not using those images")
|
||||
return self.frames()
|
||||
finally:
|
||||
# Each step runs even if an earlier one failed: eye images must
|
||||
# be removed whatever else went wrong.
|
||||
# Ctrl-C and SIGTERM (raised as KeyboardInterrupt) are held
|
||||
# until the images are gone, then re-raised.
|
||||
interrupted, failed = None, None
|
||||
for step in (lambda: self.finish(process),
|
||||
lambda: self.pool and self.pool.shutdown(wait=True, cancel_futures=True),
|
||||
self.adopt_reported):
|
||||
try:
|
||||
step()
|
||||
except BaseException as error:
|
||||
if isinstance(error, Exception):
|
||||
failed = failed or error
|
||||
else:
|
||||
interrupted = interrupted or error
|
||||
self.log = None
|
||||
self.remove()
|
||||
if interrupted:
|
||||
raise interrupted
|
||||
if failed:
|
||||
raise RuntimeError(f"the eye-camera capture did not stop cleanly: {failed}")
|
||||
|
||||
def finish(self, process):
|
||||
"""Let the capture tool end by itself, deleting its images meanwhile.
|
||||
|
||||
Never signal it: stopping the tool early leaves the headset's eye
|
||||
camera stuck streaming until the DSP service restarts (verified on the
|
||||
Frame with SIGTERM). Its run is bounded by `seconds`, so waiting is
|
||||
short. Only if it overruns by a wide margin is it killed."""
|
||||
if not process:
|
||||
return
|
||||
interrupted = None
|
||||
give_up = time.monotonic() + self.seconds + 30
|
||||
while True:
|
||||
try:
|
||||
if process.poll() is not None:
|
||||
break
|
||||
self.discard()
|
||||
if time.monotonic() > give_up:
|
||||
process.kill()
|
||||
process.wait()
|
||||
raise RuntimeError("the eye-camera capture had to be killed; "
|
||||
"the eye cameras may need a reboot to stream again")
|
||||
time.sleep(0.05)
|
||||
except KeyboardInterrupt as error: # keep cleaning up until it ends
|
||||
interrupted = interrupted or error
|
||||
if interrupted:
|
||||
raise interrupted
|
||||
|
||||
def discard(self):
|
||||
"""Delete the eye images written so far, without reading them."""
|
||||
if self.before is not None:
|
||||
self.new |= self.candidates() - self.before
|
||||
for directory in self.new | ({self.directory} if self.directory else set()):
|
||||
if str(directory).startswith(self.PREFIX) and directory.is_dir() and not directory.is_symlink():
|
||||
for entry in os.scandir(directory):
|
||||
if self.NAME.match(entry.name):
|
||||
try:
|
||||
os.unlink(entry.path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def adopt_reported(self):
|
||||
"""The directory the tool reported is ours by its own account."""
|
||||
named = self.written(self.log)
|
||||
if named and str(named).startswith(self.PREFIX):
|
||||
self.new.add(named)
|
||||
|
||||
def written(self, log):
|
||||
"""The directory the capture tool reported, once its output is flushed."""
|
||||
log.seek(0)
|
||||
match = self.WRITING.search(log.read())
|
||||
return Path(match.group(1)) if match else None
|
||||
|
||||
def reduce(self, final=False):
|
||||
"""Reduce and delete every complete image; an image is complete once a
|
||||
later one of the same eye exists, or the capture has ended."""
|
||||
if not self.directory or not self.directory.is_dir():
|
||||
return
|
||||
pending = {"left": [], "right": []}
|
||||
for entry in os.scandir(self.directory):
|
||||
match = self.NAME.match(entry.name)
|
||||
if match:
|
||||
pending[match.group(1)].append((int(match.group(2)), entry.path))
|
||||
if sum(len(files) for files in pending.values()) > self.MAX_BACKLOG:
|
||||
raise RuntimeError("eye-image processing fell behind; capture abandoned")
|
||||
for eye, files in pending.items():
|
||||
files.sort()
|
||||
ready = files if final else files[:-1]
|
||||
for index, path in ready:
|
||||
if path in self.submitted:
|
||||
continue
|
||||
self.submitted.add(path)
|
||||
if self.pool:
|
||||
self.futures[(eye, index)] = self.pool.submit(reduce_file, self.loader, path)
|
||||
else:
|
||||
self.grids[eye][index] = array("f", reduce_file(self.loader, path))
|
||||
for key, future in list(self.futures.items()):
|
||||
if final or future.done():
|
||||
self.grids[key[0]][key[1]] = array("f", future.result())
|
||||
del self.futures[key]
|
||||
|
||||
def frames(self):
|
||||
"""[(monotonic seconds, eye, grid)] joined with the capture metadata."""
|
||||
meta = json.loads((self.directory / "meta.json").read_text())
|
||||
frames = []
|
||||
for pair in meta["frames"]:
|
||||
for eye in ("left", "right"):
|
||||
info = pair.get(eye) or {}
|
||||
match = self.NAME.match(Path(info.get("fname", "")).name)
|
||||
grid = self.grids[eye].get(int(match.group(2))) if match else None
|
||||
if info.get("valid") and grid is not None:
|
||||
frames.append((float(info["tsMono"]), eye, grid))
|
||||
return frames
|
||||
|
||||
def remove(self):
|
||||
"""Remove every capture directory that appeared during the run. Eye
|
||||
images must not outlive it, so a failure to delete is an error."""
|
||||
left = []
|
||||
if self.before is not None:
|
||||
try:
|
||||
self.new |= self.candidates() - self.before # even if interrupted before the first poll
|
||||
except OSError:
|
||||
pass
|
||||
for directory in self.new | ({self.directory} if self.directory else set()):
|
||||
if str(directory).startswith(self.PREFIX) and directory.is_dir() and not directory.is_symlink():
|
||||
try:
|
||||
shutil.rmtree(directory)
|
||||
except OSError:
|
||||
left.append(str(directory))
|
||||
if left:
|
||||
raise RuntimeError("could not delete eye images in " + ", ".join(sorted(left)))
|
||||
|
||||
|
||||
# --------------------------------------------------------------- analysis ---
|
||||
|
||||
def fft(values):
|
||||
"""In-place iterative radix-2 FFT of a list whose length is a power of 2."""
|
||||
n = len(values)
|
||||
a = list(values)
|
||||
j = 0
|
||||
for i in range(1, n):
|
||||
bit = n >> 1
|
||||
while j & bit:
|
||||
j ^= bit
|
||||
bit >>= 1
|
||||
j |= bit
|
||||
if i < j:
|
||||
a[i], a[j] = a[j], a[i]
|
||||
size = 2
|
||||
while size <= n:
|
||||
step = cmath.exp(-2j * math.pi / size)
|
||||
half = size // 2
|
||||
for start in range(0, n, size):
|
||||
w = 1
|
||||
for k in range(start, start + half):
|
||||
t = w * a[k + half]
|
||||
a[k + half] = a[k] - t
|
||||
a[k] += t
|
||||
w *= step
|
||||
size *= 2
|
||||
return a
|
||||
|
||||
|
||||
def resample(times, values, rate=RATE):
|
||||
"""Average samples into 1/rate bins from the first sample; empty bins are
|
||||
filled from the previous bin."""
|
||||
if not times:
|
||||
return []
|
||||
start = times[0]
|
||||
count = int((times[-1] - start) * rate) + 1
|
||||
sums, counts = [0.0] * count, [0] * count
|
||||
for t, v in zip(times, values):
|
||||
i = min(int((t - start) * rate), count - 1)
|
||||
sums[i] += v
|
||||
counts[i] += 1
|
||||
out, last = [], None
|
||||
for s, c in zip(sums, counts):
|
||||
last = s / c if c else last
|
||||
out.append(last)
|
||||
first = next(v for v in out if v is not None)
|
||||
return [first if v is None else v for v in out]
|
||||
|
||||
|
||||
def clean(signal, rate=RATE):
|
||||
"""Relative change with slow drift removed; None if the patch is mostly
|
||||
blinks or eye movement. Spikes (blinks, saccades) become gaps at the
|
||||
local level rather than clipped steps, which would add false rhythm."""
|
||||
mean = sum(signal) / len(signal)
|
||||
x = [v / mean - 1 for v in signal]
|
||||
half = int(rate) # 2-second centred moving median removes drift and blinks
|
||||
trend = []
|
||||
for i in range(len(x)):
|
||||
chunk = sorted(x[max(0, i - half):i + half + 1])
|
||||
trend.append(chunk[len(chunk) // 2])
|
||||
residual = [v - m for v, m in zip(x, trend)]
|
||||
mad = sorted(abs(v) for v in residual)[len(residual) // 2] or 1e-9
|
||||
limit = 4 * 1.4826 * mad
|
||||
spikes = sum(1 for v in residual if abs(v) > limit)
|
||||
if spikes > 0.05 * len(residual):
|
||||
return None
|
||||
return [v if abs(v) <= limit else 0.0 for v in residual]
|
||||
|
||||
|
||||
def spectrum(signal, rate=RATE):
|
||||
"""[(bpm, power)] within the search band, Hann-windowed and zero-padded."""
|
||||
n = len(signal)
|
||||
size = 1
|
||||
while size < max(n * 4, 256):
|
||||
size *= 2
|
||||
window = [0.5 - 0.5 * math.cos(2 * math.pi * i / (n - 1)) for i in range(n)] if n > 1 else [1.0]
|
||||
padded = [s * w for s, w in zip(signal, window)] + [0.0] * (size - n)
|
||||
result = fft(padded)
|
||||
out = []
|
||||
for k in range(size // 2):
|
||||
bpm = k * rate / size * 60
|
||||
if LOW <= bpm <= HIGH:
|
||||
out.append((bpm, abs(result[k]) ** 2))
|
||||
return out
|
||||
|
||||
|
||||
def peak(spec):
|
||||
"""Peak BPM with parabolic interpolation between spectral bins."""
|
||||
i = max(range(len(spec)), key=lambda k: spec[k][1])
|
||||
if 0 < i < len(spec) - 1:
|
||||
a, b, c = spec[i - 1][1], spec[i][1], spec[i + 1][1]
|
||||
denominator = a - 2 * b + c
|
||||
offset = 0.5 * (a - c) / denominator if denominator else 0.0
|
||||
return spec[i][0] + offset * (spec[1][0] - spec[0][0])
|
||||
return spec[i][0]
|
||||
|
||||
|
||||
def snr(spec, bpm, width=4.0):
|
||||
"""Power near the pulse and its first harmonic against the rest of the band."""
|
||||
near = sum(p for f, p in spec if abs(f - bpm) <= width or abs(f - 2 * bpm) <= width)
|
||||
rest = sum(p for f, p in spec) - near
|
||||
if near <= 0:
|
||||
return 0.0
|
||||
return near / rest if rest > 0 else float("inf")
|
||||
|
||||
|
||||
def normalised(spec):
|
||||
total = sum(p for _, p in spec) or 1.0
|
||||
return [p / total for _, p in spec]
|
||||
|
||||
|
||||
def usable(values):
|
||||
"""Patches that are neither dark nor saturated for the whole recording."""
|
||||
mean = sum(values) / len(values)
|
||||
return 8 <= mean <= 245
|
||||
|
||||
|
||||
def estimate(times, patches, rate=RATE, share=0.2, window=WINDOW):
|
||||
"""Estimate pulse from patch brightness traces.
|
||||
|
||||
times: monotonic seconds per frame. patches: {name: [brightness per frame]}.
|
||||
Selects the share of usable patches with the clearest periodic signal,
|
||||
combines their spectra and reports the overall and windowed estimates.
|
||||
"""
|
||||
cleaned = {}
|
||||
for name, values in patches.items():
|
||||
if usable(values):
|
||||
signal = clean(resample(times, values, rate), rate)
|
||||
if signal is not None and any(signal): # flat patches carry no rhythm
|
||||
cleaned[name] = signal
|
||||
if not cleaned or len(next(iter(cleaned.values()))) < rate * 8:
|
||||
raise ValueError("need at least 8 seconds of usable eye-camera frames")
|
||||
quality = []
|
||||
for name, signal in cleaned.items():
|
||||
spec = spectrum(signal, rate)
|
||||
own = peak(spec)
|
||||
quality.append((snr(spec, own), name, own, spec))
|
||||
quality.sort(reverse=True)
|
||||
chosen = quality[:max(4, int(len(quality) * share))]
|
||||
combined = [sum(values) for values in zip(*(normalised(spec) for _, _, _, spec in chosen))]
|
||||
bins = [f for f, _ in chosen[0][3]]
|
||||
bpm = peak(list(zip(bins, combined)))
|
||||
top = chosen[:8]
|
||||
agree = sum(1 for _, _, own, _ in top if abs(own - bpm) <= 5) / len(top)
|
||||
series = []
|
||||
samples = int(window * rate)
|
||||
length = len(next(iter(cleaned.values())))
|
||||
for end in range(samples, length + 1, int(rate)):
|
||||
specs = [spectrum(cleaned[name][end - samples:end], rate) for _, name, _, _ in chosen]
|
||||
total = [sum(values) for values in zip(*(normalised(s) for s in specs))]
|
||||
window_bins = [f for f, _ in specs[0]]
|
||||
series.append((times[0] + end / rate, peak(list(zip(window_bins, total)))))
|
||||
return {
|
||||
"bpm": bpm,
|
||||
"agreement": agree,
|
||||
"patches": len(chosen),
|
||||
"usable": len(cleaned),
|
||||
"snr": snr(list(zip(bins, combined)), bpm),
|
||||
"series": series,
|
||||
}
|
||||
|
||||
|
||||
def analyse(frames):
|
||||
"""Estimate from Capture.frames(); both eyes' patches are analysed together
|
||||
on a shared time base, each sample taken from that eye's nearest frame."""
|
||||
times = sorted({t for t, _, _ in frames})
|
||||
patches = {}
|
||||
for eye in ("left", "right"):
|
||||
eye_frames = sorted((t, grid) for t, e, grid in frames if e == eye)
|
||||
if not eye_frames:
|
||||
continue
|
||||
eye_times = [t for t, _ in eye_frames]
|
||||
nearest = []
|
||||
for t in times:
|
||||
i = bisect.bisect_left(eye_times, t)
|
||||
if i == len(eye_times) or (i > 0 and t - eye_times[i - 1] <= eye_times[i] - t):
|
||||
i -= 1
|
||||
nearest.append(i)
|
||||
for k in range(len(eye_frames[0][1])):
|
||||
patches[f"{eye}{k}"] = [eye_frames[i][1][k] for i in nearest]
|
||||
return estimate(times, patches)
|
||||
|
||||
|
||||
def proximity(root=None):
|
||||
"""The headset's proximity reading, or None if it can't be read."""
|
||||
try:
|
||||
for device in sorted(Path(root or IIO).glob("iio:device*")):
|
||||
if (device / "name").read_text().strip() == "vcnl4000":
|
||||
return float((device / "in_proximity_raw").read_text())
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def worn(reading):
|
||||
"""False only when the sensor says the headset is not on a face."""
|
||||
return reading is None or reading >= WORN_MIN
|
||||
|
||||
|
||||
def reliable(result):
|
||||
"""Whether the estimate is clear enough to show as a reading. Thresholds
|
||||
are provisional until checked against a reference on a real wearer."""
|
||||
return result["agreement"] >= 0.75 and result["snr"] >= 0.5
|
||||
@@ -0,0 +1,438 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Frame-local tracking tools. No network destination or data log by default."""
|
||||
import argparse
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import socket
|
||||
import struct
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
|
||||
class TrackingError(RuntimeError):
|
||||
"""A safe, actionable status message containing no sensor data."""
|
||||
|
||||
|
||||
HRS = "0000180d-0000-1000-8000-00805f9b34fb"
|
||||
MEASUREMENT = "00002a37-0000-1000-8000-00805f9b34fb"
|
||||
DEVICE = "org.bluez.Device1"
|
||||
SERVICE = "org.bluez.GattService1"
|
||||
CHARACTERISTIC = "org.bluez.GattCharacteristic1"
|
||||
|
||||
|
||||
def heart_rate(data):
|
||||
"""Validate the Bluetooth HRS measurement, returning BPM/contact only.
|
||||
|
||||
Energy and RR intervals are checked for length but never retained.
|
||||
None means contact is supported and the strap reports no skin contact.
|
||||
"""
|
||||
data = bytes(data)
|
||||
if len(data) < 2 or data[0] & 0xe0:
|
||||
raise ValueError("invalid HRS measurement")
|
||||
flags = data[0]
|
||||
size = 2 if flags & 1 else 1
|
||||
end = 1 + size + (2 if flags & 8 else 0)
|
||||
if len(data) < end:
|
||||
raise ValueError("truncated HRS measurement")
|
||||
extra = len(data) - end
|
||||
if (flags & 16 and (extra < 2 or extra % 2)) or (not flags & 16 and extra):
|
||||
raise ValueError("invalid HRS optional fields")
|
||||
if flags & 4 and not flags & 2:
|
||||
return None
|
||||
bpm = int.from_bytes(data[1:1 + size], "little")
|
||||
return bpm if bpm else None
|
||||
|
||||
|
||||
def gaze_angles(quaternion):
|
||||
"""OpenXR head-relative -Z forward → VRChat degrees, down/right positive."""
|
||||
if len(quaternion) != 4 or not all(math.isfinite(v) for v in quaternion):
|
||||
raise ValueError("invalid gaze orientation")
|
||||
norm = math.sqrt(sum(v * v for v in quaternion))
|
||||
if not 0.9 < norm < 1.1:
|
||||
raise ValueError("invalid gaze orientation")
|
||||
x, y, z, w = (v / norm for v in quaternion)
|
||||
# Rotate OpenXR's forward vector (0, 0, -1) into VIEW space.
|
||||
dx, dy, dz = -2 * (x*z + w*y), 2 * (w*x - y*z), 2 * (x*x + y*y) - 1
|
||||
return math.degrees(math.atan2(-dy, math.hypot(dx, dz))), math.degrees(math.atan2(dx, -dz))
|
||||
|
||||
|
||||
def osc_message(address, values):
|
||||
if not address.startswith("/") or any(c.isspace() or c in '\0#*,?[]{}' for c in address):
|
||||
raise ValueError("OSC address must be a literal path")
|
||||
def string(value):
|
||||
encoded = value.encode("utf-8") + b"\0"
|
||||
return encoded + b"\0" * (-len(encoded) % 4)
|
||||
tags, payload = ",", b""
|
||||
for value in values:
|
||||
if type(value) is int:
|
||||
tags += "i"
|
||||
payload += struct.pack(">i", value)
|
||||
else:
|
||||
if not math.isfinite(value):
|
||||
raise ValueError("OSC value must be finite")
|
||||
tags += "f"
|
||||
payload += struct.pack(">f", value)
|
||||
return string(address) + string(tags) + payload
|
||||
|
||||
|
||||
class Osc:
|
||||
def __init__(self, endpoint=None):
|
||||
self.sock = None
|
||||
self.target = None
|
||||
if endpoint:
|
||||
import ipaddress
|
||||
address = ipaddress.ip_address(endpoint[0])
|
||||
port = int(endpoint[1])
|
||||
if address.is_unspecified or address.is_multicast or not 1 <= port <= 65535:
|
||||
raise ValueError("OSC needs a unicast IP address and port 1..65535")
|
||||
self.sock = socket.socket(socket.AF_INET6 if address.version == 6 else socket.AF_INET, socket.SOCK_DGRAM)
|
||||
self.target = (str(address), port)
|
||||
|
||||
def send(self, address, values):
|
||||
if self.sock:
|
||||
self.sock.sendto(osc_message(address, values), self.target)
|
||||
|
||||
def close(self):
|
||||
if self.sock:
|
||||
self.sock.close()
|
||||
|
||||
|
||||
class HeartSession:
|
||||
def __init__(self, osc, address, log=None, clock=time.monotonic):
|
||||
self.osc, self.address, self.clock = osc, address, clock
|
||||
self.bpm, self.updated = None, None
|
||||
self.log = None
|
||||
if log:
|
||||
# Exclusive creation refuses existing files and symlinks; mode is
|
||||
# private even with a permissive process umask.
|
||||
fd = os.open(log, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
self.log = os.fdopen(fd, "w")
|
||||
self.log.write("unix_seconds,bpm\n")
|
||||
|
||||
def notification(self, data):
|
||||
self.bpm = heart_rate(data)
|
||||
self.updated = self.clock()
|
||||
if self.bpm is not None:
|
||||
self.osc.send(self.address, [self.bpm])
|
||||
if self.log:
|
||||
self.log.write(f"{time.time():.3f},{self.bpm}\n")
|
||||
self.log.flush()
|
||||
|
||||
def current(self):
|
||||
if self.updated is None or self.clock() - self.updated > 5:
|
||||
return None
|
||||
return self.bpm
|
||||
|
||||
def close(self):
|
||||
if self.log:
|
||||
self.log.close()
|
||||
|
||||
|
||||
class BluezHeart:
|
||||
"""One explicitly selected, already discovered strap; no ambient scan."""
|
||||
def __init__(self, bus, interface, address, on_value):
|
||||
self.bus, self.interface, self.on_value = bus, interface, on_value
|
||||
self.device = self.characteristic = None
|
||||
self.connected_here = False
|
||||
self.notifying = False
|
||||
self.match = None
|
||||
objects = self.objects()
|
||||
matches = [path for path, interfaces in objects.items()
|
||||
if str(interfaces.get(DEVICE, {}).get("Address", "")).upper() == address.upper()]
|
||||
if len(matches) != 1:
|
||||
raise TrackingError("Strap not found uniquely in BlueZ; pair/discover it in SteamOS Bluetooth settings first")
|
||||
self.device = matches[0]
|
||||
self.match = bus.add_signal_receiver(self.changed, signal_name="PropertiesChanged",
|
||||
dbus_interface="org.freedesktop.DBus.Properties",
|
||||
bus_name="org.bluez", path_keyword="path")
|
||||
try:
|
||||
if not objects[self.device][DEVICE].get("Connected"):
|
||||
self.call(self.device, DEVICE).Connect(timeout=20)
|
||||
self.connected_here = True
|
||||
except Exception:
|
||||
self.close()
|
||||
raise
|
||||
|
||||
def objects(self):
|
||||
return self.call("/", "org.freedesktop.DBus.ObjectManager").GetManagedObjects()
|
||||
|
||||
def call(self, path, kind):
|
||||
return self.interface(self.bus.get_object("org.bluez", path), kind)
|
||||
|
||||
def subscribe(self):
|
||||
objects = self.objects()
|
||||
if not objects.get(self.device, {}).get(DEVICE, {}).get("ServicesResolved"):
|
||||
return False
|
||||
services = {p for p, obj in objects.items() if str(obj.get(SERVICE, {}).get("UUID", "")).lower() == HRS
|
||||
and obj[SERVICE].get("Device") == self.device}
|
||||
for path, obj in objects.items():
|
||||
props = obj.get(CHARACTERISTIC, {})
|
||||
if props.get("Service") in services and str(props.get("UUID", "")).lower() == MEASUREMENT:
|
||||
if "notify" not in props.get("Flags", []):
|
||||
raise TrackingError("Heart-rate characteristic does not support notifications")
|
||||
self.characteristic = path
|
||||
self.call(path, CHARACTERISTIC).StartNotify()
|
||||
self.notifying = True
|
||||
return True
|
||||
raise TrackingError("Selected device has no standard Heart Rate Service measurement")
|
||||
|
||||
def changed(self, kind, changes, invalidated, path=None):
|
||||
if kind == CHARACTERISTIC and path == self.characteristic and "Value" in changes:
|
||||
self.on_value(changes["Value"])
|
||||
elif kind == DEVICE and path == self.device and "Connected" in changes and not changes["Connected"]:
|
||||
self.on_value(None)
|
||||
|
||||
def close(self):
|
||||
try:
|
||||
if self.notifying:
|
||||
self.call(self.characteristic, CHARACTERISTIC).StopNotify()
|
||||
finally:
|
||||
if self.match:
|
||||
self.match.remove()
|
||||
if self.connected_here:
|
||||
self.call(self.device, DEVICE).Disconnect()
|
||||
|
||||
|
||||
def run_gaze(args, osc):
|
||||
binary = Path(__file__).with_name("gaze")
|
||||
command = [str(binary), "--seconds", str(args.seconds)]
|
||||
if not args.osc:
|
||||
return subprocess.call(command)
|
||||
read_fd, write_fd = os.pipe()
|
||||
process = None
|
||||
try:
|
||||
process = subprocess.Popen(command + ["--fd", str(write_fd)], pass_fds=(write_fd,))
|
||||
os.close(write_fd)
|
||||
write_fd = None
|
||||
with os.fdopen(read_fd) as source:
|
||||
read_fd = None
|
||||
for line in source:
|
||||
try:
|
||||
angles = gaze_angles([float(v) for v in line.split()])
|
||||
except ValueError:
|
||||
continue
|
||||
osc.send("/tracking/eye/CenterPitchYaw", angles)
|
||||
return process.wait()
|
||||
finally:
|
||||
if read_fd is not None:
|
||||
os.close(read_fd)
|
||||
if write_fd is not None:
|
||||
os.close(write_fd)
|
||||
if process and process.poll() is None:
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait()
|
||||
|
||||
|
||||
class HeartPanel:
|
||||
"""Our GTK panel, using the Frame's existing GTK4/GI platform libraries."""
|
||||
def __init__(self):
|
||||
os.environ["GDK_BACKEND"] = "x11"
|
||||
import gi
|
||||
gi.require_version("Gtk", "4.0")
|
||||
gi.require_version("GdkX11", "4.0")
|
||||
from gi.repository import Gtk, Gdk, GdkX11, GLib
|
||||
Gtk.init()
|
||||
self.running = True
|
||||
self.window = Gtk.Window(title="Frame Control · Heart rate")
|
||||
self.window.set_default_size(480, 320)
|
||||
self.window.connect("close-request", self.stop)
|
||||
Gtk.Settings.get_default().set_property("gtk-application-prefer-dark-theme", True)
|
||||
box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=16)
|
||||
box.set_valign(Gtk.Align.CENTER)
|
||||
box.set_halign(Gtk.Align.CENTER)
|
||||
box.set_size_request(440, -1)
|
||||
for side in ("top", "bottom", "start", "end"):
|
||||
getattr(box, "set_margin_" + side)(24)
|
||||
self.window.set_child(box)
|
||||
title = Gtk.Label(label="Heart rate")
|
||||
title.add_css_class("title-2")
|
||||
box.append(title)
|
||||
self.reading = Gtk.Label(label="—")
|
||||
self.reading.add_css_class("reading")
|
||||
box.append(self.reading)
|
||||
self.status = Gtk.Label(label="Waiting for strap")
|
||||
box.append(self.status)
|
||||
button = Gtk.Button(label="Stop")
|
||||
button.connect("clicked", self.stop)
|
||||
box.append(button)
|
||||
css = Gtk.CssProvider()
|
||||
css.load_from_data(b".reading { font-size: 144px; font-weight: 700; }")
|
||||
Gtk.StyleContext.add_provider_for_display(Gdk.Display.get_default(), css, Gtk.STYLE_PROVIDER_PRIORITY_APPLICATION)
|
||||
self.window.present()
|
||||
context = GLib.MainContext.default()
|
||||
while context.pending():
|
||||
context.iteration(False)
|
||||
try:
|
||||
xid = GdkX11.X11Surface.get_xid(self.window.get_surface())
|
||||
subprocess.run(["xprop", "-id", str(xid), "-f", "STEAM_GAME", "32c", "-set", "STEAM_GAME", "2000000027"],
|
||||
check=True, stdout=subprocess.DEVNULL)
|
||||
except Exception:
|
||||
self.window.destroy()
|
||||
raise
|
||||
|
||||
def stop(self, *args):
|
||||
self.running = False
|
||||
return True
|
||||
|
||||
def update(self, bpm):
|
||||
self.reading.set_label(str(bpm) if bpm is not None else "—")
|
||||
self.status.set_label("beats per minute" if bpm is not None else "Waiting for strap")
|
||||
|
||||
def close(self):
|
||||
self.window.destroy()
|
||||
|
||||
|
||||
def run_heart(args, osc):
|
||||
import dbus
|
||||
from dbus.mainloop.glib import DBusGMainLoop
|
||||
from gi.repository import GLib
|
||||
DBusGMainLoop(set_as_default=True)
|
||||
session = HeartSession(osc, args.address, args.log)
|
||||
reader, root = None, None
|
||||
failure = []
|
||||
def value(data):
|
||||
if data is None:
|
||||
session.bpm = None
|
||||
failure.append("Strap disconnected; reconnect and start again")
|
||||
return
|
||||
try:
|
||||
session.notification(data)
|
||||
except ValueError:
|
||||
session.bpm = None
|
||||
except OSError:
|
||||
failure.append("OSC or session log write failed")
|
||||
try:
|
||||
reader = BluezHeart(dbus.SystemBus(), dbus.Interface, args.device, value)
|
||||
context = GLib.MainContext.default()
|
||||
deadline = time.monotonic() + 20
|
||||
while not reader.subscribe():
|
||||
if time.monotonic() > deadline:
|
||||
raise TrackingError("Timed out waiting for the strap's GATT services")
|
||||
while context.pending():
|
||||
context.iteration(False)
|
||||
time.sleep(0.1)
|
||||
end = time.monotonic() + args.seconds
|
||||
print("Heart-rate notifications started; readings stay local unless OSC or a log was selected.")
|
||||
if args.panel:
|
||||
root = HeartPanel()
|
||||
running = lambda: root.running if root else True
|
||||
while running() and time.monotonic() < end and not failure:
|
||||
while context.pending():
|
||||
context.iteration(False)
|
||||
if root:
|
||||
root.update(session.current())
|
||||
time.sleep(0.05)
|
||||
if failure:
|
||||
raise TrackingError(failure[0])
|
||||
return 0
|
||||
finally:
|
||||
if root:
|
||||
root.close()
|
||||
try:
|
||||
if reader:
|
||||
reader.close()
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
def run_pulse(args, osc):
|
||||
"""Experimental: estimate pulse from the IR eye cameras (see pulse.py)."""
|
||||
import pulse
|
||||
if not pulse.ET_BIN.exists():
|
||||
raise TrackingError("SteamVR's eye-tracking tool is not installed on this Frame")
|
||||
print(f"Capturing {args.seconds} s from the eye cameras. Wear the headset and keep still.", flush=True)
|
||||
readings = [pulse.proximity()]
|
||||
try:
|
||||
frames = pulse.Capture(args.seconds).run()
|
||||
readings.append(pulse.proximity())
|
||||
except RuntimeError as error: # capture status only; no image data
|
||||
raise TrackingError(str(error))
|
||||
print(f"Captured {len(frames)} eye frames; images already deleted. Analysing...", flush=True)
|
||||
try:
|
||||
result = pulse.analyse(frames)
|
||||
except ValueError as error:
|
||||
raise TrackingError(str(error))
|
||||
on_face = all(pulse.worn(reading) for reading in readings)
|
||||
if not on_face:
|
||||
# Unworn, the cameras still show a steady periodic artifact that looks
|
||||
# like a pulse (verified on the Frame), so it is never called clear.
|
||||
print("The proximity sensor says the headset is not being worn; no pulse can be read.")
|
||||
clear = on_face and pulse.reliable(result)
|
||||
offset = time.time() - time.monotonic() # the capture's timestamps are CLOCK_MONOTONIC
|
||||
if args.log:
|
||||
fd = os.open(args.log, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(fd, "w") as log:
|
||||
log.write("unix_seconds,bpm\n")
|
||||
for when, bpm in result["series"]:
|
||||
log.write(f"{when + offset:.3f},{bpm:.1f}\n")
|
||||
if clear:
|
||||
osc.send(args.address, [round(result["bpm"])])
|
||||
if args.show:
|
||||
print(f"Estimate: {result['bpm']:.1f} BPM ({'clear' if clear else 'NOT clear'}; "
|
||||
f"patch agreement {result['agreement']:.0%}, signal/noise {result['snr']:.2f}, "
|
||||
f"{result['patches']} of {result['usable']} usable patches)")
|
||||
print("Per-second estimates (15 s windows): "
|
||||
+ " ".join(str(round(bpm)) for _, bpm in result["series"]))
|
||||
else:
|
||||
print("A clear pulse was found." if clear else "No clear pulse was found.")
|
||||
return 0 if clear else 3
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
gaze = commands.add_parser("gaze", help="headless OpenXR; prints counters only without --osc")
|
||||
heart = commands.add_parser("heart", help="standard BLE HRS from an explicitly selected strap")
|
||||
pulse = commands.add_parser("pulse", help="experimental pulse estimate from the IR eye cameras")
|
||||
for command in (gaze, heart, pulse):
|
||||
command.add_argument("--osc", nargs=2, metavar=("IP", "PORT"), help="explicit UDP destination; no default")
|
||||
for command in (gaze, heart):
|
||||
command.add_argument("--seconds", type=int, default=10, help="bounded run, 1..86400 seconds (default: 10)")
|
||||
pulse.add_argument("--seconds", type=int, default=60, help="capture length, 20..300 seconds (default: 60)")
|
||||
heart.add_argument("--device", required=True, help="strap Bluetooth address already discovered by BlueZ")
|
||||
heart.add_argument("--panel", action="store_true", help="show our panel on gamescope DISPLAY=:0")
|
||||
for command in (heart, pulse):
|
||||
command.add_argument("--address", default="/avatar/parameters/HeartRate", help="integer BPM OSC parameter")
|
||||
command.add_argument("--log", help="new private CSV file; disabled by default")
|
||||
pulse.add_argument("--show", action="store_true", help="print the estimate and per-second series")
|
||||
args = parser.parse_args()
|
||||
if not 1 <= args.seconds <= 86400:
|
||||
parser.error("--seconds must be 1..86400")
|
||||
if args.command == "pulse" and not 20 <= args.seconds <= 300:
|
||||
parser.error("pulse --seconds must be 20..300")
|
||||
def interrupted(signum, frame):
|
||||
raise KeyboardInterrupt
|
||||
signal.signal(signal.SIGTERM, interrupted)
|
||||
if hasattr(signal, "SIGHUP"):
|
||||
signal.signal(signal.SIGHUP, interrupted)
|
||||
osc = None
|
||||
try:
|
||||
if args.command in ("heart", "pulse"):
|
||||
osc_message(args.address, [0])
|
||||
if getattr(args, "panel", False):
|
||||
os.environ["DISPLAY"] = ":0"
|
||||
osc = Osc(args.osc)
|
||||
run = {"gaze": run_gaze, "heart": run_heart, "pulse": run_pulse}[args.command]
|
||||
return run(args, osc)
|
||||
except TrackingError as error:
|
||||
print(str(error))
|
||||
return 1
|
||||
except KeyboardInterrupt:
|
||||
return 130
|
||||
except Exception as error:
|
||||
# Never dump notifications, gaze, BLE addresses or exception payloads.
|
||||
print(f"Tracking stopped ({type(error).__name__}). Check the device, runtime and selected output.")
|
||||
return 1
|
||||
finally:
|
||||
if osc:
|
||||
osc.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,274 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check Frame Control's heart-rate readings against a reference, on your computer.
|
||||
|
||||
python3 scripts/heart-check.py listen --port 9000 --out ours.csv
|
||||
python3 scripts/heart-check.py compare ours.csv reference.csv
|
||||
python3 scripts/heart-check.py compare ours.csv ~/Downloads/export.zip
|
||||
|
||||
`listen` receives our integer-BPM OSC messages (send them here with
|
||||
`tracking-on-frame.py heart --osc <this computer's IP> 9000`) and shows each
|
||||
reading live, so you can watch it next to a reference such as your Apple
|
||||
Watch. `--out` also records `unix_seconds,bpm` to a new private file.
|
||||
|
||||
`compare` lines up two recordings by time and reports how far apart they are.
|
||||
The reference can be:
|
||||
- a CSV whose first column is a time (unix seconds or ISO 8601) and whose
|
||||
second is BPM, such as our own log, `listen --out`, or the test strap's
|
||||
output (a third `flags` column marks skin-contact loss as "no reading");
|
||||
- an Apple Health export (`export.zip` or `export.xml`, from Health → your
|
||||
profile → Export All Health Data). Only heart-rate records within the
|
||||
recording's time range are read.
|
||||
|
||||
Everything stays on this computer. Nothing is uploaded.
|
||||
"""
|
||||
import argparse
|
||||
import bisect
|
||||
import csv
|
||||
from datetime import datetime
|
||||
import io
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
import time
|
||||
import zipfile
|
||||
from xml.etree import ElementTree
|
||||
|
||||
ADDRESS = "/avatar/parameters/HeartRate"
|
||||
|
||||
|
||||
def parse_osc(packet):
|
||||
"""Return (address, values) for a single OSC message with i/f arguments."""
|
||||
def string(offset):
|
||||
end = packet.index(b"\0", offset)
|
||||
return packet[offset:end].decode("utf-8"), (end + 4) & ~3
|
||||
address, offset = string(0)
|
||||
tags, offset = string(offset)
|
||||
if not tags.startswith(","):
|
||||
raise ValueError("not an OSC message")
|
||||
values = []
|
||||
for tag in tags[1:]:
|
||||
if tag not in "if" or offset + 4 > len(packet):
|
||||
raise ValueError("unsupported OSC argument")
|
||||
values.append(struct.unpack(">i" if tag == "i" else ">f", packet[offset:offset + 4])[0])
|
||||
offset += 4
|
||||
return address, values
|
||||
|
||||
|
||||
def listen(port, address, out, seconds, clock=time.time, stream=sys.stdout):
|
||||
log = None
|
||||
if out:
|
||||
log = os.fdopen(os.open(out, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w")
|
||||
log.write("unix_seconds,bpm\n")
|
||||
received = 0
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
|
||||
sock.bind(("0.0.0.0", port))
|
||||
sock.settimeout(0.5)
|
||||
print(f"Listening for {address} on UDP port {port}. Ctrl-C stops.", file=stream, flush=True)
|
||||
end = clock() + seconds if seconds else None
|
||||
try:
|
||||
while end is None or clock() < end:
|
||||
try:
|
||||
packet = sock.recv(1024)
|
||||
except socket.timeout:
|
||||
continue
|
||||
try:
|
||||
path, values = parse_osc(packet)
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
continue
|
||||
if path != address or len(values) != 1:
|
||||
continue
|
||||
now = clock()
|
||||
bpm = int(values[0])
|
||||
received += 1
|
||||
print(f"{time.strftime('%H:%M:%S', time.localtime(now))} {bpm:3d} bpm", file=stream, flush=True)
|
||||
if log:
|
||||
log.write(f"{now:.3f},{bpm}\n")
|
||||
log.flush()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
if log:
|
||||
log.close()
|
||||
return received
|
||||
|
||||
|
||||
def parse_time(text):
|
||||
text = text.strip()
|
||||
if re.fullmatch(r"\d+(\.\d+)?", text):
|
||||
return float(text)
|
||||
# Apple Health uses "2026-09-29 09:12:03 +1000".
|
||||
text = re.sub(r" ([+-]\d{2}):?(\d{2})$", r"\1\2", text).replace("Z", "+0000")
|
||||
for pattern in ("%Y-%m-%d %H:%M:%S%z", "%Y-%m-%dT%H:%M:%S%z", "%Y-%m-%dT%H:%M:%S.%f%z"):
|
||||
try:
|
||||
return datetime.strptime(text, pattern).timestamp()
|
||||
except ValueError:
|
||||
pass
|
||||
raise ValueError(f"unrecognised time {text!r}")
|
||||
|
||||
|
||||
def read_csv(path):
|
||||
"""[(time, bpm or None)], sorted. A header row is skipped."""
|
||||
samples = []
|
||||
with open(path, newline="") as source:
|
||||
for row in csv.reader(source):
|
||||
if len(row) < 2:
|
||||
continue
|
||||
try:
|
||||
when, bpm = parse_time(row[0]), int(float(row[1]))
|
||||
except (ValueError, OverflowError):
|
||||
continue # header or unparseable line
|
||||
try:
|
||||
flags = int(float(row[2])) if len(row) > 2 else None
|
||||
except (ValueError, OverflowError):
|
||||
flags = None # an unrecognised flag leaves the reading as it is
|
||||
if flags is not None and flags & 4 and not flags & 2:
|
||||
bpm = None # contact supported and not detected: no reading
|
||||
samples.append((when, bpm))
|
||||
return sorted(samples, key=lambda s: s[0])
|
||||
|
||||
|
||||
def read_health(path, start, end):
|
||||
"""Heart-rate records from an Apple Health export between start and end."""
|
||||
samples = []
|
||||
def scan(source):
|
||||
for _, element in ElementTree.iterparse(source):
|
||||
if element.tag == "Record" and element.get("type") == "HKQuantityTypeIdentifierHeartRate":
|
||||
try:
|
||||
when = parse_time(element.get("startDate") or "")
|
||||
value = round(float(element.get("value") or ""))
|
||||
except (ValueError, OverflowError):
|
||||
continue
|
||||
if start <= when <= end:
|
||||
samples.append((when, value))
|
||||
element.clear()
|
||||
if zipfile.is_zipfile(path):
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
names = [n for n in archive.namelist() if n.endswith("/export.xml") or n == "export.xml"]
|
||||
if not names:
|
||||
raise ValueError("no export.xml in that archive; use Health's Export All Health Data")
|
||||
with archive.open(names[0]) as source:
|
||||
scan(source)
|
||||
else:
|
||||
with open(path, "rb") as source:
|
||||
scan(source)
|
||||
return sorted(samples)
|
||||
|
||||
|
||||
def read_any(path, start=None, end=None):
|
||||
path = str(path)
|
||||
if path.endswith((".zip", ".xml")):
|
||||
return read_health(path, start, end)
|
||||
return read_csv(path)
|
||||
|
||||
|
||||
def value_at(samples, when, hold, times=None):
|
||||
"""Our reading at a moment: the latest sample no older than `hold` seconds.
|
||||
`times` is the samples' time column, if the caller has already built it."""
|
||||
times = times if times is not None else [s[0] for s in samples]
|
||||
i = bisect.bisect_right(times, when) - 1
|
||||
if i < 0 or when - times[i] > hold:
|
||||
return None
|
||||
return samples[i][1]
|
||||
|
||||
|
||||
def compare(ours, reference, max_lag=10.0, hold=5.0):
|
||||
"""Compare our readings with the reference at each reference moment.
|
||||
|
||||
`lag` is the delay added to reference times before looking up ours (our
|
||||
readings arrive after the sensor's). The best lag within ±max_lag is used.
|
||||
"""
|
||||
real = [(t, b) for t, b in reference if b is not None]
|
||||
if not real or not any(b is not None for _, b in ours):
|
||||
raise ValueError("both recordings need at least one reading")
|
||||
best = None
|
||||
ours_times = [t for t, _ in ours]
|
||||
steps = int(max_lag * 4)
|
||||
for step in range(-steps, steps + 1):
|
||||
lag = step / 4
|
||||
pairs = [(value_at(ours, t + lag, hold, ours_times), b) for t, b in real]
|
||||
pairs = [(o, r) for o, r in pairs if o is not None]
|
||||
if not pairs:
|
||||
continue
|
||||
error = sum(abs(o - r) for o, r in pairs) / len(pairs)
|
||||
key = (-len(pairs), error, abs(lag))
|
||||
if best is None or key < best[0]:
|
||||
best = (key, lag, pairs)
|
||||
if best is None:
|
||||
raise ValueError("the recordings do not overlap in time")
|
||||
_, lag, pairs = best
|
||||
differences = [o - r for o, r in pairs]
|
||||
# While the reference says "no reading" (lost skin contact), we must not
|
||||
# produce new readings. Count ours that arrive during such a stretch.
|
||||
gaps = [t for t, b in reference if b is None]
|
||||
reference_times = [t for t, _ in reference]
|
||||
shown_in_gaps = 0
|
||||
for t, bpm in ours:
|
||||
i = bisect.bisect_right(reference_times, t - lag) - 1
|
||||
if bpm is not None and i >= 0 and reference[i][1] is None:
|
||||
shown_in_gaps += 1
|
||||
return {
|
||||
"reference_readings": len(real),
|
||||
"matched": len(pairs),
|
||||
"lag_seconds": lag,
|
||||
"mean_abs_error": sum(abs(d) for d in differences) / len(differences),
|
||||
"bias": sum(differences) / len(differences),
|
||||
"max_abs_error": max(abs(d) for d in differences),
|
||||
"within_5": sum(1 for d in differences if abs(d) <= 5) / len(differences),
|
||||
"exact": sum(1 for d in differences if d == 0) / len(differences),
|
||||
"no_contact_moments": len(gaps),
|
||||
"shown_during_no_contact": shown_in_gaps,
|
||||
}
|
||||
|
||||
|
||||
def report(result, tolerance, stream=sys.stdout):
|
||||
print(f"Reference readings: {result['reference_readings']}, matched: {result['matched']}", file=stream)
|
||||
print(f"Best alignment: ours {result['lag_seconds']:+.2f} s after the reference", file=stream)
|
||||
print(f"Mean absolute difference: {result['mean_abs_error']:.2f} BPM "
|
||||
f"(bias {result['bias']:+.2f}, worst {result['max_abs_error']})", file=stream)
|
||||
print(f"Within ±5 BPM: {result['within_5']:.0%}; identical: {result['exact']:.0%}", file=stream)
|
||||
if result["no_contact_moments"]:
|
||||
print(f"No-contact moments: {result['no_contact_moments']}, where we showed a stale "
|
||||
f"reading: {result['shown_during_no_contact']}", file=stream)
|
||||
coverage = result["matched"] / result["reference_readings"]
|
||||
passed = (result["mean_abs_error"] <= tolerance and coverage >= 0.8
|
||||
and not result["shown_during_no_contact"])
|
||||
print(("PASS" if passed else "FAIL") + f" (mean difference ≤ {tolerance} BPM, ≥80% of reference "
|
||||
f"readings matched, nothing shown without contact)", file=stream)
|
||||
return passed
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
heard = commands.add_parser("listen", help="show and optionally record our OSC readings live")
|
||||
heard.add_argument("--port", type=int, default=9000)
|
||||
heard.add_argument("--address", default=ADDRESS)
|
||||
heard.add_argument("--out", help="new private CSV file")
|
||||
heard.add_argument("--seconds", type=float, default=0, help="stop after this long (default: until Ctrl-C)")
|
||||
check = commands.add_parser("compare", help="compare our recording with a reference")
|
||||
check.add_argument("ours", type=Path)
|
||||
check.add_argument("reference", type=Path)
|
||||
check.add_argument("--tolerance", type=float, default=5.0, help="allowed mean difference in BPM (default 5)")
|
||||
check.add_argument("--max-lag", type=float, default=10.0, help="largest time offset to search, seconds")
|
||||
args = parser.parse_args(argv)
|
||||
if args.command == "listen":
|
||||
count = listen(args.port, args.address, args.out, args.seconds)
|
||||
print(f"Received {count} readings.")
|
||||
return 0 if count else 3
|
||||
try:
|
||||
ours = read_csv(args.ours)
|
||||
if not ours:
|
||||
raise ValueError("our recording has no readings")
|
||||
reference = read_any(args.reference, ours[0][0] - 60, ours[-1][0] + 60)
|
||||
result = compare(ours, reference, args.max_lag)
|
||||
except (ValueError, OSError, csv.Error, ElementTree.ParseError, zipfile.BadZipFile) as error:
|
||||
print(f"Could not compare: {error}")
|
||||
return 1
|
||||
return 0 if report(result, args.tolerance) else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,100 @@
|
||||
// A synthetic Bluetooth heart-rate strap for testing, run on the Mac.
|
||||
//
|
||||
// swiftc -O scripts/heart-test-strap.swift -o /tmp/heart-test-strap
|
||||
// /tmp/heart-test-strap [seconds] # default 60
|
||||
//
|
||||
// Advertises the standard Heart Rate Service (180d) as "FC Test Strap" and,
|
||||
// while a central is subscribed, sends one Heart Rate Measurement (2a37) per
|
||||
// second from a fixed, known sequence. Each sent value is printed to stdout as
|
||||
// `unix_seconds,bpm,flags` so scripts/heart-check.py can compare what the
|
||||
// Frame received with what was sent. Every value is synthetic; this is not a
|
||||
// sensor. macOS asks for Bluetooth permission the first time it runs.
|
||||
import CoreBluetooth
|
||||
import Foundation
|
||||
|
||||
let hrs = CBUUID(string: "180D")
|
||||
let measurement = CBUUID(string: "2A37")
|
||||
|
||||
/// The known sequence: an 8-bit ramp, 16-bit encodings, a skin-contact loss
|
||||
/// (which must show as no reading) and a recovery.
|
||||
func packet(_ second: Int) -> (bpm: Int, flags: UInt8) {
|
||||
switch second % 60 {
|
||||
case 0..<30: return (60 + second % 60 * 4, 0x06) // 60…176, contact detected
|
||||
case 30..<40: return (180 - (second % 60 - 30) * 3, 0x07) // 16-bit value
|
||||
case 40..<45: return (150, 0x04) // contact lost
|
||||
default: return (72 + (second % 60 - 45), 0x06)
|
||||
}
|
||||
}
|
||||
|
||||
final class Strap: NSObject, CBPeripheralManagerDelegate {
|
||||
let seconds: Int
|
||||
var manager: CBPeripheralManager!
|
||||
var characteristic: CBMutableCharacteristic!
|
||||
var subscribed = false
|
||||
var sent = 0
|
||||
|
||||
init(seconds: Int) {
|
||||
self.seconds = seconds
|
||||
super.init()
|
||||
manager = CBPeripheralManager(delegate: self, queue: nil)
|
||||
}
|
||||
|
||||
func peripheralManagerDidUpdateState(_ peripheral: CBPeripheralManager) {
|
||||
guard peripheral.state == .poweredOn else {
|
||||
FileHandle.standardError.write("Bluetooth state \(peripheral.state.rawValue)\n".data(using: .utf8)!)
|
||||
if peripheral.state == .unauthorized || peripheral.state == .unsupported || peripheral.state == .poweredOff {
|
||||
FileHandle.standardError.write("Bluetooth unavailable (state \(peripheral.state.rawValue))\n".data(using: .utf8)!)
|
||||
exit(1)
|
||||
}
|
||||
return
|
||||
}
|
||||
characteristic = CBMutableCharacteristic(type: measurement, properties: [.notify], value: nil, permissions: [])
|
||||
let service = CBMutableService(type: hrs, primary: true)
|
||||
service.characteristics = [characteristic]
|
||||
peripheral.add(service)
|
||||
}
|
||||
|
||||
func peripheralManager(_ peripheral: CBPeripheralManager, didAdd service: CBService, error: Error?) {
|
||||
if let error { fail("add service: \(error.localizedDescription)") }
|
||||
peripheral.startAdvertising([CBAdvertisementDataLocalNameKey: "FC Test Strap",
|
||||
CBAdvertisementDataServiceUUIDsKey: [hrs]])
|
||||
}
|
||||
|
||||
func peripheralManagerDidStartAdvertising(_ peripheral: CBPeripheralManager, error: Error?) {
|
||||
if let error { fail("advertise: \(error.localizedDescription)") }
|
||||
FileHandle.standardError.write("Advertising FC Test Strap\n".data(using: .utf8)!)
|
||||
Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { _ in self.tick() }
|
||||
}
|
||||
|
||||
func peripheralManager(_ peripheral: CBPeripheralManager, central: CBCentral, didSubscribeTo characteristic: CBCharacteristic) {
|
||||
subscribed = true
|
||||
FileHandle.standardError.write("Central subscribed\n".data(using: .utf8)!)
|
||||
}
|
||||
|
||||
func peripheralManager(_ peripheral: CBPeripheralManager, central: CBCentral, didUnsubscribeFrom characteristic: CBCharacteristic) {
|
||||
subscribed = false
|
||||
FileHandle.standardError.write("Central unsubscribed\n".data(using: .utf8)!)
|
||||
}
|
||||
|
||||
func tick() {
|
||||
guard subscribed else { return }
|
||||
if sent >= seconds { exit(0) }
|
||||
let (bpm, flags) = packet(sent)
|
||||
var bytes: [UInt8] = [flags, UInt8(bpm & 0xff)]
|
||||
if flags & 1 != 0 { bytes.append(UInt8(bpm >> 8)) }
|
||||
if manager.updateValue(Data(bytes), for: characteristic, onSubscribedCentrals: nil) {
|
||||
print(String(format: "%.3f,%d,%d", Date().timeIntervalSince1970, bpm, flags))
|
||||
fflush(stdout)
|
||||
sent += 1
|
||||
}
|
||||
}
|
||||
|
||||
func fail(_ message: String) -> Never {
|
||||
FileHandle.standardError.write("\(message)\n".data(using: .utf8)!)
|
||||
exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
let seconds = CommandLine.arguments.count > 1 ? Int(CommandLine.arguments[1]) ?? 60 : 60
|
||||
let strap = Strap(seconds: seconds)
|
||||
RunLoop.main.run()
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Install or run our local-only tracking tools on the Frame.
|
||||
|
||||
python3 scripts/tracking-on-frame.py install
|
||||
python3 scripts/tracking-on-frame.py gaze --seconds 10
|
||||
python3 scripts/tracking-on-frame.py gaze --seconds 3600 --osc 127.0.0.1 9000
|
||||
python3 scripts/tracking-on-frame.py heart --device AA:BB:CC:DD:EE:FF --panel
|
||||
python3 scripts/tracking-on-frame.py pulse --seconds 60 --show # experimental
|
||||
|
||||
FRAME_ALIAS overrides the SSH alias (default: frame). Runs in the foreground;
|
||||
Ctrl-C stops the reader. No service, autostart, sudo or SteamVR settings changes.
|
||||
"""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
|
||||
REMOTE = '"$HOME/.local/share/frame-control/tracking"'
|
||||
INSTALL = '''set -eu
|
||||
base="$HOME/.local/share/frame-control/tracking"
|
||||
mkdir -p "$base"
|
||||
stage=$(mktemp -d "$base/.install.XXXXXX")
|
||||
trap 'rm -rf "$stage"' EXIT
|
||||
tar -xf - -C "$stage"
|
||||
cc -O2 -Wall -Wextra -Werror "$stage/gaze.c" \\
|
||||
-L/opt/steamvr/bin/linuxarm64 -Wl,-rpath,/opt/steamvr/bin/linuxarm64 \\
|
||||
-lopenxr_loader -o "$stage/gaze"
|
||||
chmod 700 "$stage/gaze" "$stage/tracking.py"
|
||||
chmod 600 "$stage/pulse.py"
|
||||
mv "$stage/gaze" "$stage/tracking.py" "$stage/pulse.py" "$base/"
|
||||
echo 'Installed Frame Control tracking tools (no service started).'
|
||||
'''
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2 or sys.argv[1] not in ("install", "gaze", "heart", "pulse"):
|
||||
print(__doc__)
|
||||
return 2
|
||||
host = os.environ.get("FRAME_ALIAS", "frame")
|
||||
if not host or host.startswith("-"):
|
||||
raise ValueError("invalid SSH alias")
|
||||
ssh = ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", host]
|
||||
if sys.argv[1] == "install":
|
||||
import tempfile
|
||||
source = Path(__file__).resolve().parents[1] / "frame" / "tracking"
|
||||
with tempfile.TemporaryFile() as archive:
|
||||
with tarfile.open(fileobj=archive, mode="w") as tar:
|
||||
for name in ("gaze.c", "tracking.py", "pulse.py"):
|
||||
tar.add(source / name, arcname=name)
|
||||
archive.seek(0)
|
||||
return subprocess.call(ssh + ["bash -c " + shlex.quote(INSTALL)], stdin=archive)
|
||||
# Allocate a tty so SSH forwards Ctrl-C and hangup to the foreground process.
|
||||
command = 'exec python3 ' + REMOTE + '/tracking.py ' + shlex.join(sys.argv[1:])
|
||||
return subprocess.call(ssh[:-1] + ["-tt", host, command])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,472 @@
|
||||
"""Eye-camera pulse estimate and the heart-rate comparison tool; no headset needed."""
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import random
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import zipfile
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def load(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, ROOT / path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
pulse = load("pulse", "frame/tracking/pulse.py")
|
||||
check = load("heart_check", "scripts/heart-check.py")
|
||||
|
||||
|
||||
def synthetic(bpm, pulsing, seconds=40, fps=30, patches=48, noise=0.004, seed=7):
|
||||
"""Patch brightness with a faint pulse in `pulsing` patches, plus sensor
|
||||
noise, slow drift, blinks in some patches and eye movement in others."""
|
||||
rng = random.Random(seed)
|
||||
times = [i / fps for i in range(seconds * fps)]
|
||||
values = {}
|
||||
for k in range(patches):
|
||||
base, amplitude, trace = 40 + k, 0.003 if k < pulsing else 0.0, []
|
||||
for t in times:
|
||||
v = base * (1 + amplitude * math.sin(2 * math.pi * bpm / 60 * t)
|
||||
+ noise * rng.gauss(0, 1) + 0.02 * math.sin(0.1 * t + k))
|
||||
if k % 8 == 7 and int(t * 10) % 47 == 0:
|
||||
v *= 0.5 # blink
|
||||
if k % 8 == 6 and int(t * 10) % 23 == 0:
|
||||
v *= 1.3 # frequent eye movement
|
||||
trace.append(v)
|
||||
values[k] = trace
|
||||
return times, values
|
||||
|
||||
|
||||
class Estimate(unittest.TestCase):
|
||||
def test_finds_pulse(self):
|
||||
for bpm in (58, 72, 115):
|
||||
with self.subTest(bpm=bpm):
|
||||
result = pulse.estimate(*synthetic(bpm, 16))
|
||||
self.assertAlmostEqual(result["bpm"], bpm, delta=1.5)
|
||||
self.assertTrue(pulse.reliable(result))
|
||||
self.assertTrue(all(abs(b - bpm) <= 2 for _, b in result["series"]))
|
||||
|
||||
def test_noise_and_blinks_are_not_a_pulse(self):
|
||||
result = pulse.estimate(*synthetic(72, 0))
|
||||
self.assertFalse(pulse.reliable(result))
|
||||
|
||||
def test_frequent_spike_patches_are_dropped(self):
|
||||
times, values = synthetic(72, 16)
|
||||
result = pulse.estimate(times, values)
|
||||
self.assertLess(result["usable"], len(values))
|
||||
|
||||
def test_needs_enough_frames(self):
|
||||
with self.assertRaises(ValueError):
|
||||
pulse.estimate(*synthetic(72, 16, seconds=5))
|
||||
times, values = synthetic(72, 16)
|
||||
with self.assertRaises(ValueError):
|
||||
pulse.estimate(times, {k: [0.0] * len(times) for k in values}) # all dark
|
||||
|
||||
def test_series_times(self):
|
||||
times, values = synthetic(72, 16, seconds=20)
|
||||
series = pulse.estimate(times, values)["series"]
|
||||
self.assertAlmostEqual(series[0][0], pulse.WINDOW, delta=0.1)
|
||||
self.assertEqual(len(series), 20 - int(pulse.WINDOW) + 1)
|
||||
|
||||
def test_flat_patches_do_not_rank_first(self):
|
||||
times, values = synthetic(72, 16)
|
||||
values["flat"] = [100.0] * len(times)
|
||||
result = pulse.estimate(times, values)
|
||||
self.assertAlmostEqual(result["bpm"], 72, delta=1.5)
|
||||
self.assertEqual(pulse.snr([(60.0, 0.0), (61.0, 0.0)], 60), 0.0)
|
||||
|
||||
def test_analyse_uses_nearest_frame(self):
|
||||
times, values = synthetic(72, 4, patches=4, seconds=10)
|
||||
# Right-eye frames 1 ms before each left frame: nearest is that frame.
|
||||
frames = [(t, "left", [values[k][i] for k in range(4)]) for i, t in enumerate(times)]
|
||||
frames += [(t - 0.001, "right", [float(i)] * 4) for i, t in enumerate(times)]
|
||||
seen = {}
|
||||
original = pulse.estimate
|
||||
with patch.object(pulse, "estimate", lambda t, p: seen.update(p) or original(t, p)):
|
||||
pulse.analyse(frames)
|
||||
self.assertEqual(seen["right0"][:5], [0.0, 0.0, 1.0, 1.0, 2.0])
|
||||
|
||||
def test_fft_matches_dft(self):
|
||||
signal = [math.sin(i) + (i % 3) for i in range(16)]
|
||||
fast = pulse.fft(signal)
|
||||
for k in range(16):
|
||||
slow = sum(signal[n] * complex(math.cos(2 * math.pi * k * n / 16), -math.sin(2 * math.pi * k * n / 16))
|
||||
for n in range(16))
|
||||
self.assertAlmostEqual(abs(fast[k] - slow), 0, places=9)
|
||||
|
||||
def test_grid_means(self):
|
||||
# 4 × 4 image in RGB with padding: left half 10, right half 30 (channel 0).
|
||||
width, height, channels, stride = 4, 4, 3, 16
|
||||
pixels = bytearray(stride * height)
|
||||
for y in range(height):
|
||||
for x in range(width):
|
||||
pixels[y * stride + x * channels] = 10 if x < 2 else 30
|
||||
pixels[y * stride + x * channels + 1] = 255 # other channels ignored
|
||||
self.assertEqual(pulse.grid_means(bytes(pixels), width, height, stride, channels, grid=2),
|
||||
[10, 30, 10, 30])
|
||||
|
||||
def test_analyse_combines_both_eyes(self):
|
||||
times, values = synthetic(80, 16, patches=16)
|
||||
frames = []
|
||||
for i, t in enumerate(times):
|
||||
frames.append((t, "left", [values[k][i] for k in range(16)]))
|
||||
frames.append((t + 0.001, "right", [values[k][i] for k in range(16)]))
|
||||
result = pulse.analyse(frames)
|
||||
self.assertAlmostEqual(result["bpm"], 80, delta=1.5)
|
||||
self.assertEqual(result["usable"] % 2, 0)
|
||||
|
||||
|
||||
class FakeCaptureTool:
|
||||
"""Stands in for `eyetracking --calib`: writes PNG names over a few polls."""
|
||||
def __init__(self, directory, frames=6, fail=False):
|
||||
self.directory, self.frames, self.fail = Path(directory), frames, fail
|
||||
self.polls = 0
|
||||
self.returncode = None
|
||||
self.stdout = None
|
||||
self.announce = None # a different directory to report, if set
|
||||
|
||||
def __call__(self, command, cwd, stdout, stderr):
|
||||
self.command = command
|
||||
self.stdout = stdout
|
||||
if self.fail:
|
||||
stdout.write("Failed to initialize cameras\n")
|
||||
stdout.flush()
|
||||
self.returncode = 1
|
||||
return self
|
||||
# Real output is block-buffered until exit, so nothing is printed here.
|
||||
stdout.flush()
|
||||
self.directory.mkdir()
|
||||
return self
|
||||
|
||||
def poll(self):
|
||||
if self.returncode is not None:
|
||||
return self.returncode
|
||||
if self.polls < self.frames:
|
||||
for eye in ("left", "right"):
|
||||
(self.directory / f"{eye}_{self.polls}.png").write_bytes(b"png")
|
||||
self.polls += 1
|
||||
return None
|
||||
meta = {"frames": [{eye: {"fname": str(self.directory / f"{eye}_{i}.png"), "frameNum": i + 10,
|
||||
"tsMono": 100 + i / 90, "valid": i != 2} for eye in ("left", "right")}
|
||||
for i in range(self.frames)]}
|
||||
(self.directory / "meta.json").write_text(json.dumps(meta))
|
||||
# The real tool's buffered output only appears once it exits.
|
||||
self.stdout.write(f"Writing capture to: {self.announce or self.directory}\nCaptured images\n")
|
||||
self.stdout.flush()
|
||||
self.returncode = 0
|
||||
return 0
|
||||
|
||||
def terminate(self):
|
||||
raise AssertionError("the capture tool must never be signalled")
|
||||
|
||||
kill = terminate
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return self.returncode
|
||||
|
||||
|
||||
class WornCheck(unittest.TestCase):
|
||||
def sensor(self, name, raw):
|
||||
root = Path(self.enterContext(tempfile.TemporaryDirectory()))
|
||||
device = root / "iio:device2"
|
||||
device.mkdir()
|
||||
(device / "name").write_text(name + "\n")
|
||||
(device / "in_proximity_raw").write_text(raw + "\n")
|
||||
return root
|
||||
|
||||
def test_reads_the_proximity_sensor(self):
|
||||
self.assertEqual(pulse.proximity(self.sensor("vcnl4000", "3.250000000")), 3.25)
|
||||
|
||||
def test_unreadable_sensor_does_not_block(self):
|
||||
self.assertIsNone(pulse.proximity(self.sensor("other", "9")))
|
||||
self.assertIsNone(pulse.proximity(self.sensor("vcnl4000", "junk")))
|
||||
self.assertIsNone(pulse.proximity(Path("/nonexistent")))
|
||||
self.assertTrue(pulse.worn(None))
|
||||
|
||||
def test_low_reading_means_unworn(self):
|
||||
self.assertFalse(pulse.worn(3.1))
|
||||
self.assertTrue(pulse.worn(pulse.WORN_MIN))
|
||||
|
||||
|
||||
class CaptureLifecycle(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.root = Path(tempfile.mkdtemp())
|
||||
self.directory = self.root / "etcalib_test"
|
||||
(self.root / "etcalib_older").mkdir() # an earlier capture is not ours
|
||||
self.loaded = []
|
||||
|
||||
def tearDown(self):
|
||||
import shutil
|
||||
shutil.rmtree(self.root, ignore_errors=True)
|
||||
|
||||
def loader(self, path):
|
||||
self.loaded.append(Path(path).name)
|
||||
self.assertTrue(Path(path).exists())
|
||||
return [float(len(self.loaded))]
|
||||
|
||||
def capture(self, tool):
|
||||
class TestCapture(pulse.Capture):
|
||||
# A temporary directory stands in for /tmp/etcalib_*.
|
||||
PREFIX = str(self.root / "etcalib_")
|
||||
capture = TestCapture(20, runner=tool, loader=self.loader, workers=0)
|
||||
with patch.object(pulse.time, "sleep", lambda s: None):
|
||||
return capture, capture.run()
|
||||
|
||||
def test_reduces_deletes_and_joins_metadata(self):
|
||||
tool = FakeCaptureTool(self.directory)
|
||||
capture, frames = self.capture(tool)
|
||||
self.assertEqual(sorted(self.loaded), sorted(f"{e}_{i}.png" for e in ("left", "right") for i in range(6)))
|
||||
self.assertFalse(self.directory.exists()) # images and metadata removed
|
||||
self.assertTrue((self.root / "etcalib_older").exists())
|
||||
self.assertEqual(len(frames), 10) # frame 2 invalid in both eyes
|
||||
self.assertEqual(tool.command[-2:], ["--calib", "20"])
|
||||
self.assertTrue(all(isinstance(t, float) and eye in ("left", "right") for t, eye, _ in frames))
|
||||
|
||||
def test_camera_failure(self):
|
||||
tool = FakeCaptureTool(self.directory, fail=True)
|
||||
with self.assertRaises(RuntimeError) as raised:
|
||||
self.capture(tool)
|
||||
self.assertIn("cameras unavailable", str(raised.exception))
|
||||
|
||||
def test_backlog_abandons_capture_and_removes_images(self):
|
||||
class Stalled(FakeCaptureTool):
|
||||
def poll(self):
|
||||
if self.polls > 8:
|
||||
self.returncode = 0 # the bounded run ends by itself
|
||||
return 0
|
||||
for i in range(20):
|
||||
for eye in ("left", "right"):
|
||||
(self.directory / f"{eye}_{self.polls * 20 + i}.png").write_bytes(b"png")
|
||||
self.polls += 1
|
||||
return None
|
||||
tool = Stalled(self.directory)
|
||||
class TestCapture(pulse.Capture):
|
||||
PREFIX = str(self.root / "etcalib_")
|
||||
MAX_BACKLOG = 50
|
||||
capture = TestCapture(20, runner=tool, loader=self.loader, workers=0)
|
||||
capture.reduce = lambda final=False, original=capture.reduce: (
|
||||
original(final) if tool.polls > 3 else None) # reduction stalls
|
||||
with patch.object(pulse.time, "sleep", lambda s: None), self.assertRaises(RuntimeError) as raised:
|
||||
capture.run()
|
||||
self.assertIn("fell behind", str(raised.exception))
|
||||
self.assertEqual(tool.returncode, 0) # left to end by itself, never signalled
|
||||
self.assertFalse(self.directory.exists()) # no image left behind
|
||||
|
||||
def test_second_capture_directory_stops_and_removes_both(self):
|
||||
foreign = self.root / "etcalib_foreign"
|
||||
class Racing(FakeCaptureTool):
|
||||
def poll(self):
|
||||
if self.polls == 2:
|
||||
foreign.mkdir()
|
||||
(foreign / "left_0.png").write_bytes(b"png")
|
||||
return super().poll()
|
||||
tool = Racing(self.directory)
|
||||
with self.assertRaises(RuntimeError) as raised:
|
||||
self.capture(tool)
|
||||
self.assertIn("another eye-camera capture", str(raised.exception))
|
||||
self.assertFalse(self.directory.exists())
|
||||
self.assertFalse(foreign.exists()) # no eye image outlives the run
|
||||
self.assertTrue((self.root / "etcalib_older").exists())
|
||||
|
||||
def test_images_elsewhere_are_not_used_but_are_removed(self):
|
||||
tool = FakeCaptureTool(self.directory)
|
||||
tool.announce = self.root / "etcalib_reported"
|
||||
tool.announce.mkdir()
|
||||
(tool.announce / "left_0.png").write_bytes(b"png")
|
||||
with self.assertRaises(RuntimeError) as raised:
|
||||
self.capture(tool)
|
||||
self.assertIn("somewhere else", str(raised.exception))
|
||||
self.assertFalse(tool.announce.exists())
|
||||
self.assertFalse(self.directory.exists())
|
||||
|
||||
def test_interrupt_during_cleanup_still_removes_images(self):
|
||||
class Stubborn(FakeCaptureTool):
|
||||
interrupts = 0
|
||||
def poll(self):
|
||||
if self.polls == 2 and self.interrupts < 2:
|
||||
self.interrupts += 1
|
||||
raise KeyboardInterrupt # Ctrl-C mid-capture, and again while waiting
|
||||
return super().poll()
|
||||
tool = Stubborn(self.directory)
|
||||
with self.assertRaises(KeyboardInterrupt):
|
||||
self.capture(tool)
|
||||
self.assertEqual(tool.returncode, 0) # still let finish, never signalled
|
||||
self.assertFalse(self.directory.exists())
|
||||
|
||||
def test_overrunning_tool_is_killed_as_a_last_resort(self):
|
||||
class Hung(FakeCaptureTool):
|
||||
killed = False
|
||||
def poll(self):
|
||||
(self.directory / "left_0.png").write_bytes(b"png")
|
||||
return None if not self.killed else -9
|
||||
def kill(self):
|
||||
self.killed = True
|
||||
tool = Hung(self.directory)
|
||||
clock = iter(range(0, 10000, 20))
|
||||
with patch.object(pulse.time, "monotonic", lambda: next(clock)), \
|
||||
self.assertRaises(RuntimeError) as raised:
|
||||
self.capture(tool)
|
||||
self.assertIn("may need a reboot", str(raised.exception))
|
||||
self.assertTrue(tool.killed)
|
||||
self.assertFalse(self.directory.exists())
|
||||
|
||||
def test_cleanup_tries_every_directory(self):
|
||||
capture = pulse.Capture(20)
|
||||
capture.PREFIX = str(self.root / "etcalib_")
|
||||
first, second = self.root / "etcalib_a", self.root / "etcalib_b"
|
||||
for directory in (first, second):
|
||||
directory.mkdir()
|
||||
(directory / "left_0.png").write_bytes(b"png")
|
||||
capture.new = {first, second}
|
||||
real = pulse.shutil.rmtree
|
||||
def flaky(path):
|
||||
if Path(path) == first:
|
||||
raise OSError("busy")
|
||||
real(path)
|
||||
with patch.object(pulse.shutil, "rmtree", flaky), self.assertRaises(RuntimeError) as raised:
|
||||
capture.remove()
|
||||
self.assertIn("etcalib_a", str(raised.exception))
|
||||
self.assertFalse(second.exists())
|
||||
|
||||
def test_removes_a_directory_seen_only_at_the_end(self):
|
||||
capture = pulse.Capture(20)
|
||||
capture.PREFIX = str(self.root / "etcalib_")
|
||||
capture.before = capture.candidates()
|
||||
late = self.root / "etcalib_late"
|
||||
late.mkdir()
|
||||
(late / "left_0.png").write_bytes(b"png")
|
||||
capture.remove()
|
||||
self.assertFalse(late.exists())
|
||||
self.assertTrue((self.root / "etcalib_older").exists())
|
||||
|
||||
def test_never_adopts_directories_without_a_baseline(self):
|
||||
capture = pulse.Capture(20)
|
||||
capture.PREFIX = str(self.root / "etcalib_")
|
||||
capture.remove()
|
||||
self.assertTrue((self.root / "etcalib_older").exists())
|
||||
|
||||
def test_only_removes_capture_directories(self):
|
||||
capture = pulse.Capture(20)
|
||||
capture.directory = self.root
|
||||
capture.remove()
|
||||
self.assertTrue(self.root.exists())
|
||||
|
||||
|
||||
class HeartCheck(unittest.TestCase):
|
||||
def write(self, name, text):
|
||||
path = Path(self.enterContext(tempfile.TemporaryDirectory())) / name
|
||||
path.write_text(text)
|
||||
return path
|
||||
|
||||
def test_osc_round_trip(self):
|
||||
tracking = load("tracking", "frame/tracking/tracking.py")
|
||||
self.assertEqual(check.parse_osc(tracking.osc_message("/avatar/parameters/HeartRate", [72])),
|
||||
("/avatar/parameters/HeartRate", [72]))
|
||||
address, values = check.parse_osc(tracking.osc_message("/x", [1.5, -2.0]))
|
||||
self.assertEqual((address, values), ("/x", [1.5, -2.0]))
|
||||
with self.assertRaises(ValueError):
|
||||
check.parse_osc(b"/x\0\0,s\0\0abc\0")
|
||||
|
||||
def test_listen_records_readings(self):
|
||||
tracking = load("tracking", "frame/tracking/tracking.py")
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
out = Path(directory) / "ours.csv"
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as probe:
|
||||
probe.bind(("127.0.0.1", 0))
|
||||
port = probe.getsockname()[1]
|
||||
import threading
|
||||
def send():
|
||||
import time
|
||||
time.sleep(0.3)
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sender:
|
||||
sender.sendto(tracking.osc_message(check.ADDRESS, [72]), ("127.0.0.1", port))
|
||||
sender.sendto(tracking.osc_message("/other", [1]), ("127.0.0.1", port))
|
||||
threading.Thread(target=send).start()
|
||||
shown = io.StringIO()
|
||||
count = check.listen(port, check.ADDRESS, str(out), 1.5, stream=shown)
|
||||
self.assertEqual(count, 1)
|
||||
self.assertIn("72 bpm", shown.getvalue())
|
||||
self.assertEqual(out.read_text().splitlines()[1].split(",")[1], "72")
|
||||
self.assertEqual(out.stat().st_mode & 0o777, 0o600)
|
||||
|
||||
def test_compare_finds_lag_and_contact_loss(self):
|
||||
reference = [(1000.0 + i, 60 + i, 6) for i in range(40)] + [(1040.0 + i, 150, 4) for i in range(5)]
|
||||
ours = [(1002.0 + i, 60 + i) for i in range(40)] # 2 s late, nothing during contact loss
|
||||
ref = self.write("ref.csv", "unix_seconds,bpm,flags\n" + "".join(f"{t},{b},{f}\n" for t, b, f in reference))
|
||||
mine = self.write("ours.csv", "unix_seconds,bpm\n" + "".join(f"{t},{b}\n" for t, b in ours))
|
||||
result = check.compare(check.read_csv(mine), check.read_any(ref))
|
||||
self.assertEqual(result["lag_seconds"], 2.0)
|
||||
self.assertEqual(result["mean_abs_error"], 0)
|
||||
self.assertEqual(result["shown_during_no_contact"], 0)
|
||||
self.assertTrue(check.report(result, 5, stream=io.StringIO()))
|
||||
# A stale reading sent during contact loss fails the check.
|
||||
stale = check.read_csv(mine) + [(1043.0, 99)]
|
||||
result = check.compare(stale, check.read_any(ref))
|
||||
self.assertEqual(result["shown_during_no_contact"], 1)
|
||||
self.assertFalse(check.report(result, 5, stream=io.StringIO()))
|
||||
|
||||
def test_compare_against_apple_health_export(self):
|
||||
records = "".join(
|
||||
f'<Record type="HKQuantityTypeIdentifierHeartRate" unit="count/min" '
|
||||
f'startDate="2026-09-29 12:00:{s:02d} +1000" endDate="2026-09-29 12:00:{s:02d} +1000" value="{70 + s % 3}"/>'
|
||||
for s in range(0, 60, 5))
|
||||
other = '<Record type="HKQuantityTypeIdentifierStepCount" startDate="2026-09-29 12:00:00 +1000" value="9"/>'
|
||||
xml = f'<?xml version="1.0"?><HealthData>{other}{records}</HealthData>'
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
archive = Path(directory) / "export.zip"
|
||||
with zipfile.ZipFile(archive, "w") as z:
|
||||
z.writestr("apple_health_export/export.xml", xml)
|
||||
start = check.parse_time("2026-09-29 12:00:00 +1000")
|
||||
samples = check.read_any(archive, start - 60, start + 120)
|
||||
self.assertEqual(len(samples), 12)
|
||||
self.assertEqual(samples[0], (start, 70))
|
||||
ours = [(start + i + 1.0, 71) for i in range(60)]
|
||||
result = check.compare(ours, samples)
|
||||
self.assertLessEqual(result["mean_abs_error"], 1)
|
||||
|
||||
def test_unusual_flags_and_missing_export(self):
|
||||
path = self.write("ref.csv", "time,bpm,flags\n1000,70,6.0\n1001,71,yes\n1002,72,4\n")
|
||||
self.assertEqual(check.read_csv(path), [(1000.0, 70), (1001.0, 71), (1002.0, None)])
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
archive = Path(directory) / "other.zip"
|
||||
with zipfile.ZipFile(archive, "w") as z:
|
||||
z.writestr("notes.txt", "x")
|
||||
with self.assertRaises(ValueError):
|
||||
check.read_any(archive, 0, 1)
|
||||
|
||||
def test_compare_reports_bad_input_without_traceback(self):
|
||||
good = self.write("ref.csv", "1000,70\n")
|
||||
shown = io.StringIO()
|
||||
with patch("sys.stdout", shown):
|
||||
self.assertEqual(check.main(["compare", str(good.parent / "missing.csv"), str(good)]), 1)
|
||||
self.assertEqual(check.main(["compare", str(self.write("ours.csv", "1000,inf\n1001,70\n")),
|
||||
str(self.write("bad.xml", "<not closed"))]), 1)
|
||||
self.assertEqual(shown.getvalue().count("Could not compare"), 2)
|
||||
|
||||
def test_health_records_with_non_finite_values_are_skipped(self):
|
||||
record = '<Record type="HKQuantityTypeIdentifierHeartRate" startDate="2026-09-29 12:00:00 +1000" value="%s"/>'
|
||||
path = self.write("export.xml", "<HealthData>" + record % "inf" + record % "72" + "</HealthData>")
|
||||
start = check.parse_time("2026-09-29 12:00:00 +1000")
|
||||
self.assertEqual(check.read_health(path, start - 1, start + 1), [(start, 72)])
|
||||
|
||||
def test_time_formats(self):
|
||||
self.assertEqual(check.parse_time("1700000000.5"), 1700000000.5)
|
||||
self.assertEqual(check.parse_time("2023-11-14T22:13:20Z"), 1700000000)
|
||||
self.assertEqual(check.parse_time("2023-11-15 08:13:20 +1000"), 1700000000)
|
||||
with self.assertRaises(ValueError):
|
||||
check.parse_time("yesterday")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,188 @@
|
||||
"""Tracking protocols and fake-Frame BlueZ lifecycle; no headset or strap needed."""
|
||||
import importlib.util
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import stat
|
||||
import struct
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
SPEC = importlib.util.spec_from_file_location("tracking", Path(__file__).resolve().parents[1] / "frame/tracking/tracking.py")
|
||||
t = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(t)
|
||||
|
||||
|
||||
class Protocols(unittest.TestCase):
|
||||
def test_hrs_formats(self):
|
||||
self.assertEqual(t.heart_rate(b"\x00\x48"), 72)
|
||||
self.assertEqual(t.heart_rate(b"\x01\x2c\x01"), 300)
|
||||
self.assertEqual(t.heart_rate(b"\x1e\x48\x01\x00\x00\x04\x00\x04"), 72)
|
||||
self.assertEqual(t.heart_rate(b"\x02\x48"), 72) # contact not supported
|
||||
self.assertIsNone(t.heart_rate(b"\x04\x48")) # no contact
|
||||
self.assertIsNone(t.heart_rate(b"\x00\x00"))
|
||||
|
||||
def test_hrs_malformed(self):
|
||||
for packet in (b"", b"\x00", b"\x01\x48", b"\x08\x48\x00", b"\x10\x48",
|
||||
b"\x10\x48\x00", b"\x00\x48\x01", b"\xe0\x48"):
|
||||
with self.subTest(packet=packet), self.assertRaises(ValueError):
|
||||
t.heart_rate(packet)
|
||||
|
||||
def test_gaze_coordinates(self):
|
||||
self.assertEqual(t.gaze_angles([0, 0, 0, 1]), (0, 0))
|
||||
angle = math.radians(15)
|
||||
pitch, yaw = t.gaze_angles([math.sin(angle), 0, 0, math.cos(angle)])
|
||||
self.assertAlmostEqual(pitch, -30) # OpenXR +X rotation looks up
|
||||
self.assertAlmostEqual(yaw, 0)
|
||||
pitch, yaw = t.gaze_angles([0, -math.sin(angle), 0, math.cos(angle)])
|
||||
self.assertAlmostEqual(pitch, 0)
|
||||
self.assertAlmostEqual(yaw, 30) # right
|
||||
|
||||
def test_invalid_gaze(self):
|
||||
for pose in ([0, 0, 0, 0], [math.nan, 0, 0, 1], [0, 0, 0], [0, 0, 0, math.inf]):
|
||||
with self.assertRaises(ValueError):
|
||||
t.gaze_angles(pose)
|
||||
|
||||
def test_osc_wire(self):
|
||||
self.assertEqual(t.osc_message('/x', [72]), b'/x\0\0,i\0\0' + struct.pack('>i', 72))
|
||||
self.assertEqual(t.osc_message('/x', [1.0, -2.0]), b'/x\0\0,ff\0' + struct.pack('>ff', 1, -2))
|
||||
for address in ('x', '/x\0y', '/x y', '/x*'):
|
||||
with self.assertRaises(ValueError):
|
||||
t.osc_message(address, [1])
|
||||
|
||||
def test_default_never_opens_socket(self):
|
||||
with patch.object(t.socket, 'socket') as create:
|
||||
osc = t.Osc()
|
||||
osc.send('/x', [72])
|
||||
osc.close()
|
||||
create.assert_not_called()
|
||||
|
||||
def test_only_configured_endpoint(self):
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as receiver:
|
||||
receiver.bind(('127.0.0.1', 0))
|
||||
receiver.settimeout(1)
|
||||
osc = t.Osc(receiver.getsockname())
|
||||
try:
|
||||
osc.send('/tracking/eye/CenterPitchYaw', [0.0, 30.0])
|
||||
self.assertEqual(receiver.recv(1024), t.osc_message('/tracking/eye/CenterPitchYaw', [0.0, 30.0]))
|
||||
finally:
|
||||
osc.close()
|
||||
|
||||
def test_endpoint_validation(self):
|
||||
for endpoint in [('example.org', 9000), ('0.0.0.0', 9000), ('224.0.0.1', 9000), ('127.0.0.1', 0), ('::1', 65536)]:
|
||||
with self.assertRaises(ValueError):
|
||||
t.Osc(endpoint)
|
||||
|
||||
def test_heart_staleness_contact_and_log(self):
|
||||
now = [0]
|
||||
osc = Mock()
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / 'session.csv'
|
||||
session = t.HeartSession(osc, '/hr', path, lambda: now[0])
|
||||
self.assertIsNone(session.current())
|
||||
session.notification(b'\x00\x48')
|
||||
self.assertEqual(session.current(), 72)
|
||||
osc.send.assert_called_once_with('/hr', [72])
|
||||
now[0] = 6
|
||||
self.assertIsNone(session.current())
|
||||
session.notification(b'\x04\x48')
|
||||
self.assertIsNone(session.current())
|
||||
self.assertEqual(osc.send.call_count, 1)
|
||||
session.close()
|
||||
self.assertEqual(path.read_text().splitlines()[0], 'unix_seconds,bpm')
|
||||
self.assertEqual(len(path.read_text().splitlines()), 2)
|
||||
if os.name != 'nt':
|
||||
self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o600)
|
||||
with self.assertRaises(FileExistsError):
|
||||
t.HeartSession(osc, '/hr', path)
|
||||
|
||||
def test_no_log_by_default(self):
|
||||
with patch.object(t.os, 'open') as create:
|
||||
session = t.HeartSession(Mock(), '/hr')
|
||||
session.notification(b'\x00\x48')
|
||||
session.close()
|
||||
create.assert_not_called()
|
||||
|
||||
|
||||
class FakeBluez(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.device = '/org/bluez/hci0/dev_TEST'
|
||||
self.service = self.device + '/service1'
|
||||
self.char = self.service + '/char1'
|
||||
self.objects = {
|
||||
self.device: {t.DEVICE: {'Address': 'AA:BB:CC:DD:EE:FF', 'Connected': False, 'ServicesResolved': True}},
|
||||
self.service: {t.SERVICE: {'UUID': t.HRS, 'Device': self.device}},
|
||||
self.char: {t.CHARACTERISTIC: {'UUID': t.MEASUREMENT, 'Service': self.service, 'Flags': ['notify']}},
|
||||
}
|
||||
self.api = Mock()
|
||||
self.api.GetManagedObjects.side_effect = lambda: self.objects
|
||||
self.bus = Mock()
|
||||
self.interface = Mock(return_value=self.api)
|
||||
self.values = Mock()
|
||||
|
||||
def reader(self):
|
||||
return t.BluezHeart(self.bus, self.interface, 'AA:BB:CC:DD:EE:FF', self.values)
|
||||
|
||||
def test_subscribe_receive_and_cleanup(self):
|
||||
reader = self.reader()
|
||||
self.api.Connect.assert_called_once()
|
||||
self.assertTrue(reader.subscribe())
|
||||
self.api.StartNotify.assert_called_once()
|
||||
reader.changed(t.CHARACTERISTIC, {'Value': [0, 72]}, [], self.char)
|
||||
self.values.assert_called_once_with([0, 72])
|
||||
reader.changed(t.CHARACTERISTIC, {'Value': [0, 73]}, [], '/other/strap')
|
||||
self.assertEqual(self.values.call_count, 1)
|
||||
reader.close()
|
||||
self.api.StopNotify.assert_called_once()
|
||||
self.api.Disconnect.assert_called_once()
|
||||
self.bus.add_signal_receiver.return_value.remove.assert_called_once()
|
||||
|
||||
def test_preserve_existing_connection(self):
|
||||
self.objects[self.device][t.DEVICE]['Connected'] = True
|
||||
reader = self.reader()
|
||||
reader.subscribe()
|
||||
reader.close()
|
||||
self.api.Connect.assert_not_called()
|
||||
self.api.Disconnect.assert_not_called()
|
||||
|
||||
def test_only_selected_device_service(self):
|
||||
self.objects[self.service][t.SERVICE]['Device'] = '/other/device'
|
||||
reader = self.reader()
|
||||
try:
|
||||
with self.assertRaises(RuntimeError):
|
||||
reader.subscribe()
|
||||
finally:
|
||||
reader.close()
|
||||
self.api.StartNotify.assert_not_called()
|
||||
|
||||
def test_wait_for_services(self):
|
||||
self.objects[self.device][t.DEVICE]['ServicesResolved'] = False
|
||||
reader = self.reader()
|
||||
self.assertFalse(reader.subscribe())
|
||||
reader.close()
|
||||
self.api.StartNotify.assert_not_called()
|
||||
self.api.StopNotify.assert_not_called()
|
||||
|
||||
def test_disconnect_notification(self):
|
||||
reader = self.reader()
|
||||
reader.changed(t.DEVICE, {'Connected': 0}, [], self.device) # dbus.Boolean behaves as int
|
||||
self.values.assert_called_once_with(None)
|
||||
reader.close()
|
||||
|
||||
def test_failed_notify_cleans_connection(self):
|
||||
reader = self.reader()
|
||||
self.api.StartNotify.side_effect = RuntimeError('failure')
|
||||
with self.assertRaises(RuntimeError):
|
||||
reader.subscribe()
|
||||
reader.close()
|
||||
self.api.StopNotify.assert_not_called()
|
||||
self.api.Disconnect.assert_called_once()
|
||||
|
||||
def test_unknown_device_does_not_connect_or_scan(self):
|
||||
self.objects.clear()
|
||||
with self.assertRaises(RuntimeError):
|
||||
self.reader()
|
||||
self.api.Connect.assert_not_called()
|
||||
self.api.StartDiscovery.assert_not_called()
|
||||
Reference in new issue
Block a user