Hands: check the tracking cameras before recording, and watch for losing them

After the headset wakes, XRService sometimes fails to load the colour module's
VCINT FPGA image; then only the two side cameras run, without the IR light,
and the tracker finds no hands. hands/camcheck.py reads XRService's log, the
video nodes it holds and ft-camd's ring, and says ok, degraded or unknown.

The recorder won't start while degraded (--ignore-cameras overrides it), offers
a confirmed SteamVR restart, and stops the first hand-size step when the
tracker sees no hand at all. ft-camwatch (unit file only, not enabled) follows
the log, notifies, and with CAMWATCH_AUTO_RESTART=1 restarts SteamVR when the
headset isn't worn and nothing else uses VR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hands/README.md

@@ -207,6 +207,49 @@ Python, with NumPy and OpenCV. `setup/dev-container.sh` doesn't install them, be
 
 To try the hand cutouts without restarting the desktop, `screens/build/ft-handtest [--distance m] [--width m] [--seconds s]` (built by `screens/build.sh`, run in the dev container, with hand tracking on) shows a test panel of its own, a light grid 1 m wide and 0.8 m ahead by default, and cuts your hands out of it the way ft-screens cuts them out of the screens.
 
+## Camera check
+
+Hand tracking needs all four mono cameras and the headset's IR light. With the Arcturus colour module attached, SteamVR's XRService loads an FPGA image ("VCINT") onto the module every time it opens the cameras: when SteamVR starts and after every wake. When that load fails, XRService runs only the two side cameras, the frames come out darker and noisier, and ft-hands finds no hands at all. It happened on 2026-10-02 at 17:02, after the headset slept; the hand recorder then said "I can't see your hands" for a whole session.
+
+`hands/camcheck.py` (system Python, standard library) tells whether the four cameras run: `ok`, `degraded: upper cameras and IR light off (VCINT FPGA failed to load)` (or another `degraded:` reason), or `unknown` (SteamVR not running, the cameras closed while the headset sleeps). It prints the log lines and other evidence it used; `--json` is for programs; the exit status is 0, 1 or 2. It reads:
+- the running XRService's log (`~/.local/share/Steam/logs/xrservice.txt`): the last camera start (from the FPGA check to the next "Closing tracking camera interfaces"), its VCINT result, `Upper cameras FPGA interleaving support: N`, `Created N tasks (T tracking, P passthrough)` and the `TrackingCameraInit` lines. A wake that works prints no "Created N tasks", so an older one doesn't count;
+- which `/dev/video*` XRService has open (`/proc/PID/fd`; video9 and video13 are the side pair, video6 and video7 the upper pair). Only on the host: the dev container can't read another process's open files, so there it's skipped;
+- ft-camd's ring header, when it runs: how many mono cameras it publishes.
+
+The hand recorder runs it before a session (DESIGN.md, "Camera check"). `hands/tests/test_camcheck.py` runs it on the 2026-10-02 log cut at several points, and on made-up logs.
+
+### The watcher (off by default)
+
+`hands/ft-camwatch` follows the XRService log (a stat every 2 s, reading only what's new). On a VCINT failure it posts a notification in the Frametop desktop, on the desktop's own D-Bus, found through its plasmashell as `decoration/apply.sh` does. With `CAMWATCH_AUTO_RESTART=1` in `~/.config/frametop.conf` it also restarts SteamVR, but only:
+- while the headset isn't worn (frame-job's check: `vrcompositor` runs and a `/sys/class/backlight/*/brightness` is over 0), and after it has been off for `CAMWATCH_IDLE_S` (60);
+- with no app Steam launched (`SteamLaunch AppId=N` in a process's arguments; `CAMWATCH_IGNORE_APPIDS` lists ids that don't count) and nobody on the remote desktop (an established connection to the VNC port, `VNC_PORT`, 5900);
+- once per failure, and not again within `CAMWATCH_COOLDOWN_MIN` (30) of the last automatic restart. It remembers both in `~/.local/state/frametop/camwatch.json`, so its own restart doesn't reset them.
+
+It logs every decision to the journal. `ft-camwatch --once` prints the state and what it would do, and does nothing; `--dry-run` keeps watching without acting. `hands/frametop-camwatch.service` is the unit (a template, `@REPO@` as in the others; nothing installs or enables it yet). It isn't `PartOf=steamvr.service`, so it outlives the restart it asks for. `CAMWATCH_NOTIFY=0` turns the notification off. `hands/tests/test_camwatch.py` tests its decisions with made-up inputs.
+
+### What a SteamVR restart does to Frametop
+
+Read from the code on the experimental branch, not tried live:
+- ft-screens quits when SteamVR does: on `VREvent_Quit` it ends its Wayland display (`screens/vr.cpp`, `ft_vr_poll`; `screens/compositor.c`, `handle_vr_event`). It never connects to SteamVR again: `ft_vr_init` runs once, at its start.
+- KWin runs nested in ft-screens, so the Frametop desktop ends with it, every window in it too (the hand recorder's as well). Its unit, `frametop-desktop`, is a transient `systemd-run` unit with `Restart=no`, so the desktop doesn't come back by itself: start it again (Desktop in the library, or `desktops.sh start`).
+- When the unit stops, systemd ends what's left in it. `session/keep-apps.sh` moves programs started in the desktop out of the unit first, but only `desktops.sh stop` runs it; here they stop too. Programs in the dev container (ft-screens, the hand recorder) are in the container's cgroup and end when their Wayland connection goes.
+- The units that are `PartOf=steamvr.service` restart with it: `frametop-camd`, `frametop-hands`, the pointer helper, gaze and power, and the hand recorder's own transient ft-camd and ft-hands units.
+
+### Verified, and what's a guess
+
+Verified, from the XRService logs of 2026-10-01 and 2026-10-02 and the running system:
+- The failure's log lines and its effect: "Failed to load VCINT FPGA image when passthrough cameras are connected", interleaving support 0, "Created 4 tasks (2 tracking, 2 passthrough)", and only video9 and video13 opened. At 19:38-19:59 XRService held only those two of the four (plus video0 and video3), and ft-camd published two mono cameras.
+- A wake's load can work and can fail. Both wakes in the logs started from an FPGA that answered nothing ("ERROR/UNKNOWN"): the one at 2026-10-01 16:39 loaded VCINT, the one at 2026-10-02 17:02 failed ("FPGA config_done signal did not assert").
+- After a reboot the FPGA reads PASSTHRU and SteamVR's start loads VCINT (2026-10-01 21:53, 2026-10-02 13:39).
+- A SteamVR restart within a boot found VCINT still loaded and loaded nothing (2026-10-01 15:27): XRService checks the FPGA when it starts and loads only when it must.
+
+Guesses, not tested:
+- **Whether a SteamVR restart fixes it.** After a failed load the FPGA doesn't answer, so a new XRService would run the same load a wake runs, which has worked once and failed once. It's never been tried after a failure. If it doesn't help, only a reboot is known to work (the FPGA comes up as PASSTHRU, and the load at SteamVR's start has worked both times).
+- That the IR light is off because of the FPGA: the frames are darker and the illuminator ring isn't seen, and the FPGA loader lists a `room_led_en` pin, but nothing shows the light's state directly.
+- That a sleep and wake (taking the headset off long enough) would retry the load too: it should, since every wake loads VCINT, but no failure has been followed by a wake yet.
+- How the Frametop desktop behaves on a SteamVR restart (above): read from the code only.
+- That Steam-launched apps carry `SteamLaunch AppId=N`: from Steam on other Linux systems; no VR game has run on the Frame to confirm it.
+
 ## Build
 
 `hands/build.sh` builds in the dev container on the Frame, into `hands/build/`, with `hands/Makefile`. The first build fetches ncnn at a pinned tag (`NCNN_TAG` in the Makefile) and builds it into `hands/build/ncnn`, which takes a few minutes; `NCNN=DIR` points at an ncnn install already built instead. ft-camd is linked statically, because it runs on the host, which has an older glibc than the container.
@@ -218,4 +261,5 @@ To try the hand cutouts without restarting the desktop, `screens/build/ft-handte
 - **The colour calibration mapping isn't settled.** Which colour camera is `passthrough_left` (`HANDS_COLOR_LEFT`) and how the module's crop applies (`HANDS_COLOR_CROP`) still need `tools/check_color.py` on a recording with a lit, textured view.
 - **Depth when one camera loses the hand.** A hand seen in one camera drifts 10% per update toward the one-camera depth guess (`kMonoDepthGain`, 0.1, in `track/tracker.cpp`). In the 2026-09-30 replays that was worse than keeping the last distance (see "3D" above). A smaller gain, such as 0.02, is the next thing to try.
 - **Pinches aren't reliable enough for everyday use yet.** That's why hand tracking stays off until `ft-handsctl on`, and `POINTER_HANDS` is 0 by default.
+- **SteamVR can leave the upper cameras and the IR light off after a wake**, and then no hands are found. See "Camera check" above: `camcheck.py` tells, the hand recorder won't start a session, and the fix is a SteamVR restart or a reboot.
 - **Floating windows don't get hand cutouts.** Their panels show crops of the client buffer, which the cutouts' side-by-side buffer doesn't match (`screens/vr.cpp`, `UpdateCutouts`).

hands/camcheck.py

@@ -0,0 +1,422 @@
+#!/usr/bin/env python3
+"""camcheck: are the headset's four mono tracking cameras running, so that ft-camd and
+ft-hands see them all?
+
+The Frame has four mono IR tracking cameras: the side pair slam_left and slam_right
+(/dev/video9 and /dev/video13) and the upper pair (/dev/video6 and /dev/video7). With the
+Arcturus colour module attached, SteamVR's XRService loads an FPGA image ("VCINT") onto the
+module whenever it opens the cameras (at start and after every wake). When that load fails
+(seen 2026-10-02 17:02, after a sleep), XRService runs only the two side cameras, the IR
+illuminator seems to stay off, and ft-hands finds no hands at all.
+
+What it looks at, cheapest first, all read-only:
+  1. The newest XRService log (~/.local/share/Steam/logs/xrservice.txt, a symlink to the
+     running instance's log): the last camera start, its VCINT result, "Upper cameras FPGA
+     interleaving support: N", "Created N tasks (T tracking, P passthrough)" and the
+     TrackingCameraInit lines. A wake doesn't always print "Created N tasks", so the parser
+     tracks each camera start ("episode") from the FPGA check to the next close.
+  2. Which /dev/video* XRService has open (/proc/PID/fd). Only on the host: the dev container
+     can't read another process's fd table (checked 2026-10-02), and then this is skipped.
+  3. A running ft-camd's ring header (/run/user/UID/frametop-hands/cam-ring): how many mono
+     cameras it publishes.
+
+Status: "ok", "degraded: <why>" or "unknown" (SteamVR not running, the cameras closed while
+the headset sleeps, no log). Exit status 0, 1, 2 for those.
+
+  python3 hands/camcheck.py             # the status and its evidence
+  python3 hands/camcheck.py --json      # for programs
+  python3 hands/camcheck.py --log FILE --no-proc --no-ring   # a saved log only (tests)
+
+System Python, standard library only; session.py and ft-camwatch import it.
+"""
+import argparse
+import glob
+import json
+import os
+import re
+import struct
+import sys
+import time
+
+LOG_DIR = os.path.expanduser("~/.local/share/Steam/logs")
+LOG_LINK = os.path.join(LOG_DIR, "xrservice.txt")
+SIDE_NODES = (9, 13)     # slam_left, slam_right (TrackingCameraInit index 0 and 1)
+UPPER_NODES = (6, 7)     # the upper pair (index 2 and 3)
+TRACKING = 4
+
+VCINT_REASON = "upper cameras and IR light off (VCINT FPGA failed to load)"
+DEGRADED_VCINT = "degraded: " + VCINT_REASON
+
+# What the window and the recorder say when the check fails this way.
+USER_TEXT = ("The headset's upper cameras and IR light are off. SteamVR couldn't start the colour camera "
+             "module (it happens sometimes after the headset sleeps). Restart SteamVR, or restart the headset "
+             "if that doesn't fix it.")
+
+ANSI = re.compile(r"\x1b\[[0-9;]*m")
+STAMP = re.compile(r"^\w{3} \w{3} \d{2} \d{4} (\d{2}:\d{2}:\d{2})\.\d+ (\w+): ?(.*)$")
+# Lines worth reading; anything else is skipped before the regexes (the log grows by MBs a day).
+KEYS = ("FPGA", "VCINT", "Created", "TrackingCameraInit", "Closing tracking camera", "Streaming",
+        "systemd suspend", "systemd resume", "XRService logging to", "Exiting XRService")
+RE_PASSTHRU = re.compile(r"Passthrough connected but FPGA is (\S+) - loading VCINT")
+RE_INTERLEAVE = re.compile(r"Upper cameras FPGA interleaving support: (\d)")
+RE_TASKS = re.compile(r"Created (\d+) tasks \((\d+) tracking, (\d+) passthrough\)")
+RE_INIT = re.compile(r"TrackingCameraInit: index: (\d+)\. video device: /dev/video(\d+)")
+RE_STREAM = re.compile(r"Streaming resumed \(FPGA: (\S+), VC interleaving: (\w+)\)")
+RE_STATE = re.compile(r"FPGA state check: (\S+)")
+
+
+class LogState:
+    """Reads an XRService log line by line (feed), so the watcher can follow it as it grows.
+
+    An episode is one opening of the cameras: from the first FPGA, task or camera-init line
+    after the log starts or after "Closing tracking camera interfaces", to the next close."""
+
+    def __init__(self, path=""):
+        self.path = path
+        self.instance = ""        # the "XRService logging to" line's time
+        self.exited = False
+        self.closed = False       # the cameras were closed and haven't opened again
+        self.closed_at = ""
+        self.episode = None
+        self.nodes = {}           # TrackingCameraInit index -> /dev/videoN, from the whole log
+        self.failures = []        # [(time, line)]: every VCINT failure in this log
+        self.lines = 0
+
+    def _new_episode(self, t):
+        self.closed = False
+        self.episode = {"start": t, "fpga_before": "", "vcint": "", "interleave": None, "tasks": None,
+                        "inits": {}, "stream": "", "failure": "", "evidence": []}
+        if self.closed_at:
+            self.episode["evidence"].append(self.closed_at)
+
+    def _ep(self, t):
+        if self.episode is None or self.closed:
+            self._new_episode(t)
+        return self.episode
+
+    def feed(self, raw):
+        self.lines += 1
+        if not any(k in raw for k in KEYS):
+            return
+        line = ANSI.sub("", raw).rstrip("\n")
+        m = STAMP.match(line)
+        if not m:
+            return   # the FPGA loader's own output, without a time
+        t, _level, text = m.groups()
+        short = ("%s %s" % (t, text))[:220]
+        if "XRService logging to" in text:
+            lines = self.lines
+            self.__init__(self.path)
+            self.instance, self.lines = t, lines
+            return
+        if "Exiting XRService" in text:
+            self.exited = True
+            return
+        if "Closing tracking camera interfaces" in text:
+            self.closed, self.closed_at = True, short
+            return
+        if "systemd suspend notification" in text or "systemd resume notification" in text:
+            if "resume" in text:
+                self.closed_at = (self.closed_at + " / " if self.closed_at else "") + short
+            return
+        m = RE_PASSTHRU.search(text)
+        if m:
+            ep = self._ep(t)
+            ep["fpga_before"] = m.group(1)
+            ep["evidence"].append(short)
+            return
+        if "FPGA image VCINT loaded and verified successfully" in text:
+            ep = self._ep(t)
+            ep["vcint"] = "ok"
+            ep["evidence"].append(short)
+            return
+        if "Failed to load VCINT FPGA image" in text:
+            ep = self._ep(t)
+            ep["vcint"] = "failed"
+            ep["failure"] = t
+            ep["evidence"].append(short)
+            self.failures.append((t, short))
+            return
+        if "FPGA load failed" in text:
+            self._ep(t)["evidence"].append(short)
+            return
+        m = RE_INTERLEAVE.search(text)
+        if m:
+            ep = self._ep(t)
+            ep["interleave"] = int(m.group(1))
+            ep["evidence"].append(short)
+            return
+        m = RE_TASKS.search(text)
+        if m:
+            ep = self._ep(t)
+            ep["tasks"] = tuple(int(v) for v in m.groups())
+            ep["evidence"].append(short)
+            return
+        m = RE_INIT.search(text)
+        if m:
+            ep = self._ep(t)
+            idx, node = int(m.group(1)), int(m.group(2))
+            ep["inits"][idx] = node
+            self.nodes[idx] = node
+            ep["evidence"].append(short)
+            return
+        m = RE_STREAM.search(text)
+        if m:
+            ep = self._ep(t)
+            ep["stream"] = "%s, interleaving %s" % m.groups()
+            ep["evidence"].append(short)
+            return
+        m = RE_STATE.search(text)
+        if m:
+            ep = self._ep(t)
+            if not ep["fpga_before"] and not ep["vcint"]:
+                ep["fpga_before"] = m.group(1)
+                if m.group(1) == "VCINT":
+                    ep["vcint"] = "loaded"   # already there: no load needed (a SteamVR restart in the same boot)
+                ep["evidence"].append(short)
+
+    def feed_text(self, text):
+        for line in text.splitlines():
+            self.feed(line)
+        return self
+
+    def upper_nodes(self):
+        got = tuple(self.nodes[i] for i in (2, 3) if i in self.nodes)
+        return got if len(got) == 2 else UPPER_NODES
+
+    def tracking_nodes(self):
+        got = tuple(self.nodes[i] for i in range(TRACKING) if i in self.nodes)
+        return got if len(got) == TRACKING else SIDE_NODES + UPPER_NODES
+
+    def verdict(self):
+        """(status, reason, evidence): status "ok", "degraded" or "unknown"."""
+        if self.lines == 0:
+            return "unknown", "the XRService log is empty", []
+        if self.exited:
+            return "unknown", "XRService has exited (SteamVR isn't running)", []
+        if self.episode is None:
+            return "unknown", "the cameras haven't started yet in this log", []
+        ep = self.episode
+        ev = ep["evidence"][-14:]
+        if self.closed:
+            return "unknown", "the cameras are closed (the headset is asleep, or SteamVR is stopping)", \
+                ev + [self.closed_at]
+        tracking = len(ep["inits"]) if ep["inits"] else (ep["tasks"][1] if ep["tasks"] else None)
+        if ep["vcint"] == "failed":
+            return "degraded", VCINT_REASON, ev
+        if tracking is not None and tracking < TRACKING:
+            why = "only %d of %d tracking cameras running" % (tracking, TRACKING)
+            if ep["interleave"] == 0:
+                why += " (upper cameras' FPGA interleaving off)"
+            return "degraded", why, ev
+        if tracking == TRACKING:
+            return "ok", "%d tracking cameras running" % TRACKING, ev
+        return "unknown", "the cameras are starting", ev
+
+    def snapshot(self):
+        status, reason, ev = self.verdict()
+        ep = self.episode or {}
+        return {"status": status, "reason": reason, "evidence": ev, "log": self.path, "instance": self.instance,
+                "episode": {k: (list(v) if isinstance(v, tuple) else v) for k, v in ep.items() if k != "evidence"},
+                "failure": "%s@%s" % (self.path, ep["failure"]) if ep.get("vcint") == "failed" else ""}
+
+
+# ------------------------------------------------------------------------------------------
+# Processes
+
+def proc_argv(pid):
+    try:
+        with open("/proc/%s/cmdline" % pid, "rb") as f:
+            return [a.decode(errors="replace") for a in f.read().split(b"\0") if a]
+    except OSError:
+        return []
+
+
+def xrservice_pid():
+    """XRService's pid (its main thread renames itself XRServiceLoopTh), or None."""
+    for pid in os.listdir("/proc"):
+        if not pid.isdigit():
+            continue
+        try:
+            with open("/proc/%s/comm" % pid) as f:
+                if not f.read().startswith("XRService"):
+                    continue
+        except OSError:
+            continue
+        argv = proc_argv(pid)
+        if argv and os.path.basename(argv[0]) == "XRService":
+            return int(pid)
+    return None
+
+
+def xrservice_fds(pid):
+    """{"videos": [N, ...], "log": path or ""} from /proc/PID/fd, or None if it can't be read
+    (the dev container can't)."""
+    try:
+        fds = os.listdir("/proc/%d/fd" % pid)
+    except OSError:
+        return None
+    videos, log = set(), ""
+    for fd in fds:
+        try:
+            target = os.readlink("/proc/%d/fd/%s" % (pid, fd))
+        except OSError:
+            continue
+        m = re.match(r"/dev/video(\d+)$", target)
+        if m:
+            videos.add(int(m.group(1)))
+        elif re.search(r"/XRService-[^/]*\.log$", target):
+            log = target
+    if not videos and not log:
+        return None   # nothing readable: as good as no access
+    return {"videos": sorted(videos), "log": log}
+
+
+def newest_log():
+    """The running XRService's log: the xrservice.txt symlink, else the newest by time."""
+    if os.path.exists(LOG_LINK):
+        return os.path.realpath(LOG_LINK)
+    found = glob.glob(os.path.join(LOG_DIR, "XRService-*", "XRService-*.log"))
+    found += glob.glob(os.path.join(LOG_DIR, "XRService-*.log"))
+    found = [p for p in found if os.path.isfile(p)]
+    return max(found, key=os.path.getmtime) if found else ""
+
+
+def read_log(path):
+    st = LogState(path)
+    with open(path, "r", errors="replace") as f:
+        for line in f:
+            st.feed(line)
+    return st
+
+
+# ------------------------------------------------------------------------------------------
+# ft-camd's ring (camd/fhring.h; the header only, as session.py's Ring reads it)
+
+RING_HDR = struct.Struct("<8sIIIIQqQ16x")
+RING_CAM = struct.Struct("<32s32siIIIIIQQQQQIf24x")
+FH_CAM_DARK, FH_CAM_COLOR = 1, 2
+
+
+def default_ring():
+    return "/run/user/%d/frametop-hands/cam-ring" % os.getuid()
+
+
+def read_ring(path):
+    """{"alive", "writer_pid", "mono": [{"name", "sensor", "node"}]} or None (no ring)."""
+    try:
+        with open(path, "rb") as f:
+            data = f.read(RING_HDR.size + 8 * RING_CAM.size)
+    except OSError:
+        return None
+    if len(data) < RING_HDR.size:
+        return None
+    magic, version, _, ncams, _, _, writer, _ = RING_HDR.unpack_from(data, 0)
+    if magic != b"FHRING01" or version != 1:
+        return None
+    hb = struct.unpack_from("<Q", data, 40)[0]
+    alive = hb != 0 and (time.clock_gettime_ns(time.CLOCK_MONOTONIC) - hb) / 1e9 < 2.0
+    mono = []
+    for i in range(min(ncams, 8)):
+        off = RING_HDR.size + i * RING_CAM.size
+        if off + RING_CAM.size > len(data):
+            break
+        f = RING_CAM.unpack_from(data, off)
+        sensor = f[0].split(b"\0", 1)[0].decode(errors="replace")
+        name = f[1].split(b"\0", 1)[0].decode(errors="replace")
+        if f[13] & (FH_CAM_DARK | FH_CAM_COLOR) or name.endswith("_dk") or name.startswith("color"):
+            continue
+        mono.append({"name": name, "sensor": sensor, "node": f[2]})
+    return {"alive": alive, "writer_pid": writer, "mono": mono}
+
+
+# ------------------------------------------------------------------------------------------
+# The check
+
+def check(log=None, proc=True, ring=True, ring_path=None):
+    """The cameras' state: {"status": "ok"|"degraded"|"unknown", "summary": "ok" or
+    "degraded: ..." or "unknown: ...", "reason", "evidence": [lines], "log", "xrservice", "ring"}."""
+    evidence = []
+    pid = xrservice_pid() if proc else None
+    fds = xrservice_fds(pid) if pid else None
+    path = log or (fds or {}).get("log") or newest_log()
+    state = None
+    if path:
+        try:
+            state = read_log(path)
+        except OSError as e:
+            evidence.append("log %s: %s" % (path, e))
+    if state:
+        status, reason, ev = state.verdict()
+        evidence += ["log %s:" % path] + ["  " + e for e in ev]
+    else:
+        status, reason = "unknown", "no XRService log in %s" % LOG_DIR
+    out = {"log": path, "xrservice": None, "ring": None,
+           "episode": state.snapshot()["episode"] if state else {},
+           "failure": state.snapshot()["failure"] if state else ""}
+
+    if proc:
+        if pid is None:
+            # The log can't tell a killed XRService from a running one; no process settles it.
+            evidence.append("XRService isn't running")
+            status, reason = "unknown", "SteamVR isn't running (no XRService)"
+        elif fds is None:
+            evidence.append("XRService pid %d: its open files can't be read here (in the dev container?)" % pid)
+            out["xrservice"] = {"pid": pid, "videos": None}
+        else:
+            videos = fds["videos"]
+            want = state.tracking_nodes() if state else SIDE_NODES + UPPER_NODES
+            upper = state.upper_nodes() if state else UPPER_NODES
+            have = [n for n in want if n in videos]
+            evidence.append("XRService pid %d has open: %s (tracking cameras: %s; upper: %s)" % (
+                pid, " ".join("video%d" % n for n in videos) or "no cameras",
+                " ".join("video%d" % n for n in want), " ".join("video%d" % n for n in upper)))
+            out["xrservice"] = {"pid": pid, "videos": videos, "tracking_open": len(have)}
+            closed = state is not None and state.closed
+            if not closed and len(have) == TRACKING and status == "unknown":
+                status, reason = "ok", "XRService has all %d tracking cameras open" % TRACKING
+            elif not closed and videos and not all(n in videos for n in upper) and status != "degraded":
+                status, reason = "degraded", ("XRService has %d of %d tracking cameras open (the upper pair "
+                                              "is missing)" % (len(have), TRACKING))
+
+    if ring:
+        r = read_ring(ring_path or default_ring())
+        out["ring"] = r
+        if r is None:
+            evidence.append("ft-camd: no camera ring (not running)")
+        else:
+            names = " ".join(c["name"] for c in r["mono"]) or "none"
+            evidence.append("ft-camd (pid %d, %s): %d mono cameras: %s" % (
+                r["writer_pid"], "running" if r["alive"] else "stale ring", len(r["mono"]), names))
+            if r["alive"] and len(r["mono"]) < TRACKING and status == "ok":
+                status, reason = "degraded", ("ft-camd publishes only %d of %d mono cameras (it started while "
+                                              "they were missing: restart it)" % (len(r["mono"]), TRACKING))
+    out.update(status=status, reason=reason, evidence=evidence,
+               summary="ok" if status == "ok" else "%s: %s" % (status, reason))
+    return out
+
+
+def is_vcint_failure(result):
+    return bool(result) and result.get("status") == "degraded" and result.get("reason") == VCINT_REASON
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(description="Are the headset's four mono tracking cameras running?")
+    ap.add_argument("--json", action="store_true", help="print the result as JSON")
+    ap.add_argument("--log", help="read this XRService log (default: the running instance's)")
+    ap.add_argument("--no-proc", action="store_true", help="don't look at XRService's process")
+    ap.add_argument("--no-ring", action="store_true", help="don't look at ft-camd's ring")
+    ap.add_argument("--ring", help="ft-camd's ring (default /run/user/UID/frametop-hands/cam-ring)")
+    a = ap.parse_args(argv)
+    r = check(log=a.log, proc=not a.no_proc, ring=not a.no_ring, ring_path=a.ring)
+    if a.json:
+        print(json.dumps(r, indent=1))
+    else:
+        print(r["summary"])
+        for line in r["evidence"]:
+            print("  " + line)
+    return {"ok": 0, "degraded": 1}.get(r["status"], 2)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

hands/frametop-camwatch.service

@@ -0,0 +1,21 @@
+# Template: the installer replaces @REPO@ with the repo path on the Frame. Not installed or
+# enabled by anything yet (hands/README.md, "Camera check").
+[Unit]
+Description=Frametop camera watch: tells you when SteamVR leaves the headset's upper cameras off
+Documentation=file://@REPO@/hands/README.md
+# Not PartOf=steamvr.service: it has to outlive the SteamVR restart it may ask for.
+
+[Service]
+# On the host, system Python, standard library only. It stats the XRService log every 2 s and
+# reads only what's new. Notifications go to the Frametop desktop's own D-Bus (found through its
+# plasmashell). With CAMWATCH_AUTO_RESTART=1 in ~/.config/frametop.conf it may restart SteamVR
+# while the headset isn't worn: that also closes the Frametop desktop.
+ExecStart=/usr/bin/python3 @REPO@/hands/ft-camwatch
+Restart=on-failure
+RestartSec=30
+Nice=10
+CPUQuota=5%
+MemoryMax=64M
+
+[Install]
+WantedBy=default.target

hands/ft-camwatch

@@ -0,0 +1,378 @@
+#!/usr/bin/env python3
+"""ft-camwatch: watches SteamVR's XRService log for the camera failure that turns off the
+headset's upper cameras and IR light (a VCINT FPGA load that fails, usually after a wake;
+hands/camcheck.py), tells the Frametop desktop, and can restart SteamVR by itself.
+
+Off by default: hands/frametop-camwatch.service is a template the installer doesn't enable.
+Settings in ~/.config/frametop.conf:
+  CAMWATCH_AUTO_RESTART=1    restart SteamVR on a failure (default 0: only a notification). Only
+                             while the headset isn't worn (frame-job's check: vrcompositor runs and
+                             a backlight is on), after it has been off for CAMWATCH_IDLE_S, with no
+                             Steam-launched app running and nobody on the remote desktop (VNC).
+                             At most once per failure, and not again within CAMWATCH_COOLDOWN_MIN.
+  CAMWATCH_IDLE_S=60         how long the headset must be off first
+  CAMWATCH_COOLDOWN_MIN=30   the least time between two automatic restarts
+  CAMWATCH_IGNORE_APPIDS=    Steam app ids that don't count as a running VR app (comma-separated)
+  CAMWATCH_NOTIFY=1          post a notification in the Frametop desktop (0: log only)
+
+A SteamVR restart also closes the Frametop desktop and every window in it (ft-screens quits
+with SteamVR, and the desktop's unit doesn't restart: see hands/README.md, "Camera check").
+
+It follows the log with a stat every 2 s (no inotify, nothing else while all is well), and
+logs every decision to stdout (the journal). --dry-run never notifies or restarts; --once
+prints the state and what it would do now, and exits.
+"""
+import argparse
+import json
+import os
+import subprocess
+import sys
+import time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+import camcheck  # noqa: E402
+
+CONF = os.path.expanduser("~/.config/frametop.conf")
+STATE = os.path.expanduser("~/.local/state/frametop/camwatch.json")
+BACKLIGHTS = "/sys/class/backlight"
+POLL_S = 2.0
+MAX_READ = 16 << 20     # a poll reads at most this much of a log that grew
+NOTIFY_TITLE = "Hand tracking: cameras off"
+NOTIFY_TEXT = ("The headset's upper cameras and IR light are off: SteamVR couldn't start the colour camera "
+               "module (it happens sometimes after the headset sleeps). Restart SteamVR, or the headset if "
+               "that doesn't fix it. Restarting SteamVR closes this desktop and its windows.")
+DEFAULTS = {"CAMWATCH_AUTO_RESTART": "0", "CAMWATCH_IDLE_S": "60", "CAMWATCH_COOLDOWN_MIN": "30",
+            "CAMWATCH_IGNORE_APPIDS": "", "CAMWATCH_NOTIFY": "1", "VNC_PORT": "5900"}
+
+
+def log(text):
+    print("%s %s" % (time.strftime("%H:%M:%S"), text), flush=True)
+
+
+def read_conf(path=CONF):
+    """KEY=VALUE lines of a shell-style file (comments and quotes stripped), over DEFAULTS."""
+    conf = dict(DEFAULTS)
+    try:
+        with open(path) as f:
+            for line in f:
+                line = line.split("#", 1)[0].strip()
+                if "=" not in line:
+                    continue
+                k, v = line.split("=", 1)
+                k, v = k.strip(), v.strip().strip("'\"")
+                if k.replace("_", "").isalnum():
+                    conf[k] = v
+    except OSError:
+        pass
+    return conf
+
+
+def conf_int(conf, key):
+    try:
+        return int(float(conf.get(key, DEFAULTS.get(key, "0"))))
+    except ValueError:
+        return int(DEFAULTS.get(key, "0") or 0)
+
+
+# ------------------------------------------------------------------------------------------
+# What's going on around (all read-only, from /proc and /sys)
+
+def process_running(name):
+    for pid in os.listdir("/proc"):
+        if pid.isdigit():
+            try:
+                with open("/proc/%s/comm" % pid) as f:
+                    if f.read().strip() == name:
+                        return True
+            except OSError:
+                pass
+    return False
+
+
+def headset_worn(backlights=BACKLIGHTS):
+    """frame-job's check: vrcompositor runs and any panel's backlight is on (SteamVR turns the
+    panels off 5 s after the headset comes off)."""
+    lit = False
+    try:
+        for name in os.listdir(backlights):
+            try:
+                with open(os.path.join(backlights, name, "brightness")) as f:
+                    lit = lit or int(f.read().strip()) > 0
+            except (OSError, ValueError):
+                pass
+    except OSError:
+        return False
+    return lit and process_running("vrcompositor")
+
+
+def vr_apps(ignore=()):
+    """Apps Steam launched ("SteamLaunch AppId=N" in a process's arguments, as Steam's reaper
+    runs them): ["AppId=N ..."]. Steam's own processes, SteamVR's and Frametop's services aren't
+    launched this way. A guess: no VR game has been run on the Frame to confirm the form."""
+    out = set()
+    for pid in os.listdir("/proc"):
+        if not pid.isdigit():
+            continue
+        argv = camcheck.proc_argv(pid)
+        if "SteamLaunch" not in argv:
+            continue
+        ids = [a.split("=", 1)[1] for a in argv if a.startswith("AppId=")]
+        if ids and ids[0] not in ignore:
+            out.add("AppId=" + ids[0])
+    return sorted(out)
+
+
+def remote_viewers(port=5900, tables=("/proc/net/tcp", "/proc/net/tcp6")):
+    """Established connections to the remote desktop's VNC port, from /proc/net/tcp(6)."""
+    out = []
+    for path in tables:
+        try:
+            with open(path) as f:
+                next(f)
+                for line in f:
+                    p = line.split()
+                    if len(p) > 3 and p[3] == "01" and int(p[1].rsplit(":", 1)[1], 16) == port:
+                        out.append(p[2])
+        except (OSError, StopIteration, ValueError):
+            pass
+    return out
+
+
+def frametop_bus():
+    """The Frametop desktop's D-Bus (from its plasmashell, as decoration/apply.sh finds it), or None."""
+    for pid in os.listdir("/proc"):
+        if not pid.isdigit():
+            continue
+        try:
+            with open("/proc/%s/comm" % pid) as f:
+                if f.read().strip() != "plasmashell":
+                    continue
+            with open("/proc/%s/environ" % pid, "rb") as f:
+                env = dict(kv.split(b"=", 1) for kv in f.read().split(b"\0") if b"=" in kv)
+        except OSError:
+            continue
+        if env.get(b"XDG_RUNTIME_DIR", b"").endswith(b"/frametop") and b"DBUS_SESSION_BUS_ADDRESS" in env:
+            return env[b"DBUS_SESSION_BUS_ADDRESS"].decode()
+    return None
+
+
+def notify(title, text):
+    bus = frametop_bus()
+    if not bus:
+        log("no Frametop desktop running: no notification")
+        return False
+    r = subprocess.run(["notify-send", "-a", "Frametop", "-u", "critical", "-i", "dialog-warning", title, text],
+                       env=dict(os.environ, DBUS_SESSION_BUS_ADDRESS=bus), capture_output=True, text=True,
+                       timeout=10)
+    if r.returncode:
+        log("notify-send failed (%d): %s" % (r.returncode, r.stderr.strip()))
+    return r.returncode == 0
+
+
+def restart_steamvr():
+    # --no-block: the job runs in systemd; the Frametop desktop closing doesn't cut it short.
+    r = subprocess.run(["systemctl", "--user", "restart", "--no-block", "steamvr.service"],
+                       capture_output=True, text=True, timeout=30)
+    log("systemctl --user restart steamvr.service: exit %d %s" % (r.returncode, (r.stderr or "").strip()))
+    return r.returncode == 0
+
+
+def environment(conf):
+    ignore = tuple(a.strip() for a in conf.get("CAMWATCH_IGNORE_APPIDS", "").replace(",", " ").split() if a.strip())
+    return {"steamvr": camcheck.xrservice_pid() is not None, "worn": headset_worn(),
+            "vr_apps": vr_apps(ignore), "remote": remote_viewers(conf_int(conf, "VNC_PORT"))}
+
+
+# ------------------------------------------------------------------------------------------
+# The decision (pure: tests feed it made-up states)
+
+def new_memory():
+    return {"pending": "", "notified": [], "restarted": {}, "last_restart": 0.0, "idle_since": None, "why": ""}
+
+
+def decide(now, failure, env, mem, conf):
+    """What to do now. failure: the current VCINT failure's id ("" if none: the cameras run,
+    or they're closed); env: {"steamvr", "worn", "vr_apps", "remote"} (only looked at while a
+    failure is current); mem: new_memory(), updated in place; now: wall-clock seconds.
+    Returns [("log", text) | ("notify", failure) | ("restart", failure)]."""
+    acts = []
+    if not failure:
+        if mem["pending"]:
+            acts.append(("log", "failure %s is no longer current" % mem["pending"]))
+        mem.update(pending="", idle_since=None, why="")
+        return acts
+    if mem["pending"] != failure:
+        mem.update(pending=failure, why="")
+        acts.append(("log", "VCINT failure: %s (upper cameras and IR light off)" % failure))
+    if failure not in mem["notified"]:
+        mem["notified"] = (mem["notified"] + [failure])[-20:]
+        if conf.get("CAMWATCH_NOTIFY", "1") != "0":
+            acts.append(("notify", failure))
+    if env.get("worn"):
+        mem["idle_since"] = None
+    elif mem["idle_since"] is None:
+        mem["idle_since"] = now
+    idle_s = conf_int(conf, "CAMWATCH_IDLE_S")
+    cooldown = conf_int(conf, "CAMWATCH_COOLDOWN_MIN") * 60
+    if conf.get("CAMWATCH_AUTO_RESTART", "0") != "1":
+        why = "no automatic restart (CAMWATCH_AUTO_RESTART=1 in ~/.config/frametop.conf turns it on)"
+    elif failure in mem["restarted"]:
+        why = "SteamVR was restarted once for this failure already: restart the headset"
+    elif mem["last_restart"] and now - mem["last_restart"] < cooldown:
+        why = "no restart: the last automatic one was %d min ago (cooldown %d min)" % (
+            (now - mem["last_restart"]) // 60, cooldown // 60)
+    elif not env.get("steamvr"):
+        why = "no restart: SteamVR isn't running"
+    elif env.get("worn"):
+        why = "no restart while the headset is worn"
+    elif env.get("vr_apps"):
+        why = "no restart: a VR app is running (%s)" % ", ".join(env["vr_apps"])
+    elif env.get("remote"):
+        why = "no restart: someone is on the remote desktop (%s)" % ", ".join(env["remote"])
+    elif now - mem["idle_since"] < idle_s:
+        why = "waiting for the headset to stay off for %d s" % idle_s
+    else:
+        mem["restarted"][failure] = now
+        mem["last_restart"] = now
+        why = "restarting SteamVR (the headset is off, nothing else in VR)"
+        acts.append(("log", why))
+        acts.append(("restart", failure))
+        mem["why"] = why
+        return acts
+    if why != mem["why"]:
+        mem["why"] = why
+        acts.append(("log", why))
+    return acts
+
+
+def load_memory(path=STATE):
+    mem = new_memory()
+    try:
+        with open(path) as f:
+            saved = json.load(f)
+        mem["notified"] = list(saved.get("notified", []))[-20:]
+        mem["restarted"] = dict(saved.get("restarted", {}))
+        mem["last_restart"] = float(saved.get("last_restart", 0.0))
+    except (OSError, ValueError, TypeError, AttributeError):
+        pass
+    return mem
+
+
+def save_memory(mem, path=STATE):
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    tmp = path + ".tmp"
+    with open(tmp, "w") as f:
+        json.dump({"notified": mem["notified"], "restarted": mem["restarted"],
+                   "last_restart": mem["last_restart"]}, f)
+    os.replace(tmp, path)
+
+
+# ------------------------------------------------------------------------------------------
+# Following the log
+
+class Follower:
+    """The running XRService's log, read as it grows. A new log (SteamVR restarted) starts afresh."""
+
+    def __init__(self, path=None):
+        self.fixed = path
+        self.path, self.ino, self.offset, self.rest = "", None, 0, ""
+        self.state = None
+
+    def poll(self):
+        path = self.fixed or camcheck.newest_log()
+        if not path:
+            return None
+        try:
+            st = os.stat(path)
+        except OSError:
+            return self.state
+        if path != self.path or st.st_ino != self.ino or st.st_size < self.offset:
+            self.path, self.ino, self.offset, self.rest = path, st.st_ino, 0, ""
+            self.state = camcheck.LogState(path)
+            log("following %s" % path)
+        if st.st_size > self.offset:
+            with open(path, "rb") as f:
+                f.seek(self.offset)
+                data = f.read(MAX_READ)
+            self.offset += len(data)
+            text = self.rest + data.decode(errors="replace")
+            lines = text.split("\n")
+            self.rest = lines.pop()
+            for line in lines:
+                self.state.feed(line)
+        return self.state
+
+
+def current_failure(state, steamvr_running=True):
+    """The failure id if the log's current camera start is a VCINT failure, else ""."""
+    if state is None or not steamvr_running:
+        return ""
+    snap = state.snapshot()
+    return snap["failure"] if snap["status"] == "degraded" and snap["reason"] == camcheck.VCINT_REASON else ""
+
+
+def act(acts, dry):
+    changed = False
+    for kind, arg in acts:
+        if kind == "log":
+            log(arg)
+        elif kind == "notify":
+            changed = True
+            if dry:
+                log("(dry run) would notify: %s" % NOTIFY_TITLE)
+            else:
+                notify(NOTIFY_TITLE, NOTIFY_TEXT)
+        elif kind == "restart":
+            changed = True
+            if dry:
+                log("(dry run) would restart SteamVR")
+            else:
+                restart_steamvr()
+    return changed
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(description="Watch for the camera failure that turns off the upper cameras.")
+    ap.add_argument("--dry-run", action="store_true", help="log what it would do; never notify or restart")
+    ap.add_argument("--once", action="store_true", help="look once, print the decision (a dry run), and exit")
+    ap.add_argument("--log", help="follow this file instead of the running XRService's log (tests)")
+    ap.add_argument("--state", default=STATE, help="where it remembers past failures (default %(default)s)")
+    a = ap.parse_args(argv)
+    dry = a.dry_run or a.once
+    mem = load_memory(a.state)
+    follower = Follower(a.log)
+    conf = read_conf()
+    conf_mtime = 0.0
+    log("watching; automatic restart %s" % ("on" if conf.get("CAMWATCH_AUTO_RESTART") == "1" else "off"))
+    while True:
+        try:
+            m = os.stat(CONF).st_mtime
+        except OSError:
+            m = 0.0
+        if m != conf_mtime:
+            conf, conf_mtime = read_conf(), m
+        state = follower.poll()
+        failure = current_failure(state)
+        env = {}
+        if failure or mem["pending"]:
+            env = environment(conf)
+            if not env["steamvr"] and not a.log:
+                failure = ""   # the log's last word, but XRService is gone
+        acts = decide(time.time(), failure, env, mem, conf)
+        if a.once:
+            snap = state.snapshot() if state else {"status": "unknown", "reason": "no log"}
+            print("log: %s\nstate: %s %s\nenvironment: %s" % (follower.path, snap["status"], snap["reason"],
+                                                              json.dumps(env)))
+            act(acts, True)
+            return 0
+        if act(acts, dry) and not dry:
+            save_memory(mem, a.state)
+        time.sleep(POLL_S)
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main())
+    except KeyboardInterrupt:
+        sys.exit(0)

hands/rec/DESIGN.md

@@ -18,6 +18,7 @@ The plan this belongs to is `~/Desktop/Projects/frame-hands/notes/hands-plan.md`
 | `hands/rec/build.sh` | Builds `ft-handpanel` into `hands/rec/build/`, like `gaze/build.sh`. |
 | `hands/rec/CONSENT.md`, `hands/rec/UPLOAD.md` | The texts the window shows. |
 | ft-hands `--record-hz N` (done) | Records at most N frame sets a second. The recorder uses 10. |
+| `hands/camcheck.py` (shared, standard library) | Are all four mono cameras running? The recorder runs it before a session and when a step sees no hands (below; `hands/README.md`, "Camera check"). |
 
 Every part runs in the dev container, as ft-hands and Input Settings do. The host Python isn't used: it lacks PySide6 and zstd. `setup/dev-container.sh` gains `zstd`.
 
@@ -37,6 +38,7 @@ Test hooks:
 - `--next-after S` presses Next by itself after S seconds of waiting (real time), so a step-mode session runs unattended. Lines on stdin steer it too: `n` or an empty line is Next, `p` pause or resume, `r` redo, `s` skip the section, `q` stop.
 - `--no-headset-button` leaves the headset's button alone (`ft-handrec --no-headset-button` too). `--button-device PATH` reads it from PATH instead, an event device or a FIFO of `input_event` structs, also in a dry run: a simulated button for tests.
 - `--auto` runs the timed flow; `--poses DIR` takes the pose pictures from DIR; `--plan` prints the sections, their steps and length.
+- `--ignore-cameras` (`ft-handrec --ignore-cameras` too) starts even when the camera check fails. A dry run and `--ring` skip the check by themselves.
 
 ## Files
 
@@ -76,9 +78,12 @@ No name, email, or account. The contributor id is random, so several sessions fr
  "checklist": {"objects": ["pencil", "phone", "cup", "keyboard", "mouse", "gamepad", "small"], "own_objects": ["..."],
                "controllers": "straps|none", "sleeves": "short|long|", "rings": false, "watch": false, "notes": ""},
  "device": {"steamos": "<VERSION_ID from /etc/os-release>", "steamvr": "<version if known>", "cameras": [{"name", "width", "height"}]},
- "mode": "step|auto", "takes": ["01-hand-size", "..."]}
+ "mode": "step|auto", "takes": ["01-hand-size", "..."],
+ "camera": {"status": "ok|unknown|degraded", "reason": "..."}, "stop_reason": "..."}
 ```
 
+`camera` is the camera check's verdict at the start (no paths or log lines: those go to `session.log`). `stop_reason` is there when a session was stopped at the no-hands screen, with the check's result.
+
 A session started before step mode existed has no `mode`: it ran as `auto`.
 
 ### calibration.json
@@ -112,6 +117,7 @@ One JSON object per line:
 {"t": 123, "event": "ready", "id": "static-poses/fist/left/near", "seconds": 3}
 {"t": 123, "event": "wait"}
 {"t": 123, "event": "redo", "id": "static-poses/fist/left/near", "from": 123, "to": 123}
+{"t": 123, "event": "nohands", "id": "hand-size/flat/both", "reads": 120, "published": 118}
 {"t": 123, "event": "end", "status": "complete|stopped|skipped"}
 ```
 
@@ -126,7 +132,7 @@ prompt     the hold: its labels start here
 wait       the hold is over: no labels from here; part N stops
 ```
 
-The touch-the-dot targets after the first follow straight on: no `ready` or `wait` between them. `redo` marks a try done again (R): `from` is that step's `ready` (or its `prompt` if it had none), `to` its end. Its prompt is skipped; the sets stay. In auto mode there's no `ready` or `wait`, and the intro is a recorded prompt `<section>/intro`. Readers that knew only `prompt` and `end` keep working, but they'd give the countdown to the step before: `hub_review.py` (frame-hands `train/hub`) shows it as "(countdown)" and redone prompts as "(redone)"; `FORMAT.md` in the dataset repo has the rules.
+The touch-the-dot targets after the first follow straight on: no `ready` or `wait` between them. `redo` marks a try done again (R): `from` is that step's `ready` (or its `prompt` if it had none), `to` its end. Its prompt is skipped; the sets stay. In auto mode there's no `ready` or `wait`, and the intro is a recorded prompt `<section>/intro`. `nohands` marks the first hand-size step stopped because no hand was seen (below, "Camera check"); a `redo` over the same range follows it, so the try gets no labels. Readers that knew only `prompt` and `end` keep working, but they'd give the countdown to the step before: `hub_review.py` (frame-hands `train/hub`) shows it as "(countdown)" and redone prompts as "(redone)"; `FORMAT.md` in the dataset repo has the rules.
 
 ### poses.jsonl
 
@@ -245,6 +251,14 @@ Before section 8: "Put on both controllers and tighten the straps". Before secti
 - **Controller tracking:** in sections 8 and 9, `devices` is polled once a second. A result other than 200 for more than 1 s says "The left controller lost tracking: turn your palm slightly toward you". Each such stretch goes into `prompts.jsonl` as `feedback` with `"controller": {...}`.
 - **Lighting check at session start:** the mean of every mono camera's `mean` and `dark_mean` from the ring (`hands/tools/ring.py` layout; struct only, no numpy). It's compared with the person's earlier sessions. If the chosen lighting matches an earlier round's within 15%, the window says so before starting.
 
+### Camera check
+
+On 2026-10-02 a whole session showed "I can't see your hands": after the headset slept, SteamVR had failed to load the colour module's FPGA image, which left the upper cameras and the IR light off (`hands/README.md`, "Camera check"). Two checks keep that from wasting a session:
+
+- **Before the session.** The window runs `hands/camcheck.py` when the checklist page opens ("Tracking cameras:", with the evidence under Details and Check again), and again when Start is pressed; `session.py` runs it first thing, before it makes the session's folder or starts anything (`Session._preflight`). If it finds the cameras degraded, the session doesn't start. The window says: "The headset's upper cameras and IR light are off. SteamVR couldn't start the colour camera module (it happens sometimes after the headset sleeps). Restart SteamVR, or restart the headset if that doesn't fix it." (another `degraded:` reason gets a sentence naming it), and Start stays off. `unknown` (SteamVR not running, the cameras asleep) doesn't stop it: the session's own start fails clearly then. From the command line, `session.py` prints the check and exits with status 3.
+- **Restart SteamVR.** The message has a Restart SteamVR button. After a confirmation it runs `systemctl --user restart --no-block steamvr.service` on the host (through `distrobox-host-exec`), then checks the cameras every 3 s, for up to 2 minutes, until a new XRService has opened its cameras. The confirmation says what really happens: every VR app closes, and so does the Frametop desktop with all its windows, this one included, and it doesn't come back by itself (`hands/README.md`, "What a SteamVR restart does to Frametop"). So the check after the restart mostly happens when the recorder is opened again; it re-runs when the checklist page opens. `--no-block` lets the restart finish after the window is gone.
+- **No hands in the first step.** The first step of the hand-size section has both hands up, about 40 cm away. During its hold, the session counts the hands file's reads (about 20 a second). If the tracker published in at least half of at least 10 reads and never saw a hand, the step stops: a `wait` and the recording stops as usual, then a `nohands` event and a `redo` over the try. The session runs the camera check and shows "I can't see your hands" with its result (the camera text above when it's the VCINT failure, else "The camera check found nothing wrong"). The state is `nohands`, waiting: Next (the headset button, Space, Try again) or R starts the same step's countdown at once; Stop (Esc) ends the session with `stop_reason` set; S skips the section. One missed step costs a retry, not the session, and every hold of that step is logged ("hands check ...: a hand in N of M reads"). Without a tracker publishing the session can't tell, logs that, and goes on. Only that one step is checked: later steps have their notes ("I can't see your left hand") as before. Auto mode does the same; its recording pauses meanwhile.
+
 ### Controls
 
 - The window has Start, a big Next (while a step waits; its hint is the panel's), Pause/Resume, Redo step, Skip section and Stop. While it has focus: Space is Next, P pauses or resumes, R redoes, S skips the section, Esc stops. The panel's bottom line and the window list them. The window also shows the step's picture, diagram, countdown and "Hold".

hands/rec/ft_handrec.py

@@ -5,7 +5,9 @@ A Kirigami (QML) app with a Python backend. It runs in the dev container:
   - Welcome: the consent text (CONSENT.md), shown the first time and again when its version
     changes. Agreeing writes profile.json with a random contributor id.
   - Before you start: the checklist (objects, controller straps, lighting, sleeves, privacy,
-    free space) and what will happen. Start hands it to the session runner (session.py).
+    free space), the camera check (hands/camcheck.py: Start stays off while the upper cameras
+    are off, with a Restart SteamVR button; --ignore-cameras overrides it) and what will happen.
+    Start hands it to the session runner (session.py).
   - Session: the runner's live status with the step's pose picture and where-to diagram, Next,
     Pause/Resume, Redo, Skip section and Stop (Space: Next, P, R, S and Esc while the window has
     focus; the button on the headset's right side is Next, pause and resume). The prompts appear
@@ -33,6 +35,7 @@ import signal
 import subprocess
 import sys
 import threading
+import time
 import uuid
 
 from PySide6.QtCore import Property, QObject, Qt, QTimer, QUrl, Signal, Slot
@@ -68,7 +71,10 @@ LIGHTING = [("dim", "Dim: one lamp only"), ("room", "Normal room light"), ("dayl
 SLEEVES = [("short", "Short sleeves or bare arms"), ("long", "Long sleeves"), ("", "Rather not say")]
 HANDEDNESS = [("", "Rather not say"), ("right", "Right-handed"), ("left", "Left-handed"),
               ("both", "Both (ambidextrous)")]
-ACTIVE_STATES = ("starting", "intro", "ready", "countdown", "running", "paused", "between")
+ACTIVE_STATES = ("starting", "intro", "ready", "countdown", "running", "paused", "between", "nohands")
+# After Restart SteamVR: look at the cameras again this often, for at most this long.
+RECHECK_S = 3
+RECHECK_FOR_S = 120
 # Shown side by side in the viewer at this height; thumbnails are smaller.
 SET_HEIGHT = 480
 THUMB_HEIGHT = 96
@@ -208,8 +214,10 @@ class Backend(QObject):
     _loginArrived = Signal(dict)
     _uploadLine = Signal(dict)
     _uploadFinished = Signal(dict)
+    cameraChanged = Signal()
+    _cameraArrived = Signal(dict)
 
-    def __init__(self, store, session_options=None, hub_dry_run=False):
+    def __init__(self, store, session_options=None, hub_dry_run=False, ignore_cameras=False):
         super().__init__()
         self.store = store
         self._session_options = session_options or {}  # Session options: dry_run, speed, poses_dir (tests), button
@@ -233,6 +241,11 @@ class Backend(QObject):
         self._export_text = ""
         self._export_session = ""
         self._statusArrived.connect(self._on_status)
+        self._ignore_cameras = ignore_cameras
+        self._camera = {}            # the last camera check (session.camera_check)
+        self._camera_busy = False
+        self._restarting = False
+        self._cameraArrived.connect(self._on_camera)
         self._lightingArrived.connect(self._on_lighting)
         self._exportProgress.connect(self._on_export_progress)
         self._exportFinished.connect(self._on_export_finished)
@@ -373,6 +386,99 @@ class Backend(QObject):
         self._lighting_note = note
         self.lightingChanged.emit()
 
+    # --- the camera check (hands/camcheck.py through session.camera_check)
+    @Property(str, notify=cameraChanged)
+    def cameraState(self):
+        """ok, degraded, unknown; checking while it runs; "" before the first check."""
+        if self._camera_busy and not self._camera:
+            return "checking"
+        return self._camera.get("status", "")
+
+    @Property(str, notify=cameraChanged)
+    def cameraText(self):
+        """What to tell the person when the cameras aren't all running ("" otherwise)."""
+        mod = self._runner()
+        return mod.camera_text(self._camera) if mod and self._camera else ""
+
+    @Property(str, notify=cameraChanged)
+    def cameraSummary(self):
+        return self._camera.get("summary", "")
+
+    @Property(str, notify=cameraChanged)
+    def cameraEvidence(self):
+        return "\n".join(self._camera.get("evidence", []))
+
+    @Property(bool, constant=True)
+    def camerasIgnored(self):
+        return self._ignore_cameras
+
+    @Property(bool, notify=cameraChanged)
+    def cameraBusy(self):
+        return self._camera_busy
+
+    @Property(bool, notify=cameraChanged)
+    def restartingSteamVR(self):
+        return self._restarting
+
+    @Property(bool, notify=cameraChanged)
+    def camerasBlockStart(self):
+        """Start stays off: the check found the cameras degraded (unless --ignore-cameras)."""
+        return not self._ignore_cameras and self._camera.get("status") == "degraded"
+
+    def _check_now(self):
+        mod = self._runner()
+        if not mod or self._session_options.get("dry_run"):
+            return {"status": "unknown", "summary": "not checked (dry run)", "reason": "dry run", "evidence": []}
+        return mod.camera_check()
+
+    @Slot()
+    def checkCameras(self):
+        """Run the camera check off this thread (it reads the XRService log: a few MB)."""
+        if self._camera_busy:
+            return
+        self._camera_busy = True
+        self.cameraChanged.emit()
+        self._thread(lambda: self._cameraArrived.emit(self._check_now()))
+
+    def _on_camera(self, result):
+        self._camera = result
+        self._camera_busy = self._restarting
+        self.cameraChanged.emit()
+
+    @Slot()
+    def restartSteamVR(self):
+        """systemctl --user restart steamvr.service on the host (--no-block: the job runs in
+        systemd, so it finishes even when this window closes with the Frametop desktop), then
+        the camera check every few seconds until the new XRService has opened its cameras."""
+        mod = self._runner()
+        if not mod or self._restarting or self.sessionActive:
+            return
+        self._restarting = self._camera_busy = True
+        self.cameraChanged.emit()
+        before = self._camera.get("log", "") if self._camera else ""   # each XRService start writes a new log
+
+        def run():
+            result = None
+            try:
+                r = subprocess.run(mod.host_command("systemctl", "--user", "restart", "--no-block",
+                                                    "steamvr.service"), capture_output=True, text=True, timeout=60)
+                if r.returncode != 0:
+                    raise RuntimeError((r.stderr or r.stdout).strip() or "exit %d" % r.returncode)
+                end = time.monotonic() + RECHECK_FOR_S
+                time.sleep(RECHECK_S * 2)
+                while time.monotonic() < end:
+                    result = self._check_now()
+                    # Done once a new XRService (a new log) has opened its cameras, ok or not.
+                    if result.get("status") in ("ok", "degraded") and result.get("log") != before:
+                        break
+                    time.sleep(RECHECK_S)
+            except Exception as e:
+                result = {"status": "unknown", "summary": f"unknown: couldn't restart SteamVR ({e})",
+                          "reason": str(e), "evidence": []}
+            self._restarting = False
+            self._cameraArrived.emit(result or self._check_now())
+        self._thread(run)
+
     # --- the session
     @Property("QVariantMap", notify=statusChanged)
     def status(self):
@@ -410,6 +516,12 @@ class Backend(QObject):
         if self.needsConsent:
             self.message.emit("Agree to the consent text first (Welcome page)", True)
             return False
+        if not self._ignore_cameras and not self._session_options.get("dry_run"):
+            self._camera = self._check_now()   # fresh: it takes a fraction of a second
+            self.cameraChanged.emit()
+            if self._camera.get("status") == "degraded":
+                self.message.emit(mod.camera_text(self._camera), True)
+                return False
         checklist = dict(checklist)
         checklist["objects"] = [o for o in checklist.get("objects", []) if o in dict(OBJECTS)]
         checklist["own_objects"] = [o.strip() for o in checklist.get("own_objects", []) if str(o).strip()]
@@ -435,6 +547,9 @@ class Backend(QObject):
 
     def _on_status(self, status):
         self._status = status
+        if status.get("camera") and status.get("camera") is not self._camera:
+            self._camera = status["camera"]   # the session's own check (at its start, or after no hands)
+            self.cameraChanged.emit()
         if self._session is not None and not self._session_id:
             self._session_id = os.path.basename(str(getattr(self._session, "session_dir", "") or ""))
         self.statusChanged.emit()
@@ -927,6 +1042,8 @@ def main():
                     help="sessions don't read the headset's button (Next, pause, resume)")
     ap.add_argument("--hub-dry-run", action="store_true",
                     help="test: Upload checks the export and says what it would send, with no network calls")
+    ap.add_argument("--ignore-cameras", action="store_true",
+                    help="start sessions even if the camera check (hands/camcheck.py) finds the upper cameras off")
     a, qt_args = ap.parse_known_args()
     app = QGuiApplication([sys.argv[0]] + qt_args)
     app.setApplicationName("ft-handrec")
@@ -943,7 +1060,9 @@ def main():
         options["poses_dir"] = a.poses
     if a.no_headset_button:
         options["button"] = False
-    backend = Backend(store, options, hub_dry_run=a.hub_dry_run)
+    if a.ignore_cameras:
+        options["check_cameras"] = False
+    backend = Backend(store, options, hub_dry_run=a.hub_dry_run, ignore_cameras=a.ignore_cameras)
     app.aboutToQuit.connect(backend.shutdown)
     engine.rootContext().setContextProperty("backend", backend)
     engine.rootContext().setContextProperty("startPage", a.page)

hands/rec/main.qml

@@ -109,6 +109,41 @@ Kirigami.ApplicationWindow {
         }
     }
 
+    // Restart SteamVR (the camera check's fix). What it does to Frametop, read from the code
+    // (hands/README.md, "Camera check"): ft-screens quits when SteamVR does, which ends the
+    // desktop; its unit doesn't restart, and stopping it ends what was started in it.
+    function askRestartSteamVR() {
+        confirm.ask("Restart SteamVR?",
+                    "SteamVR starts its cameras again, and with them the colour camera module. This closes every "
+                    + "VR app. It also closes the Frametop desktop with all its windows, this one too: Frametop's "
+                    + "screens don't come back by themselves. Save your work first; programs started from the "
+                    + "desktop, such as terminals and what runs in them, stop too.\n\nOnce SteamVR is back, start "
+                    + "the desktop again (Desktop in the library) and open the Hand recorder: it checks the cameras "
+                    + "again. If they're still off, restart the headset.",
+                    () => backend.restartSteamVR())
+    }
+
+    // The camera check's problem, with its fix (a header message on the checklist and session pages).
+    component CameraMessage: Kirigami.InlineMessage {
+        position: Kirigami.InlineMessage.Position.Header
+        type: backend.restartingSteamVR ? Kirigami.MessageType.Information : Kirigami.MessageType.Error
+        text: backend.restartingSteamVR ? "Restarting SteamVR, then checking the cameras again…" : backend.cameraText
+        actions: [
+            Kirigami.Action {
+                text: "Restart SteamVR…"
+                icon.name: "system-reboot"
+                enabled: !backend.restartingSteamVR && !backend.sessionActive
+                onTriggered: root.askRestartSteamVR()
+            },
+            Kirigami.Action {
+                text: "Check again"
+                icon.name: "view-refresh"
+                enabled: !backend.cameraBusy
+                onTriggered: backend.checkCameras()
+            }
+        ]
+    }
+
     // Closing during a session asks first. Quitting closes the window again (Qt 6), so the
     // answer is remembered; the backend stops the session as the app quits.
     property bool quitting: false
@@ -250,7 +285,12 @@ Kirigami.ApplicationWindow {
                     text: "A session is running."
                     actions: [ Kirigami.Action { text: "Go to it"; onTriggered: root.show(sessionPage) } ]
                 }
+                CameraMessage {
+                    Layout.fillWidth: true
+                    visible: !backend.sessionActive && (backend.cameraText !== "" || backend.restartingSteamVR)
+                }
             }
+            Component.onCompleted: backend.checkCameras()
 
             Kirigami.FormLayout {
                 Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Within reach" }
@@ -379,6 +419,49 @@ Kirigami.ApplicationWindow {
                           + "or free up space, first."
                 }
 
+                Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Cameras" }
+                RowLayout {
+                    Kirigami.FormData.label: "Tracking cameras:"
+                    Controls.Label {
+                        Layout.maximumWidth: Kirigami.Units.gridUnit * 20
+                        wrapMode: Text.Wrap
+                        text: backend.camerasIgnored ? "Not checked (--ignore-cameras)"
+                              : backend.cameraState === "ok" ? "All four are running."
+                              : backend.cameraState === "degraded" ? "Not all running: see the message at the top."
+                              : backend.cameraState === "unknown" ? "Couldn't tell (" + backend.cameraSummary.replace(/^unknown: /, "")
+                                                                    + "). The session checks again when it starts."
+                              : "Checking…"
+                    }
+                    Controls.ToolButton {
+                        visible: backend.cameraEvidence !== ""
+                        icon.name: "documentinfo"
+                        text: "Details"
+                        checkable: true
+                        id: cameraDetails
+                        display: Controls.AbstractButton.IconOnly
+                        Controls.ToolTip.text: "What the check looked at"
+                        Controls.ToolTip.visible: hovered
+                    }
+                    Controls.ToolButton {
+                        icon.name: "view-refresh"
+                        text: "Check again"
+                        display: Controls.AbstractButton.IconOnly
+                        enabled: !backend.cameraBusy
+                        onClicked: backend.checkCameras()
+                        Controls.ToolTip.text: text
+                        Controls.ToolTip.visible: hovered
+                    }
+                }
+                Controls.Label {
+                    visible: cameraDetails.checked && backend.cameraEvidence !== ""
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 34
+                    wrapMode: Text.WrapAnywhere
+                    font: Kirigami.Theme.smallFont
+                    opacity: 0.7
+                    textFormat: Text.PlainText
+                    text: backend.cameraSummary + "\n" + backend.cameraEvidence
+                }
+
                 Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "What will happen" }
                 Controls.Label {
                     Layout.maximumWidth: Kirigami.Units.gridUnit * 26
@@ -412,17 +495,18 @@ Kirigami.ApplicationWindow {
                 Controls.Button {
                     text: "Start"
                     icon.name: "media-record"
-                    enabled: checklist.ready
+                    enabled: checklist.ready && !backend.camerasBlockStart && !backend.restartingSteamVR
                     onClicked: {
                         if (backend.startSession(checklist.answers(), lighting.currentValue, autoAdvance.checked))
                             root.show(sessionPage)
                     }
                 }
                 Controls.Label {
-                    visible: !checklist.ready && !backend.sessionActive
+                    visible: (!checklist.ready || backend.camerasBlockStart) && !backend.sessionActive
                     opacity: 0.7
                     text: !checklist.privacyOk ? "Tick the three privacy checks to start."
-                          : lighting.currentIndex < 0 ? "Choose the lighting to start." : ""
+                          : lighting.currentIndex < 0 ? "Choose the lighting to start."
+                          : backend.camerasBlockStart ? "The headset's cameras aren't all running: see the message at the top." : ""
                 }
             }
         }
@@ -441,7 +525,7 @@ Kirigami.ApplicationWindow {
             readonly property var stateText: ({
                 starting: "Starting…", intro: "Get ready", ready: "Get ready", countdown: "Starting",
                 running: "Recording", paused: "Paused", between: "Between sections", done: "Done",
-                stopped: "Stopped", error: "Error"
+                stopped: "Stopped", error: "Error", nohands: "No hands seen"
             })
 
             // The keys while the window has focus (the panel in the headset lists them too).
@@ -594,11 +678,21 @@ Kirigami.ApplicationWindow {
                 }
             }
 
-            header: Kirigami.InlineMessage {
-                visible: backend.runnerError !== "" || sessionView.state === "error"
-                position: Kirigami.InlineMessage.Position.Header
-                type: Kirigami.MessageType.Error
-                text: backend.runnerError || ("The session stopped with an error: " + (sessionView.st.error || "unknown"))
+            header: ColumnLayout {
+                spacing: 0
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    visible: backend.runnerError !== "" || (sessionView.state === "error" && sessionView.st.error !== backend.cameraText)
+                    position: Kirigami.InlineMessage.Position.Header
+                    type: Kirigami.MessageType.Error
+                    text: backend.runnerError || ("The session stopped with an error: " + (sessionView.st.error || "unknown"))
+                }
+                // The camera check stopped the session (at its start, or after a step with no hands).
+                CameraMessage {
+                    Layout.fillWidth: true
+                    visible: (!backend.sessionActive && sessionView.state !== "" && backend.cameraText !== "")
+                             || backend.restartingSteamVR
+                }
             }
 
             Kirigami.PlaceholderMessage {
@@ -702,8 +796,8 @@ Kirigami.ApplicationWindow {
                         Controls.Button {
                             visible: sessionView.waiting && sessionView.state !== "paused"
                             focusPolicy: Qt.NoFocus
-                            text: "Next"
-                            icon.name: "go-next"
+                            text: sessionView.state === "nohands" ? "Try again" : "Next"
+                            icon.name: sessionView.state === "nohands" ? "edit-undo" : "go-next"
                             font.pointSize: Kirigami.Theme.defaultFont.pointSize * 1.5
                             Layout.preferredWidth: Kirigami.Units.gridUnit * 10
                             Layout.preferredHeight: Kirigami.Units.gridUnit * 3
@@ -712,8 +806,11 @@ Kirigami.ApplicationWindow {
                         Controls.Label {
                             visible: sessionView.waiting && sessionView.state !== "paused"
                             opacity: 0.7
-                            text: (sessionView.st.ready_text || "Ready? Press Space or click Next")
-                                  + ". A 3-2-1 countdown starts the recording."
+                            text: sessionView.state === "nohands"
+                                  ? "Try again starts the same step (Space or the headset button do too). Stop ends the session, "
+                                    + "keeping what's recorded."
+                                  : (sessionView.st.ready_text || "Ready? Press Space or click Next")
+                                    + ". A 3-2-1 countdown starts the recording."
                             wrapMode: Text.Wrap
                             Layout.fillWidth: true
                         }

hands/rec/session.py

@@ -14,6 +14,10 @@ for testing (in the dev container):
   python3 hands/rec/session.py --dry-run --speed 20 --next-after 0.2   # no processes: prints the panel commands
   python3 hands/rec/session.py --ring /tmp/ring --base /tmp/hr        # ft-ringplay's frames, no headset needed
 
+Before anything starts it runs the camera check (hands/camcheck.py) and won't start while the
+upper cameras are off (--ignore-cameras overrides it). If the hand-size section's first step sees
+no hands at all, it stops that step and asks: try again, or stop (DESIGN.md, "Camera check").
+
 Step mode (the default) shows each step and waits for Next (Space in the window, n or Enter
 here); a 3-2-1 countdown, recorded, then the hold. Only the countdowns and holds are recorded:
 each is a part of the take's recording (sets-N.bin). --auto is the old timed flow.
@@ -41,6 +45,8 @@ import traceback
 HERE = os.path.dirname(os.path.abspath(__file__))
 REPO = os.path.dirname(os.path.dirname(HERE))
 HANDS = os.path.join(REPO, "hands")
+sys.path.insert(0, HANDS)
+import camcheck  # noqa: E402  (hands/camcheck.py: are all four mono cameras running?)
 FT_HANDS = os.path.join(HANDS, "build", "ft-hands")
 FT_CAMD = os.path.join(HANDS, "build", "ft-camd")
 PANEL_BIN = os.path.join(HERE, "build", "ft-handpanel")
@@ -70,6 +76,18 @@ KEYS_AUTO = "Hand recorder window:  P pause  \u00b7  R redo  \u00b7  S skip sect
 KEYS_STEP_BUTTON = ("Headset button: next, pause  \u00b7  Window: Space next  \u00b7  P pause  \u00b7  R redo  "
                     "\u00b7  S skip  \u00b7  Esc stop")
 KEYS_AUTO_BUTTON = "Headset button: pause  \u00b7  Window: P pause  \u00b7  R redo  \u00b7  S skip section  \u00b7  Esc stop"
+# The early no-hands stop (DESIGN.md, "Camera check"): the first step of this section has both
+# hands up. If the live tracker publishes through its hold and never sees a hand, the session
+# stops that step and asks: try again, or stop. Only when the tracker published in at least
+# HANDS_CHECK_PUBLISHED of at least HANDS_CHECK_MIN_READS reads (about 20 a second): without
+# a tracker it can't tell.
+HANDS_CHECK_SECTION = "hand-size"
+HANDS_CHECK_MIN_READS = 10
+HANDS_CHECK_PUBLISHED = 0.5
+NO_HANDS_TITLE = "I can't see your hands"
+NO_HANDS_TEXT = "The hand tracker didn't see either of your hands during that whole step."
+NO_HANDS_RETRY = ("Try again: hold both hands up in front of you, about 40 cm away. To stop instead: "
+                  "Esc or Stop in the Hand recorder window.")
 
 
 def mono_ns():
@@ -921,6 +939,26 @@ def _steamvr_version():
     return ""
 
 
+def camera_check():
+    """camcheck.check() (the XRService log, its open cameras where readable, ft-camd's ring);
+    never raises: a failure is "unknown"."""
+    try:
+        return camcheck.check()
+    except Exception as e:
+        return {"status": "unknown", "summary": "unknown: the camera check failed (%s)" % e,
+                "reason": str(e), "evidence": []}
+
+
+def camera_text(result):
+    """What to tell the person about a camera check that isn't ok ("" if it's ok or unknown)."""
+    if not result or result.get("status") != "degraded":
+        return ""
+    if camcheck.is_vcint_failure(result):
+        return camcheck.USER_TEXT
+    return ("Not all of the headset's tracking cameras are running (%s). Restart SteamVR, or restart the "
+            "headset if that doesn't fix it." % result.get("reason", ""))
+
+
 class Session:
     """One recording session. start() runs it in its own thread; next_step(), pause(), resume(),
     redo(), skip() and stop() steer it from any thread. on_status(dict) is called from the
@@ -931,11 +969,14 @@ class Session:
     advancing by itself, one recording per take. next_after (a test hook): press Next by itself
     after that many seconds of waiting. poses_dir: the pose pictures (poses/). button: read the
     headset's button (not in a dry run unless button_device, a test hook, names the device or
-    a FIFO of input_event structs)."""
+    a FIFO of input_event structs). check_cameras: run the camera check first and don't start
+    if it fails (not in a dry run, nor with ring). hands_reader (a test hook): an object whose
+    read() stands in for the hands file, read in a dry run too."""
 
     def __init__(self, base_dir, profile, checklist, lighting_choice, script_path, *, ring=None,
                  start_processes=True, dry_run=False, speed=1.0, on_status=None, hands_dir=None, panel_bin=None,
-                 auto=False, next_after=None, poses_dir=None, button=True, button_device=None):
+                 auto=False, next_after=None, poses_dir=None, button=True, button_device=None,
+                 check_cameras=True, hands_reader=None):
         self.base_dir = os.path.abspath(os.path.expanduser(base_dir or BASE_DIR))
         self.profile = dict(profile or {})
         self.checklist = dict(checklist or {})
@@ -951,6 +992,9 @@ class Session:
         self.auto = bool(auto)
         self.next_after = next_after
         self.button, self.button_device = bool(button), button_device
+        self.check_cameras = bool(check_cameras)
+        # the camera check (a test hook: tests swap it); a dry run looks at no real cameras
+        self.camera_check_fn = (lambda: None) if dry_run else camera_check
         self.input_devices = INPUT_DEVICES   # where mice are looked for (a test hook)
         self._button = None                  # the ButtonReader
         self.print = print   # where dry-run panel commands go (the CLI's stdout)
@@ -971,7 +1015,8 @@ class Session:
                         "prompt": "", "seconds_left": 0.0, "note": "", "hands": {"left": None, "right": None},
                         "take": None, "error": "", "waiting": False, "countdown": 0, "big": "", "can_redo": False,
                         "image": "", "image_mode": "", "caption": "", "position": "", "distance": "",
-                        "ready_text": READY_TEXT, "button": False, "mouse": True}
+                        "ready_text": READY_TEXT, "button": False, "mouse": True,
+                        "camera": None, "nohands": False}
         self._last_emit = 0.0
         self._log_file = None
         self._panel = None
@@ -982,7 +1027,10 @@ class Session:
         self._recorder = None
         self._prompt = None        # the prompt shown: dict (hands, controllers)
         self._live = None          # the hands file's last read
-        self._hands_file = HandsFile(os.path.join(self.hands_dir, "hands"))
+        self._hands_file = hands_reader or HandsFile(os.path.join(self.hands_dir, "hands"))
+        self._read_hands = not dry_run or hands_reader is not None
+        self._hold_watch = None    # the no-hands check's counts during a hold
+        self._stop_note = ""       # why the session stopped, when it says more than "stopped"
         self._paused = False
         self._recording = False    # a recording part is running
         self._paused_recording = False
@@ -1083,6 +1131,8 @@ class Session:
     # --- the run
     def _run(self):
         try:
+            if self._preflight():
+                return
             self._make_dir()
             self._emit(state="starting")
             self._setup()
@@ -1098,6 +1148,20 @@ class Session:
             self._log(traceback.format_exc())
             self._finish("error", "%s: %s" % (type(e).__name__, e))
 
+    def _preflight(self):
+        """The camera check, before anything is made or started (camcheck.py): with the upper
+        cameras off a session records nothing useful. True if it stopped the session (state
+        "error", no session folder)."""
+        if self.dry_run or self.ring or not self.check_cameras:
+            return False
+        r = self.camera_check_fn()
+        self._status["camera"] = r
+        text = camera_text(r)
+        if not text:
+            return False
+        self._emit(state="error", error=text, prompt=text, camera=r)
+        return True
+
     def _make_dir(self):
         sessions = os.path.join(self.base_dir, "sessions")
         os.makedirs(sessions, exist_ok=True)
@@ -1117,6 +1181,13 @@ class Session:
                                                 self.script_path))
         for s in self.skipped:
             self._log("skipping %s: %s" % (s["section"], s["reason"]))
+        cam = self._status.get("camera")
+        if cam:
+            self._log("camera check: %s" % cam.get("summary", cam.get("status")))
+            for line in cam.get("evidence", []):
+                self._log("  " + line)
+        elif not self.check_cameras and not self.dry_run:
+            self._log("camera check skipped (--ignore-cameras)")
 
     def _setup(self):
         ring_path = self.ring or os.path.join(self.hands_dir, "cam-ring")
@@ -1149,6 +1220,9 @@ class Session:
             "script": {"version": self.script.get("version"), "sections": [s["id"] for s in self.plan],
                        "skipped": self.skipped},
             "mode": "auto" if self.auto else "step", "takes": [], "status": "recording"}
+        cam = self._status.get("camera")
+        if cam:
+            self._session_json["camera"] = {"status": cam.get("status"), "reason": cam.get("reason", "")}
         if self.dry_run:
             self._session_json["dry_run"] = True
         if self.speed != 1:
@@ -1325,8 +1399,12 @@ class Session:
             self._session_json["ended"] = time.strftime("%Y-%m-%dT%H:%M:%S%z")
             if error:
                 self._session_json["error"] = error
+            if self._stop_note and state == "stopped":
+                self._session_json["stop_reason"] = self._stop_note
             self._save_session()
         self._log("%s%s" % (state, ": " + error if error else ""))
+        if self._stop_note and state == "stopped":
+            self._log(self._stop_note)
         self._stop_units()
         if self._button:
             self._button.stop()
@@ -1349,7 +1427,8 @@ class Session:
             self._log_file = None
         screen = {"done": self.script.get("done"), "stopped": self.script.get("stopped")}.get(state) or {}
         self._emit(state=state, error=error, seconds_left=0.0, note="", take=None, section="",
-                   prompt=error or screen.get("text", "").replace("|", "\n"), hands={"left": None, "right": None},
+                   prompt=error or (self._stop_note if state == "stopped" else "") or screen.get("text", "").replace("|", "\n"),
+                   hands={"left": None, "right": None}, nohands=False,
                    waiting=False, countdown=0, big="", can_redo=False, image="", image_mode="", caption="",
                    position="", distance="")
 
@@ -1455,8 +1534,15 @@ class Session:
             fb["disk"] = now
             if shutil.disk_usage(self.session_dir).free < MIN_FREE:
                 raise _Fail("The disk is nearly full: the session stopped")
-        live = None if self.dry_run else self._hands_file.read()
+        live = self._hands_file.read() if self._read_hands else None
         self._live = live
+        w = self._hold_watch
+        if w is not None:
+            w["reads"] += 1
+            if live is not None:
+                w["published"] += 1
+                if live["left"] or live["right"]:
+                    w["seen"] += 1
         p = self._prompt or {}
         asked = p.get("hands", "")
         # While a step waits for Next the chips show what's seen, with no notes yet.
@@ -1767,6 +1853,7 @@ class Session:
         steps = self._step_list(s)
         done = []      # the steps finished in this take: (index, id, from_ns, to_ns), for R
         again = set()  # steps to do again: they wait for Next too
+        retry = False  # after the no-hands stop: Next was pressed there, go straight to the countdown
         k = 0
         while k < len(steps):
             st, p = steps[k], steps[k]["p"]
@@ -1777,13 +1864,26 @@ class Session:
                 if not self.auto and (st["ready"] or k in again):
                     self._redo_ok = bool(done)
                     self._ready(s, st, where, can_redo=bool(done))
-                    self._await_next()
+                    if not retry:
+                        self._await_next()
+                    retry = False
                     self._redo_ok = True
                     started = self._countdown(i, s, st)
                 else:
+                    retry = False
                     self._redo_ok = True
                     self._emit(force=False, can_redo=True)
-                self._run_step(s, st, where)
+                check = self._read_hands and s["id"] == HANDS_CHECK_SECTION and k == 0 and st["kind"] == "prompt"
+                self._hold_watch = {"reads": 0, "published": 0, "seen": 0} if check else None
+                try:
+                    self._run_step(s, st, where)
+                finally:
+                    watch, self._hold_watch = self._hold_watch, None
+                if check and self._no_hands_seen(p, watch):
+                    self._no_hands(s, st, where, started if started is not None else self._step_t0, watch)
+                    again.add(k)
+                    retry = True
+                    continue
                 done.append((k, p["id"], started if started is not None else self._step_t0, mono_ns()))
                 if not self.auto and (k + 1 == len(steps) or steps[k + 1]["ready"] or k + 1 in again):
                     self._hold_end()
@@ -1809,6 +1909,66 @@ class Session:
                     self._log("    redo %s" % pid)
                     again.add(k)
 
+    def _no_hands_seen(self, p, w):
+        """The no-hands check's verdict on a hold (logged either way): True if the tracker
+        published through it and never saw a hand."""
+        enough = w["reads"] >= HANDS_CHECK_MIN_READS and w["published"] >= HANDS_CHECK_PUBLISHED * w["reads"]
+        if not enough:
+            self._log("    hands check %s: the tracker published in %d of %d reads: can't tell"
+                      % (p["id"], w["published"], w["reads"]))
+            return False
+        self._log("    hands check %s: a hand in %d of %d reads (%d published)"
+                  % (p["id"], w["seen"], w["reads"], w["published"]))
+        return w["seen"] == 0
+
+    def _no_hands(self, s, st, where, from_ns, w):
+        """The first step saw no hands at all: stop it (its range marked as redone, so it gets
+        no labels), run the camera check, say so, and wait. Next or R tries the step again; Stop
+        (Esc) ends the session with the camera check's result; S skips the section."""
+        p = st["p"]
+        if not self.auto:
+            self._hold_end()
+        elif self._recording:
+            self._stop_recording()
+            self._event("pause")
+        self._event("nohands", id=p["id"], reads=w["reads"], published=w["published"])
+        if from_ns is not None:
+            self._event("redo", id=p["id"], **{"from": from_ns, "to": mono_ns()})
+        cam = self.camera_check_fn() if self.check_cameras else None
+        cam_text = camera_text(cam)
+        summary = (cam or {}).get("summary", "not run")
+        self._log("    no hands seen in %s: asking to try again or stop; camera check: %s" % (p["id"], summary))
+        for line in (cam or {}).get("evidence", []):
+            self._log("      " + line)
+        text = "%s|%s|%s" % (NO_HANDS_TEXT, cam_text or "The camera check found nothing wrong (%s)." % summary,
+                             NO_HANDS_RETRY)
+        self._stop_note = "Stopped: no hands were seen in the first step. Camera check: %s." % summary
+        if cam_text:
+            self._stop_note += " " + cam_text
+        self._prompt = None
+        self._view()
+        self._show(NO_HANDS_TITLE, where, text)
+        for key, c in (("big", "big "), ("countdown", "countdown off"), ("bar", "bar off"), ("target", "target off"),
+                       ("note", "note "), ("hands", "hands off off")):
+            self._panel.set(key, c)
+        self._emit(state="nohands", prompt=text.replace("|", "\n"), camera=cam, nohands=True, can_redo=True,
+                   big="", countdown=0, seconds_left=0.0)
+        self._redo_ok = True
+        try:
+            self._await_next()
+        except _Redo:
+            pass   # R here is the same as Next: try again
+        except _Skip:
+            self._stop_note = ""
+            self._emit(nohands=False)
+            raise
+        self._stop_note = ""
+        self._log("    trying %s again" % p["id"])
+        self._emit(nohands=False)
+        if self.auto and self._take and not self._recording:
+            self._event("resume")
+            self._start_recording()
+
     def _ready(self, s, st, where, can_redo):
         """Step mode: show the step (text, picture, diagram) with "Ready?"."""
         p = st["p"]
@@ -2002,6 +2162,8 @@ def main():
                     help="don't read the headset's button (gpio-keys KEY_SELECT: Next, pause, resume)")
     ap.add_argument("--button-device", metavar="PATH",
                     help="test: read the button from this event device or FIFO of input_event structs (also in a dry run)")
+    ap.add_argument("--ignore-cameras", action="store_true",
+                    help="start even if the camera check (hands/camcheck.py) finds the upper cameras off")
     a = ap.parse_args()
 
     known = ("pencil", "phone", "cup", "keyboard", "mouse", "gamepad", "small")
@@ -2049,7 +2211,7 @@ def main():
     s = Session(base, profile, checklist, a.lighting, a.script, ring=a.ring, start_processes=not a.no_start,
                 dry_run=a.dry_run, speed=a.speed, on_status=on_status, hands_dir=a.hands_dir, panel_bin=a.panel,
                 auto=a.auto, next_after=a.next_after, poses_dir=a.poses, button=not a.no_headset_button,
-                button_device=a.button_device)
+                button_device=a.button_device, check_cameras=not a.ignore_cameras)
     est = plan_seconds(s.script, s.plan, auto=a.auto)
     worst = plan_seconds(s.script, s.plan, worst=True, auto=a.auto)
     print("%d sections, %s mode: about %.1f min%s (at most %.1f)%s" % (
@@ -2065,6 +2227,16 @@ def main():
             print("  %-18s %-9s %3d prompts %3d steps %5.0f s" % (sec["id"], sec["kind"], len(sec["prompts"]),
                                                                 section_steps(sec), section_seconds(sec, auto=a.auto)))
         return 0
+    if not a.dry_run and not a.ring:
+        cam = camera_check()
+        print("cameras: %s" % cam["summary"])
+        for line in cam.get("evidence", []):
+            print("  " + line)
+        if camera_text(cam):
+            print(camera_text(cam))
+            if not a.ignore_cameras:
+                print("Not starting (--ignore-cameras starts anyway).")
+                return 3
     if not a.dry_run:
         light = ring_lighting(a.ring)
         match = similar_lighting(base, {"chosen": a.lighting, "ring": light}) if light else None

hands/rec/tests/test_nohands.py

@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+"""Tests for the session's camera check: the preflight that keeps a session from starting
+with the upper cameras off, and the early no-hands stop in the hand-size section's first step
+(dry runs with a made-up hands file; no processes, no real cameras).
+
+  python3 hands/rec/tests/test_nohands.py
+"""
+import json
+import os
+import sys
+import threading
+import unittest
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(HERE))
+sys.path.insert(0, HERE)
+import session  # noqa: E402
+import camcheck  # noqa: E402  (session.py put hands/ on the path)
+from test_session import SessionBase  # noqa: E402
+
+SCRIPT = {
+    "version": 1, "intro_s": 1, "between_s": 1,
+    "welcome": {"title": "Test", "seconds": 1, "text": "A test."},
+    "done": {"title": "Done", "seconds": 0, "text": "Done."},
+    "stopped": {"title": "Stopped", "seconds": 0, "text": "Stopped."},
+    "sections": [
+        {"id": "hand-size", "title": "Hand size", "intro": "Hand size.", "go": "Hold",
+         "defaults": {"hands": "both"},
+         "prompts": [{"text": "Both hands flat.", "seconds": 6, "pose": "flat"},
+                     {"text": "Backs.", "seconds": 2, "pose": "flat-back"}]}]}
+
+HAND = {"palm": [0.0, 0.0, -0.4], "palm_m": 0.4, "tip": [0.0, 0.05, -0.45]}
+DEGRADED = {"status": "degraded", "reason": camcheck.VCINT_REASON, "summary": camcheck.DEGRADED_VCINT,
+            "evidence": ["17:02:47 Failed to load VCINT FPGA image when passthrough cameras are connected"]}
+OK = {"status": "ok", "reason": "4 tracking cameras running", "summary": "ok", "evidence": []}
+
+
+class FakeHands:
+    """Stands in for the hands file: what read() gives can change mid-session."""
+
+    def __init__(self, value):
+        self.value = value
+        self.lock = threading.Lock()
+
+    def set(self, value):
+        with self.lock:
+            self.value = value
+
+    def read(self):
+        with self.lock:
+            return None if self.value is None else dict(self.value)
+
+
+class NoHandsTest(SessionBase):
+    def setUp(self):
+        super().setUp()
+        with open(self.script, "w") as f:
+            json.dump(SCRIPT, f)
+
+    def events(self, s):
+        with open(os.path.join(s.session_dir, "takes", "01-hand-size", "prompts.jsonl")) as f:
+            # feedback lines (twice a second, as the tracker publishes) aren't what's tested here
+            return [e for e in map(json.loads, f) if e["event"] != "feedback"]
+
+    def run_to_first_hold(self, hands, camera=DEGRADED, **kw):
+        s = self.session(hands_reader=hands, speed=3, **kw)
+        s.camera_check_fn = lambda: camera
+        s.start()
+        self.waiting(s, "starting")
+        s.next_step()
+        self.waiting(s, "intro")
+        s.next_step()
+        self.waiting(s, "ready", "Both hands flat.")
+        s.next_step()
+        return s
+
+    def session_json(self, s):
+        with open(os.path.join(s.session_dir, "session.json")) as f:
+            return json.load(f)
+
+    def log(self, s):
+        with open(os.path.join(s.session_dir, "session.log")) as f:
+            return f.read()
+
+    def test_no_hands_then_redo(self):
+        hands = FakeHands({"left": None, "right": None})   # the tracker publishes, sees nothing
+        s = self.run_to_first_hold(hands)
+        st = self.wait_for(s, lambda st: st["state"] == "nohands" and st["waiting"], "the no-hands stop")
+        self.assertTrue(st["nohands"])
+        self.assertTrue(st["can_redo"])
+        self.assertEqual(st["camera"]["status"], "degraded")
+        self.assertIn(camcheck.USER_TEXT, st["prompt"])
+        self.assertIn("panel: title " + session.NO_HANDS_TITLE, self.panel)
+        hands.set({"left": HAND, "right": HAND})
+        s.next_step()   # try again: straight to the countdown, no second ready wait
+        self.wait_for(s, lambda st: st["state"] == "countdown", "the countdown again")
+        self.waiting(s, "ready", "Backs.")
+        s.next_step()
+        s.join(20)
+        self.assertEqual(s.state, "done")
+        ev = self.events(s)
+        names = [e["event"] for e in ev]
+        self.assertEqual(names, ["take", "ready", "prompt", "wait", "nohands", "redo",
+                                 "ready", "prompt", "wait", "ready", "prompt", "wait", "end"])
+        nohands, redo = ev[4], ev[5]
+        self.assertGreaterEqual(nohands["reads"], session.HANDS_CHECK_MIN_READS)
+        self.assertEqual(redo["from"], ev[1]["t"])   # the failed try gets no labels
+        self.assertTrue(ev[2]["t"] < redo["to"])
+        # the retry wasn't stopped: a hand was seen
+        self.assertRegex(self.log(s), r"hands check hand-size/flat/both: a hand in [1-9][0-9]* of")
+        self.assertNotIn("stop_reason", self.session_json(s))
+
+    def test_no_hands_then_stop(self):
+        s = self.run_to_first_hold(FakeHands({"left": None, "right": None}))
+        self.wait_for(s, lambda st: st["state"] == "nohands" and st["waiting"], "the no-hands stop")
+        s.stop(wait=10)
+        self.assertEqual(s.state, "stopped")
+        sj = self.session_json(s)
+        self.assertIn("no hands were seen", sj["stop_reason"])
+        self.assertIn(camcheck.DEGRADED_VCINT, sj["stop_reason"])
+        self.assertIn(camcheck.USER_TEXT, s._status["prompt"])
+        self.assertIn("camera check: " + camcheck.DEGRADED_VCINT, self.log(s))
+
+    def test_redo_key_and_skip(self):
+        hands = FakeHands({"left": None, "right": None})
+        s = self.run_to_first_hold(hands, camera=OK)
+        st = self.wait_for(s, lambda st: st["state"] == "nohands" and st["waiting"], "the no-hands stop")
+        self.assertIn("The camera check found nothing wrong", st["prompt"])
+        s.redo()   # R is the same as Next there
+        self.wait_for(s, lambda st: st["state"] == "countdown", "the countdown again")
+        self.wait_for(s, lambda st: st["state"] == "nohands" and st["waiting"], "no hands again")
+        s.skip()   # S skips the section; the no-hands note doesn't stick
+        s.join(20)
+        self.assertEqual(s.state, "done")
+        self.assertNotIn("stop_reason", self.session_json(s))
+        self.assertEqual([e["event"] for e in self.events(s)].count("nohands"), 2)
+
+    def test_hand_seen_sometimes(self):
+        hands = FakeHands({"left": None, "right": None})
+        s = self.run_to_first_hold(hands)
+        self.wait_for(s, lambda st: st["state"] == "running", "the hold")
+        hands.set({"left": HAND, "right": None})   # one hand, for part of the hold: fine
+        self.waiting(s, "ready", "Backs.")
+        s.next_step()
+        s.join(20)
+        self.assertEqual(s.state, "done")
+        self.assertNotIn("nohands", [e["event"] for e in self.events(s)])
+
+    def test_no_tracker_cant_tell(self):
+        s = self.run_to_first_hold(FakeHands(None))   # nothing published: can't tell, go on
+        self.waiting(s, "ready", "Backs.")
+        s.next_step()
+        s.join(20)
+        self.assertEqual(s.state, "done")
+        self.assertNotIn("nohands", [e["event"] for e in self.events(s)])
+        self.assertIn("can't tell", self.log(s))
+
+    def test_only_the_first_step(self):
+        hands = FakeHands({"left": HAND, "right": HAND})
+        s = self.run_to_first_hold(hands)
+        self.wait_for(s, lambda st: st["state"] == "running", "the hold")
+        self.waiting(s, "ready", "Backs.")
+        hands.set({"left": None, "right": None})   # gone in step 2: notes, but no stop
+        s.next_step()
+        s.join(20)
+        self.assertEqual(s.state, "done")
+        self.assertNotIn("nohands", [e["event"] for e in self.events(s)])
+
+    def test_auto_mode(self):
+        hands = FakeHands({"left": None, "right": None})
+        s = self.session(hands_reader=hands, speed=3, auto=True)
+        s.camera_check_fn = lambda: DEGRADED
+        s.start()
+        self.wait_for(s, lambda st: st["state"] == "nohands" and st["waiting"], "the no-hands stop")
+        hands.set({"left": HAND, "right": HAND})
+        s.next_step()
+        s.join(20)
+        self.assertEqual(s.state, "done")
+        names = [e["event"] for e in self.events(s)]
+        self.assertEqual(names, ["take", "prompt", "prompt", "pause", "nohands", "redo", "resume", "prompt",
+                                 "prompt", "end"])
+
+
+class PreflightTest(SessionBase):
+    def test_degraded_stops_before_anything(self):
+        s = self.session()
+        s.dry_run = False            # as a real session; the check is the only step run
+        s.camera_check_fn = lambda: DEGRADED
+        self.assertTrue(s._preflight())
+        self.assertEqual(s.state, "error")
+        self.assertEqual(s._status["error"], camcheck.USER_TEXT)
+        self.assertEqual(s.session_dir, "")   # no session folder made
+        self.assertFalse(os.path.exists(os.path.join(self.tmp, "base", "sessions")))
+
+    def test_other_degraded_text(self):
+        r = dict(DEGRADED, reason="only 3 of 4 tracking cameras running")
+        self.assertIn("only 3 of 4", session.camera_text(r))
+        self.assertEqual(session.camera_text(OK), "")
+        self.assertEqual(session.camera_text({"status": "unknown"}), "")
+        self.assertEqual(session.camera_text(None), "")
+
+    def test_ok_unknown_ignored(self):
+        for result in (OK, {"status": "unknown", "summary": "unknown: x", "evidence": []}):
+            s = self.session()
+            s.dry_run = False
+            s.camera_check_fn = lambda: result
+            self.assertFalse(s._preflight())
+        s = self.session(check_cameras=False)
+        s.dry_run = False
+        s.camera_check_fn = lambda: DEGRADED
+        self.assertFalse(s._preflight())   # --ignore-cameras
+        s = self.session(ring="/tmp/some-ring")
+        s.dry_run = False
+        s.camera_check_fn = lambda: DEGRADED
+        self.assertFalse(s._preflight())   # ft-ringplay's frames: no headset cameras involved
+
+    def test_camera_check_never_raises(self):
+        orig = camcheck.check
+        try:
+            camcheck.check = lambda: 1 / 0
+            r = session.camera_check()
+        finally:
+            camcheck.check = orig
+        self.assertEqual(r["status"], "unknown")
+
+
+if __name__ == "__main__":
+    unittest.main()

hands/tests/test_camcheck.py

@@ -0,0 +1,274 @@
+#!/usr/bin/env python3
+"""Tests for hands/camcheck.py: the real XRService log of 2026-10-02 (a good start at 13:39,
+the VCINT failure after the 17:02 wake) cut at several points, synthetic logs for the other
+cases, and a made-up ft-camd ring. Nothing here touches SteamVR or the cameras.
+
+  python3 hands/tests/test_camcheck.py
+"""
+import io
+import json
+import os
+import shutil
+import struct
+import sys
+import tempfile
+import time
+import unittest
+from contextlib import redirect_stdout
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(HERE))
+import camcheck  # noqa: E402
+
+REAL_LOG = os.path.expanduser("~/.local/share/Steam/logs/XRService-2026.10.02/XRService-13-39-32.log")
+
+
+def L(t, text, level="INFO"):
+    """A log line as XRService writes it (with its colour codes on INFO lines)."""
+    if level == "INFO":
+        return "Fri Oct 02 2026 %s.000000 INFO: \x1b[0;36m%s\x1b[0m" % (t, text)
+    return "Fri Oct 02 2026 %s.000000 %s: %s" % (t, level, text)
+
+
+START = [L("10:00:00", "XRService logging to /tmp/x.log. Use --showLogToConsole to see it here as well")]
+GOOD_OPEN = [
+    L("10:00:01", "FPGA state check: PASSTHRU (register value: 0x00011212)"),
+    L("10:00:01", "Passthrough connected but FPGA is PASSTHRU - loading VCINT"),
+    L("10:00:01", "Loading FPGA image: VCINT"),
+    L("10:00:02", "FPGA image VCINT loaded and verified successfully"),
+    L("10:00:02", "Upper cameras FPGA interleaving support: 1 (Driver features available = 1 | VCINT loaded = 1)"),
+    L("10:00:02", "[buildMediaCtlSetupTasks] Created 6 tasks (4 tracking, 2 passthrough)"),
+] + [L("10:00:03", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: /dev/v4l-subdev3%d"
+       % (i, n, i)) for i, n in enumerate((9, 13, 6, 7))]
+CLOSE = [L("11:00:00", "[SystemdInhibitor] Received systemd suspend notification"),
+         L("11:00:00", "[DeckardCaptureSource] Closing tracking camera interfaces camerasToUse: 1111"),
+         L("11:00:01", "[DeckardCaptureSource] Streaming paused")]
+RESUME = [L("11:30:00", "[SystemdInhibitor] Received systemd resume notification"),
+          L("11:30:00", "Failed to read FPGA register 0x12 (exit code 256)", "ERROR"),
+          L("11:30:00", "Passthrough connected but FPGA is ERROR/UNKNOWN - loading VCINT"),
+          L("11:30:00", "Loading FPGA image: VCINT")]
+FAIL = [L("11:30:01", "FPGA load failed (exit code 256): Loading VC Interleaving FPGA image", "ERROR"),
+        "Loading FPGA image from bitstream file /usr/lib/deckard-fpga/images/csi_agg_deckard_ev1_1_vcint.binx",
+        "  FAILED: FPGA config_done signal did not assert",
+        L("11:30:01", "Failed to load VCINT FPGA image when passthrough cameras are connected", "ERROR"),
+        L("11:30:01", "Upper cameras FPGA interleaving support: 0 (Driver features available = 1 | VCINT loaded = 0)"),
+        L("11:30:01", "[buildMediaCtlSetupTasks] Created 4 tasks (2 tracking, 2 passthrough)"),
+        L("11:30:01", "TrackingCameraInit: index: 0. video device: /dev/video9. v4l subdevice: /dev/v4l-subdev31"),
+        L("11:30:01", "TrackingCameraInit: index: 1. video device: /dev/video13. v4l subdevice: /dev/v4l-subdev30"),
+        L("11:30:02", "[DeckardCaptureSource] Streaming resumed (FPGA: PASSTHRU, VC interleaving: disabled)")]
+# A wake that works prints no "Created N tasks" (2026-10-01 16:39), so an older one must not count.
+RESUME_OK = [L("12:30:00", "[SystemdInhibitor] Received systemd resume notification"),
+             L("12:30:00", "Passthrough connected but FPGA is ERROR/UNKNOWN - loading VCINT"),
+             L("12:30:01", "FPGA image VCINT loaded and verified successfully"),
+             L("12:30:01", "FPGA state check: VCINT (register value: 0x00021211)"),
+             L("12:30:01", "Upper cameras FPGA interleaving support: 1 (Driver features available = 1 | VCINT loaded = 1)")] + \
+            [L("12:30:01", "TrackingCameraInit: index: %d. video device: /dev/video%d. v4l subdevice: x" % (i, n))
+             for i, n in enumerate((9, 13, 6, 7))] + \
+            [L("12:30:02", "[DeckardCaptureSource] Streaming resumed (FPGA: VCINT, VC interleaving: enabled)")]
+EXIT = [L("13:00:00", "XRService - main thread exiting"), L("13:00:00", "Exiting XRService")]
+
+
+def state(lines):
+    return camcheck.LogState("test").feed_text("\n".join(lines))
+
+
+class SyntheticLogs(unittest.TestCase):
+    def test_good_start(self):
+        st = state(START + GOOD_OPEN)
+        self.assertEqual(st.verdict()[:2], ("ok", "4 tracking cameras running"))
+        self.assertEqual(st.episode["vcint"], "ok")
+        self.assertEqual(st.upper_nodes(), (6, 7))
+
+    def test_good_start_tasks_only(self):
+        # "Created 6 tasks (4 tracking ...)" and no camera init yet: 4 tracking cameras planned
+        self.assertEqual(state(START + GOOD_OPEN[:6]).verdict()[0], "ok")
+
+    def test_starting(self):
+        self.assertEqual(state(START + GOOD_OPEN[:3]).verdict()[:2], ("unknown", "the cameras are starting"))
+
+    def test_vcint_already_loaded(self):
+        # A SteamVR restart in the same boot (2026-10-01 15:27): the FPGA still has VCINT.
+        lines = START + [L("10:00:01", "FPGA state check: VCINT (register value: 0x00021211)"),
+                         L("10:00:01", "Upper cameras FPGA interleaving support: 1 (Driver features available = 1 | VCINT loaded = 1)"),
+                         L("10:00:01", "[buildMediaCtlSetupTasks] Created 6 tasks (4 tracking, 2 passthrough)")]
+        st = state(lines)
+        self.assertEqual(st.verdict()[0], "ok")
+        self.assertEqual(st.episode["vcint"], "loaded")
+
+    def test_asleep(self):
+        status, reason, ev = state(START + GOOD_OPEN + CLOSE).verdict()
+        self.assertEqual(status, "unknown")
+        self.assertIn("closed", reason)
+        self.assertTrue(any("Closing tracking camera interfaces" in e for e in ev))
+
+    def test_vcint_failure(self):
+        st = state(START + GOOD_OPEN + CLOSE + RESUME + FAIL)
+        status, reason, ev = st.verdict()
+        self.assertEqual((status, reason), ("degraded", camcheck.VCINT_REASON))
+        self.assertTrue(any("Created 4 tasks (2 tracking, 2 passthrough)" in e for e in ev))
+        self.assertTrue(any("Closing tracking camera interfaces" in e for e in ev))
+        self.assertFalse(any("\x1b" in e for e in ev))
+        self.assertEqual(st.snapshot()["failure"], "test@11:30:01")
+        self.assertEqual(len(st.failures), 1)
+
+    def test_failure_seen_before_the_cameras_start(self):
+        # the failure line alone (camera init not logged yet): degraded already
+        st = state(START + GOOD_OPEN + CLOSE + RESUME + FAIL[:4])
+        self.assertEqual(st.verdict()[0], "degraded")
+
+    def test_wake_after_failure_works(self):
+        st = state(START + GOOD_OPEN + CLOSE + RESUME + FAIL + CLOSE + RESUME_OK)
+        self.assertEqual(st.verdict()[0], "ok")
+        self.assertEqual(st.snapshot()["failure"], "")
+        self.assertEqual(len(st.failures), 1)   # still remembered for the log's history
+
+    def test_fewer_cameras_without_vcint(self):
+        lines = START + [L("10:00:01", "[buildMediaCtlSetupTasks] Created 4 tasks (2 tracking, 2 passthrough)")]
+        status, reason, _ = state(lines).verdict()
+        self.assertEqual(status, "degraded")
+        self.assertEqual(reason, "only 2 of 4 tracking cameras running")
+
+    def test_exited_and_empty(self):
+        self.assertEqual(state(START + GOOD_OPEN + EXIT).verdict()[0], "unknown")
+        self.assertEqual(state([]).verdict()[0], "unknown")
+        self.assertEqual(state(START).verdict()[:2], ("unknown", "the cameras haven't started yet in this log"))
+
+    def test_new_instance_resets(self):
+        st = state(START + GOOD_OPEN + CLOSE + RESUME + FAIL + START + GOOD_OPEN)
+        self.assertEqual(st.verdict()[0], "ok")
+        self.assertEqual(st.failures, [])
+
+    def test_incremental_equals_whole(self):
+        text = "\n".join(START + GOOD_OPEN + CLOSE + RESUME + FAIL)
+        st = camcheck.LogState("test")
+        for line in text.split("\n"):
+            st.feed(line + "\n")
+        self.assertEqual(st.snapshot(), state(START + GOOD_OPEN + CLOSE + RESUME + FAIL).snapshot())
+
+
+@unittest.skipUnless(os.path.exists(REAL_LOG), "the 2026-10-02 XRService log isn't on this machine")
+class RealLog(unittest.TestCase):
+    """The log with both the good 13:39 start and the 17:02 failure, cut in copies in /tmp."""
+
+    @classmethod
+    def setUpClass(cls):
+        with open(REAL_LOG, errors="replace") as f:
+            cls.lines = f.read().split("\n")
+        cls.tmp = tempfile.mkdtemp(prefix="camcheck-test-")
+
+    @classmethod
+    def tearDownClass(cls):
+        shutil.rmtree(cls.tmp, ignore_errors=True)
+
+    def cut_after(self, needle, nth=1):
+        """The log up to and including the nth line containing needle, as a file in /tmp."""
+        seen = 0
+        for i, line in enumerate(self.lines):
+            if needle in line:
+                seen += 1
+                if seen == nth:
+                    path = os.path.join(self.tmp, "cut-%d.log" % i)
+                    with open(path, "w") as f:
+                        f.write("\n".join(self.lines[:i + 1]) + "\n")
+                    return path
+        self.fail("no line %d with %r" % (nth, needle))
+
+    def check(self, path):
+        return camcheck.check(log=path, proc=False, ring=False)
+
+    def test_cut_points(self):
+        cases = [
+            ("Loading FPGA image: VCINT", 1, "unknown", "the cameras are starting"),
+            ("FPGA image VCINT loaded and verified successfully", 1, "unknown", "the cameras are starting"),
+            ("Created 6 tasks (4 tracking, 2 passthrough)", 1, "ok", None),
+            ("TrackingCameraInit: index: 3.", 1, "ok", None),
+            ("Upper cameras are not in sync", 1, "ok", None),         # 15:45: a resync, not a failure
+            ("Closing tracking camera interfaces", 1, "unknown", None),   # 16:41: asleep
+            ("Passthrough connected but FPGA is ERROR/UNKNOWN", 1, "unknown", "the cameras are starting"),
+            ("Failed to load VCINT FPGA image", 1, "degraded", camcheck.VCINT_REASON),
+            ("Created 4 tasks (2 tracking, 2 passthrough)", 1, "degraded", camcheck.VCINT_REASON),
+            ("Streaming resumed (FPGA: PASSTHRU", 1, "degraded", camcheck.VCINT_REASON),
+        ]
+        for needle, nth, status, reason in cases:
+            with self.subTest(cut=needle):
+                r = self.check(self.cut_after(needle, nth))
+                self.assertEqual(r["status"], status, r["summary"])
+                if reason:
+                    self.assertEqual(r["reason"], reason)
+        r = self.check(self.cut_after("Closing tracking camera interfaces"))
+        self.assertIn("closed", r["reason"])
+
+    def test_whole_log(self):
+        r = self.check(REAL_LOG)
+        self.assertEqual(r["summary"], camcheck.DEGRADED_VCINT)
+        ev = "\n".join(r["evidence"])
+        for want in ("17:02:47 Failed to load VCINT FPGA image", "interleaving support: 0",
+                     "Created 4 tasks (2 tracking, 2 passthrough)", "/dev/video9", "/dev/video13",
+                     "Closing tracking camera interfaces"):
+            self.assertIn(want, ev)
+        self.assertNotIn("/dev/video6", ev)   # the 13:39 start isn't this episode's evidence
+        self.assertTrue(r["failure"].endswith("@17:02:47"))
+        self.assertTrue(camcheck.is_vcint_failure(r))
+
+    def test_cli(self):
+        out = io.StringIO()
+        with redirect_stdout(out):
+            code = camcheck.main(["--log", REAL_LOG, "--no-proc", "--no-ring", "--json"])
+        self.assertEqual(code, 1)
+        self.assertEqual(json.loads(out.getvalue())["status"], "degraded")
+        out = io.StringIO()
+        with redirect_stdout(out):
+            code = camcheck.main(["--log", self.cut_after("TrackingCameraInit: index: 3."), "--no-proc", "--no-ring"])
+        self.assertEqual(code, 0)
+        self.assertEqual(out.getvalue().split("\n")[0], "ok")
+
+
+def make_ring(path, mono_names, alive=True):
+    """A ring header as ft-camd writes it (camd/fhring.h), no frames."""
+    cams = [(b"og01a1b", n.encode(), 9 + i) for i, n in enumerate(mono_names)]
+    hb = time.clock_gettime_ns(time.CLOCK_MONOTONIC) if alive else 1
+    data = bytearray(camcheck.RING_HDR.pack(b"FHRING01", 1, 0, len(cams), 0, 0, 4242, 0))
+    struct.pack_into("<Q", data, 40, hb)
+    for sensor, name, node in cams:
+        data += camcheck.RING_CAM.pack(sensor, name, node, 0, 1056, 1024, 1056, 16, 0, 0, 0, 0, 0, 0, 0.0)
+    with open(path, "wb") as f:
+        f.write(bytes(data))
+
+
+class Ring(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.mkdtemp(prefix="camcheck-ring-")
+        self.ring = os.path.join(self.tmp, "cam-ring")
+        self.log = os.path.join(self.tmp, "x.log")
+        with open(self.log, "w") as f:
+            f.write("\n".join(START + GOOD_OPEN) + "\n")
+
+    def tearDown(self):
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def test_two_cameras_in_ring(self):
+        make_ring(self.ring, ["slam_video9", "slam_video13"])
+        r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
+        self.assertEqual(r["status"], "degraded")
+        self.assertIn("ft-camd publishes only 2 of 4", r["reason"])
+
+    def test_four_cameras_in_ring(self):
+        make_ring(self.ring, ["a_video9", "b_video13", "c_video6", "d_video7", "a_video9_dk"])
+        r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
+        self.assertEqual(r["status"], "ok")
+        self.assertEqual(len(r["ring"]["mono"]), 4)   # the dark twin doesn't count
+
+    def test_stale_ring_ignored(self):
+        make_ring(self.ring, ["slam_video9"], alive=False)
+        r = camcheck.check(log=self.log, proc=False, ring_path=self.ring)
+        self.assertEqual(r["status"], "ok")
+        self.assertIn("stale ring", "\n".join(r["evidence"]))
+
+    def test_no_ring(self):
+        r = camcheck.check(log=self.log, proc=False, ring_path=os.path.join(self.tmp, "none"))
+        self.assertEqual(r["status"], "ok")
+        self.assertIsNone(r["ring"])
+
+
+if __name__ == "__main__":
+    unittest.main()

hands/tests/test_camwatch.py

@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+"""Tests for hands/ft-camwatch: its decision logic with made-up inputs (worn or not, VR apps,
+remote viewers, cooldown, once per failure), following a growing log, and its settings. It
+never notifies or restarts anything here.
+
+  python3 hands/tests/test_camwatch.py
+"""
+import importlib.machinery
+import importlib.util
+import os
+import shutil
+import sys
+import tempfile
+import unittest
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+HANDS = os.path.dirname(HERE)
+sys.path.insert(0, HANDS)
+sys.path.insert(0, HERE)
+from test_camcheck import CLOSE, FAIL, GOOD_OPEN, RESUME, RESUME_OK, START  # noqa: E402
+
+loader = importlib.machinery.SourceFileLoader("camwatch", os.path.join(HANDS, "ft-camwatch"))
+spec = importlib.util.spec_from_loader("camwatch", loader)
+cw = importlib.util.module_from_spec(spec)
+loader.exec_module(cw)
+
+ON = dict(cw.DEFAULTS, CAMWATCH_AUTO_RESTART="1", CAMWATCH_IDLE_S="60", CAMWATCH_COOLDOWN_MIN="30")
+OFF = dict(cw.DEFAULTS)
+IDLE = {"steamvr": True, "worn": False, "vr_apps": [], "remote": []}
+WORN = dict(IDLE, worn=True)
+F1, F2 = "a.log@17:02:47", "b.log@18:10:00"
+
+
+def kinds(acts):
+    return [k for k, _ in acts]
+
+
+class Decide(unittest.TestCase):
+    def setUp(self):
+        self.mem = cw.new_memory()
+
+    def d(self, now, failure, env=IDLE, conf=ON):
+        return cw.decide(now, failure, env, self.mem, conf)
+
+    def test_nothing_wrong(self):
+        self.assertEqual(self.d(0, "", {}), [])
+        self.assertEqual(self.mem["pending"], "")
+
+    def test_notify_once(self):
+        acts = self.d(0, F1, WORN, OFF)
+        self.assertEqual(kinds(acts), ["log", "notify", "log"])
+        self.assertIn("CAMWATCH_AUTO_RESTART=1", acts[2][1])
+        self.assertEqual(self.d(2, F1, WORN, OFF), [])          # same failure, same reason: quiet
+        self.assertEqual(self.d(4, F1, IDLE, OFF), [])          # off: never restarts, whatever the state
+        self.assertEqual(self.d(1000, F1, IDLE, OFF), [])
+
+    def test_notify_off(self):
+        acts = self.d(0, F1, WORN, dict(OFF, CAMWATCH_NOTIFY="0"))
+        self.assertNotIn("notify", kinds(acts))
+
+    def test_never_while_worn(self):
+        self.d(0, F1, WORN)
+        for t in range(2, 4000, 2):
+            self.assertNotIn("restart", kinds(self.d(t, F1, WORN)))
+        self.assertIn("worn", self.mem["why"])
+
+    def test_restart_after_idle_then_once(self):
+        self.d(0, F1, WORN)
+        self.assertNotIn("restart", kinds(self.d(10, F1, IDLE)))   # just taken off: wait
+        self.assertIn("waiting", self.mem["why"])
+        self.assertNotIn("restart", kinds(self.d(69, F1, IDLE)))
+        acts = self.d(70, F1, IDLE)
+        self.assertEqual(kinds(acts), ["log", "restart"])
+        self.assertEqual(acts[1][1], F1)
+        # the same failure again (the new XRService hasn't logged yet, or failed the same way)
+        acts = self.d(72, F1, IDLE)
+        self.assertNotIn("restart", kinds(acts))
+        self.assertIn("restart the headset", self.mem["why"])
+        for t in range(74, 10000, 500):
+            self.assertNotIn("restart", kinds(self.d(t, F1, IDLE)))
+
+    def test_put_on_resets_the_wait(self):
+        self.d(0, F1, IDLE)
+        self.d(50, F1, WORN)                                        # back on at 50 s
+        self.assertNotIn("restart", kinds(self.d(70, F1, IDLE)))   # off again: the 60 s start over
+        self.assertNotIn("restart", kinds(self.d(129, F1, IDLE)))
+        self.assertIn("restart", kinds(self.d(131, F1, IDLE)))
+
+    def test_cooldown(self):
+        self.d(0, F1, IDLE)
+        self.assertIn("restart", kinds(self.d(60, F1, IDLE)))
+        # The restart's new XRService fails too: a new failure, but within the cooldown.
+        self.d(90, "", IDLE)
+        acts = self.d(120, F2, IDLE)
+        self.assertIn("notify", kinds(acts))
+        self.assertNotIn("restart", kinds(acts))
+        self.assertNotIn("restart", kinds(self.d(60 + 30 * 60 - 2, F2, IDLE)))
+        self.assertIn("cooldown", self.mem["why"])
+        self.assertIn("restart", kinds(self.d(60 + 30 * 60, F2, IDLE)))
+
+    def test_vr_app_remote_and_no_steamvr(self):
+        for env, word in ((dict(IDLE, vr_apps=["AppId=450390"]), "VR app"),
+                          (dict(IDLE, remote=["0A00000B:D2F0"]), "remote desktop"),
+                          (dict(IDLE, steamvr=False), "SteamVR isn't running")):
+            with self.subTest(word=word):
+                self.mem = cw.new_memory()
+                for t in range(0, 600, 2):
+                    self.assertNotIn("restart", kinds(self.d(t, F1, env)))
+                self.assertIn(word, self.mem["why"])
+                # once it's gone (and the headset still off), the restart happens
+                self.assertIn("restart", kinds(self.d(600, F1, IDLE)))
+
+    def test_resolved(self):
+        self.d(0, F1, WORN)
+        acts = self.d(10, "", {})
+        self.assertEqual(kinds(acts), ["log"])
+        self.assertIn("no longer current", acts[0][1])
+        self.assertEqual(self.mem["pending"], "")
+        self.assertEqual(self.d(12, "", {}), [])
+
+    def test_memory_round_trip(self):
+        tmp = tempfile.mkdtemp(prefix="camwatch-test-")
+        try:
+            path = os.path.join(tmp, "state", "camwatch.json")
+            self.d(0, F1, IDLE)
+            self.d(60, F1, IDLE)
+            cw.save_memory(self.mem, path)
+            mem = cw.load_memory(path)
+            self.assertEqual(mem["restarted"], {F1: 60})
+            self.assertEqual(mem["notified"], [F1])
+            # the watcher restarted: the same failure neither notifies nor restarts again
+            acts = cw.decide(200, F1, IDLE, mem, ON)
+            self.assertNotIn("notify", kinds(acts))
+            self.assertNotIn("restart", kinds(acts))
+            self.assertEqual(cw.load_memory(os.path.join(tmp, "missing.json"))["restarted"], {})
+        finally:
+            shutil.rmtree(tmp, ignore_errors=True)
+
+
+class Follow(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.mkdtemp(prefix="camwatch-follow-")
+
+    def tearDown(self):
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def test_growing_log(self):
+        path = os.path.join(self.tmp, "x.log")
+        f = cw.Follower(path)
+        text = "\n".join(START + GOOD_OPEN) + "\n"
+        with open(path, "w") as out:
+            out.write(text)
+        self.assertEqual(cw.current_failure(f.poll()), "")
+        self.assertEqual(f.state.verdict()[0], "ok")
+        # half a line, then the rest: read as one line
+        more = "\n".join(CLOSE + RESUME + FAIL) + "\n"
+        cut = more.index("Failed to load VCINT") + 5
+        with open(path, "a") as out:
+            out.write(more[:cut])
+        self.assertEqual(cw.current_failure(f.poll()), "")
+        with open(path, "a") as out:
+            out.write(more[cut:])
+        self.assertTrue(cw.current_failure(f.poll()).endswith("@11:30:01"))
+        self.assertEqual(cw.current_failure(f.state, steamvr_running=False), "")
+        with open(path, "a") as out:
+            out.write("\n".join(CLOSE + RESUME_OK) + "\n")
+        self.assertEqual(cw.current_failure(f.poll()), "")
+
+    def test_new_log_starts_afresh(self):
+        a, b = os.path.join(self.tmp, "a.log"), os.path.join(self.tmp, "b.log")
+        with open(a, "w") as out:
+            out.write("\n".join(START + GOOD_OPEN + CLOSE + RESUME + FAIL) + "\n")
+        f = cw.Follower(a)
+        self.assertTrue(cw.current_failure(f.poll()))
+        with open(b, "w") as out:
+            out.write("\n".join(START + GOOD_OPEN) + "\n")
+        f.fixed = b   # as when xrservice.txt points at a new XRService's log
+        self.assertEqual(cw.current_failure(f.poll()), "")
+        self.assertEqual(f.path, b)
+
+
+class Settings(unittest.TestCase):
+    def test_read_conf(self):
+        tmp = tempfile.mkdtemp(prefix="camwatch-conf-")
+        try:
+            path = os.path.join(tmp, "frametop.conf")
+            with open(path, "w") as f:
+                f.write("# Frametop\nREMOTE=1   # comment\nCAMWATCH_AUTO_RESTART=1\nCAMWATCH_IDLE_S='90'\n"
+                        "CAMWATCH_IGNORE_APPIDS=\"1, 2\"\nnot a setting\n")
+            conf = cw.read_conf(path)
+            self.assertEqual(conf["CAMWATCH_AUTO_RESTART"], "1")
+            self.assertEqual(cw.conf_int(conf, "CAMWATCH_IDLE_S"), 90)
+            self.assertEqual(conf["CAMWATCH_IGNORE_APPIDS"], "1, 2")
+            self.assertEqual(cw.conf_int(conf, "CAMWATCH_COOLDOWN_MIN"), 30)   # the default
+            self.assertEqual(cw.read_conf(os.path.join(tmp, "none"))["CAMWATCH_AUTO_RESTART"], "0")
+            self.assertEqual(cw.conf_int({"CAMWATCH_IDLE_S": "soon"}, "CAMWATCH_IDLE_S"), 60)
+        finally:
+            shutil.rmtree(tmp, ignore_errors=True)
+
+    def test_headset_worn(self):
+        tmp = tempfile.mkdtemp(prefix="camwatch-bl-")
+        orig = cw.process_running
+        try:
+            os.makedirs(os.path.join(tmp, "panel0"))
+            bright = os.path.join(tmp, "panel0", "brightness")
+            for value, compositor, worn in (("95", True, True), ("0", True, False), ("95", False, False)):
+                with open(bright, "w") as f:
+                    f.write(value + "\n")
+                cw.process_running = lambda name, c=compositor: c and name == "vrcompositor"
+                self.assertEqual(cw.headset_worn(tmp), worn, (value, compositor))
+            self.assertFalse(cw.headset_worn(os.path.join(tmp, "none")))
+        finally:
+            cw.process_running = orig
+            shutil.rmtree(tmp, ignore_errors=True)
+
+    def test_remote_viewers(self):
+        tmp = tempfile.mkdtemp(prefix="camwatch-tcp-")
+        try:
+            path = os.path.join(tmp, "tcp")
+            with open(path, "w") as f:
+                f.write("  sl  local_address rem_address   st tx_queue rx_queue\n"
+                        "   0: 0100007F:170C 00000000:0000 0A 00000000:00000000\n"      # listening
+                        "   1: 0B00640A:170C 0C00640A:D2F0 01 00000000:00000000\n"      # a viewer
+                        "   2: 0B00640A:0016 0C00640A:D2F1 01 00000000:00000000\n")     # ssh
+            self.assertEqual(cw.remote_viewers(5900, (path,)), ["0C00640A:D2F0"])
+            self.assertEqual(cw.remote_viewers(5901, (path,)), [])
+        finally:
+            shutil.rmtree(tmp, ignore_errors=True)
+
+
+if __name__ == "__main__":
+    unittest.main()