Stop stalled OBS relays and handle intentional shutdown

docs/evidence/obs-followup-2026-10-25.md

@@ -0,0 +1,58 @@
+# OBS relay and MP4 follow-up, 25 October 2026 UTC
+
+This check used a Linux x86-64 host (`lxso2`) and the real Frame running
+SteamOS 0.5.6, build `20261022.6153909`. The host reached `ssh frame`. Before
+device work, the task atomically acquired `/tmp/frame-test.lock`, wrote its
+owner marker, and after cleanup removed only that marker and its lock. Battery
+was 99–100%; SteamVR 2.18.2 reported `activity_level: 3` (standby). Steam and
+SteamVR were left running. No passthrough session or game was started.
+
+The production `scripts/obs-headset-source.py` relay reached its first local
+UDP packet in 0.65–1.15 seconds across several runs. A 12-second wall-clock
+capture of its MPEG-TS packets contained 325 H.264 frames at 1280×720,
+spanning 10.83 seconds of video timestamps: 30.0 fps. All 324 adjacent frame
+intervals were 33.333–33.334 ms, with none over 50 ms. The 826 captured TS
+packets had zero sync-byte, transport-error-indicator, or per-PID continuity
+errors. Six final TS packets arrived after the receiver's 12-second cutoff and
+were not included in those counts. Static standby pixels cannot establish
+whether a moving game would drop frames.
+
+After the fix, the `--height 1080 --fps 60 --no-audio` fallback reached its
+first packet in 0.597 seconds. An eight-second wall-clock capture contained
+447 H.264 frames at 1920×1080 over 7.45 seconds of video timestamps: 60.0 fps.
+All frame intervals were 16.666–16.667 ms, none over 25 ms, and its 756 TS
+packets had zero continuity or transport-error-indicator errors. It had no
+audio stream, as requested. The helper exited 0 on Stop, and the owned lock
+was released.
+
+The stream also contained 48 kHz stereo AAC. A 440 Hz synthetic tone played
+through the Frame's default speaker output with real-time pacing appeared in
+the capture for about 0.89 seconds; decoded audio had a -21.0 dB peak. This
+proves an audio signal reached the output monitor and OBS transport. It does
+not prove audible monitoring, real game audio, or audio/video sync: there was
+no simultaneous visible marker to compare with the tone.
+
+The production MP4 recording commands were run separately for six seconds,
+then Stop, remux, and copy to the host completed. Stop returned after 7.27
+seconds from Start, including six seconds of recording; the remote ffmpeg
+exited 0. The copied 36,929-byte MP4 contained 188 H.264 frames at 1920×1080
+and 30 fps (6.266 seconds), plus 281 AAC packets (5.994 seconds). One decoded
+frame at two seconds had a single color, RGB `(22, 27, 53)`, across all
+2,073,600 pixels. The test's remote files were removed; the host copy stayed
+under `/tmp` for inspection.
+
+The relay previously returned an error when the app deliberately closed its
+control pipe, because its SSH child commonly exits 255 after termination.
+The same test reproduced this (`helper_exit: 1`) on the real Frame. The follow-up
+fix recognizes deliberate Stop and adds bounded startup and stall timeouts,
+with fake-SSH regression tests. A new locked real-Frame check after the fix
+received its first 1,316-byte UDP datagram in 0.697 seconds and returned
+`helper_exit_after_stop: 0` after closing the app control pipe. The longer
+transport and MP4 measurements above were taken before the code change; the
+fix does not alter codecs or capture commands.
+
+Prior [OBS 30 Media Source evidence](obs-media-source-real-2026-10-25.md)
+records a real Frame stream in OBS on Linux. This follow-up measured the relay
+with an FFmpeg receiver; OBS itself was not relaunched. Windows and macOS
+application paths, wearer-visible image latency, real game sound, audible
+playback, perceptual A/V sync, and active passthrough pixels remain unverified.

docs/obs-headset-source.md

@@ -27,6 +27,9 @@ Ctrl+C stops the checkout helper. The app closes a control pipe when you click
 Stop or quit; the helper then closes its SSH child, including on Windows. On
 Windows, install Python 3 and OpenSSH Client if Frame Control cannot find them.
 The Windows app Start/Stop path still needs a physical Windows/Frame test.
+If no first packet arrives within 20 seconds, or an active stream sends no
+packet for 10 seconds, the helper closes SSH and shows an error instead of
+leaving **Start OBS source** waiting or a frozen source running indefinitely.
 
 To share the OBS view in a call, see the short
 [Discord setup guide](streaming-to-discord.md).
@@ -49,6 +52,9 @@ default output was present in the saved audio. The headset was in standby, so
 the video showed the standby mirror. Nobody listened to OBS's monitoring output,
 and audio/video sync and active passthrough remain unverified. See the
 [test evidence](evidence/obs-media-source-real-2026-10-25.md).
+The [follow-up Linux/Frame transport check](evidence/obs-followup-2026-10-25.md)
+measured packet continuity and a short MP4 recording on the current SteamOS
+build. The headset was still in standby.
 
 ## Known issue: game audio in OBS
 

scripts/obs-headset-source.py

@@ -11,10 +11,13 @@ import socket
 import subprocess
 import sys
 import threading
+import time
 
 
 PACKET = 188
 DATAGRAM = PACKET * 7
+STARTUP_TIMEOUT = 20
+STREAM_STALL = 10
 
 
 def remote_command(height=720, fps=30, audio=True):
@@ -37,7 +40,7 @@ def remote_command(height=720, fps=30, audio=True):
             f'{maps} {encode} -f mpegts -flush_packets 1 -')
 
 
-def relay(source, destination):
+def relay(source, destination, on_packet=None):
     """Preserve 188-byte MPEG-TS packet boundaries in loopback datagrams."""
     pending = bytearray()
     total = 0
@@ -46,6 +49,8 @@ def relay(source, destination):
         size = min(len(pending) // PACKET * PACKET, DATAGRAM)
         if size:
             destination.send(bytes(pending[:size]))
+            if on_packet:
+                on_packet()
             del pending[:size]
             total += size
     if pending:
@@ -53,7 +58,7 @@ def relay(source, destination):
     return total
 
 
-def watch_control(stream, proc):
+def watch_control(stream, proc, stopped):
     """A desktop app closes its pipe to stop SSH, including on Windows.
 
     Windows TerminateProcess does not run Python's finally block. The pipe also
@@ -63,10 +68,53 @@ def watch_control(stream, proc):
         stream.read(1)
     except OSError:
         pass
+    stopped.set()
     if proc.poll() is None:
         proc.terminate()
 
 
+class StreamWatchdog:
+    """Close SSH if the first packet or a later packet takes too long."""
+
+    def __init__(self, proc, startup=None, stall=None):
+        self.proc = proc
+        self.startup = STARTUP_TIMEOUT if startup is None else startup
+        self.stall = STREAM_STALL if stall is None else stall
+        self.condition = threading.Condition()
+        self.last_packet = time.monotonic()
+        self.ready = False
+        self.closed = False
+        self.reason = ''
+        self.thread = threading.Thread(target=self._watch, daemon=True)
+        self.thread.start()
+
+    def packet(self):
+        with self.condition:
+            self.ready = True
+            self.last_packet = time.monotonic()
+            self.condition.notify()
+
+    def close(self):
+        with self.condition:
+            self.closed = True
+            self.condition.notify()
+        self.thread.join(timeout=1)
+
+    def _watch(self):
+        with self.condition:
+            while not self.closed:
+                left = self.last_packet + (self.stall if self.ready else self.startup) - time.monotonic()
+                if left <= 0:
+                    self.reason = (f'The Frame sent no OBS video for {self.startup:g} seconds' if not self.ready
+                                   else f'The Frame OBS video stopped for {self.stall:g} seconds')
+                    break
+                self.condition.wait(left)
+            else:
+                return
+        if self.proc.poll() is None:
+            self.proc.terminate()
+
+
 def main(argv=None):
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument('--frame', default='frame', help='SSH host or alias (default: frame)')
@@ -87,11 +135,17 @@ def main(argv=None):
     signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
     with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as udp:
         proc = subprocess.Popen(ssh, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE)
+        stopped = threading.Event()
+        watchdog = StreamWatchdog(proc)
         if args.control_stdin:
-            threading.Thread(target=watch_control, args=(sys.stdin.buffer, proc), daemon=True).start()
+            threading.Thread(target=watch_control, args=(sys.stdin.buffer, proc, stopped), daemon=True).start()
         try:
-            amount = relay(proc.stdout, _Sender(udp, target))
+            amount = relay(proc.stdout, _Sender(udp, target), watchdog.packet)
             code = proc.wait(timeout=10)
+            if stopped.is_set():
+                return 0
+            if watchdog.reason:
+                raise RuntimeError(watchdog.reason)
             if code or not amount:
                 raise RuntimeError(f'Frame stream stopped (SSH exit {code}; {amount} bytes). '
                                    'Check SteamVR and the audio monitor; try --no-audio if needed.')
@@ -99,6 +153,7 @@ def main(argv=None):
         except KeyboardInterrupt:
             return 0
         finally:
+            watchdog.close()
             proc.stdout.close()
             if proc.poll() is None:
                 proc.terminate()

tests/test_obs_headset_source.py

@@ -9,8 +9,10 @@ import socket
 import subprocess
 import sys
 import tempfile
+import threading
 import time
 import unittest
+from unittest import mock
 
 
 SCRIPT = Path(__file__).resolve().parents[1] / 'scripts' / 'obs-headset-source.py'
@@ -55,6 +57,29 @@ class ObsSourceTests(unittest.TestCase):
         self.assertIn('-f mpegts', with_audio)
         self.assertNotIn('@DEFAULT_MONITOR@', obs.remote_command(audio=False))
 
+    def test_watchdog_ends_a_stalled_child_on_every_host_platform(self):
+        class Child:
+            def __init__(self):
+                self.stopped = threading.Event()
+
+            def poll(self):
+                return 255 if self.stopped.is_set() else None
+
+            def terminate(self):
+                self.stopped.set()
+
+        for ready in (False, True):
+            with self.subTest(ready=ready):
+                child = Child()
+                watchdog = obs.StreamWatchdog(child, startup=0.1, stall=0.1)
+                if ready:
+                    watchdog.packet()
+                try:
+                    self.assertTrue(child.stopped.wait(1), 'a stalled SSH child was not terminated')
+                    self.assertIn('stopped' if ready else 'no OBS video', watchdog.reason)
+                finally:
+                    watchdog.close()
+
     @unittest.skipIf(sys.platform == 'win32', 'the packaged Start/Stop control is Linux/macOS only')
     def test_relay_stops_its_ssh_child_on_app_signal_or_control_pipe(self):
         for mode in ('signal', 'control-pipe'):
@@ -70,7 +95,7 @@ class ObsSourceTests(unittest.TestCase):
                             "import os, signal, sys, time\n"
                             "def stop(*_):\n"
                             "    open(os.environ['FAKE_SSH_STOPPED'], 'w').close()\n"
-                            "    sys.exit(0)\n"
+                            "    sys.exit(255)\n"
                             "signal.signal(signal.SIGTERM, stop)\n"
                             "sys.stdout.buffer.write(b'G' * 1316)\n"
                             "sys.stdout.buffer.flush()\n"
@@ -112,6 +137,33 @@ class ObsSourceTests(unittest.TestCase):
                 process.stdout.close()
                 process.stderr.close()
 
+    @unittest.skipIf(sys.platform == 'win32', 'the stand-in SSH script uses a POSIX shebang')
+    def test_startup_and_stream_stalls_end_ssh_with_a_clear_error(self):
+        for mode in ('startup', 'stream'):
+            with self.subTest(mode=mode), tempfile.TemporaryDirectory() as folder:
+                directory = Path(folder)
+                fake = directory / 'ssh'
+                fake.write_text("#!/usr/bin/env python3\n"
+                                "import os, signal, sys, time\n"
+                                "def stop(*_):\n"
+                                "    open(os.environ['FAKE_SSH_STOPPED'], 'w').close()\n"
+                                "    sys.exit(255)\n"
+                                "signal.signal(signal.SIGTERM, stop)\n"
+                                "if os.environ['FAKE_SSH_MODE'] == 'stream':\n"
+                                "    sys.stdout.buffer.write(b'G' * 1316)\n"
+                                "    sys.stdout.buffer.flush()\n"
+                                "while True: time.sleep(1)\n")
+                fake.chmod(0o755)
+                stopped = directory / 'stopped'
+                env = {'PATH': str(directory) + os.pathsep + os.environ.get('PATH', ''),
+                       'FAKE_SSH_STOPPED': str(stopped), 'FAKE_SSH_MODE': mode}
+                with mock.patch.dict(os.environ, env), \
+                     mock.patch.object(obs, 'STARTUP_TIMEOUT', 0.3), \
+                     mock.patch.object(obs, 'STREAM_STALL', 0.3):
+                    with self.assertRaisesRegex(RuntimeError, 'no OBS video' if mode == 'startup' else 'video stopped'):
+                        obs.main(['--frame', 'fake-frame'])
+                self.assertTrue(stopped.exists(), 'SSH survived the relay timeout')
+
 
 if __name__ == '__main__':
     unittest.main()