Mac in the headset: stream Mac windows and screens into the Frame as panels

Frame Control can now show any Mac window, or a whole screen, as its own
SteamVR panel on the Steam Frame (Tools -> Mac in the headset, macOS only).
Place it with the SteamVR dashboard; the laser clicks and scrolls, and the
Mac's own keyboard types.

- mac/frame-mac-view (Swift, no dependencies): ScreenCaptureKit capture per
  window or display, VideoToolbox H.264 with low-latency rate control (JPEG
  fallback), a loopback HTTP/WebSocket server, CGEvent/AX input playback,
  and a display-awake assertion while anyone watches.
- ui/frame_macview.py: starts the agent, runs an ssh -R tunnel with a
  supervisor that reopens it on the same port, and launches a Chromium app
  window per stream on gamescope's :0, tagged with STEAM_GAME for its own panel.
- Frame Control's key never leaves the Mac: viewers get single-use,
  per-source tickets and reconnect keys that Stop revokes.
- ui/mac-view.html: WebCodecs decode, keyframe recovery, pointer/wheel/keys back.
- Bundled in the Mac app build; tests/test_macview.py builds and drives the
  agent on macOS.

Verified on the Frame (build 20260925.6191901) with the test pattern: panel
in about 1.5 s, about 60 fps, Mac-to-window about 11-17 ms, tunnel recovery
in 4 s. Laser input and real window capture still need a person in the headset.

Also commits the other thread's first-party rule (steam-frame skill) and
the first-party options table in docs/streaming.md.

Reviewed by GPT-6 Astra (xhigh, read-only) over six rounds; all findings fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
saphidandClaude Opus 5.5 committed 2026-09-28 13:33:46 +10:00
1 parent 0016d9200c
commit a3c6e5c984
20 files changed
+2708 -6

No files matched your search

+45 -1
View File
@@ -39,6 +39,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
import frame_android # noqa: E402
import frame_catalog # noqa: E402
import frame_host # noqa: E402
import frame_macview # noqa: E402
import frame_store # noqa: E402
import frame_titles # noqa: E402
import frame_webinstall # noqa: E402
@@ -1213,10 +1214,50 @@ def _sweep_one(prefix, d):
pass
# ---- Mac in the headset (frame_macview.py) ----------------------------------
# The tunnel gets its own connection: the shared master's options would win
# over anything added after them.
macview = frame_macview.MacView(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=8"],
lambda remote, stdin=None, timeout=30: ssh(remote, stdin=stdin, timeout=timeout),
FRAME, track=_live_tunnels.add)
def macview_state(query=None):
if LOCAL:
return {"available": False, "reason": "Runs on your computer, not the headset."}
try:
# Refresh restarts the helper, which picks up a permission just granted.
if (query or {}).get("restart") == ["1"]:
macview.restart_agent()
return macview.state()
except frame_macview.MacViewError as e:
raise Failure(str(e), 500)
def macview_action(body):
"""{action: show|stop|permissions, src, quality, w, h}."""
if LOCAL:
raise Failure("Runs on your computer, not the headset.", 400)
action = body.get("action")
try:
if action == "show":
w, h = body.get("w"), body.get("h")
return macview.show(str(body.get("src") or ""), str(body.get("quality") or "balanced"),
int(w) if w else None, int(h) if h else None)
if action == "stop":
return macview.stop(body.get("src") or None)
if action == "permissions":
return macview.request_permissions()
except frame_macview.MacViewError as e:
raise Failure(str(e), 502)
raise Failure("unknown action", 400)
POST = {"/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard,
"/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots,
"/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start,
"/api/webinstall/cancel": webinstall_cancel}
"/api/webinstall/cancel": webinstall_cancel, "/api/macview": macview_action}
# ---- HTTP ------------------------------------------------------------------
@@ -1336,6 +1377,8 @@ class Handler(BaseHTTPRequestHandler):
"shared": frame_catalog.compat_db.shared()})
elif path == "/api/android/catalog":
self.send_json({"apps": frame_catalog.catalog()})
elif path == "/api/macview":
self.send_json(macview_state(parse_qs(url.query)))
elif path == "/api/status":
self.send_json(status({}))
elif path == "/api/steam/owned":
@@ -1529,6 +1572,7 @@ def main():
if not frame_host.WINDOWS:
signal.signal(signal.SIGTERM, signal.SIG_IGN)
webinstall_shutdown()
macview.shutdown() # close the headset's viewers before the agent goes
# The master was started with -N, so it stays up until told to exit.
if CONTROL:
subprocess.run([*MUX, "-O", "exit", FRAME], capture_output=True, stdin=subprocess.DEVNULL)