Mac in the headset: measure every frame, adapt to the network, separate windows

- Per-frame timing on the Mac's clock (capture, encode, network, decode,
  draw), viewer clock sync and reports, input echo, /stats and a HUD.
- scripts/macview-bench.py: repeatable runs on the real Frame, a shaping
  relay (no sudo), interleaved A/B between agent settings; results in
  bench/results/.
- Adaptive controller: ack-based send gate with jitter-aware slack, AIMD
  bitrate that knows when a stream is app-limited, fps then size tiers.
  On a 50->3->50 Mbit/s step, scroll p95 went from 4.7 s to 72 ms; no cost
  on a clean link.
- Separate mode: real AppKit event loop (HiDPI and NSScreen now work),
  cropped capture for fixed-size windows, windows kept on their display,
  graceful quit restores windows; stop/start races fixed.
- Encoder timeline clamp (no oversized frame after a pause).
- Frame Control shows each live stream's fps, delay, bitrate and tier.

Reviewed by GPT-6 Astra xhigh (read-only), 7 rounds; findings fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/how-the-frame-works.md

@@ -52,6 +52,7 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600
 | Power actions need `sudo`, which asks for the Developer Mode password over SSH. | Frame Control's power buttons |
 | **SSH server:** OpenSSH 9.7p1. It offers `publickey,password` (keyboard-interactive is off, PAM on) and also asks `userdbctl ssh-authorized-keys` for keys. OpenSSH ≥ 8.8 rejects SHA-1 `ssh-rsa` signatures by default, so a client whose RSA support is SHA-1 only (the Swift library Citadel, for one) can't log in with the RSA key that devkit pairing installs; use ed25519 (**inferred** from OpenSSH defaults). **Verified 2026-09-27**, BUILD_ID 20260922.6101926. | [iphone.md](iphone.md), `ui/frame_connect.py` |
 | **A Chromium app window on gamescope's `:0` with its own `STEAM_GAME` becomes a panel, even when started over SSH.** `chromium-xr/chrome --ozone-platform=x11 --app=URL --window-size=1280,720` got a 1920×1080 window, and tagging it produced `[Overlays] Created: valve.steam.desktopgame.<id>` in `~/.local/share/Steam/logs/vrwebhelper_systemui.txt`. Chromium XR decoded a 1080p H.264 WebCodecs stream from the Mac at about 60 fps. XTest events sent to `:0` (libXtst through Python ctypes) did not reach the page. The Frame has `libXtst`, `xprop`, `xwininfo`, `curl` and the `C.utf8` locale. **Verified 2026-09-28**, BUILD_ID 20260925.6191901. | [mac-in-headset.md](mac-in-headset.md) |
+| **Streaming video into a panel: what the Frame adds.** Chromium XR decodes H.264 in **software** (1920×1290 at about 9 Mbit/s took 4–6 ms per frame); its GL is ANGLE → zink → Turnip on the Adreno 750, and it logs `GetVSyncParametersIfAvailable() failed`. An idle page's `requestAnimationFrame` runs at about 140 Hz; when the headset is worn or woken, SteamVR picks the 4320×2160 @ 144 Hz mode. **An unworn headset throttles panel apps** whatever they draw: a local canvas page ran at 58 fps for about 6 s, then 28, then about 15 once in standby (vrserver logs `entering standby`, with `power.pauseCompositorOnStandby 1`). `vrcmd --handlewakeup` gave 137–144 fps for about 2 s, then 36. So a panel's frame rate and compositor delay can only be measured while it's worn. `tailscaled` runs with `--tun=userspace-networking` and cost about 8% of a core at 9 Mbit/s (0.5% over the LAN address). Wi-Fi power saving is on (`iw … get power_save`). **Verified 2026-09-28**, BUILD_ID 20260925.6191901. | [mac-in-headset.md](mac-in-headset.md#measuring) |
 | **Tools on the image:** Python 3.12.3, `ffmpeg`, `openssl`, `curl`, `rsync`, `zip`/`unzip`, `flatpak`, `wpctl`, `podman`. **No `adb`.** `steamos` is uid 1000, in `wheel`, and sudoers has `%wheel ALL=(ALL) ALL`, so `sudo -S` takes the Developer Mode password on stdin. **Verified 2026-09-27.** | Running Frame Control's server on the Frame (`FRAME_LOCAL=1`, [iphone.md](iphone.md)) |
 | **Each Lepton instance is a podman container** named `lepton-steamlaunch-<instance id>`, labelled with its ADB port (`podman ps --format '{{.Names}} {{.Labels.adb_port}}'`). `podman exec <container> /system/bin/sh -c '…'` runs Android's shell inside it with no adb at all (used for `pidof` and `logcat` by the app tester). Running `wm size`/`wm density` that way is untested. **Verified 2026-09-27.** | `ui/frame_android.py`, the iPhone app's display settings |
 | **Asleep means off the network.** In standby the Frame stops answering on its LAN address, `frame.local` and Tailscale alike (`Host is down`, `No route to host`, timeouts), and ping fails. It was unreachable for about 2.5 hours until woken. Nothing over SSH can wake it. **Verified 2026-09-27.** | Frame Control's offline banner and retries |

docs/mac-in-headset.md

@@ -99,9 +99,10 @@ ScreenCaptureKit (one window or display)
     keyframe instead of showing old frames late.
   - It falls back to JPEG stills (**Compatible** quality) where H.264 isn't
     available.
-- **Flow control.** When the link backs up, the agent skips capture frames
-  *before* encoding, so no reference frame goes missing. Once the link drains,
-  it sends the newest picture.
+- **Flow control.** The agent never lets frames queue up anywhere on the
+  way. It skips capture frames *before* encoding, so no reference frame goes
+  missing, and it lowers the bitrate, then the frame rate, then the size, to
+  fit the link (see [Adapting to the network](#adapting-to-the-network)).
 - **Input.**
   - A click on a window's panel brings that Mac window to the front
     (Accessibility API), then clicks at the same point. Double clicks, right
@@ -133,6 +134,157 @@ may ask again after an update.
 | Light | 1280 | 30 | H.264 | Weak Wi-Fi or Tailscale off the LAN |
 | Compatible | 1280 | 20 | JPEG | A Frame browser without H.264 |
 
+## Measuring
+
+Every frame carries a sequence number, and the agent records its journey on
+the Mac's clock (`Sources/Stats.swift`):
+
+| Stage | From → to |
+|---|---|
+| capture | the Mac composited it (ScreenCaptureKit's display time) → the agent got it |
+| queue, encode | → encoding started → the encoder finished |
+| network | → the viewer received it |
+| decode, draw | → WebCodecs decoded it → it was drawn on the page's canvas |
+| present | → the page's next animation frame |
+
+- **Clock sync.** The viewer syncs its clock to the Mac's the way NTP does:
+  it pings over the stream's own WebSocket and keeps the sample with the
+  shortest round trip. It then reports, in Mac time, when each frame arrived
+  (right away, so the agent can pace itself) and when it was decoded and
+  drawn (in batches every 250 ms).
+- **Input.** The first frame captured after a click or key carries that
+  event's id. So input latency is the viewer's event → injected on the Mac →
+  the first frame after it → drawn in the headset.
+- **Where to see it.**
+  - `GET /stats` (key required) returns every frame and input record.
+  - `/status` includes a two-second summary, which Frame Control's card
+    shows next to each live stream.
+  - In the headset, add `?stats=1` to the viewer or press
+    Ctrl+Alt+Shift+S for an overlay.
+- **The benchmark.** `scripts/macview-bench.py` runs fixed scenarios on the
+  real Frame, from the Mac, with nobody wearing the headset:
+  - **test** is the moving test pattern.
+  - **scroll** is a Chrome page on its own display scrolling at 240 pt/s,
+    which gives about 9 Mbit/s of real 1920×1290 video.
+  - **type** types into a Chrome text box, first fast and then with pauses.
+
+  It writes `bench/results/<date>-<commit>-<label>.json`, compares two
+  results, and runs interleaved A/B tests between agent settings (`ab`).
+  Wi-Fi changes from minute to minute, so single runs at different times
+  aren't comparable. Throttled links come from a shaping relay on the Mac,
+  which needs no sudo (`--net 50@0,3@8,50@16` means 50 Mbit/s, then 3 from
+  8 s, then 50 from 16 s).
+- **What's graded.** "Content" runs from when the Mac composited a frame
+  (or when ScreenCaptureKit delivered it, if that was earlier) to when it was
+  drawn in the viewer. The Frame's compositor adds its own delay after that.
+  That part is reported, but not graded: an unworn Frame throttles panels to
+  about 36 fps after a few seconds, and to 15 fps in standby, whatever they
+  draw. This was **verified** with a local canvas page that ran on the Frame
+  with no network involved (`bench/pages/present.html`). The compositor's
+  share needs a run with the headset worn.
+
+Targets: content p50 ≤ 25 ms (p95 ≤ 40), click to photon p50 ≤ 50 ms
+(p95 ≤ 70), 60 fps with ≤ 1% late frames, no stall over 100 ms, and adapting
+to a new link rate within 1 s.
+
+Baseline on 2026-09-28 (**verified**, home Wi-Fi, Tailscale, Balanced,
+`bench/results/2026-09-28-a3c6e5c-dirty-baseline-fixed.json`; ms p50/p95):
+
+| Scenario | Content | Input to drawn | fps drawn | Notes |
+|---|---|---|---|---|
+| test (1280×720) | 9.9 | 28.8 / 39.0 | 59 | encode 4.0, network 4.0, decode 1.3 |
+| scroll (1920×1290) | 14.7 | – | 50.4 | encode 6.7, decode 6.4 (software), 9.4 Mbit/s, 16% late |
+| type (1920×1290) | 16.7 | 51.2 / 73.2 | – | most of the input time is the Mac app reacting |
+
+What was learned (all **verified**, unless marked):
+
+- The biggest costs are encoding (4–7 ms), network (4–6 ms), and decoding
+  on the Frame. Chromium XR on the Frame decodes H.264 in **software**.
+- Wi-Fi alone stalls for 240–580 ms now and then, over Tailscale and over
+  the LAN alike. Over the LAN (`--host 192.168.1.237`) latency was no
+  better, but the Frame used about 8% less CPU, because tailscaled runs in
+  userspace there.
+- With no controller, a link that slows down queues without limit. In one
+  run, frames arrived 1.9 s late, and at worst 9.4 s late.
+- Tried, and no help, so not kept: VideoToolbox options (require hardware,
+  no frame delay, prioritise speed, a hard data-rate cap), Chromium flags
+  (`--disable-gpu-vsync`, `--disable-frame-rate-limit`,
+  `--use-angle=vulkan`), and a 120 Hz virtual display.
+- Inconclusive, so off by default: keeping the Frame's Wi-Fi awake during
+  typing (`FRAME_MAC_VIEW_WARM=40`, a tiny message every 40 ms for 5 s
+  after input). Over three interleaved runs each, input p50 went 44 → 47 ms
+  and p95 92 → 71 ms, and the ranges overlapped widely
+  (`…-ab-keepwarm.json`). In that run, and in the baseline's typing, the
+  harness typed spaces as "+" (a URL-encoding bug, since fixed), so they
+  went through the text path rather than as space keys.
+- Pointer moves now go out on an 8 ms timer, not on the page's next
+  animation frame, which an unworn Frame slows to 15–36 Hz. This is
+  **inferred** to help dragging; the benchmark has no drag scenario yet.
+- Kept: the encoder's timestamps never jump more than two frame intervals.
+  Before this, the first frame after a pause got a quarter of a second's
+  bit budget, and one P-frame reached 204 KB.
+
+## Adapting to the network
+
+`Sources/Controller.swift`, per stream, latency first:
+
+- **The gate.** The viewer acknowledges every frame as it arrives. A new
+  frame is sent only while the oldest unacknowledged one is younger than
+  the path's usual round trip, plus one frame interval, plus room for this
+  link's normal jitter (1.5 times its recent spread, 25–80 ms). So frames
+  never queue in SSH, TCP or the Wi-Fi driver. While the link is stuck, the
+  newest picture waits and goes out as soon as it moves.
+- **The bitrate.** The link counts as congested when, for two checks in a
+  row (100 ms apart), round trips grow by more than 40 ms while the stream
+  uses much of its budget, or the gate holds frames back, or a frame is
+  stuck for 100 ms. Then the bitrate drops to a bit under what actually got
+  through: at least a fifth off, and at most half. Once the link has been
+  clear for a second, it rises by 10% steps, never above the quality
+  setting's bitrate.
+- **The tier.** When the bitrate stays low, and the stream is really
+  limited by the link rather than having little to send, it steps down:
+  60 → 45 → 30 fps, then 75%, then 50% of the pixels. It goes straight to
+  the tier the bitrate supports after half a second, and steps back up one
+  tier at a time after two seconds with room to spare.
+- `FRAME_MAC_VIEW_ADAPT=0` turns it off, for comparison.
+
+Measured on the real Frame, 2026-09-28 (**verified**; interleaved A/B, off
+versus on, medians of the runs, ms):
+
+| Link | Scenario | Content p95, off → on | fps drawn, off → on | Result file |
+|---|---|---|---|---|
+| Clean Wi-Fi (3 runs each) | scroll | 32.3 → 33.6 | 57 → 56.2 | `…-ab-adapt-clean2.json` |
+| Clean Wi-Fi | test | 15 → 14.2 | 60 → 59.7 | same |
+| 50 → 3 → 50 Mbit/s at 8 s and 16 s (2 runs each) | scroll | **4670 → 72** | 45 → 42 | `…-ab-adapt-step3.json` |
+| 50 → 3 → 50 Mbit/s | test | 66 → 16 | 60 → 60 | same |
+
+- **Clean link.** On a clean link it costs nothing measurable. An earlier
+  version with a fixed gate slack lost 9 fps to Wi-Fi jitter while
+  scrolling (47 → 38 fps), and that's why the slack now follows the link's
+  jitter.
+- **Throttled link.** Without the controller, frames queued for up to 5.6 s
+  and never caught up while the link was slow (p95 1.8–5.6 s, second by
+  second). With it, in the two runs:
+
+  | | Run 1 | Run 2 |
+  |---|---|---|
+  | Worst second's p95 just after the drop | 219 ms | 428 ms |
+  | p95 back under 100 ms for 3 s in a row | after 1 s | after 4 s |
+  | Stepped down to 1440 px at 30 fps | 1.8 s after the drop | 3.5 s after |
+  | p95 per second after that, on the 3 Mbit/s link | 55–94 ms | 60–148 ms |
+
+  Sending one frame takes about 27 ms on that link by itself. After the
+  link recovered, the stream was back at full size and 60 fps in about
+  7.5 s. It steps up one tier every two seconds, on purpose, so it doesn't
+  bounce. The 1 s adaptation target was met in one run of two.
+- **A hiccup on a small stream.** In one clean run, a Wi-Fi hiccup made an
+  earlier version halve the test pattern's bitrate five times and drop it
+  to half size. That cut couldn't help: the stream only sends
+  0.47 Mbit/s. Now the controller estimates what a stream wants (captures
+  per second × average frame size). While a stream wants about half its
+  budget or less, no cut takes it below twice what it wants, so it doesn't change
+  tier.
+
 ## Checked so far (2026-09-28)
 
 - **Mac (checked by hand, macOS 26.5.2).**
@@ -191,6 +343,9 @@ may ask again after an update.
   - Whether Flathub Chromium has H.264. Chromium XR is used when it's
     installed, as it is on this Frame.
   - Latency with real, busy windows at Sharp.
+  - A benchmark run while wearing the headset. Only then does the Frame show
+    panels at full rate, so only then can the compositor's share of the
+    latency, and the frame rate you actually see, be measured.
 
 ## Limits
 

docs/mac-window-separation.md

@@ -0,0 +1,200 @@
+# Real separation of Mac windows in the headset
+
+Research and experiments on 2026-09-28 (macOS 26.5.2, Apple M5 Pro), for
+making each streamed Mac window truly independent. Builds on
+[mac-in-headset.md](mac-in-headset.md). Labels: **verified** (tried here),
+**documented**, **source** (read in someone's code) or **reported**.
+
+## What "separate" lacks today
+
+Today each window is captured on its own
+(`SCContentFilter(desktopIndependentWindow:)`), but it still lives on the
+Mac's one desktop:
+
+- **Clicks.** A click has to raise the window first, so it reorders the
+  Mac's windows, steals focus and moves the real cursor.
+- **Child windows.** A window's menus, popovers, sheets and tooltips are
+  separate windows, so they aren't in its panel. Apple documents that the
+  single-window filter leaves them out.
+- **Size.** A panel's size is tied to the window's size on the Mac screen.
+- **Hidden windows.** Minimised windows, and windows on other Spaces, can't
+  be shown live.
+
+## How the Mac draws, and where pixels can be read
+
+Apps draw with Core Animation into IOSurfaces. WindowServer's compositor
+stacks every window onto each display, including virtual ones, and sends
+the result to the screen. Pixels can be read at three points:
+
+| Where | How | Gets | Doesn't get |
+|---|---|---|---|
+| One window's content | ScreenCaptureKit `desktopIndependentWindow` (public, what we use) | Live, zero-copy, even when covered by other windows (**documented**) | Menus, popovers, sheets. It pauses while the window is minimised (**reported**) |
+| One window plus its children | `SCStreamConfiguration.includeChildWindows` (macOS 14.2+, **reported**) | Menus, popovers and sheets attached to the window | Anything the app puts outside the window's bounds |
+| A whole display | ScreenCaptureKit display filter, with apps or windows included or excluded | Everything on that display, cursor included | Only what's on that display |
+| A snapshot of any window | Private `CGSHWCaptureWindowList`, which AltTab uses for minimised windows and other Spaces (**source**: `alt-tab-macos` `PrivateApis.swift`) | Minimised windows and other Spaces | It's one still image, not a live stream |
+| Another app's layer tree | Private `CALayerHost` with a context id | A live, zero-copy picture | It only works when the other app cooperates, so it's no good for arbitrary windows (**reported**) |
+
+`CGWindowListCreateImage` and `CGDisplayStream` are obsolete in the macOS 15
+SDK (**reported** by MacPorts and JUCE). Nothing reads another app's pixels
+without the Screen Recording permission.
+
+## The idea: each window gets its own virtual display
+
+A virtual display (the private `CGVirtualDisplay`, as used by BetterDisplay,
+DeskPad and quest-display) is a compositor target with no physical screen.
+Put one streamed window alone on its own virtual display, sized to its
+panel, and capture the whole display:
+
+- **Nothing can overlap it.** A plain click at that point always lands on
+  that window, so input doesn't need raising or background-event tricks.
+- **Menus, sheets, popovers, tooltips and context menus appear on the same
+  display, so they're in the panel.** With "Displays have separate Spaces"
+  on (it is on this Mac), each display also has its own menu bar, so the
+  app's menu bar can be part of the panel.
+- **The panel's size is the display's size,** in HiDPI. Resizing the panel
+  means changing the display mode and resizing the window to fill it
+  (Accessibility API).
+- **Minimised windows and other Spaces stop being a problem,** because
+  streamed windows live on their own displays.
+- **The cursor goes where the laser points.** That display's panel is the
+  one being used, much as Vision Pro's Mac Virtual Display works. Keys from
+  the Mac's keyboard go to the window last clicked.
+
+### Verified here (no permission needed)
+
+- **Creating one.** It needs no permission or entitlement. 1920×1080 HiDPI
+  gives a 3840×2160-pixel, 60 Hz display, placed next to the built-in
+  screen, and `NSScreen.screensHaveSeparateSpaces` was true.
+- **Many at once.** 16 were created at once with no error.
+- **Placement.** `CGConfigureDisplayOrigin` far away failed with error
+  1014: macOS keeps displays edge to edge. Disabling one with the private
+  `CGSConfigureDisplayEnabled` from a command-line tool also failed with 1014.
+- **Removal is deferred while the physical screen sleeps.** With the
+  built-in display asleep, virtual displays were not removed when released,
+  or when their process exited, even from their own `.app`. A second display
+  with the same vendor, product and serial couldn't be created. All of them
+  disappeared as soon as the screen woke (`caffeinate -u`). The helper must
+  therefore:
+  - keep a fixed pool of displays and reuse them rather than create new ones;
+  - keep the screen awake while they exist, which it already does while
+    streaming.
+
+### Built: "Give each window its own display"
+
+Frame Control's helper now does this (`mac/frame-mac-view/Sources/Separate.swift`),
+and it's the default in the Tools card. Streams of this kind are named
+`separate:<window id>`.
+
+- For each window shown, it creates a HiDPI virtual display. The display is
+  sized to the window plus a menu bar, with a fresh serial each time, so a
+  display left over while the screen slept can't block a new one.
+- The window is moved onto the display and resized to fill it (Accessibility
+  API), and the whole display is captured.
+- On **Stop** the window goes back where it was, and the display is released.
+- Plain per-window capture is still there: untick "Give each window its own
+  display". Separate mode needs Accessibility (to move the window), and
+  without it, Show says so rather than quietly falling back.
+
+Verified 2026-09-28 (macOS 26.5.2, M5 Pro), using a TextEdit test document
+and the benchmark's Chrome windows:
+
+- **Separation works.** The window moved onto its own display and streamed,
+  with its first frames in 3.8 s. The display showed TextEdit's own menu bar.
+- **Child windows come along.** A context menu and the Page Setup sheet
+  opened on the same display and were in the capture.
+- **Input.** Typing through the stream reached the window. The benchmark's
+  typing scenario does this on every run.
+- **Stop.** Stop put the window back at exactly its old size (656×422), and
+  the display was removed.
+- **The helper must run a real Cocoa event loop.** Earlier, it ran only a
+  `RunLoop`. With that, AppKit never learned about new displays:
+  - `NSScreen.screens` never listed them, so placing the window always waited
+    3 s and then gave up;
+  - the display's HiDPI mode never applied, so windows were captured at 1×
+    (1280×860 instead of 1920×1290 pixels) and text was soft.
+
+  Running `NSApplication` (with no Dock icon) fixed both. The screen now
+  appears within 100 ms, and captures are at 2× (**verified** in
+  `bench/results/*-baseline.json` against `*-baseline-fixed.json`).
+- **Frame rate.** Chrome drew at 60–61 fps on the virtual display, but
+  ScreenCaptureKit delivered only about 46–53 fps from it, whereas the
+  built-in ProMotion screen gave 121 fps (**verified**). Neither a 120 Hz
+  virtual display (`FRAME_MAC_VIEW_VD_HZ=120`) nor a looser
+  `minimumFrameInterval` changed that. Cause unknown.
+- **Windows that keep their own size** (Calculator, 230×408). The window
+  moved and streamed, but stayed small in the display's corner, so the panel
+  was mostly wallpaper. Now only that corner is captured
+  (`SCStreamConfiguration.sourceRect`): the menu bar and the window, at least
+  480×360 points so menus fit. Clicks map to the cropped area. The first
+  frame arrived in 0.6–1.1 s, and on Stop the window went back and the
+  display was removed. A display's menu bar names the active app, so it
+  shows the window's own menus only once the panel has been clicked.
+- **Several windows at once** (on the Mac, with a local stand-in viewer that
+  acknowledges frames). Three and four Chrome windows, each scrolling on its
+  own display at 1920×1290 pixels, streamed at 53–56 fps and about
+  9 Mbit/s each. The helper used 20% (three) or 24% (four) of one CPU core.
+  The hardware encoder is shared: its time per frame went from 6.7 ms for
+  one stream to 7–15 ms for three and 11–25 ms for four, so each extra
+  moving window adds latency to the others. Once in four runs, before a fix,
+  one window left its display when several displays appeared at the same
+  moment, and its stream stopped: nothing on that display was changing any
+  more. The helper now checks every second and puts the window back. Three
+  further runs with four windows were clean, and every display was gone
+  afterwards (checked with `CGGetOnlineDisplayList`).
+- **Quitting.** On SIGTERM, or when Frame Control goes away, the helper ends
+  every stream first, so windows go back before their displays disappear.
+  Verified: a 900×600 window was back at 900×600 after SIGTERM. Closing a
+  viewer 0.05–1.5 s into startup left the window at its old size and no
+  extra display.
+- **A consent prompt.** macOS 26 asks whether to let ScreenCaptureKit apps
+  "bypass the system private window picker". The prompt appeared *on the
+  virtual display*, so it would show up inside the headset panel. Allow it
+  once on the Mac.
+
+### Still to check
+
+1. That a context menu opens over the text being clicked, not just somewhere
+   on the display. This needs a retest after the consent prompt above is
+   allowed.
+2. Stage Manager, which is reported to undo Accessibility resizes.
+3. Where the Dock and the cursor go on a virtual display.
+4. Closing the lid with virtual displays attached (clamshell).
+5. Many moving windows at once in the headset: whether to lower the bitrate
+   or frame rate of panels you aren't using, so the one you are using keeps
+   the encoder to itself.
+6. All of it in the headset, with the laser.
+
+## Input without disturbing the Mac (a finer option)
+
+For windows that stay on the Mac's own screen, input can go to a window
+behind others without raising it:
+
+- **Background click.** `CGEventPostToPid` with the CGEvent fields 91 and
+  92, which say which window a click is for (`kCGMouseEventWindowUnderMousePointer`
+  and `...ThatCanHandleThisEvent`), plus a window-relative location
+  (**reported**, reverse-engineered, needs testing).
+- **Focus without raise.** yabai makes a window key without raising it by
+  posting event records with the private `SLPSPostEventRecordTo` (**source**:
+  `yabai/src/window_manager.c`).
+- **Known failures.**
+  - Chromium and Electron ignore background clicks unless they're built
+    with the 2026 `acceptsFirstMouse` fix (electron/electron#54493).
+  - Games and canvas apps often need real activation.
+  - Password fields (Secure Event Input) drop synthetic keys.
+  - IME composition, as for Chinese or Japanese input, isn't reliable.
+
+With a virtual display per window, most of this isn't needed. It's worth
+having for hovering over one panel while typing in another.
+
+## Encoding and transport
+
+- **Codec.** Keep H.264 4:2:0. Apple's High Performance Screen Sharing uses
+  4:4:4 over two virtual displays (**documented**), but the Frame's Chromium
+  decodes H.264 in software, and no 4:4:4 decode path is confirmed there.
+- **Sharper static text.** When a panel has been still for a moment, send a
+  high-quality refresh, either a keyframe at low quantisation or a lossless
+  WebP overlay, so static text is sharp. Moving content stays as video.
+- **Transport.** Keep WebSocket over SSH for now, as it works everywhere.
+  WebRTC (UDP, congestion control) is the proven step up for Wi-Fi.
+  WebTransport over QUIC is promising, but its server side on macOS is
+  unverified.