saphid--frame-control/docs/mac-window-separation.md

Real separation of Mac windows in the headset

Research and experiments on 2026-09-28 (macOS 26.5.2, Apple M5 Pro), for making each streamed Mac window truly independent. Builds on mac-in-headset.md. Labels: verified (tried here), documented, source (read in someone's code) or reported.

What "separate" lacks today

Today each window is captured on its own (SCContentFilter(desktopIndependentWindow:)), but it still lives on the Mac's one desktop:

• Clicks. A click has to raise the window first, so it reorders the Mac's windows, steals focus and moves the real cursor.
• Child windows. A window's menus, popovers, sheets and tooltips are separate windows, so they aren't in its panel. Apple documents that the single-window filter leaves them out.
• Size. A panel's size is tied to the window's size on the Mac screen.
• Hidden windows. Minimised windows, and windows on other Spaces, can't be shown live.

How the Mac draws, and where pixels can be read

Apps draw with Core Animation into IOSurfaces. WindowServer's compositor stacks every window onto each display, including virtual ones, and sends the result to the screen. Pixels can be read at three points:

Where	How	Gets	Doesn't get
One window's content	ScreenCaptureKit desktopIndependentWindow (public, what we use)	Live, zero-copy, even when covered by other windows (documented)	Menus, popovers, sheets. It pauses while the window is minimised (reported)
One window plus its children	SCStreamConfiguration.includeChildWindows (macOS 14.2+, reported)	Menus, popovers and sheets attached to the window	Anything the app puts outside the window's bounds
A whole display	ScreenCaptureKit display filter, with apps or windows included or excluded	Everything on that display, cursor included	Only what's on that display
A snapshot of any window	Private CGSHWCaptureWindowList, which AltTab uses for minimised windows and other Spaces (source: alt-tab-macos PrivateApis.swift)	Minimised windows and other Spaces	It's one still image, not a live stream
Another app's layer tree	Private CALayerHost with a context id	A live, zero-copy picture	It only works when the other app cooperates, so it's no good for arbitrary windows (reported)

CGWindowListCreateImage and CGDisplayStream are obsolete in the macOS 15 SDK (reported by MacPorts and JUCE). Nothing reads another app's pixels without the Screen Recording permission.

The idea: each window gets its own virtual display

A virtual display (the private CGVirtualDisplay, as used by BetterDisplay, DeskPad and quest-display) is a compositor target with no physical screen. Put one streamed window alone on its own virtual display, sized to its panel, and capture the whole display:

• Nothing can overlap it. A plain click at that point always lands on that window, so input doesn't need raising or background-event tricks.
• Menus, sheets, popovers, tooltips and context menus appear on the same display, so they're in the panel. With "Displays have separate Spaces" on (it is on this Mac), each display also has its own menu bar, so the app's menu bar can be part of the panel.
• The panel's size is the display's size, in HiDPI. Resizing the panel means changing the display mode and resizing the window to fill it (Accessibility API).
• Minimised windows and other Spaces stop being a problem, because streamed windows live on their own displays.
• The cursor goes where the laser points. That display's panel is the one being used, much as Vision Pro's Mac Virtual Display works. Keys from the Mac's keyboard go to the window last clicked.

Verified here (no permission needed)

• Creating one. It needs no permission or entitlement. 1920×1080 HiDPI gives a 3840×2160-pixel, 60 Hz display, placed next to the built-in screen, and NSScreen.screensHaveSeparateSpaces was true.
• Many at once. 16 were created at once with no error.
• Placement. CGConfigureDisplayOrigin far away failed with error 1014: macOS keeps displays edge to edge. Disabling one with the private CGSConfigureDisplayEnabled from a command-line tool also failed with 1014.
• Removal is deferred while the physical screen sleeps. With the built-in display asleep, virtual displays were not removed when released, or when their process exited, even from their own .app. A second display with the same vendor, product and serial couldn't be created. All of them disappeared as soon as the screen woke (caffeinate -u). The helper must therefore:
  • keep a fixed pool of displays and reuse them rather than create new ones;
  • keep the screen awake while they exist, which it already does while streaming.

Built: "Give each window its own display"

Frame Control's helper now does this (mac/frame-mac-view/Sources/Separate.swift), and it's the default in the Tools card. Streams of this kind are named separate:<window id>.

• For each window shown, it creates a HiDPI virtual display. The display is sized to the window plus a menu bar, with a fresh serial each time, so a display left over while the screen slept can't block a new one.
• The window is moved onto the display and resized to fill it (Accessibility API), and the whole display is captured.
• On Stop the window goes back where it was, and the display is released.
• Plain per-window capture is still there: untick "Give each window its own display". Separate mode needs Accessibility (to move the window), and without it, Show says so rather than quietly falling back.

Verified 2026-09-28 (macOS 26.5.2, M5 Pro), using a TextEdit test document and the benchmark's Chrome windows:

• Separation works. The window moved onto its own display and streamed, with its first frames in 3.8 s. The display showed TextEdit's own menu bar.

• Child windows come along. A context menu and the Page Setup sheet opened on the same display and were in the capture.

• Input. Typing through the stream reached the window. The benchmark's typing scenario does this on every run.

• Stop. Stop put the window back at exactly its old size (656×422), and the display was removed.

• The helper must run a real Cocoa event loop. Earlier, it ran only a RunLoop. With that, AppKit never learned about new displays:

  • NSScreen.screens never listed them, so placing the window always waited 3 s and then gave up;
  • the display's HiDPI mode never applied, so windows were captured at 1× (1280×860 instead of 1920×1290 pixels) and text was soft.

  Running NSApplication (with no Dock icon) fixed both. The screen now appears within 100 ms, and captures are at 2× (verified in bench/results/*-baseline.json against *-baseline-fixed.json).

• Frame rate. Chrome drew at 60–61 fps on the virtual display, but ScreenCaptureKit delivered only about 46–53 fps from it, whereas the built-in ProMotion screen gave 121 fps (verified). Neither a 120 Hz virtual display (FRAME_MAC_VIEW_VD_HZ=120) nor a looser minimumFrameInterval changed that. Cause unknown.

• Windows that keep their own size (Calculator, 230×408). The window moved and streamed, but stayed small in the display's corner, so the panel was mostly wallpaper. Now only that corner is captured (SCStreamConfiguration.sourceRect): the menu bar and the window, at least 480×360 points so menus fit. Clicks map to the cropped area. The first frame arrived in 0.6–1.1 s, and on Stop the window went back and the display was removed. A display's menu bar names the active app, so it shows the window's own menus only once the panel has been clicked.

• Several windows at once (on the Mac, with a local stand-in viewer that acknowledges frames). Three and four Chrome windows, each scrolling on its own display at 1920×1290 pixels, streamed at 53–56 fps and about 9 Mbit/s each. The helper used 20% (three) or 24% (four) of one CPU core. The hardware encoder is shared: its time per frame went from 6.7 ms for one stream to 7–15 ms for three and 11–25 ms for four, so each extra moving window adds latency to the others. Once in four runs, before a fix, one window left its display when several displays appeared at the same moment, and its stream stopped: nothing on that display was changing any more. The helper now checks every second and puts the window back. Three further runs with four windows were clean, and every display was gone afterwards (checked with CGGetOnlineDisplayList).

• Quitting. On SIGTERM, or when Frame Control goes away, the helper ends every stream first, so windows go back before their displays disappear. Verified: a 900×600 window was back at 900×600 after SIGTERM. Closing a viewer 0.05–1.5 s into startup left the window at its old size and no extra display.

• A consent prompt. macOS 26 asks whether to let ScreenCaptureKit apps "bypass the system private window picker". The prompt appeared on the virtual display, so it would show up inside the headset panel. Allow it once on the Mac.

Still to check

1. That a context menu opens over the text being clicked, not just somewhere on the display. This needs a retest after the consent prompt above is allowed.
2. Stage Manager, which is reported to undo Accessibility resizes.
3. Where the Dock and the cursor go on a virtual display.
4. Closing the lid with virtual displays attached (clamshell).
5. Many moving windows at once in the headset: whether to lower the bitrate or frame rate of panels you aren't using, so the one you are using keeps the encoder to itself.
6. All of it in the headset, with the laser.

Input without disturbing the Mac (a finer option)

For windows that stay on the Mac's own screen, input can go to a window behind others without raising it:

• Background click. CGEventPostToPid with the CGEvent fields 91 and 92, which say which window a click is for (kCGMouseEventWindowUnderMousePointer and ...ThatCanHandleThisEvent), plus a window-relative location (reported, reverse-engineered, needs testing).
• Focus without raise. yabai makes a window key without raising it by posting event records with the private SLPSPostEventRecordTo (source: yabai/src/window_manager.c).
• Known failures.
  • Chromium and Electron ignore background clicks unless they're built with the 2026 acceptsFirstMouse fix (electron/electron#54493).
  • Games and canvas apps often need real activation.
  • Password fields (Secure Event Input) drop synthetic keys.
  • IME composition, as for Chinese or Japanese input, isn't reliable.

With a virtual display per window, most of this isn't needed. It's worth having for hovering over one panel while typing in another.

Encoding and transport

• Codec. Keep H.264 4:2:0. Apple's High Performance Screen Sharing uses 4:4:4 over two virtual displays (documented), but the Frame's Chromium decodes H.264 in software, and no 4:4:4 decode path is confirmed there.
• Sharper static text. When a panel has been still for a moment, send a high-quality refresh, either a keyframe at low quantisation or a lossless WebP overlay, so static text is sharp. Moving content stays as video.
• Transport. Keep WebSocket over SSH for now, as it works everywhere. WebRTC (UDP, congestion control) is the proven step up for Wi-Fi. WebTransport over QUIC is promising, but its server side on macOS is unverified.