* Live view: a Desktop view that stays still, and Control to tap on the Frame
The live view gets a second source and a way to use the Frame from it:
- Desktop: the app panel in use in the headset, streamed from its own window
(x11grab of gamescope's redirected window), so it doesn't move as the
wearer looks around. A picker shows any other panel, view only.
- Control: on the Desktop view a tap or click lands exactly where you put it;
drag is a mouse drag, press and hold right-clicks, two fingers scroll, and
on a computer the mouse, wheel and keyboard work directly. On the headset
view the view is a trackpad. A text field and key row type from a phone.
Input goes through gamescope's own EIS socket (the way Steam feeds Remote
Play input) with the libei already on the image: ui/frame_touch.py, over
the same long-lived ssh machinery as the keyboard agent, nothing to
install. It reaches the panel that has focus on either X display, which
the KDE Connect route can't. Verified on the Frame and from the iPhone app
in the Simulator.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Control: fixes from review
- Keys held on the Frame are released with buttons when Control stops or
the view loses focus; keys for the Frame no longer trigger Frame Control's
own shortcuts.
- Taps only act when the picture on screen is the panel in use; positions,
presses, keys, text and scrolls name their panel (display and window: ids
repeat across :0 and :1, told apart by pid), and the Frame drops them if
focus has moved on. Releases always go.
- While connecting, a tap keeps its position; on an error only releases wait
and retries back off; trimming a long queue never drops a release.
- Lifting one of two scrolling fingers ends the scroll; a cancelled touch
isn't a tap; clicks and holds on the bars around the picture do nothing.
- A capture loop from before a Live restart can't stop the new video.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Control: close the targeting gaps from the second review
- The focused panel's display comes from GAMESCOPE_FOCUS_DISPLAY (gamescope
packs ":1" into the first value), so a window id repeated across :0 and :1
can't be mistaken; the pid is only the fallback.
- Presses, keys, text and scrolls read focus afresh on the Frame; only moves
use a reading up to a second old.
- A gesture remembers the panel it started on and does nothing more if that
stops being the one in use; a press with no panel to aim at isn't sent.
- Opening a screenshot clears the panel Control would act on; switching to
another app releases held keys and buttons.
- Trimming keeps a click with its position; the error backoff holds for new
input too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Control: fixes from the SWE-2 Max review
- The Frame side tracks keys as well as buttons and lets go of both when the
session ends.
- A stale tap tells the page, which re-reads the panels at once.
- A paused input device waits instead of ending the session; only a
disconnect does. An OS error on one event skips it.
- Presses check focus with two property reads and do the full lookup only
when it changed.
- Writes to an agent's stdin are serialized, so two devices sending at once
can't tear a line (the keyboard agent too).
- Connecting gives up with a message after 15 s instead of hanging on
"Connecting…"; text goes in 100-character pieces so releases don't wait
behind a long paste; a cancelled mouse gesture releases what's held.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Control: stale clears, pauses reconverge, pastes split per request
- The Frame says it has caught up as soon as an aimed event lands after a
stale one, so the page stops re-reading the panels.
- After a device pause it lets go of everything it holds (releases that
arrived while paused were dropped), and waits for the device once per
batch, not once per event.
- The quick focus check no longer freshens the panel geometry's age.
- Each request carries at most about 100 characters of text.
- Turning Control off while it connects doesn't report an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* frame_touch: build the socket path on the Frame, so Windows can import it for tests
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* frame_touch: any event that goes through clears the stale flag
A trackpad move names no panel, so waiting for an aimed event could leave
the page re-reading panels for the rest of the session; a release still
aimed at the old panel doesn't count.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Plugged into the Mac, the Frame is a USB network device ("Steam Frame",
usb0 at ~0.9 ms). The tunnel uses it when the Frame's usb0 answers,
with the usual host key; otherwise the normal path. Interleaved runs:
content p50 7 vs 10 ms, click to drawn 17 vs 27 ms, scroll p95 23 vs
31-37 ms. FRAME_MACVIEW_USB=0 turns it off; the card says "over USB-C".
- Bench: --usb, and the route is recorded per run.
- Docs: Steam's own streaming (no SteamVR host on macOS; Remote Play pairs
but streams games, not windows; test blocked); the Frame's USB network;
the 2026-09-28 health-check boot-loop recurrence.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The keyboard and trackpad no longer fetch KDE Connect from Valve's package
repository on the Frame. The desktop apps and the iPhone app's Frame bundle
carry Valve's arm64 build of kdeconnect 24.02.2-1 and the five libraries it
links (kcontacts, kpeople, modemmanager-qt, pulseaudio-qt, libfakekey),
pinned by SHA-256 in frame/kdeconnect/packages.json and downloaded at build
time from the kdeconnect-frame-24.02.2-1 release, which also holds Valve's
complete source package for each.
On first use the computer copies them over its SSH connection (the iPhone
bundle already has them on the Frame); the agent checks each SHA-256,
unpacks them and stamps which build it is, so later starts copy nothing.
No internet on the Frame, 3.6 MB instead of 8 MB, 18 MB unpacked instead of
82 MB (ModemManager and friends were packaging-only dependencies).
GPL/LGPL compliance: frame/kdeconnect/NOTICE.md names each exact version,
licence and source; per-project licence texts in frame/kdeconnect/LICENSES;
THIRD_PARTY_NOTICES.md; an About and licences dialog in the app.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Brings in #4, #5, #8 (fbl100's verified Remmina/VNC mirror), APK
alternatives and the website. docs/streaming.md: Mac in the headset stays
the recommendation (now verified on the Frame); Remmina keeps #8's verified
evidence as the whole-screen fallback. docs/mac-in-headset.md cites #8's
lag finding.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Per-frame timing on the Mac's clock (capture, encode, network, decode,
draw), viewer clock sync and reports, input echo, /stats and a HUD.
- scripts/macview-bench.py: repeatable runs on the real Frame, a shaping
relay (no sudo), interleaved A/B between agent settings; results in
bench/results/.
- Adaptive controller: ack-based send gate with jitter-aware slack, AIMD
bitrate that knows when a stream is app-limited, fps then size tiers.
On a 50->3->50 Mbit/s step, scroll p95 went from 4.7 s to 72 ms; no cost
on a clean link.
- Separate mode: real AppKit event loop (HiDPI and NSScreen now work),
cropped capture for fixed-size windows, windows kept on their display,
graceful quit restores windows; stop/start races fixed.
- Encoder timeline clamp (no oversized frame after a pause).
- Frame Control shows each live stream's fps, delay, bitrate and tier.
Reviewed by GPT-6 Astra xhigh (read-only), 7 rounds; findings fixed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Home → Keyboard and trackpad, in every version of Frame Control (Mac,
Windows, Linux, iPhone, iPad) with nothing to install on the device in
your hand. On a phone: a trackpad (drag, tap, two-finger scroll and
right-click) and a field that types on the Frame. On a computer: click
the pad to pass the mouse and keyboard through; Esc to stop.
First-party route: ui/frame_input_agent.py runs on the Frame and talks
KDE Connect's LAN protocol (v7) to kdeconnectd as if it were a phone.
KDE Connect isn't on the image, but Valve's package repository has it;
the agent fetches it and four libraries into ~ (no root, survives
updates), starts it, pairs by itself (accepting over D-Bus), and stops
it again when the last device disconnects. Each device has its own
identity; a stuck KDE Connect is restarted once.
Verified against the real Frame (SteamOS 0.4.1): first-time install,
pairing, pointer moves from the Mac's server and the iPhone app
(Simulator), Mac and iPhone at once, two installs at once, a frozen
daemon replaced, and the daemon stopping when the app quits.
Reviewed by GPT-6 Astra (xhigh) over seven rounds; all findings fixed
except per-event delivery acknowledgement (documented known limit).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Steam's own idle timer (60 min on AC, 15 on battery) suspends the Frame,
and SSH work doesn't count as activity. scripts/keep-awake.sh on sets both
timers to Never through Steam's DevTools (reusing ui/frame_steam.py) and
holds a logind sleep inhibitor as a user unit; off releases the inhibitor
and restores the saved timers. Findings recorded in how-the-frame-works.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Frame Control can now show any Mac window, or a whole screen, as its own
SteamVR panel on the Steam Frame (Tools -> Mac in the headset, macOS only).
Place it with the SteamVR dashboard; the laser clicks and scrolls, and the
Mac's own keyboard types.
- mac/frame-mac-view (Swift, no dependencies): ScreenCaptureKit capture per
window or display, VideoToolbox H.264 with low-latency rate control (JPEG
fallback), a loopback HTTP/WebSocket server, CGEvent/AX input playback,
and a display-awake assertion while anyone watches.
- ui/frame_macview.py: starts the agent, runs an ssh -R tunnel with a
supervisor that reopens it on the same port, and launches a Chromium app
window per stream on gamescope's :0, tagged with STEAM_GAME for its own panel.
- Frame Control's key never leaves the Mac: viewers get single-use,
per-source tickets and reconnect keys that Stop revokes.
- ui/mac-view.html: WebCodecs decode, keyframe recovery, pointer/wheel/keys back.
- Bundled in the Mac app build; tests/test_macview.py builds and drives the
agent on macOS.
Verified on the Frame (build 20260925.6191901) with the test pattern: panel
in about 1.5 s, about 60 fps, Mac-to-window about 11-17 ms, tunnel recovery
in 4 s. Laser input and real window capture still need a person in the headset.
Also commits the other thread's first-party rule (steam-frame skill) and
the first-party options table in docs/streaming.md.
Reviewed by GPT-6 Astra (xhigh, read-only) over six rounds; all findings fixed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- docs/recovery-and-images.md: where Valve's Frame images are (not linked from
the SteamOS download page), file names, sizes and our checksums, the GPT
layout with exact start sectors, what's in rootfs-A (btrfs, SteamOS 0.3.0
build 20260922.5152327, users, sudo and sshd config), extracting it, running
it without the headset, and Valve/Collabora's Holo Core aarch64 preview.
- how-the-frame-works.md: correct the recovery image file names; add verified
facts on the SSH server, tools on the image (no adb), Lepton instances as
podman containers, going off the network when asleep, and the battery
reading at full charge.
- ssh.md: pairing from an iPhone and why devkit RSA pairing doesn't fit it.
- open-questions.md, README.md and the steam-frame skill point to the new pages.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Recovery menu (AUX + Power), USB and EDL re-imaging, from Valve's docs.
- Verified cause and fix of a boot loop: steamvr-health-check wipes
~/.local/share/Steam after repeated SteamVR start failures, which then
repeat while Steam re-downloads, until the Frame reboots.
- T3 Code desktop running natively as a panel (from an earlier session).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Live in the headset view now streams video instead of polling stereo
screenshots (~2 fps). SteamVR's steamvr-v4l2cam.service mirrors the headset
view into /dev/video99; ffmpeg on the Frame encodes it with x264 (720p30 by
default, AUD + repeated SPS/PPS), /api/stream relays the raw H.264 over SSH,
and the page splits it on access unit delimiters and decodes it with
WebCodecs into the existing viewer. Capture still takes a stereo still; the
desktop panel keeps capture polling, and the page falls back to it if the
video can't start.
The remote ffmpeg runs under a shell that kills it when the SSH channel
closes, stderr goes to a temp file, and a 10 s stall ends the stream. One
stream at a time; a new one supersedes the last.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Screenshots: list the Frame's Steam screenshots, open them in the
viewer, and save new ones to ~/Pictures/SteamFrame. Ids are validated
before any shell, and copies land atomically.
- Tailscale: scripts/tailscale-on-frame.sh installs a userspace tailscaled
as a lingering systemd --user service with no sudo, SHA-256 checked, safe
to re-run, with --uninstall. docs/tailscale.md covers setup and warns that
in userspace mode every Frame port, including loopback-only DevTools and
ADB, is reachable from the tailnet.
- push-vr-video.sh: filenames starting with "-" are safe, symlinks are
followed, and a real Videos\VR directory triggers a warning.
- Tests cover the screenshot routes (19 total).
Docs keep placeholder addresses for the headset and tailnet.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- app: startup shell, python and ssh probes run asynchronously so a slow
shell profile can't freeze the window; PATH comes from the user's real
login shell and a failed lookup isn't cached; a server that never
answers is killed; the setup offer runs once per launch, only after the
UI loads, and decides from HostName alone; connect.sh is started through
`env ... zsh` so it works whatever the login shell is.
- server: volume validates the level before muting or changing anything.
- Steam: null-safe install-manager fields, http.client errors caught in
store ratings, price fallback when a sale has no final price.
- tests: server output kept for diagnosis, any startup error retried, and
captures asserted non-cacheable.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Package the Frame Control web UI as an installable Electron Mac app and
bring in the tooling built alongside it.
- app/: Electron wrapper that starts ui/server.py on a free loopback port,
hardened window (sandbox, no navigation, runAsNode fuse off), login-shell
PATH so Homebrew tools work from Finder, first-run offer to run
connect.sh, ad-hoc signed DMG/zip via electron-builder.
- ui/: headset view (OpenVR screenshots), device status, library, Steam
"Get games" (owned games, install, store search), Android apps as
persistent Lepton instances with a rated F-Droid catalogue and a private
compatibility database, Android display controls over ADB, file and
clipboard transfer, Flatpaks, remote and power actions.
- apk-catalog/, compat-db/, frame/: catalogue build pipeline, Lakebed
capsule for compatibility reports, Frame-side launchers.
- tests/ and CI: server guard and validation tests plus Steam helper tests,
run on Python 3.9 with script and app syntax checks.
- Docs: README leads with the Mac app; new Android, panels, Steam games and
field-notes docs; security notes on LAN-exposed ADB ports.
Screenshot values for the headset's IP and Wi-Fi name are placeholders.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
