Alex SouthwellandClaude Opus 5.5 e702adbd77 Live view: a Desktop view that stays still, and Control to tap on the Frame (#46)
* Live view: a Desktop view that stays still, and Control to tap on the Frame

The live view gets a second source and a way to use the Frame from it:

- Desktop: the app panel in use in the headset, streamed from its own window
  (x11grab of gamescope's redirected window), so it doesn't move as the
  wearer looks around. A picker shows any other panel, view only.
- Control: on the Desktop view a tap or click lands exactly where you put it;
  drag is a mouse drag, press and hold right-clicks, two fingers scroll, and
  on a computer the mouse, wheel and keyboard work directly. On the headset
  view the view is a trackpad. A text field and key row type from a phone.

Input goes through gamescope's own EIS socket (the way Steam feeds Remote
Play input) with the libei already on the image: ui/frame_touch.py, over
the same long-lived ssh machinery as the keyboard agent, nothing to
install. It reaches the panel that has focus on either X display, which
the KDE Connect route can't. Verified on the Frame and from the iPhone app
in the Simulator.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from review

- Keys held on the Frame are released with buttons when Control stops or
  the view loses focus; keys for the Frame no longer trigger Frame Control's
  own shortcuts.
- Taps only act when the picture on screen is the panel in use; positions,
  presses, keys, text and scrolls name their panel (display and window: ids
  repeat across :0 and :1, told apart by pid), and the Frame drops them if
  focus has moved on. Releases always go.
- While connecting, a tap keeps its position; on an error only releases wait
  and retries back off; trimming a long queue never drops a release.
- Lifting one of two scrolling fingers ends the scroll; a cancelled touch
  isn't a tap; clicks and holds on the bars around the picture do nothing.
- A capture loop from before a Live restart can't stop the new video.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: close the targeting gaps from the second review

- The focused panel's display comes from GAMESCOPE_FOCUS_DISPLAY (gamescope
  packs ":1" into the first value), so a window id repeated across :0 and :1
  can't be mistaken; the pid is only the fallback.
- Presses, keys, text and scrolls read focus afresh on the Frame; only moves
  use a reading up to a second old.
- A gesture remembers the panel it started on and does nothing more if that
  stops being the one in use; a press with no panel to aim at isn't sent.
- Opening a screenshot clears the panel Control would act on; switching to
  another app releases held keys and buttons.
- Trimming keeps a click with its position; the error backoff holds for new
  input too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from the SWE-2 Max review

- The Frame side tracks keys as well as buttons and lets go of both when the
  session ends.
- A stale tap tells the page, which re-reads the panels at once.
- A paused input device waits instead of ending the session; only a
  disconnect does. An OS error on one event skips it.
- Presses check focus with two property reads and do the full lookup only
  when it changed.
- Writes to an agent's stdin are serialized, so two devices sending at once
  can't tear a line (the keyboard agent too).
- Connecting gives up with a message after 15 s instead of hanging on
  "Connecting…"; text goes in 100-character pieces so releases don't wait
  behind a long paste; a cancelled mouse gesture releases what's held.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: stale clears, pauses reconverge, pastes split per request

- The Frame says it has caught up as soon as an aimed event lands after a
  stale one, so the page stops re-reading the panels.
- After a device pause it lets go of everything it holds (releases that
  arrived while paused were dropped), and waits for the device once per
  batch, not once per event.
- The quick focus check no longer freshens the panel geometry's age.
- Each request carries at most about 100 characters of text.
- Turning Control off while it connects doesn't report an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: build the socket path on the Frame, so Windows can import it for tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: any event that goes through clears the stale flag

A trackpad move names no panel, so waiting for an aimed event could leave
the page re-reading panels for the rest of the session; a release still
aimed at the old panel doesn't count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:13:21 +10:00

Frame Control icon

Frame Control

Manage your Valve Steam Frame from your computer.
See what the headset sees, install games and Android apps, move files and text across, and check battery and status, all over SSH.

Latest release Platforms Checks License: MIT

Website · Download · Trailer · Features · Set up the headset · Feedback · Docs


Frame Control's Games tab: installed games, sideloaded titles, and your Steam library with Frame ratings

Watch the Frame Control trailer

The trailer: 66 seconds, with sound. Downloads the MP4 from the trailer release.

Unofficial hobby project, not affiliated with Valve. Free and open source.


Features

👓 Headset view and Desktop
Live video of what the lenses show, or of the app panel in use, flat and still however the wearer looks around. Turn on Control and tap or click right on it to use the Frame from your phone or computer.

🔋 Battery and status
Charge, charging watts and time left, storage, memory, temperature, Wi-Fi, and what's running.

🎮 Steam games
Everything you own with its Steam Frame rating. Install onto the headset with live progress, and search the store.

🤖 Android apps
About 4,500 F-Droid apps rated for the Frame. One click installs each as its own app in your Steam library.

📁 Files, games and clipboard
Drag files onto the window to send them. Drop a game's .zip, folder or .exe to add it to the Steam library, with Proton or the Linux runtime picked for you. Send text or your clipboard straight to the headset's desktop.

📸 Screenshots
Browse the shots you take in the headset and save them to your Pictures folder.

⌨️ Keyboard and trackpad
Type and point in the Frame's apps from your computer or phone, through KDE Connect, which Frame Control brings along and sets up on the Frame. Nothing else to install, anywhere.

🧩 Flatpaks and display
Install desktop apps like Moonlight or VLC, and set each Android app's resolution and text size.

⚡ One-click tools
SSH, SFTP, Steam Link, remote desktop, volume, sleep, restart and shut down.

Nothing is installed on the Frame for any of this: the app uses what SteamOS already ships (sideloading a game copies Valve's own devkit scripts to ~/devkit-utils, as Valve's Devkit Client does). How each feature works.

Install

Download Needs
macOS (Apple Silicon) Frame-Control-mac-arm64.dmg Nothing extra
Windows 10 / 11 (x64) Frame-Control-Setup-x64.exe · portable .zip Nothing extra
Linux (x64) AppImage · .deb ssh (most desktops have it)
Linux (arm64) AppImage · .deb ssh, and adb for Android apps (sudo apt install adb)

iPhone and iPad: the same features from your phone, with nothing to install on a computer. Build it from ios/ in Xcode; see docs/iphone.md.

The app brings its own Python and adb; SSH is built into macOS and Windows. From 0.4 it updates itself: when a new version is published, a banner offers Update and restart. It sends anonymous usage statistics, which you can turn off. Sharing compatibility results and error details is opt-in. See docs/privacy.md. Google doesn't publish adb for arm64 Linux, so that build uses your distribution's. If you already have adb, the app uses yours.

macOS: the app isn't notarized

There's no paid Apple developer account behind it, so macOS says the app is damaged or can't be checked. Drag it to Applications, then clear the download quarantine once:

xattr -dr com.apple.quarantine "/Applications/Frame Control.app"

The first time, macOS also asks to allow local network access (for SSH) and control of Terminal (for the password prompts).

Windows: SmartScreen warning

The installer isn't code-signed, so Windows SmartScreen may say it protected your PC. Choose More info → Run anyway. The portable .zip avoids the installer: unzip it anywhere and run Frame Control.exe.

Linux: running the AppImage
chmod +x Frame-Control-linux-*.AppImage && ./Frame-Control-linux-*.AppImage

If it complains about FUSE, install libfuse2 (Ubuntu 24.04+: libfuse2t64), or run it with --appimage-extract-and-run.

Set up the headset

You type one password on the headset, once. Everything else happens on your computer.

  1. On the Frame: Steam Settings → System → Enable Developer Mode, then in the Developer section, Set User Password. Pick something short: you'll type it once more on your computer and then never again.

  2. On your computer: open Frame Control. It offers to Set Up Connection, which finds the headset, creates an SSH key, and asks for that password once in a terminal window. If it can't find the Frame, type the IP address from the Frame's Quick Settings.

    Before asking for the password it tries Valve's SteamOS devkit pairing: in the headset, open Steam Settings → Developer → Pair new host and approve the request, and no password is needed. (The service and the pairing-mode step are verified on a Frame; the approval itself isn't yet. See SSH.)

  3. That's it. The app now reaches the headset whenever it's awake and on the same network. For anywhere else, see Tailscale.

What it changes: only what you click. Installs go to your user account on the Frame (--user Flatpaks, Lepton instances, Steam downloads, sideloaded games in ~/devkit-game), and nothing needs sudo except the power buttons. On your computer it adds a Host frame entry to ~/.ssh/config and keys at ~/.ssh/id_ed25519_frame and ~/.ssh/id_rsa_frame_devkit (the pairing service only takes RSA keys).

Feedback

This is a first public test, so reports are really useful, especially from Windows and Linux. The quickest way is Report a problem in the app (the warning-sign button at the top, or Help → Report a Problem…). It adds diagnostics with personal details removed, shows you exactly what's included, and sends it privately to the maintainer; nothing is published. Without the app, use the feedback form. Please include:

  • what you tried and what happened
  • your computer's OS and your SteamOS build (Steam Settings → System)
  • the server log: Frame → Show Server Log in the app

Issues and PRs opened directly on GitHub by new contributors are auto-closed until a maintainer approves them; see CONTRIBUTING.md.

Going further

This repo also holds the scripts behind the app and field notes on how the Frame's software fits together, all checked against a real headset and labelled verified or inferred.

Frame Control in detail Every feature, how it works, per-platform notes, building
Scripts and headset setup The command-line helpers, minimum typing, streaming options, floating panels
How the Frame works SteamVR → gamescope → Plasma, verified facts, debugging
Android apps (Lepton) Sideloading, the rated F-Droid catalogue, per-app instances
Sideloading Linux and Windows games A .zip, folder or .exe as a Steam Devkit Game, runtime detection
Install links for websites frame-control://install links and manifests, the rules, a button to paste
Steam games · VR video · WebXR in Chromium Installing and buying, watching VR180/360, the Chromium build
Mac in the headset Mac windows and screens as panels in the Frame, with laser and keyboard input
VR mods and custom songs Per-game feasibility, real-Frame results and blockers; no installer yet
SSH · Streaming · Files · Panels · Tailscale Topic notes
Frame Control for iPhone The iPhone and iPad app, how it runs the server on the Frame, pairing
Recovery and OS images Where to download the Frame's OS, what's inside, testing without the headset
AI agents and assistant Key-free MCP tools, human approvals, and an opt-in assistant panel
Testing Unit tests, end-to-end tests against a fake Frame in Docker, and the headset smoke test
Open questions What's still unchecked
Security notes
  • With Developer Mode on, sshd, ADB and xrdp are all reachable on your LAN. Each running Lepton (Android) instance opens its own ADB port in 5555–5599, listening on 0.0.0.0 rather than only loopback. This was seen on the device on 2026-09-25, so anyone on the network can reach it. Use trusted networks only, and turn Developer Mode off when you don't need it.
  • Frame Control reaches ADB and the Steam client's DevTools port (Frame loopback 127.0.0.1:8080) only through SSH tunnels. The compatibility database key (maintainer-only) is never written to the repo.
  • steamos has sudo, protected by the same Developer Mode password. Once you've switched to key auth, a short password still protects sudo and RDP, so pick one that isn't trivially guessable.
  • Don't port-forward 22, 3389, or 5555–5599 from your router. For remote access, use Tailscale: scripts/tailscale-on-frame.sh (no sudo). In its userspace mode every Frame port is reachable from your tailnet, including Steam's DevTools on loopback 8080; see docs/tailscale.md.

Development

python3 -m unittest discover -s tests   # server tests; no headset needed
scripts/e2e.sh                          # end-to-end against a fake Frame (Linux with Docker)
cd app && npm install && npm start      # run the app from the checkout

The server is Python stdlib only; the app is Electron. GitHub Actions runs the tests on macOS, Windows and Linux, and a v* tag builds all three installers into a draft release, which reaches users once published. See building and releasing.

License

MIT. The apps also ship other people's software under its own licence, notably KDE Connect (GPL) for the keyboard and trackpad; see THIRD_PARTY_NOTICES.md. Steam, Steam Frame and SteamVR are trademarks of Valve Corporation. This project isn't affiliated with or endorsed by Valve.

S
Description
GitHub pull mirror of saphid/frame-control. GitHub repository ID: 1388213277. Gitea import mode: github-full.
https://github.com/saphid/frame-control Readme MIT
39 MiB
0 Stars 1 Watchers 0 Forks
Languages
JavaScript 53.7%
Python 32%
HTML 7.7%
Swift 3.9%
Shell 1.4%
Other 1%