steam-frame/saphid--frame-control
1
0 Fork 0
mirror of https://github.com/saphid/frame-control.git synced 2026-10-06 06:00:33 +02:00
Code Issues Packages Projects Releases 6 Wiki Activity

Frame Control: Mac app, web UI, Android and Steam tooling

Browse Source 
Package the Frame Control web UI as an installable Electron Mac app and
bring in the tooling built alongside it.

- app/: Electron wrapper that starts ui/server.py on a free loopback port,
  hardened window (sandbox, no navigation, runAsNode fuse off), login-shell
  PATH so Homebrew tools work from Finder, first-run offer to run
  connect.sh, ad-hoc signed DMG/zip via electron-builder.
- ui/: headset view (OpenVR screenshots), device status, library, Steam
  "Get games" (owned games, install, store search), Android apps as
  persistent Lepton instances with a rated F-Droid catalogue and a private
  compatibility database, Android display controls over ADB, file and
  clipboard transfer, Flatpaks, remote and power actions.
- apk-catalog/, compat-db/, frame/: catalogue build pipeline, Lakebed
  capsule for compatibility reports, Frame-side launchers.
- tests/ and CI: server guard and validation tests plus Steam helper tests,
  run on Python 3.9 with script and app syntax checks.
- Docs: README leads with the Mac app; new Android, panels, Steam games and
  field-notes docs; security notes on LAN-exposed ADB ports.

Screenshot values for the headset's IP and Wi-Fi name are placeholders.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
saphidandClaude Opus 5.5 committed 2026-09-25 22:21:20 +10:00
1 parent 6ccf562756
commit d4486a7681
56 files changed
+19298 -11
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+150 -10
View File
@@ -4,10 +4,14 @@ This repo holds notes and Mac-side helpers for controlling a Valve Steam Frame
(standalone VR headset: SteamOS 3, Arch-based, arm64, Snapdragon 8 Gen 3) from
this Mac, with as little typing on the headset's virtual keyboard as possible.
Status: research written 2026-09-25, then checked against a real Frame the same
day (SteamOS 0.3.0, variant `vr`, build 20260922). `connect.sh`, `push.sh`,
`paste-to-frame.sh` and `install-apps.sh` work; the rest are still untested. See
[docs/open-questions.md](docs/open-questions.md#verified-on-device-2026-09-25).
Status: written 2026-09-25 and checked against a real Frame the same day
(SteamOS 0.3.0, variant `vr`, build 20260922). The **Frame Control** Mac app
and most scripts are **verified** on the device. The scripts table below marks
each one, and [docs/open-questions.md](docs/open-questions.md#verified-on-device-2026-09-25)
lists what's still unchecked.
**Quick start:** set up SSH once (next section), then install
[Frame Control](#frame-control-mac-app) from the DMG.
## Minimum typing on the headset
@@ -67,7 +71,121 @@ enables `sshd`. See [docs/ssh.md](docs/ssh.md#fallback-bootstrap-one-liner).
Details: [docs/ssh.md](docs/ssh.md), [docs/streaming.md](docs/streaming.md),
[docs/file-transfer.md](docs/file-transfer.md),
[docs/open-questions.md](docs/open-questions.md).
[docs/open-questions.md](docs/open-questions.md). For how the Frame's software
fits together, see [docs/how-the-frame-works.md](docs/how-the-frame-works.md).
## Windows anywhere in the room
The in-headset Linux desktop is a single 1280×800 panel, and its windows can't
leave it. Each Steam app, though, gets its own SteamVR panel. That also works
for any Linux app tagged with an app id of its own:
```sh
./scripts/panel-on-frame.sh konsole
./scripts/panel-on-frame.sh mac-screen # the Mac's screen, in its own panel
```
Then use the SteamVR dashboard's **Float in World**, **Move** and **Size**
controls to place each panel. See [docs/panels.md](docs/panels.md).
## Frame Control (Mac app)
As of 2026-09-25 no other Mac app manages the Frame end to end.
[Stream Frame](https://streamframe.app/) (macOS 14+, free) records and screenshots
the headset over SSH. [FrameDrop](https://framedropvr.com) sideloads but is
Windows-only. Steam Link views the headset. **Frame Control** is a Mac app over
the scripts below. Install it from the DMG (see [Mac app](#mac-app)), or run
the same UI in a browser without packaging:
```sh
./scripts/frame-ui.sh # opens http://127.0.0.1:47810 in its own window
```
![Frame Control](docs/img/frame-control.png)
- **Headset view**: what the lenses show, as SteamVR composites it (the room,
floating panels, dashboard and controllers). Shows the left eye, like pointing
a camera into one lens, or both eyes; single shot or about 2 fps live; saves
as PNG. The viewer fits the whole frame; zoom with − / + (or scroll, or
double-click), drag to pan, `0` to fit, `F` for full screen. It uses OpenVR's `IVRScreenshots` API through Python `ctypes`
(`ui/frame_vrshot.py`), so nothing is installed on the Frame. **Desktop panel**
captures gamescope's flat layer instead.
- Battery with charging state: charge rate in watts, time to full or empty,
charger type and wattage (for example USB-C PD 20 W), and battery temperature
- Storage, memory, temperature, Wi-Fi, uptime, and whether SteamVR, the desktop,
Lepton and xrdp are running
- Library shelf with Steam cover art and a Play button (`steam://rungameid`)
- **Get games**: every game you own with its Steam Frame rating (Verified,
Playable, Unsupported, Unknown). Install on Frame downloads it to the headset
with live progress. Search the Steam store with prices and Frame ratings; Buy
opens the store page in your browser, or Store on Frame opens it in the
headset. It drives the Frame's own Steam client through its DevTools port;
see `docs/steam-games.md`
- Volume and mute (`wpctl`)
- **Android apps**: search about 4,500 F-Droid apps rated for the Frame, install
one with a click as its own Lepton instance (it keeps its data and shows in the
Steam library), then launch, stop, test or remove it. **Report an APK** records whether any APK
worked (F-Droid or not: pick a file, type a package, or use an installed app). The
ratings go into our private compatibility database (a Lakebed capsule only the
app can use, backed up daily to Google Drive; see `compat-db/README.md`)
- **Android display**: pick a running Lepton instance (by the app in it) and set
its resolution (Native 1920×1080, or Sharp 2560×1440 with density scaled to
match), UI scale (Smaller / Default / Larger, or an exact dpi) and text size
(0.85–1.3×) over ADB (`wm size`, `wm density`, `font_scale`). Reset puts all three
back. Whether the settings survive the app relaunching is untested
- Drag and drop files to `~/Downloads`; `.apk` files install as their own Android app
- Send typed text, or the Mac clipboard, to the Frame clipboard
- Install and remove Flatpaks (quick picks: Moonlight, Firefox, VLC, Remmina)
- One-click SSH or SFTP in Terminal, Steam Link, and Windows App (RDP).
Sleep, restart and shut down open Terminal because SteamOS asks for the
sudo password over SSH.
The server is Python stdlib only and listens on 127.0.0.1. It rejects requests
with a non-local `Host` header, and any `/api/` request without a custom
header, so other websites can't drive it or read captures. It keeps a single multiplexed SSH connection open, so
status and each capture take about 0.3s. Headset captures are deleted from the
Frame as soon as they're copied, because they show everything on screen,
including anything private. The look follows the Steam client: its palette,
Motiva Sans (loaded from Valve's CDN), portrait library capsules and green
Play buttons. **Verified on the Frame 2026-09-25:** status and charging details,
both capture modes (headset view while in use, and a blank frame in standby,
which the UI labels), clipboard, volume, file push, and input validation. **Not yet exercised from the UI:** Launch, Flatpak
install/remove, APK drop, and the power buttons. Each of these calls a
command or script that was verified separately.
### Mac app
`app/` wraps the same UI as a standalone Mac app (Electron). The app bundles
`ui/`, `scripts/`, `frame/android/` and the rated catalogue from `apk-catalog/`.
It starts `ui/server.py` on a free loopback port and shows it in its own window.
The server stops when you quit the app. A prebuilt DMG for Apple Silicon is
attached to each [GitHub release](https://github.com/saphid/steam-frame/releases).
```sh
cd app
npm install
npm run dist # → app/dist/Frame Control-<version>-arm64.dmg (and a .zip)
npm start # run from the checkout without packaging
```
Open the DMG and drag **Frame Control** to Applications. You need `python3` on
the Mac (Xcode Command Line Tools or Homebrew). The app reads `PATH` from your
login shell, so Homebrew's `rsync` and `adb` work when you launch it from
Finder. On first launch, if there's no `frame` SSH alias yet, the app offers
to run `connect.sh` in Terminal. **Frame → Set Up Connection…** does the same
at any time. The Frame menu also shows the server log at
`~/Library/Logs/Frame Control/server.log`. Installing APKs needs `adb`
(`brew install android-platform-tools`). The Android ratings database needs
its key in the Keychain (see `compat-db/README.md`); without it, the app uses
its offline copy.
The build is ad-hoc signed and not notarized. A copy you build yourself opens
normally. A copy downloaded from GitHub Releases is quarantined; clear it with
`xattr -dr com.apple.quarantine "/Applications/Frame Control.app"`. The first
time you use them, macOS asks to allow local network access (for SSH) and
control of Terminal (for SSH and power actions). **Verified 2026-09-25:**
installed from the DMG, launched from Finder, connected to the Frame, and
showed live status and the library.
## Scripts
@@ -76,20 +194,42 @@ Details: [docs/ssh.md](docs/ssh.md), [docs/streaming.md](docs/streaming.md),
| `scripts/connect.sh` | Mac | Discover, set up key and `~/.ssh/config`, copy key, optional `--harden` (**verified**; `--harden` untested) |
| `scripts/install-apps.sh` | Mac → Frame | Install Flatpaks (Remmina, Moonlight, …) on the Frame over SSH as `--user` (**verified** with Remmina) |
| `scripts/paste-to-frame.sh` | Mac → Frame | Send the Mac clipboard (or stdin) to the Frame clipboard (**verified**) |
| `scripts/install-apk.sh` | Mac → Frame | Install APKs into Lepton with ADB tunnelled over SSH; starts Lepton Development if needed (**verified**; see [docs/apks.md](docs/apks.md) for app compatibility) |
| `scripts/install-apk.sh` | Mac → Frame | Install APKs, each as its own persistent Lepton instance with a Steam library shortcut (`--dev`: old ADB path into Lepton Development) (**verified**; see [docs/apks.md](docs/apks.md)) |
| `scripts/panel-on-frame.sh` | Mac → Frame | Start an app as its own floating VR panel, outside the desktop (**verified**: overlays created; in-headset placement not yet checked) |
| `scripts/run-on-frame.sh` | Mac → Frame | Start an app on the headset desktop, e.g. `mac-screen` opens Remmina straight into the Mac (**verified**) |
| `scripts/frame-ui.sh` | Mac | Start the Frame Control web UI (`ui/server.py`) and open it (**verified**) |
| `scripts/apk-catalog.sh` | Mac | Refresh the rated F-Droid catalogue that Frame Control's Android section shows (**verified**) |
| `scripts/compat-db-backup.sh` | Mac | Back up the compatibility database locally and to Google Drive (daily LaunchAgent) (**verified**) |
| `scripts/push.sh` | Mac → Frame | `rsync` files to `~/Downloads` (or a given path) on the Frame (**verified**) |
| `scripts/serve-bootstrap.sh` | Mac | Fallback: serve `bootstrap-on-frame.sh` with your public key embedded |
| `scripts/bootstrap-on-frame.sh` | Frame | Fallback: install the key and enable `sshd` |
## Security notes
- With Developer Mode on, `sshd`, ADB (Wi-Fi, port 5555, while a Lepton
session is running), and xrdp are all reachable on your LAN. Use trusted
networks only. Turn Developer Mode off when you don't need it.
- With Developer Mode on, `sshd`, ADB and xrdp are all reachable on your LAN.
Each running Lepton (Android) instance opens its own ADB port in 5555–5599,
listening on `0.0.0.0` rather than only loopback. This was seen on the
device on 2026-09-25, so anyone on the network can reach it. Use trusted
networks only, and turn Developer Mode off when you don't need it.
- Frame Control reaches ADB and the Steam client's DevTools port (Frame
loopback `127.0.0.1:8080`) only through SSH tunnels. The compatibility
database key lives in the macOS Keychain and is never written to the repo.
- `steamos` has `sudo`, protected by the same Developer Mode password. Once
you've switched to key auth, a short password still protects `sudo` and
RDP, so pick one that isn't trivially guessable.
- Don't port-forward 22, 3389, or 5555 from your router. For remote access,
- Don't port-forward 22, 3389, or 5555–5599 from your router. For remote access,
use Tailscale (Flatpak/package availability for the Frame hasn't been
checked).
## Development
```sh
python3 -m unittest discover -s tests # server guards, validation, Steam helpers; no headset needed
cd app && npm install && npm run dist # build the DMG
```
GitHub Actions runs the tests on Python 3.9, which is the oldest `python3` the app
may find (Xcode Command Line Tools), plus syntax checks for every script and the
Electron main process (`.github/workflows/checks.yml`). Anything that touches the
headset is verified by hand against a real Frame, and the docs label it
**verified** or **inferred**.