diff --git a/.claude/skills/steam-frame/SKILL.md b/.claude/skills/steam-frame/SKILL.md index ed74648..0437825 100644 --- a/.claude/skills/steam-frame/SKILL.md +++ b/.claude/skills/steam-frame/SKILL.md @@ -27,6 +27,9 @@ desktop or panels. | Flatpaks | `docs/streaming.md` | `scripts/install-apps.sh` | | Launch an app inside the desktop panel | the script's header comment | `scripts/run-on-frame.sh` | | Mac GUI over all of this | `README.md` → Frame Control | `scripts/frame-ui.sh` | +| iPhone/iPad app (server runs on the Frame, `FRAME_LOCAL=1`) | `docs/iphone.md` | `ios/`, `ui/local-bin/ssh` | +| Recovery images, factory reset, boot loops | `docs/recovery-and-images.md`, `docs/how-the-frame-works.md` | `~/Downloads/steam-frame-recovery/` | +| Test without the headset (the Frame OS image's own sshd) | `tests/frame-container/README.md` | `tests/frame-container/frame-image.sh` | | What's still unverified | `docs/open-questions.md` | — | Each script's usage is in its header comment. Read the header rather than diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000..2aa60c9 --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1 @@ +ko_fi: alexsouthwell diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index b0d46ca..f131fe4 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -20,6 +20,7 @@ jobs: run: sudo apt-get update -qq && sudo apt-get install -y -qq zsh - name: Script syntax run: | + sh -n ui/local-bin/ssh for f in scripts/*.sh frame/*/*.sh; do case "$(head -n 1 "$f")" in *zsh*) zsh -n "$f" ;; @@ -28,7 +29,7 @@ jobs: done - name: Python compiles run: | - python -m py_compile ui/*.py apk-catalog/*.py frame/android/*.py + python -m py_compile ui/*.py apk-catalog/*.py frame/android/*.py ios/scripts/*.py # Valve's devkit-utils (vendored; run by the Frame's python3). Most have no .py suffix. python -m py_compile $(find frame/devkit-utils -type f ! -name '*.*' ! -name LICENSE) frame/devkit-utils/devkit_utils/*.py - name: Server tests @@ -61,3 +62,28 @@ jobs: python-version: ${{ matrix.python }} - name: Server tests run: python -m unittest discover -s tests -v + + # The iPhone app: builds for the Simulator and runs its unit tests. + ios: + runs-on: macos-latest + steps: + - uses: actions/checkout@v4 + - name: Generate the project + run: brew install xcodegen && cd ios && xcodegen generate + - name: Build and test + run: | + cd ios + udid=$(xcrun simctl list devices available -j | python3 -c 'import json,sys; d=json.load(sys.stdin)["devices"]; print(next(x["udid"] for r in d for x in d[r] if x["name"].startswith("iPhone")))') + xcodebuild -project FrameControl.xcodeproj -scheme FrameControl -destination "platform=iOS Simulator,id=$udid" CODE_SIGNING_ALLOWED=NO test + + # End-to-end tests against the fake Frame (tests/fakeframe): Arch Linux ARM + # in Docker, on a native arm64 runner like the headset. See docs/testing.md. + e2e: + runs-on: ubuntu-24.04-arm + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - name: Install zsh + run: sudo apt-get update -qq && sudo apt-get install -y -qq zsh + - name: End-to-end tests + run: scripts/e2e.sh diff --git a/.gitignore b/.gitignore index ad5b005..d89ecc2 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ apk-catalog/data/cache/ apk-catalog/data/index-v2.json* compat-db/.env.lakebed.server compat-db/.lakebed/ +tests/smoke/results/ diff --git a/README.md b/README.md index 2acc3df..c4574f7 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ See what the headset sees, install games and Android apps, move files and text a
-Frame Control showing the headset view, battery and status, and the Steam library +Frame Control's Games tab: installed games, sideloaded titles, and your Steam library with Frame ratings Watch the Frame Control trailer @@ -103,6 +103,9 @@ already ships (sideloading a game copies Valve's own devkit scripts to | **Linux** (x64) | [AppImage](https://github.com/saphid/steam-frame/releases/latest/download/Frame-Control-linux-x86_64.AppImage) · [.deb](https://github.com/saphid/steam-frame/releases/latest/download/Frame-Control-linux-amd64.deb) | `ssh` (most desktops have it) | | **Linux** (arm64) | [AppImage](https://github.com/saphid/steam-frame/releases/latest/download/Frame-Control-linux-arm64.AppImage) · [.deb](https://github.com/saphid/steam-frame/releases/latest/download/Frame-Control-linux-arm64.deb) | `ssh`, and `adb` for Android apps (`sudo apt install adb`) | +**iPhone and iPad:** the same features from your phone, with nothing to install on +a computer. Build it from [`ios/`](ios) in Xcode; see [docs/iphone.md](docs/iphone.md). + The app brings its own Python and `adb`; SSH is built into macOS and Windows. From 0.4 it updates itself: when a new version is published, a banner offers **Update and restart**. It sends anonymous usage statistics, which you can turn @@ -205,6 +208,9 @@ Frame's software fits together, all checked against a real headset and labelled | [Install links for websites](docs/web-install.md) | `frame-control://install` links and manifests, the rules, a button to paste | | [Steam games](docs/steam-games.md) · [VR video](docs/vr-video.md) · [WebXR in Chromium](docs/webxr-chromium.md) | Installing and buying, watching VR180/360, the Chromium build | | [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes | +| [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing | +| [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset | +| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, and the headset smoke test | | [Open questions](docs/open-questions.md) | What's still unchecked |
@@ -231,6 +237,7 @@ Frame's software fits together, all checked against a real headset and labelled ```sh python3 -m unittest discover -s tests # server tests; no headset needed +scripts/e2e.sh # end-to-end against a fake Frame (Linux with Docker) cd app && npm install && npm start # run the app from the checkout ``` diff --git a/app/main.js b/app/main.js index f3463b8..1c66379 100644 --- a/app/main.js +++ b/app/main.js @@ -248,6 +248,7 @@ function fromUi(e) { } ipcMain.handle("clipboard:read", (e) => fromUi(e) ? clipboard.readText() : ""); +ipcMain.handle("connection:setup", (e) => { if (fromUi(e)) setUpConnection(); }); ipcMain.handle("update:get", (e) => fromUi(e) ? publicUpdate() : null); ipcMain.handle("update:check", (e) => fromUi(e) ? checkForUpdate({ manual: true }).then(publicUpdate) : null); ipcMain.handle("update:install", (e) => { if (fromUi(e)) installUpdate(); }); diff --git a/app/preload.js b/app/preload.js index aaf5290..d6d935b 100644 --- a/app/preload.js +++ b/app/preload.js @@ -2,6 +2,7 @@ // to the Frame needs no pbpaste, PowerShell, xclip or wl-clipboard. Also tells // the page where a dropped file or folder lives, so a folder can be sideloaded // as a title without zipping it (the local server reads it from there). +// It can open Set Up Connection when the headset can't be reached. // It also receives frame-control://install links (docs/web-install.md): only // what the link asked for, never an install; the page asks the user first. // And it passes update state both ways: see app/updater.js. @@ -9,6 +10,7 @@ const { contextBridge, ipcRenderer, webUtils } = require("electron"); contextBridge.exposeInMainWorld("frameApp", { readClipboard: () => ipcRenderer.invoke("clipboard:read"), + setUpConnection: () => ipcRenderer.invoke("connection:setup"), pathForFile: (file) => { try { return webUtils.getPathForFile(file) || ""; } catch { return ""; } }, // Updates (app/updater.js): the page shows a banner and an Update button. update: { diff --git a/docs/frame-control.md b/docs/frame-control.md index 34796d9..28f9e69 100644 --- a/docs/frame-control.md +++ b/docs/frame-control.md @@ -17,6 +17,15 @@ python3 ui/server.py # anywhere: then open http://127.0.0.1:47810 ## Features +The window has four tabs: **Home** (headset view, status, screenshots), +**Games** (installed games, sideloaded titles, getting games), **Android** (apps, +the catalogue, display settings, reports) and **Tools** (sending files and text, +Flatpaks, remote and power). Keys 1–4 switch between them. Files can be dropped +anywhere in the window. When the Frame can't be reached, one banner says why in +plain words and the app retries every few seconds, filling everything in once it +answers. Flatpak and Android installs run in the background; the bottom bar +counts them while they run. + - **Headset view**: what the lenses show, as SteamVR composites it (the room, floating panels, dashboard and controllers). Shows the left eye, like pointing a camera into one lens, or both eyes, as a single shot; saves as PNG. **Live** diff --git a/docs/how-the-frame-works.md b/docs/how-the-frame-works.md index b897a18..5c392d1 100644 --- a/docs/how-the-frame-works.md +++ b/docs/how-the-frame-works.md @@ -51,7 +51,13 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600 | Tailscale runs without root as a userspace `tailscaled` user service (static arm64 build in `~/.local/share/tailscale`, lingering on). In userspace mode, inbound tailnet connections reach the Frame's **loopback**, so every port, including DevTools on 8080, is reachable from the tailnet. **Verified 2026-09-25.** | [tailscale.md](tailscale.md), `scripts/tailscale-on-frame.sh` | | **T3 Code desktop runs natively.** The stock release `T3-Code-0.0.42-arm64.AppImage` in `~/Applications/T3CodeDesktop/` starts with no extra setup: glibc 2.39, `libfuse.so.2`, GTK 3, NSS and libsecret are on the image. `panel-on-frame.sh --name t3code-desktop -- '~/Applications/T3CodeDesktop/T3-Code.AppImage'` gives it its own panel (`valve.steam.desktopgame.2000281357`, `--ozone-platform=x11`). Its bundled server listens on `127.0.0.1:3773` and shows up in onboarding as the `frame` computer, with `passwordStore: gnome-libsecret`. The image has no agent CLI and no `node`. Agents run through the LAN CLIProxyAPI (`llm-proxy.lan:8317`, which resolves on the Frame). Claude Code 2.1.283 comes from `claude.ai/install.sh`, and Codex 0.157.1 from the `codex-aarch64-unknown-linux-musl` release tarball, both into `~/.local/bin`. `with-cliproxy` and a mode-600 `~/.config/cliproxyapi/secrets.env` are copied from the Mac. The wrappers `claude-cliproxy` and `codex-cliproxy` (a `-c model_provider=cliproxy`, `wire_api="responses"`, `env_key="CLIPROXY_API_KEY"`) are set as `providers.claudeAgent.binaryPath` and `providers.codex.binaryPath` in `~/.t3/userdata/settings.json`, and T3 picked that up without a restart. Through the wrappers, `claude auth status` reports `loggedIn: true` (`oauth_token`), and both CLIs answered a prompt with `kimi-k3`. `gamescopectl screenshot` captured another layer (the Lepton T3 app) rather than this panel. `DISPLAY=:0 xwd -id ` piped to `ffmpeg` captures the window itself (1920×1080). **Verified 2026-09-26**, BUILD_ID 20260922.6101926. | Running T3 Code as a host on the Frame | | Power actions need `sudo`, which asks for the Developer Mode password over SSH. | Frame Control's power buttons | -| **Boot / recovery menu.** Hold Power ~10 s until the LED goes off, then power on while holding the **AUX button on top of the Power button** (not the volume keys) until a text menu appears. Entries: `Current` (SteamOS-A/B + build), `Previous` (the other A/B slot), `Boot from USB`, `Repair Steam Installation`, `Erase User Data` (factory reset), `ADB mode`, `Battery Ship Mode`. It auto-boots `Current` after a ~15 s countdown. **Volume Up/Down (left side) move, AUX (right side) selects.** For a boot loop, Valve says pick `Previous` (keeps user data); then `Repair Steam Installation`; `Erase User Data` wipes `~` (SSH keys, Tailscale, Flatpaks, T3 setup). Last resort is a full re-image, two ways: (1) USB: write `steamframe-repair-latest.img.bz2` to an 8 GB+ USB-C stick (Balena Etcher on the Mac), pick `Boot from USB`, then use "Wipe Device & Install SteamOS" / "Repair SteamOS" (keeps games and personal content) from the recovery desktop; (2) cable/EDL: `steamframe-repair-qdl-latest.tar.gz`, run `flash.sh` (Linux) or `flash.cmd` (Windows), then with the Frame off for 10 s hold Power + Vol Up + Vol Down for 10 s and plug it in; it reflashes and reboots. Both images: `https://steamdeck-images.steamos.cloud/recovery/` (build 20260922.5153644, 0.3.0, ~4 GB each, no published checksums); local copies in `~/Downloads/steam-frame-recovery/`. Source: Valve's [SteamOS Recovery FAQ](https://help.steampowered.com/en/faqs/view/1B71-EDF2-EB6D-2BB3) and [Installation and Repair FAQ](https://help.steampowered.com/en/faqs/view/65B4-2AA3-5F37-4227), plus a menu photo in [EloiStree/HelloSteamFrame#9](https://github.com/EloiStree/HelloSteamFrame/issues/9). **Inferred** (Valve docs, 2026-09-26); not yet tried on our Frame. | Recovering from a boot loop | +| **SSH server:** OpenSSH 9.7p1. It offers `publickey,password` (keyboard-interactive is off, PAM on) and also asks `userdbctl ssh-authorized-keys` for keys. OpenSSH ≥ 8.8 rejects SHA-1 `ssh-rsa` signatures by default, so a client whose RSA support is SHA-1 only (the Swift library Citadel, for one) can't log in with the RSA key that devkit pairing installs; use ed25519 (**inferred** from OpenSSH defaults). **Verified 2026-09-27**, BUILD_ID 20260922.6101926. | [iphone.md](iphone.md), `ui/frame_connect.py` | +| **Tools on the image:** Python 3.12.3, `ffmpeg`, `openssl`, `curl`, `rsync`, `zip`/`unzip`, `flatpak`, `wpctl`, `podman`. **No `adb`.** `steamos` is uid 1000, in `wheel`, and sudoers has `%wheel ALL=(ALL) ALL`, so `sudo -S` takes the Developer Mode password on stdin. **Verified 2026-09-27.** | Running Frame Control's server on the Frame (`FRAME_LOCAL=1`, [iphone.md](iphone.md)) | +| **Each Lepton instance is a podman container** named `lepton-steamlaunch-`, labelled with its ADB port (`podman ps --format '{{.Names}} {{.Labels.adb_port}}'`). `podman exec /system/bin/sh -c '…'` runs Android's shell inside it with no adb at all (used for `pidof` and `logcat` by the app tester). Running `wm size`/`wm density` that way is untested. **Verified 2026-09-27.** | `ui/frame_android.py`, the iPhone app's display settings | +| **Asleep means off the network.** In standby the Frame stops answering on its LAN address, `frame.local` and Tailscale alike (`Host is down`, `No route to host`, timeouts), and ping fails. It was unreachable for about 2.5 hours until woken. Nothing over SSH can wake it. **Verified 2026-09-27.** | Frame Control's offline banner and retries | +| **Battery at full on a charger** can read `Discharging` at about 0 W (for example 99 %, 0.0 W, USB-C PD 18 W). Treat under 0.5 W on a charger as "not charging", not "draining". **Verified 2026-09-27.** | Frame Control's battery card | +| **The OS image is downloadable.** Valve's recovery images for the Frame are at `https://steamdeck-images.steamos.cloud/recovery/`. The root filesystem inside is btrfs, and it runs as an SSH test target on ARM64 Linux without the headset (`tests/frame-container/frame-image.sh`). **Verified 2026-09-27.** | [recovery-and-images.md](recovery-and-images.md) | +| **Boot / recovery menu.** Hold Power ~10 s until the LED goes off, then power on while holding the **AUX button on top of the Power button** (not the volume keys) until a text menu appears. Entries: `Current` (SteamOS-A/B + build), `Previous` (the other A/B slot), `Boot from USB`, `Repair Steam Installation`, `Erase User Data` (factory reset), `ADB mode`, `Battery Ship Mode`. It auto-boots `Current` after a ~15 s countdown. **Volume Up/Down (left side) move, AUX (right side) selects.** For a boot loop, Valve says pick `Previous` (keeps user data); then `Repair Steam Installation`; `Erase User Data` wipes `~` (SSH keys, Tailscale, Flatpaks, T3 setup). Last resort is a full re-image, two ways: (1) USB: write `steamframe-oobe-repair-.img.bz2` to an 8 GB+ USB-C stick (Balena Etcher on the Mac), pick `Boot from USB`, then use "Wipe Device & Install SteamOS" / "Repair SteamOS" (keeps games and personal content) from the recovery desktop; (2) cable/EDL: `steamframe-oobe-repair-qdl-.tar.gz`, run `flash.sh` (Linux) or `flash.cmd` (Windows), then with the Frame off for 10 s hold Power + Vol Up + Vol Down for 10 s and plug it in; it reflashes and reboots. Both images: `https://steamdeck-images.steamos.cloud/recovery/` (build 20260922.5153644, 0.3.0, 3.8 GiB each, no published checksums); local copies in `~/Downloads/steam-frame-recovery/`. File names, checksums and what's inside: [recovery-and-images.md](recovery-and-images.md). Source: Valve's [SteamOS Recovery FAQ](https://help.steampowered.com/en/faqs/view/1B71-EDF2-EB6D-2BB3) and [Installation and Repair FAQ](https://help.steampowered.com/en/faqs/view/65B4-2AA3-5F37-4227), plus a menu photo in [EloiStree/HelloSteamFrame#9](https://github.com/EloiStree/HelloSteamFrame/issues/9). **Inferred** (Valve docs, 2026-09-26); not yet tried on our Frame. | Recovering from a boot loop | | **Boot loop cause: the SteamVR health check.** `steamvr.service` runs `/usr/share/deckard/steamvr-health-check`, which appends `frog:glasses:` to `$XDG_RUNTIME_DIR/steamvr-short-session-tracker` on every failed or <10 s SteamVR run. At 3 it runs `steam-health-check --repair-now`, which **deletes all of `~/.local/share/Steam` (games, login, Developer Mode) and `~/.steam`**, keeping only `registry.vdf`. At 4 it also tries `steamos-bootconf set-mode reboot-other` (fails as the user: `bootenv: Permission denied`). SteamVR normally fails 1–2 times per boot while it waits for the Steam client (`SteamAPI_InitEx failed … Steam is probably not running`, then `fatal stalled cross-thread pipe`). Once Steam has been wiped, it has to re-download a ~210 MB client on every boot, so SteamVR keeps failing, Steam keeps getting wiped and the Frame reboots, in a loop. Also, the Steam updater can deadlock at `Installing update...` (main process blocked writing to the `-child-update-ui` process, which is stuck in `drm_syncobj_array_wait_timeout`). Killing only the `-child-update-ui` process lets the install finish (`package/*.installed` appears). **Fix without sudo:** over USB-C ADB (`adb -s frame shell` works as `steamos` while the Frame is looping; SSH is refused once Developer Mode is lost), truncate both `/run/user/1000/steam{,vr}-short-session-tracker` files and `chmod 444` them (the health check then logs `Permission denied` and does nothing; this is tmpfs, so it resets on reboot). Unstick the updater if needed, let Steam finish installing, then hold Power 10 s and start the Frame normally. `systemctl reboot` over ADB needs interactive auth. After the fix, sign in to Steam and turn Developer Mode back on. **Verified 2026-09-26**, BUILD_ID 20260922.6101926, slot B (clean boot: 0 SteamVR failures, SSH and Tailscale back). | Diagnosing a boot loop | ## Debug recipes @@ -79,4 +85,6 @@ ssh frame 'cat /opt/steamvr/resources/webinterface/dashboard/localization/dashbo - Installing and buying Steam games: [steam-games.md](steam-games.md) - Remote access from anywhere: [tailscale.md](tailscale.md) - Floating windows in space: [panels.md](panels.md) +- Recovery images, what's in them, testing without the headset: [recovery-and-images.md](recovery-and-images.md) +- Frame Control on iPhone (the server running on the Frame itself): [iphone.md](iphone.md) - What's still unverified: [open-questions.md](open-questions.md) diff --git a/docs/img/frame-control.png b/docs/img/frame-control.png index ae62e25..a2a5ae4 100644 Binary files a/docs/img/frame-control.png and b/docs/img/frame-control.png differ diff --git a/docs/img/iphone-tabs.jpg b/docs/img/iphone-tabs.jpg new file mode 100644 index 0000000..3d1e6ed Binary files /dev/null and b/docs/img/iphone-tabs.jpg differ diff --git a/docs/iphone.md b/docs/iphone.md new file mode 100644 index 0000000..b09920d --- /dev/null +++ b/docs/iphone.md @@ -0,0 +1,109 @@ +# Frame Control for iPhone + +The iPhone (and iPad) app does what the desktop app does, from the phone: +headset view and live video, battery and status, screenshots, Steam games, +Android apps and their display settings, sideloading, files, clipboard, +Flatpaks, and power. Source: [`ios/`](../ios). + +## How it works + +An iPhone can't run Python or `ssh`, but the Frame can. So the app: + +1. connects to the Frame over SSH itself (the [Citadel](https://github.com/orlandos-nl/Citadel) + Swift SSH library), with its own ed25519 key from the Keychain; +2. copies Frame Control's server and helpers (`ios/scripts/make_frame_bundle.py`, + under 1 MB) to `~/.cache/frame-control/` on the Frame, once per version; +3. starts `ui/server.py` there with `FRAME_LOCAL=1`. It listens only on the + Frame's own 127.0.0.1, and it stops when the phone disconnects (`--exit-on-eof`); +4. tunnels to it through the SSH session and shows the same page as the desktop + app, in a web view. The page carries a fresh key each session, which the + server requires on every request. + +With `FRAME_LOCAL=1`, every `ssh frame COMMAND` the server runs goes to +`ui/local-bin/ssh`, which runs the command on the Frame directly (rsync uses it +as its transport too), so the desktop and phone share one code path. Android +display settings use `podman exec` into each Lepton container instead of adb, +which the Frame doesn't have. + +Nothing is left running on the Frame after the phone disconnects; the copied +files stay in `~/.cache/frame-control` (delete it any time). + +## Pairing + +On the Frame, turn on Developer Mode and set a user password (Steam Settings → +System, then Developer → Set User Password). In the app, enter the headset's +address (`frame.local`, its IP, or its Tailscale name) and that password once. +The app adds its own key to `~/.ssh/authorized_keys` and remembers the Frame's +host key; the password isn't saved. If you already reach the Frame over SSH, +**Or add the key yourself** shows the phone's key to paste into +`authorized_keys`, and connects without a password. + +Valve's tap-to-approve devkit pairing isn't used: it only takes RSA keys, and +the Frame's OpenSSH 9.7 rejects the SHA-1 RSA signatures the Swift SSH library +makes. + +## What's different on the phone + +| Desktop | iPhone | +|---|---| +| Drop files anywhere | Tap **Send to Frame** (or Add a game) and pick files; folders need zipping | +| Screenshots save to `~/Pictures/SteamFrame` | Save opens the share sheet: Save Image puts it in Photos | +| SSH and SFTP open a terminal | They open an app that handles `ssh://` / `sftp://` (Blink Shell, Termius) | +| Steam Link, remote desktop | Open the Steam Link and Windows App apps | +| Sleep, restart, shut down ask in a terminal | The page asks for the Developer Mode password | +| Compatibility reports kept on the computer | Kept on the Frame (`~/.local/share/Frame Control`) | + +## Building + +```sh +cd ios +xcodegen generate # after changing project.yml +open FrameControl.xcodeproj +``` + +The build packs the Frame bundle from the checkout, so the phone always runs +the page and server from the same commit. Running on a phone needs your own +signing team in Xcode (Signing & Capabilities). + +## Verified + +The four tabs in the iPhone app, connected to a Frame + +In the iOS Simulator (iOS 26.5) against a real Frame, 2026-09-27: the app connected +with its key, copied the bundle over SFTP, started the server on the Frame and +showed all four tabs with live data. In the app's web view, Capture returned a +headset still and Live played H.264 video at 31 fps (WebCodecs works in +WKWebView). Through the app's tunnel: status, games, Steam library, Android apps, +screenshots, a file upload (checked on the Frame), a background install job, and +the power password check (a wrong password is refused). The server on the Frame +exits within seconds of the app closing. + +Against Valve's own Steam Frame OS (SteamOS 0.3.0 build 20260922.5152327, the +`rootfs-A` partition of the Frame recovery image, run with its own sshd; see +[tests/frame-container](../tests/frame-container)), and a Holo Core stand-in: +pairing with the password (key added with the right +permissions, host key pinned, password stored nowhere), the power password +check (a wrong or missing password refused; the right one reaches `systemctl`), +a changed host key refused with "Pair with the Frame again", and a wrong +pairing password reported the same way. + +Also verified in the Simulator against the Frame (2026-09-27): the setup screen +found the Frame by itself over Bonjour (`frame · 192.168.1.237`); a paired app +waiting for a sleeping Frame connected 4 s after it answered; an upload from the +app's web view landed in `~/Downloads`; the share sheet offers Save Image +(needs `NSPhotoLibraryAddUsageDescription`, now declared); an install link opens +the confirm dialog and downloads nothing until Install; Steam Link without the +app installed opens its App Store page. + +Things iOS asks the first time: **Local Network** (tap Allow, or the app can't +see the Frame), and **Paste** when you send the iPhone's clipboard (tap Allow +Paste, or set Settings → Apps → Frame Control → Paste from Other Apps → Allow). +Sending text to the Frame's clipboard needs the desktop panel open in the +headset, as on the desktop app. + +Not yet exercised: Android display changes through podman (no Android app was +running), a real sleep/restart/shut down on the Frame, and a physical iPhone. + +Debug builds have Simulator test hooks (`FRAME_TEST_HOST`, `FRAME_TEST_PAGE`, +`FRAME_TEST_JS`, and the tunnel URL in the app's Caches folder); release builds +don't. diff --git a/docs/open-questions.md b/docs/open-questions.md index 67f4645..13b4c45 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -33,14 +33,19 @@ build 20260922.6101926, kernel 6.18, aarch64): - **10.** `install-apps.sh remmina --vnc-host .local` installed Remmina as a `--user` Flatpak over SSH and wrote the profile. The desktop's `XDG_DATA_DIRS` includes the user Flatpak exports, so it shows up in the menu. - The Frame can reach the Mac's Screen Sharing port (5900). The Remmina - connection itself hasn't been tried in the headset yet (part of 11). + The Frame can reach the Mac's Screen Sharing port (5900). +- **11.** Answered 2026-09-27 (BUILD_ID 20260925.6191901, macOS 27.0): the + pre-seeded profile connects and shows the Mac in its own panel. It asks for + the Mac account login rather than the VNC password, needs scale-to-fit at + Retina resolutions, and doesn't show the Mac cursor without + `scripts/mac-cursor-ring.lua`. It's usable but noticeably laggy. See + [streaming.md](streaming.md). - **Panels.** An X11 window on gamescope's `:0` with its own `STEAM_GAME` id gets its own SteamVR overlay (`valve.steam.desktopgame.`). Three were created side by side with `panel-on-frame.sh`. See [panels.md](panels.md). -Still open: 4, 6, 7, 11 (in-headset connect), 12–15, 16 (off-LAN and after a reboot), 17–21. +Still open: 4, 6, 7, 12–15, 16 (off-LAN and after a reboot), 17–21. ## Check on the headset (in order) @@ -70,12 +75,10 @@ Still open: 4, 6, 7, 11 (in-headset connect), 12–15, 16 (off-LAN and after a r `ssh frame 'command -v wl-copy xclip rsync flatpak'`. 10. **Can Flatpaks be installed `--user` over SSH, and do they appear in the headset's desktop?** Test with `./scripts/install-apps.sh remmina`. -11. **Remmina → macOS Screen Sharing:** does it connect, and is it usable at - Retina resolutions? Is the pre-seeded profile path - (`~/.var/app/org.remmina.Remmina/data/remmina/`) the one Remmina - actually reads? -12. **Moonlight Flatpak (aarch64) + Sunshine on macOS:** worth trying only if - VNC is too slow. +11. ~~**Remmina → macOS Screen Sharing**~~: answered 2026-09-27; see above + and [streaming.md](streaming.md). +12. **Moonlight Flatpak (aarch64) + Sunshine on macOS:** VNC works but is + noticeably laggy, so this is worth trying. 13. **KDE Connect**: is it preinstalled or installable on the Frame, and does it pair with KDE Connect for macOS? 14. **Bluetooth keyboard pairing** on the Frame, for the rare times you do need @@ -86,8 +89,9 @@ Still open: 4, 6, 7, 11 (in-headset connect), 12–15, 16 (off-LAN and after a r runs as a lingering user service with no sudo; see [tailscale.md](tailscale.md). Still open: reaching the Frame from outside the home network, and the service starting after a reboot. -17. **Floating panels in the headset** (see [panels.md](panels.md)): do the - panels from `panel-on-frame.sh` show up, take input, and offer **Float in +17. **Floating panels in the headset** (see [panels.md](panels.md)): panels + from `panel-on-frame.sh` show up and take controller input (verified + 2026-09-27 with `mac-screen`). Still open: do they offer **Float in World** / **Move** / **Size**? Do floating positions survive closing and reopening the app, or a reboot? 18. **`LEPTON_NO_CLEANUP=1 %command%`** as Lepton Development's launch @@ -103,12 +107,32 @@ Still open: 4, 6, 7, 11 (in-headset connect), 12–15, 16 (off-LAN and after a r the colour-coded test clips play in 3D (red left eye, cyan right) for both H.264 and H.265? Does the DLNA browser find a server on the Mac? +## Verified 2026-09-27 + +- **Recovery images exist** for the Frame at + `https://steamdeck-images.steamos.cloud/recovery/`; the root filesystem inside + is btrfs and runs, as a userland, on ARM64 Linux. See + [recovery-and-images.md](recovery-and-images.md). +- **Frame Control's server runs on the Frame itself** (the iPhone app does + this), including headset capture, 31 fps live video and file uploads. See + [iphone.md](iphone.md). +- **Password pairing and `sudo -S`** work against the recovery image's own + sshd and sudo (not yet against the headset, whose password we don't hold). + +## Still open (2026-09-27) + +- Does `podman exec /system/bin/sh -c 'wm size'` change an + instance's display the way `adb shell wm size` does? +- Can the recovery image, or its kernel, boot in a VM at all? +- Does a real sleep, restart or shut down from the iPhone app work (via + `sudo -S systemctl`)? +- The Mac EDL flashing script in `~/Downloads/steam-frame-recovery/` hasn't + been run against a Frame. + ## Unconfirmed claims made in these docs - `/home` and `/etc` persist across Frame OS updates. This is inferred from Steam Deck behaviour. -- The whole Mac → Frame desktop path (VNC → Remmina). Each part is documented - separately, but the combination is untested. - Steam Remote Play with a Mac as host is broken. That's based on community reports, not tested with the Frame. - `connect.sh --harden`, `serve-bootstrap.sh` and diff --git a/docs/recovery-and-images.md b/docs/recovery-and-images.md new file mode 100644 index 0000000..ad170d6 --- /dev/null +++ b/docs/recovery-and-images.md @@ -0,0 +1,109 @@ +# Recovery images and OS images for the Frame + +Where to get the Steam Frame's operating system, what's inside it, and how to +run it for testing without the headset. For recovering a Frame that won't boot, +see the boot menu and boot-loop entries in +[how-the-frame-works.md](how-the-frame-works.md#facts-worth-knowing). + +## Downloads + +Valve's SteamOS download page (`store.steampowered.com/steamos/download`) +redirects to the [Installation and Repair FAQ](https://help.steampowered.com/en/faqs/view/65B4-2AA3-5F37-4227), +which offers the Steam Deck image. The **Steam Frame images are on the same +server** but aren't linked from that page: +**https://steamdeck-images.steamos.cloud/recovery/** (a plain directory +listing, checked 2026-09-27). + +| File | Size | Use | +|---|---|---| +| `steamframe-oobe-repair-20260922.5153644-0.3.0.img.bz2` (or `.img.zip`) | 3.8 GiB | Write to an 8 GB+ USB-C stick, then **Boot from USB** in the Frame's boot menu | +| `steamframe-oobe-repair-qdl-20260922.5153644-0.3.0.tar.gz` (or `.zip`) | 3.8 GiB | Flash over a USB-C cable in Qualcomm EDL mode with `flash.sh` (Linux) or `flash.cmd` (Windows), which use [qdl](https://github.com/linux-msm/qdl). **Wipes everything** | + +All four are dated 2026-09-22. Everything else there is for the Steam Deck +(`steamdeck-…`, x86-64), which won't run on the Frame. Valve publishes **no +checksums**. These are the SHA-256s of our downloads (2026-09-26), which passed +`bzip2 -t` and `tar -t`: + +``` +3a4a077f1b1f40688ab3279affcb56776bd97c54db1573e7c65fc52a97106676 steamframe-oobe-repair-20260922.5153644-0.3.0.img.bz2 +d3323bfa8efe9ece1954948421cdf5f705e8942eb50c960e2916d935d1b850ab steamframe-oobe-repair-qdl-20260922.5153644-0.3.0.tar.gz +``` + +Our copies, with a Mac EDL flashing script built on qdl (untested), are in +`~/Downloads/steam-frame-recovery/` on the Mac. + +## What's inside the USB image + +A GPT disk with 512-byte sectors and one A slot (a Frame has A and B slots; +the installer makes the rest). **Verified 2026-09-27** from +`steamframe-oobe-repair-20260922.5153644-0.3.0.img.bz2`: + +| # | Name | Start sector | Size | Type GUID | +|---|---|---|---|---| +| 1 | `esp` | 34 | 256 MiB | `c12a7328-f81f-11d2-ba4b-00a0c93ec93b` (EFI system) | +| 2 | `efi-A` | 524322 | 64 MiB | `ebd0a0a2-b9e5-4433-87c0-68b6b72699c7` | +| 3 | `rootfs-A` | 655394 | 5120 MiB | `4f68bce3-e8cd-4db1-96e7-fbcaf984b709` | +| 4 | `var-A` | 11141154 | 256 MiB | `4d21b016-b534-45c2-a9fb-5c16e091fd2d` | +| 5 | `home` | 11665442 | 100 MiB | `933ac7e1-2eb4-4f13-b844-0e14e2aef915` | + +The partitions start at sector 34, not on MiB boundaries, so compute offsets +from the table (sector × 512), not from rounded sizes. `rootfs-A` is **btrfs** +(label `rootfs-A`, 9.2 GB of files), mounted read-only on the Frame. +Its `/etc/os-release` says `NAME="SteamOS"`, `ID=steamos`, `ID_LIKE=arch`, +`VERSION_CODENAME=holo`; the running system reports version 0.3.0, variant +`vr`, build **20260922.5152327**, which is a different number from the +`5153644` in the file name. Our headset reports build 20260922.6101926. + +Inside, it matches a real Frame: + +- User `steamos` (uid 1000) is in `wheel` (gid 998), and sudoers has + `%wheel ALL=(ALL) ALL`, so sudo asks for the Developer Mode password. +- `sshd_config` includes `sshd_config.d/*.conf`, uses `.ssh/authorized_keys` + plus `AuthorizedKeysCommand /usr/bin/userdbctl ssh-authorized-keys %u`, + and sets `KbdInteractiveAuthentication no` and `UsePAM yes`. So sshd offers + `publickey,password`, the same as the headset. +- `/usr/bin` has `sshd`, `sudo`, `python3` and `podman`. + +Get just the root filesystem without unpacking the whole 5.8 GB image (the +partition's start and size, in sectors, come from the table above): + +```sh +bzcat steamframe-oobe-repair-*.img.bz2 | tail -c +$((655394 * 512 + 1)) | head -c $((10485760 * 512)) > rootfs-A.img +``` + +A Mac can't mount btrfs; a Linux machine or VM can (`mount -o ro -t btrfs`). + +## Running it without the headset + +The image can't boot in a generic virtual machine: its kernel and bootloader +are built for the Frame's Qualcomm Snapdragon 8 Gen 3 (**inferred**; not +attempted). Its **userland** runs fine on any ARM64 Linux, which covers +anything that talks to the Frame over SSH. + +[`tests/frame-container/frame-image.sh`](../tests/frame-container/frame-image.sh) +extracts `rootfs-A`, mounts it read-only with a throwaway writable layer, and +starts the image's own `sshd` on port 2223 (user `steamos`, a test password; +`systemctl` only records requests). On a Mac, run it in Colima's ARM64 VM (see +[tests/frame-container/README.md](../tests/frame-container/README.md)). +**Verified 2026-09-27:** the iPhone app paired with it by password (the image's +sshd logged `Accepted password`, then `Accepted publickey … ED25519`), ran +Frame Control's server on the image's Python, and the image's sudo rejected a +wrong power password and passed the right one to `systemctl`. Without the +Frame's hardware there's no SteamVR, Steam client, battery or Lepton, so those +parts stay untested this way. + +## Holo Core aarch64 (Valve and Collabora) + +The ARM64 port of Arch Linux that the Frame's SteamOS is built on, published as +a preview in July 2026 ([Collabora's announcement](https://www.collabora.com/news-and-blog/news-and-events/building-an-arch-linux-aarch64-port-for-holo-core.html)). +It's a base system and build environment, not the Frame's OS: + +- Source: `https://gitlab.steamos.cloud/holo/holo-core-aarch64-preview` +- Packages: `https://holo-packages.steamos.cloud/holo-core-aarch64-preview/mash-20251118` +- Container: `registry.gitlab.steamos.cloud/holo/holo-core-aarch64-preview/base-devel:latest` + (1.7 GB; `/etc/os-release` says "Holo core Aarch64 port (preview)"; `pacman` + installs OpenSSH 10.2, Python 3.13 and sudo from its repositories. Checked 2026-09-27.) + +[`tests/frame-container/Dockerfile`](../tests/frame-container/Dockerfile) builds a +lighter Frame stand-in on it (a `steamos` user with a password and sudo, sshd +with keys and passwords), handy when you don't have the 4 GB image. diff --git a/docs/scripts.md b/docs/scripts.md index 464c5a6..ad11ebd 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -93,7 +93,8 @@ controls to place each panel. See [docs/panels.md](panels.md). | `scripts/install-apps.sh` | Mac → Frame | Install Flatpaks (Remmina, Moonlight, …) on the Frame over SSH as `--user` (**verified** with Remmina) | | `scripts/paste-to-frame.sh` | Mac → Frame | Send the Mac clipboard (or stdin) to the Frame clipboard (**verified**) | | `scripts/install-apk.sh` | Mac → Frame | Install APKs, each as its own persistent Lepton instance with a Steam library shortcut (`--dev`: old ADB path into Lepton Development) (**verified**; see [docs/apks.md](apks.md)) | -| `scripts/panel-on-frame.sh` | Mac → Frame | Start an app as its own floating VR panel, outside the desktop (**verified**: overlays created; in-headset placement not yet checked) | +| `scripts/panel-on-frame.sh` | Mac → Frame | Start an app as its own floating VR panel, outside the desktop (**verified**, including `mac-screen` in the headset) | +| `scripts/mac-cursor-ring.lua` | Mac | Hammerspoon script: a ring around the Mac pointer so it shows in the VNC mirror (**verified**) | | `scripts/run-on-frame.sh` | Mac → Frame | Start an app on the headset desktop, e.g. `mac-screen` opens Remmina straight into the Mac (**verified**) | | `scripts/frame-ui.sh` | Mac | Start the Frame Control web UI (`ui/server.py`) and open it (**verified**) | | `scripts/apk-catalog.sh` | Mac | Refresh the rated F-Droid catalogue that Frame Control's Android section shows (**verified**) | diff --git a/docs/sideloading.md b/docs/sideloading.md index 6ff60e4..53a0ad8 100644 --- a/docs/sideloading.md +++ b/docs/sideloading.md @@ -21,7 +21,8 @@ desktop app, which knows where a dropped folder lives; in a plain browser, zip it.) A dialog shows: - **Name**: what Steam shows. Steam uses the title id as the name, so it's - limited to letters, digits, `_` and `-`; the dialog shows the result. + limited to letters, digits and `_`, and can't start with a digit; the + dialog shows the result. - **Launches**: the program picked to start the game, with the other candidates in the list. - **Runtime**: picked from the program, see below. Windows programs can switch @@ -133,10 +134,15 @@ splits that string is **not checked**. with the same rule, because `scp -r` would follow a link out of the folder and upload whatever it points at. - Installs run one at a time, and Remove is refused while one runs. -- The title id is limited to `[A-Za-z0-9_-]`, at most 64 characters. Valve's - scripts pass it to a shell (`steamos-delete` runs `rm -r` on it). Valve's +- The title id is limited to letters, digits and `_`, doesn't start with a + digit (one that would gets `_` in front), and is 2 to 64 characters. That's + what Steam's `create-shortcut` accepts: on the Frame it refused + `fc-smoke-exe` with `missing/invalid arguments` and registered the same + program as `FCSmokeProbe` (2026-09-27, BUILD_ID 20260922.6101926), and + Valve's client only allows `^[A-Za-z_][A-Za-z0-9_.]+$`. Valve's scripts + also pass the id to a shell (`steamos-delete` runs `rm -r` on it). Valve's reserved sideload names (`steam`, `steamvr`, and their `deckard` forms, - which would replace the Steam client itself) get `-game` added. + which would replace the Steam client itself) get `_game` added. - Nothing needs `sudo`; everything goes to your home folder on the Frame. - In the app, a dropped folder is read from its local path by the app's own server, which only accepts requests from its own page (see diff --git a/docs/ssh.md b/docs/ssh.md index 93aac3b..7833bb5 100644 --- a/docs/ssh.md +++ b/docs/ssh.md @@ -102,6 +102,18 @@ started. `curl http://:32000/properties.json` shows whether the servic `~/.ssh/authorized_keys` lives under `/home`, which SteamOS keeps across OS updates (inferred from Deck; the Frame uses the same A/B image scheme). +## From an iPhone or iPad + +The iPhone app ([iphone.md](iphone.md)) makes its own ed25519 key and adds it +with the Developer Mode password, once, over a password login; the Frame's sshd +offers `publickey,password` (OpenSSH 9.7p1, keyboard-interactive off). It can't +use the devkit pairing above: that installs an RSA key, and the Swift SSH +library signs RSA only with SHA-1, which OpenSSH 8.8 and later refuse by default. +The app pins the Frame's host key on first use and asks you to pair again if it +changes. **Verified 2026-09-27** against the Frame's recovery image +([recovery-and-images.md](recovery-and-images.md)); on the headset, the add-the-key-yourself +route was used. + ## Keeping `sshd` enabled across updates - **Frame**: SSH is tied to the Developer Mode toggle, so it should survive diff --git a/docs/streaming.md b/docs/streaming.md index 42222b9..707544e 100644 --- a/docs/streaming.md +++ b/docs/streaming.md @@ -1,9 +1,11 @@ # Screen and desktop streaming -This covers two directions: +This covers three directions, plus input: - **A. Frame → Mac**: see and control the headset from the Mac. - **B. Mac → Frame**: use the Mac's desktop inside the headset. +- **C. iPhone → Frame**: mirror the phone inside the headset. +- **Input**: type and point in the Frame from the Mac or iPhone. The confidence labels are the same as in [ssh.md](ssh.md). @@ -32,7 +34,7 @@ flat 2D desktop streaming into a window on the Frame's Linux desktop. | Option | Setup | Confidence | Verdict | |---|---|---|---| -| **macOS Screen Sharing (VNC) → Remmina on the Frame** | **Mac:** System Settings → General → Sharing → Screen Sharing on → (i) → enable "VNC viewers may control screen with password". **Frame:** `./scripts/install-apps.sh remmina` from the Mac, then open Remmina in the headset and connect to `vnc://.local` | **Inferred.** Remmina is on Flathub for **aarch64** with VNC and RDP ([Flathub](https://flathub.org/apps/org.remmina.Remmina)). The Frame desktop runs Flatpaks ([UploadVR](https://www.uploadvr.com/flatpaks-open-source-steam-frame/)). macOS VNC is built in. | **Recommended.** Nothing to install on the Mac, and it's easy to set up. Latency is fine for productivity but not for games. You'll type the Mac's hostname once in Remmina on the headset, then save the profile. To avoid even that, the script can pre-seed a Remmina profile over SSH (see below). | +| **macOS Screen Sharing (VNC) → Remmina on the Frame** | **Mac:** System Settings → General → Sharing → Screen Sharing on → (i) → enable "VNC viewers may control screen with password". **Frame:** `./scripts/install-apps.sh remmina` from the Mac, then open Remmina in the headset and connect to `vnc://.local` | **Verified 2026-09-27** (Frame BUILD_ID 20260925.6191901, macOS 27.0), in its own panel via `panel-on-frame.sh mac-screen`. Remmina is on Flathub for **aarch64** with VNC and RDP ([Flathub](https://flathub.org/apps/org.remmina.Remmina)). The Frame desktop runs Flatpaks ([UploadVR](https://www.uploadvr.com/flatpaks-open-source-steam-frame/)). macOS VNC is built in. | **Recommended.** Nothing to install on the Mac, and it's easy to set up. Noticeable lag, even at lower Remmina quality settings on a good 5 GHz link, where neither Wi-Fi nor the Frame's CPU was the bottleneck. Usable for reading and coding, but not for games. You'll type the Mac's hostname once in Remmina on the headset, then save the profile. To avoid even that, the script can pre-seed a Remmina profile over SSH (see below). | | Sunshine (Mac) → Moonlight (Frame Flatpak) | `brew install` Sunshine on the Mac, then `./scripts/install-apps.sh moonlight` | Moonlight Flatpak supports **aarch64** ([Flathub](https://flathub.org/apps/com.moonlight_stream.Moonlight)). **Sunshine on macOS is poorly supported**: install problems on Apple Silicon/Sequoia, and no virtual gamepads ([LizardByte discussion #777](https://github.com/orgs/LizardByte/discussions/777)). | Try it if VNC is too laggy. Expect some friction. | | Steam Remote Play with the Mac as host | Steam on the Mac, Steam Link/Remote Play on the Frame | macOS-hosted Remote Play is reported broken or flaky in 2024–2026 ([Steam discussion](https://steamcommunity.com/groups/homestream/discussions/1/574921459914429988/)) | Not recommended. It's only for games, if it works at all. | | Immersed / Virtual Desktop | Vendor apps | Immersed has a Mac agent but no known Frame client. Virtual Desktop's developer said he'd "try" to port it ([NewsBreak](https://www.newsbreak.com/news/4892834783961-virtual-desktop-dev-says-he-ll-try-to-bring-the-app-to-steam-frame)). | Not available as of 2026-09-25. Check again later. | @@ -45,20 +47,70 @@ them on the Frame in DeoVR instead: see [vr-video.md](vr-video.md). `scripts/install-apps.sh remmina --vnc-host .local` writes `~/.var/app/org.remmina.Remmina/data/remmina/mac-screen-sharing.remmina` on the Frame over -SSH. The profile then appears in Remmina's list, and you just click it. You'll -still be asked for the VNC password in the headset the first time, unless you -choose to save it. Remmina stores passwords encrypted with a per-install key, -so the script doesn't try to write the password. (The Remmina file format is -standard; the Flatpak data path is inferred.) +SSH. The profile then appears in Remmina's list, and you just click it. It +scales the Mac's desktop to fit the window (`scale=1`, `viewmode=1`). Without +that, Remmina shows a Retina Mac's native pixels 1:1, so you see a zoomed-in +corner. (Verified 2026-09-27.) -## Input and text entry without the virtual keyboard +**Expect a Mac login prompt, not the VNC password.** macOS offers Apple's own +authentication (RFB security type 30) ahead of plain VNC auth (type 2), and +Remmina picks it. So Remmina asks for your **Mac account name and login +password**; the "VNC viewers may control screen" password isn't used. To store +the password without typing it in the headset, run on the Frame: -- **A Bluetooth keyboard and mouse** paired to the Frame is the obvious way to - avoid the virtual keyboard. Road to VR says there are "only a few things - you'd actually want to do" on the Linux desktop unless you connect a - keyboard and mouse. - (Pairing a BT keyboard on the Frame is inferred from SteamOS; not verified.) -- **Clipboard from the Mac**: `scripts/paste-to-frame.sh` (see - [file-transfer.md](file-transfer.md#clipboard)). +```sh +printf '%s' "$PASSWORD" | flatpak run org.remmina.Remmina \ + --update-profile ~/.var/app/org.remmina.Remmina/data/remmina/mac-screen-sharing.remmina \ + --set-option password +``` + +Remmina encrypts it into the profile with its own key, because there's no +secret service in the SSH session. (Verified 2026-09-27.) + +### The Mac's cursor + +The mirror doesn't show the Mac's pointer, with either `showcursor` value. +macOS keeps the pointer out of the picture it sends, and Remmina's cursor mode +draws the cursor shape only at the Frame's own pointer, which doesn't follow +the Mac trackpad. `scripts/mac-cursor-ring.lua` works around this: a +[Hammerspoon](https://www.hammerspoon.org/) script that draws a ring around the +Mac pointer as a real window, so it's part of the mirrored picture. Setup is in +its header. (Verified 2026-09-27.) + +Going the other way, pointing a controller at the panel moves the Mac's mouse, +because Remmina forwards input (`viewonly=0`). + +## C. Show the iPhone's screen inside the Frame + +iOS only shares its screen two ways: **AirPlay** (Screen Mirroring in Control +Centre) or a **ReplayKit broadcast extension** in an app. Nothing else can +capture it. + +| Option | What it takes | Confidence | Verdict | +|---|---|---|---| +| **UxPlay** (an open-source AirPlay receiver) on the Frame | Build it for aarch64 (no Flathub package; there's a Snap and distro packages), run it in `~` or a podman container, and advertise it over mDNS. The iPhone *and* the Mac then see "Frame" in Screen Mirroring, with nothing to install on either | **Inferred.** It runs on ARM64 Linux such as the Raspberry Pi ([UxPlay](https://github.com/FDH2/UxPlay)). Not tried on the Frame: needs mDNS registration and its ports (7000, 7001, 7100 and a UDP range) reachable | **Recommended to try first.** It's the only receiver-side option, and it covers the Mac too. The window shows in the Frame's Linux desktop panel | +| A broadcast extension in Frame Control | ReplayKit sends the screen to a small extension (50 MB memory limit), which encodes H.264 and sends it through the app's SSH tunnel to the page, shown the same way as the Frame's live view in reverse | **Inferred** from Apple's ReplayKit docs | Full control and no network setup, but several days' work, and the picture only shows where Frame Control's page is open in the headset | + +## Input: type and point in the Frame from the Mac or iPhone + +**Verified 2026-09-27** on the headset: `steamos` is in the `input` group and +`/dev/uinput` is `crw-rw-r-- root input`, so **our own code can create a +virtual keyboard and mouse without sudo**. The Frame has no `python-evdev`, +`ydotool`, `wtype` or KDE Connect; `kwin_wayland` and `plasmashell` run only +while the desktop panel is open in the headset. + +| Option | Mac | iPhone | Notes | +|---|---|---|---| +| **A uinput keyboard and mouse in Frame Control's server** | ✓ | ✓ | **Recommended.** The server opens `/dev/uinput` with `ctypes` (standard library only) and the page sends key and pointer events through the tunnel it already has. On the phone: a trackpad area (drag to move, tap to click, two fingers to scroll) and the iOS keyboard for typing. On the Mac: a "control the Frame" mode that captures the keyboard and pointer (Esc to release). Uinput devices look like real hardware to the kernel, so libinput, KWin and gamescope should take them; [frame-voice](https://github.com/DeeJanuz/frame-voice) already types into a Frame through a uinput keyboard. **Untested**: which surfaces in VR (desktop panel, SteamVR dashboard, games, Android apps in Lepton) accept the pointer. About a day or two of work | +| **Bluetooth keyboard and mouse** | – | – | Real hardware paired in SteamOS settings. The iPhone can't pretend to be a Bluetooth keyboard: iOS won't advertise the HID service ([Apple forums](https://developer.apple.com/forums/thread/733916)) | +| **Deskflow** (formerly Input Leap / Barrier) | ✓ | – | Moves the Mac's own mouse and keyboard onto the Frame's screen edge. Flathub has an aarch64 build ([Flathub](https://flathub.org/apps/org.deskflow.deskflow)); on Wayland it needs the InputCapture/libei portal, and only works while Plasma is running. No iPhone client | +| **KDE Connect** | ~ | ✓ | Its iOS app has a remote touchpad and keyboard, but the Frame would need KDE Connect installed (not on Flathub; `pacman` on a read-only root). More moving parts than the uinput route | +| **Remmina / Steam Link / RDP** | ✓ | – | Input only reaches the streamed session, not the headset's own apps | + +Other ways to get text in: + +- **Clipboard from the Mac**: `scripts/paste-to-frame.sh`, or Frame Control's + clipboard box (see [file-transfer.md](file-transfer.md#clipboard)). Needs + the desktop panel open. - **RDP session**: Windows App syncs the clipboard with xrdp, but only inside that RDP session. diff --git a/docs/support-matrix.html b/docs/support-matrix.html new file mode 100644 index 0000000..8051ce1 --- /dev/null +++ b/docs/support-matrix.html @@ -0,0 +1,190 @@ + + + + + +Frame Control 0.3.1: features by OS + + + +
+

Frame Control 0.3.1: features by OS

+

Which features are built for each OS, and which were tested against a real Steam Frame + (SteamOS 0.3.0, build 20260922.6101926). Status as of 26 September 2026, for + PR #2 (0.3.1). Every build now bundles its own + Python 3.12, adb and CA certificates, so nothing else needs installing (only ssh on Linux, + plus the system adb on arm64 Linux).

+ +

Builds and test machines

+
+

macOS

+
Built
Apple Silicon (arm64): .dmg, .zip. No Intel build.
+
Signing
Ad-hoc signed, not notarised
+
Tested on
Apple Silicon Mac, macOS 26, using a local 0.3.1 build with the bundled Python and adb. All 15 calls it made to the Frame at startup returned OK.
+
+

Windows

+
Built
x64: NSIS installer .exe and .zip
+
Signing
Unsigned. SmartScreen shows a warning.
+
Tested on
Windows 11 x64 VM. Real-Frame results below are from 0.3.0. The 0.3.1 installer from CI installs cleanly (31 s) and reinstalls over itself (38 s). The server starts on the bundled Python, and HTTPS to Steam and F-Droid works. The Frame went offline before its 0.3.1 run on the headset.
+
+

Linux

+
Built
x86_64 and arm64: AppImage and .deb
+
Signing
n/a
+
Tested on
x86_64 Ubuntu 26.04 with no adb and no clipboard tools, using the 0.3.1 AppImage under Xvfb with the bundled Python and adb. The arm64 builds and the .deb packages weren't run; the arm64 package was only checked to contain an ARM Python.
+
+
+ +

Features

+
+ Tested: worked against the real Frame on that OS + Partial: only part of the feature was tested (see note) + Automated: covered by CI tests on that OS, not tried on a real Frame + Built: in the build, not tested + Not built +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FeaturemacOSWindowsLinux
Connection
Set Up Connection
Finds the Frame, writes the frame SSH alias, copies your key using the Frame's password. macOS runs connect.sh in Terminal; Windows and Linux run frame_connect.py.
Partial
Existing alias used, script not re-run
TestedTested
Shared SSH connection
A single SSH connection is reused, so each request takes about 0.3 s. Windows OpenSSH can't do this, so there each request opens its own connection (about 0.5 to 1 s).
TestedNot built
OpenSSH limitation
Tested
Headset view
Capture headset view
The left eye or both eyes as the lenses show them, saved as PNG
TestedTestedTested
Capture desktop panel
gamescope's flat layer
TestedTestedTested
Live view
720p H.264 at about 30 fps, decoded with WebCodecs
TestedTestedTested
Headset screenshots
Browse the screenshots you took with Steam's shortcut, and save them to ~/Pictures/SteamFrame
TestedPartial
Listed (5 found); saving not tried
Partial
Listed with thumbnails; saving not tried
Status
Battery and charging
Percentage, watts, time to full or empty, charger type, temperature
TestedTestedTested
System status
Storage, memory, temperature, Wi-Fi, uptime, running services
TestedTestedTested
Volume and mute
TestedPartial
Read only
Partial
Read only
Games
Owned games with Frame ratings
Verified, Playable, Unsupported or Unknown
TestedTestedTested
Install a game on the Frame
Uses the headset's Steam client, with live progress
TestedBuiltBuilt
Store search, Buy, Store on Frame
TestedBuiltBuilt
Library shelf and Play button
BuiltBuiltBuilt
Android apps
Installed Android apps list
TestedTestedTested
F-Droid catalogue search
About 4,500 apps with Frame ratings, bundled with the app
TestedTestedTested
Install, launch, stop, test, remove an app
Each app runs as its own Lepton instance, using the bundled adb. APK files are read by a built-in parser (no aapt2) that matched aapt2 on 9 F-Droid APKs.
Tested
Diary: read, install, launch, test, remove
BuiltPartial
Launch and stop
Report an APK
Reports are saved on your computer; the shared database is maintainer-only
AutomatedAutomatedAutomated
Android display settings
Resolution, UI scale, text size
Tested
Density and text size set, then reset
BuiltPartial
Read over the bundled adb
Transfer
Send files to ~/Downloads
Test files had non-English characters in their names (é, ✓). macOS and Linux copy with rsync; Windows uses scp.
TestedTestedTested
Drop an APK to install it
TestedBuiltBuilt
Send text or clipboard to the Frame
Needs the headset desktop open. The app reads your clipboard through Electron, so no extra tools are needed.
Tested
Reading the clipboard retested in 0.3.1
Partial
Reached the Frame; desktop was closed
Partial
Clipboard read with no xclip; Frame desktop was closed
Flatpak install and remove
BuiltBuiltBuilt
One-click tools
SSH or SFTP in a terminal
macOS: Terminal. Windows: cmd. Linux: GNOME Terminal, Konsole, xterm and others.
BuiltBuiltBuilt
Steam Link
BuiltBuiltBuilt
Remote desktop
macOS: Windows App. Windows: Remote Desktop. Linux: Remmina or FreeRDP.
BuiltBuiltBuilt
Sleep, restart, shut down
Opens a terminal because SteamOS asks for the sudo password
BuiltBuiltBuilt
App
Local server test suite
Runs in GitHub Actions on every push (Python 3.12 on macOS and Windows, Python 3.13 on Ubuntu), including the APK reader tests
AutomatedAutomatedAutomated
Mac or PC wording
The UI says Finder or File Explorer, and Mac or PC, to match your system
TestedTestedTested
+ +

Notes

+
    +
  • Tested means the app, running on that OS, got a successful response from the real Frame: for example, a PNG from a capture, 868 owned games, or the Android apps listed.
    • +
  • The Windows VM tests ran in its desktop session. ssh.exe hangs when it's started from a remote SSH session, but a normal desktop user won't hit that.
    • +
  • The macOS test from 25 September also covered the capture shown when the headset is in standby, input validation, and using the clipboard with the headset desktop open.
    • +
  • Everything marked Built runs a command that works on its own. It just hasn't been tried end to end from the app on that OS yet.
    • +
+
Frame Control is an unofficial tool, not made by Valve. MIT licence.
+
+ + diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..b06577d --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,150 @@ +# Testing + +Frame Control is tested in three layers, from fast and fake to slow and real. +A fourth, a SteamOS VM, may come later ([issue #6](https://github.com/saphid/steam-frame/issues/6)). + +| Layer | Runs | Needs | Covers | +|---|---|---|---| +| Unit tests (`tests/*.py`) | `python3 -m unittest discover -s tests` | Nothing | Parsing, validation, request guards; SSH and HTTP are mocked | +| Fake Frame (`tests/e2e`) | `scripts/e2e.sh` | Linux with Docker | The real server and scripts against a container that behaves like a Frame | +| Headset smoke test | `scripts/frame-smoke.sh` | A Frame on the `frame` alias | Install, launch and remove on the real device, recorded with its BUILD_ID | + +## Unit tests + +```sh +python3 -m unittest discover -s tests +``` + +About 120 tests, a few seconds, on Python 3.9 and newer. GitHub Actions runs +them on macOS, Windows and Linux. They don't pick up `tests/e2e`. + +## The fake Frame + +`tests/fakeframe/` builds a container that stands in for the headset, and a +second one for the computer Frame Control runs on. `scripts/e2e.sh` builds +both, starts them with `docker compose`, runs `tests/e2e` in the host +container and takes everything down, exiting with the tests' status: + +```sh +scripts/e2e.sh # everything, about 2 minutes plus the first build +scripts/e2e.sh test_titles # one module +scripts/e2e.sh test_faults.Faults.test_disk_full # one test +FAKEFRAME_KEEP=1 scripts/e2e.sh # leave it running afterwards +``` + +It needs a Linux host with Docker and `docker compose`, and zsh. The images +are `fakeframe-frame` and `fakeframe-host`; the compose project, network and +volumes are `fakeframe-e2e*`. CI runs it on a native arm64 runner +(`ubuntu-24.04-arm`, the `e2e` job in `.github/workflows/checks.yml`). + +The host container exists because OpenSSH reads `~/.ssh/config` from the +passwd home directory, not `$HOME`. There, `ssh frame` reaches the fake Frame +through the same `Host frame` block `ui/frame_connect.py` writes, the +repository is mounted read-only at `/repo`, and each test module starts the +real `ui/server.py` (Python 3.9) and talks to it over HTTP with the headers +its guards want. + +### What's real and what's fake + +| On the fake Frame | | +|---|---| +| Arch Linux (`archlinux:base`, or Valve's Holo Core aarch64 preview on arm64), user `steamos`, `/etc/os-release` with BUILD_ID 20260922.6101926 | Real OS, Frame's identity | +| `sshd` with key and password logins, `rsync`, `python3` | Real | +| Valve's steamos-devkit-service on port 32000 and its hooks, vendored unmodified in `tests/fakeframe/steamos-devkit-service` | Real; only its `dbus` import (for mDNS through systemd-resolved) is a stand-in that logs the registration | +| Valve's devkit-utils, copied over by Frame Control itself | Real | +| **fakesteam**: `~/.steam/steam.pid`, `steam.token` and the `steam.pipe` FIFO; answers `approve-ssh-key`, `create-shortcut`, `run-game`, `list-shortcuts` and `delete-shortcut` with the response files devkit-utils waits for; takes `steam://rungameid`, `install` and `store` URLs | Fake | +| DevTools on `127.0.0.1:8080` with a `SharedJSContext` target. The JavaScript Frame Control sends runs for real in Node against stand-in `SteamClient`, `appStore` and `downloadsStore` objects (`cef_shim.js`), so async functions, optional chaining and `Map`s behave as in Steam's CEF | The JS engine is real; the objects are fake | +| `steam`, `wpctl`, `flatpak`, `podman`, `nmcli`, `qdbus6`, `gamescopectl`, SteamOS's `steamos-enable-sshd` helper, and Lepton's launcher | Stubs that record their calls | +| Battery, charger and thermal zones under `/sys/class` | Files the supervisor writes. `/sys` is read-only in a container and Docker's AppArmor profile refuses writes under it, so each folder is a volume mounted twice: over `/sys/class/...` for `frame_status.py` to read, and under `/var/lib/fakeframe/sys` for the supervisor to write | +| `vrserver` and `plasmashell` | Renamed `sleep` processes, so the status page and the clipboard find them | + +Every fake behaviour copied from the device has a comment citing the doc or +observation and the BUILD_ID it came from; anything not seen on a headset is +marked as a guess. The fake keeps its state in `/var/lib/fakeframe/state.json` +(shortcuts, devkit titles, compat tool mapping, launches, pairing requests, +Lepton instances, volume, Flatpaks, clipboard) and logs stub calls to +`calls.jsonl` beside it. + +Native programs really run: a launched aarch64 title executes on an arm64 +host, and an x86-64 one on x86-64 (the container shares the host's kernel). +Proton titles are recorded with the command Steam would run, not run. + +### Fault switches + +`fakeframe-ctl` works over SSH (`ssh frame fakeframe-ctl help`) and from the +host container (`FAKEFRAME_CTL=http://fakeframe:9999`), so a test can flip a +switch while SSH is down: + +| Command | Effect | +|---|---| +| `pairing on\|off` | Steam's **Pair new host** screen open or not; off gives the device's 403 text | +| `answer approve\|deny\|timeout` | How the pairing prompt is answered | +| `steam on\|off` | Steam client running (pid file, pipe, DevTools) | +| `sleep on\|off` | Headset asleep: ports 22 and 32000 accept and never answer, so SSH times out | +| `sshd on\|off` | sshd stopped: new connections are refused, open ones stay | +| `devkit-service on\|off` | Port 32000 closed | +| `disk-full on\|off` | Fills the small (64 MB) filesystem on `~/devkit-game` | +| `runtime NAME installed\|missing` | Proton, the Steam Linux Runtimes, Lepton | +| `battery KEY=VALUE...` | e.g. `capacity=15 status=Discharging current_now=-900000` | +| `keys harness\|none`, `authorized-keys` | Set or read `~/.ssh/authorized_keys` | +| `reset`, `state`, `calls [TOOL]` | Start over; read the state and call log | + +### What the fake can't show + +- Rendering: the headset view, desktop capture content, live video, SteamVR, + gamescope and panels. The capture stub returns a placeholder PNG. +- Proton and FEX: whether a Windows or x86-64 program actually runs. +- Android: there's no Android in the Lepton stand-in, so no ADB, display + settings, probes or app crashes. +- The real Steam client's UI and anything it does that isn't modelled, and + mDNS discovery. +- `sudo` and the power buttons, Tailscale, and the Windows and macOS sides of + the app (the host container is Linux, so the `rsync` paths are tested and the + `scp` fallback isn't). + +## Headset smoke test + +```sh +scripts/frame-smoke.sh # needs `ssh frame` to work without a password +scripts/frame-smoke.sh --pair # also pairs a throwaway key: approve it in the headset +``` + +It checks `properties.json` and the status, then installs, launches and +removes three tiny titles built from bytes by `tests/smoke/tiny_programs.py` +(an ARM64 and an x86-64 static Linux program that sleep for ten seconds, and +an x86-64 `.exe` that exits at once). A launch passes only with fresh evidence: +the ARM64 program running, the `.exe` started (its process or Steam's log), +and the x86-64 program running or Steam logging that its runtime isn't +installed, which is what the Frame does today. Steam's log lines about each +title are kept. + +Everything it installs is removed again, also after a failure: the titles and +their Steam shortcuts, a paired key, and `~/devkit-utils` if it wasn't there +before (if it was, it stays, synced to this checkout as Frame Control always +does). A cleanup that fails counts as a failed step. Results go to +`tests/smoke/results/