steam-frame/saphid--frame-control
1
0 Fork 0
mirror of https://github.com/saphid/frame-control.git synced 2026-10-06 06:00:33 +02:00
Code Issues Packages Projects Releases 6 Wiki Activity

Frame Control: Mac app, web UI, Android and Steam tooling

Browse Source 
Package the Frame Control web UI as an installable Electron Mac app and
bring in the tooling built alongside it.

- app/: Electron wrapper that starts ui/server.py on a free loopback port,
  hardened window (sandbox, no navigation, runAsNode fuse off), login-shell
  PATH so Homebrew tools work from Finder, first-run offer to run
  connect.sh, ad-hoc signed DMG/zip via electron-builder.
- ui/: headset view (OpenVR screenshots), device status, library, Steam
  "Get games" (owned games, install, store search), Android apps as
  persistent Lepton instances with a rated F-Droid catalogue and a private
  compatibility database, Android display controls over ADB, file and
  clipboard transfer, Flatpaks, remote and power actions.
- apk-catalog/, compat-db/, frame/: catalogue build pipeline, Lakebed
  capsule for compatibility reports, Frame-side launchers.
- tests/ and CI: server guard and validation tests plus Steam helper tests,
  run on Python 3.9 with script and app syntax checks.
- Docs: README leads with the Mac app; new Android, panels, Steam games and
  field-notes docs; security notes on LAN-exposed ADB ports.

Screenshot values for the headset's IP and Wi-Fi name are placeholders.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
saphidandClaude Opus 5.5 committed 2026-09-25 22:21:20 +10:00
1 parent 6ccf562756
commit d4486a7681
56 files changed
+19298 -11
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/apks.md
+300
View File
@@ -0,0 +1,300 @@
# Installing APKs (Lepton)
The confidence labels are the same as in [ssh.md](ssh.md). Android apps run
in **Lepton**, Valve's Waydroid-based container. Lepton is built for games,
not general Android use
([GamingOnLinux](https://www.gamingonlinux.com/2026/09/lepton-from-valve-to-run-android-games-on-linux-is-now-open-source/)).
## Install from the Mac: one app, one Lepton instance (verified 2026-09-25)
Use Frame Control's **Android apps** section (search, Install, Test, Report), drop
an `.apk` on **Send to Frame**, or:
```sh
./scripts/install-apk.sh some-app.apk # own instance, Steam shortcut
python3 ui/frame_android.py list|launch|stop|remove|probe <package>
```
Each APK becomes its own app, the way T3 Code is set up (see the instance
section below), instead of going into Lepton Development:
1. `aapt2` reads the package, label, version, ABIs and icon. APKs that need
API > 30 or have no `arm64-v8a` build are refused.
2. The APK, `frame/android/lepton-app.sh` (as `launch.sh`), `instance.id`,
`meta.json`, the icon and the `lepton-show-flatscreen` marker go to
`~/Applications/Android/<package>/` on the Frame.
3. A non-Steam shortcut is added through Steam's CEF debug port
(`frame/android/steam_shortcuts.py`), with no Steam restart.
4. Launching the shortcut runs Lepton directly with `SteamAppId` set to the
instance id (`2800000000 + crc32(package) % 70000000`). That's a
"steamlaunch" context, so app data in `compatdata/<id>/internal` survives
restarts and updates, and each app gets its own SteamVR panel. Several can
run at once alongside Lepton Development, each in its own container
(`lepton-steamlaunch-<id>`, ADB on 5556, 5557, …).
Verified with AntennaPod and Tabletop Tools: installed in about 7 s, launched
from the shortcut, stopped, and relaunched with their data intact. ADB and
Lepton Development aren't involved.
`--dev` keeps the old path: ADB into Lepton Development over an SSH tunnel
(first free Mac port from 15555), which starts Lepton Development if needed.
Apps installed that way are deleted when it exits (see below). No pairing or
"Allow debugging?" prompt is needed for either path.
Lepton Development must be installed once. Over SSH,
`ssh frame 'steam steam://install/3056000'` queues it, but the install still
needs to be confirmed or started in the headset.
## Installed apps disappear when Lepton Development closes (verified 2026-09-25)
Lepton Development runs in a throwaway "dev" context. When it exits for any
reason (you close it, or it crashes), the launcher script
`~/.local/share/Steam/steamapps/common/Lepton/lepton` calls
`clear_baked_app_data "non steamlaunch container"` and **deletes every app
installed over ADB**. The journal shows `Clearing baked app data due to non
steamlaunch container`, and `pm list packages -3` is empty afterwards.
The script skips the wipe when `LEPTON_NO_CLEANUP` is set
(`liblepton/liblepton.sh`, `clear_baked_app_data`). To keep your apps, set
Lepton Development's Steam launch options to:
```
LEPTON_NO_CLEANUP=1 %command%
```
(Steam → Library → Lepton Development → Properties → Launch Options.) Inferred
from the script, not yet tested across a restart.
## Which APKs work (verified 2026-09-25, SteamOS build 20260922.6101926)
Lepton is LineageOS 18.1 (`lepton_arm64_only`): Android 11, API 30,
`abilist=arm64-v8a` only, Mesa (Turnip, Adreno 750) with GLES 3.2 and
Vulkan 1.4. About 30 F-Droid apps were installed and opened on the Frame to
check each rule. The results are in the compatibility database (see compat-db/README.md).
**Won't install** (the installer refuses):
| Rule | Seen on device |
|---|---|
| `minSdkVersion` > 30 | `INSTALL_FAILED_OLDER_SDK: Requires newer sdk version #33 (current version is #30)` |
| Native code without `arm64-v8a` (32-bit ARM or x86 only) | `INSTALL_FAILED_NO_MATCHING_ABIS` |
**Crash on launch.** Lepton has no `clipboard` system service, so
`getSystemService(CLIPBOARD_SERVICE)` returns null:
| What | Result |
|---|---|
| **Jetpack Compose UI < 1.11** | Crashes as soon as a Compose screen appears: `null cannot be cast to non-null type android.content.ClipboardManager` in `AndroidComposeView`. Seen with 1.5, 1.6, 1.7, 1.8 and 1.10 apps. |
| Jetpack Compose UI 1.11, 1.12, 1.13 | **Works.** Six apps opened fine, including Aurora Store and NewPipe. |
| Old Compose, but the first screen uses classic Views | Opens (FoCal, Compose 1.3), and crashes only on Compose screens |
| SDL2 apps, including Kivy | Crash: SDL calls `ClipboardManager.addPrimaryClipChangedListener` at start-up |
| Godot 4.3 | Crashes (clipboard cast). Godot 4.6.1 works. |
**Works:** classic Android Views apps, Flutter (2 of 2), libGDX (2 of 2),
Compose 1.11+, Firebase-using apps. React Native: 2 of 3 opened; one
(controlloid) died with SIGSEGV on the Hermes JS thread. A Qt 6 app
(AusweisApp) failed on a missing libc++ symbol.
**Missing pieces:** an app may open but fail when you use one of these:
- No Google Play Services.
- No activity for `VIEW` of web links, `OPEN_DOCUMENT`/`GET_CONTENT` (no file
picker), `IMAGE_CAPTURE`, or text-to-speech. WebView (Chromium 152) is there.
- No Downloads or Contacts providers.
- Missing system services also include `accessibility`, `vibrator`, `phone`,
`print`, `usb`, `nfc` and `autofill`. The declared features lack
`touchscreen.multitouch`, `bluetooth_le` and `telephony`.
- No on-screen keyboard (IME) is installed. How text entry reaches Android
apps in the headset hasn't been checked.
**Lepton itself can crash.** Three times during testing, the graphics HAL
(`android.hardware.graphics.composer@2.1-service`) aborted right after an app
crashed. SurfaceFlinger died, the whole `lepton-dev` container exited, and the
installed apps were wiped (see above). A retry of the same app worked, so it's
intermittent rather than app-specific.
**How good are the predictions?** In a random sample of 12 apps rated "Should
work", all 12 installed, opened and were still running 12 s later (two needed
a retry because Lepton crashed mid-install). "Opened" isn't the same as fully
working: see the missing pieces above.
## Catalogue and compatibility reports
Frame Control's **Android apps** section lists every F-Droid app with a
verdict: Works on Frame, Should work, Might work, Probably crashes, or Won't
work, with the reasons. As of 2026-09-25 that's 30 working, 3,323 should work,
223 might work, 723 probably crash (mostly Compose < 1.11) and 156 won't
install. Each app is rated on its newest version that Lepton can install,
because F-Droid often publishes separate per-ABI builds and the newest is
frequently x86_64. **Install** downloads the APK (SHA-256 checked against the
F-Droid index) and sets it up as its own instance.
No ProtonDB-style database for sideloaded Android apps on the Frame existed
as of 2026-09-25. [Steam Frame Hub](https://verified.steamframehub.com/)
collects community reports for Steam games only and has no public API, and
Valve's "Great on Frame" badges and each Steam app's `recommended_runtime`
(for example `lepton-stable`) also cover Steam games only
([VR.org](https://vr.org/articles/steam-frame-lepton-android-runtime-52-of-130-certified-2026)).
So we keep our own, in a private Lakebed database
(`https://frame-compat.lakebed.app`) that only Frame Control can read or write.
**Test** records whether the app stays up in its own instance, and **Report**
(for any APK, F-Droid or not) records whether it worked, how it was run, where it came from, and notes, each with the SteamOS and Lepton build ids.
A daily job backs it up locally and to Google Drive. See
[compat-db/README.md](../compat-db/README.md) and
[apk-catalog/README.md](../apk-catalog/README.md).
## In-headset app store: F-Droid 1.17 (verified 2026-09-25)
F-Droid 1.23 uses an old Compose and crashes on launch. **F-Droid 1.17.2**, the
newest archived build without Compose, runs, loads the full catalogue (the
first repo update takes about 90s), and can install apps. F-Droid 2.0 uses
Compose 1.12, so it should work, but it hasn't been tried. The catalogue's
Install button for F-Droid installs 1.17.2. To let F-Droid install apps without
a settings prompt, with the tunnel open:
```sh
adb -s $S shell appops set org.fdroid.fdroid REQUEST_INSTALL_PACKAGES allow
```
## Handy commands
Open a tunnel by hand (use any free local port):
```sh
ssh -f -N -M -S /tmp/frame-adb.sock -L 127.0.0.1:15555:127.0.0.1:5555 frame
adb connect 127.0.0.1:15555
S=127.0.0.1:15555
# when done: adb disconnect $S; ssh -S /tmp/frame-adb.sock -O exit frame
```
Then:
```sh
adb -s $S shell pm list packages -3 # installed third-party apps
adb -s $S shell monkey -p <pkg> -c android.intent.category.LAUNCHER 1
# If monkey exits with -5 (it did for T3 Code), start the activity directly:
adb -s $S shell am start -W -n "$(adb -s $S shell cmd package resolve-activity --brief -c android.intent.category.LAUNCHER <pkg> | tail -n 1)"
adb -s $S logcat -d -b crash # why an app died
adb -s $S uninstall <pkg>
adb -s $S exec-out screencap -p > shot.png # the Lepton window
```
## Reaching a Mac service from Lepton (T3 Code v2, verified 2026-09-25)
Lepton runs in podman with `pasta` networking. It has **its own loopback**, so
a port on the Frame's `127.0.0.1` isn't visible as `127.0.0.1` inside Android.
But pasta runs with `--map-gw`, so the **gateway address inside Lepton
(`192.168.1.1` on the home network) maps to the Frame host's loopback**.
T3 Code v2 on the Mac listens only on `127.0.0.1:3873`. To reach it:
1. The LaunchAgent `~/Library/LaunchAgents/frame-t3-tunnel.plist`
keeps `ssh -N -R 127.0.0.1:3873:127.0.0.1:3873 frame` running. launchd
restarts it if it drops. Log: `~/Library/Logs/frame-t3-tunnel.log`.
2. In the app on the Frame, the environment host is `192.168.1.1:3873`.
The app on the Frame was built from the v2 nightly source (fork commit
`d0c468e3`) with `expo prebuild` and `gradlew assembleRelease
-PreactNativeArchitectures=arm64-v8a`, using Homebrew `openjdk@17` and the
`android-commandlinetools` SDK. It's signed with the debug key.
To pair again, issue a one-time code on the Mac and type it into
**Add environment**:
```sh
A="/Applications/T3 Code (V2 Preview).app"
ELECTRON_RUN_AS_NODE=1 "$A/Contents/MacOS/T3 Code (Alpha)" \
"$A/Contents/Resources/app.asar/apps/server/dist/bin.mjs" \
auth pairing create --base-dir "$HOME/.t3-v2" --ttl 15m --label "Steam Frame"
```
The app is deleted whenever Lepton Development closes (see above), so reinstall
it afterwards or set `LEPTON_NO_CLEANUP=1`. The gateway address comes from the Frame's network when Lepton starts. On a
different network, check it with `adb shell ip route` and edit the host.
## Lepton Development forgets apps; give an app its own instance (verified 2026-09-25)
**Lepton Development wipes every installed app when it exits.** Its launcher
logs `Clearing baked app data due to non steamlaunch container`, unless
`LEPTON_NO_CLEANUP` is set. A Steam-style launch (with `SteamAppId` set) is a
"steamlaunch" context and keeps its data:
- App data lives in `STEAM_COMPAT_DATA_PATH/internal/<package>` (symlinked to
`/data/data/<package>`) and survives everything, including APK updates.
- `STEAM_COMPAT_DATA_PATH/baked` is Lepton's Android snapshot. It's rebuilt when
the APK changes, or when the app exits within 30 seconds of starting.
- `STEAM_COMPAT_DATA_PATH` must be under `~/.local/share/Steam` (use
`steamapps/compatdata/<id>`). Only that tree is mounted in the container. Put
it anywhere else and the symlinks dangle, so the app crashes with
`ENOENT` on its first file write.
- Lepton runs apps headless unless an empty `lepton-show-flatscreen` file sits
next to the APK (`STEAM_COMPAT_INSTALL_PATH`).
- Several instances can run at once. Each gets ADB on `5555 + offset`
(`podman ps --format "{{.Names}} {{.Labels.adb_port}}"`).
- Outside Steam, Lepton's `setpgid --foreground` re-exec fails with no
terminal. Set `IS_PARENT=true` and start it with `setsid --wait`.
[`frame/t3code/launch.sh`](../frame/t3code/launch.sh) does all this for T3
Code (context `steamlaunch-2873873873`; ADB is the first free `5555 + n`, e.g. 5557). It needs the Steam client
running (it mounts `~/.steam/steam.pipe`).
**T3 Code in the Steam library (verified 2026-09-25).** The wrapper lives on
the Frame at `~/Applications/T3Code/launch.sh`, with `t3code.apk` and the
flatscreen marker next to it. It's a non-Steam shortcut called "T3 Code"
(shortcut app id `3130509679`). Launching it from Steam gets its own SteamVR
panel, `valve.steam.desktopgame.3130509679`, and opens already paired.
- The shortcut was added without restarting Steam, through Steam's CEF debug
port (`127.0.0.1:8080` on the Frame, target `SharedJSContext`):
`SteamClient.Apps.AddShortcut(name, exe, "", "")`, then `SetShortcutName`
and `SetShortcutStartDir`. `steam steam://addnonsteamgame/<path>` only logged
the URL and added nothing.
- To launch it over SSH: `steam steam://rungameid/13445436691150012416`, which
is `(3130509679 << 32) | 0x02000000`.
- Steam sets `STEAM_FOSSILIZE_DUMP_PATH` for shortcut launches but not
`STEAM_COMPAT_SHADER_PATH`. Lepton then dies with "unbound variable", so the
wrapper sets both.
- To update T3, replace `t3code.apk`. Lepton rebuilds its snapshot on the next
launch, and the pairing survives.
## Crashing apps can take down the headset session (verified 2026-09-25)
Some apps crash Android's graphics composer HAL, which kills the Lepton
container. On 2026-09-25 a batch crash-test also coincided with `steamvr.service`
restarting "on client request", which stops and SIGKILLs `gamescope-session`.
After one of those kills, gamescope crash-looped about once a second on
`rendervulkan.cpp:2181 ... Assertion '!modifiers.empty()'` because it kept
attaching to the SteamVR processes orphaned from the dead session. The fix
without sudo was to `for p in vrdashboard vrcompositor vrserver; do pkill -TERM -x $p; done` (pkill takes one pattern). The
next session then started SteamVR fresh and recovered within a minute.
## Android display: resolution, UI scale, text size (verified 2026-09-25, SteamOS 0.3.0, build 20260922.6101926)
Each running Lepton instance has its own ADB port on the Frame, assigned at
launch: 5555 is Lepton Development, and own-instance apps take the next free
port (T3 Code was on 5557). Find them with `ss -ltn` (5555–5599) and identify
each with `pm list packages -3`. Both instances reported `Physical size:
1920x1080`. Their densities were 180 dpi (Lepton Development) and 213 dpi
(T3 Code), and `settings get system font_scale` returned `null` (1.0).
These all apply immediately and read back as set. Tested on Lepton Development
only:
```sh
adb -s $S shell wm size 2560x1440 # or: wm size reset
adb -s $S shell wm density 240 # or: wm density reset
adb -s $S shell settings put system font_scale 1.15
adb -s $S shell settings delete system font_scale
```
After a `wm` reset, Android writes `font_scale=1.0` back asynchronously, so a
single delete that follows one reads back `1.0`. A second delete a second later
leaves it `null`. Frame Control's **Android display** card does this for you
(`/api/android/display`).
**Inferred, not yet checked in the headset:** a bigger Android resolution with
density scaled to match (2560×1440 at 4/3 of the density) gives sharper text,
because gamescope scales Lepton's surface to fit the same panel. Also unverified:
whether the settings survive the app or its Lepton instance relaunching.
Lepton Development rebuilds its Android data on exit, so there they probably
don't.
docs/how-the-frame-works.md
+73
View File
@@ -0,0 +1,73 @@
# How the Frame is put together (field notes)
What we learnt by poking at a real Frame over SSH. Unless a line says
otherwise, it was **verified 2026-09-25** on SteamOS 0.3.0 (`VARIANT_ID=vr`,
build 20260922.6101926, kernel 6.18, aarch64). Topic docs go deeper. This page
is the map.
## The layer cake
```
SteamVR (vrserver, vrcompositor, vrdashboard) ← renders the room + panels
└─ gamescope --backend openvr ← one SteamVR overlay per app id
├─ Xwayland :0 (Steam UI, games, tagged apps) ← STEAM_GAME property = app id
├─ Xwayland :1 (STEAM_GAME_DISPLAY_0)
├─ Wayland socket gamescope-0
└─ steamos-nested-desktop ← "the Linux desktop" panel
└─ dbus-run-session startplasma-wayland
└─ kwin_wayland 1280×800, Wayland wayland-0, Xwayland :2
└─ plasmashell, Konsole, Dolphin, Flatpaks you open there
Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 3056000
```
## Facts worth knowing
| Fact | Where it matters |
|---|---|
| The desktop is a **nested** Plasma session: runtime dir `/run/user/1000/nested_plasma`, its own D-Bus bus, `WAYLAND_DISPLAY=wayland-0`, `DISPLAY=:2`. A plain `ssh frame app` can't find it. Copy the env from `plasmashell`'s `/proc/<pid>/environ`. | `run-on-frame.sh`, `paste-to-frame.sh` |
| The desktop size is hard-coded to 1280×800 in `/usr/bin/steamos-nested-desktop` (read-only rootfs). | [panels.md](panels.md) |
| gamescope runs with `--virtual-connector-strategy PerAppId`. Each app id becomes a SteamVR overlay `valve.steam.desktopgame.<id>`, which is a panel you can float. Setting `STEAM_GAME` on an X11 window on `:0` makes a new panel. | `panel-on-frame.sh`, [panels.md](panels.md) |
| Handy gamescope root properties on `:0`: `GAMESCOPE_FOCUSABLE_APPS`, `GAMESCOPE_FOCUSABLE_WINDOWS` (triples: window, app id, pid), `GAMESCOPE_FOCUSED_APP`. Read them with `DISPLAY=:0 xprop -root`. | Debugging panels |
| `gamescopectl screenshot <file>` (with `WAYLAND_DISPLAY=gamescope-0`) captures gamescope's flat layer. | Frame Control's capture |
| The **headset view** (both eyes, fully composited: room, panels, dashboard, controllers) comes from OpenVR `IVRScreenshots::RequestScreenshot(VRScreenshotType_Stereo)`. It's callable from `python3` with `ctypes` against `/opt/steamvr/bin/linuxarm64/libopenvr_api.so` as an overlay app. The compositor appends `.png`, writing a 1920×1080 side-by-side image (960×1080 per eye) plus a left-eye preview, in about 0.3s. In standby the frame is blank. `vrcmd --screenshot` and `vrcmd --compositorcmd screenshot_request` wrote nothing, even with `steamvr/rawCapturePath` set. | `ui/frame_vrshot.py` |
| Battery: `/sys/class/power_supply/max1720x_bat_7-36` gives µV/µA (current is positive while charging), `time_to_full_now`/`time_to_empty_now` in seconds, and `temp` in tenths of °C. The charger shows up as `tcpm-source-psy-…` (`type=USB`, `usb_type=C PD [PD_PPS]`), for example 12 V × 1.67 A. | Frame Control's battery card |
| `vrcmd --stats` reports `activity_level` (3 = standby). | Telling whether the headset is being worn |
| The SteamVR dashboard has docking: Float in World, Move, Size, Curvature, controller docking, Theater, Multitasking View. **Inferred** from `/opt/steamvr/resources/webinterface/dashboard/` and not yet driven by hand. | [panels.md](panels.md) |
| SteamVR settings live in `~/.config/openvr/config/steamvr.vrsettings`, not under `~/.local/share/Steam/config/`. `dashboard.lastAccessedExternalOverlayKey` names the last panel you used. | Settings tweaks |
| The Steam client's journal (`journalctl --user`) carries SteamVR system UI lines such as `[Overlays] Created: …` and `vroverlay_uid<appid>`. It's the quickest way to see panels come and go. | Debugging |
| Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale`, `krfb`, `wayvnc`. | Script design |
| Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` |
| `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things |
| Clipboard: Klipper over the nested D-Bus bus (`qdbus6 org.kde.klipper …`). | `paste-to-frame.sh` |
| Lepton listens for ADB on the Frame's loopback `5555`, so tunnel it over SSH. It's Android 11 (API 30), 64-bit ARM only, with no `clipboard` service: Compose < 1.11, SDL/Kivy and Godot 4.3 apps crash on launch. | [apks.md](apks.md), `apk-catalog/` |
| Lepton Development deletes every ADB-installed app when it exits (`clear_baked_app_data "non steamlaunch container"` in `…/common/Lepton/lepton`) unless `LEPTON_NO_CLEANUP` is set. | [apks.md](apks.md) |
| Any APK can run as its own Lepton instance: run `…/common/Lepton/lepton waitforexitandrun -- app.apk` with `SteamAppId` set and `STEAM_COMPAT_DATA_PATH` under `~/.local/share/Steam`. Data persists and each gets its own container and panel. `frame/android/lepton-app.sh`, `ui/frame_android.py`. | [apks.md](apks.md) |
| The Steam client runs with `-cef-enable-debugging`, so its UI answers Chrome DevTools on loopback `127.0.0.1:8080`. The `SharedJSContext` page has `appStore` (owned apps), `downloadsStore` and `SteamClient.*`. `steam steam://install/<appid>` over SSH installs an owned game; when the options dialog shows (state 7), `SteamClient.Installs.ContinueInstall()` accepts it. **Verified 2026-09-25** with Balatro and Broforce. The Frame rating is `steam_hw_compat_category_packed >> 8 & 3`. | [steam-games.md](steam-games.md), `ui/frame_steam.py` |
| Chromium Flatpak 154 has **no immersive WebXR**: `navigator.xr` exists, but `isSessionSupported("immersive-vr")` returns `false`. Web VR180 players (DL8/DeoVR embeds) still play video inline as a flat, pannable view, and their VR button opens a tab on immersiveweb.dev. Forcing it doesn't help. `--enable-features=OpenXR,WebXR --force-webxr-runtime=openxr`, with `/opt/steamvr` and `XR_RUNTIME_JSON` exposed to the Flatpak, still returns `false`. The aarch64 Linux binary has no OpenXR code at all (no `xrCreateInstance`, no loader), even though `chrome://flags` lists `#webxr-runtime` → OpenXR. To watch in 3D, use a native player. Started with `--remote-debugging-port=9222`, Chromium answers DevTools on loopback. **Verified 2026-09-25**, BUILD_ID 20260922.6101926. | Web video, [panels.md](panels.md) |
| Power actions need `sudo`, which asks for the Developer Mode password over SSH. | Frame Control's power buttons |
## Debug recipes
```sh
# Which panels (app ids) exist right now?
ssh frame 'DISPLAY=:0 xprop -root GAMESCOPE_FOCUSABLE_APPS GAMESCOPE_FOCUSED_APP'
# Watch panels being created
ssh frame 'journalctl --user -f | grep --line-buffered "\[Overlays\]"'
# gamescope's full flags (in case Valve changes them)
ssh frame 'tr "\0" " " < /proc/$(pgrep -x gamescope | head -n 1)/cmdline'
# Everything the SteamVR dashboard can say (find hidden features)
ssh frame 'cat /opt/steamvr/resources/webinterface/dashboard/localization/dashboard_english.json'
```
## Where the rest lives
- Access and SSH: [ssh.md](ssh.md)
- Seeing the Frame from the Mac, and the Mac from the Frame: [streaming.md](streaming.md)
- Files and clipboard: [file-transfer.md](file-transfer.md)
- Android apps: [apks.md](apks.md)
- Installing and buying Steam games: [steam-games.md](steam-games.md)
- Floating windows in space: [panels.md](panels.md)
- What's still unverified: [open-questions.md](open-questions.md)
docs/img/frame-control.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 892 KiB

docs/open-questions.md
+16 -1
View File
@@ -36,7 +36,11 @@ build 20260922.6101926, kernel 6.18, aarch64):
The Frame can reach the Mac's Screen Sharing port (5900). The Remmina
connection itself hasn't been tried in the headset yet (part of 11).
Still open: 4, 6, 7, 11 (in-headset connect), 12–16.
- **Panels.** An X11 window on gamescope's `:0` with its own `STEAM_GAME` id
gets its own SteamVR overlay (`valve.steam.desktopgame.<id>`). Three were
created side by side with `panel-on-frame.sh`. See [panels.md](panels.md).
Still open: 4, 6, 7, 11 (in-headset connect), 12–17.
## Check on the headset (in order)
@@ -80,6 +84,17 @@ Still open: 4, 6, 7, 11 (in-headset connect), 12–16.
reach the Linux side? Does USB power from the Mac cope?
16. **Tailscale**: can it be installed persistently (Flatpak? a
userspace `tailscaled` in `~`?) for access off the home LAN?
17. **Floating panels in the headset** (see [panels.md](panels.md)): do the
panels from `panel-on-frame.sh` show up, take input, and offer **Float in
World** / **Move** / **Size**? Do floating positions survive closing and
reopening the app, or a reboot?
18. **`LEPTON_NO_CLEANUP=1 %command%`** as Lepton Development's launch
option: do ADB-installed apps survive closing and reopening it?
19. **Typing in Android apps:** Lepton has no IME installed. Does the SteamVR
keyboard or a Bluetooth keyboard reach Android text fields, or does an
F-Droid keyboard (installed and enabled with `ime enable`/`ime set`) work?
20. **F-Droid 2.0** (Compose 1.12): does it run? If so, the catalogue can
install it instead of 1.17.2.
## Unconfirmed claims made in these docs
docs/panels.md
+125
View File
@@ -0,0 +1,125 @@
# Arranging windows in space
The confidence labels are the same as in [ssh.md](ssh.md).
## The short version
- The in-headset **Linux desktop is one flat panel**: a nested Plasma session,
fixed at 1280×800, drawn into a single SteamVR overlay. Windows *inside* it
are arranged by KWin inside that rectangle. They can't leave it.
- Every **Steam app gets its own panel**. gamescope runs with
`--virtual-connector-strategy PerAppId`, so each distinct app id becomes a
separate SteamVR overlay named `valve.steam.desktopgame.<appid>`.
- To float a Linux app on its own, run it on gamescope's X display (`:0`)
instead of in Plasma, and tag its window with an app id of its own.
`scripts/panel-on-frame.sh` does this:
```sh
./scripts/panel-on-frame.sh konsole # a terminal, as its own panel
./scripts/panel-on-frame.sh --name notes -- kate '~/notes.md' # quote ~ so the Frame expands it
./scripts/panel-on-frame.sh org.mozilla.firefox # a Flatpak
./scripts/panel-on-frame.sh mac-screen # the Mac's screen (Remmina/VNC)
```
- Then **place each panel with the SteamVR dashboard's docking controls**:
**Float in World**, **Move**, **Size**, **Toggle Curvature**, dock on the
left or right controller, **View in Theater**, and **Multitasking View**.
## How a panel is born (verified 2026-09-25)
gamescope's command line on the Frame includes:
```
--backend openvr --xwayland-count 2 --virtual-connector-strategy PerAppId
--vr-overlay-key valve.steam.gamepadui.fallback
--vr-app-overlay-key valve.steam.desktopgame
--vr-overlay-physical-width 2.67 --vr-overlay-enable-control-bar
--nested-width 1280 --nested-height 720
```
gamescope reads each X11 window's `STEAM_GAME` property as its app id. That's
the same property Steam sets on games it launches. On a new id, Steam's
SteamVR system UI logs:
```
[Overlays] Created: valve.steam.desktopgame.7777777
[Overlays] Created: valve.steam.desktopgame.7777777.layer1 … layer7
```
The test: an `xterm` on `DISPLAY=:0`, tagged with
`xprop -id <win> -f STEAM_GAME 32c -set STEAM_GAME 7777777`, produced the
overlay above. Two more apps with different ids (`konsole`, `xterm`) produced
two more overlays, and all three were listed together in the root property
`GAMESCOPE_FOCUSABLE_APPS`. **Not yet checked by eye:** how the new panels
look in the headset and how they handle input.
Untagged windows on `:0` get app id 0 and share the default panel. Plasma
itself (`kwin_wayland`, pid in `GAMESCOPE_FOCUSABLE_WINDOWS`) is one of those.
### What `panel-on-frame.sh` does
1. Sets `DISPLAY=:0`, unsets `WAYLAND_DISPLAY`, and forces X11 in the
toolkits (`QT_QPA_PLATFORM=xcb`, `GDK_BACKEND=x11`, `SDL_VIDEODRIVER=x11`,
`MOZ_ENABLE_WAYLAND=0`). A Wayland-only app would connect to gamescope's
own Wayland socket and not get tagged.
2. Starts the app detached (`setsid nohup`), so it outlives SSH.
3. Diffs the root window's children before and after, and sets `STEAM_GAME`
on each new mapped top-level window. It keeps watching about 3s after the
first window (for splash screens), up to 20s in total (for slow Flatpaks).
It gives up early if the app exits before showing a window.
4. The id comes from `--id`, or is derived from `--name`/the command in the
range 2,000,000,000–2,000,999,999, far above real Steam app ids. The same
label always gives the same id.
Limits:
- **Single-instance apps** (Remmina, most KDE apps with a running copy in
Plasma) hand the request to the existing process, so the window opens
wherever that process lives. Close the app in Plasma first.
- A window the app opens later (a dialog, a second window) isn't tagged, so it
lands on the default panel. Tag it by hand:
`ssh frame 'DISPLAY=:0 xprop -id <win> -f STEAM_GAME 32c -set STEAM_GAME <id>'`
(find `<win>` with `DISPLAY=:0 xwininfo -root -children`).
- The script tags *any* new window on `:0` during its watch window, so a
Steam popup that opens in those few seconds would join the panel too. For
the same reason, run one `panel-on-frame.sh` at a time. If a stray window
is tagged first, the script can report success while the app's own window
stays on the default panel; check in the headset.
- Each panel renders at gamescope's nested size (1280×720), not the Plasma
desktop's 1280×800.
- Steam treats the tagged id as "the current game": it applies a generic
controller config and logs `Failed to get app info` for the made-up id. So
far this hasn't caused anything worse.
## Placing panels: the SteamVR dashboard (inferred from SteamVR's UI code)
The Frame's SteamVR dashboard
(`/opt/steamvr/resources/webinterface/dashboard/`) wraps each overlay in a
frame with a **dock location**: `Dashboard`, `World`, `Theater`,
`LeftController`, `RightController`. The strings and handlers are there
(`dashboard_english.json`, `systemui.js`):
| Control | What it does |
|---|---|
| **Float in World** | Only shown while the panel is docked on the dashboard. Detaches it into the room, where it stays after the dashboard closes. |
| **Move** / grab handle | Push, pull and drag the panel. *Grab Handle Acceleration* in SteamVR settings speeds up push and pull. |
| **Size** | Resize the floating panel. |
| **Toggle Curvature** | Flat vs curved. |
| **Dock on Left/Right Controller** | Attach to a controller, like a wrist screen. |
| **Dock on Dashboard / Return to Dashboard** | Put it back. |
| **View in Theater** / Show/Hide Theater Screen | Shows the panel as a large theater screen. |
| **Multitasking View** | Shows every open panel together (only if `VRHTML.BSupportsMultitaskingView()`). |
| **More Options** (…) | Where the less common docking actions live. |
**Still to check in the headset:** where exactly each control appears, whether
floating positions survive a panel closing and reopening, and whether there's
a limit on the number of floating panels.
## Other routes
- **Just the desktop somewhere else**: float the Plasma panel itself. No
script needed.
- **Inside the desktop panel**: KWin tiling (Meta+arrow keys with a Bluetooth
keyboard) or virtual desktops arrange windows within the 1280×800 rectangle.
- **Windows-only overlay tools** (Desktop+, OVR Toolkit, OVRdrop) do this for a
PC's desktop in SteamVR. They don't run on the Frame's standalone Linux.
docs/steam-games.md
+108
View File
@@ -0,0 +1,108 @@
# Installing and buying Steam games from the Mac
Frame Control's **Get games** section lists the games you own with each one's
Steam Frame rating, installs them on the Frame, and searches the Steam store.
This page covers how it works underneath, so you can do the same from a shell.
## How it works
The Frame's Steam client runs with `-cef-enable-debugging`. So its UI, a
Chromium page, answers the Chrome DevTools protocol on the Frame's loopback,
`127.0.0.1:8080`. The page titled **SharedJSContext** holds the client's own
state and API:
| Object | What it gives you |
|---|---|
| `appStore.allApps` | Every app the account owns (868 games here), with `local_per_client_data.installed`, playtime, `vr_supported`/`vr_only` and `steam_hw_compat_category_packed` |
| `downloadsStore.m_DownloadOverview` | A Map keyed by client ID. `"0"` is this machine: current app, percent, ETA, bytes/s |
| `SteamClient.Installs.*` | The install wizard: `GetInstallManagerInfo`, `ContinueInstall`, `CancelInstall`, `OpenInstallWizard` |
| `SteamClient.User.GetIPCountry()` | The store country (`AU` here), which store search needs |
`ui/frame_steam.py` is a stdlib-only WebSocket client for this page. It's piped
over SSH like the other helpers:
```sh
ssh frame 'python3 - owned' < ui/frame_steam.py # owned games + download
ssh frame 'python3 - install 274190' < ui/frame_steam.py # install Broforce
ssh frame 'python3 - store 1145360' < ui/frame_steam.py # store page in the headset
```
The debugger port only listens on the Frame's loopback, so it's reachable over
SSH and not from the network.
## Installing a game you own
`steam steam://install/<appid>`, run over SSH, hands the URL to the running
client, which opens its install wizard. The wizard's state
(`GetInstallManagerInfo().eInstallState`) then tells you what happens next:
| State | Meaning | What `frame_steam.py` does |
|---|---|---|
| 14 complete | Steam skipped the options dialog and queued the download | Reports "queued" |
| 7 config | The options dialog is showing in the headset (library folder, compatibility note) | Calls `ContinueInstall()` when the game fits on disk, as the headset's Install button does |
| 3, 4, 6, 8, 13 | Free license, CD key, password, EULA, signup | Leaves them for you to answer in the headset |
| 15 failed | Error | Reports `errorDetail` |
**Verified 2026-09-25 (SteamOS 0.3.0, build 20260922.6101926):**
- Balatro (2379780, 67 MB) went straight to state 14 and installed in about 7 s,
with nothing to answer in the headset.
- Broforce (274190, 0.6 GB) stopped at state 7. Calling `ContinueInstall()` over
DevTools queued the download, and the game installed.
- Calling `SteamClient.Installs.OpenInstallWizard([appid])` directly did nothing:
the state stayed at 0. Go through the `steam://install` URL instead.
**Inferred** from the client's JS: Steam skips the options dialog when there's
one library folder, the game fits, and there's no compatibility note to show.
Broforce is Deck "Playable", which probably explains why it stopped.
## Frame ratings
`steam_hw_compat_category_packed` holds two bits per device. The client decodes
it like this (from `steamui/chunk~2dcc5aaf7.js`):
| Device | Bits |
|---|---|
| Steam Deck | `packed & 3` |
| SteamOS | `packed >> 4 & 3` |
| Steam Machine | `packed >> 6 & 3` |
| **Steam Frame** | `packed >> 8 & 3` |
The values are 0 unknown, 1 unsupported, 2 playable and 3 verified. On
2026-09-25 this library had 12 Frame Verified, 2 Playable, 6 Unsupported and 848
Unknown games.
For games you don't own, the store's public
`saleaction/ajaxgetdeckappcompatibilityreport?nAppID=<id>` returns
`frame_resolved_category` on the same scale, along with `resolved_category`
(Deck), `steamos_resolved_category` and `machine_resolved_category`. No key or
login is needed.
## Buying
Frame Control doesn't buy anything. Purchases happen on Steam's own store page,
signed in as you:
- **Buy on Steam ↗** opens `store.steampowered.com/app/<id>/` in the Mac's
browser (the Electron app sends `target=_blank` links there).
- **Store on Frame** runs `steam steam://store/<id>`, which opens the page in the
Steam client on the headset. **Verified 2026-09-25:** a "Hades on Steam" page
appeared in the DevTools page list. It wasn't visible in the headset capture
because an app was in the foreground; it opens in Steam's dashboard.
After buying, press **Refresh** in Get games. The game shows up as owned, and
**Install on Frame** installs it.
Store search uses `store.steampowered.com/api/storesearch/?term=…&cc=…`. It
returns nothing without `cc`, so Frame Control takes the country from
`SteamClient.User.GetIPCountry()` on the Frame.
## Not yet checked
- Free-to-play games: `steam://install` should stop at state 3 (free license)
for you to accept in the headset. Not tried, because it adds a license to the
account.
- Games with a EULA (state 8).
- Installing when there's more than one library folder, such as a microSD card.
- Uninstalling. `steam://uninstall/<appid>` should open a confirmation in the
headset.