From 0181071b8313b6826841930d1dad30a702ce763f Mon Sep 17 00:00:00 2001 From: saphid <4596216+saphid@users.noreply.github.com> Date: Sun, 27 Sep 2026 17:06:40 +1000 Subject: [PATCH] Tests against a fake Frame, and a headset smoke test tests/fakeframe: a container that stands in for the Frame (Arch Linux, or Arch Linux ARM on arm64) with sshd, rsync, Valve's steamos-devkit-service and hooks (vendored unmodified), a fake Steam client for the devkit pipe and the DevTools port (the app's JavaScript runs in Node against stand-in SteamClient/appStore objects), stubs for steam, wpctl, flatpak, podman, Lepton and friends, battery and thermal files under /sys, and fault switches (fakeframe-ctl): pairing mode, approve/deny/timeout, Steam not running, headset asleep, sshd off, disk full, runtimes missing. A second container is the computer running Frame Control. tests/e2e: 29 tests driving the real ui/server.py, frame_connect.py and frame_titles.py against it; skipped unless FRAME_E2E=1. scripts/e2e.sh builds, runs and tears down; CI runs it on ubuntu-24.04-arm. tests/smoke + scripts/frame-smoke.sh: the core cases against a real Frame, recorded with its BUILD_ID, cleaning up after itself; --pair to pair a throwaway key. docs/testing.md describes the layers. Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/workflows/checks.yml | 12 + .gitignore | 1 + README.md | 2 + docs/testing.md | 142 +++++ scripts/e2e.sh | 64 +++ scripts/frame-smoke.sh | 8 + tests/e2e/harness.py | 198 +++++++ tests/e2e/test_android.py | 76 +++ tests/e2e/test_device.py | 86 +++ tests/e2e/test_faults.py | 94 ++++ tests/e2e/test_pairing.py | 111 ++++ tests/e2e/test_titles.py | 191 +++++++ tests/fakeframe/Containerfile | 40 ++ tests/fakeframe/compose.yaml | 54 ++ tests/fakeframe/host.Containerfile | 14 + tests/fakeframe/host/entrypoint.sh | 25 + tests/fakeframe/rootfs/etc/ssh/sshd_config | 21 + .../steamos-enable-sshd | 14 + tests/fakeframe/rootfs/usr/lib/os-release | 9 + .../rootfs/usr/local/bin/fakeframe-ctl | 31 ++ tests/fakeframe/rootfs/usr/local/bin/flatpak | 38 ++ .../rootfs/usr/local/bin/gamescopectl | 26 + tests/fakeframe/rootfs/usr/local/bin/nmcli | 11 + tests/fakeframe/rootfs/usr/local/bin/podman | 47 ++ tests/fakeframe/rootfs/usr/local/bin/qdbus6 | 19 + tests/fakeframe/rootfs/usr/local/bin/steam | 26 + tests/fakeframe/rootfs/usr/local/bin/wpctl | 21 + .../usr/local/lib/fakeframe/cef_shim.js | 157 ++++++ .../local/lib/fakeframe/fakeframe_state.py | 183 ++++++ .../usr/local/lib/fakeframe/fakesteam.py | 490 ++++++++++++++++ .../rootfs/usr/local/lib/fakeframe/init.py | 468 ++++++++++++++++ .../rootfs/usr/local/lib/fakeframe/lepton.py | 83 +++ .../lib/fakeframe/pystubs/dbus/__init__.py | 50 ++ .../lib/fakeframe/pystubs/dbus/exceptions.py | 3 + .../fakeframe/steamos-devkit-service/LICENSE | 504 +++++++++++++++++ .../steamos-devkit-service/README.md | 21 + .../hooks/approve-ssh-key | 71 +++ .../hooks/devkit-1-identify | 151 +++++ .../hooks/devkit_utils/__init__.py | 265 +++++++++ .../hooks/install-ssh-key | 74 +++ .../src/steamos-devkit-service.py | 522 ++++++++++++++++++ tests/smoke/frame_smoke.py | 235 ++++++++ tests/smoke/tiny_programs.py | 89 +++ 43 files changed, 4747 insertions(+) create mode 100644 docs/testing.md create mode 100755 scripts/e2e.sh create mode 100755 scripts/frame-smoke.sh create mode 100644 tests/e2e/harness.py create mode 100644 tests/e2e/test_android.py create mode 100644 tests/e2e/test_device.py create mode 100644 tests/e2e/test_faults.py create mode 100644 tests/e2e/test_pairing.py create mode 100644 tests/e2e/test_titles.py create mode 100644 tests/fakeframe/Containerfile create mode 100644 tests/fakeframe/compose.yaml create mode 100644 tests/fakeframe/host.Containerfile create mode 100755 tests/fakeframe/host/entrypoint.sh create mode 100644 tests/fakeframe/rootfs/etc/ssh/sshd_config create mode 100755 tests/fakeframe/rootfs/usr/bin/steamos-polkit-helpers/steamos-enable-sshd create mode 100644 tests/fakeframe/rootfs/usr/lib/os-release create mode 100755 tests/fakeframe/rootfs/usr/local/bin/fakeframe-ctl create mode 100755 tests/fakeframe/rootfs/usr/local/bin/flatpak create mode 100755 tests/fakeframe/rootfs/usr/local/bin/gamescopectl create mode 100755 tests/fakeframe/rootfs/usr/local/bin/nmcli create mode 100755 tests/fakeframe/rootfs/usr/local/bin/podman create mode 100755 tests/fakeframe/rootfs/usr/local/bin/qdbus6 create mode 100755 tests/fakeframe/rootfs/usr/local/bin/steam create mode 100755 tests/fakeframe/rootfs/usr/local/bin/wpctl create mode 100644 tests/fakeframe/rootfs/usr/local/lib/fakeframe/cef_shim.js create mode 100755 tests/fakeframe/rootfs/usr/local/lib/fakeframe/fakeframe_state.py create mode 100755 tests/fakeframe/rootfs/usr/local/lib/fakeframe/fakesteam.py create mode 100755 tests/fakeframe/rootfs/usr/local/lib/fakeframe/init.py create mode 100755 tests/fakeframe/rootfs/usr/local/lib/fakeframe/lepton.py create mode 100644 tests/fakeframe/rootfs/usr/local/lib/fakeframe/pystubs/dbus/__init__.py create mode 100644 tests/fakeframe/rootfs/usr/local/lib/fakeframe/pystubs/dbus/exceptions.py create mode 100644 tests/fakeframe/steamos-devkit-service/LICENSE create mode 100644 tests/fakeframe/steamos-devkit-service/README.md create mode 100755 tests/fakeframe/steamos-devkit-service/hooks/approve-ssh-key create mode 100755 tests/fakeframe/steamos-devkit-service/hooks/devkit-1-identify create mode 100644 tests/fakeframe/steamos-devkit-service/hooks/devkit_utils/__init__.py create mode 100755 tests/fakeframe/steamos-devkit-service/hooks/install-ssh-key create mode 100755 tests/fakeframe/steamos-devkit-service/src/steamos-devkit-service.py create mode 100644 tests/smoke/frame_smoke.py create mode 100644 tests/smoke/tiny_programs.py diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 51eea2d..314bf4f 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -71,3 +71,15 @@ jobs: cd ios udid=$(xcrun simctl list devices available -j | python3 -c 'import json,sys; d=json.load(sys.stdin)["devices"]; print(next(x["udid"] for r in d for x in d[r] if x["name"].startswith("iPhone")))') xcodebuild -project FrameControl.xcodeproj -scheme FrameControl -destination "platform=iOS Simulator,id=$udid" CODE_SIGNING_ALLOWED=NO test + + # End-to-end tests against the fake Frame (tests/fakeframe): Arch Linux ARM + # in Docker, on a native arm64 runner like the headset. See docs/testing.md. + e2e: + runs-on: ubuntu-24.04-arm + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - name: Install zsh + run: sudo apt-get update -qq && sudo apt-get install -y -qq zsh + - name: End-to-end tests + run: scripts/e2e.sh diff --git a/.gitignore b/.gitignore index ad5b005..d89ecc2 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ apk-catalog/data/cache/ apk-catalog/data/index-v2.json* compat-db/.env.lakebed.server compat-db/.lakebed/ +tests/smoke/results/ diff --git a/README.md b/README.md index 93cabeb..5f25235 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,7 @@ Frame's software fits together, all checked against a real headset and labelled | [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes | | [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing | | [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset | +| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, and the headset smoke test | | [Open questions](docs/open-questions.md) | What's still unchecked |
@@ -221,6 +222,7 @@ Frame's software fits together, all checked against a real headset and labelled ```sh python3 -m unittest discover -s tests # server tests; no headset needed +scripts/e2e.sh # end-to-end against a fake Frame (Linux with Docker) cd app && npm install && npm start # run the app from the checkout ``` diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..e70a4a7 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,142 @@ +# Testing + +Frame Control is tested in three layers, from fast and fake to slow and real. +A fourth, a SteamOS VM, may come later ([issue #6](https://github.com/saphid/steam-frame/issues/6)). + +| Layer | Runs | Needs | Covers | +|---|---|---|---| +| Unit tests (`tests/*.py`) | `python3 -m unittest discover -s tests` | Nothing | Parsing, validation, request guards; SSH and HTTP are mocked | +| Fake Frame (`tests/e2e`) | `scripts/e2e.sh` | Linux with Docker | The real server and scripts against a container that behaves like a Frame | +| Headset smoke test | `scripts/frame-smoke.sh` | A Frame on the `frame` alias | Install, launch and remove on the real device, recorded with its BUILD_ID | + +## Unit tests + +```sh +python3 -m unittest discover -s tests +``` + +About 120 tests, a few seconds, on Python 3.9 and newer. GitHub Actions runs +them on macOS, Windows and Linux. They don't pick up `tests/e2e`. + +## The fake Frame + +`tests/fakeframe/` builds a container that stands in for the headset, and a +second one for the computer Frame Control runs on. `scripts/e2e.sh` builds +both, starts them with `docker compose`, runs `tests/e2e` in the host +container and takes everything down, exiting with the tests' status: + +```sh +scripts/e2e.sh # everything, about 2 minutes plus the first build +scripts/e2e.sh test_titles # one module +scripts/e2e.sh test_faults.Faults.test_disk_full # one test +FAKEFRAME_KEEP=1 scripts/e2e.sh # leave it running afterwards +``` + +It needs a Linux host with Docker and `docker compose`, and zsh. The images +are `fakeframe-frame` and `fakeframe-host`; the compose project, network and +volumes are `fakeframe-e2e*`. CI runs it on a native arm64 runner +(`ubuntu-24.04-arm`, the `e2e` job in `.github/workflows/checks.yml`). + +The host container exists because OpenSSH reads `~/.ssh/config` from the +passwd home directory, not `$HOME`. There, `ssh frame` reaches the fake Frame +through the same `Host frame` block `ui/frame_connect.py` writes, the +repository is mounted read-only at `/repo`, and each test module starts the +real `ui/server.py` (Python 3.9) and talks to it over HTTP with the headers +its guards want. + +### What's real and what's fake + +| On the fake Frame | | +|---|---| +| Arch Linux (`archlinux:base`, or `menci/archlinuxarm:base` on arm64), user `steamos`, `/etc/os-release` with BUILD_ID 20260922.6101926 | Real OS, Frame's identity | +| `sshd` with key and password logins, `rsync`, `python3` | Real | +| Valve's steamos-devkit-service on port 32000 and its hooks, vendored unmodified in `tests/fakeframe/steamos-devkit-service` | Real; only its `dbus` import (for mDNS through systemd-resolved) is a stand-in that logs the registration | +| Valve's devkit-utils, copied over by Frame Control itself | Real | +| **fakesteam**: `~/.steam/steam.pid`, `steam.token` and the `steam.pipe` FIFO; answers `approve-ssh-key`, `create-shortcut`, `run-game`, `list-shortcuts` and `delete-shortcut` with the response files devkit-utils waits for; takes `steam://rungameid`, `install` and `store` URLs | Fake | +| DevTools on `127.0.0.1:8080` with a `SharedJSContext` target. The JavaScript Frame Control sends runs for real in Node against stand-in `SteamClient`, `appStore` and `downloadsStore` objects (`cef_shim.js`), so async functions, optional chaining and `Map`s behave as in Steam's CEF | The JS engine is real; the objects are fake | +| `steam`, `wpctl`, `flatpak`, `podman`, `nmcli`, `qdbus6`, `gamescopectl`, SteamOS's `steamos-enable-sshd` helper, and Lepton's launcher | Stubs that record their calls | +| Battery, charger and thermal zones under `/sys/class` | Files the supervisor writes. `/sys` is read-only in a container and Docker's AppArmor profile refuses writes under it, so each folder is a volume mounted twice: over `/sys/class/...` for `frame_status.py` to read, and under `/var/lib/fakeframe/sys` for the supervisor to write | +| `vrserver` and `plasmashell` | Renamed `sleep` processes, so the status page and the clipboard find them | + +Every fake behaviour copied from the device has a comment citing the doc or +observation and the BUILD_ID it came from; anything not seen on a headset is +marked as a guess. The fake keeps its state in `/var/lib/fakeframe/state.json` +(shortcuts, devkit titles, compat tool mapping, launches, pairing requests, +Lepton instances, volume, Flatpaks, clipboard) and logs stub calls to +`calls.jsonl` beside it. + +Native programs really run: a launched aarch64 title executes on an arm64 +host, and an x86-64 one on x86-64 (the container shares the host's kernel). +Proton titles are recorded with the command Steam would run, not run. + +### Fault switches + +`fakeframe-ctl` works over SSH (`ssh frame fakeframe-ctl help`) and from the +host container (`FAKEFRAME_CTL=http://fakeframe:9999`), so a test can flip a +switch while SSH is down: + +| Command | Effect | +|---|---| +| `pairing on\|off` | Steam's **Pair new host** screen open or not; off gives the device's 403 text | +| `answer approve\|deny\|timeout` | How the pairing prompt is answered | +| `steam on\|off` | Steam client running (pid file, pipe, DevTools) | +| `sleep on\|off` | Headset asleep: ports 22 and 32000 accept and never answer, so SSH times out | +| `sshd on\|off` | sshd stopped: new connections are refused, open ones stay | +| `devkit-service on\|off` | Port 32000 closed | +| `disk-full on\|off` | Fills the small (64 MB) filesystem on `~/devkit-game` | +| `runtime NAME installed\|missing` | Proton, the Steam Linux Runtimes, Lepton | +| `battery KEY=VALUE...` | e.g. `capacity=15 status=Discharging current_now=-900000` | +| `keys harness\|none`, `authorized-keys` | Set or read `~/.ssh/authorized_keys` | +| `reset`, `state`, `calls [TOOL]` | Start over; read the state and call log | + +### What the fake can't show + +- Rendering: the headset view, desktop capture content, live video, SteamVR, + gamescope and panels. The capture stub returns a placeholder PNG. +- Proton and FEX: whether a Windows or x86-64 program actually runs. +- Android: there's no Android in the Lepton stand-in, so no ADB, display + settings, probes or app crashes. +- The real Steam client's UI and anything it does that isn't modelled, and + mDNS discovery. +- `sudo` and the power buttons, Tailscale, and the Windows and macOS sides of + the app (the host container is Linux, so the `rsync` paths are tested and the + `scp` fallback isn't). + +## Headset smoke test + +```sh +scripts/frame-smoke.sh # needs `ssh frame` to work without a password +scripts/frame-smoke.sh --pair # also pairs a throwaway key: approve it in the headset +``` + +It checks `properties.json` and the status, then installs, launches and +removes three tiny titles built from bytes by `tests/smoke/tiny_programs.py` +(an ARM64 and an x86-64 static Linux program that sleep for five seconds, and +an x86-64 `.exe` that exits at once), keeping what Steam logged about each. +Everything it installs is removed again, also after a failure. Results go to +`tests/smoke/results/