diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index 51eea2d..314bf4f 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -71,3 +71,15 @@ jobs: cd ios udid=$(xcrun simctl list devices available -j | python3 -c 'import json,sys; d=json.load(sys.stdin)["devices"]; print(next(x["udid"] for r in d for x in d[r] if x["name"].startswith("iPhone")))') xcodebuild -project FrameControl.xcodeproj -scheme FrameControl -destination "platform=iOS Simulator,id=$udid" CODE_SIGNING_ALLOWED=NO test + + # End-to-end tests against the fake Frame (tests/fakeframe): Arch Linux ARM + # in Docker, on a native arm64 runner like the headset. See docs/testing.md. + e2e: + runs-on: ubuntu-24.04-arm + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - name: Install zsh + run: sudo apt-get update -qq && sudo apt-get install -y -qq zsh + - name: End-to-end tests + run: scripts/e2e.sh diff --git a/.gitignore b/.gitignore index ad5b005..d89ecc2 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ apk-catalog/data/cache/ apk-catalog/data/index-v2.json* compat-db/.env.lakebed.server compat-db/.lakebed/ +tests/smoke/results/ diff --git a/README.md b/README.md index 93cabeb..5f25235 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,7 @@ Frame's software fits together, all checked against a real headset and labelled | [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes | | [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing | | [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset | +| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, and the headset smoke test | | [Open questions](docs/open-questions.md) | What's still unchecked |
@@ -221,6 +222,7 @@ Frame's software fits together, all checked against a real headset and labelled ```sh python3 -m unittest discover -s tests # server tests; no headset needed +scripts/e2e.sh # end-to-end against a fake Frame (Linux with Docker) cd app && npm install && npm start # run the app from the checkout ``` diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..e70a4a7 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,142 @@ +# Testing + +Frame Control is tested in three layers, from fast and fake to slow and real. +A fourth, a SteamOS VM, may come later ([issue #6](https://github.com/saphid/steam-frame/issues/6)). + +| Layer | Runs | Needs | Covers | +|---|---|---|---| +| Unit tests (`tests/*.py`) | `python3 -m unittest discover -s tests` | Nothing | Parsing, validation, request guards; SSH and HTTP are mocked | +| Fake Frame (`tests/e2e`) | `scripts/e2e.sh` | Linux with Docker | The real server and scripts against a container that behaves like a Frame | +| Headset smoke test | `scripts/frame-smoke.sh` | A Frame on the `frame` alias | Install, launch and remove on the real device, recorded with its BUILD_ID | + +## Unit tests + +```sh +python3 -m unittest discover -s tests +``` + +About 120 tests, a few seconds, on Python 3.9 and newer. GitHub Actions runs +them on macOS, Windows and Linux. They don't pick up `tests/e2e`. + +## The fake Frame + +`tests/fakeframe/` builds a container that stands in for the headset, and a +second one for the computer Frame Control runs on. `scripts/e2e.sh` builds +both, starts them with `docker compose`, runs `tests/e2e` in the host +container and takes everything down, exiting with the tests' status: + +```sh +scripts/e2e.sh # everything, about 2 minutes plus the first build +scripts/e2e.sh test_titles # one module +scripts/e2e.sh test_faults.Faults.test_disk_full # one test +FAKEFRAME_KEEP=1 scripts/e2e.sh # leave it running afterwards +``` + +It needs a Linux host with Docker and `docker compose`, and zsh. The images +are `fakeframe-frame` and `fakeframe-host`; the compose project, network and +volumes are `fakeframe-e2e*`. CI runs it on a native arm64 runner +(`ubuntu-24.04-arm`, the `e2e` job in `.github/workflows/checks.yml`). + +The host container exists because OpenSSH reads `~/.ssh/config` from the +passwd home directory, not `$HOME`. There, `ssh frame` reaches the fake Frame +through the same `Host frame` block `ui/frame_connect.py` writes, the +repository is mounted read-only at `/repo`, and each test module starts the +real `ui/server.py` (Python 3.9) and talks to it over HTTP with the headers +its guards want. + +### What's real and what's fake + +| On the fake Frame | | +|---|---| +| Arch Linux (`archlinux:base`, or `menci/archlinuxarm:base` on arm64), user `steamos`, `/etc/os-release` with BUILD_ID 20260922.6101926 | Real OS, Frame's identity | +| `sshd` with key and password logins, `rsync`, `python3` | Real | +| Valve's steamos-devkit-service on port 32000 and its hooks, vendored unmodified in `tests/fakeframe/steamos-devkit-service` | Real; only its `dbus` import (for mDNS through systemd-resolved) is a stand-in that logs the registration | +| Valve's devkit-utils, copied over by Frame Control itself | Real | +| **fakesteam**: `~/.steam/steam.pid`, `steam.token` and the `steam.pipe` FIFO; answers `approve-ssh-key`, `create-shortcut`, `run-game`, `list-shortcuts` and `delete-shortcut` with the response files devkit-utils waits for; takes `steam://rungameid`, `install` and `store` URLs | Fake | +| DevTools on `127.0.0.1:8080` with a `SharedJSContext` target. The JavaScript Frame Control sends runs for real in Node against stand-in `SteamClient`, `appStore` and `downloadsStore` objects (`cef_shim.js`), so async functions, optional chaining and `Map`s behave as in Steam's CEF | The JS engine is real; the objects are fake | +| `steam`, `wpctl`, `flatpak`, `podman`, `nmcli`, `qdbus6`, `gamescopectl`, SteamOS's `steamos-enable-sshd` helper, and Lepton's launcher | Stubs that record their calls | +| Battery, charger and thermal zones under `/sys/class` | Files the supervisor writes. `/sys` is read-only in a container and Docker's AppArmor profile refuses writes under it, so each folder is a volume mounted twice: over `/sys/class/...` for `frame_status.py` to read, and under `/var/lib/fakeframe/sys` for the supervisor to write | +| `vrserver` and `plasmashell` | Renamed `sleep` processes, so the status page and the clipboard find them | + +Every fake behaviour copied from the device has a comment citing the doc or +observation and the BUILD_ID it came from; anything not seen on a headset is +marked as a guess. The fake keeps its state in `/var/lib/fakeframe/state.json` +(shortcuts, devkit titles, compat tool mapping, launches, pairing requests, +Lepton instances, volume, Flatpaks, clipboard) and logs stub calls to +`calls.jsonl` beside it. + +Native programs really run: a launched aarch64 title executes on an arm64 +host, and an x86-64 one on x86-64 (the container shares the host's kernel). +Proton titles are recorded with the command Steam would run, not run. + +### Fault switches + +`fakeframe-ctl` works over SSH (`ssh frame fakeframe-ctl help`) and from the +host container (`FAKEFRAME_CTL=http://fakeframe:9999`), so a test can flip a +switch while SSH is down: + +| Command | Effect | +|---|---| +| `pairing on\|off` | Steam's **Pair new host** screen open or not; off gives the device's 403 text | +| `answer approve\|deny\|timeout` | How the pairing prompt is answered | +| `steam on\|off` | Steam client running (pid file, pipe, DevTools) | +| `sleep on\|off` | Headset asleep: ports 22 and 32000 accept and never answer, so SSH times out | +| `sshd on\|off` | sshd stopped: new connections are refused, open ones stay | +| `devkit-service on\|off` | Port 32000 closed | +| `disk-full on\|off` | Fills the small (64 MB) filesystem on `~/devkit-game` | +| `runtime NAME installed\|missing` | Proton, the Steam Linux Runtimes, Lepton | +| `battery KEY=VALUE...` | e.g. `capacity=15 status=Discharging current_now=-900000` | +| `keys harness\|none`, `authorized-keys` | Set or read `~/.ssh/authorized_keys` | +| `reset`, `state`, `calls [TOOL]` | Start over; read the state and call log | + +### What the fake can't show + +- Rendering: the headset view, desktop capture content, live video, SteamVR, + gamescope and panels. The capture stub returns a placeholder PNG. +- Proton and FEX: whether a Windows or x86-64 program actually runs. +- Android: there's no Android in the Lepton stand-in, so no ADB, display + settings, probes or app crashes. +- The real Steam client's UI and anything it does that isn't modelled, and + mDNS discovery. +- `sudo` and the power buttons, Tailscale, and the Windows and macOS sides of + the app (the host container is Linux, so the `rsync` paths are tested and the + `scp` fallback isn't). + +## Headset smoke test + +```sh +scripts/frame-smoke.sh # needs `ssh frame` to work without a password +scripts/frame-smoke.sh --pair # also pairs a throwaway key: approve it in the headset +``` + +It checks `properties.json` and the status, then installs, launches and +removes three tiny titles built from bytes by `tests/smoke/tiny_programs.py` +(an ARM64 and an x86-64 static Linux program that sleep for five seconds, and +an x86-64 `.exe` that exits at once), keeping what Steam logged about each. +Everything it installs is removed again, also after a failure. Results go to +`tests/smoke/results/