mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-06 03:00:18 +02:00
Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
73b97fedc5 | ||
|
|
924e37aa30 | ||
|
|
bd24436fbc | ||
|
|
895cc0ebbd | ||
|
|
f7dbad500a | ||
|
|
3f20e5f9d3 |
No files matched your search
@@ -222,7 +222,7 @@ Frame's software fits together, all checked against a real headset and labelled
|
||||
| [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing |
|
||||
| [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset |
|
||||
| [AI agents and assistant](docs/agents.md) | Key-free MCP tools, human approvals, and an opt-in assistant panel |
|
||||
| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, and the headset smoke test |
|
||||
| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, the headset smoke test and a Windows test VM |
|
||||
| [Open questions](docs/open-questions.md) | What's still unchecked |
|
||||
|
||||
<details>
|
||||
|
||||
@@ -21,6 +21,20 @@ The confidence labels are the same as in [ssh.md](ssh.md).
|
||||
| **ADB + scrcpy (Lepton only)** | A mirror of the Android container | **Guess** | `brew install scrcpy android-platform-tools`, then `adb connect frame.local:5555` while Lepton Development is running ([adb_lepton](https://partner.steamgames.com/doc/steamhardware/steamframe/adb_lepton)), then `scrcpy`. This only shows Android apps, not SteamOS. |
|
||||
| VNC server on the Frame (krfb / wayvnc) | A mirror of the Plasma desktop | **Inferred (SteamOS)** | Deck users run krfb in Desktop Mode ([one.vg](https://one.vg/blog/remote-control-your-steam-deck)). On the Frame, the in-headset desktop is a virtual screen, and krfb isn't known to be preinstalled. RDP and Steam Link cover this case, so it's not recommended. |
|
||||
|
||||
**RDP from Windows, verified 2026-09-30:** Windows 11 25H2's Remote Desktop
|
||||
(`mstsc`) against BUILD_ID 20260925.6191901. xrdp picks TLS, not NLA, so
|
||||
Remote Desktop never asks for a user or password. It warns that the certificate
|
||||
(`www.xrdp.org`) can't be verified. After **Yes**, xrdp shows its own "Login to
|
||||
frame" box with the username blank. Any user but `steamos` gets "User does not
|
||||
exist, or could not be authenticated". Signing in as `steamos` with the
|
||||
Developer Mode password opens a Plasma (X11) desktop within about 6 seconds.
|
||||
It's a new session on display `:10`, separate from what the headset shows,
|
||||
and it uses about 1.3 GB of the Frame's memory. Closing Remote Desktop leaves
|
||||
it running, and the next sign-in reconnects to it. To end it, find its
|
||||
session with `loginctl list-sessions` over SSH (its leader is `xrdp-sesexec`)
|
||||
and run `loginctl terminate-session <id>`; the headset's own session keeps
|
||||
running.
|
||||
|
||||
**Recommendation for A:** start with Steam Link for macOS, because Valve
|
||||
documents it. Use Windows App (RDP) when you want a proper Linux desktop on the
|
||||
Mac with keyboard, mouse, and clipboard.
|
||||
|
||||
@@ -8,6 +8,7 @@ A fourth, a SteamOS VM, may come later ([issue #6](https://github.com/saphid/ste
|
||||
| Unit tests (`tests/*.py`) | `python3 -m unittest discover -s tests` | Nothing | Parsing, validation, request guards; SSH and HTTP are mocked |
|
||||
| Fake Frame (`tests/e2e`) | `scripts/e2e.sh` | Linux with Docker | The real server and scripts against a container that behaves like a Frame |
|
||||
| Headset smoke test | `scripts/frame-smoke.sh` | A Frame on the `frame` alias | Install, launch and remove on the real device, recorded with its BUILD_ID |
|
||||
| [Windows test VM](#windows-test-vm) | `scripts/windows-vm.sh` | A Linux machine with KVM and Docker | The Windows build on a real Windows desktop, against a real Frame when needed |
|
||||
|
||||
## Unit tests
|
||||
|
||||
@@ -163,6 +164,86 @@ refuses ids with a hyphen (`missing/invalid arguments`), which the fake had
|
||||
accepted. The fake now refuses them the same way, and Frame Control makes ids
|
||||
Steam accepts.
|
||||
|
||||
## Windows test VM
|
||||
|
||||
The unit tests run on Windows in CI, but the app itself doesn't. Anything that
|
||||
depends on the Windows desktop (Remote Desktop, the installer, the bundled
|
||||
Python, file dialogs) needs a real Windows machine. This is a Windows 11 VM in a
|
||||
[dockur/windows](https://github.com/dockur/windows) container on a Linux machine
|
||||
with KVM, driven from the Mac with `scripts/windows-vm.sh`.
|
||||
|
||||
Setting it up, once, on the Linux machine:
|
||||
|
||||
- **Windows comes from Microsoft.** The container downloads the Windows 11
|
||||
image from Microsoft on first start. Windows runs unactivated, which is fine
|
||||
for testing. Don't use activation workarounds or third-party Windows images.
|
||||
- **Publish its ports on loopback only.** Map `127.0.0.1:2222:22` (SSH),
|
||||
`127.0.0.1:8006:8006` (the web console) and, if you need it,
|
||||
`127.0.0.1:13389:3389`. The Mac reaches them through `ssh -J`. Use
|
||||
`restart: "no"` and `stop_grace_period: 2m` so it only runs when someone is
|
||||
testing, and a `docker stop` shuts Windows down cleanly.
|
||||
- **Give it SSH on first sign-in.** The container runs `/oem/install.bat`
|
||||
once. Have it add the OpenSSH Server capability, start `sshd`, set PowerShell
|
||||
as its default shell, and put a dedicated public key (for example
|
||||
`~/.ssh/id_ed25519_winvm` on the Mac) in
|
||||
`C:\ProgramData\ssh\administrators_authorized_keys`.
|
||||
- Give it 4 cores, 8 GB of memory and a 32 GB disk. That's enough for the app
|
||||
and the tests.
|
||||
|
||||
Using it, with `WINVM_HOST` set to the Linux machine's ssh alias:
|
||||
|
||||
```sh
|
||||
scripts/windows-vm.sh up # start it and wait for SSH (1-2 minutes)
|
||||
scripts/windows-vm.sh put Frame-Control-Setup-x64.exe
|
||||
scripts/windows-vm.sh ps 'Start-Process "$env:USERPROFILE\Downloads\Frame-Control-Setup-x64.exe" /S -Wait'
|
||||
scripts/windows-vm.sh shot screen.png # what's on its screen
|
||||
scripts/windows-vm.sh click 723 359 # screen pixels, as in the screenshot
|
||||
scripts/windows-vm.sh keys s t e a m o s ret # QEMU key names
|
||||
scripts/windows-vm.sh down # shut Windows down
|
||||
```
|
||||
|
||||
- **Clicks go through a scheduled task.** Commands over SSH run in a
|
||||
session with no desktop, so `click` and `scroll` write the position to a
|
||||
file, and a scheduled task running as the signed-in user replays it. The
|
||||
VM's screen must be signed in; it is after `up`. Clicks and scrolls sent at
|
||||
the same time run one after another. QEMU's own `mouse_move` is relative
|
||||
and drifts, so the script doesn't use it.
|
||||
- **Screenshots may not show the pointer.** Check the result of a click (a
|
||||
menu that opens, a button that changes) rather than the pointer's position.
|
||||
- **Windows' `ssh` waits for stdin.** The script closes it for every command.
|
||||
Do the same if you run `ssh` in the VM by hand.
|
||||
- **Starting the app.** Run Frame Control in the signed-in session, not over
|
||||
SSH. Use its Start-menu shortcut via `click`, or a scheduled task like the
|
||||
one `click` uses.
|
||||
|
||||
**Testing against a real Frame.** The VM reaches the headset on the LAN like
|
||||
any other computer. Run **Set Up Connection** in the VM's Frame Control once;
|
||||
that makes the VM's keys. So the VM doesn't keep access between test runs,
|
||||
afterwards take the lines it added out of the headset's
|
||||
`~/.ssh/authorized_keys`: the ones matching the VM's
|
||||
`~/.ssh/id_ed25519_frame.pub` and, if pairing made one,
|
||||
`~/.ssh/id_rsa_frame_devkit.pub`. Check that `ssh -o BatchMode=yes frame true`
|
||||
in the VM now fails. Then, around each run:
|
||||
|
||||
```sh
|
||||
scripts/windows-vm.sh frame-key add # let the VM's key into the headset
|
||||
# ... test ...
|
||||
scripts/windows-vm.sh frame-key remove # and take it out again
|
||||
```
|
||||
|
||||
`add` appends the VM's key tagged `windows-vm-test`, unless the headset
|
||||
already trusts that key through another line. `remove` deletes only the exact
|
||||
line `add` wrote, so it doesn't undo what Set Up Connection did, and it leaves
|
||||
the file alone if it can't rewrite it. Follow the shared-device
|
||||
procedure in [Headset smoke test](#headset-smoke-test) before installing or
|
||||
launching anything on the headset.
|
||||
|
||||
**Verified 2026-09-30:** Windows 11 Pro 25H2 (build 26200) against a Frame on
|
||||
BUILD_ID 20260925.6191901. The script was used to install Frame Control 0.4.0,
|
||||
connect it to the Frame and follow Remote Desktop through to the Frame's
|
||||
desktop ([what it does](streaming.md#a-see-and-control-the-frame-from-the-mac)).
|
||||
`frame-key` left `authorized_keys` byte-for-byte as it was.
|
||||
|
||||
## Owned media player
|
||||
|
||||
`tests/test_media.py` covers layout evidence and overrides, OU eye ordering,
|
||||
|
||||
Executable
+161
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env zsh
|
||||
# Mac or Linux: drive a Windows 11 test VM for Frame Control's Windows build.
|
||||
# The VM is a dockur/windows container on another machine; this reaches it over
|
||||
# SSH through that machine. See docs/testing.md#windows-test-vm.
|
||||
#
|
||||
# Usage: scripts/windows-vm.sh <command> [args]
|
||||
# up | down | status start it (waits for SSH), shut Windows down cleanly, show state
|
||||
# ps '<PowerShell>' run PowerShell as the VM's user
|
||||
# put <file> [<dir>] copy a file in (default: the user's Downloads)
|
||||
# shot <out.png> save the VM's screen
|
||||
# click <x> <y> click screen pixel x,y in the signed-in session
|
||||
# scroll <x> <y> <n> turn the wheel n notches at x,y (negative scrolls down)
|
||||
# keys <key>... type QEMU key names: a, shift-a, ret, tab, esc, spc ...
|
||||
# frame-key add|remove let the VM's Frame Control key into the headset (`frame`
|
||||
# alias) for a test run, then take it out again
|
||||
#
|
||||
# Set WINVM_HOST to the ssh alias of the machine running the container. Optional:
|
||||
# WINVM_CONTAINER (frame-winvm), WINVM_USER (frame), WINVM_PORT (2222: the VM's
|
||||
# sshd, published on that machine's loopback), WINVM_KEY (~/.ssh/id_ed25519_winvm).
|
||||
set -euo pipefail
|
||||
setopt extendedglob
|
||||
|
||||
host=${WINVM_HOST:?set WINVM_HOST to the ssh alias of the machine running the VM}
|
||||
ctr=${WINVM_CONTAINER:-frame-winvm}
|
||||
user=${WINVM_USER:-frame}
|
||||
port=${WINVM_PORT:-2222}
|
||||
key=${WINVM_KEY:-$HOME/.ssh/id_ed25519_winvm}
|
||||
opts=(-o ConnectTimeout=20 -o StrictHostKeyChecking=accept-new
|
||||
-o UserKnownHostsFile=${TMPDIR:-/tmp}/windows-vm-known_hosts -i $key -J $host)
|
||||
TAG=windows-vm-test # comment on the VM's key in the headset's authorized_keys
|
||||
|
||||
die() { print -u2 "windows-vm: $*"; exit 1 }
|
||||
int() { [[ $1 == (-|)<-99999> ]] || die "not a whole number (up to 99999): $1" }
|
||||
# These go into commands run by a shell on the other machine, so keep them plain.
|
||||
for v in $host $ctr $user; do [[ $v == [A-Za-z0-9_.]##[A-Za-z0-9_.-]# ]] || die "not a plain name: $v"; done
|
||||
[[ $port == <1-65535> ]] || die "WINVM_PORT isn't a port: $port"
|
||||
|
||||
# Windows' OpenSSH waits for stdin to close, so it always gets /dev/null. The
|
||||
# script travels UTF-16 base64-encoded, so no quoting survives two shells.
|
||||
vm_ps() {
|
||||
local b64=$(print -rn -- "\$ProgressPreference = 'SilentlyContinue'"$'\n'"$1" |
|
||||
iconv -f UTF-8 -t UTF-16LE | base64 | tr -d '\n')
|
||||
ssh $opts -p $port $user@127.0.0.1 "powershell -NoProfile -NonInteractive -OutputFormat Text -EncodedCommand $b64" </dev/null
|
||||
}
|
||||
|
||||
# QEMU's monitor inside the container: one command per line on stdin.
|
||||
monitor() { ssh $host "docker exec -i $ctr nc -q 1 -U /run/shm/monitor.sock" >/dev/null }
|
||||
|
||||
# Input has to come from the signed-in desktop session, not SSH's session 0, so
|
||||
# a scheduled task running as the user replays one click or wheel turn written
|
||||
# to input.txt, then deletes the file to say it's done. Any error stops it
|
||||
# before that, so the caller times out instead of reporting a click.
|
||||
INPUT_PS1='$ErrorActionPreference = "Stop"
|
||||
$a = (Get-Content "$PSScriptRoot\input.txt").Trim() -split "\s+"
|
||||
Add-Type -Namespace WinVm -Name Input -MemberDefinition @"
|
||||
[DllImport("user32.dll")] public static extern bool SetProcessDPIAware();
|
||||
[DllImport("user32.dll")] public static extern bool SetCursorPos(int x, int y);
|
||||
[DllImport("user32.dll")] public static extern void mouse_event(uint flags, int dx, int dy, int data, System.IntPtr extra);
|
||||
"@
|
||||
[WinVm.Input]::SetProcessDPIAware() | Out-Null
|
||||
if (-not [WinVm.Input]::SetCursorPos([int]$a[0], [int]$a[1])) { throw "SetCursorPos failed" }
|
||||
Start-Sleep -Milliseconds 150
|
||||
if ($a[2] -eq "click") {
|
||||
[WinVm.Input]::mouse_event(0x2, 0, 0, 0, [IntPtr]::Zero); Start-Sleep -Milliseconds 60
|
||||
[WinVm.Input]::mouse_event(0x4, 0, 0, 0, [IntPtr]::Zero)
|
||||
} else { [WinVm.Input]::mouse_event(0x800, 0, 0, 120 * [int]$a[3], [IntPtr]::Zero) }
|
||||
Remove-Item "$PSScriptRoot\input.txt"'
|
||||
|
||||
pointer() { # x y click|wheel [notches]
|
||||
local b64=$(print -rn -- $INPUT_PS1 | base64 | tr -d '\n')
|
||||
vm_ps '$ErrorActionPreference = "Stop"
|
||||
$d = Join-Path $env:LOCALAPPDATA "windows-vm"; $req = "$d\input.txt"; $mine = $false
|
||||
# Giving up: end the helper first, or it could wake up and act in a later request.
|
||||
function Abandon {
|
||||
Stop-ScheduledTask -TaskName WindowsVmInput -ErrorAction SilentlyContinue
|
||||
foreach ($i in 1..25) {
|
||||
if ((Get-ScheduledTask -TaskName WindowsVmInput -ErrorAction SilentlyContinue).State -ne "Running") { break }
|
||||
Start-Sleep -Milliseconds 200
|
||||
}
|
||||
Remove-Item $req -ErrorAction SilentlyContinue
|
||||
}
|
||||
trap { [Console]::Error.WriteLine("windows-vm: $_"); if ($mine) { Abandon }; exit 1 }
|
||||
# One request at a time: the helper, the task and input.txt are shared. Windows
|
||||
# frees the mutex when this process ends, however it ends.
|
||||
$lock = New-Object Threading.Mutex($false, "windows-vm-input")
|
||||
try { $mine = $lock.WaitOne(15000) } catch [Threading.AbandonedMutexException] { $mine = $true }
|
||||
if (-not $mine) { [Console]::Error.WriteLine("windows-vm: another click or scroll is still in progress"); exit 1 }
|
||||
New-Item -ItemType Directory -Force $d | Out-Null
|
||||
[IO.File]::WriteAllText($req, "'"$*"'")
|
||||
[IO.File]::WriteAllText("$d\input.ps1", [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String("'$b64'")))
|
||||
$act = New-ScheduledTaskAction -Execute powershell.exe -Argument "-NoProfile -WindowStyle Hidden -ExecutionPolicy Bypass -File `"$d\input.ps1`""
|
||||
$who = New-ScheduledTaskPrincipal -UserId $env:USERNAME -LogonType Interactive
|
||||
Register-ScheduledTask -TaskName WindowsVmInput -Action $act -Principal $who -Force | Out-Null
|
||||
Start-ScheduledTask -TaskName WindowsVmInput
|
||||
foreach ($i in 1..50) { if (-not (Test-Path $req)) { exit 0 }; Start-Sleep -Milliseconds 200 }
|
||||
Abandon
|
||||
[Console]::Error.WriteLine("windows-vm: no input after 10 s: is $env:USERNAME signed in on the VM screen, and is x,y on it?"); exit 1'
|
||||
}
|
||||
|
||||
(( $# )) || die "usage: see the top of $0"
|
||||
cmd=$1; shift
|
||||
case $cmd in
|
||||
up)
|
||||
ssh $host "docker start $ctr" >/dev/null
|
||||
for i in {1..60}; do
|
||||
vm_ps 'exit 0' 2>/dev/null && { print "up"; exit 0 }
|
||||
sleep 5
|
||||
done
|
||||
die "Windows didn't answer on SSH within 5 minutes" ;;
|
||||
down) # the container turns SIGTERM into an ACPI shutdown and waits for Windows
|
||||
ssh $host "docker stop -t 150 $ctr" >/dev/null && print "down" ;;
|
||||
status) ssh $host "docker ps -a --filter 'name=^$ctr\$' --format '{{.Names}}: {{.Status}}'" ;;
|
||||
ps) (( $# == 1 )) || die "usage: ps '<PowerShell>'"; vm_ps "$1" ;;
|
||||
put)
|
||||
[[ -f ${1:-} ]] || die "usage: put <file> [<dir>]"
|
||||
scp -q $opts -P $port $1 "$user@127.0.0.1:${2:-C:/Users/$user/Downloads}/${1:t}" </dev/null ;;
|
||||
shot)
|
||||
(( $# == 1 )) || die "usage: shot <out.png>"
|
||||
print "screendump /tmp/windows-vm-shot.ppm" | monitor
|
||||
sleep 1
|
||||
ssh $host "docker exec $ctr sh -c 'cat /tmp/windows-vm-shot.ppm && rm /tmp/windows-vm-shot.ppm'" | python3 -c '
|
||||
import re, struct, sys, zlib
|
||||
d = sys.stdin.buffer.read()
|
||||
m = re.match(rb"P6\s+(\d+)\s+(\d+)\s+255\s", d) or sys.exit("windows-vm: no screen dump")
|
||||
w, h = int(m[1]), int(m[2]); px = d[m.end():]
|
||||
raw = b"".join(b"\0" + px[y * w * 3:(y + 1) * w * 3] for y in range(h))
|
||||
chunk = lambda t, b: struct.pack(">I", len(b)) + t + b + struct.pack(">I", zlib.crc32(t + b))
|
||||
open(sys.argv[1], "wb").write(b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
|
||||
+ chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b""))
|
||||
' $1
|
||||
print $1 ;;
|
||||
click) (( $# == 2 )) || die "usage: click <x> <y>"; int $1; int $2; pointer $1 $2 click ;;
|
||||
scroll) (( $# == 3 )) || die "usage: scroll <x> <y> <n>"; int $1; int $2; int $3; pointer $1 $2 wheel $3 ;;
|
||||
keys)
|
||||
(( $# )) || die "usage: keys <key>..."
|
||||
for k; do [[ $k == [a-z0-9_.,/=-]## ]] || die "not a QEMU key name: $k"; done
|
||||
for k; do print "sendkey $k"; done | monitor ;;
|
||||
frame-key)
|
||||
[[ ${1:-} == (add|remove) ]] || die "usage: frame-key add|remove"
|
||||
pub=(${=$(vm_ps 'Get-Content (Join-Path $env:USERPROFILE ".ssh\id_ed25519_frame.pub")' | tr -d '\r')})
|
||||
[[ ${pub[1]:-} == ssh-ed25519 && ${pub[2]:-} == [A-Za-z0-9+/=]## ]] ||
|
||||
die "the VM has no Frame Control key yet: run Set Up Connection in the app first"
|
||||
# $1: add|remove, $2: the key's base64, $3: the exact line this script owns.
|
||||
ssh frame "sh -s -- $1 '$pub[2]' '$pub[1] $pub[2] $TAG'" <<'EOF'
|
||||
f=~/.ssh/authorized_keys
|
||||
if [ "$1" = add ]; then
|
||||
if grep -qxF "$3" "$f"; then exit 0; fi
|
||||
if grep -qF "$2" "$f"; then echo "the headset already trusts this key through another entry; left as it is"; exit 0; fi
|
||||
[ -z "$(tail -c 1 "$f")" ] || echo >> "$f" # a last line without a newline would swallow ours
|
||||
echo "$3" >> "$f"
|
||||
else
|
||||
t=$(mktemp "$f.XXXXXX") || exit 1
|
||||
grep -vxF "$3" "$f" > "$t" # 0: lines left, 1: none left, more: couldn't read or write
|
||||
if [ $? -gt 1 ] || ! chmod 600 "$t" || ! mv "$t" "$f"; then
|
||||
rm -f "$t"; echo "couldn't rewrite $f; it's unchanged" >&2; exit 1
|
||||
fi
|
||||
fi
|
||||
EOF
|
||||
print "frame-key $1: done" ;;
|
||||
*) die "unknown command: $cmd (see the top of $0)" ;;
|
||||
esac
|
||||
Reference in new issue
Block a user