Compare commits

..
Author SHA1 Message Date
saphidandClaude Opus 5.5 3f280ba59a Merge main into vr-utilities: the performance HUD alongside comfort, keyboard, panels and media
Also from review: a SteamVR build without the timing exports can't break status
(AttributeError), and the device test class runs when the file is run directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:59:15 +10:00
Alex Southwell 83fa5c6c5b Merge pull request #41 from saphid/panel-workspaces
Add panel switchers and document workspace feasibility
2026-09-29 11:56:15 +10:00
saphidandClaude Opus 5.5 3ef7312654 Merge main into panel-workspaces (testing notes)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:50:17 +10:00
Alex Southwell 70e7f7f5e5 Merge pull request #39 from saphid/family-comfort
Family and comfort: safe timers, casting, alerts and breaks
2026-09-29 11:38:47 +10:00
saphidandClaude Opus 5.5 2bd86207c7 Merge main into panel-workspaces: the panel switcher alongside media, analytics and the Mac view
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:37:18 +10:00
saphidandClaude Opus 5.5 be1ab129c9 Tests: read the page as UTF-8 (Windows' default codec can't read its arrows)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:33:34 +10:00
saphid a38d0cda6e Document shared Frame test procedure and blocked recheck 2026-09-29 11:16:29 +10:00
Alex SouthwellandClaude Opus 5.5 e702adbd77 Live view: a Desktop view that stays still, and Control to tap on the Frame (#46)
* Live view: a Desktop view that stays still, and Control to tap on the Frame

The live view gets a second source and a way to use the Frame from it:

- Desktop: the app panel in use in the headset, streamed from its own window
  (x11grab of gamescope's redirected window), so it doesn't move as the
  wearer looks around. A picker shows any other panel, view only.
- Control: on the Desktop view a tap or click lands exactly where you put it;
  drag is a mouse drag, press and hold right-clicks, two fingers scroll, and
  on a computer the mouse, wheel and keyboard work directly. On the headset
  view the view is a trackpad. A text field and key row type from a phone.

Input goes through gamescope's own EIS socket (the way Steam feeds Remote
Play input) with the libei already on the image: ui/frame_touch.py, over
the same long-lived ssh machinery as the keyboard agent, nothing to
install. It reaches the panel that has focus on either X display, which
the KDE Connect route can't. Verified on the Frame and from the iPhone app
in the Simulator.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from review

- Keys held on the Frame are released with buttons when Control stops or
  the view loses focus; keys for the Frame no longer trigger Frame Control's
  own shortcuts.
- Taps only act when the picture on screen is the panel in use; positions,
  presses, keys, text and scrolls name their panel (display and window: ids
  repeat across :0 and :1, told apart by pid), and the Frame drops them if
  focus has moved on. Releases always go.
- While connecting, a tap keeps its position; on an error only releases wait
  and retries back off; trimming a long queue never drops a release.
- Lifting one of two scrolling fingers ends the scroll; a cancelled touch
  isn't a tap; clicks and holds on the bars around the picture do nothing.
- A capture loop from before a Live restart can't stop the new video.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: close the targeting gaps from the second review

- The focused panel's display comes from GAMESCOPE_FOCUS_DISPLAY (gamescope
  packs ":1" into the first value), so a window id repeated across :0 and :1
  can't be mistaken; the pid is only the fallback.
- Presses, keys, text and scrolls read focus afresh on the Frame; only moves
  use a reading up to a second old.
- A gesture remembers the panel it started on and does nothing more if that
  stops being the one in use; a press with no panel to aim at isn't sent.
- Opening a screenshot clears the panel Control would act on; switching to
  another app releases held keys and buttons.
- Trimming keeps a click with its position; the error backoff holds for new
  input too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from the SWE-2 Max review

- The Frame side tracks keys as well as buttons and lets go of both when the
  session ends.
- A stale tap tells the page, which re-reads the panels at once.
- A paused input device waits instead of ending the session; only a
  disconnect does. An OS error on one event skips it.
- Presses check focus with two property reads and do the full lookup only
  when it changed.
- Writes to an agent's stdin are serialized, so two devices sending at once
  can't tear a line (the keyboard agent too).
- Connecting gives up with a message after 15 s instead of hanging on
  "Connecting…"; text goes in 100-character pieces so releases don't wait
  behind a long paste; a cancelled mouse gesture releases what's held.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: stale clears, pauses reconverge, pastes split per request

- The Frame says it has caught up as soon as an aimed event lands after a
  stale one, so the page stops re-reading the panels.
- After a device pause it lets go of everything it holds (releases that
  arrived while paused were dropped), and waits for the device once per
  batch, not once per event.
- The quick focus check no longer freshens the panel geometry's age.
- Each request carries at most about 100 characters of text.
- Turning Control off while it connects doesn't report an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: build the socket path on the Frame, so Windows can import it for tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: any event that goes through clears the stale flag

A trackpad move names no panel, so waiting for an aimed event could leave
the page re-reading panels for the rest of the session; a release still
aimed at the old panel doesn't count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:13:21 +10:00
saphidandClaude Opus 5.5 ab1985b6b3 Comfort: a state missing 'started' can't crash status (timestamps of 0 still count)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:12:41 +10:00
saphidandClaude Opus 5.5 1203ec0a9e Merge main into family-comfort; a session state without 'started' can't crash status
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:09:35 +10:00
Alex Southwell ef56025ad1 Merge pull request #19 from saphid/frame-input
Keyboard and trackpad for the Frame, through its own KDE Connect
2026-09-29 10:33:35 +10:00
Alex Southwell 697452e5a9 Merge pull request #42 from saphid/theatre-media
Add owned Frame-side theatre and stereo media previews
2026-09-29 10:27:40 +10:00
saphid ae7f733b90 Merge remote-tracking branch 'origin/main' into theatre-media
# Conflicts:
#	ui/server.py
2026-09-29 10:21:27 +10:00
saphidandClaude Opus 5.5 b00db2484d Keep the backslash upload-name test POSIX-only
Windows treats the backslash as a separator, so that name can't occur there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:15:37 +10:00
saphid b0d6039eec Merge remote-tracking branch 'origin/main' into theatre-media
# Conflicts:
#	docs/testing.md
#	ui/server.py
2026-09-29 10:08:45 +10:00
saphidandClaude Opus 5.5 86b8ab1d82 Assert the start-failure test reaches systemd-run
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:01:17 +10:00
saphidandClaude Opus 5.5 020e3386e1 Make media stop race-free and cover start failures
Stop always calls systemctl and treats exit 5 (unit already collected)
as done, so there is no is-active/stop race. Cleanup never masks the
copy error, the play ssh timeout covers the remote worst case, and
tests cover stop exit codes and systemd-run stderr reporting.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:30:03 +10:00
saphidandClaude Opus 5.5 f81f70ed5d Fix media stop, upload cleanup and review findings
- Stop is a no-op when the collected player unit is already gone
  (raw systemctl stop exits 5 on the Frame; verified 2026-09-29).
- Surface systemd-run stderr when the player can't start.
- Keep the copy error if the cleanup ssh also fails; reject upload
  names that the play path can never accept.
- Allow 60 s for play (ffprobe 30 s + systemd-run 15 s remote).
- Docs: four-hour cap is unconditional; no delete action yet; fix a
  garbled timing sentence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:22:22 +10:00
saphid 222d5eccc9 fix(comfort): harden timer recovery and notification UX after review 2026-09-29 08:31:00 +10:00
saphid 08c306c3a6 docs: record CI results and incomplete independent review 2026-09-28 22:51:03 +10:00
saphid 4d4e45f622 Pin fake Frame data directory modes 2026-09-28 22:46:00 +10:00
saphid 6ecd0cea39 Match fake Frame user data ownership to the headset 2026-09-28 22:44:14 +10:00
saphid 6a63a5a597 docs: record VR device evidence and paused comfort controls 2026-09-28 22:41:54 +10:00
saphid 3fb541dce7 feat: add OpenVR performance HUD and optional VR utilities 2026-09-28 22:41:45 +10:00
saphid c335cd5130 Fix media test portability and e2e discovery 2026-09-28 22:37:28 +10:00
saphid 91c5853627 Refresh panel switcher screenshot from final device check 2026-09-28 22:34:53 +10:00
saphid 0d448ab510 Add owned OpenVR media playback and stereo previews 2026-09-28 22:33:50 +10:00
saphid 72352c5914 Add companion and headset panel switchers with feasibility evidence 2026-09-28 22:33:23 +10:00
saphid d3fa282377 docs(comfort): include verified desktop and iPhone notification evidence 2026-09-28 22:32:54 +10:00
saphid 0622afcae9 feat(ui): add family controls, casting and native notifications 2026-09-28 22:29:58 +10:00
saphid 522c46ed6f feat(comfort): run safe session timers and alerts on the Frame 2026-09-28 22:29:58 +10:00
53 changed files with 5128 additions and 215 deletions

No files matched your search

+11 -5
View File
@@ -35,8 +35,8 @@ See what the headset sees, install games and Android apps, move files and text a
<tr> <tr>
<td width="50%" valign="top"> <td width="50%" valign="top">
**👓 Headset view**<br> **👓 Headset view and Desktop**<br>
Live video of what the lenses show (about 30 fps), or a still of both eyes. Zoom, pan, full screen, save as PNG. Live video of what the lenses show, or of the app panel in use, flat and still however the wearer looks around. Turn on Control and tap or click right on it to use the Frame from your phone or computer.
</td> </td>
<td width="50%" valign="top"> <td width="50%" valign="top">
@@ -93,9 +93,15 @@ SSH, SFTP, Steam Link, remote desktop, volume, sleep, restart and shut down.
</tr> </tr>
</table> </table>
Nothing is installed on the Frame for any of this: the app uses what SteamOS The optional [Family and comfort](docs/family-comfort.md) card adds session
limits, breaks, local alerts and one-click casting. A session copies a small
Frame Control worker into your headset user account.
For the other features, nothing is installed on the Frame: the app uses what SteamOS
already ships (sideloading a game copies Valve's own devkit scripts to already ships (sideloading a game copies Valve's own devkit scripts to
`~/devkit-utils`, as Valve's Devkit Client does). [How each feature works](docs/frame-control.md). `~/devkit-utils`, as Valve's Devkit Client does). The optional
[performance HUD](docs/vr-utilities.md) copies our own Python helpers into
`~/.local/share/frame-control/vr/`. [How each feature works](docs/frame-control.md).
## Install ## Install
@@ -209,7 +215,7 @@ Frame's software fits together, all checked against a real headset and labelled
| [Android apps (Lepton)](docs/apks.md) | Sideloading, the rated F-Droid catalogue, per-app instances | | [Android apps (Lepton)](docs/apks.md) | Sideloading, the rated F-Droid catalogue, per-app instances |
| [Sideloading Linux and Windows games](docs/sideloading.md) | A .zip, folder or .exe as a Steam Devkit Game, runtime detection | | [Sideloading Linux and Windows games](docs/sideloading.md) | A .zip, folder or .exe as a Steam Devkit Game, runtime detection |
| [Install links for websites](docs/web-install.md) | `frame-control://install` links and manifests, the rules, a button to paste | | [Install links for websites](docs/web-install.md) | `frame-control://install` links and manifests, the rules, a button to paste |
| [Steam games](docs/steam-games.md) · [VR video](docs/vr-video.md) · [WebXR in Chromium](docs/webxr-chromium.md) | Installing and buying, watching VR180/360, the Chromium build | | [VR comfort and HUD](docs/vr-utilities.md) · [Steam games](docs/steam-games.md) · [VR video](docs/vr-video.md) · [WebXR in Chromium](docs/webxr-chromium.md) | Installing and buying, watching VR180/360, the Chromium build |
| [Mac in the headset](docs/mac-in-headset.md) | Mac windows and screens as panels in the Frame, with laser and keyboard input | | [Mac in the headset](docs/mac-in-headset.md) | Mac windows and screens as panels in the Frame, with laser and keyboard input |
| [VR mods and custom songs](docs/mods.md) | Per-game feasibility, real-Frame results and blockers; no installer yet | | [VR mods and custom songs](docs/mods.md) | Per-game feasibility, real-Frame results and blockers; no installer yet |
| [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes | | [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes |
+17 -2
View File
@@ -1,7 +1,7 @@
// Frame Control as a desktop app (macOS, Windows, Linux): starts ui/server.py on // Frame Control as a desktop app (macOS, Windows, Linux): starts ui/server.py on
// a free loopback port and shows it in a native window. The server does all the // a free loopback port and shows it in a native window. The server does all the
// work over the `frame` SSH alias; this file only hosts it. // work over the `frame` SSH alias; this file only hosts it.
const { app, BrowserWindow, Menu, clipboard, dialog, ipcMain, shell } = require("electron"); const { app, BrowserWindow, Menu, Notification, clipboard, dialog, ipcMain, shell } = require("electron");
const { execFile, spawn } = require("child_process"); const { execFile, spawn } = require("child_process");
const { promisify } = require("util"); const { promisify } = require("util");
const fs = require("fs"); const fs = require("fs");
@@ -254,6 +254,21 @@ ipcMain.handle("update:get", (e) => fromUi(e) ? publicUpdate() : null);
ipcMain.handle("update:check", (e) => fromUi(e) ? checkForUpdate({ manual: true }).then(publicUpdate) : null); ipcMain.handle("update:check", (e) => fromUi(e) ? checkForUpdate({ manual: true }).then(publicUpdate) : null);
ipcMain.handle("update:install", (e) => { if (fromUi(e)) installUpdate(); }); ipcMain.handle("update:install", (e) => { if (fromUi(e)) installUpdate(); });
ipcMain.handle("comfort:notify", (e, message) => {
if (!fromUi(e) || typeof message !== "string" || message.length > 500) throw new Error("Invalid notification");
if (!Notification.isSupported()) throw new Error("System notifications are unavailable");
return new Promise((resolve, reject) => {
const notification = new Notification({title: "Frame Control", body: message});
const timer = setTimeout(() => reject(new Error("Notification delivery was not confirmed. Check system notification settings.")), 5000);
notification.once("show", () => { clearTimeout(timer); resolve(true); });
notification.once("failed", (_event, error) => {
clearTimeout(timer);
reject(new Error("Notification delivery failed. Check system notification settings: " + error));
});
notification.show();
});
});
// frame-control://install links from websites (docs/web-install.md). They can // frame-control://install links from websites (docs/web-install.md). They can
// arrive before the window or server exists (macOS open-url on a cold launch), // arrive before the window or server exists (macOS open-url on a cold launch),
// so they wait here until the page asks for them. The page checks the link with // so they wait here until the page asks for them. The page checks the link with
@@ -376,7 +391,7 @@ function createWindow() {
title: "Frame Control", backgroundColor: BG, show: false, title: "Frame Control", backgroundColor: BG, show: false,
...(IS_MAC ? { titleBarStyle: "hiddenInset", trafficLightPosition: { x: 18, y: 26 } } ...(IS_MAC ? { titleBarStyle: "hiddenInset", trafficLightPosition: { x: 18, y: 26 } }
: { icon: path.join(__dirname, "build", "icon.png") }), : { icon: path.join(__dirname, "build", "icon.png") }),
webPreferences: { contextIsolation: true, nodeIntegration: false, sandbox: true, webPreferences: { contextIsolation: true, nodeIntegration: false, sandbox: true, backgroundThrottling: false,
preload: path.join(__dirname, "preload.js") }, preload: path.join(__dirname, "preload.js") },
}); });
win.once("ready-to-show", () => win.show()); win.once("ready-to-show", () => win.show());
+1
View File
@@ -9,6 +9,7 @@
const { contextBridge, ipcRenderer, webUtils } = require("electron"); const { contextBridge, ipcRenderer, webUtils } = require("electron");
contextBridge.exposeInMainWorld("frameApp", { contextBridge.exposeInMainWorld("frameApp", {
notify: (message, request) => ipcRenderer.invoke("comfort:notify", message, request),
readClipboard: () => ipcRenderer.invoke("clipboard:read"), readClipboard: () => ipcRenderer.invoke("clipboard:read"),
setUpConnection: () => ipcRenderer.invoke("connection:setup"), setUpConnection: () => ipcRenderer.invoke("connection:setup"),
// While the keyboard-and-trackpad panel holds the keyboard, ⌘W, ⌘R and the rest go to the Frame. // While the keyboard-and-trackpad panel holds the keyboard, ⌘W, ⌘R and the rest go to the Frame.
+53
View File
@@ -0,0 +1,53 @@
{
"date": "2026-09-28",
"os": {
"version": "0.4.1",
"build": "20260925.6191901",
"variant": "vr"
},
"performance": {
"compositorFps": 72.0,
"frameMs": 13.89,
"appFps": null,
"gpuMs": 3.07,
"compositorCpuMs": 0.61,
"cpuPercent": 40.6,
"gpuMHz": 903.0
},
"batteryPercent": 16,
"maxTempC": 73.5,
"ownership": [
{
"id": 1009850,
"owned": false,
"installed": false,
"frame": 0
},
{
"id": 1173510,
"owned": false,
"installed": false,
"frame": 0
},
{
"id": 1068820,
"owned": false,
"installed": false,
"frame": 0
},
{
"id": 908520,
"owned": false,
"installed": false,
"frame": 0
},
{
"id": 1494460,
"owned": false,
"installed": false,
"frame": 0
}
],
"hudLifecycle": "open, duplicate-open, close passed before control-test pause; overlay probe removal verified",
"comfort": "Initial 1cm seated probe restored exactly. Later commits/readback disagreed while Frame in use; Alex paused control tests. No controls shipped."
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 269 KiB

+58
View File
@@ -0,0 +1,58 @@
# Independent review attempts — 2026-09-28
Target: the implementation and evidence in
[3fb541d](https://github.com/saphid/frame-control/commit/3fb541d) and
[6a63a5a](https://github.com/saphid/frame-control/commit/6a63a5a), supplied as a
frozen diff before those commits were made. No executable code changed after
the final review snapshot.
Requested model: **SWE-2 Max**, explicitly selected with `--model swe-2-max`.
No completed verdict or model self-identification was returned. This is not a
passed review, and there is no "no actionable findings" claim.
## First attempt
Launcher:
```sh
devin -p --model swe-2-max --permission-mode auto --respect-workspace-trust false --prompt-file /tmp/frame-vr-review-prompt.txt
```
The prompt required read-only review, no delegation, no edits and no Frame
access. The reviewer inspected surrounding code, then stopped while checking
the public OpenVR header. The tool runner reported:
> warning: rejected a tool call that requires confirmation. Running in non-interactive mode.
The real launcher exit status was **0**, but no verdict was returned. A zero
process status here is not evidence that the review completed.
## Tool-free retry
Launcher:
```sh
devin -p --model swe-2-max --permission-mode auto --respect-workspace-trust false --prompt-file /tmp/frame-vr-review-final-prompt.txt
```
The self-contained prompt supplied the complete frozen changes, surrounding
code, standards and locally fetched authoritative OpenVR header excerpts. It
explicitly prohibited tools, edits, delegation and device access. This avoided
the first attempt's permission boundary without escalating permissions.
No output or verdict arrived within the fifteen-minute review window. The
process was sent SIGTERM at 918 seconds; the shell recorded real exit status
**143**. Findings are unavailable. Review must be completed before considering
this partial draft ready; playspace feasibility is also still paused.
## Other validation
- 166 Python unit tests passed locally.
- 8 website tests and UI JavaScript syntax passed locally.
- Desktop and phone-width attached-preview checks passed using live telemetry;
paid optional install buttons were disabled, and unavailable metrics cleared.
- [Fake-Frame CI](https://github.com/saphid/frame-control/actions/runs/36423548487/job/108931878908)
passed, as did Windows/Linux server tests and the main checks job. Docker was
unavailable locally. macOS/iOS jobs were still queued at this handoff.
- Real-device evidence and the paused-control limitation are in
[VR utilities](../../vr-utilities.md).
+122
View File
@@ -0,0 +1,122 @@
# Family and comfort
Frame Control's Home tab has a **Family and comfort** card, on desktop and
on iPhone. No third-party notification or parental-control app is needed.
This is Frame Control code using Python, Steam and SteamVR already on the Frame.
![Family and comfort controls in the desktop app](img/comfort-desktop.png)
## Sessions
Set a limit of 1–240 minutes, optional break and check-in intervals, then
**Start session**. Break and check-in intervals of 0 turn those reminders off.
**Cancel session** cancels the timer and monitoring without changing the game.
Cancel before starting a session with different settings.
The Frame shows a one-minute warning, then opens Steam Home in its dashboard.
**Games stay running**: save and pause before the limit. Some games pause when
the dashboard opens; others do not. There is no kill, power-off, Steam restart,
account restriction or parental lock. The wearer can return to the game.
**Documented implementation:** the timer is a single, opt-in Python worker in
the Frame user's account. Desktop and iPhone share its state. It keeps going
when the companion disconnects, closes or is suspended. It exits after
completion or cancellation (normally within five seconds). Cancellation waits
for any in-flight SteamVR action to finish within its timeout; it is not a boot
service. A Frame reboot invalidates the session. Suspend counts toward the
limit, using Linux's boot-time clock. If a warning was delayed by suspend or a
SteamVR failure, Home waits until at least a full minute after a successful
warning. A failed Home transition remains active and retries, with an error
shown in the companion. A stale worker is reported as unverified enforcement.
## Alerts and breaks
During a session, battery, overheating and check-in alerts go to connected
companions. Break reminders and session warnings also appear on the headset.
- **Low battery:** 15% or below while discharging. One alert until charging or
recovery to 20%, so values around 15% do not produce repeated notifications.
- **Overheating:** a thermal zone reaches its own kernel-reported hot/critical
trip, or the battery reports `Overheat`. Missing sensors mean unknown, not
safe. These are status alerts, not medical advice or an extra thermal governor.
- **Check in:** an alert after the chosen number of active minutes.
- **Breaks:** a SteamVR reminder and companion notification at the chosen interval.
**Inferred:** SteamVR activity levels 1 and 2 are a useful proxy for use, not
proof someone is wearing the headset. Inactive readings reset continuous use;
missing readings add no time. Long gaps count at most 30 seconds. Breaks and
check-ins are distinct from the elapsed-time session limit.
Click **Enable / test notifications** on each companion. iOS asks for permission;
macOS, Windows and Linux follow their notification settings. The page also shows
recent events and errors. Keep Frame Control open and connected for companion
alerts. **Phone alerts are local, not push notifications:** iOS suspension,
force-quit or a lost SSH connection prevents live delivery. Old alerts are not
replayed as a notification burst on reconnect. Headset warnings and the session
limit continue without the phone. A physical iPhone's background delivery has
not been verified and is not guaranteed.
## Casting
**Cast headset view** starts the existing headset Live view and requests full
screen where supported. Show that screen to people in the room, or use the
computer/phone's own screen mirroring. It creates no new stream transport,
public URL or LAN server. iPhone uses the inline viewer if full screen is not
available. The image includes private content visible to the wearer.
## What is installed
The shared authenticated `/api/comfort` endpoint copies three bundled Python
files to `~/.cache/frame-control/comfort/<content-hash>/`. Session state and
locks live in `~/.local/state/frame-control/comfort/`, with a private directory
and 0600 state file. There is no network listener or system service. Cancel a
session before removing these directories. The iPhone's normal server still
exits on disconnect; the explicitly started comfort worker is the exception.
## Verification
**Verified 2026-09-28**, SteamOS 0.4.1, build `20260925.6191901`: shipped
`/opt/steamvr/bin/linuxarm64/vrcmd --notify TEXT` reported success for a custom
reminder. Steam's CDP `SteamUIStore.Navigate('/library/home')` and
`SteamClient.OpenVR.VROverlay.ShowDashboard('valve.steam.gamepadui.main')`
opened Home while the running app ID stayed unchanged. Prior page and dashboard
visibility were restored. Kernel hot/critical trips and SteamVR activity were
read from the real device. No temperature or battery fault was induced.
**Verified locally:** deterministic fake-Frame tests cover late warnings,
failed warnings/Home actions, cancellation, activity gaps, thresholds, duplicate
suppression, reboot invalidation, shared session state and the exact Home
JavaScript. `python3 -m unittest discover -s tests` runs them. The iOS Simulator
build tests notification content and bounds. Physical iPhone delivery and
wearer-perceived headset notification visibility remain unverified.
**Verified end to end on the same Frame:** a two-minute session with no companion
connection for 135 seconds emitted its warning, break and check-in, then opened
Home. The running app ID was unchanged; the test restored the previous page and
dashboard visibility and confirmed the worker exited. Casting through the Home
shortcut decoded the existing headset stream at 30 fps.
**Verified on the iOS 26.5 Simulator:** connected to the real Frame, approved the
notification prompt, and saw the native Frame Control test banner. Seven iOS
tests passed.
![Native test notification in the iOS Simulator](img/comfort-notification-ios.png)
Desktop and 390-pixel phone layouts had no horizontal overflow.
On macOS the development Electron app's real notification attempt was denied
(`UNErrorDomain` 1); the bridge now returns that failure instead of reporting
success. Successful macOS/Windows/Linux notification display remains unverified.
**Verified on the real Frame:** its naturally discharging 15% battery produced
one low-battery event during a short session; the test then cancelled the
session. Overheating alerts use fake sensor samples in tests: the shared
headset was not deliberately overheated.
**Verified 2026-09-29 on the same Frame:** a fresh one-minute session opened
Home more than 60 seconds after the successful warning. The test restored the
previous page and dashboard visibility. Local regression coverage now includes
slow notification delivery, a total Home-action timeout, failed worker startup,
unreadable saved state, malformed activity samples and notification UX: 173
Python tests passed. Desktop and 390-pixel layouts were checked again; system
notification-denial guidance stayed visible across polls. Initial event history
did not replay notifications, and only the latest new event was announced.
+2
View File
@@ -40,6 +40,8 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600
| Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale` (installable in `~`, see below), `krfb`, `wayvnc`. | Script design | | Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale` (installable in `~`, see below), `krfb`, `wayvnc`. | Script design |
| **SteamOS updates arrive on their own.** The Frame went from 0.3.0 (build 20260922.6101926) to **0.4.1, build 20260925.6191901**, between 2026-09-27 and 2026-09-28 with no action from us; `~` (keys, user Flatpaks, `~/.local/share`) survived. **Verified 2026-09-28.** | Keep changes in `~` | | **SteamOS updates arrive on their own.** The Frame went from 0.3.0 (build 20260922.6101926) to **0.4.1, build 20260925.6191901**, between 2026-09-27 and 2026-09-28 with no action from us; `~` (keys, user Flatpaks, `~/.local/share`) survived. **Verified 2026-09-28.** | Keep changes in `~` |
| **Valve's package repository has more than the image.** `pacman -Si` / `pacman -Sp` work as `steamos` without root and list Valve's own builds, such as `kdeconnect` 24.02.2 and `python-evdev` 1.7.0 in `extra`. Unpacking those packages into `~` runs them without touching the read-only root. The repository URLs say not to share them, so never write them down; Valve also publishes each build's source package there (`sources/packages/`), which is how Frame Control got the complete source for the KDE Connect it ships. **Verified 2026-09-28**, SteamOS 0.4.1. | [streaming.md](streaming.md#input-type-and-point-in-the-frame-from-the-mac-or-iphone) | | **Valve's package repository has more than the image.** `pacman -Si` / `pacman -Sp` work as `steamos` without root and list Valve's own builds, such as `kdeconnect` 24.02.2 and `python-evdev` 1.7.0 in `extra`. Unpacking those packages into `~` runs them without touching the read-only root. The repository URLs say not to share them, so never write them down; Valve also publishes each build's source package there (`sources/packages/`), which is how Frame Control got the complete source for the KDE Connect it ships. **Verified 2026-09-28**, SteamOS 0.4.1. | [streaming.md](streaming.md#input-type-and-point-in-the-frame-from-the-mac-or-iphone) |
| **gamescope has its own input injection.** An EIS socket at `/run/user/1000/gamescope-0-ei` (libei 1.4.1 is on the image) offers "Gamescope Virtual Input": relative and absolute pointer, buttons, scroll, keyboard. It drives the panel that has focus in the headset, on either X display. Focus moves only with the controller's laser (or to a new panel when none has it); `gamescopectl focus_info` prints the focus state to the journal. **Verified 2026-09-29.** | [streaming.md](streaming.md#live-view-and-control-watch-a-panel-and-tap-on-it) |
| **A panel's own pixels:** `ffmpeg -f x11grab -window_id <window> -i :<display>` captures one window (x11grab of the root is black under gamescope). Panels live on `:0` (Steam's UI, windows tagged by `panel-on-frame.sh`) or `:1` (apps Steam starts). **Verified 2026-09-29.** | Frame Control's Desktop view |
| **gamescope runs two Xwayland displays.** `:0` holds Steam's VR bar and menus (`valve.steam.gamepadui.*`) and ignores XTest pointer motion; `:1` holds apps such as Chromium and takes it. There's also a libei socket, `/run/user/1000/gamescope-0-ei`. **Verified 2026-09-28**, SteamOS 0.4.1. | Keyboard and trackpad | | **gamescope runs two Xwayland displays.** `:0` holds Steam's VR bar and menus (`valve.steam.gamepadui.*`) and ignores XTest pointer motion; `:1` holds apps such as Chromium and takes it. There's also a libei socket, `/run/user/1000/gamescope-0-ei`. **Verified 2026-09-28**, SteamOS 0.4.1. | Keyboard and trackpad |
| Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` | | Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` |
| `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things | | `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things |
Binary file not shown.

After

Width:  |  Height:  |  Size: 192 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 71 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 409 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 901 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

+12 -1
View File
@@ -26,7 +26,10 @@ as its transport too), so the desktop and phone share one code path. Android
display settings use `podman exec` into each Lepton container instead of adb, display settings use `podman exec` into each Lepton container instead of adb,
which the Frame doesn't have. which the Frame doesn't have.
Nothing is left running on the Frame after the phone disconnects; the copied The app server stops after the phone disconnects. An explicitly started
[comfort session](family-comfort.md) keeps its timer and headset reminders running
until the session ends or is cancelled; phone notifications require the app to
remain connected and running. The copied
files stay in `~/.cache/frame-control` (delete it any time). files stay in `~/.cache/frame-control` (delete it any time).
## Pairing ## Pairing
@@ -108,3 +111,11 @@ running), a real sleep/restart/shut down on the Frame, and a physical iPhone.
Debug builds have Simulator test hooks (`FRAME_TEST_HOST`, `FRAME_TEST_PAGE`, Debug builds have Simulator test hooks (`FRAME_TEST_HOST`, `FRAME_TEST_PAGE`,
`FRAME_TEST_JS`, and the tunnel URL in the app's Caches folder); release builds `FRAME_TEST_JS`, and the tunnel URL in the app's Caches folder); release builds
don't. don't.
## Family and comfort
The shared Home card sets session limits, breaks and check-ins, and offers
**Cast headset view**. **Enable / test notifications** requests iOS notification
permission and sends a local test. These are local notifications, not APNs push;
iOS background suspension can interrupt phone alerts. The headset timer still
runs. See [the behavior and verification limits](family-comfort.md).
+15
View File
@@ -468,3 +468,18 @@ versus on, medians of the runs, ms):
window to the front first. window to the front first.
- Ctrl stays Ctrl. On the Mac, copy is ⌘C, so use Meta+C on a keyboard paired - Ctrl stays Ctrl. On the Mac, copy is ⌘C, so use Meta+C on a keyboard paired
with the Frame. with the Frame.
## Switching panels and workspace limits
**Tools → Panel switcher** lists open SteamVR panels, including Mac viewers.
Use **Show** to request focus or **Open in headset** for Frame Control's own
switcher panel. It uses SteamVR/gamescope and Chromium, with no third-party
overlay app. [Device checks and limits](panels.md#frame-controls-panel-switcher)
include the difference between a panel surviving a scene launch and staying
visible over it.
Saved spatial layouts are blocked on this build: the public OpenVR transform
setter denies access to gamescope-owned panels. Reconnecting an existing viewer
is supported; restoring its room position after a reboot is not. We do not
save short-lived Mac window IDs or viewer access keys as if they were a durable
workspace. See [the feasibility evidence](panels.md#saved-spatial-layouts-blocked-on-the-current-panel-route).
+5 -4
View File
@@ -102,10 +102,11 @@ Still open: 4, 6, 7, 12–15, 16 (off-LAN and after a reboot), 17–21.
20. **F-Droid 2.0** (Compose 1.12): does it run? If so, the catalogue can 20. **F-Droid 2.0** (Compose 1.12): does it run? If so, the catalogue can
install it instead of 1.17.2. install it instead of 1.17.2.
21. **DeoVR local files:** does DeoVR's file browser show `Videos → VR` 21. **Owned media player:** worn-headset comfort, audio quality/lip sync, long
(the symlink from `push-vr-video.sh`) or `Z:\home\steamos\Videos\VR`, and do movies and 4K/8K decoding remain to check. Native spatial-photo container
the colour-coded test clips play in 3D (red left eye, cyan right) for both extraction, large/immersive splats and existing-panel theatre docking need
H.264 and H.265? Does the DLNA browser find a server on the Mac? further implementation. Remote eye isolation, short hardware decode and
owned-overlay cleanup are verified; see [vr-video.md](vr-video.md).
## Verified 2026-09-27 ## Verified 2026-09-27
+168 -2
View File
@@ -121,5 +121,171 @@ a limit on the number of floating panels.
script needed. script needed.
- **Inside the desktop panel**: KWin tiling (Meta+arrow keys with a Bluetooth - **Inside the desktop panel**: KWin tiling (Meta+arrow keys with a Bluetooth
keyboard) or virtual desktops arrange windows within the 1280×800 rectangle. keyboard) or virtual desktops arrange windows within the 1280×800 rectangle.
- **Windows-only overlay tools** (Desktop+, OVR Toolkit, OVRdrop) do this for a - **Optional overlay tools:** Desktop+, OVR Toolkit and similar software are
PC's desktop in SteamVR. They don't run on the Frame's standalone Linux. separate from Frame Control. Public reports describe some Proton support;
Windows-only does not by itself prove a Frame app cannot run. Local status
and sources are in [VR utilities](vr-utilities.md).
- **Our performance HUD:** Home → VR comfort and performance → Open HUD in
headset creates its own gamescope panel using built-in tools. It needs no
third-party overlay app. [Metrics and verification](vr-utilities.md).
## Frame Control's panel switcher
**Verified 2026-09-28**, SteamOS 0.4.1, BUILD_ID `20260925.6191901`,
SteamVR 2.18.1: **Tools → Panel switcher** lists SteamVR's open main panels,
including panels that are currently hidden. **Show** asks SteamVR to bring one
forward. **Open in headset** opens the same switcher as its own panel; choose
it again from Steam's dashboard after switching away. Refresh updates the list.
This is a list, not thumbnail Exposé.
![Frame Control's switcher rendered on the Frame](img/panel-switcher.png)
This is our own Python/HTML implementation (`ui/frame_panels.py`), using the
Frame's shipped `vrcmd` OpenVR client and gamescope. The headset page uses
Chromium (Chromium XR when present, then system Chromium, then the existing
Chromium Flatpak). No XSOverlay, OVR Toolkit, WayVR or other overlay application
is needed. This dependency boundary also applies to future layout and panel
persistence work: platform APIs and bundled libraries are fine; another app
must not implement the feature for us.
The companion runs the helper over SSH. Opening it in the headset installs a
copy under `~/.local/share/frame-control/panels/` and starts a loopback HTTP
server and an isolated Chromium profile. There is no startup service or global
setting change. Close the switcher to stop its server and browser. Other
Chromium profiles, Steam and SteamVR are left alone. If the window or runtime
closes, use **Open in headset** again.
The page carries a random, per-process access key in its URL fragment, removes
it from the address bar, keeps it in tab session storage for page reloads, and
sends it in a header. Panel lists and actions need
that key; Host and Origin checks reject other sites. The key permits only
listing panels, requesting focus and closing this switcher. Like Mac viewer
launch tickets, it is initially readable by another process running as the
same Frame user. Panel titles are rendered as text, never HTML. The companion
retains its existing request guards. No Mac capture credentials cross this API.
**Verified:** the real headset page rendered its panel list (image above), its
HTTP focus request changed `GAMESCOPE_FOCUSED_APP` to `2000999030`, a request
without the key returned HTTP 403, and Close stopped the helper and its browser.
Opening an already running switcher requests its focus rather than creating a
second one. The companion uses the same list/focus helper. **Unverified:** laser
selection while wearing the headset, physical placement, and non-XR Chromium.
The API reports that focus was *requested*: another action can take focus before
we observe the result. Closed panels are rejected after re-enumeration.
### Shared-device recheck, 2026-09-29
**Verified:** the follow-up's atomic `mkdir /tmp/frame-test.lock` attempts
failed because another thread held the lock. The existing lock was left alone;
no applications were installed, launched or stopped in this follow-up. The last
read-only battery check showed 62%, charging. The 180 Python and 8 website tests
passed again locally.
**Unverified in this follow-up:** the prepared browser-button test (Refresh,
selection, reload and Close) and repeated OpenXR transition could not run under
the shared lock. The device results elsewhere in this page are the earlier
2026-09-28 observations, not results from this blocked recheck. In particular,
HTTP focus is not evidence of worn-headset laser input. Follow the
[shared-device test procedure](testing.md#headset-smoke-test) for the next run.
## Saved spatial layouts: blocked on the current panel route
**Verified 2026-09-28**, same build, using a temporary xterm panel with
`STEAM_GAME=2000999031` and `FnTable:IVROverlay_028` from
`/opt/steamvr/bin/linuxarm64/libopenvr_api.so`:
| OpenVR call | Result |
|---|---|
| `FindOverlay("valve.steam.desktopgame.2000999031")` | Success |
| `GetOverlayWidthInMeters` | Success, 2.67 m |
| `SetOverlayWidthInMeters` (same width) | Success |
| `GetOverlayTransformType` | Success, type 5 (`VROverlayTransform_DashboardTab`) |
| `GetOverlayTransformAbsolute` | 18 (`WrongTransformType`) |
| `SetOverlayTransformAbsolute` (identity rotation, 1.2 m up, 1.5 m forward) | 12 (`PermissionDenied`); type remained 5 |
The public interface names type 5 **DashboardTab**; SteamVR's dashboard code
places these panels through its scene graph. It owns the frame/docking
transforms. A successful width setter does not grant permission to restore the
position. `vrcmd --dock-overlay world <key>` dispatched a docking request but
the dashboard logged `Failed to get SGTransform in setInitialTransformForLocation.
Invalid transform ID`. This does not establish working world placement.
**Inferred:** saving X11 pixel rectangles or Mac window IDs would not restore
this spatial arrangement. Mac window IDs also change when an application
reopens; viewer tickets and reconnect keys must not go into a layout file.
The base Mac stream reconnects after a network break, but that is different
from recreating windows and their room positions after a reboot.
There is consequently no Save/Restore control yet. A durable layout needs a
working transform restore path, stable source identity, and a fresh capture
permission/ticket flow. The tested gamescope-owned overlay route denies that
transform operation. A future Frame Control-owned overlay renderer, or a
supported platform API for dashboard frame transforms, needs its own device
proof before building layout UI. This is a blocker for the current approach,
not a claim that all possible implementations are impossible. Reboot recovery
was not tested: the shared headset was not rebooted.
## Panels during an immersive session
**Verified 2026-09-28**, same build: our Chromium switcher panel remained in
OpenVR's overlay list before, during and after the Frame's shipped `helloxr -g
Vulkan` sample. During the test `vrcmd --stats` identified
`system.generated.openxr.helloxr.helloxr`, with 242 frame submissions. The test
ended only its own sample process; no SteamVR, Steam, power or global settings
were changed. The switcher was still selectable afterwards.
This proves survival of that panel across an OpenXR scene session, **not** that
it stayed visibly composited over the scene: OpenVR reported it `not_visible`
before, during and after. **Verified:** calling `ShowOverlay` on our
*gamescope-owned* switcher overlay returns 12 (`PermissionDenied`). A helper
cannot force that panel visible using the public overlay call. Use the
switcher/dashboard to request access to it; we do not fight the runtime with a
repeated force-focus loop.
**Verified in a second controlled run:** a live H.264 test-pattern stream from
this checkout's Mac helper, through its own SSH tunnel and a temporary Chromium
profile, survived the same OpenXR sample (257 scene-frame submissions). Its
panel `2000999032` changed from `visible` before launch to `not_visible` during
and after the scene. The Mac helper still reported the same `test` stream;
captured frames increased from 35 to 232, with 29.5 decoded/drawn fps afterwards.
The test did not capture personal Mac windows or inject Mac input. The sample,
viewer, temporary profile, tunnel and Mac helper were cleaned up. This proves
stream survival, and also shows why it must not be advertised as always visible.
**Unverified:** persistent visible placement while playing a Steam-launched VR
game, Plasma desktop and real Mac-window behavior during that launch, and worn
headset input. Other threads were launching games and changing the runtime on
the shared device, so those transitions were not treated as controlled evidence.
A runtime/X-server restart can destroy the viewer windows; a network reconnect
cannot recreate them. No “always visible during games” guarantee is shipped.
## Keyboard passthrough feasibility
**Verified 2026-09-28**, same build, using `FnTable:IVRTrackedCamera_006`:
`HasCamera(0)` returned success and true. `GetCameraFrameSize` returned 100
(`OperationFailed`), with zero dimensions, for all three public frame types
(distorted, undistorted and maximum-undistorted), including after acquiring the
video service. Acquisition returned success and a handle; release returned 101
(`InvalidHandle`). The probe shut down its OpenVR client afterwards. No camera
frames were captured and no camera settings were changed.
**Documented:** the public OpenVR camera interface provides camera frame sizes,
intrinsics, projections and streaming handles; these are prerequisites for a
spatially aligned camera cutout. See Valve's
[OpenVR C API](https://github.com/ValveSoftware/openvr/blob/master/headers/openvr_capi.h).
**Inferred:** camera presence alone does not establish access to camera pixels.
The failed frame-size path blocks a keyboard cutout in our current panel
implementation. We have not established a keyboard detector or a calibrated
camera-to-panel mapping. Built-in full-room passthrough is not proof of a
public, selectively masked camera stream. No keyboard cutout is offered, and
no third-party camera/overlay app is substituted for it.
## Frame Control's media theatre
[The owned media player](vr-video.md) can show its video or stereo image on a
larger, head-relative screen with its own dark surround. **Verified remotely
2026-09-28**, SteamOS 0.4.1 / BUILD_ID 20260925.6191901: screen, eye isolation,
surround and cleanup. It does not alter panel docking or global settings.
Its `Overlay` RGBA rendering hook is available to stream producers; applying
SteamVR theatre docking to existing Mac/PC panels is still unverified here.
+1 -1
View File
@@ -99,7 +99,7 @@ controls to place each panel. See [docs/panels.md](panels.md).
| `scripts/frame-ui.sh` | Mac | Start the Frame Control web UI (`ui/server.py`) and open it (**verified**) | | `scripts/frame-ui.sh` | Mac | Start the Frame Control web UI (`ui/server.py`) and open it (**verified**) |
| `scripts/apk-catalog.sh` | Mac | Refresh the rated F-Droid catalogue that Frame Control's Android section shows (**verified**) | | `scripts/apk-catalog.sh` | Mac | Refresh the rated F-Droid catalogue that Frame Control's Android section shows (**verified**) |
| `scripts/compat-db-backup.sh` | Mac | Maintainer-only: back up the shared compatibility database locally and to Google Drive (**verified**) | | `scripts/compat-db-backup.sh` | Mac | Maintainer-only: back up the shared compatibility database locally and to Google Drive (**verified**) |
| `scripts/push-vr-video.sh` | Mac → Frame | Upload VR180/360 videos to `~/Videos/VR`, linked into DeoVR's Proton prefix; `--launch` starts DeoVR (**verified**: upload and link; in-headset playback of local files not yet checked). See [docs/vr-video.md](vr-video.md) | | `scripts/push-vr-video.sh` | Computer → Frame | Upload movies, stereo PNG/JPEG or small `.splat` files to `~/Videos/FrameControl`; `--launch` starts our OpenVR player, `--theatre` adds a larger screen and dark surround. No separate viewer. **Verified remotely**: decode, stereo output and cleanup; worn-headset checks remain. See [docs/vr-video.md](vr-video.md) |
| `scripts/push.sh` | Mac → Frame | `rsync` files to `~/Downloads` (or a given path) on the Frame (**verified**) | | `scripts/push.sh` | Mac → Frame | `rsync` files to `~/Downloads` (or a given path) on the Frame (**verified**) |
| `scripts/serve-bootstrap.sh` | Mac | Fallback: serve `bootstrap-on-frame.sh` with your public key embedded | | `scripts/serve-bootstrap.sh` | Mac | Fallback: serve `bootstrap-on-frame.sh` with your public key embedded |
| `scripts/bootstrap-on-frame.sh` | Frame | Fallback: install the key and enable `sshd` | | `scripts/bootstrap-on-frame.sh` | Frame | Fallback: install the key and enable `sshd` |
+8
View File
@@ -110,3 +110,11 @@ returns nothing without `cc`, so Frame Control takes the country from
- Installing when there's more than one library folder, such as a microSD card. - Installing when there's more than one library folder, such as a microSD card.
- Uninstalling. `steam://uninstall/<appid>` should open a confirmation in the - Uninstalling. `steam://uninstall/<appid>` should open a confirmation in the
headset. headset.
## Optional VR software
The [VR utilities list](vr-utilities.md#optional-software) is separate from our
controls and HUD. It checks software ownership as well as games; paid utilities
are installable only when already in the loaded Frame account library. No
purchase flow is added. Public reports, local results and ownership are shown
separately, and untested tools remain untested.
+66 -4
View File
@@ -8,6 +8,7 @@ This covers three directions, plus input:
- **PC VR from Linux**: [feasibility and options](linux-vr-streaming.md), - **PC VR from Linux**: [feasibility and options](linux-vr-streaming.md),
including Valve's streaming and USB support. No Linux host tested yet. including Valve's streaming and USB support. No Linux host tested yet.
- **Input**: type and point in the Frame from the Mac or iPhone. - **Input**: type and point in the Frame from the Mac or iPhone.
- **Live view and Control**: watch a panel flat and tap on it to use it.
The confidence labels are the same as in [ssh.md](ssh.md). The confidence labels are the same as in [ssh.md](ssh.md).
@@ -45,8 +46,10 @@ flat 2D desktop streaming into a window on the Frame's Linux desktop.
| Immersed / Virtual Desktop | Vendor apps | Immersed has a Mac agent but no known Frame client. Virtual Desktop's developer said he'd "try" to port it ([NewsBreak](https://www.newsbreak.com/news/4892834783961-virtual-desktop-dev-says-he-ll-try-to-bring-the-app-to-steam-frame)). | Not available as of 2026-09-25. Check again later. | | Immersed / Virtual Desktop | Vendor apps | Immersed has a Mac agent but no known Frame client. Virtual Desktop's developer said he'd "try" to port it ([NewsBreak](https://www.newsbreak.com/news/4892834783961-virtual-desktop-dev-says-he-ll-try-to-bring-the-app-to-steam-frame)). | Not available as of 2026-09-25. Check again later. |
| WiVRn / ALVR | VR streaming from a Linux or Windows PC | Irrelevant for a Mac host (no SteamVR/OpenXR runtime on macOS) | N/A | | WiVRn / ALVR | VR streaming from a Linux or Windows PC | Irrelevant for a Mac host (no SteamVR/OpenXR runtime on macOS) | N/A |
For **VR video files** (180°/360° stereo), don't stream the Mac's screen. Play For **local movies and stereo photos**, Frame Control's own OpenVR player
them on the Frame in DeoVR instead: see [vr-video.md](vr-video.md). runs on the Frame; see [vr-video.md](vr-video.md). It currently renders a flat
stereo screen. VR180/360 projection is not implemented; the same page records
DeoVR only as an optional, independently installed alternative.
### Pre-seeding the Remmina profile (no typing in the headset) ### Pre-seeding the Remmina profile (no typing in the headset)
@@ -182,13 +185,72 @@ on the Frame, which talks KDE Connect's own LAN protocol to the Frame's
and ignores injected pointer motion; `:1` holds apps such as Chromium and and ignores injected pointer motion; `:1` holds apps such as Chromium and
takes it. KDE Connect runs on `:1`, so it reaches apps, not Steam's own menus. takes it. KDE Connect runs on `:1`, so it reaches apps, not Steam's own menus.
There's also a `gamescope-0-ei` (libei) socket. There's also a `gamescope-0-ei` (libei) socket.
- **Not yet tested:** typing and clicking as seen in the headset, and whether - Typing through KDE Connect lands in a Chromium panel on `:1` (seen in the
it reaches the KDE desktop panel (Plasma is its own session). panel's own capture, 2026-09-29). It **can't reach panels on `:0`** (Frame
Control's own panels, Steam's UI) and, since XTest positions are clamped to
`:1`'s 1280×720 root, can't reach beyond that in a bigger window. Control on
the live view (below) has neither limit.
- **Not yet tested:** whether it reaches the KDE desktop panel (Plasma is its
own session).
- **Known limit:** keys and clicks typed while the link is reconnecting wait - **Known limit:** keys and clicks typed while the link is reconnecting wait
and are sent once it's back, but anything sent in the moment the Wi-Fi and are sent once it's back, but anything sent in the moment the Wi-Fi
drops, before SSH notices, can be lost. Confirming every event would add a drops, before SSH notices, can be lost. Confirming every event would add a
round trip to each pointer move. round trip to each pointer move.
## Live view and Control: watch a panel and tap on it
**Built: Home → Desktop / Headset view → Control.** The live view has two
sources:
- **Headset view**: what the lenses show (SteamVR's mirror, `/dev/video99`). It
moves with the wearer's head, so Control makes the view a trackpad: drag to
move the pointer, tap to click, press and hold to right-click, two fingers to
scroll. With a mouse, moving over the view moves the pointer.
- **Desktop**: the app panel in use in the headset, from its own window, so it
stays still however the wearer looks around. Control makes taps and clicks
land exactly where you put them. Dragging is a mouse drag, press and hold is a
right-click, two fingers scroll, and on a computer the mouse, wheel and
keyboard work directly on it (⌘ is sent as Ctrl on a Mac). A picker shows any
other panel, view only.
Below the view, a text field and key buttons type on the Frame from a phone.
How (**verified 2026-09-29**, SteamOS 0.4.1, build 20260925.6191901):
- **Input goes through gamescope's own injection.** gamescope serves an EIS
socket (`/run/user/1000/gamescope-0-ei`; Steam feeds Remote Play input through
it), and `libei` 1.4.1 is on the image. [`ui/frame_touch.py`](../ui/frame_touch.py)
talks to it with `ctypes`: nothing to install. gamescope offers one device,
"Gamescope Virtual Input", with relative and absolute pointer, buttons,
scroll and keyboard (Linux key codes; no text capability, so the text field
types printable ASCII on a US layout). Its absolute region is unbounded; the
pointer uses the focused panel's display coordinates, and gamescope fits each
window to its display, so a 1920×1080 window on the 1280×720 `:1` takes
positions at two thirds scale. Taps on a 1280×720 page landed on the exact
pixel.
- **It reaches the panel that has focus** (`GAMESCOPE_FOCUSED_WINDOW` on `:0`'s
root), on either X display. In the OpenVR backend focus moves only on SteamVR
overlay events (the controller's laser entering or clicking a panel), or to a
new panel when none holds it (read from gamescope's `OpenVRBackend.cpp`, seen
with `gamescopectl focus_info`, which writes to the journal). Neither
`GAMESCOPECTRL_BASELAYER_WINDOW`/`_APPID` nor X focus moves it, and no
gamescope command does. So Control follows the wearer: whatever they last
used is what your taps reach. A window without a Steam app id (`STEAM_GAME`)
gets a connector of its own and doesn't hold focus.
- **Keys in a burst can arrive out of order**, so the helper paces them (8 ms
apart).
- **Known limit:** if focus moves to another panel in the middle of a drag, the
release goes to the panel that has focus then. Whether gamescope hands it to
the window that got the press isn't known yet. When the session ends, the
helper lets go of every button and key it still holds.
- **The Desktop picture is the window's own pixels**: `ffmpeg -f x11grab
-window_id <window> -i :<display>` works on gamescope's redirected windows,
while grabbing the root gives black. It streams as H.264 like the headset view
(about 30 fps at 720p).
- Tested from the iPhone app (Simulator): a tap on the Desktop view focused a
text box in the panel and the text field typed into it; a trackpad move went
exactly (+40, +25).
Our own `uinput` keyboard and mouse would also work (`steamos` is in the Our own `uinput` keyboard and mouse would also work (`steamos` is in the
`input` group and `/dev/uinput` is group-writable, verified 2026-09-27), and `input` group and `/dev/uinput` is group-writable, verified 2026-09-27), and
remains the fallback if the bundled KDE Connect ever stops working on a new SteamOS. remains the fallback if the bundled KDE Connect ever stops working on a new SteamOS.
+54
View File
@@ -104,6 +104,20 @@ switch while SSH is down:
## Headset smoke test ## Headset smoke test
**Documented shared-device procedure:** before a test installs, launches or
stops an application, acquire `ssh frame 'mkdir /tmp/frame-test.lock'`. If it
fails, leave that lock alone and continue offline work. Only the thread that
acquired it releases it with `ssh frame 'rmdir /tmp/frame-test.lock'`, after
cleanup. Keep each device session to a few minutes.
Check battery capacity and charging state under `/sys/class/power_supply`
before and after; keep capacity above 20%. Stop only processes started by the
test, remove temporary installs and profiles, and restore the prior dashboard
state. Leave Steam and SteamVR running. Do not reboot or change global settings.
Record the build, actual interaction results, cleanup and any unworn-headset
limits alongside screenshots or logs. These are caller responsibilities; the
smoke script below does not acquire this shared lock itself.
```sh ```sh
scripts/frame-smoke.sh # needs `ssh frame` to work without a password scripts/frame-smoke.sh # needs `ssh frame` to work without a password
scripts/frame-smoke.sh --pair # also pairs a throwaway key: approve it in the headset scripts/frame-smoke.sh --pair # also pairs a throwaway key: approve it in the headset
@@ -149,6 +163,20 @@ refuses ids with a hyphen (`missing/invalid arguments`), which the fake had
accepted. The fake now refuses them the same way, and Frame Control makes ids accepted. The fake now refuses them the same way, and Frame Control makes ids
Steam accepts. Steam accepts.
## Owned media player
`tests/test_media.py` covers layout evidence and overrides, OU eye ordering,
hardware-decoder command construction, malformed splats, stereo parallax and
fake-Frame library/process ownership. `tests/e2e/test_media_transfer.py` checks the real
HTTP/SSH upload and library listing without pretending the fake renders VR.
Real decode timings and captured stereo output are recorded in
[vr-video.md](vr-video.md). Generated media only; no external player required.
**Verified 2026-09-28**, real Frame BUILD_ID 20260925.6191901: `~/.local`
and `~/.local/share` are `steamos:steamos`, mode 0755. The fake supervisor
sets those parent owners too; previously its root-created Steam manifests
left the parents root-owned and incorrectly prevented user runtime installs.
## Agent interfaces ## Agent interfaces
`tests/test_agent.py` exercises MCP stdio, exact-action human approvals and the `tests/test_agent.py` exercises MCP stdio, exact-action human approvals and the
@@ -156,3 +184,29 @@ assistant against an in-process HTTP endpoint with canned responses (no keys or
external calls). `tests/e2e/test_agents.py` runs the MCP/HTTP/SSH path against the external calls). `tests/e2e/test_agents.py` runs the MCP/HTTP/SSH path against the
fake Frame for approved installs, clipboard and file transfer. Headset Chromium fake Frame for approved installs, clipboard and file transfer. Headset Chromium
rendering and real screenshots still need a device; see [agent evidence](agents.md#evidence-and-limits). rendering and real screenshots still need a device; see [agent evidence](agents.md#evidence-and-limits).
## Family and comfort
`tests/test_comfort.py` uses an injected clock, fake headset sensor readings and
actions, plus a Node fake of Steam's Home API. It covers warnings before Home,
late/suspended sessions, cancellation, failed actions, duplicate alerts, reboot
invalidation, per-zone thermal trips and shared on-headset state. The server
guards reject invalid session settings before SSH. See
[real-device evidence and limits](family-comfort.md#verification).
## Panel switcher
`tests/test_panels.py` supplies fake-Frame `vrcmd --overlays` output, checks
main-panel filtering (including hidden panels), revalidates closed panels before
focus, and drives the headset helper's real loopback HTTP server to test access
keys, Host/Origin guards, malformed requests, offline errors and Close. It runs
in the normal unit suite without OpenVR or a headset. The fixture format comes
from SteamVR 2.18.1, BUILD_ID `20260925.6191901`; it does not simulate rendering.
On the Frame, run `python3 -` over SSH with `ui/frame_panels.py` on stdin to
list panels. `--focus <key>` rechecks the list and requests focus. In Frame
Control, **Tools → Panel switcher → Open in headset** exercises installation,
Chromium rendering and the same helper through HTTP. Close the switcher after
testing. [The recorded device checks](panels.md#frame-controls-panel-switcher)
cover actual focus, HTTP guards and an OpenXR sample transition, and separately
identify the unverified Steam-game, spatial layout, reboot and laser behaviors.
+139
View File
@@ -0,0 +1,139 @@
# VR comfort and performance
Frame Control owns its HUD and telemetry. They use SteamVR/OpenVR, gamescope,
Python and xterm already on the Frame, plus the Frame's sensors. No feature
requires fpsVR, OVR Advanced Settings, XSOverlay or another third-party app.
The software list is a separate, optional convenience.
This is **part of [#25](https://github.com/saphid/frame-control/issues/25)**,
not completion of the issue. Playspace controls remain blocked on the device
checks below. The PR stays draft.
## Our performance HUD
On **Home → VR comfort and performance**, the app shows a timestamped sample
with each status refresh (30 seconds, or Refresh). **Open HUD in headset**
starts our text HUD as a gamescope panel, refreshed every two seconds. In the
SteamVR dashboard, select **Frame Control HUD**, then Float in World or dock
it to a controller. **Close HUD**, or closing its terminal, ends it. Opening
it twice reuses the existing process.
| Value | Meaning and source |
|---|---|
| Compositor FPS / period | Differences between two `IVRCompositor_029::GetFrameTiming` frame indices and monotonic compositor timestamps, sampled 200 ms apart. Output cadence, not game FPS or a long-term average. |
| Application FPS | Reciprocal of OpenVR's client frame interval. Unavailable if there is no positive interval; not inferred from refresh rate. |
| Render GPU time | OpenVR total render GPU milliseconds, not GPU utilisation. |
| Compositor CPU | OpenVR compositor render CPU milliseconds, not game CPU time. |
| System CPU | `/proc/stat` busy-time delta across the sample, with guest time counted once and iowait treated as idle. |
| GPU clock | `3d00000.gpu/cur_freq`, converted from Hz to MHz; frequency is not load. |
| Hottest sensor / battery | Existing thermal-zone and battery sysfs reads from `frame_status.py`. |
OpenVR uses background application mode, which does not start SteamVR or keep
it running. This mode also returned live timing in a read-only device probe.
Missing sensors, a stopped or incompatible SteamVR runtime, and non-advancing
frame indices display **Unavailable**, never invented zero FPS. Failed status
refreshes clear the HUD card rather than keeping a stale live-looking sample.
The HUD itself adds CPU/GPU work; it is a diagnostic, not a zero-overhead benchmark.
**Verified 2026-09-28**, SteamOS 0.4.1, build `20260925.6191901`, SteamVR
2.18.1: the exact OpenVR interface and 192-byte timing layout returned advancing
frame indices and live GPU/CPU timing. Sensor reads, creation of overlay
`valve.steam.desktopgame.2000250025`, duplicate-open handling, and closing the
HUD passed. The temporary probe overlay disappeared after its process closed.
[Sanitized device sample](evidence/vr-utilities/device.json).
**Unverified:** visual placement while wearing the headset, controller docking,
and overhead during gameplay. The companion card was checked in the attached
preview using live Frame data. Creating an overlay does not establish that it
was visible to the wearer.
The optional HUD copies only `frame_status.py` and `frame_vr.py` into
`~/.local/share/frame-control/vr/`. It tags only the window whose X11 PID matches
its own xterm, avoiding other threads' windows. Stop checks both PID and Linux
process start time before sending SIGTERM. It does not stop Steam or SteamVR,
edit their settings, install a service, or need sudo.
## Playspace, seated height and recenter: paused
**Verified 2026-09-28:** SteamVR exposes `IVRChaperoneSetup_006` and
`IVRChaperone_004`. An initial 1 cm seated zero-pose translation committed,
read back and restored numerically. A later trial, while the shared Frame was
in use, changed universe IDs after commits and returned transforms that did
not match the requested write or restore. Journal entries reported
`CommitWorkingCopy`, `VREvent_ChaperoneUniverseHasChanged` and
`VREvent_ChaperoneRoomSetupCommitted`. The collision-bound arrays and play-area
size matched in the saved before/after records, but the origin matrices did not.
Alex confirmed the headset was in use and asked to pause control tests. No
further playspace writes were made. The exploratory control implementation was
removed from the shipping API; `recenter`, `adjust` and `restore` are rejected.
This is an **unresolved feasibility check**, not evidence that OpenVR controls
cannot work. Concurrent use and the Frame driver's coordinate-system handling
still need to be separated.
The probe's original and last-read poses remain in the Frame's
`~/.local/share/frame-control/vr/comfort.json` for investigation. This draft
neither reads nor applies that baseline. Do not blindly replay it into a room
that may have changed. No recenter test was reached in the later trial.
Before adding controls, on an idle Frame:
1. Establish current room and tracking state, and inspect the retained probe
evidence before considering any restoration.
2. Prove seated and standing height/move operations in the Frame driver's
current coordinates, including delayed readback, coordinate rebasing and
recovery. Show that the physical safety boundary stays correct.
3. Verify recenter independently, and test a full apply/restore cycle plus a
concurrent room-change refusal. Add a fake OpenVR test for those contracts.
4. Check the apparent result in a seated and a standing app before exposing UI.
**Documented:** seated and standing are tracking origins selected by an app;
changing a seated origin cannot force every game to support seated play.
**Inferred from the installed SteamVR defaults:** there is no generic snap-turn
or locomotion-vignette setting. `dashboard.verticalOffsetCm_2` and
`steamvr.panelMaskVignette` affect panels, not the player's height or game
locomotion. The app gives game-setting hints for snap-turn, teleport movement
and movement vignette instead of writing these unrelated settings.
## Optional software
**Verified 2026-09-28 (same build):** a read-only query of Steam's loaded
`appStore.allApps` found none of these five apps on the Frame account. The query
includes software, which the existing games-only library filter excludes.
Steam's public app-details API listed only Desktop+ as free. No software was
purchased, installed or launched during these ownership checks.
| Utility | Local Frame status | Public evidence / optional source |
|---|---|---|
| OVR Advanced Settings (1009850) | **Untested**, not owned | Steam edition is paid. Developer's [free source and releases](https://github.com/OpenVR-Advanced-Settings/OpenVR-AdvancedSettings). No verified Frame result in this work. |
| XSOverlay (1173510) | **Untested**, not owned | Supplied research attributes Proton support with tweaks to [Road to VR](https://www.roadtovr.com/valve-steam-frame-review/). This is a public report, not our verification. |
| OVR Toolkit (1068820) | **Untested**, not owned | Same [public Frame report](https://www.roadtovr.com/valve-steam-frame-review/); no local verification. |
| fpsVR (908520) | **Untested**, not owned | [Steam listing](https://store.steampowered.com/app/908520/). No Frame-specific result established in the supplied research. General PC VR reviews do not verify Frame support. |
| Desktop+ (1494460) | **Untested**, free | [Developer source](https://github.com/elvissteinjr/DesktopPlus). No verified Frame result in this work. |
**Documented (supplied research):** the XSOverlay/OVR Toolkit claims are kept as
attributed leads. **Verified source check 2026-09-28:** fetching the linked
review returned HTTP 200 and the expected review title, but neither utility name
appeared in the fetched HTML or extracted text. The claims could not be
corroborated from that page; this is not proof of incompatibility. They do not make those apps dependencies or mark them
locally verified. No paid utility is auto-acquired. A server-side check blocks
installation through the Steam endpoint if the paid utility is absent from the
loaded Frame library; an empty/unavailable library fails closed. Desktop+ uses
the existing free Steam install flow, which may need a license confirmation in
the headset. None is advertised as known-good without local evidence.
Compatibility reports reuse the existing `compat-db` storage and validation
with `package: "steam:<appid>"`, rather than colliding with Android package IDs.
The optional list shows the latest `works`, `issues` or `broken` report with its
date, build and notes, separately from public sources and ownership. With no
report the status stays **untested**. No shared database schema change or
production deployment is needed; no reports were published by this work.
## Validation and review
166 unit tests and 8 website tests passed locally. The new fake-Frame cases
passed in GitHub CI (Docker was unavailable locally). Desktop and phone-width
preview checks passed. The two independent review attempts did not produce a
verdict; [commands, real exit statuses and limitations](evidence/vr-utilities/review.md).
+139 -72
View File
@@ -1,81 +1,148 @@
# Watching VR video (180°/360°) on the Frame # Movies, stereo photos and splats
The confidence labels are the same as in [ssh.md](ssh.md). Everything here Frame Control has its own Frame-side OpenVR player. It uses SteamOS's ffmpeg
was checked on SteamOS 0.3.0, build 20260922.6101926. and V4L2 hardware decoder, Python and SteamVR. **No separate player or viewer
is required.** Chromium and immersive WebXR are not in this playback path.
## The short version ## Use it
1. Install **DeoVR Video Player** from Steam (free, app 837380) and start it once In **Tools → Media in the headset**, send a file, choose its layout and press
in the headset. That creates its Proton prefix. **Play**. **Theatre** gives it a larger screen and an 85% black surround.
2. On the Mac: `scripts/push-vr-video.sh --launch ~/Movies/beach_180_LR.mp4` **Stop** removes both. Refresh reads the library and the player's state.
3. In the headset, open DeoVR's local file browser → **Videos → VR** and The screen follows your head; it isn't a saved world-space panel.
pick the file.
## Why DeoVR The command-line route uses the same player:
The Frame's Chromium can't play VR video in 3D. It has no immersive WebXR
([how-the-frame-works.md](how-the-frame-works.md)). DeoVR's Windows build
runs under Proton ARM64 + FEX as a real SteamVR app. **Verified 2026-09-25:**
it found the Steam Frame headset and controller over OpenVR, and decoded
7680×3840 and 8192×4096 H.265 VR180 side-by-side streams through AVPro's
hardware Media Foundation path, mapped onto a 180° dome or fisheye mesh.
Known quirks (verified):
- The first launch takes about 45 s while it compiles shaders.
- Grid thumbnails stay blank. Unity's own video player, which DeoVR uses for
previews, fails with `0xc00d36bb` under Proton. Full playback uses AVPro
and isn't affected.
- The in-app store and web content are separate from local files. You don't
need an account to play your own files.
## Getting files onto the headset
`scripts/push-vr-video.sh` copies files with `rsync --partial`, so an
interrupted upload resumes. They go to `~/Videos/VR` on the Frame (`/home`
has about 860 GB free). The script also links that folder into DeoVR's prefix
as `C:\users\steamuser\Videos\VR`. It's reachable at
`Z:\home\steamos\Videos\VR` as well. **Verified** that the upload and link
work. **Not yet checked** whether DeoVR's file browser lands there
(open question 21).
Speed: a test upload over Wi-Fi ran at about 3–5 MB/s (verified 2026-09-25,
one sample). At that rate an 8K file of several GB takes tens of minutes, so
start big uploads before you put the headset on.
## Naming files so they play correctly
DeoVR guesses the projection from the file name. Its binary contains the tags
`_180`, `_360`, `_fisheye`, `_fisheye190`, `_mkx200`, `_vrca220` and `_rf52`
(verified). For stereo layout, the common DeoVR convention is `_LR`/`_SBS`
(side by side) and `_TB` (top/bottom) (inferred). If a video looks wrong
(doubled, warped, or flat), change the projection and stereo mode in DeoVR's
player menu.
Examples: `trip_180_LR.mp4`, `concert_360_TB.mp4`, `hike_fisheye190_LR.mp4`.
Codecs: H.265 at 8K played (verified, streamed). Local H.264 and H.265
files haven't been played yet. The test clips below cover that.
## Test clips
The script doesn't include these. To check a setup, make two 20 s clips:
3840×1920, 180° side by side, with the left eye tinted red and the right eye
cyan. In the headset each eye should see only its own colour. A single mixed
colour means the stereo split is wrong.
```sh ```sh
ffmpeg -f lavfi -i testsrc2=size=1920x1920:rate=30:duration=20 \ scripts/push-vr-video.sh --launch --theatre ~/Movies/film_SBS.mp4
-filter_complex "[0:v]split[a][b];[a]colorchannelmixer=rr=1:gg=0.3:bb=0.3[l];[b]colorchannelmixer=rr=0.3:gg=1:bb=1[r];[l][r]hstack" \ scripts/push-vr-video.sh --launch --layout ou ~/Pictures/stereo.png
-c:v libx264 -pix_fmt yuv420p -b:v 20M frame-test_180_LR_h264.mp4 scripts/push-vr-video.sh --launch capture.splat
scripts/push-vr-video.sh frame-test_180_LR_h264.mp4 scripts/push-vr-video.sh --list
scripts/push-vr-video.sh --stop
``` ```
## Streaming from the Mac instead of copying (untested) Uploads live in `~/Videos/FrameControl/<id>/` on the Frame. Each upload gets
its own directory, so sending another file with the same name doesn't replace
it. The UI's existing upload limit is 8 GiB. Transfers use the app's rsync/scp
path; resumable large uploads are not implemented here yet. Existing
`~/Videos/VR` files and Proton prefixes are left alone. The script no longer
launches DeoVR, links a Proton prefix, or accepts directories.
DeoVR has a DLNA browser (the binary contains `Searching for DLNA | Media | Supported preview |
devices...` and a UPnP ContentDirectory client). A DLNA server on the Mac |---|---|
should therefore appear in DeoVR without copying anything, for example | Movies | H.264 / H.265, mono or left-first SBS / top-first OU; audio through the Frame's PulseAudio-compatible server |
`brew install rclone` then `rclone serve dlna ~/Movies/VR`. This is **inferred**, not | Stereo photos | PNG / JPEG containing both eyes, SBS or OU |
tried. 8K VR video needs roughly 50–100 Mbit/s sustained, and the Wi-Fi | Gaussian splats | Common 32-byte `.splat` records, 1–20,000 Gaussians; a stationary stereo preview |
sample above (about 30–40 Mbit/s) suggests copying first is the safer default.
Auto layout reads delimited filename tags: `_SBS`, `_HSBS`, `_LR`, `_OU`,
`_TB`, `_HOU`, `_HTB`, `_FSBS`, `_FOU`, `_FTB`. SBS/OU without `F` means
half-resolution packing. Full packing preserves each eye's original aspect.
It also accepts FFmpeg's `stereo_mode` metadata (`left_right`, `top_bottom`,
`mono`); left/right and top/bottom metadata are treated as full packing.
Choose an explicit layout if that assumption doesn't match the file.
Unknown or conflicting tags ask for a choice, rather than silently flattening
stereo. The explicit selector always wins. Layout is not guessed from resolution.
## What was verified
**Verified remotely, 2026-09-28:** SteamOS 0.4.1, BUILD_ID
`20260925.6191901`, SteamVR 2.18.1. Nobody wore the headset for these checks.
All test media were generated by us. No paid content or DRM was involved.
- Stock `ffmpeg` `h264_v4l2m2m` decoded 150 frames of our 1920×1080 H.264
test in 0.128 s (decode only); converting all frames to RGBA took 0.242 s.
- Our ffmpeg → Python → OpenVR path submitted all 150 frames and exited 0.
The 1280×720 prototype took 4.775 s; the full 1920×1080 player took
4.618 s. These are wall times, not in-headset frame-rate measurements.
Finishing a 5 s clip early showed those probes weren't paced, so the final
player paces output at 30 fps: all 150 frames then completed in **5.025 s**.
- The H.265 hardware decoder also completed the generated 1080p clip
(60 frames, exit 0); this is a short compatibility check, not a 4K/8K benchmark.
- Our actual player, launched through Frame Control's media API, displayed
SBS and OU photos with red only in the left-eye capture and cyan only in
the right. OpenVR's `SideBySide_Parallel` flag separates the eyes; we
rearrange OU rows ourselves.
- A generated 48-Gaussian `.splat` rendered in our own CPU renderer and appeared
in both eyes with separate perspective projections.
- Theatre's owned dark surround and screen appeared in captures. Steam's
dashboard remained available over them. Stop removed the owned overlays
and ended the dedicated user service. No global SteamVR setting was changed.
- A generated H.264/AAC clip reached the Pulse audio output and completed
all 100 video frames with exit 0. This proves the output path, not audible
quality or lip sync.
![Frame Control SBS eye-isolation proof: red left, cyan right](img/media-sbs-proof.png)
![Our Gaussian-splat stereo preview on the Frame](img/media-splat-proof.png)
**Verified failed route:** GStreamer 1.24.2's `playbin` selected
`v4l2h264dec`, delivered the first RGBA sample and then segfaulted (exit 139)
in the basic appsink probe and the OpenVR probe. We do not ship that route.
The ffmpeg path above completed instead.
## Limits and blockers
- **Not verified:** worn-headset comfort, sound quality/lip sync, long movies,
4K/8K decode, HDR, controller interaction or behaviour on other OS builds.
Output is bounded to 1920×1080 packed pixels before OU rearrangement.
- **Not implemented:** pause, seeking, subtitles, playlists, right-eye-first
layouts, non-square pixel correction, VR180/360 projection and fisheye.
This preview is a flat stereo screen, not a dome player.
- **Native spatial-photo blocker:** HEIC/HEIF/AVIF/MPO stereo-container
extraction isn't implemented in our viewer. We reject these rather than
displaying one image and calling it spatial. Export both eyes to PNG/JPEG
first. This is a current implementation gap, not a claim that the Frame
cannot support these containers.
- **Large/immersive splat blocker:** the owned CPU rasterizer projects 3D
covariance into each eye and alpha-composites Gaussians, but renders a
fixed view at 320×240 per eye. It caps each Gaussian footprint at 32 pixels
and normalizes the scene to a two-metre box. Large scenes, PLY/SPZ, live
head-position parallax and navigation need a GPU scene renderer; this
version rejects files above 20,000 records. It is a stereo preview, not
an immersive walk-through.
- A single player can run at a time. It stops at EOF, on **Stop**, or after
four hours in any case, so longer movies are cut off. Still images remain
until Stop or that timeout. There is no delete action yet: remove old
uploads from `~/Videos/FrameControl/` over SSH. The log is
`~/.local/share/frame-control/media/player.log` on the Frame.
- **Panel/stream theatre remains outside this media slice:** SteamVR's
`vrcmd --dock-overlay` accepts `theater`, but docking an existing panel
and its dimming behaviour were not verified here. The shared headset had
workspace and stream tests active. We did not reposition their panels.
Integration with [#22](https://github.com/saphid/frame-control/issues/22)
and [#31](https://github.com/saphid/frame-control/issues/31) can use the
owned rendering hook below; no second stream/window manager is introduced.
## Integration hook
`POST /api/upload` with `X-Mode: media` and `X-Filename` sends a file and
returns its `id`. `POST /api/media` accepts:
```json
{"action":"play","id":"<32 hex characters>/film_SBS.mp4","layout":"auto","theatre":true}
```
Other actions are `list`, `status`, `stop`. These use the normal `X-Frame-UI`
guard. Play reports **starting**, not a claim that frames reached the headset;
read status for `playing`, `ended`, `stopped` or `error`.
The own-code files are copied to `~/.local/share/frame-control/media/` and
run in the `frame-control-media.service` systemd user unit. Stop affects only
that unit, including its decoder child.
For a Frame-side stream producer, `frame_media_player.Overlay` exposes
`create(key, width, distance, stereo=False, aspect=1, order=1)`,
`pixels(handle, rgba, width, height)` and `close()`. Use an owned unique key,
pass interleaved RGBA bytes (left/right halves for stereo) and always close in
`finally`. `create` uses a head-relative transform; it does not move another
app's panel. The player demonstrates a separate black surround overlay.
`frame_media.stereo_pixels` converts OU to SBS. This is the narrow rendering
hook for stream/workspace work; it doesn't capture or manage a Mac/PC stream.
## Optional separate app
You can independently install **DeoVR Video Player** (free Steam app 837380)
if you prefer its VR180/360 features. Earlier tests on SteamOS 0.3.0,
build `20260922.6101926` (2026-09-25), verified its OpenVR initialization and
8K H.265 streamed VR180 decoding under Proton. Frame Control's media features
neither install nor launch it, and do not depend on it. Its behaviour and
file-naming conventions are not evidence about our player.
@@ -17,6 +17,7 @@
78427FC66780623F31E7501E /* FrameControlApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */; }; 78427FC66780623F31E7501E /* FrameControlApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */; };
84423CB45629465420180A64 /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 8F2CB550FC81C01E6BDD5A71 /* Assets.xcassets */; }; 84423CB45629465420180A64 /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 8F2CB550FC81C01E6BDD5A71 /* Assets.xcassets */; };
9657F7BC23E3352E5AB30777 /* SetupView.swift in Sources */ = {isa = PBXBuildFile; fileRef = DB544223FC60A59CC3E8EF5F /* SetupView.swift */; }; 9657F7BC23E3352E5AB30777 /* SetupView.swift in Sources */ = {isa = PBXBuildFile; fileRef = DB544223FC60A59CC3E8EF5F /* SetupView.swift */; };
A0C5B00E257230A38DBD9E54 /* ComfortNotificationTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */; };
A8C7AED25A6280682FCE45DC /* Citadel in Frameworks */ = {isa = PBXBuildFile; productRef = 6BA549B6CC0A0CB847126456 /* Citadel */; }; A8C7AED25A6280682FCE45DC /* Citadel in Frameworks */ = {isa = PBXBuildFile; productRef = 6BA549B6CC0A0CB847126456 /* Citadel */; };
DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */; }; DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */; };
E6898C714A92D3979F73B6E1 /* FrameFinder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2F288DF6636A417F0CA3A6CD /* FrameFinder.swift */; }; E6898C714A92D3979F73B6E1 /* FrameFinder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2F288DF6636A417F0CA3A6CD /* FrameFinder.swift */; };
@@ -48,6 +49,7 @@
BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InstallLink.swift; sourceTree = "<group>"; }; BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InstallLink.swift; sourceTree = "<group>"; };
D6C4E6C28315CA8729FCAAEA /* WebShell.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WebShell.swift; sourceTree = "<group>"; }; D6C4E6C28315CA8729FCAAEA /* WebShell.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WebShell.swift; sourceTree = "<group>"; };
DB544223FC60A59CC3E8EF5F /* SetupView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SetupView.swift; sourceTree = "<group>"; }; DB544223FC60A59CC3E8EF5F /* SetupView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SetupView.swift; sourceTree = "<group>"; };
E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ComfortNotificationTests.swift; sourceTree = "<group>"; };
EDC7BA8014DBC302D08FD397 /* HeadsetServer.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = HeadsetServer.swift; sourceTree = "<group>"; }; EDC7BA8014DBC302D08FD397 /* HeadsetServer.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = HeadsetServer.swift; sourceTree = "<group>"; };
F3E2F5607DD877272483D64E /* FrameControl.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = FrameControl.app; sourceTree = BUILT_PRODUCTS_DIR; }; F3E2F5607DD877272483D64E /* FrameControl.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = FrameControl.app; sourceTree = BUILT_PRODUCTS_DIR; };
/* End PBXFileReference section */ /* End PBXFileReference section */
@@ -107,6 +109,7 @@
75A17B1C79C8C3C60FEABBA6 /* FrameControlTests */ = { 75A17B1C79C8C3C60FEABBA6 /* FrameControlTests */ = {
isa = PBXGroup; isa = PBXGroup;
children = ( children = (
E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */,
1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */, 1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */,
); );
path = FrameControlTests; path = FrameControlTests;
@@ -274,6 +277,7 @@
isa = PBXSourcesBuildPhase; isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647; buildActionMask = 2147483647;
files = ( files = (
A0C5B00E257230A38DBD9E54 /* ComfortNotificationTests.swift in Sources */,
DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */, DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */,
); );
runOnlyForDeploymentPostprocessing = 0; runOnlyForDeploymentPostprocessing = 0;
+44 -2
View File
@@ -1,6 +1,7 @@
import SwiftUI import SwiftUI
import UIKit import UIKit
import WebKit import WebKit
import UserNotifications
/// The Frame Control page, served by the server on the headset, in a web view. /// The Frame Control page, served by the server on the headset, in a web view.
/// window.frameApp (the same bridge the desktop app's preload.js provides) lets /// window.frameApp (the same bridge the desktop app's preload.js provides) lets
@@ -48,6 +49,7 @@ struct WebShell: UIViewRepresentable {
let installCb = null; let installCb = null;
window.frameApp = { window.frameApp = {
platform: "ios", platform: "ios",
notify: (message, request) => call("notify", { message, request }),
readClipboard: () => call("readClipboard"), readClipboard: () => call("readClipboard"),
setUpConnection: () => call("setUpConnection"), setUpConnection: () => call("setUpConnection"),
open: (what) => call("open", what), open: (what) => call("open", what),
@@ -58,13 +60,31 @@ struct WebShell: UIViewRepresentable {
})(); })();
""" """
final class Coordinator: NSObject, WKScriptMessageHandlerWithReply, WKNavigationDelegate, WKUIDelegate { final class Coordinator: NSObject, WKScriptMessageHandlerWithReply, WKNavigationDelegate, WKUIDelegate, UNUserNotificationCenterDelegate {
let model: AppModel let model: AppModel
weak var web: WKWebView? weak var web: WKWebView?
var loaded: URL? var loaded: URL?
private var installReady = false private var installReady = false
init(model: AppModel) { self.model = model } init(model: AppModel) {
self.model = model
super.init()
UNUserNotificationCenter.current().delegate = self
}
func userNotificationCenter(_ center: UNUserNotificationCenter, willPresent notification: UNNotification,
withCompletionHandler completionHandler: @escaping (UNNotificationPresentationOptions) -> Void) {
completionHandler([.banner, .sound, .list])
}
static func notificationContent(_ message: String) -> UNMutableNotificationContent? {
guard !message.isEmpty, message.count <= 500 else { return nil }
let content = UNMutableNotificationContent()
content.title = "Frame Control"
content.body = message
content.sound = .default
return content
}
// MARK: bridge // MARK: bridge
@@ -76,6 +96,28 @@ struct WebShell: UIViewRepresentable {
} }
let arg = body["arg"] let arg = body["arg"]
switch name { switch name {
case "notify":
guard message.frameInfo.isMainFrame,
message.frameInfo.securityOrigin.host == "127.0.0.1",
let args = arg as? [String: Any], let text = args["message"] as? String,
let content = Self.notificationContent(text) else {
return replyHandler(nil, "Invalid notification")
}
let center = UNUserNotificationCenter.current()
let send: (Bool, Error?) -> Void = { allowed, error in
guard allowed else {
return replyHandler(nil, error?.localizedDescription ?? "Notifications are off. Enable them in iOS Settings.")
}
let request = UNNotificationRequest(identifier: UUID().uuidString, content: content, trigger: nil)
center.add(request) { error in replyHandler(error == nil, error?.localizedDescription) }
}
if args["request"] as? Bool == true {
center.requestAuthorization(options: [.alert, .sound], completionHandler: send)
} else {
center.getNotificationSettings { settings in
send(settings.authorizationStatus == .authorized || settings.authorizationStatus == .provisional, nil)
}
}
case "readClipboard": case "readClipboard":
replyHandler(UIPasteboard.general.string ?? "", nil) replyHandler(UIPasteboard.general.string ?? "", nil)
case "setUpConnection": case "setUpConnection":
@@ -0,0 +1,14 @@
import XCTest
import UserNotifications
@testable import Frame_Control
final class ComfortNotificationTests: XCTestCase {
func testNotificationContentAndBounds() {
let content = WebShell.Coordinator.notificationContent("Time for a break")
XCTAssertEqual(content?.title, "Frame Control")
XCTAssertEqual(content?.body, "Time for a break")
XCTAssertNotNil(content?.sound)
XCTAssertNil(WebShell.Coordinator.notificationContent(""))
XCTAssertNil(WebShell.Coordinator.notificationContent(String(repeating: "x", count: 501)))
}
}
+6 -63
View File
@@ -1,65 +1,8 @@
#!/usr/bin/env zsh #!/usr/bin/env zsh
# Mac-side: upload VR videos to the Steam Frame so DeoVR can play them in 3D. # Upload media for Frame Control's own OpenVR player (no external player).
# # Usage: scripts/push-vr-video.sh [--launch] [--layout auto|mono|sbs|ou|full-sbs|full-ou] [--theatre] FILE...
# Files go to ~/Videos/VR on the Frame. The script links that folder into # scripts/push-vr-video.sh --list | --stop
# DeoVR's Proton prefix as C:\users\steamuser\Videos\VR, so DeoVR's file # Files go to ~/Videos/FrameControl/<id>/; --launch accepts exactly one file.
# browser finds it under Videos. (It's also reachable as Z:\home\steamos\Videos\VR.) # Directories, automatic DeoVR launching and Proton-prefix links are no longer used.
# Uploads resume if interrupted.
#
# Name files so DeoVR picks the projection: include _180 or _360 (or _fisheye190,
# _mkx200, _rf52 ...) plus the stereo layout (_LR / _SBS side by side, _TB over-
# under), e.g. "beach_180_LR.mp4". You can also change it in DeoVR's player.
#
# Usage:
# scripts/push-vr-video.sh FILE_OR_DIR... # upload
# scripts/push-vr-video.sh --launch FILE... # upload, then start DeoVR
# scripts/push-vr-video.sh --launch # just start DeoVR
# scripts/push-vr-video.sh --list # what's on the Frame
set -euo pipefail set -euo pipefail
exec python3 "${0:A:h}/../ui/frame_media_cli.py" "$@"
FRAME_ALIAS=${FRAME_ALIAS:-frame}
DEOVR_APPID=837380
REMOTE_DIR="Videos/VR"
PREFIX_VIDEOS=".local/share/Steam/steamapps/compatdata/$DEOVR_APPID/pfx/drive_c/users/steamuser/Videos"
launch=0 list=0
while (( $# )); do
case "$1" in
-h|--help) sed -n '2,17p' "$0"; exit 0 ;;
--launch) launch=1; shift ;;
--list) list=1; shift ;;
--) shift; break ;;
-*) print -u2 "push-vr-video: unknown option $1"; exit 2 ;;
*) break ;;
esac
done
(( $# || launch || list )) || { sed -n '2,17p' "$0" >&2; exit 2; }
for f in "$@"; do
[[ -e "$f" ]] || { print -u2 "push-vr-video: no such file: $f"; exit 2; }
done
# Create the folder and link it into DeoVR's prefix (the prefix exists once
# DeoVR has run). Refresh a stale link, but never replace a real directory.
if (( $# || launch )); then
ssh "$FRAME_ALIAS" "mkdir -p ~/$REMOTE_DIR
p=~/$PREFIX_VIDEOS
if [ -d \"\$p\" ] && { [ -L \"\$p/VR\" ] || [ ! -e \"\$p/VR\" ]; }; then ln -sfn ~/$REMOTE_DIR \"\$p/VR\"
elif [ -d \"\$p/VR\" ]; then echo \"warning: \$p/VR is a real folder, so uploads won't show under DeoVR's Videos; browse Z:\\\\home\\\\steamos\\\\Videos\\\\VR instead\" >&2; fi
[ -d \"\$p\" ] || echo 'note: DeoVR has not run yet; use Z:\\home\\steamos\\Videos\\VR or run this again after starting it once' >&2"
fi
if (( $# )); then
# -L: send what a symlink points at; a Mac-side link would dangle on the Frame
rsync -aL --partial --progress -- "$@" "$FRAME_ALIAS:$REMOTE_DIR/"
fi
if (( list )); then
ssh "$FRAME_ALIAS" "cd ~/$REMOTE_DIR && ls -lhR"
fi
if (( launch )); then
ssh "$FRAME_ALIAS" "command -v steam >/dev/null || { echo 'steam not found on the Frame' >&2; exit 1; }
steam steam://rungameid/$DEOVR_APPID </dev/null >/dev/null 2>&1 &"
print "DeoVR starting on the Frame. Open Local files / the file browser → Videos → VR."
fi
+22
View File
@@ -83,5 +83,27 @@ class Device(harness.FrameTestCase):
self.assertEqual(state()['steam']['pages'][0]['title'], 'Hades on Steam') self.assertEqual(state()['steam']['pages'][0]['title'], 'Hades on Steam')
class VRUtilities(harness.FrameTestCase):
def test_missing_vr_runtime_is_unavailable_not_zero_fps(self):
data = ok('GET', '/api/status')
self.assertIsNone(data['performance']['compositorFps'])
self.assertIsNone(data['performance']['appFps'])
self.assertEqual(data['temp'], 41.5)
self.assertEqual(data['battery']['percent'], 76)
def test_optional_paid_utilities_are_not_installed(self):
# The fake library contains games but none of these paid software titles.
for appid in (1009850, 1173510, 1068820, 908520):
code, body, _ = api('POST', '/api/steam', {'action': 'install', 'appid': appid})
self.assertEqual(code, 502, body)
self.assertIn('not owned', body['error'])
self.assertEqual(launches('install'), [])
def test_unverified_controls_are_not_exposed(self):
for action in ('recenter', 'adjust', 'restore'):
code, body, _ = api('POST', '/api/vr', {'action': action, 'origin': 'seated', 'y': .1})
self.assertEqual(code, 400, body)
if __name__ == '__main__': if __name__ == '__main__':
unittest.main() unittest.main()
+21
View File
@@ -0,0 +1,21 @@
"""Media transfer through the real HTTP/SSH path; the fake has no VR renderer."""
import harness
from harness import ok, ssh
harness.require()
class Media(harness.FrameTestCase):
def test_upload_and_list_without_launching_a_viewer(self):
# The transfer endpoint doesn't decode. Rendering belongs to the real
# headset smoke checks documented in docs/vr-video.md.
self.assertEqual(ssh('stat -c "%U:%G %a" ~/.local/share').strip(), 'steamos:steamos 755')
result = ok('POST', '/api/upload', raw=b'fake-media', headers={
'X-Mode': 'media', 'X-Filename': 'test_SBS.png'})
identity = result['id']
try:
files = ok('POST', '/api/media', {'action': 'list'})['files']
self.assertIn(identity, [f['id'] for f in files])
self.assertEqual(ssh('cat ~/Videos/FrameControl/'+identity), 'fake-media')
finally:
ssh('rm -rf ~/Videos/FrameControl/'+identity.split('/')[0])
@@ -27,7 +27,7 @@ function newShortcutId(steam) {
function build(steam) { function build(steam) {
const findShortcut = id => steam.shortcuts.find(s => s.appid === Number(id)); const findShortcut = id => steam.shortcuts.find(s => s.appid === Number(id));
const gameOverview = a => ({ const gameOverview = a => ({
appid: a.appid, display_name: a.display_name, sort_as: a.display_name, app_type: 1, appid: a.appid, display_name: a.display_name, sort_as: a.display_name, app_type: a.app_type ?? 1,
steam_hw_compat_category_packed: a.packed || 0, vr_supported: !!a.vr, vr_only: !!a.vr_only, steam_hw_compat_category_packed: a.packed || 0, vr_supported: !!a.vr, vr_only: !!a.vr_only,
size_on_disk: String(a.installed ? a.size : 0), minutes_playtime_forever: a.minutes || 0, size_on_disk: String(a.installed ? a.size : 0), minutes_playtime_forever: a.minutes || 0,
rt_last_time_played: a.last_played || 0, rt_last_time_played: a.last_played || 0,
@@ -231,6 +231,13 @@ def sysfs(state):
def runtimes(state): def runtimes(state):
"""Installed compat tools as Steam app manifests, and the Lepton launcher itself.""" """Installed compat tools as Steam app manifests, and the Lepton launcher itself."""
# Verified 2026-09-28, BUILD_ID 20260925.6191901: both parents are
# steamos:steamos 0755. The root supervisor must not leave them root-owned
# when creating Steam's fake manifests; user-account app installs need them.
for directory in (HOME + '/.local', HOME + '/.local/share'):
os.makedirs(directory, mode=0o755, exist_ok=True)
chown(directory)
os.chmod(directory, 0o755)
apps = fs.STEAM_ROOT + '/steamapps' apps = fs.STEAM_ROOT + '/steamapps'
for alias, installed in state['runtimes'].items(): for alias, installed in state['runtimes'].items():
acf = f'{apps}/appmanifest_{fs.RUNTIME_APPIDS[alias]}.acf' acf = f'{apps}/appmanifest_{fs.RUNTIME_APPIDS[alias]}.acf'
+244
View File
@@ -0,0 +1,244 @@
// Control on the live view (tap, drag, hold, scroll, type), run in node against the real
// functions from ui/index.html with a fake canvas and a fake server.
import { readFileSync } from "fs";
const src = readFileSync(new URL("../../ui/index.html", import.meta.url), "utf8");
const grab = name => {
const one = src.match(new RegExp(`\\n((?:async )?function ${name}\\(.*\\}\\n)`)); // one-line function
if (one) return one[1];
const m = src.match(new RegExp(`(?:\\nconst ${name} = [^\\n]*\\n)|((?:async )?function ${name}\\([\\s\\S]*?\\n}\\n)`));
if (!m) throw new Error("not found: " + name);
return m[0];
};
const NAMES = ["panelKey", "ctrlAimedAt", "ctrlSameTarget", "isMoveEvent", "isRelease", "ctrlKeepable", "TAP_MOVE", "ctrlAim", "ctrlKeyEvent", "ctrlTouchCancel", "ctrlSend", "ctrlFlush", "ctrlMoveTo", "ctrlMoveBy", "ctrlSchedule",
"ctrlFlushMoves", "ctrlButton", "ctrlClick", "ctrlRelease", "ctrlFraction", "ctrlTouchDown", "centroid",
"ctrlTouchMove", "ctrlTouchUp", "ctrlTap", "ctrlText"];
const code = NAMES.map(grab).join("");
const fail = msg => { console.log("FAIL " + msg); process.exit(1); };
const tick = (ms = 0) => new Promise(r => setTimeout(r, ms));
function page({ mode = "abs", rect = { left: 0, top: 0, width: 640, height: 360 }, api } = {}) {
const target = { panel: { window: 42, display: ":1" } };
const ctrl = { on: true, queue: [], sending: false, state: "ready", message: "", move: null, rel: [0, 0], raf: 0,
held: new Set(), keys: new Set(), pointers: new Map(), g: null, retry: null };
const sent = [];
const canvas = { width: 1280, height: 720, getBoundingClientRect: () => rect };
const env = {
ctrl, $: () => canvas, ctrlMode: () => mode, ctrlShow: () => {}, toast: () => {},
ctrlTarget: () => (mode !== "abs" ? { ok: true } : target.panel ? { ok: true, panel: target.panel } : { why: "gone" }),
api: api || (async (path, body) => { sent.push(...body.events); return { state: "ready", sent: true }; }),
requestAnimationFrame: cb => { setTimeout(cb, 0); return 1; },
navigator: {},
};
const fns = new Function(...Object.keys(env), `let ctrlWarned = false;\n${code}
return { ctrlTouchDown, ctrlTouchMove, ctrlTouchUp, ctrlTouchCancel, ctrlSend, ctrlText, ctrlButton, ctrlKeyEvent, ctrlRelease, ctrlFlushMoves };`)(...Object.values(env));
const at = (id, x, y) => ({ pointerId: id, clientX: x, clientY: y });
return { ctrl, sent, target, ...fns, at };
}
// A tap lands where it was tapped: pointer there first, then the click.
{
const p = page();
p.ctrlTouchDown(p.at(1, 320, 90)); p.ctrlTouchUp(p.at(1, 320, 90));
await tick(10);
const [move, down, up] = p.sent;
if (!(move.fx === 0.5 && Math.abs(move.fy - 0.25) < 1e-9 && move.window === 42 && move.display === ":1")) fail("tap position " + JSON.stringify(move));
if (!(down.window === 42 && down.display === ":1")) fail("a press names its panel " + JSON.stringify(down));
if (!(down.button === "left" && down.down && up.button === "left" && up.down === false && p.sent.length === 3))
fail("tap click " + JSON.stringify(p.sent));
}
// The panel is letterboxed in a taller view: taps map inside it, taps on the bars do nothing.
{
const p = page({ rect: { left: 0, top: 0, width: 640, height: 480 } }); // 640x360 picture, 60px bars
p.ctrlTouchDown(p.at(1, 320, 150)); p.ctrlTouchUp(p.at(1, 320, 150));
p.ctrlTouchDown(p.at(2, 320, 10)); p.ctrlTouchUp(p.at(2, 320, 10));
await tick(10);
if (!(p.sent.length === 3 && Math.abs(p.sent[0].fy - 0.25) < 1e-9)) fail("letterbox " + JSON.stringify(p.sent));
}
// A drag is a mouse drag: down where it started, moves, up at the end.
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 100));
p.ctrlTouchMove(p.at(1, 140, 100)); await tick(5);
p.ctrlTouchMove(p.at(1, 200, 120)); await tick(5);
p.ctrlTouchUp(p.at(1, 200, 120)); await tick(10);
const kinds = p.sent.map(e => "fx" in e ? "move" : `${e.button}-${e.down ? "down" : "up"}`);
if (!(kinds[0] === "move" && kinds[1] === "left-down" && kinds.at(-1) === "left-up" && kinds.includes("move", 2)))
fail("drag " + kinds.join(","));
if (Math.abs(p.sent[0].fx - 100 / 640) > 1e-9) fail("drag starts where the finger went down");
if (p.ctrl.held.size) fail("drag left the button held");
}
// Press and hold is a right-click where the finger is.
{
const p = page();
p.ctrlTouchDown(p.at(1, 64, 36));
await tick(620);
p.ctrlTouchUp(p.at(1, 64, 36)); await tick(10);
const buttons = p.sent.filter(e => "button" in e).map(e => `${e.button}-${e.down}`);
if (buttons.join() !== "right-true,right-false") fail("hold " + buttons.join());
if (!(p.sent[0].fx === 0.1)) fail("hold position " + JSON.stringify(p.sent[0]));
}
// Two fingers scroll, and the content follows them (fingers up scrolls down).
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 200)); p.ctrlTouchDown(p.at(2, 200, 200));
p.ctrlTouchMove(p.at(1, 100, 180)); p.ctrlTouchMove(p.at(2, 200, 180));
p.ctrlTouchUp(p.at(1, 100, 180)); p.ctrlTouchUp(p.at(2, 200, 180));
await tick(10);
const dy = p.sent.filter(e => e.scroll).reduce((a, e) => a + e.scroll[1], 0);
if (!(dy === 40 && !p.sent.some(e => "button" in e))) fail("scroll " + JSON.stringify(p.sent));
}
// On the headset view it's a trackpad: drags move the pointer, taps click where it is.
{
const p = page({ mode: "rel" });
p.ctrlTouchDown(p.at(1, 100, 100));
p.ctrlTouchMove(p.at(1, 110, 100)); p.ctrlTouchMove(p.at(1, 120, 105));
p.ctrlTouchUp(p.at(1, 120, 105)); await tick(10);
p.ctrlTouchDown(p.at(1, 50, 50)); p.ctrlTouchUp(p.at(1, 50, 50)); await tick(10);
const moved = p.sent.filter(e => "dx" in e).reduce((a, e) => [a[0] + e.dx, a[1] + e.dy], [0, 0]);
if (!(moved[0] === 32 && moved[1] === 8)) fail("trackpad move " + JSON.stringify(moved));
if (p.sent.some(e => "fx" in e)) fail("trackpad sent an absolute position");
if (p.sent.filter(e => e.button === "left").length !== 2) fail("trackpad tap " + JSON.stringify(p.sent));
}
// Text: plain ASCII only, in chunks the server takes.
{
const p = page();
p.ctrlText("héllo " + "x".repeat(600));
await tick(10);
const text = p.sent.map(e => e.text).join("");
if (!(text === "hllo " + "x".repeat(600) && p.sent.every(e => e.text.length <= 500))) fail("text " + text.length);
}
// Not connected yet: clicks and keys wait with their position. The request fails: only releases wait.
{
let calls = 0;
const p = page({ api: async () => { calls++; if (calls === 1) return { state: "starting", sent: false }; throw new Error("offline"); } });
p.ctrlSend([{ fx: 0.1, fy: 0.1, window: 42 }, { button: "left", down: true }, { key: 30, down: true }]);
await tick(5);
const q = p.ctrl.queue;
if (!(q.length === 3 && "fx" in q[0] && q[1].button === "left" && q[2].key === 30)) fail("kept while starting " + JSON.stringify(q));
clearTimeout(p.ctrl.retry);
p.ctrl.queue = []; p.ctrl.retryAt = 0;
p.ctrlSend([{ button: "left", down: false }, { key: 31, down: true }, { dx: 3, dy: 1 }]);
await tick(5);
if (!(p.ctrl.queue.length === 1 && p.ctrl.queue[0].button === "left" && p.ctrl.queue[0].down === false))
fail("release kept on failure " + JSON.stringify(p.ctrl.queue));
if (calls !== 2) fail("retried in a tight loop: " + calls + " requests");
clearTimeout(p.ctrl.retry);
}
// Turning Control off (or the view losing focus) lets go of anything held.
{
const p = page();
p.ctrlButton("left", true);
p.ctrlRelease(); await tick(10);
if (!(p.sent.at(-1).button === "left" && p.sent.at(-1).down === false && !p.ctrl.held.size)) fail("release " + JSON.stringify(p.sent));
}
// Keys held on the Frame are let go too.
{
const p = page();
p.ctrlKeyEvent(42, true);
p.ctrlRelease(); await tick(10);
if (!(p.sent.at(-1).key === 42 && p.sent.at(-1).down === false && !p.ctrl.keys.size)) fail("key release " + JSON.stringify(p.sent));
}
// A long queue sheds moves and old scrolls, never a release.
{
const p = page({ api: () => new Promise(() => {}) }); // stuck request
p.ctrlSend([{ dx: 1, dy: 1 }]);
p.ctrlSend([{ button: "left", down: false }]);
for (let i = 0; i < 320; i++) p.ctrlSend([{ scroll: [0, 1] }]);
if (!p.ctrl.queue.some(e => e.button === "left" && e.down === false)) fail("trim dropped a release");
if (p.ctrl.queue.length > 300) fail("trim kept " + p.ctrl.queue.length);
}
// Broken on the Frame side: no hammering, and only releases wait.
{
let calls = 0;
const p = page({ api: async () => { calls++; return { state: "error", sent: false }; } });
p.ctrlSend([{ button: "left", down: true }, { button: "left", down: false }]);
await tick(50);
if (calls !== 1) fail("error response reposted " + calls + " times");
if (!(p.ctrl.queue.length === 1 && p.ctrl.queue[0].down === false)) fail("error kept " + JSON.stringify(p.ctrl.queue));
clearTimeout(p.ctrl.retry);
}
// Connecting: a tap keeps its position, so it lands where it was made.
{
const p = page({ api: async () => ({ state: "starting", sent: false }) });
p.ctrlTouchDown(p.at(1, 320, 90)); p.ctrlTouchUp(p.at(1, 320, 90));
await tick(10);
const q = p.ctrl.queue;
if (!("fx" in q[0] && q[1].button === "left")) fail("connecting tap " + JSON.stringify(q));
clearTimeout(p.ctrl.retry);
}
// Lifting one of two scrolling fingers doesn't jump the scroll or start a drag.
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 200)); p.ctrlTouchDown(p.at(2, 300, 200));
p.ctrlTouchUp(p.at(1, 100, 200));
p.ctrlTouchMove(p.at(2, 300, 199));
p.ctrlTouchUp(p.at(2, 300, 199)); await tick(10);
if (p.sent.length) fail("one finger left after scrolling " + JSON.stringify(p.sent));
}
// A cancelled touch isn't a tap.
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 100)); p.ctrlTouchCancel(p.at(1, 100, 100)); await tick(10);
if (p.sent.length) fail("cancel clicked " + JSON.stringify(p.sent));
}
// Press and hold on the bars around the picture does nothing.
{
const p = page({ rect: { left: 0, top: 0, width: 640, height: 480 } });
p.ctrlTouchDown(p.at(1, 320, 10)); await tick(620); p.ctrlTouchUp(p.at(1, 320, 10)); await tick(10);
if (p.sent.length) fail("hold on the bars " + JSON.stringify(p.sent));
}
// Taps only reach the panel in use, and only once its picture is the one on screen.
{
const fns = ["panelKey", "deskPanel", "ctrlTarget"].map(grab).join("");
const check = (desk, live = true) => new Function("desk", "live", "ctrlMode", fns + "; return ctrlTarget();")(desk, live, () => "abs");
const a = { display: ":1", window: 5 }, b = { display: ":0", window: 5 };
const base = { panels: [a, b], loaded: true, pick: "", focus: ":1/5", shown: ":1/5" };
if (!check(base).ok) fail("target: shown and in use");
if (check({ ...base, shown: ":0/5" }).ok) fail("target: the picture is another panel with the same id");
if (!/Capture or Live/.test(check({ ...base, shown: null }, false).why)) fail("target: stale picture message");
if (check({ ...base, pick: ":0/5", shown: ":0/5" }).ok) fail("target: a watched panel that isn't in use");
if (check({ ...base, focus: null }).ok) fail("target: nothing in use");
}
// Focus moves to another panel mid-gesture: the tap or hold does nothing.
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 100));
p.target.panel = { window: 43, display: ":1" };
p.ctrlTouchUp(p.at(1, 100, 100)); await tick(10);
p.target.panel = { window: 42, display: ":1" };
p.ctrlTouchDown(p.at(1, 100, 100));
p.target.panel = null;
await tick(620); p.ctrlTouchUp(p.at(1, 100, 100)); await tick(10);
if (p.sent.some(e => "button" in e)) fail("gesture outlived its panel " + JSON.stringify(p.sent));
}
// Trimming keeps a click together with its position.
{
const p = page({ api: () => new Promise(() => {}) });
p.ctrlSend([{ dx: 1, dy: 0 }]); // in flight forever
for (let i = 0; i < 100; i++) p.ctrlSend([{ scroll: [0, 1] }]);
p.ctrlSend([{ fx: 0.5, fy: 0.5, window: 42, display: ":1" }, { button: "left", down: true }, { button: "left", down: false }]);
for (let i = 0; i < 198; i++) p.ctrlSend([{ scroll: [0, 1] }]);
const q = p.ctrl.queue, i = q.findIndex(e => e.button === "left" && e.down);
if (i < 1 || !("fx" in q[i - 1])) fail("trim split a click from its position");
}
// Backing off holds for new input too.
{
let calls = 0;
const p = page({ api: async () => { calls++; return { state: "error", sent: false }; } });
p.ctrlSend([{ button: "left", down: false }]);
await tick(5);
for (let i = 0; i < 10; i++) { p.ctrlSend([{ key: 30, down: false }]); await tick(2); }
if (calls !== 1) fail("new input bypassed the backoff: " + calls + " requests");
clearTimeout(p.ctrl.retry);
}
// A long paste goes in several requests, so a release never waits behind all of it.
{
const batches = [];
const p = page({ api: async (path, body) => { batches.push(body.events); return { state: "ready", sent: true }; } });
p.ctrlText("y".repeat(450));
p.ctrlButton("left", false);
await tick(30);
if (!batches.every(b => b.reduce((a, e) => a + (e.text?.length || 0), 0) <= 100)) fail("a batch carried too much text");
if (batches.length < 5) fail("paste went in " + batches.length + " requests");
}
console.log("control gestures ok");
+257
View File
@@ -0,0 +1,257 @@
"""Fake-Frame session clock/actions plus real helper serialization and sensor probes."""
import os
import shutil
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import Mock, patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui'))
import frame_comfort as comfort
import frame_status as status
OPTIONS = {'action': 'start', 'minutes': 3, 'breakMinutes': 1, 'stillMinutes': 1,
'batteryAlert': True, 'heatAlert': True}
class SessionTests(unittest.TestCase):
def setUp(self):
self.s = comfort.new_session(OPTIONS, 0, 'boot-one')
self.warn, self.home = Mock(), Mock()
def step(self, now, **sample):
comfort.tick(self.s, now, sample, self.warn, self.home, read_clock=lambda: now)
def test_warning_then_home_never_closes_a_game(self):
self.step(119)
self.warn.assert_not_called()
self.step(120)
self.warn.assert_called_once()
self.step(179)
self.home.assert_not_called()
self.step(180)
self.home.assert_called_once()
self.assertFalse(self.s['active'])
self.step(181)
self.home.assert_called_once()
def test_late_wakeup_always_gets_a_full_warning_minute(self):
self.step(400)
self.home.assert_not_called()
self.step(459)
self.home.assert_not_called()
self.step(460)
self.home.assert_called_once()
def test_slow_warning_still_leaves_a_full_minute(self):
comfort.tick(self.s, 120, {}, self.warn, self.home, read_clock=lambda: 140)
self.assertEqual(self.s['warned'], 140)
self.step(180)
self.home.assert_not_called()
self.step(199)
self.home.assert_not_called()
self.step(200)
self.home.assert_called_once()
def test_failed_warning_never_stops_session(self):
self.warn.side_effect = RuntimeError('offline')
with self.assertRaises(RuntimeError):
self.step(200)
self.assertIsNone(self.s['warned'])
self.home.assert_not_called()
self.warn.side_effect = None
self.step(300)
self.step(359)
self.home.assert_not_called()
self.step(360)
self.home.assert_called_once()
def test_failed_home_stays_active_and_retries(self):
self.step(120)
self.home.side_effect = RuntimeError('Steam offline')
with self.assertRaises(RuntimeError):
self.step(180)
self.assertTrue(self.s['active'])
self.home.side_effect = None
self.step(185)
self.assertFalse(self.s['active'])
def test_cancel_prevents_all_actions(self):
self.s['active'] = False
self.step(999, battery={'percent': 1, 'status': 'Discharging'})
self.warn.assert_not_called()
self.home.assert_not_called()
self.assertEqual(self.s['events'], [])
def test_breaks_and_checkin_require_measured_activity(self):
self.step(20, activity=1)
self.step(40, activity=2)
self.step(60, activity=1)
self.assertEqual([e['kind'] for e in self.s['events']], ['break', 'still'])
self.step(70, activity=1)
self.assertEqual(len(self.s['events']), 2)
self.step(75, activity=3)
self.assertEqual(self.s['used'], 0)
self.assertFalse(self.s['stillSent'])
def test_unknown_activity_and_gaps_do_not_count_as_wear(self):
self.step(25)
self.assertEqual(self.s['used'], 0)
self.assertEqual(self.s['unavailable'], ['battery', 'temperature', 'activity'])
self.step(100, activity=1)
self.assertEqual(self.s['used'], 30)
def test_alerts_latch_and_rearm_without_battery_chatter(self):
low = {'percent': 10, 'status': 'Discharging'}
self.step(1, battery=low, thermal=['cpu'])
self.step(2, battery=low, thermal=['cpu'])
self.step(3)
self.assertEqual(len(self.s['events']), 2)
self.step(4, battery={'percent': 16, 'status': 'Discharging'}, thermal=[])
self.step(5, battery=low, thermal=[])
self.assertEqual(len(self.s['events']), 2)
self.step(6, battery={'percent': 22, 'status': 'Discharging'}, thermal=[])
self.step(7, battery=low, thermal=['cpu'])
self.assertEqual([e['kind'] for e in self.s['events']], ['battery', 'heat', 'battery', 'heat'])
def test_disabled_alerts_and_charging(self):
self.s['options']['heatAlert'] = False
self.step(1, battery={'percent': 2, 'status': 'Charging'}, thermal=['cpu'])
self.assertEqual(self.s['events'], [])
def test_invalid_options(self):
for key, value in [('minutes', 0), ('minutes', 241), ('minutes', True), ('minutes', 2.5),
('breakMinutes', -1), ('stillMinutes', '1'), ('heatAlert', 1)]:
with self.subTest(key=key, value=value), self.assertRaises(ValueError):
comfort.validate({**OPTIONS, key: value})
for value in (None, [], {'action': 'shutdown'}):
with self.assertRaises(ValueError):
comfort.validate(value)
def test_restart_invalidates_session_and_stale_worker_is_explicit(self):
with patch.object(comfort, 'boot', return_value='boot-one'), patch.object(comfort.time, 'time', return_value=999):
current = comfort.current(self.s, 100)
self.assertIn('not responding', current['error'])
self.assertEqual(current['time'], 999)
with patch.object(comfort, 'boot', return_value='boot-two'):
result = comfort.current(self.s, 100)
self.assertFalse(result['active'])
self.assertIn('restarted', result['error'])
@unittest.skipUnless(os.name == "posix", "on-headset state uses POSIX flock")
def test_real_state_commands_share_one_session_and_cancel(self):
with tempfile.TemporaryDirectory() as tmp, patch.object(comfort, 'ROOT', Path(tmp)), \
patch.object(comfort, 'boot', return_value='boot-one'), \
patch.object(comfort, 'clock', return_value=0), patch.object(comfort.subprocess, 'Popen') as spawn:
started = comfort.command(OPTIONS)
self.assertEqual(comfort.command({'action': 'status'})['id'], started['id'])
with self.assertRaises(ValueError):
comfort.command(OPTIONS)
self.assertFalse(comfort.command({'action': 'cancel'})['active'])
spawn.assert_called_once()
self.assertEqual((Path(tmp) / 'session.json').stat().st_mode & 0o777, 0o600)
@unittest.skipUnless(os.name == "posix", "on-headset state uses POSIX flock")
def test_cancel_clears_stale_worker_error(self):
with tempfile.TemporaryDirectory() as tmp, patch.object(comfort, 'ROOT', Path(tmp)), \
patch.object(comfort, 'boot', return_value='boot-one'), \
patch.object(comfort, 'clock', return_value=200):
with comfort.locked():
comfort.save(self.s)
self.assertIn('not responding', comfort.command({'action': 'status'})['error'])
cancelled = comfort.command({'action': 'cancel'})
self.assertFalse(cancelled['active'])
self.assertIsNone(cancelled['error'])
self.assertIsNone(comfort.command({'action': 'status'})['error'])
@unittest.skipUnless(os.name == "posix", "on-headset state uses POSIX flock")
def test_failed_spawn_leaves_session_inactive_and_retryable(self):
with tempfile.TemporaryDirectory() as tmp, patch.object(comfort, 'ROOT', Path(tmp)), \
patch.object(comfort, 'boot', return_value='boot-one'), \
patch.object(comfort, 'clock', return_value=0), patch.object(comfort.subprocess, 'Popen') as spawn:
spawn.side_effect = OSError('process limit')
with self.assertRaises(OSError):
comfort.command(OPTIONS)
failed = comfort.command({'action': 'status'})
self.assertFalse(failed['active'])
self.assertIn('Could not start', failed['error'])
spawn.side_effect = None
self.assertTrue(comfort.command(OPTIONS)['active'])
@unittest.skipUnless(os.name == "posix", "on-headset state uses POSIX flock")
def test_unreadable_state_is_preserved_and_can_be_replaced(self):
for contents in (b'{broken', b'\xff', b'null', b'[]', b'42', b'"x"'):
with self.subTest(contents=contents):
with tempfile.TemporaryDirectory() as tmp, patch.object(comfort, 'ROOT', Path(tmp)), \
patch.object(comfort, 'boot', return_value='boot-one'), \
patch.object(comfort, 'clock', return_value=0), patch.object(comfort.subprocess, 'Popen'):
(Path(tmp) / 'session.json').write_bytes(contents)
failed = comfort.command({'action': 'status'})
self.assertFalse(failed['active'])
self.assertIn('unreadable', failed['error'])
backups = list(Path(tmp).glob('session-unreadable-*.json'))
self.assertEqual(len(backups), 1)
self.assertEqual(backups[0].read_bytes(), contents)
self.assertTrue(comfort.command(OPTIONS)['active'])
def test_home_has_total_process_deadline_and_propagates_timeout(self):
with patch.object(comfort.subprocess, 'run', side_effect=subprocess.TimeoutExpired('home', 15)) as run:
with self.assertRaises(subprocess.TimeoutExpired):
comfort.home()
self.assertEqual(run.call_args.kwargs['timeout'], 15)
self.assertEqual(run.call_args.args[0][-1], '--home')
def test_native_warning_reports_failures_and_quotes_as_one_argument(self):
with patch.object(comfort.subprocess, 'run') as run:
run.return_value = subprocess.CompletedProcess([], 0, 'Notification succeeded', '')
comfort.notify('Save "now"; $(nothing)')
args = run.call_args.args[0]
self.assertEqual(args, [comfort.VRCMD, '--notify', 'Frame Control: Save "now"; $(nothing)'])
run.return_value.stdout = 'Notification failed with error 1'
with self.assertRaises(RuntimeError):
comfort.notify('test')
@unittest.skipUnless(shutil.which("node"), "Node exercises the fake Steam JS context")
def test_home_javascript_against_fake_steam_preserves_game(self):
# Same JS runs in Steam CDP. This fake records navigation and refuses any
# unexpected API call; it offers no shutdown or terminate-game primitive.
js = '''let running = [123], path = '/routes/library/app/123', visible = false;
const location = {get pathname() {return path;}};
const SteamUIStore = {Navigate(p) {path = '/routes' + p;}};
const SteamClient = {OpenVR: {VROverlay: {
async ShowDashboard(key) {if (key !== 'valve.steam.gamepadui.main') throw Error(key); visible = true;},
async IsDashboardVisible() {return visible;}
}}};
'''
js += comfort.HOME_JS + '.then(result => console.log(JSON.stringify({result, running, visible})));'
r = subprocess.run(['node', '-e', js], capture_output=True, text=True, check=True)
result = json.loads(r.stdout)
self.assertEqual(result['running'], [123])
self.assertTrue(result['visible'])
self.assertEqual(result['result']['path'], '/routes/library/home')
class SensorTests(unittest.TestCase):
def test_hot_trip_uses_its_own_zone_not_hottest_unrelated_chip(self):
values = {'/z/a/temp': '90000', '/z/a/trip_point_0_type': 'hot', '/z/a/trip_point_0_temp': '110000',
'/z/b/temp': '45000', '/z/b/trip_point_0_type': 'hot', '/z/b/trip_point_0_temp': '44000', '/z/b/type': 'battery'}
def glob(pattern):
if pattern.endswith('thermal_zone*'):
return ['/z/a', '/z/b']
return [pattern.replace('*', '0')]
with patch.object(status.glob, 'glob', side_effect=glob), patch.object(status, 'read', side_effect=values.get):
self.assertEqual(status.thermal_alerts(), [{'zone': 'battery', 'tempC': 45, 'limitC': 44}])
def test_missing_thermal_and_activity_are_unknown(self):
with patch.object(status.glob, 'glob', return_value=[]):
self.assertIsNone(status.thermal_alerts())
with patch.object(status, 'run', return_value='unavailable'):
self.assertIsNone(status.activity_level())
for malformed in ('{}', '[null, 42, "bad"]'):
with patch.object(status, 'run', return_value=malformed):
self.assertIsNone(status.activity_level())
with patch.object(status, 'run', return_value='[{"operation":"status","activity_level":3}]'):
self.assertEqual(status.activity_level(), 3)
+60
View File
@@ -0,0 +1,60 @@
"""Run the actual shared page's comfort renderer against a minimal DOM/bridge."""
import pathlib
import shutil
import subprocess
import unittest
ROOT = pathlib.Path(__file__).resolve().parents[1]
@unittest.skipUnless(shutil.which('node'), 'Node exercises the shared page JS')
class ComfortUI(unittest.TestCase):
def test_notification_failure_survives_poll_until_success(self):
page = (ROOT / 'ui/index.html').read_text(encoding='utf-8')
code = page[page.index('let comfortBusy ='):page.index('async function pollComfort()')]
setup = r'''
const assert = require('node:assert/strict');
const elements = new Map();
const $ = id => {
if (!elements.has(id)) elements.set(id, {textContent:'', hidden:true, disabled:false, type: 'number'});
return elements.get(id);
};
let denied = 0;
const window = {frameApp:{notify:async()=>{denied++;throw Error('permission denied');}}};
const log = ()=>{}, toast = ()=>{};
'''
checks = r'''
(async()=>{
const active = {id:'session-one',active:true,time:100,remaining:120,
options:{minutes:2,breakMinutes:1,stillMinutes:1,batteryAlert:true,heatAlert:true},
events:[{id:'event-one',kind:'battery',time:99,message:'Low battery'}]};
renderComfort(active); // initial history must not replay even a fresh event
assert.equal(denied,0);
assert.equal($('comfortAnnouncement').textContent,'');
active.events.push({id:'event-two',kind:'break',time:100,message:'Take a break'});
renderComfort(active);
await new Promise(resolve=>setImmediate(resolve));
assert.equal(denied,1);
assert.equal($('comfortAnnouncement').textContent,'Take a break');
assert.equal($('comfortNotificationStatus').hidden,false);
assert.match($('comfortNotificationStatus').textContent,/notification settings/);
renderComfort({...active,time:105}); // the next normal poll must not erase failure
assert.equal($('comfortNotificationStatus').hidden,false);
assert.equal($('sessionStart').disabled,true);
assert.equal($('sessionMinutes').disabled,true);
assert.equal($('sessionCancel').disabled,false);
let requests=0;
window.frameApp.notify=async()=>{requests++;};
renderComfort({...active,time:106});
assert.equal($('comfortAnnouncement').textContent,'Take a break');
assert.equal(requests,0); // polling does not replay an already-seen event
await localNotification('test',true);
assert.equal($('comfortNotificationStatus').hidden,true);
renderComfort({...active,active:false});
assert.equal($('sessionStart').disabled,false);
assert.equal($('sessionMinutes').disabled,false);
assert.equal($('sessionCancel').disabled,true);
})().catch(e=>{console.error(e);process.exitCode=1;});
'''
result = subprocess.run(['node', '-e', setup + code + checks], capture_output=True, text=True)
self.assertEqual(result.returncode, 0, result.stderr)
+141
View File
@@ -0,0 +1,141 @@
"""Owned media planning, eye isolation, decoder choice and fake-Frame ownership."""
import json
import os
from pathlib import Path
import struct
import sys
import tempfile
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui'))
import frame_media as media
import frame_media_player as player
import frame_media_remote as remote
import frame_splat as splat
import server
class Media(unittest.TestCase):
def test_layout_evidence_and_override(self):
for name, layout in [('film_SBS.mp4', 'sbs'), ('film.OU.mkv', 'ou'),
('film_FSBS.mp4', 'full-sbs'), ('photo_TB.png', 'ou')]:
self.assertEqual(media.plan(name)['layout'], layout)
self.assertEqual(media.plan('film_SBS_OU.mp4', 'mono')['source'], 'explicit')
self.assertEqual(media.plan('film.mkv', metadata={'stereo_mode': 'top_bottom'})['layout'], 'full-ou')
for name in ('film.mp4', 'film_SBS_OU.mp4', 'businessbs.mp4'):
with self.assertRaises(ValueError):
media.plan(name)
with self.assertRaises(ValueError):
media.plan('film.mkv', metadata={'stereo_mode': 'right_left'})
def test_unsupported_containers_do_not_flatten_spatial_photos(self):
for name in ('spatial.HEIC', 'stereo.mpo', 'cloud.ply', 'cloud.spz', 'app.exe'):
with self.assertRaises(ValueError):
media.plan(name, 'sbs')
def test_ou_pixels_keep_each_eye_and_row(self):
a, b, c, d = [bytes([n])*8 for n in (1, 2, 3, 4)]
data, width, height = media.stereo_pixels(a+b+c+d, 2, 4, 'ou')
self.assertEqual((data, width, height), (a+c+b+d, 4, 2))
with self.assertRaises(ValueError):
media.stereo_pixels(b'bad', 2, 4, 'sbs')
self.assertEqual(media.geometry(3840, 2160, 'sbs'), (1920, 1080, 2))
self.assertEqual(media.geometry(1920, 1080, 'ou'), (1920, 1080, .5))
def test_decode_is_hardware_and_one_clock_for_audio(self):
for codec, decoder in [('h264', 'h264_v4l2m2m'), ('hevc', 'hevc_v4l2m2m')]:
cmd = player.decoder_command(Path('/tmp/a file.mp4'), {'codec_name': codec}, 1280, 720, True)
self.assertIn(decoder, cmd)
self.assertIn('-re', cmd)
self.assertIn('pulse', cmd)
self.assertIn(str(Path('/tmp/a file.mp4')), cmd)
with self.assertRaises(ValueError):
player.decoder_command(Path('x.webm'), {'codec_name': 'vp9'}, 640, 480, False)
cmd = player.decoder_command(Path('x.png'), {'codec_name': 'png'}, 640, 480, False, True)
self.assertNotIn('-re', cmd)
self.assertIn('-frames:v', cmd)
def test_fake_frame_library_and_traversal(self):
with tempfile.TemporaryDirectory() as d, patch.object(remote, 'ROOT', Path(d)):
identity = 'a'*32+'/space and quote\'.png'
path = Path(d)/identity
path.parent.mkdir()
path.write_bytes(b'test')
self.assertEqual(remote.media_path(identity), path.resolve())
for bad in ('../../etc/passwd', '/etc/passwd', 'a'*32+'/..', 'a'*32+'/x/y', None):
with self.assertRaises((ValueError, FileNotFoundError)):
remote.media_path(bad)
link = path.parent/'link.png'
link.symlink_to(path)
with self.assertRaises(ValueError):
remote.media_path('a'*32+'/link.png')
with patch.object(remote, 'status', return_value={'state': 'idle'}):
files = remote.run({'action': 'list'})['files']
self.assertEqual([f['id'] for f in files], [identity])
def test_server_rejects_bad_actions_before_ssh(self):
with patch.object(server, 'ssh') as ssh:
for body in ({'action': 'delete'}, {'action': 'play', 'id': '../x'},
{'action': 'play', 'id': 'a'*32+'/x', 'layout': 'invalid'},
{'action': 'play', 'id': 'a'*32+'/x', 'theatre': 'false'}):
with self.assertRaises(server.Failure):
server.media(body)
ssh.assert_not_called()
def test_fake_frame_stop_only_owns_our_unit(self):
for rc in (0, 5): # 5: already collected ("not loaded"), a no-op
with patch.object(remote.subprocess, 'run') as run, patch.object(remote, 'status', return_value={'state': 'ended'}):
run.return_value.returncode = rc
self.assertEqual(remote.run({'action': 'stop'})['state'], 'ended')
self.assertEqual(run.call_args.args[0], ['systemctl', '--user', 'stop', 'frame-control-media.service'])
with patch.object(remote.subprocess, 'run') as run, patch.object(remote, 'status', return_value={}):
run.return_value.returncode, run.return_value.stderr = 1, 'Access denied'
with self.assertRaisesRegex(RuntimeError, 'Access denied'):
remote.run({'action': 'stop'})
def test_start_failure_reports_systemd_error(self):
with tempfile.TemporaryDirectory() as d, patch.object(remote, 'ROOT', Path(d)), \
patch.object(remote, 'STATUS', Path(d)/'status.json'), \
patch.object(remote, 'active', return_value=False), \
patch.object(remote.subprocess, 'run') as run:
identity = 'b'*32+'/still_SBS.png'
(Path(d)/identity).parent.mkdir()
(Path(d)/identity).write_bytes(b'x')
run.return_value.returncode, run.return_value.stderr = 1, 'Unit already exists'
with patch.object(remote, 'probe', return_value=({}, False)), \
self.assertRaisesRegex(RuntimeError, 'Unit already exists'):
remote.run({'action': 'play', 'id': identity})
self.assertEqual(run.call_args.args[0][0], 'systemd-run') # reset-failed's result is ignored
def test_upload_rejects_unplayable_names_and_keeps_copy_error(self):
with patch.object(server, 'ssh') as ssh, patch.object(server, 'push_file') as push:
# On Windows a backslash is a separator, so such a name can't reach here.
for name in ('.hidden.mp4',) + (('a\\b_SBS.mp4',) if os.sep == '/' else ()):
with self.assertRaises(server.Failure):
server.push_media(Path('/tmp')/name)
ssh.assert_not_called()
push.side_effect = server.Failure('copy failed')
ssh.side_effect = [None, server.Failure('link down')]
with self.assertRaisesRegex(server.Failure, 'copy failed'):
server.push_media(Path('/tmp/film_SBS.mp4'))
def test_splat_invalid_records_and_stereo_parallax(self):
with tempfile.TemporaryDirectory() as d:
path = Path(d)/'small.splat'
path.write_bytes(struct.pack('<6f8B', 0, 0, 0, .001, .001, .001,
255, 0, 0, 255, 255, 128, 128, 128))
data, w, h = splat.render(path, 64, 48)
self.assertEqual((len(data), w, h), (128*48*4, 128, 48))
def centroid(eye):
weights = [(x, data[(y*w+x+eye*64)*4]) for y in range(h) for x in range(64)]
return sum(x*v for x,v in weights)/sum(v for x,v in weights)
self.assertGreater(centroid(0), centroid(1))
for bad in (b'', b'bad', struct.pack('<6f8B', float('nan'), 0, 0, 1, 1, 1, *([128]*8))):
path.write_bytes(bad)
with self.assertRaises(ValueError):
splat.read(path)
if __name__ == '__main__':
unittest.main()
+102
View File
@@ -0,0 +1,102 @@
"""Fake-Frame panel responses and the headset page's real HTTP guards."""
import http.client
import json
from pathlib import Path
import sys
import threading
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / 'ui'))
import frame_panels as panels
# Shape verified with vrcmd on SteamVR 2.18.1 / BUILD_ID 20260925.6191901.
OVERLAYS = """---- OVERLAYS ----
'valve.steam.desktopgame.12' -- 'Alex's <notes>', 1920x1080 visible VROverlayType_Dashboard_Main
'valve.steam.desktopgame.12.thumb' -- 'Thumb', not_visible VROverlayType_Dashboard_Thumbnail
'valve.steam.desktopgame.12.layer1' -- 'Layer', visible VROverlayType_Subview
'system.pointer' -- 'Pointer', visible VROverlayType_Basic
'valve.steam.desktopgame.13' -- 'Other', not_visible VROverlayType_Dashboard_Main
"""
class Panels(unittest.TestCase):
def test_enumerates_only_main_panels_including_hidden(self):
rows = panels.parse_overlays(OVERLAYS)
self.assertEqual(len(rows), 2)
self.assertEqual(rows[0]['title'], "Alex's <notes>")
self.assertTrue(rows[0]['visible'])
self.assertFalse(rows[1]['visible'])
def test_unavailable_runtime_is_not_an_empty_workspace(self):
with self.assertRaises(panels.PanelError):
panels.parse_overlays('SteamVR not running')
self.assertEqual(panels.parse_overlays('---- OVERLAYS ----'), [])
@patch.object(panels, 'run', return_value=OVERLAYS)
def test_focus_revalidates_and_dispatches_without_shell(self, run):
key = 'valve.steam.desktopgame.12'
self.assertEqual(panels.focus(key)['requested'], key)
self.assertEqual(run.call_args.args[0], [panels.VRCMD, '--showdashboard', key])
@patch.object(panels, 'run', return_value=OVERLAYS)
def test_closed_or_injected_panel_never_dispatches(self, run):
for key in (None, 12, 'x;touch /tmp/bad', '../other', 'missing'):
with self.assertRaises(panels.PanelError):
panels.focus(key)
self.assertEqual(run.call_count, 1) # only valid-looking 'missing' enumerates
class HeadsetHTTP(unittest.TestCase):
def setUp(self):
self.server = panels.HTTPServer(('127.0.0.1', 0), panels.Handler)
self.server.key = 'test-key'
self.server.closing = False
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
self.thread.start()
def tearDown(self):
self.server.shutdown()
self.thread.join()
self.server.server_close()
def request(self, path, body=None, headers=None):
c = http.client.HTTPConnection('127.0.0.1', self.server.server_port, timeout=5)
try:
c.request('POST' if body is not None else 'GET', path, body=body, headers=headers or {})
r = c.getresponse()
return r.status, r.read().decode()
finally:
c.close()
def test_page_is_public_but_contains_no_key_or_private_titles(self):
status, page = self.request('/')
self.assertEqual(status, 200)
self.assertNotIn('test-key', page)
self.assertIn('textContent=p.title', page) # titles never become HTML
self.assertEqual(self.request('/panels')[0], 403)
@patch.object(panels, 'state', return_value={'panels': []})
def test_auth_host_and_origin_checks(self, state):
auth = {'X-Panel-Key': 'test-key'}
self.assertEqual(self.request('/panels', headers=auth)[0], 200)
for extra in ({'Host': 'evil.test'}, {'Origin': 'https://evil.test'}, {'X-Panel-Key': 'wrong'}):
self.assertEqual(self.request('/panels', headers={**auth, **extra})[0], 403)
self.assertEqual(state.call_count, 1)
@patch.object(panels, 'focus', return_value={'requested': 'panel'})
def test_post_validation_and_close(self, focus):
auth = {'X-Panel-Key': 'test-key'}
for body in ('[]', '{broken', '0', '"text"', 'x' * 1025):
self.assertEqual(self.request('/focus', body, auth)[0], 400)
self.assertEqual(focus.call_count, 0)
self.assertEqual(self.request('/focus', '{"key":"panel"}', auth)[0], 200)
self.assertEqual(focus.call_args.args, ('panel',))
self.assertEqual(self.request('/close', '{}', auth)[0], 200)
self.assertTrue(self.server.closing)
@patch.object(panels, 'state', side_effect=panels.PanelError('offline'))
def test_offline_is_an_error_not_a_successful_empty_list(self, state):
status, body = self.request('/panels', headers={'X-Panel-Key': 'test-key'})
self.assertEqual(status, 502)
self.assertEqual(json.loads(body), {'error': 'offline'})
+3
View File
@@ -84,6 +84,7 @@ class ServerGuards(unittest.TestCase):
def test_api_needs_custom_header(self): def test_api_needs_custom_header(self):
# <img src> and plain form posts from other sites can't set it. # <img src> and plain form posts from other sites can't set it.
self.assertEqual(self.request("POST", "/api/comfort", {"action": "start"})[0], 403)
self.assertEqual(self.request("GET", "/api/status")[0], 403) self.assertEqual(self.request("GET", "/api/status")[0], 403)
self.assertEqual(self.request("GET", "/api/screenshot?view=headset")[0], 403) self.assertEqual(self.request("GET", "/api/screenshot?view=headset")[0], 403)
self.assertEqual(self.request("GET", "/api/shots")[0], 403) self.assertEqual(self.request("GET", "/api/shots")[0], 403)
@@ -99,6 +100,8 @@ class ServerGuards(unittest.TestCase):
def test_input_validation(self): def test_input_validation(self):
cases = [ cases = [
("/api/comfort", {"action": "poweroff"}),
("/api/comfort", {"action": "start", "minutes": 0}),
("/api/launch", {"appid": "620; rm -rf ~"}), ("/api/launch", {"appid": "620; rm -rf ~"}),
("/api/launch", {"appid": ""}), ("/api/launch", {"appid": ""}),
("/api/flatpak", {"id": "org.example.App;id", "action": "install"}), ("/api/flatpak", {"id": "org.example.App;id", "action": "install"}),
+283
View File
@@ -0,0 +1,283 @@
"""Touch: the live view's Control (ui/frame_touch.py on the Frame, and the server's checks).
Run: python3 -m unittest discover -s tests
"""
import shutil
import subprocess
import sys
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "ui"))
class Mapping(unittest.TestCase):
@classmethod
def setUpClass(cls):
import frame_touch
cls.t = frame_touch
def test_panel_bigger_than_its_display_is_scaled(self):
# Verified 2026-09-29: a 1920x1080 window on :1 (1280x720) took Accept at
# window (1828, 1020) as pointer (1219, 680).
panel = {"root": [1280, 720], "width": 1920, "height": 1080}
x, y = self.t.to_root(panel, 1828 / 1920, 1020 / 1080)
self.assertAlmostEqual(x, 1218.7, delta=1)
self.assertAlmostEqual(y, 679.4, delta=1)
def test_same_size_is_one_to_one_and_clamped(self):
panel = {"root": [1280, 720], "width": 1280, "height": 720}
self.assertEqual(self.t.to_root(panel, 0, 0), (0, 0))
self.assertEqual(self.t.to_root(panel, 1, 1), (1279, 719))
self.assertEqual(self.t.to_root(panel, -3, 7), (0, 719))
def test_other_shapes_are_letterboxed(self):
panel = {"root": [1280, 720], "width": 800, "height": 800} # square: bars left and right
x, y = self.t.to_root(panel, 0, 0.5)
self.assertAlmostEqual(x, 280, delta=0.5)
self.assertAlmostEqual(y, 359.5, delta=0.5)
def test_same_id_on_both_displays_is_told_apart_by_pid(self):
t = self.t
saved = t.displays, t.window_info, t.window_pid
self.addCleanup(lambda: (setattr(t, "displays", saved[0]), setattr(t, "window_info", saved[1]),
setattr(t, "window_pid", saved[2])))
t.displays = lambda: [":0", ":1"]
t.window_info = lambda d, w: {"name": f"on {d}", "width": 1280 if w != "root" else 1920, "height": 720}
t.window_pid = lambda d, w: {":0": 111, ":1": 222}[d]
self.assertEqual(t.locate(5, 222)["display"], ":1")
self.assertEqual(t.locate(5, 111)["display"], ":0")
self.assertEqual(t.locate(5, None)["display"], ":0")
def test_focus_display_is_decoded(self):
t = self.t
saved = t.xprop_root
self.addCleanup(lambda: setattr(t, "xprop_root", saved))
for values, want in (([12602, 0, 58], ":1"), ([12346], ":0"), ([], None), ([0x41], None)):
t.xprop_root = lambda d, n, v=values: v
self.assertEqual(t.focus_display(), want)
def test_ascii_table_covers_printable_characters(self):
for code in range(0x20, 0x7F):
self.assertIn(chr(code), self.t.ASCII, chr(code))
self.assertEqual(self.t.ASCII["a"], (30, False))
self.assertEqual(self.t.ASCII["A"], (30, True))
self.assertEqual(self.t.ASCII["?"], (53, True))
self.assertEqual(self.t.ASCII["1"], (2, False))
self.assertEqual(self.t.ASCII["0"], (11, False))
def test_events_parsing(self):
self.assertEqual(self.t.events(b'{"dx": 1}'), [{"dx": 1}])
self.assertEqual(self.t.events(b'[{"dx": 1}, 5]'), [{"dx": 1}])
self.assertEqual(self.t.events(b"nope"), [])
class FakeGamescope:
def __init__(self):
self.calls = []
def __getattr__(self, name):
return lambda *a: self.calls.append((name, *a))
class Apply(unittest.TestCase):
@classmethod
def setUpClass(cls):
import frame_touch
cls.t = frame_touch
def setUp(self):
self.focused = {"window": 7, "display": ":1", "root": [1280, 720], "width": 1280, "height": 720, "name": "x"}
saved = self.t.focus, self.t.say, self.t.focus_now
self.t.STALE[0] = False
self.said = []
self.t.focus = lambda: dict(self.focused)
self.t.focus_now = lambda: (self.focused["window"], self.focused["display"])
self.t.say = lambda state, **more: self.said.append((state, more))
self.addCleanup(lambda: (setattr(self.t, "focus", saved[0]), setattr(self.t, "say", saved[1]),
setattr(self.t, "focus_now", saved[2])))
def test_tap_moves_then_clicks_in_order(self):
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"}, {"button": "left", "down": True, "window": 7, "display": ":1"},
{"button": "left", "down": False}):
panel = self.t.apply(gs, e, panel)
self.assertEqual([c[0] for c in gs.calls], ["move_to", "button", "button"])
self.assertEqual(gs.calls[1][1:], ("left", True))
def test_a_tap_meant_for_another_panel_goes_nowhere(self):
# Focus moved on: the position and the press are dropped; the release still goes.
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, {"button": "left", "down": True, "window": 99, "display": ":1"},
{"key": 30, "down": True, "window": 7, "display": ":0"}, {"button": "left", "down": False, "window": 99, "display": ":1"}):
panel = self.t.apply(gs, e, panel)
self.assertEqual(gs.calls, [("button", "left", False)])
self.assertEqual(self.said[0], ("ready", {"focus": 7, "display": ":1", "stale": True}))
def test_presses_read_focus_afresh(self):
# A move may use a recent reading; a press checks again, so a panel that just
# took focus doesn't get a click meant for another.
gs, calls = FakeGamescope(), []
self.t.focus = lambda: calls.append(1) or dict(self.focused)
panel = self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"}, None)
panel = self.t.apply(gs, {"fx": 0.6, "fy": 0.5, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 1)
self.t.apply(gs, {"button": "left", "down": True, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 1) # same panel still: the quick check was enough
self.focused["window"] = 8
self.t.apply(gs, {"button": "left", "down": True, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 2)
self.assertEqual([c[0] for c in gs.calls], ["move_to", "move_to", "button"])
def test_stale_is_said_once_and_cleared(self):
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"},
{"fx": 0.6, "fy": 0.5, "window": 7, "display": ":1"}):
panel = self.t.apply(gs, e, panel)
self.assertEqual(self.said, [("ready", {"focus": 7, "display": ":1", "stale": True}),
("ready", {"focus": 7, "display": ":1"})])
def test_stale_clears_on_a_trackpad_move_but_not_on_a_stale_release(self):
gs = FakeGamescope()
panel = self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, None)
panel = self.t.apply(gs, {"button": "left", "down": False, "window": 99, "display": ":1"}, panel)
self.assertTrue(self.t.STALE[0]) # that release was still aimed at the old panel
self.t.apply(gs, {"dx": 3, "dy": 0}, panel)
self.assertFalse(self.t.STALE[0])
self.assertEqual(self.said[-1][1].get("stale"), None)
def test_same_window_id_on_the_other_display_is_another_panel(self):
gs = FakeGamescope()
self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":0"}, None)
self.assertEqual(gs.calls, [])
def test_relative_scroll_keys_text(self):
gs = FakeGamescope()
for e in ({"dx": 5, "dy": -3}, {"scroll": [0, 120]}, {"key": 30, "down": True}, {"text": "hi"},
{"key": True}, {"button": "sideways"}):
self.t.apply(gs, e, None)
self.assertEqual([c[0] for c in gs.calls], ["move_by", "scroll", "key", "text"])
def test_everything_held_is_let_go(self):
class Held(self.t.Gamescope):
def __init__(self):
self.held, self.keys, self.log = set(), set(), []
def frame(self):
pass
g = Held()
g.L = type("L", (), {"ei_device_button_button": lambda *a: g.log.append(("button",) + a[2:]),
"ei_device_keyboard_key": lambda *a: g.log.append(("key",) + a[2:])})()
g.device = object()
g.button("left", True)
g.key(42, True)
g.release_all()
self.assertEqual(g.log[-2:], [("button", 0x110, False), ("key", 42, False)])
self.assertEqual((g.held, g.keys), (set(), set()))
def test_resuming_after_a_pause_lets_go_of_everything(self):
t = self.t
log, queue = [], [1]
class L:
def __getattr__(self, name):
if name == "ei_get_event":
return lambda ei: queue.pop(0) if queue else None
return {"ei_event_get_type": lambda ev: t.EV_DEVICE_RESUMED, "ei_event_get_device": lambda ev: "dev",
"ei_device_has_capability": lambda d, c: True,
"ei_device_button_button": lambda d, c, down: log.append(("button", c, down)),
"ei_device_keyboard_key": lambda d, c, down: log.append(("key", c, down))}.get(name, lambda *a: 0)
g = t.Gamescope.__new__(t.Gamescope)
g.L, g.ei, g.fd, g.device, g.sequence, g.alive = L(), None, None, None, 0, True
g.held, g.keys = {0x110}, {42}
g.frame = lambda: None
old = t.select.select
t.select.select = lambda *a: ([], [], [])
try:
g.pump()
finally:
t.select.select = old
self.assertEqual(sorted(log), [("button", 0x110, False), ("key", 42, False)])
self.assertEqual(g.device, "dev")
def test_text_uses_shift_for_capitals(self):
class Keys(self.t.Gamescope):
def __init__(self):
self.pressed = []
def key(self, code, down):
self.pressed.append((code, down))
k = Keys()
k.text("Hié") # the é has no key on a US layout and is left out
self.assertEqual(k.pressed, [(42, True), (35, True), (35, False), (42, False), (23, True), (23, False)])
class ServerChecks(unittest.TestCase):
@classmethod
def setUpClass(cls):
import server
cls.s = server
def test_touch_event_keeps_known_fields(self):
ev = self.s.touch_event
self.assertEqual(ev({"fx": 0.5, "fy": 2, "window": 5, "display": ":1"}), {"fx": 0.5, "fy": 1.0, "window": 5, "display": ":1"})
self.assertEqual(ev({"button": "left", "window": 5, "display": ":0"}), {"button": "left", "down": True, "window": 5, "display": ":0"})
self.assertEqual(ev({"button": "right"}), {"button": "right", "down": True})
self.assertEqual(ev({"key": 30, "down": False}), {"key": 30, "down": False})
self.assertEqual(ev({"scroll": [0, 1e9]}), {"scroll": [0.0, 5000.0]})
self.assertEqual(ev({"dx": 3, "other": 1}), {"dx": 3.0})
def test_touch_event_rejects_bad_ones(self):
for bad in (None, {}, {"fx": 0.5, "fy": 0.5}, {"fx": "1", "fy": 0, "window": 1, "display": ":1"}, {"button": "side"},
{"fx": 0.5, "fy": 0.5, "window": 1}, {"fx": 0.5, "fy": 0.5, "window": 1, "display": ":1;x"},
{"window": 1, "display": ":1"},
{"key": 0}, {"key": 999}, {"key": True}, {"scroll": [1]}, {"text": ""}, {"text": "x" * 501},
{"fx": 0.1, "fy": 0.1, "window": True, "display": ":1"}):
with self.assertRaises(self.s.Failure, msg=repr(bad)):
self.s.touch_event(bad)
def test_panel_stream_is_the_window_and_checked(self):
cmd = self.s.stream_command("src=panel&window=10485777&display=:1&h=720&fps=30")
self.assertIn("DISPLAY=:1 ffmpeg", cmd)
self.assertIn("-window_id 10485777 -i :1", cmd)
for bad in ("src=panel&window=1;rm&display=:1", "src=panel&window=1&display=:1;x", "src=panel&display=:1"):
with self.assertRaises(self.s.Failure):
self.s.stream_command(bad + "&h=720&fps=30")
def test_touch_agent_runs_the_helper_with_nothing_to_copy(self):
agent = self.s.TouchAgent()
self.assertEqual(agent.deliver(lambda m: None), "")
cmd = agent.command()
self.assertTrue(cmd.startswith("python3 -u -c '"))
self.assertIn("frame_touch", cmd)
def test_batch_limit(self):
with self.assertRaises(self.s.Failure):
self.s.remote_touch({"events": [{"dx": 1}] * (self.s.INPUT_BATCH_LIMIT + 1)})
@unittest.skipUnless(shutil.which("node"), "needs node")
class PageGestures(unittest.TestCase):
"""Control's gestures and queue (tests/page/ctrl_gestures.mjs runs the real functions from index.html)."""
def test_gestures(self):
r = subprocess.run(["node", str(ROOT / "tests" / "page" / "ctrl_gestures.mjs")], capture_output=True, text=True,
timeout=60)
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
class Script(unittest.TestCase):
def test_helper_compiles_on_the_frames_python(self):
# The Frame runs it with its own python3 (3.13 on SteamOS 0.4.1); stdlib and ctypes only.
src = (ROOT / "ui/frame_touch.py").read_text()
compile(src, "frame_touch.py", "exec")
for mod in ("import ctypes", "import json", "import select"):
self.assertIn(mod, src)
self.assertNotIn("import requests", src)
if __name__ == "__main__":
unittest.main()
+152
View File
@@ -0,0 +1,152 @@
"""Own VR telemetry, optional catalogue and entitlement guards; no headset needed."""
import ctypes
import json
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
import unittest
from unittest import mock
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui'))
import frame_status as status
import frame_steam as steam
import frame_utilities as utilities
import frame_vr as vr
import server
class Telemetry(unittest.TestCase):
def test_background_client_does_not_request_steamvr_start(self):
with mock.patch.object(status.C, 'CDLL') as load:
with status.OpenVR():
pass
self.assertEqual(load.return_value.VR_InitInternal2.call_args.args[1], 3)
load.return_value.VR_ShutdownInternal.assert_called_once()
def test_linux_arm64_timing_layout(self):
self.assertEqual(ctypes.sizeof(status.Timing), 192)
self.assertEqual(status.Timing.pose.offset, 96)
def test_cpu_excludes_guest_and_counts_iowait_idle(self):
with mock.patch.object(status, 'read', return_value='cpu 20 0 10 50 20 0 0 0 7 1\n'):
self.assertEqual(status.cpu_ticks(), (100, 70))
self.assertEqual(status.cpu_percent((100, 70), (200, 110)), 60)
self.assertIsNone(status.cpu_percent((100, 70), (100, 70)))
self.assertIsNone(status.cpu_percent(None, (100, 70)))
def test_compositor_and_app_fps_are_distinct(self):
a, b = status.Timing(), status.Timing()
a.index, a.time = 100, 10
b.index, b.time, b.interval = 118, 10.2, 22.2222
b.gpu, b.compositorCpu = 4, 1
result = status.timing_values(a, b)
self.assertEqual(result['compositorFps'], 90)
self.assertEqual(result['appFps'], 45)
self.assertEqual(result['frameMs'], 11.11)
b.interval = 0
self.assertIsNone(status.timing_values(a, b)['appFps'])
b.index = a.index
self.assertEqual(status.timing_values(a, b), {})
b.index = 5 # compositor restart / wrap
self.assertEqual(status.timing_values(a, b), {})
def test_missing_openvr_keeps_sensor_values(self):
with mock.patch.object(status, 'OpenVR', side_effect=OSError('missing')), \
mock.patch.object(status, 'num', return_value=629), \
mock.patch.object(status, 'cpu_ticks', side_effect=[(100, 50), (200, 100)]), \
mock.patch.object(status.time, 'sleep'):
data = status.performance()
self.assertEqual(data['gpuMHz'], 629)
self.assertEqual(data['cpuPercent'], 50)
self.assertIsNone(data['compositorFps'])
self.assertIsNone(data['appFps'])
def test_import_has_no_device_side_effect(self):
out = subprocess.run([sys.executable, '-c', 'import frame_status'],
cwd=Path(__file__).resolve().parents[1] / 'ui', capture_output=True, text=True)
self.assertEqual(out.returncode, 0, out.stderr)
self.assertEqual(out.stdout, '')
class UtilityGuards(unittest.TestCase):
def test_all_paid_utilities_refused_before_steam_url(self):
with mock.patch.object(steam, 'Page') as page, mock.patch.object(steam, 'steam_url') as url:
page.return_value.eval.return_value = [{'id': i, 'owned': False} for i in steam.UTILITY_IDS]
for appid in set(steam.UTILITY_IDS) - set(steam.FREE_UTILITIES):
with self.assertRaisesRegex(steam.Fail, 'not owned'):
steam.install(appid)
url.assert_not_called()
def test_unknown_ownership_fails_closed(self):
with mock.patch.object(steam, 'Page') as page, mock.patch.object(steam, 'steam_url') as url:
page.return_value.eval.side_effect = steam.Fail('library unavailable')
with self.assertRaises(steam.Fail):
steam.install(908520)
url.assert_not_called()
def test_owned_software_uses_existing_install_flow(self):
with mock.patch.object(steam, 'Page') as page, mock.patch.object(steam, 'steam_url') as url:
page.return_value.eval.side_effect = [[{'id': 908520, 'owned': True}], {'name': 'fpsVR', 'installed': True}]
self.assertEqual(steam.install(908520)['state'], 'installed')
url.assert_not_called()
def test_catalogue_separates_public_claims_ownership_and_reports(self):
data = utilities.catalogue([{'id': 1173510, 'owned': True}], [
{'package': 'com.android.app', 'rating': 'works', 'date': '2026-09-28'},
{'package': 'steam:908520', 'rating': 'broken', 'date': '2026-09-28', 'notes': 'test report'}])
rows = {r['id']: r for r in data['utilities']}
self.assertEqual(rows[1173510]['status'], 'untested')
self.assertTrue(rows[1173510]['canInstall'])
self.assertFalse(rows[908520]['canInstall'])
self.assertEqual(rows[908520]['status'], 'broken')
self.assertTrue(rows[1494460]['canInstall'])
self.assertFalse(rows[1009850]['free'])
@unittest.skipUnless(shutil.which('node'), 'Node needed to evaluate Steam JavaScript')
def test_ownership_js_includes_software_and_rejects_empty_library(self):
script = 'let appStore = {allApps: [{appid:908520,app_type:2,local_per_client_data:{installed:true}}]};\n'
script += 'console.log(JSON.stringify(' + steam.UTILITIES_JS + '));'
out = subprocess.run(['node', '-e', script], capture_output=True, text=True, check=True)
rows = {r['id']: r for r in json.loads(out.stdout)}
self.assertTrue(rows[908520]['owned'])
self.assertTrue(rows[908520]['installed'])
self.assertFalse(rows[1173510]['owned'])
out = subprocess.run(['node', '-e', 'let appStore={allApps:[]};' + steam.UTILITIES_JS], capture_output=True)
self.assertNotEqual(out.returncode, 0)
class HUD(unittest.TestCase):
def test_invalid_action_never_reaches_frame(self):
with mock.patch.object(server, 'ssh') as ssh:
for action in ('hud-start; reboot', 'adjust', 'recenter', 'restore'):
with self.assertRaises(server.Failure) as err:
server.vr({'action': action})
self.assertEqual(err.exception.status, 400)
ssh.assert_not_called()
def test_stop_wont_kill_reused_pid(self):
with tempfile.TemporaryDirectory() as tmp, mock.patch.object(vr, 'ROOT', Path(tmp)), \
mock.patch.object(vr, 'process_identity', return_value='different'), mock.patch.object(vr.os, 'kill') as kill:
(Path(tmp) / 'hud.json').write_text(json.dumps({'pid': 1234, 'start': 'old'}))
vr.panel('hud-stop')
kill.assert_not_called()
self.assertFalse((Path(tmp) / 'hud.json').exists())
def test_duplicate_start_does_not_spawn(self):
with tempfile.TemporaryDirectory() as tmp, mock.patch.object(vr, 'ROOT', Path(tmp)), \
mock.patch.object(vr, 'process_identity', return_value='same'), mock.patch.object(vr.subprocess, 'Popen') as popen:
(Path(tmp) / 'hud.json').write_text(json.dumps({'pid': 1234, 'start': 'same'}))
self.assertIn('already open', vr.panel('hud-start')['message'])
popen.assert_not_called()
def test_start_failure_cleans_up_process(self):
with tempfile.TemporaryDirectory() as tmp, mock.patch.object(vr, 'ROOT', Path(tmp)), \
mock.patch.object(vr.subprocess, 'Popen') as popen, \
mock.patch.object(vr.subprocess, 'check_output', side_effect=OSError('no display')):
popen.return_value.poll.return_value = None
with self.assertRaises(OSError):
vr.panel('hud-start')
popen.return_value.terminate.assert_called_once()
popen.return_value.wait.assert_called_once()
+264
View File
@@ -0,0 +1,264 @@
"""Opt-in session worker ON the Frame; no root, extra apps, or power actions.
One worker per user, shared by desktop and phone. State survives companion
connections, not headset reboots. See docs/family-comfort.md for guarantees.
"""
import contextlib
import json
import os
from pathlib import Path
import subprocess
import sys
import time
import uuid
from frame_steam import Page
from frame_status import battery, thermal_alerts, activity_level
ROOT = Path.home() / '.local/state/frame-control/comfort'
VRCMD = '/opt/steamvr/bin/linuxarm64/vrcmd'
HOME_JS = """(async () => {
SteamUIStore.Navigate('/library/home');
await SteamClient.OpenVR.VROverlay.ShowDashboard('valve.steam.gamepadui.main');
if (!await SteamClient.OpenVR.VROverlay.IsDashboardVisible()) throw Error('Steam dashboard did not open');
return {path: location.pathname};
})()"""
def clock():
# CLOCK_BOOTTIME includes headset suspend; wall-clock corrections don't alter limits.
return time.clock_gettime(time.CLOCK_BOOTTIME)
def boot():
return Path('/proc/sys/kernel/random/boot_id').read_text().strip()
def validate(body):
if not isinstance(body, dict) or body.get('action') not in ('status', 'start', 'cancel'):
raise ValueError('Choose status, start or cancel')
if body['action'] == 'start':
for key, low, high in (('minutes', 1, 240), ('breakMinutes', 0, 120), ('stillMinutes', 0, 240)):
n = body.get(key)
if type(n) is not int or not low <= n <= high:
raise ValueError(f'{key} must be a whole number from {low} to {high}')
for key in ('batteryAlert', 'heatAlert'):
if type(body.get(key)) is not bool:
raise ValueError(f'{key} must be true or false')
return body
def new_session(body, now, boot_id):
return {'id': uuid.uuid4().hex, 'boot': boot_id, 'active': True,
'options': {k: body[k] for k in ('minutes', 'breakMinutes', 'stillMinutes', 'batteryAlert', 'heatAlert')},
'started': now, 'deadline': now + body['minutes'] * 60, 'lastSample': now,
'used': 0, 'nextBreak': body['breakMinutes'] * 60, 'stillSent': False,
'warned': None, 'events': [], 'seq': 0, 'latched': [], 'error': None}
def event(s, kind, message):
s['seq'] += 1
s['events'].append({'id': s['id'] + ':' + str(s['seq']), 'kind': kind,
'message': message, 'time': time.time()})
s['events'] = s['events'][-40:]
def notify(message):
r = subprocess.run([VRCMD, '--notify', 'Frame Control: ' + message],
capture_output=True, text=True, timeout=20)
if r.returncode or 'succeeded' not in r.stdout:
raise RuntimeError('SteamVR could not show the reminder: ' + (r.stderr or r.stdout)[-300:])
def home():
# A total process deadline also bounds a CDP peer that keeps sending events
# without completing the request. Keep cancellation ordered after this action.
r = subprocess.run([sys.executable, str(Path(__file__).resolve()), '--home'],
capture_output=True, text=True, timeout=15)
if r.returncode:
raise RuntimeError('Steam Home failed: ' + (r.stdout or r.stderr)[-300:])
def open_home():
page = Page()
try:
result = page.eval(HOME_JS)
if result.get('path') != '/routes/library/home':
raise RuntimeError('Steam did not navigate Home')
finally:
page.sock.close()
def tick(s, now, sample, warn=notify, go_home=home, read_clock=clock):
"""One deterministic step; injected actions/samples also exercise a fake Frame."""
if not s.get('active'):
return
o = s['options']
s['heartbeat'] = now
delta = max(0, min(30, now - s['lastSample']))
s['lastSample'] = now
level = sample.get('activity')
b = sample.get('battery') or {}
s['unavailable'] = []
if o['batteryAlert'] and b.get('percent') is None:
s['unavailable'].append('battery')
if o['heatAlert'] and sample.get('thermal') is None:
s['unavailable'].append('temperature')
if (o['breakMinutes'] or o['stillMinutes']) and level is None:
s['unavailable'].append('activity')
s['activity'] = level
if level in (1, 2):
s['used'] += delta
elif level is not None:
s['used'] = 0
s['nextBreak'] = o['breakMinutes'] * 60
s['stillSent'] = False
# Missing samples never count as time worn. No catch-up burst after a disconnect.
if now >= s['deadline'] - 60 and s['warned'] is None:
warn('One minute left. Save your progress; Steam Home will open.')
s['warned'] = max(now, read_clock())
event(s, 'warning', 'One minute left. Save your progress; Steam Home will open.')
if s['warned'] is not None and now >= max(s['deadline'], s['warned'] + 60):
go_home()
s['active'] = False
event(s, 'finished', 'Session ended: Steam Home opened. Your game is still running.')
return
if o['breakMinutes'] and s['used'] >= s['nextBreak']:
warn('Time for a break. Take off the headset and rest your eyes.')
event(s, 'break', 'Time for a break. Take off the headset and rest your eyes.')
s['nextBreak'] = s['used'] + o['breakMinutes'] * 60
if o['stillMinutes'] and not s['stillSent'] and s['used'] >= o['stillMinutes'] * 60:
event(s, 'still', f"Headset still active after {o['stillMinutes']} active minute(s). Check in with the wearer.")
s['stillSent'] = True
low = b.get('percent') is not None and b['percent'] <= 15 and b.get('status') == 'Discharging'
hot = sample.get('thermal')
for kind, enabled, value, message in (
('battery', o['batteryAlert'], low if b else None, 'Frame battery is low (15% or less).'),
('heat', o['heatAlert'], bool(hot) if hot is not None else None,
'Frame reports a hot/critical thermal trip or battery overheat. Ask the wearer to take a break.')):
if enabled and value and kind not in s['latched']:
event(s, kind, message)
s['latched'].append(kind)
elif value is False and kind in s['latched']:
# Battery hysteresis prevents repeated alerts around 15%.
if kind != 'battery' or b.get('status') == 'Charging' or (b.get('percent') or 0) >= 20:
s['latched'].remove(kind)
@contextlib.contextmanager
def locked(name='state.lock', nonblocking=False):
import fcntl # only needed ON the Linux headset, not by desktop validation/tests
ROOT.mkdir(parents=True, exist_ok=True, mode=0o700)
with (ROOT / name).open('a') as f:
fcntl.flock(f, fcntl.LOCK_EX | (fcntl.LOCK_NB if nonblocking else 0))
yield f
def read_state():
try:
state = json.loads((ROOT / 'session.json').read_text())
if not isinstance(state, dict):
raise ValueError('Saved session must be an object')
return state
except FileNotFoundError:
return {'active': False, 'events': []}
except (ValueError, UnicodeDecodeError):
# Preserve the unreadable state for diagnosis, then allow a new session.
(ROOT / 'session.json').replace(ROOT / ('session-unreadable-' + uuid.uuid4().hex + '.json'))
return {'active': False, 'events': [],
'error': 'Saved session was unreadable. Start a new session.'}
def save(s):
p = ROOT / 'session.tmp'
p.write_text(json.dumps(s))
p.chmod(0o600)
p.replace(ROOT / 'session.json')
def current(s, now):
if s.get('active') and s.get('boot') != boot():
s['active'] = False
s['error'] = 'Headset restarted. Start a new session.'
out = dict(s)
out['time'] = time.time() # event age uses the Frame's clock, not the phone's
out['remaining'] = max(0, max(s.get('deadline', now), (s.get('warned') or 0) + 60) - now) if s.get('active') else 0
beat = s.get('heartbeat', s.get('started', now)) # a hand-edited state may lack either
if s.get('active') and now - beat > 90:
out['error'] = 'Session worker is not responding. Timer enforcement is unverified; cancel and start again.'
return out
def watch():
try:
with locked('worker.lock', nonblocking=True) as worker:
while True:
with locked():
s = current(read_state(), clock())
if not s.get('active'):
save(s)
# Release ownership before state.lock: a concurrent start
# cannot miss the gap between an old worker and its exit.
import fcntl
fcntl.flock(worker, fcntl.LOCK_UN)
return
try:
b = battery()
hot = thermal_alerts()
if b and b.get('health') == 'Overheat':
hot = (hot or []) + ['battery']
tick(s, clock(), {'battery': b, 'thermal': hot, 'activity': activity_level()})
s['error'] = None
except Exception as e:
error = str(e)
if s.get('error') != error:
event(s, 'error', 'Session action failed: ' + error)
s['error'] = error
save(s)
time.sleep(5)
except BlockingIOError:
pass # another connection already started the single worker
def command(body):
validate(body)
with locked():
s = current(read_state(), clock())
if body['action'] == 'start':
if s.get('active'):
raise ValueError('A session is already running. Cancel it before starting another.')
s = new_session(body, clock(), boot())
event(s, 'started', 'Session started. Steam Home opens at the limit; games are not closed.')
elif body['action'] == 'cancel':
s['active'] = False
s['error'] = None
if s.get('id'):
event(s, 'cancelled', 'Session timer and monitoring cancelled.')
save(s)
if body['action'] == 'start':
try:
subprocess.Popen([sys.executable, str(Path(__file__).resolve()), '--watch'],
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
start_new_session=True, close_fds=True)
except OSError as e:
s['active'] = False
s['error'] = 'Could not start session worker: ' + str(e)
event(s, 'error', s['error'])
save(s)
raise
return current(s, clock())
if __name__ == '__main__':
if sys.argv[1:] == ['--watch']:
watch()
else:
try:
if sys.argv[1:] == ['--home']:
open_home()
print(json.dumps({'home': True}))
else:
print(json.dumps(command(json.loads(sys.argv[1]))))
except Exception as e:
print(json.dumps({'error': str(e)}))
sys.exit(1)
+67
View File
@@ -0,0 +1,67 @@
"""Media planning shared by Frame Control and its own Frame-side player.
No viewer dependencies. Filename hints are suggestions, never guesses from
resolution. Explicit layout wins; conflicting hints require a choice.
"""
import re
from pathlib import Path
LAYOUTS = ('auto', 'mono', 'sbs', 'ou', 'full-sbs', 'full-ou')
VIDEO = {'.mp4', '.mkv', '.mov', '.webm', '.m4v'}
PHOTO = {'.png', '.jpg', '.jpeg'}
def plan(name, layout='auto', metadata=None):
if layout not in LAYOUTS:
raise ValueError('Choose auto, mono, sbs, ou, full-sbs or full-ou')
suffix = Path(name).suffix.lower()
if suffix in {'.heic', '.heif', '.avif', '.mpo'}:
raise ValueError('Native spatial-photo containers are not supported yet; export both eyes as SBS or OU PNG/JPEG')
if suffix == '.splat':
return {'kind': 'splat', 'layout': 'sbs', 'source': 'renderer'}
if suffix not in VIDEO | PHOTO:
raise ValueError('Use MP4/MKV/MOV/WebM video, PNG/JPEG stereo photos, or a .splat file')
source = 'explicit'
if layout == 'auto':
tokens = set(re.split(r'[^a-z0-9]+', Path(name).stem.lower()))
hints = set()
for value, tags in [('full-sbs', {'fsbs'}), ('full-ou', {'fou', 'ftb'}),
('sbs', {'sbs', 'hsbs', 'lr'}), ('ou', {'ou', 'hou', 'tb', 'htb'})]:
if tokens & tags:
hints.add(value)
if len(hints) > 1:
raise ValueError('Conflicting stereo filename tags; choose the layout explicitly')
layout = next(iter(hints), None)
source = 'filename'
if not layout:
# Matroska StereoMode/FFmpeg stereo_mode: only known left-first modes.
mode = (metadata or {}).get('stereo_mode')
layout = {'left_right': 'full-sbs', 'top_bottom': 'full-ou', 'mono': 'mono'}.get(mode)
source = 'metadata'
if mode and layout is None:
raise ValueError('Unsupported stereo metadata; choose the eye order/layout explicitly')
if not layout:
raise ValueError('No stereo layout found; choose mono, SBS or OU (left/top eye first)')
return {'kind': 'video' if suffix in VIDEO else 'photo', 'layout': layout, 'source': source}
def geometry(width, height, layout):
"""Bound transfer to 1920x1080; return packed dimensions and texel aspect."""
if not 0 < width <= 32768 or not 0 < height <= 32768:
raise ValueError('Invalid media dimensions')
if layout not in LAYOUTS[1:]:
raise ValueError('Resolve the layout before playback')
scale = min(1, 1920 / width, 1080 / height)
w, h = max(2, int(width * scale) // 2 * 2), max(2, int(height * scale) // 2 * 2)
return w, h, {'mono': 1, 'sbs': 2, 'ou': .5, 'full-sbs': 1, 'full-ou': 1}[layout]
def stereo_pixels(data, width, height, layout):
"""Normalize top/bottom to OpenVR's left/right texture; preserve eye order."""
if len(data) != width * height * 4:
raise ValueError('Incomplete RGBA frame')
if layout not in ('ou', 'full-ou'):
return data, width, height
stride, half = width * 4, height // 2
return b''.join(data[y*stride:(y+1)*stride] +
data[(y+half)*stride:(y+half+1)*stride] for y in range(half)), width*2, half
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env python3
"""Send local media to Frame Control's own OpenVR player."""
import argparse
import json
from pathlib import Path
import sys
sys.path.insert(0, str(Path(__file__).resolve().parent))
import frame_media
import server
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument('files', nargs='*', type=Path)
ap.add_argument('--launch', action='store_true', help='play the one file being sent')
ap.add_argument('--layout', choices=frame_media.LAYOUTS, default='auto')
ap.add_argument('--theatre', action='store_true', help='bigger screen and dark surround')
ap.add_argument('--list', action='store_true')
ap.add_argument('--stop', action='store_true')
args = ap.parse_args()
if args.launch and len(args.files) != 1:
ap.error('--launch needs exactly one file')
if not args.files and not (args.list or args.stop):
ap.error('choose files, --list or --stop')
for path in args.files:
if not path.is_file():
ap.error('not a file: %s' % path)
frame_media.plan(path.name, 'mono')
if args.stop:
print(json.dumps(server.media({'action': 'stop'})))
for path in args.files:
result = server.push_media(path.resolve())
print(json.dumps(result))
if args.launch:
print(json.dumps(server.media({'action': 'play', 'id': result['id'],
'layout': args.layout, 'theatre': args.theatre})))
if args.list:
print(json.dumps(server.media({'action': 'list'})))
if __name__ == '__main__':
try:
main()
except (ValueError, server.Failure) as e:
sys.exit(str(e))
+213
View File
@@ -0,0 +1,213 @@
#!/usr/bin/env python3
"""Frame Control's local-media OpenVR player. Runs on the Frame, no third-party app.
SteamOS ffmpeg does hardware video decoding, scaling and audio output. OpenVR
owns only our screen and optional black surround. Exiting destroys both.
"""
import argparse
import ctypes as C
import json
import os
from pathlib import Path
import signal
import subprocess
import time
import frame_media
import frame_splat
LIB = '/opt/steamvr/bin/linuxarm64/libopenvr_api.so'
H = C.c_uint64
# Slots from Valve's openvr_capi.h, IVROverlay_028. Fail closed on another ABI.
SLOTS = {
'CreateOverlay': (1, [C.c_char_p, C.c_char_p, C.POINTER(H)]),
'DestroyOverlay': (3, [H]),
'SetOverlayFlag': (11, [H, C.c_int, C.c_bool]),
'SetOverlayAlpha': (16, [H, C.c_float]),
'SetOverlayTexelAspect': (18, [H, C.c_float]),
'SetOverlaySortOrder': (20, [H, C.c_uint32]),
'SetOverlayWidthInMeters': (22, [H, C.c_float]),
'SetOverlayTransformTrackedDeviceRelative': (35, [H, C.c_uint32, C.c_void_p]),
'ShowOverlay': (43, [H]),
'SetOverlayRaw': (62, [H, C.c_void_p, C.c_uint32, C.c_uint32, C.c_uint32]),
}
class Overlay:
def __init__(self):
self.handles = []
self.vr = C.CDLL(LIB)
self.vr.VR_InitInternal2.argtypes = [C.POINTER(C.c_int), C.c_int, C.c_char_p]
self.vr.VR_GetGenericInterface.argtypes = [C.c_char_p, C.POINTER(C.c_int)]
self.vr.VR_GetGenericInterface.restype = C.c_void_p
err = C.c_int()
self.vr.VR_InitInternal2(C.byref(err), 2, None)
if err.value:
raise RuntimeError('SteamVR init failed: %s' % err.value)
ptr = self.vr.VR_GetGenericInterface(b'FnTable:IVROverlay_028', C.byref(err))
if not ptr or err.value:
self.vr.VR_ShutdownInternal()
raise RuntimeError('SteamVR needs IVROverlay_028: %s' % err.value)
self.table = C.cast(ptr, C.POINTER(C.c_void_p))
def call(self, name, *values):
slot, args = SLOTS[name]
rc = C.CFUNCTYPE(C.c_int, *args)(self.table[slot])(*values)
if rc:
raise RuntimeError('OpenVR %s failed: %s' % (name, rc))
def create(self, key, width, distance, stereo=False, aspect=1, order=1):
handle = H()
self.call('CreateOverlay', key.encode(), b'Frame Control media', C.byref(handle))
self.handles.append(handle)
self.call('SetOverlayWidthInMeters', handle, width)
self.call('SetOverlaySortOrder', handle, order)
self.call('SetOverlayTexelAspect', handle, aspect)
if stereo:
self.call('SetOverlayFlag', handle, 1024, True) # SideBySide_Parallel
matrix = (C.c_float * 12)(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, -distance)
self.call('SetOverlayTransformTrackedDeviceRelative', handle, 0, matrix)
return handle
def pixels(self, handle, data, width, height):
buf = C.create_string_buffer(data)
self.call('SetOverlayRaw', handle, buf, width, height, 4)
self.call('ShowOverlay', handle)
def close(self):
try:
for h in reversed(self.handles):
self.call('DestroyOverlay', h)
finally:
self.vr.VR_ShutdownInternal()
def probe(path):
result = subprocess.run(['ffprobe', '-v', 'error', '-show_streams', '-of', 'json', str(path)],
capture_output=True, text=True, timeout=30)
if result.returncode:
raise ValueError(result.stderr[-2000:] or 'Cannot read media')
streams = json.loads(result.stdout)['streams']
video = next((s for s in streams if s['codec_type'] == 'video'), None)
if not video:
raise ValueError('No image or video stream')
return video, any(s['codec_type'] == 'audio' for s in streams)
def decoder_command(path, info, width, height, audio, photo=False):
cmd = ['ffmpeg', '-nostdin', '-hide_banner', '-loglevel', 'error']
if not photo:
cmd += ['-re', '-readrate_initial_burst', '0']
codec = {'h264': 'h264_v4l2m2m', 'hevc': 'hevc_v4l2m2m'}.get(info['codec_name'])
if not codec:
raise ValueError('Hardware playback currently supports H.264 and H.265 only')
cmd += ['-c:v', codec]
cmd += ['-i', str(path), '-map', '0:v:0', '-vf', 'scale=%s:%s' % (width, height),
'-pix_fmt', 'rgba']
if photo:
cmd += ['-frames:v', '1']
else:
cmd += ['-r', '30']
cmd += ['-f', 'rawvideo', 'pipe:1']
if audio and not photo:
cmd += ['-map', '0:a:0', '-f', 'pulse', 'Frame Control Media']
return cmd
def write_status(path, **values):
tmp = path.with_suffix('.tmp')
tmp.write_text(json.dumps(values))
tmp.replace(path)
def play(args):
path = Path(args.file).resolve(strict=True)
status = Path(args.status)
splat = path.suffix.lower() == '.splat'
if splat:
data, width, height = frame_splat.render(path)
plan = frame_media.plan(path.name)
aspect, photo, command = 1, True, None
else:
info, audio = probe(path)
plan = frame_media.plan(path.name, args.layout, info.get('tags'))
width, height, aspect = frame_media.geometry(info['width'], info['height'], plan['layout'])
photo = plan['kind'] == 'photo'
command = decoder_command(path, info, width, height, audio, photo)
vr, proc, frames, started = None, None, 0, time.monotonic()
# systemd sends SIGTERM to the whole unit, including ffmpeg. Python unwinds
# ownership; no unrelated Steam/SteamVR process or setting is touched.
def stop(signum, frame):
raise InterruptedError('Stopped')
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
try:
vr = Overlay()
if args.theatre:
surround = vr.create('framecontrol.media.surround', 40, 4, order=0)
vr.call('SetOverlayAlpha', surround, .85)
vr.pixels(surround, b'\x00\x00\x00\xff', 1, 1)
screen = vr.create('framecontrol.media.screen', 3 if args.theatre else 1.6, 2,
plan['layout'] != 'mono', aspect)
if splat:
vr.pixels(screen, data, width, height)
write_status(status, state='playing', file=path.name, frames=1, **plan)
while True:
time.sleep(1)
proc = subprocess.Popen(command, stdout=subprocess.PIPE)
video_start = time.monotonic()
while True:
data = proc.stdout.read(width * height * 4)
if not data:
break
data, outw, outh = frame_media.stereo_pixels(data, width, height, plan['layout'])
if not photo:
time.sleep(max(0, video_start + frames/30 - time.monotonic()))
vr.pixels(screen, data, outw, outh)
frames += 1
if frames == 1 or frames % 30 == 0:
write_status(status, state='playing', file=path.name, frames=frames,
seconds=time.monotonic()-started, **plan)
if not photo:
time.sleep(max(0, video_start + frames/30 - time.monotonic()))
rc = proc.wait(timeout=10)
if rc:
raise RuntimeError('ffmpeg exited %s; see media log' % rc)
if not frames:
raise RuntimeError('Decoder produced no frames')
if photo:
while True:
time.sleep(1)
write_status(status, state='ended', frames=frames, seconds=time.monotonic()-started)
except InterruptedError:
write_status(status, state='stopped', frames=frames)
finally:
if proc:
if proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait()
proc.stdout.close()
if vr:
vr.close()
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument('file')
ap.add_argument('--layout', choices=frame_media.LAYOUTS, default='auto')
ap.add_argument('--theatre', action='store_true')
ap.add_argument('--status', required=True)
args = ap.parse_args()
try:
play(args)
except Exception as e:
write_status(Path(args.status), state='error', error=str(e))
raise
if __name__ == '__main__':
main()
+109
View File
@@ -0,0 +1,109 @@
"""Frame-side library and process ownership for Frame Control media.
Only the dedicated systemd user unit is controlled. No SteamVR settings change.
"""
import argparse
import json
from pathlib import Path
import re
import subprocess
import sys
import frame_media
from frame_media_player import probe
ROOT = Path.home() / 'Videos' / 'FrameControl'
RUNTIME = Path.home() / '.local' / 'share' / 'frame-control' / 'media'
UNIT = 'frame-control-media.service'
STATUS = RUNTIME / 'status.json'
def media_path(identity):
if not isinstance(identity, str) or '\\' in identity or '\x00' in identity:
raise ValueError('Invalid media id')
parts = Path(identity).parts
if len(parts) != 2 or not re.fullmatch('[0-9a-f]{32}', parts[0]) or parts[1].startswith('.'):
raise ValueError('Invalid media id')
candidate = ROOT / identity
if candidate.is_symlink() or candidate.parent.is_symlink():
raise ValueError('Media links are not supported')
path = candidate.resolve(strict=True)
if not path.is_file() or ROOT.resolve() not in path.parents:
raise ValueError('Media file is outside the library')
return path
def active():
return subprocess.run(['systemctl', '--user', 'is-active', '--quiet', UNIT]).returncode == 0
def status():
running = active()
try:
state = json.loads(STATUS.read_text())
except (OSError, ValueError):
state = {'state': 'idle'}
if not running and state.get('state') in ('playing', 'starting', 'paused'):
state = {'state': 'stopped', 'message': 'Player exited; check the media log if this was unexpected'}
return dict(state, running=running)
def run(body):
action = body.get('action')
if action == 'list':
files = []
if ROOT.exists():
for folder in sorted(ROOT.iterdir()):
if not re.fullmatch('[0-9a-f]{32}', folder.name) or not folder.is_dir() or folder.is_symlink():
continue
for path in sorted(folder.iterdir()):
if path.is_file() and not path.is_symlink() and not path.name.startswith('.'):
files.append({'id': folder.name+'/'+path.name, 'name': path.name, 'bytes': path.stat().st_size})
return {'files': files, 'player': status()}
if action == 'status':
return status()
if action == 'stop':
# --collect unloads the unit after it exits; systemctl then exits 5
# ("not loaded", verified on the Frame). That's a finished player, not an error.
stopped = subprocess.run(['systemctl', '--user', 'stop', UNIT], capture_output=True, text=True, timeout=15)
if stopped.returncode not in (0, 5):
raise RuntimeError('Could not stop the media player: ' + (stopped.stderr.strip() or 'exit %s' % stopped.returncode))
return {'message': 'Media player stopped', **status()}
if action != 'play':
raise ValueError('Media action must be list, status, play or stop')
path = media_path(body.get('id'))
if type(body.get('theatre', False)) is not bool:
raise ValueError('theatre must be true or false')
info = {} if path.suffix.lower() == '.splat' else probe(path)[0]
plan = frame_media.plan(path.name, body.get('layout', 'auto'), info.get('tags'))
if active():
raise ValueError('Stop the current media before starting another file')
# systemd owns the process group and refuses a concurrent start of this name.
# The runtime cap also cleans up if the controlling computer disconnects.
subprocess.run(['systemctl', '--user', 'reset-failed', UNIT], stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=10)
STATUS.write_text(json.dumps({'state': 'starting', 'file': path.name}))
command = ['systemd-run', '--user', '--quiet', '--collect', '--unit='+UNIT,
'--property=RuntimeMaxSec=14400', '--property=TimeoutStopSec=8',
'--property=StandardOutput=append:'+str(RUNTIME/'player.log'),
'--property=StandardError=append:'+str(RUNTIME/'player.log'),
'python3', str(RUNTIME/'frame_media_player.py'), str(path),
'--layout', plan['layout'], '--status', str(STATUS)]
if body.get('theatre'):
command.append('--theatre')
started = subprocess.run(command, capture_output=True, text=True, timeout=15)
if started.returncode:
raise RuntimeError('Could not start the media player: ' + (started.stderr.strip() or 'systemd-run exited %s' % started.returncode))
return {'message': 'Starting Frame Control media', 'plan': plan}
def main():
try:
print(json.dumps(run(json.load(sys.stdin))))
except Exception as e:
print(json.dumps({'error': str(e)}))
sys.exit(1)
if __name__ == '__main__':
main()
+261
View File
@@ -0,0 +1,261 @@
"""Panel switcher. Runs on the Frame, either piped over SSH or installed with
--open for its loopback-only headset page. Uses Valve's shipped vrcmd and
Chromium, not an overlay app. Spatial layout limitations: docs/panels.md.
"""
import argparse
import hmac
import json
import os
from pathlib import Path
import re
import secrets
import signal
import subprocess
import sys
import time
from http.server import BaseHTTPRequestHandler, HTTPServer
VRCMD = '/opt/steamvr/bin/linuxarm64/vrcmd'
PANEL_ID = 2000999030
PANEL_KEY = 'valve.steam.desktopgame.' + str(PANEL_ID)
KEY = re.compile(r'[A-Za-z0-9_.:-]{1,200}\Z')
class PanelError(Exception):
pass
def run(args):
try:
p = subprocess.run(args, capture_output=True, text=True, timeout=10,
env={**os.environ, 'DISPLAY': ':0', 'LC_ALL': 'C.UTF-8'})
except (OSError, subprocess.TimeoutExpired) as e:
raise PanelError('The panel service did not answer: ' + str(e))
if p.returncode:
raise PanelError((p.stderr or p.stdout).strip()[-400:] or 'Panel command failed')
return p.stdout
def parse_overlays(text):
"""Only main dashboard panels, never their thumbnails, layers or cursors.
vrcmd output verified on SteamVR 2.18.1, BUILD_ID 20260925.6191901.
"""
if '---- OVERLAYS ----' not in text:
raise PanelError('SteamVR did not return its panel list. Is the headset awake?')
panels = []
for line in text.splitlines():
m = re.fullmatch(r"'([^']+)' -- '(.*)', (.*?) VROverlayType_Dashboard_Main\s*", line)
if m and KEY.fullmatch(m[1]):
panels.append({'key': m[1], 'title': 'Panel switcher' if m[1] == PANEL_KEY else m[2] or m[1],
'visible': 'not_visible' not in m[3]})
return panels
def state():
return {'panels': parse_overlays(run([VRCMD, '--overlays']))}
def focus(key):
if not isinstance(key, str) or not KEY.fullmatch(key):
raise PanelError('Choose an open panel.')
if key not in {p['key'] for p in state()['panels']}:
raise PanelError('That panel has closed. Refresh the list.')
run([VRCMD, '--showdashboard', key])
# vrcmd acknowledges dispatch, not final focus (a game or the user can
# switch again). Do not report a focus success without observing it.
return {'requested': key, 'message': 'Asked SteamVR to show the panel.'}
PAGE = '''<!doctype html><html lang="en"><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1"><title>Panel switcher [fc-panels]</title>
<style>body{background:#101b27;color:#eee;font:24px system-ui;margin:36px;max-width:1000px}
h1{font-size:36px}button{font:inherit;padding:16px 24px;border:1px solid #546574;border-radius:10px;
background:#23384b;color:white;cursor:pointer}button:focus-visible{outline:4px solid #66c0f4}
#panels{display:grid;gap:14px;margin:24px 0}#panels button{text-align:left}p{color:#bac8d5}</style>
<h1>Panel switcher</h1><p>Choose a panel to show it. Open this panel again from Steam's dashboard.</p>
<button id="refresh">Refresh</button> <button id="close">Close switcher</button>
<p id="status" role="status"></p><div id="panels"></div>
<script>
const token=location.hash.slice(1)||sessionStorage.getItem('panelKey')||'';
if(token)sessionStorage.setItem('panelKey',token);history.replaceState(null,'',location.pathname);
async function api(path,body){const r=await fetch(path,{method:body?'POST':'GET',
headers:{'X-Panel-Key':token,'Content-Type':'application/json'},body:body?JSON.stringify(body):undefined});
const s=await r.json();if(!r.ok)throw Error(s.error||'Panel request failed');return s;}
const status=document.getElementById('status');
async function refresh(){try{const s=await api('/panels');const list=document.getElementById('panels');list.replaceChildren();
for(const p of s.panels){const b=document.createElement('button');b.textContent=p.title;
b.onclick=async()=>{b.disabled=true;try{const r=await api('/focus',{key:p.key});status.textContent=r.message;}
catch(e){status.textContent=e.message;}finally{b.disabled=false;}};list.append(b);}
status.textContent=s.panels.length?'':'No open panels.';}catch(e){status.textContent=e.message;}}
document.getElementById('refresh').onclick=refresh;
document.getElementById('close').onclick=async()=>{try{await api('/close',{});window.close();}catch(e){status.textContent=e.message;}};
refresh();
</script></html>'''
class Handler(BaseHTTPRequestHandler):
def setup(self):
super().setup()
self.connection.settimeout(5)
def log_message(self, *args):
pass # never log the page's access key
def reply(self, code, value, html=False):
data = value.encode() if html else json.dumps(value).encode()
self.send_response(code)
self.send_header('Content-Type', 'text/html; charset=utf-8' if html else 'application/json')
self.send_header('Content-Length', str(len(data)))
self.send_header('Cache-Control', 'no-store')
self.send_header('X-Content-Type-Options', 'nosniff')
self.send_header('Referrer-Policy', 'no-referrer')
self.send_header('Content-Security-Policy', "default-src 'none'; script-src 'unsafe-inline'; style-src 'unsafe-inline'; connect-src 'self'; frame-ancestors 'none'")
self.end_headers()
self.wfile.write(data)
def allowed(self):
host = '127.0.0.1:' + str(self.server.server_port)
origin = self.headers.get('Origin')
return (self.headers.get('Host') == host and
(origin is None or origin == 'http://' + host) and
hmac.compare_digest(self.headers.get('X-Panel-Key', '').encode(), self.server.key.encode()))
def do_GET(self):
if self.path == '/':
return self.reply(200, PAGE, html=True) # no data or access key in the page
if not self.allowed():
return self.reply(403, {'error': 'Open the switcher from Frame Control.'})
try:
if self.path == '/panels':
return self.reply(200, state())
self.reply(404, {'error': 'Not found'})
except PanelError as e:
self.reply(502, {'error': str(e)})
def do_POST(self):
if not self.allowed():
return self.reply(403, {'error': 'Forbidden'})
try:
size = int(self.headers.get('Content-Length', '0'))
if not 0 < size <= 1024:
raise ValueError('Invalid request size')
body = json.loads(self.rfile.read(size))
if not isinstance(body, dict):
raise ValueError('Expected an object')
if self.path == '/focus':
return self.reply(200, focus(body.get('key')))
if self.path == '/close':
self.server.closing = True
return self.reply(200, {'closed': True})
self.reply(404, {'error': 'Not found'})
except (ValueError, PanelError) as e:
self.reply(400, {'error': str(e)})
def serve():
"""Own only our Chromium profile and process group. No changes to Steam,
SteamVR, other Chromium sessions, or global power settings.
"""
import fcntl # only on the Frame; module/tests also import on Windows
folder = Path.home() / '.local/share/frame-control/panels'
folder.mkdir(parents=True, exist_ok=True, mode=0o700)
with (folder / 'lock').open('w') as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
print(json.dumps(focus(PANEL_KEY)), flush=True)
return
chrome = Path.home() / 'chromium-xr/chrome'
if chrome.is_file():
command = [str(chrome)]
profile = folder / 'chromium'
elif Path('/usr/bin/chromium').is_file():
command = ['/usr/bin/chromium']
profile = folder / 'chromium'
elif subprocess.run(['flatpak', 'info', 'org.chromium.Chromium'],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=10).returncode == 0:
command = ['flatpak', 'run', 'org.chromium.Chromium']
profile = Path.home() / '.var/app/org.chromium.Chromium/data/frame-panel-switcher'
else:
raise PanelError('The headset switcher needs Chromium. The companion switcher still works.')
server = HTTPServer(('127.0.0.1', 0), Handler)
server.key = secrets.token_urlsafe(32)
server.closing = False
server.timeout = .5
url = 'http://127.0.0.1:%d/#%s' % (server.server_port, server.key)
env = {**os.environ, 'DISPLAY': ':0'}
env.pop('WAYLAND_DISPLAY', None)
browser = subprocess.Popen([*command, '--ozone-platform=x11',
'--user-data-dir=' + str(profile),
'--no-first-run', '--no-default-browser-check',
'--password-store=basic', '--window-size=1200,800', '--app=' + url],
env=env, start_new_session=True, stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
def stop(signum, frame):
server.closing = True
signal.signal(signal.SIGTERM, stop)
win = None
try:
deadline = time.monotonic() + 30
while not win and time.monotonic() < deadline and browser.poll() is None:
server.handle_request() # Chromium must fetch the page before it has a title
for line in run(['xwininfo', '-root', '-children']).splitlines():
m = re.match(r'\s*(0x[0-9a-fA-F]+) .*\[fc-panels\]', line)
if m:
win = m[1]
break
if not win:
raise PanelError('The switcher window did not appear within 30 seconds.')
run(['xprop', '-id', win, '-f', 'STEAM_GAME', '32c', '-set', 'STEAM_GAME', str(PANEL_ID)])
print(json.dumps({'message': 'Opened the panel switcher in the headset.'}), flush=True)
# Caller reads exactly one line, then disconnects; no more stdout.
while not server.closing and browser.poll() is None:
server.handle_request()
# Closing the last app window need not exit Chromium.
if win not in run(['xwininfo', '-root', '-children']):
break
finally:
server.server_close()
if browser.poll() is None:
os.killpg(browser.pid, signal.SIGTERM)
try:
browser.wait(timeout=5)
except subprocess.TimeoutExpired:
os.killpg(browser.pid, signal.SIGKILL)
browser.wait()
def open_switcher():
# This command runs from an installed path, never from the SSH stdin copy.
proc = subprocess.Popen([sys.executable, str(Path(__file__).resolve()), '--serve'],
stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
text=True, start_new_session=True)
line = proc.stdout.readline()
proc.stdout.close()
if not line:
raise PanelError('The headset switcher could not start.')
result = json.loads(line)
if 'error' in result:
raise PanelError(result['error'])
return result
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--focus')
parser.add_argument('--open', action='store_true')
parser.add_argument('--serve', action='store_true')
args = parser.parse_args()
try:
if args.serve:
serve()
else:
print(json.dumps(open_switcher() if args.open else focus(args.focus) if args.focus else state()))
except (PanelError, OSError) as e:
print(json.dumps({'error': str(e)}), flush=True)
return 1
return 0
if __name__ == '__main__':
sys.exit(main())
+83
View File
@@ -0,0 +1,83 @@
"""Small, bounded CPU Gaussian-splat preview renderer (Frame Control-owned).
Reads the common 32-byte .splat record: position/scale float32 triplets,
RGBA bytes, then normalized quaternion bytes (wxyz). Two perspective cameras,
projected 3D covariance, back-to-front alpha compositing. This is a stationary
stereo preview, not a six-degree-of-freedom scene or a large-scene renderer.
"""
import math
from pathlib import Path
import struct
MAX_SPLATS = 20000
RECORD = struct.Struct('<6f8B')
def read(path):
size = Path(path).stat().st_size
if not size or size % RECORD.size or size > MAX_SPLATS * RECORD.size:
raise ValueError('Use a 32-byte .splat file with 1–20,000 Gaussians; PLY/SPZ and larger scenes are not supported yet')
values = []
with open(path, 'rb') as stream:
for row in RECORD.iter_unpack(stream.read(MAX_SPLATS * RECORD.size + 1)):
xyz, scales = row[:3], row[3:6]
if not all(math.isfinite(v) and abs(v) <= 1e6 for v in row[:6]) or min(scales) <= 0:
raise ValueError('Invalid splat position or scale')
q = [(v - 128) / 128 for v in row[10:14]]
length = math.sqrt(sum(v*v for v in q))
if length < .01:
raise ValueError('Invalid splat quaternion')
w, x, y, z = [v / length for v in q]
rotation = ((1-2*(y*y+z*z), 2*(x*y-z*w), 2*(x*z+y*w)),
(2*(x*y+z*w), 1-2*(x*x+z*z), 2*(y*z-x*w)),
(2*(x*z-y*w), 2*(y*z+x*w), 1-2*(x*x+y*y)))
cov = [[sum(rotation[i][k]*rotation[j][k]*scales[k]**2 for k in range(3))
for j in range(3)] for i in range(3)]
values.append((xyz, cov, row[6:10]))
return values
def render(path, width=320, height=240):
values = read(path)
lo = [min(p[0][i] for p in values) for i in range(3)]
hi = [max(p[0][i] for p in values) for i in range(3)]
center = [(a+b)/2 for a, b in zip(lo, hi)]
radius = max(max(b-a for a, b in zip(lo, hi))/2, .01)
# Normalize captures to a two-metre box. Source units are not assumed metres.
normalized = [([(xyz[i]-center[i])/radius for i in range(3)],
[[v/radius**2 for v in row] for row in cov], color)
for xyz, cov, color in values]
normalized.sort(key=lambda p: p[0][2]) # camera is at z=3; farthest first
focal = width * .8
eyes = []
for eye in (-.032, .032):
pixels = bytearray(b'\x00\x00\x00\xff' * (width*height))
for (x, y, z), cov, color in normalized:
x -= eye
depth = 3-z
px, py = width/2+focal*x/depth, height/2-focal*y/depth
jac = ((focal/depth, 0, focal*x/depth**2),
(0, -focal/depth, -focal*y/depth**2))
screen = [[sum(jac[i][a]*cov[a][b]*jac[j][b] for a in range(3) for b in range(3))
for j in range(2)] for i in range(2)]
a, b, c = screen[0][0]+.3, screen[0][1], screen[1][1]+.3
det = a*c-b*b
if det <= 0 or not math.isfinite(det):
raise ValueError('Splat covariance is not renderable')
# A footprint cap bounds work on malformed or oversized Gaussians.
rx, ry = min(32, math.ceil(3*math.sqrt(a))), min(32, math.ceil(3*math.sqrt(c)))
for sy in range(max(0, int(py)-ry), min(height, int(py)+ry+1)):
dy = sy+.5-py
for sx in range(max(0, int(px)-rx), min(width, int(px)+rx+1)):
dx = sx+.5-px
power = (c*dx*dx-2*b*dx*dy+a*dy*dy)/det
if power > 9:
continue
alpha = color[3]/255 * math.exp(-.5*power)
offset = (sy*width+sx)*4
for k in range(3):
pixels[offset+k] = round(color[k]*alpha+pixels[offset+k]*(1-alpha))
eyes.append(pixels)
stride = width*4
return b''.join(eyes[0][y*stride:(y+1)*stride]+eyes[1][y*stride:(y+1)*stride]
for y in range(height)), width*2, height
+157 -25
View File
@@ -1,9 +1,12 @@
"""Runs ON the Steam Frame (piped over SSH as `python3 -`); prints one JSON object. """Runs ON the Steam Frame (piped over SSH as `python3 -`); prints one JSON object.
Read-only. Every probe is best effort: a missing tool or file gives null, not an Read-only. Every probe is best effort: a missing tool or file gives null, not an
error. Paths verified on SteamOS 0.3.0 (vr), build 20260922. error. Sensor paths verified on SteamOS 0.3.0; OpenVR timing on 0.4.1
(build 20260925.6191901). See docs/vr-utilities.md.
""" """
import ctypes as C
import glob import glob
import math
import json import json
import os import os
import re import re
@@ -156,27 +159,156 @@ def flatpaks():
return out return out
uptime = read("/proc/uptime") def thermal_alerts():
procs = process_names() """Use the kernel's per-zone hot/critical trips, never a guessed chip limit."""
print(json.dumps({ alerts, known = [], False
"time": time.time(), for z in glob.glob("/sys/class/thermal/thermal_zone*"):
"hostname": socket.gethostname(), t = num(z + "/temp", 0.001)
"os": os_release(), for trip in glob.glob(z + "/trip_point_*_type"):
"uptime": float(uptime.split()[0]) if uptime else None, if read(trip) not in ("hot", "critical"):
"battery": battery(), continue
"power": power_source(), limit = num(trip[:-4] + "temp", 0.001)
"disk": {"root": disk("/"), "home": disk("/home")}, if t is not None and limit is not None and limit > 0:
"memory": memory(), known = True
"temp": max_temp(), if t >= limit:
"wifi": wifi(), alerts.append({"zone": read(z + "/type"), "tempC": t, "limitC": limit})
"ip": ip_addr(), return alerts if known else None
"volume": volume(),
"services": {
"steamvr": "vrserver" in procs, def activity_level():
"desktop": "plasmashell" in procs, try:
"lepton": port_listening(5555), rows = json.loads(run("/opt/steamvr/bin/linuxarm64/vrcmd", "--stats"))
"rdp": "xrdp" in procs, if not isinstance(rows, list):
}, return None
"games": games(), return next((r.get("activity_level") for r in rows
"flatpaks": flatpaks(), if isinstance(r, dict) and r.get("operation") == "status"), None)
})) except (ValueError, TypeError):
return None
# OpenVR C ABI, ValveSoftware/openvr headers/openvr_capi.h (IVRCompositor_029).
# Exact interface version: never guess an index against a different table.
class Pose(C.Structure):
_fields_ = [('matrix', C.c_float * 12), ('velocity', C.c_float * 3),
('angular', C.c_float * 3), ('result', C.c_int),
('valid', C.c_bool), ('connected', C.c_bool)]
class Timing(C.Structure):
_fields_ = [(n, C.c_uint32) for n in ('size', 'index', 'presents', 'mis', 'dropped', 'flags')] + [
('time', C.c_double)] + [(n, C.c_float) for n in (
'gpuPre', 'gpuPost', 'gpu', 'compositorGpu', 'compositorCpu', 'idleCpu', 'interval',
'presentCpu', 'waitCpu', 'submit', 'posesCalled', 'posesReady', 'frameReady',
'updateStart', 'updateEnd', 'renderStart')] + [
('pose', Pose), ('ready', C.c_uint32), ('first', C.c_uint32), ('transfer', C.c_float)]
class OpenVR:
def __enter__(self):
self.lib = C.CDLL('/opt/steamvr/bin/linuxarm64/libopenvr_api.so')
self.lib.VR_InitInternal2.argtypes = [C.POINTER(C.c_int), C.c_int, C.c_char_p]
self.lib.VR_GetGenericInterface.argtypes = [C.c_char_p, C.POINTER(C.c_int)]
self.lib.VR_GetGenericInterface.restype = C.c_void_p
err = C.c_int()
self.lib.VR_InitInternal2(C.byref(err), 3, None) # background: never start or keep SteamVR running
if err.value:
raise RuntimeError(f'SteamVR unavailable (init {err.value})')
return self
def __exit__(self, *args):
self.lib.VR_ShutdownInternal()
def function(self, interface, index, result, *args):
err = C.c_int()
ptr = self.lib.VR_GetGenericInterface(('FnTable:' + interface).encode(), C.byref(err))
if err.value or not ptr:
raise RuntimeError(f'{interface} unavailable ({err.value})')
return C.CFUNCTYPE(result, *args)(C.cast(ptr, C.POINTER(C.c_void_p))[index])
def cpu_ticks():
line = (read('/proc/stat') or '').splitlines()
if not line or not line[0].startswith('cpu '):
return None
try:
# guest/guest_nice are already included in user/nice.
ticks = [int(x) for x in line[0].split()[1:9]]
return sum(ticks), ticks[3] + ticks[4]
except (ValueError, IndexError):
return None
def cpu_percent(before, after):
if before is None or after is None or after[0] <= before[0]:
return None
return round(max(0, min(100, 100 * (1 - (after[1] - before[1]) / (after[0] - before[0])))), 1)
def positive(value):
return round(value, 2) if math.isfinite(value) and value > 0 else None
def timing_values(before, after):
dt, frames = after.time - before.time, after.index - before.index
if dt <= 0 or frames <= 0 or frames / dt > 1000:
return {} # standby or old data; never present stale timings as live
return {'compositorFps': positive(frames / dt),
'frameMs': positive(1000 * dt / frames),
'appFps': positive(1000 / after.interval) if after.interval > 0 else None,
'gpuMs': positive(after.gpu), 'compositorCpuMs': positive(after.compositorCpu)}
def performance():
out = {'compositorFps': None, 'frameMs': None, 'appFps': None,
'gpuMs': None, 'compositorCpuMs': None, 'cpuPercent': None,
'gpuMHz': num('/sys/class/devfreq/3d00000.gpu/cur_freq', 1e-6)}
before = cpu_ticks()
try:
with OpenVR() as vr:
get = vr.function('IVRCompositor_029', 10, C.c_bool, C.POINTER(Timing), C.c_uint32)
a, b = Timing(), Timing()
a.size = b.size = C.sizeof(Timing)
first = get(C.byref(a), 0)
time.sleep(0.2)
if first and get(C.byref(b), 0):
out.update(timing_values(a, b))
except (OSError, RuntimeError, AttributeError): # AttributeError: a SteamVR build without these exports
time.sleep(0.2)
out['cpuPercent'] = cpu_percent(before, cpu_ticks())
return out
def status():
uptime = read("/proc/uptime")
procs = process_names()
return {
"time": time.time(),
"performance": performance(),
"hostname": socket.gethostname(),
"os": os_release(),
"uptime": float(uptime.split()[0]) if uptime else None,
"battery": battery(),
"power": power_source(),
"disk": {"root": disk("/"), "home": disk("/home")},
"memory": memory(),
"temp": max_temp(),
"wifi": wifi(),
"ip": ip_addr(),
"volume": volume(),
"services": {
"steamvr": "vrserver" in procs,
"desktop": "plasmashell" in procs,
"lepton": port_listening(5555),
"rdp": "xrdp" in procs,
},
"games": games(),
"flatpaks": flatpaks(),
}
def main():
print(json.dumps(status()))
if __name__ == "__main__":
main()
+21
View File
@@ -141,6 +141,21 @@ OWNED_JS = r"""
})() })()
""" """
# Software has app_type 2, so utility ownership must not use OWNED_JS's games filter.
# Steam prices checked 2026-09-28: OVR Advanced Settings is paid on Steam too.
UTILITY_IDS = (1009850, 1173510, 1068820, 908520, 1494460)
FREE_UTILITIES = (1494460,)
UTILITIES_JS = """(() => {
const apps = appStore.allApps;
if (!apps || !apps.length) throw new Error("Steam library is not loaded; ownership is unknown");
return [1009850,1173510,1068820,908520,1494460].map(id => {
const a = apps.find(a => a.appid === id);
return {id, owned: !!a, installed: !!a?.local_per_client_data?.installed,
frame: a ? (a.steam_hw_compat_category_packed >> 8) & 3 : 0};
});
})()"""
WIZARD_JS = """SteamClient.Installs.GetInstallManagerInfo().then(i => ({ WIZARD_JS = """SteamClient.Installs.GetInstallManagerInfo().then(i => ({
state: i?.eInstallState ?? 0, app: i?.currentAppID ?? 0, need: i?.nDiskSpaceRequired || 0, state: i?.eInstallState ?? 0, app: i?.currentAppID ?? 0, need: i?.nDiskSpaceRequired || 0,
free: i?.nDiskSpaceAvailable || 0, error: i?.eAppError, detail: i?.errorDetail }))""" free: i?.nDiskSpaceAvailable || 0, error: i?.eAppError, detail: i?.errorDetail }))"""
@@ -158,6 +173,10 @@ def owned():
def install(appid): def install(appid):
page = Page() page = Page()
if appid in UTILITY_IDS and appid not in FREE_UTILITIES:
rows = page.eval(UTILITIES_JS)
if not any(r['id'] == appid and r['owned'] for r in rows):
raise Fail("This paid utility is not owned by the Frame account. No install or store action was taken.")
app = page.eval(f"(a => a && {{name: a.display_name, installed: !!a.local_per_client_data?.installed}})" app = page.eval(f"(a => a && {{name: a.display_name, installed: !!a.local_per_client_data?.installed}})"
f"(appStore.GetAppOverviewByAppID({appid}))") f"(appStore.GetAppOverviewByAppID({appid}))")
if app and app["installed"]: if app and app["installed"]:
@@ -216,6 +235,8 @@ def main():
cmd = sys.argv[1] if len(sys.argv) > 1 else "" cmd = sys.argv[1] if len(sys.argv) > 1 else ""
if cmd == "owned": if cmd == "owned":
out = owned() out = owned()
elif cmd == "utilities":
out = {"utilities": Page().eval(UTILITIES_JS)}
elif cmd in ("install", "store") and len(sys.argv) == 3 and sys.argv[2].isdigit(): elif cmd in ("install", "store") and len(sys.argv) == 3 and sys.argv[2].isdigit():
out = (install if cmd == "install" else store)(int(sys.argv[2])) out = (install if cmd == "install" else store)(int(sys.argv[2]))
else: else:
+446
View File
@@ -0,0 +1,446 @@
"""Touch and direct input for the Steam Frame's panels. Frame Control's server runs this ON the Frame.
gamescope, the Frame's compositor, serves Valve's own input injection: an EIS
socket (libei's server side), which Steam uses to feed it Remote Play input.
This connects to it with libei, which is on the SteamOS image, and points,
clicks, scrolls and types into the panel that has focus in the headset: the
one the wearer last used. No install, and it reaches every panel, on either
of gamescope's X displays (see docs/streaming.md).
python3 frame_touch.py focus print the focused panel as JSON
python3 frame_touch.py panels print every app panel as JSON, and which has focus
python3 frame_touch.py read events on stdin, one JSON object (or list) per line:
{"fx": 0.5, "fy": 0.2, "window": 123, "display": ":1"}
pointer to that fraction of that panel; any event can
name its panel, and goes nowhere if another has focus
{"dx": 4, "dy": -2} pointer by that much
{"button": "left", "down": true} left, right or middle; "down" false releases
{"scroll": [0, 120]} by pixels; positive y scrolls down
{"key": 30, "down": true} a Linux (evdev) key code, as the page maps KeyboardEvent.code
{"text": "hello"} printable ASCII, typed on a US layout
Status goes to stdout, one JSON object per line: {"state": "ready" | "error", ...}.
Standard library only (ctypes for libei), like the rest of what runs on the Frame.
"""
import ctypes
import json
import os
import select
import subprocess
import sys
import time
SOCKET = "/run/user/{uid}/gamescope-0-ei" # filled in on the Frame (Windows has no getuid; the tests import this)
BUTTONS = {"left": 0x110, "right": 0x111, "middle": 0x112} # BTN_LEFT, BTN_RIGHT, BTN_MIDDLE
SHIFT = 42 # KEY_LEFTSHIFT
# Printable ASCII on a US layout: character -> (evdev key code, shifted).
ROWS = [("1234567890-=", "!@#$%^&*()_+", 2), ("qwertyuiop[]", "QWERTYUIOP{}", 16),
("asdfghjkl;'`", 'ASDFGHJKL:"~', 30), ("\\zxcvbnm,./", "|ZXCVBNM<>?", 43)]
ASCII = {" ": (57, False), "\n": (28, False), "\t": (15, False)}
for plain, shifted, first in ROWS:
for i, (a, b) in enumerate(zip(plain, shifted)):
ASCII[a], ASCII[b] = (first + i, False), (first + i, True)
# libei's event types and device capabilities (libei.h, libei 1.4).
EV_CONNECT, EV_DISCONNECT, EV_SEAT_ADDED, EV_DEVICE_ADDED, EV_DEVICE_REMOVED = 1, 2, 3, 5, 6
EV_DEVICE_PAUSED, EV_DEVICE_RESUMED = 7, 8
CAP_POINTER, CAP_ABSOLUTE, CAP_KEYBOARD, CAP_SCROLL, CAP_BUTTON = 1, 2, 4, 16, 32
def say(state, **more):
print(json.dumps({"state": state, **more}), flush=True)
# ---- which panel has focus -----------------------------------------------------
def xprop_root(display, name):
try:
out = subprocess.run(["xprop", "-root", name], env=dict(os.environ, DISPLAY=display),
capture_output=True, text=True, timeout=5).stdout
except (OSError, subprocess.SubprocessError):
return []
values = out.split("=", 1)[1] if "=" in out else ""
return [int(v) for v in values.replace(",", " ").split() if v.isdigit()]
def window_info(display, window):
"""Name and geometry of a window on one X display, or None if it isn't there."""
try:
which = ["-root"] if window == "root" else ["-id", str(window)]
out = subprocess.run(["xwininfo", *which], env=dict(os.environ, DISPLAY=display),
capture_output=True, text=True, timeout=5).stdout
except (OSError, subprocess.SubprocessError):
return None
if "IsViewable" not in out:
return None
info = {}
for line in out.splitlines():
line = line.strip()
if line.startswith("xwininfo: Window id:"):
info["name"] = line.split('"', 1)[1].rsplit('"', 1)[0] if '"' in line else ""
for key, field in (("Absolute upper-left X:", "x"), ("Absolute upper-left Y:", "y"),
("Width:", "width"), ("Height:", "height")):
if line.startswith(key):
info[field] = int(line.split(":", 1)[1])
return info if "width" in info else None
def displays():
return sorted(f":{n[1:]}" for n in os.listdir("/tmp/.X11-unix") if n[1:].isdigit())
def window_pid(display, window):
try:
out = subprocess.run(["xprop", "-id", str(window), "_NET_WM_PID"], env=dict(os.environ, DISPLAY=display),
capture_output=True, text=True, timeout=5).stdout
except (OSError, subprocess.SubprocessError):
return None
value = out.rsplit("=", 1)[-1].strip() if "=" in out else ""
return int(value) if value.isdigit() else None
def locate(window, pid):
"""The display a focusable window is on, with its name and geometry.
Window ids are per X server, so :0 and :1 can both have one; the pid gamescope
lists with it (GAMESCOPE_FOCUSABLE_WINDOWS) tells them apart.
"""
found = []
for display in displays():
info = window_info(display, window)
if info:
found.append((display, info))
if len(found) > 1 and pid:
found = [f for f in found if window_pid(f[0], window) == pid] or found
if not found:
return None
display, info = found[0]
root = window_info(display, "root") or {}
return {"window": window, "display": display, **info,
"root": [root.get("width", info["width"]), root.get("height", info["height"])]}
def focusable():
"""gamescope's focusable windows as (window, app id, pid)."""
t = xprop_root(":0", "GAMESCOPE_FOCUSABLE_WINDOWS")
return [tuple(t[i:i + 3]) for i in range(0, len(t) - 2, 3)]
def focus_display():
"""The display of the focused window, from GAMESCOPE_FOCUS_DISPLAY on :0's root.
gamescope writes the name (":1") as 32-bit items, so its first four bytes land,
little-endian, in the first value: 12602 is 0x313A, ":1" (steamcompmgr.cpp;
seen 2026-09-29).
"""
values = xprop_root(":0", "GAMESCOPE_FOCUS_DISPLAY")
if not values:
return None
name = (values[0] & 0xFFFFFFFF).to_bytes(4, "little").split(b"\0", 1)[0].decode("ascii", "replace")
return name if name[:1] == ":" and name[1:].isdigit() else None
def focus_now():
"""Just which window and display have focus: two property reads, for checking a press."""
window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
return (window or None, focus_display() if window else None)
def focus():
"""The panel that has focus in the headset: window, display, name and sizes (gamescope
publishes the window and its display on :0's root)."""
window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
if not window:
return {"window": None}
app, pid = next(((a, p) for w, a, p in focusable() if w == window), (None, None))
display = focus_display()
info = window_info(display, window) if display else None
if info:
root = window_info(display, "root") or {}
panel = {"window": window, "display": display, **info,
"root": [root.get("width", info["width"]), root.get("height", info["height"])]}
else:
panel = locate(window, pid) # no display published: tell them apart by pid
return {**panel, "app": app} if panel else {"window": None}
def panels():
"""Every app panel (gamescope's focusable windows), for watching one that hasn't focus."""
now = focus()
found = []
for window, app, pid in focusable():
panel = locate(window, pid)
if panel and panel["width"] > 1 and panel["height"] > 1 and \
not any(f["window"] == window and f["display"] == panel["display"] for f in found):
panel.pop("root", None)
found.append({**panel, "app": app, "focused": (window, panel["display"]) ==
(now.get("window"), now.get("display"))})
return {"focus": now.get("window"), "focus_display": now.get("display"), "panels": found}
def to_root(panel, fx, fy):
"""A point given as a fraction of the panel, in the root coordinates gamescope's pointer uses.
gamescope fits each panel's window to its display, so a 1920x1080 window on a
1280x720 display takes pointer positions at two thirds scale (verified 2026-09-29).
"""
rw, rh = panel["root"]
w, h = panel["width"], panel["height"]
s = min(rw / w, rh / h)
ox, oy = (rw - w * s) / 2, (rh - h * s) / 2
fx, fy = min(max(fx, 0.0), 1.0), min(max(fy, 0.0), 1.0)
return ox + fx * (w * s - 1), oy + fy * (h * s - 1)
# ---- gamescope's input socket ----------------------------------------------------
def libei():
L = ctypes.CDLL("libei.so.1")
vp, c = ctypes.c_void_p, ctypes
sig = {
"ei_new_sender": (vp, [vp]), "ei_configure_name": (None, [vp, c.c_char_p]),
"ei_setup_backend_socket": (c.c_int, [vp, c.c_char_p]), "ei_get_fd": (c.c_int, [vp]),
"ei_dispatch": (None, [vp]), "ei_get_event": (vp, [vp]), "ei_event_get_type": (c.c_int, [vp]),
"ei_event_unref": (vp, [vp]), "ei_event_get_seat": (vp, [vp]), "ei_event_get_device": (vp, [vp]),
"ei_device_has_capability": (c.c_bool, [vp, c.c_int]), "ei_now": (c.c_uint64, [vp]),
"ei_device_start_emulating": (None, [vp, c.c_uint32]), "ei_device_stop_emulating": (None, [vp]),
"ei_device_frame": (None, [vp, c.c_uint64]),
"ei_device_pointer_motion": (None, [vp, c.c_double, c.c_double]),
"ei_device_pointer_motion_absolute": (None, [vp, c.c_double, c.c_double]),
"ei_device_button_button": (None, [vp, c.c_uint32, c.c_bool]),
"ei_device_scroll_delta": (None, [vp, c.c_double, c.c_double]),
"ei_device_keyboard_key": (None, [vp, c.c_uint32, c.c_bool]),
"ei_unref": (vp, [vp]),
}
for name, (res, args) in sig.items():
f = getattr(L, name)
f.restype, f.argtypes = res, args
return L
class Gamescope:
"""One connection to gamescope's EIS socket and its virtual input device."""
def __init__(self):
self.L = L = libei()
self.ei = L.ei_new_sender(None)
L.ei_configure_name(self.ei, b"Frame Control")
if L.ei_setup_backend_socket(self.ei, SOCKET.format(uid=os.getuid()).encode()) != 0:
raise RuntimeError("Couldn't reach gamescope's input socket. Is the headset on?")
self.fd = L.ei_get_fd(self.ei)
self.device, self.sequence, self.held, self.keys, self.alive = None, 0, set(), set(), True
def pump(self, wait=0.0):
"""Handle gamescope's events; False once it has disconnected."""
select.select([self.fd], [], [], wait)
self.L.ei_dispatch(self.ei)
alive = True
while True:
ev = self.L.ei_get_event(self.ei)
if not ev:
return alive
kind = self.L.ei_event_get_type(ev)
if kind == EV_SEAT_ADDED:
seat = self.L.ei_event_get_seat(ev)
# Variadic, ending in 0 (NULL): ask for everything we send.
self.L.ei_seat_bind_capabilities(ctypes.c_void_p(seat), *map(ctypes.c_int, (
CAP_POINTER, CAP_ABSOLUTE, CAP_BUTTON, CAP_SCROLL, CAP_KEYBOARD, 0)))
elif kind == EV_DEVICE_RESUMED:
device = self.L.ei_event_get_device(ev)
if self.L.ei_device_has_capability(device, CAP_ABSOLUTE):
self.sequence += 1
self.L.ei_device_start_emulating(device, self.sequence)
self.device = device
# Releases that arrived while it was paused were dropped: let go of
# everything now, so the headset and this agent agree nothing is held.
if self.held or self.keys:
self.release_all()
elif kind in (EV_DEVICE_PAUSED, EV_DEVICE_REMOVED):
if self.L.ei_event_get_device(ev) == self.device:
self.device = None
elif kind == EV_DISCONNECT:
self.device, alive, self.alive = None, False, False
self.L.ei_event_unref(ev)
def wait_ready(self, timeout=5):
end = time.time() + timeout
while self.device is None and time.time() < end:
if not self.pump(0.1):
break
if self.device is None:
raise RuntimeError("gamescope closed its input socket" if not self.alive
else "gamescope didn't offer an input device")
def frame(self):
self.L.ei_device_frame(self.device, self.L.ei_now(self.ei))
self.L.ei_dispatch(self.ei)
def move_to(self, x, y):
self.L.ei_device_pointer_motion_absolute(self.device, x, y)
self.frame()
def move_by(self, dx, dy):
self.L.ei_device_pointer_motion(self.device, dx, dy)
self.frame()
def button(self, name, down):
code = BUTTONS[name]
if down == (code in self.held):
return # already in that state
self.L.ei_device_button_button(self.device, code, down)
self.frame()
(self.held.add if down else self.held.discard)(code)
def scroll(self, dx, dy):
self.L.ei_device_scroll_delta(self.device, dx, dy)
self.frame()
def key(self, code, down):
self.L.ei_device_keyboard_key(self.device, code, down)
self.frame()
(self.keys.add if down else self.keys.discard)(code)
# Paced: a burst of keys can reach the app out of order (seen 2026-09-29).
time.sleep(0.008)
def text(self, text):
for ch in text:
if ch not in ASCII:
continue
code, shifted = ASCII[ch]
if shifted:
self.key(SHIFT, True)
self.key(code, True)
self.key(code, False)
if shifted:
self.key(SHIFT, False)
def release_all(self):
"""Let go of every button and key still down, so nothing stays held in the headset."""
for code in list(self.held):
name = next(n for n, c in BUTTONS.items() if c == code)
self.button(name, False)
for code in list(self.keys):
self.key(code, False)
# ---- events from the server --------------------------------------------------------
def events(line):
try:
data = json.loads(line)
except ValueError:
return []
return [e for e in (data if isinstance(data, list) else [data]) if isinstance(e, dict)]
def number(value, limit=100000.0):
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
raise ValueError("not a number")
return max(-limit, min(limit, float(value)))
STALE = [False] # whether the last status said a tap went nowhere
def aimed_elsewhere(event, panel):
"""Whether an event names a panel that isn't the one with focus now."""
if "window" not in event:
return False
return (panel.get("window"), panel.get("display")) != (event.get("window"), event.get("display"))
def apply(gs, event, panel):
"""Send one event; returns the focused panel it checked against (looked up at most once a second).
Positions and presses name the panel they were meant for. If focus has moved to
another panel since, they go nowhere, so a tap can't land on the wrong one;
releases always go, so nothing stays held.
"""
# Moves may use a focus reading up to a second old; anything that acts (a press, key,
# text or scroll) reads it afresh, so it can't land on a panel that took focus since.
acts = any(k in event for k in ("button", "key", "text", "scroll")) and event.get("down") is not False
if "window" in event:
if panel and acts and focus_now() == (panel.get("window"), panel.get("display")):
pass # still the same panel (its geometry is re-read on the usual one-second schedule)
elif acts or not panel or time.time() - panel.get("_at", 0) > 1 or aimed_elsewhere(event, panel):
panel = {**focus(), "_at": time.time()}
stale = aimed_elsewhere(event, panel) if "window" in event else False
if stale and not (event.get("down") is False and ("button" in event or "key" in event)):
say("ready", focus=panel.get("window"), display=panel.get("display"), stale=True) # the page re-syncs
STALE[0] = True
return panel
if STALE[0] and not stale:
# Anything that goes through (a trackpad move names no panel) means caught up: stop re-syncing.
STALE[0] = False
say("ready", focus=(panel or {}).get("window"), display=(panel or {}).get("display"))
if "fx" in event and panel and panel.get("window"):
gs.move_to(*to_root(panel, number(event["fx"], 1), number(event["fy"], 1)))
if "dx" in event or "dy" in event:
gs.move_by(number(event.get("dx", 0), 2000), number(event.get("dy", 0), 2000))
if event.get("button") in BUTTONS:
gs.button(event["button"], event.get("down") is not False)
if isinstance(event.get("scroll"), list) and len(event["scroll"]) == 2:
gs.scroll(number(event["scroll"][0], 5000), number(event["scroll"][1], 5000))
if isinstance(event.get("key"), int) and not isinstance(event["key"], bool) and 0 < event["key"] < 768:
gs.key(event["key"], event.get("down") is not False)
if isinstance(event.get("text"), str):
gs.text(event["text"][:500])
return panel
def main():
if sys.argv[1:] == ["focus"]:
print(json.dumps(focus()))
return 0
if sys.argv[1:] == ["panels"]:
print(json.dumps(panels()))
return 0
try:
gs = Gamescope()
gs.wait_ready()
except (OSError, RuntimeError) as e:
say("error", message=str(e))
return 1
say("ready", focus=focus().get("window"))
stdin, pending, panel = sys.stdin.fileno(), b"", None
try:
while True:
ready, _, _ = select.select([stdin, gs.fd], [], [], 30)
if gs.fd in ready and not gs.pump():
say("error", message="gamescope closed its input socket")
return 1
if stdin not in ready:
continue
chunk = os.read(stdin, 65536)
if not chunk:
return 0 # the server went away
*lines, pending = (pending + chunk).split(b"\n")
for line in lines:
waited = False
for event in events(line):
if gs.device is None and waited:
continue # still paused: don't wait again for each event of this batch
if gs.device is None:
waited = True
# Paused (gamescope can pause the device): wait a moment; drop this
# event if it doesn't come back. Only a disconnect ends the session.
try:
gs.wait_ready(2)
except RuntimeError:
if not gs.alive:
raise
continue
try:
panel = apply(gs, event, panel)
except (ValueError, KeyError, TypeError, OSError):
continue # the server checks events; skip anything odd
except RuntimeError as e:
say("error", message=str(e))
return 1
finally:
if gs.device is not None:
gs.release_all() # never leave a button held down in the headset
if __name__ == "__main__":
sys.exit(main())
+29
View File
@@ -0,0 +1,29 @@
"""Optional software, separate from Frame Control's own controls and HUD.
Public reports are attributed leads, never local verification. Compatibility
reports reuse the existing database with steam:<appid> package keys.
"""
REPORT = 'https://www.roadtovr.com/valve-steam-frame-review/'
UTILITIES = (
(1009850, 'OVR Advanced Settings', False, 'No verified Frame result. Steam edition is paid; the developer also publishes free source/releases.', 'https://github.com/OpenVR-Advanced-Settings/OpenVR-AdvancedSettings'),
(1173510, 'XSOverlay', False, 'Supplied research reports Proton support, but the linked review did not corroborate it on recheck. Untested.', REPORT),
(1068820, 'OVR Toolkit', False, 'Supplied research reports Proton support, but the linked review did not corroborate it on recheck. Untested.', REPORT),
(908520, 'fpsVR', False, 'No Frame-specific result established in the supplied public research. Untested here.', 'https://store.steampowered.com/app/908520/'),
(1494460, 'Desktop+', True, 'Free, developer-published software. No verified Frame result.', 'https://github.com/elvissteinjr/DesktopPlus'),
)
def catalogue(ownership, reports):
owned = {r['id']: r for r in ownership}
out = []
for appid, name, free, note, source in UTILITIES:
local = owned.get(appid, {})
matches = [r for r in reports if r.get('package') == f'steam:{appid}' and r.get('rating') in ('works', 'issues', 'broken')]
latest = max(matches, key=lambda r: r.get('date') or '') if matches else None
out.append({'id': appid, 'name': name, 'free': free, 'owned': local.get('owned'),
'installed': local.get('installed', False), 'frame': local.get('frame', 0),
'status': latest['rating'] if latest else 'untested',
'report': {k: latest.get(k) for k in ('notes', 'date', 'steamos', 'source')} if latest else None,
'note': note, 'source': source,
'canInstall': bool(free or local.get('owned'))})
return {'utilities': out}
+126
View File
@@ -0,0 +1,126 @@
"""Frame Control's optional performance HUD, using Frame platform tools only.
Controls remain blocked pending idle-headset verification; no playspace writes
are exposed. See docs/vr-utilities.md for the probe evidence and follow-up.
"""
import json
import os
from pathlib import Path
import re
import signal
import subprocess
import sys
import time
from frame_status import battery, max_temp, performance
ROOT = Path.home() / '.local/share/frame-control/vr'
APPID = '2000250025'
def validate(body):
action = body.get('action')
if action not in ('hud-start', 'hud-stop'):
raise ValueError('VR action must be hud-start or hud-stop; playspace controls are not yet verified')
return action
def save(path, data):
tmp = path.with_suffix('.tmp')
tmp.write_text(json.dumps(data))
os.replace(tmp, path)
def process_identity(pid):
try:
stat = Path(f'/proc/{pid}/stat').read_text().rsplit(')', 1)[1].split()
args = Path(f'/proc/{pid}/cmdline').read_bytes().split(b'\0')
if b'frame-control-hud' in args and b'xterm' in Path(f'/proc/{pid}/comm').read_bytes():
return stat[19] # field 22, starttime; protects against PID reuse
except OSError:
pass
return None
def panel(action):
path = ROOT / 'hud.json'
saved = json.loads(path.read_text()) if path.exists() else {}
pid = saved.get('pid')
running = bool(pid and saved.get('start') and process_identity(pid) == saved['start'])
if action == 'hud-stop':
if running:
os.kill(pid, signal.SIGTERM) # xterm closes the PTY; child exits on HUP
path.unlink(missing_ok=True)
return {'message': 'HUD closed.'}
if running:
return {'message': 'HUD is already open. Find Frame Control HUD in the SteamVR dashboard.'}
env = {**os.environ, 'DISPLAY': ':0'}
p = subprocess.Popen(['xterm', '-name', 'frame-control-hud', '-title', 'Frame Control HUD',
'-fa', 'Monospace', '-fs', '20', '-geometry', '56x16',
'-bg', '#171d25', '-fg', '#d6d7d8', '-e',
sys.executable, str(ROOT / 'frame_vr.py'), 'hud'],
env=env, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, start_new_session=True)
try:
deadline = time.monotonic() + 8
while time.monotonic() < deadline and p.poll() is None:
tree = subprocess.check_output(['xwininfo', '-display', ':0', '-root', '-tree'],
text=True, timeout=2)
for w in re.findall(r'(0x[0-9a-f]+).*"frame-control-hud"', tree):
owner = subprocess.check_output(['xprop', '-display', ':0', '-id', w, '_NET_WM_PID'],
text=True, timeout=2)
if owner.strip().endswith('= ' + str(p.pid)):
subprocess.run(['xprop', '-display', ':0', '-id', w, '-f', 'STEAM_GAME', '32c',
'-set', 'STEAM_GAME', APPID], check=True, timeout=2)
identity = process_identity(p.pid)
if not identity:
raise RuntimeError('HUD process exited before its panel was ready')
save(path, {'pid': p.pid, 'start': identity})
return {'message': 'HUD opened. In SteamVR, select Frame Control HUD and Float in World or dock it to a controller.'}
time.sleep(.1)
raise RuntimeError('HUD did not create a window; is gamescope running?')
except BaseException:
if p.poll() is None:
p.terminate()
p.wait(timeout=3)
raise
def hud():
def fmt(v, unit=''):
return 'unavailable' if v is None else f'{v:.1f}{unit}'
while True:
p, b = performance(), battery() or {}
lines = ['FRAME CONTROL HUD', '',
'Compositor FPS ' + fmt(p['compositorFps']),
'Compositor period ' + fmt(p['frameMs'], ' ms'),
'Application FPS ' + fmt(p['appFps']),
'Render GPU time ' + fmt(p['gpuMs'], ' ms'),
'Compositor CPU ' + fmt(p['compositorCpuMs'], ' ms'),
'System CPU ' + fmt(p['cpuPercent'], '%'),
'GPU clock ' + fmt(p['gpuMHz'], ' MHz'),
'Hottest sensor ' + fmt(max_temp(), ' C'),
'Battery ' + fmt(b.get('percent'), '%'), '',
time.strftime('Updated %H:%M:%S'), 'Close this window to stop.']
print('\033[2J\033[H' + '\n'.join(lines), flush=True)
time.sleep(2)
def dispatch(body):
import fcntl # Frame only; validation is also imported by Windows hosts
action = validate(body)
ROOT.mkdir(parents=True, exist_ok=True, mode=0o700)
with (ROOT / 'control.lock').open('a') as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
return panel(action)
if __name__ == '__main__':
if sys.argv[1:] == ['hud']:
hud()
else:
try:
print(json.dumps(dispatch(json.load(sys.stdin))))
except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as e:
print(json.dumps({'error': str(e)}))
sys.exit(1)
+782 -26
View File
File diff suppressed because it is too large. Load diff
+292 -7
View File
@@ -44,13 +44,19 @@ import frame_assistant # noqa: E402
import frame_android # noqa: E402 import frame_android # noqa: E402
import frame_apk_versions # noqa: E402 import frame_apk_versions # noqa: E402
import frame_catalog # noqa: E402 import frame_catalog # noqa: E402
import frame_comfort # noqa: E402
import frame_host # noqa: E402 import frame_host # noqa: E402
import frame_macview # noqa: E402 import frame_macview # noqa: E402
import frame_media # noqa: E402
import frame_panels # noqa: E402
import frame_report # noqa: E402 import frame_report # noqa: E402
import frame_store # noqa: E402 import frame_store # noqa: E402
import frame_telemetry # noqa: E402 import frame_telemetry # noqa: E402
import frame_titles # noqa: E402 import frame_titles # noqa: E402
import frame_webinstall # noqa: E402 import frame_webinstall # noqa: E402
import frame_vr # noqa: E402
import frame_utilities # noqa: E402
import frame_compat_db # noqa: E402
frame_host.trust_bundled_cas() frame_host.trust_bundled_cas()
@@ -269,6 +275,37 @@ def status(_body):
return s return s
def comfort(body):
try:
frame_comfort.validate(body)
except ValueError as e:
raise Failure(str(e), 400)
# Content-addressed, user-only helper bundle. Desktop and phone use the same
# on-headset state/lock; no listener, service registration or third-party app.
import hashlib
files = {name: (HERE / name).read_text() for name in
("frame_comfort.py", "frame_status.py", "frame_steam.py")}
version = hashlib.sha256(json.dumps(files, sort_keys=True).encode()).hexdigest()[:16]
script = """import json, os, pathlib, subprocess, sys
os.umask(0o077)
files = %r
root = pathlib.Path.home() / '.cache/frame-control/comfort' / %r
root.mkdir(parents=True, exist_ok=True)
for name, source in files.items():
path = root / name
if not path.exists():
tmp = root / (name + '.' + str(os.getpid()))
tmp.write_text(source)
tmp.replace(path)
r = subprocess.run([sys.executable, str(root / 'frame_comfort.py'), %r], capture_output=True, text=True)
print(r.stdout, end='')
""" % (files, version, json.dumps(body))
out = json.loads(ssh("python3 -", stdin=script, timeout=65))
if out.get("error") and "active" not in out:
raise Failure(out["error"], 409)
return out
def headset_view(): def headset_view():
"""Both eyes as SteamVR composites them (see frame_vrshot.py); PNG bytes.""" """Both eyes as SteamVR composites them (see frame_vrshot.py); PNG bytes."""
# `timeout`: VR_Init can block if SteamVR is restarting. # `timeout`: VR_Init can block if SteamVR is restarting.
@@ -399,6 +436,7 @@ _stream_proc = None
def stream_command(query): def stream_command(query):
"""The ffmpeg that streams H.264: the headset view, or one panel's own window (src=panel)."""
q = parse_qs(query) q = parse_qs(query)
try: try:
height = int((q.get("h") or ["720"])[0]) height = int((q.get("h") or ["720"])[0])
@@ -408,6 +446,16 @@ def stream_command(query):
if height not in STREAM_HEIGHTS or fps not in STREAM_FPS: if height not in STREAM_HEIGHTS or fps not in STREAM_FPS:
raise Failure(f"h must be one of {STREAM_HEIGHTS} and fps one of {STREAM_FPS}", 400) raise Failure(f"h must be one of {STREAM_HEIGHTS} and fps one of {STREAM_FPS}", 400)
rate = 3 if height == 720 else 6 # Mbit/s rate = 3 if height == 720 else 6 # Mbit/s
if q.get("src") == ["panel"]:
# The panel's own pixels (x11grab of its window: gamescope keeps them), fitted
# to the height asked for. It stays still however the wearer moves their head.
window, display = panel_target(q)
return (f"DISPLAY={display} ffmpeg -hide_banner -loglevel error -nostdin -f x11grab -framerate {fps} "
f"-window_id {window} -i {display} "
f"-vf \"scale=-2:'trunc(min({height},ih)/2)*2',format=yuv420p\" -c:v libx264 -preset ultrafast "
f"-tune zerolatency -g {fps * 2} -bf 0 -b:v {rate}M -maxrate {rate}M -bufsize {rate // 2 or 1}M "
f"-x264-params aud=1:repeat-headers=1 -f h264 - & p=$!; "
f"exec >&-; cat >/dev/null; kill $p 2>/dev/null; wait $p")
return (f"[ -e {STREAM_DEVICE} ] || {{ echo 'No headset view device ({STREAM_DEVICE}). Is SteamVR running?' >&2; exit 3; }}; " return (f"[ -e {STREAM_DEVICE} ] || {{ echo 'No headset view device ({STREAM_DEVICE}). Is SteamVR running?' >&2; exit 3; }}; "
f"ffmpeg -hide_banner -loglevel error -nostdin -f v4l2 -video_size 1920x1080 -i {STREAM_DEVICE} " f"ffmpeg -hide_banner -loglevel error -nostdin -f v4l2 -video_size 1920x1080 -i {STREAM_DEVICE} "
f"-vf fps={fps},scale=-2:{height},format=yuv420p -c:v libx264 -preset ultrafast -tune zerolatency " f"-vf fps={fps},scale=-2:{height},format=yuv420p -c:v libx264 -preset ultrafast -tune zerolatency "
@@ -460,6 +508,27 @@ def steam(body):
return res return res
def vr(body):
try:
action = frame_vr.validate(body)
except ValueError as e:
raise Failure(str(e), 400)
sources = {name: (HERE / (name + '.py')).read_text() for name in ('frame_status', 'frame_vr')}
script = "import sys, types, json, os\n"
for name, source in sources.items():
script += f"m = types.ModuleType({name!r}); sys.modules[{name!r}] = m\nexec({source!r}, m.__dict__)\n"
# Only the optional HUD needs files: its terminal child survives this SSH call.
if action == 'hud-start':
script += "m.ROOT.mkdir(parents=True, exist_ok=True, mode=0o700)\n"
for name, source in sources.items():
script += f"p = m.ROOT / {name + '.py'!r}; t = p.with_suffix('.' + str(os.getpid()) + '.tmp')\nt.write_text({source!r}); os.replace(t, p)\n"
script += f"try:\n print(json.dumps(m.dispatch({body!r})))\nexcept Exception as e:\n print(json.dumps({{'error': str(e)}}))\n"
result = json.loads(ssh('python3 -', stdin=script, timeout=20))
if result.get('error'):
raise Failure(result['error'])
return result
def steam_search(query): def steam_search(query):
q = parse_qs(query) q = parse_qs(query)
cc = (q.get("cc") or [""])[0].upper() cc = (q.get("cc") or [""])[0].upper()
@@ -627,7 +696,7 @@ class InputAgent:
""" """
def __init__(self, source=HERE / "frame_input_agent.py", packages=None): def __init__(self, source=HERE / "frame_input_agent.py", packages=None):
self.source, self.proc, self.lock = source, None, threading.Lock() self.source, self.proc, self.lock, self.write_lock = source, None, threading.Lock(), threading.Lock()
self.packages = kdeconnect_packages() if packages is None else packages self.packages = kdeconnect_packages() if packages is None else packages
# generation counts stop()s; launching is the generation a launch is under way for. # generation counts stop()s; launching is the generation a launch is under way for.
self.status, self.launching, self.generation = {"state": "off"}, None, 0 self.status, self.launching, self.generation = {"state": "off"}, None, 0
@@ -800,8 +869,10 @@ class InputAgent:
self.start() self.start()
elif ready and events: elif ready and events:
try: try:
proc.stdin.write((json.dumps(events) + "\n").encode()) # One writer at a time: two devices sending at once mustn't tear a line.
proc.stdin.flush() with self.write_lock:
proc.stdin.write((json.dumps(events) + "\n").encode())
proc.stdin.flush()
sent = True sent = True
except (BrokenPipeError, OSError, ValueError): except (BrokenPipeError, OSError, ValueError):
pass # _watch reports how it ended pass # _watch reports how it ended
@@ -841,6 +912,115 @@ def remote_input(body):
return _input.send([input_event(e) for e in events]) return _input.send([input_event(e) for e in events])
# ---- touch: the headset's panels, through gamescope's own input (frame_touch.py) ----
PANEL_WINDOW = re.compile(r"^\d{1,10}$")
PANEL_DISPLAY = re.compile(r"^:\d{1,2}$")
TOUCH_BUTTONS = ("left", "right", "middle")
def panels():
"""The headset's app panels (window, display, name, size) and which has focus."""
return json.loads(ssh("python3 - panels", stdin=(HERE / "frame_touch.py").read_text(), timeout=20))
def panel_target(q):
window, display = (q.get("window") or [""])[0], (q.get("display") or [""])[0]
if not PANEL_WINDOW.match(window) or not PANEL_DISPLAY.match(display):
raise Failure("window must be a window id and display an X display such as :1", 400)
return window, display
def panel_capture(query):
"""One frame of a panel's own window, as PNG (its pixels, without the room around it)."""
window, display = panel_target(parse_qs(query))
return ssh(f"DISPLAY={display} timeout 10 ffmpeg -hide_banner -loglevel error -nostdin -f x11grab "
f"-window_id {window} -i {display} -frames:v 1 -f image2pipe -c:v png -", timeout=20, text=False)
def touch_event(event):
"""A frame_touch.py event with only the fields it knows, in range."""
if not isinstance(event, dict):
raise Failure("each touch event must be an object", 400)
def num(name, limit):
value = event.get(name)
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
raise Failure(f"{name} must be a number", 400)
return max(-limit, min(limit, round(float(value), 4)))
out = {}
if "fx" in event or "fy" in event:
if "window" not in event:
raise Failure("a position needs the panel's window and display", 400)
out.update(fx=num("fx", 1), fy=num("fy", 1))
# Any event can name the panel it's meant for; the Frame drops it if another has focus.
if "window" in event:
window, display = event.get("window"), event.get("display")
if isinstance(window, bool) or not isinstance(window, int) or window <= 0:
raise Failure("window must be the panel's window id", 400)
if not isinstance(display, str) or not PANEL_DISPLAY.match(display):
raise Failure("display must be an X display such as :1", 400)
out.update(window=window, display=display)
for name in ("dx", "dy"):
if name in event:
out[name] = num(name, INPUT_MOVE_LIMIT)
if "button" in event:
if event["button"] not in TOUCH_BUTTONS:
raise Failure(f"button must be one of {', '.join(TOUCH_BUTTONS)}", 400)
out["button"], out["down"] = event["button"], event.get("down") is not False
if "scroll" in event:
sc = event["scroll"]
if not isinstance(sc, list) or len(sc) != 2:
raise Failure("scroll must be [dx, dy]", 400)
out["scroll"] = [num_value(v, 5000) for v in sc]
if "key" in event:
key = event["key"]
if isinstance(key, bool) or not isinstance(key, int) or not 0 < key < 768:
raise Failure("key must be a Linux key code", 400)
out["key"], out["down"] = key, event.get("down") is not False
if "text" in event:
text = event["text"]
if not isinstance(text, str) or not 0 < len(text) <= INPUT_TEXT_LIMIT:
raise Failure(f"text must be 1 to {INPUT_TEXT_LIMIT} characters", 400)
out["text"] = text
if not set(out) - {"window", "display"}:
raise Failure("touch event has nothing to do", 400)
return out
def num_value(value, limit):
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
raise Failure("scroll values must be numbers", 400)
return max(-limit, min(limit, round(float(value), 2)))
class TouchAgent(InputAgent):
"""frame_touch.py on the Frame, fed events over one long-lived ssh. Nothing to install:
it uses gamescope's own input socket and the libei that's on the image."""
def __init__(self):
super().__init__(source=HERE / "frame_touch.py", packages=[])
def deliver(self, report, force=False):
return ""
def command(self, folder=""):
code = base64.b64encode(self.source.read_bytes()).decode()
return "python3 -u -c " + shlex.quote(
f"import base64;exec(compile(base64.b64decode('{code}'),'frame_touch','exec'))")
_touch = TouchAgent()
def remote_touch(body):
"""{"events": [...]} points, clicks, scrolls and types into the focused panel."""
events = body.get("events", [])
if not isinstance(events, list) or len(events) > INPUT_BATCH_LIMIT:
raise Failure(f"events must be a list of at most {INPUT_BATCH_LIMIT}", 400)
return _touch.send([touch_event(e) for e in events])
def flatpak(body): def flatpak(body):
app, action = str(body.get("id", "")), body.get("action") app, action = str(body.get("id", "")), body.get("action")
if not FLATPAK_ID.match(app): if not FLATPAK_ID.match(app):
@@ -1609,6 +1789,101 @@ def _sweep_one(prefix, d):
pass pass
# ---- Panel switcher (same helper on the companion and in the headset) ----
def panels_action(body):
action = body.get("action", "list")
script = (HERE / "frame_panels.py").read_text()
if action == "open":
# Installed in the user account so the page can outlive this SSH call.
remote = ('umask 077; mkdir -p ~/.local/share/frame-control/panels && '
'tmp=$(mktemp ~/.local/share/frame-control/panels/install.XXXXXX) && '
'cat > "$tmp" && mv "$tmp" ~/.local/share/frame-control/panels/switcher.py && '
'python3 ~/.local/share/frame-control/panels/switcher.py --open')
elif action == "list":
remote = "python3 -"
elif action == "focus":
key = body.get("key")
if not isinstance(key, str) or not frame_panels.KEY.fullmatch(key):
raise Failure("Choose an open panel.", 400)
remote = "python3 - --focus " + shlex.quote(key)
else:
raise Failure("Unknown panel action", 400)
result = json.loads(ssh(remote, stdin=script, timeout=45))
if "error" in result:
raise Failure(result["error"], 502)
return result
# ---- Our Frame-side media player -----------------------------------------
_MEDIA_LOCK = threading.Lock()
def media(body):
action = body.get("action")
if action not in ("list", "status", "play", "stop"):
raise Failure("Media action must be list, status, play or stop", 400)
if action == "play":
identity = body.get("id")
if not isinstance(identity, str) or not re.fullmatch(r"[0-9a-f]{32}/[^/\\\x00]+", identity):
raise Failure("Invalid media id", 400)
if body.get("layout", "auto") not in frame_media.LAYOUTS:
raise Failure("Invalid media layout", 400)
if type(body.get("theatre", False)) is not bool:
raise Failure("theatre must be true or false", 400)
with _MEDIA_LOCK:
# Ship only our small stdlib modules, atomically, to the user account.
sources = {name: (HERE / name).read_text() for name in (
"frame_media.py", "frame_media_player.py", "frame_media_remote.py", "frame_splat.py")}
installer = """import json, os, pathlib, sys, tempfile
root = pathlib.Path.home()/'.local/share/frame-control/media'
root.mkdir(parents=True, exist_ok=True)
for name, source in json.load(sys.stdin).items():
path = root/name
fd, temp = tempfile.mkstemp(dir=root, prefix=name+'.')
with os.fdopen(fd, 'w') as f:
f.write(source)
os.replace(temp, path)
"""
ssh("python3 -c " + shlex.quote(installer), stdin=json.dumps(sources))
try:
out = ssh("python3 ~/.local/share/frame-control/media/frame_media_remote.py",
stdin=json.dumps(body), timeout=75) # remote worst case: ffprobe 30 + reset-failed 10 + systemd-run 15 s
except Failure as e:
for line in reversed(getattr(e, "stdout", "").splitlines()):
try:
detail = json.loads(line).get("error")
except (ValueError, AttributeError):
continue
if detail:
raise Failure(detail) from None
raise
return json.loads(out)
def push_media(path):
# Validate the format, but leave layout selection until playback (ffprobe
# can then read metadata on the Frame, where it is installed).
name = Path(path).name
frame_media.plan(name, "mono")
if name.startswith(".") or "\\" in name:
raise Failure("Rename the file: media names can't start with a dot or contain a backslash", 400)
token = secrets.token_hex(16)
dest = "Videos/FrameControl/" + token + "/"
ssh("mkdir -p ~/" + dest)
try:
push_file(path, dest)
except Exception:
try:
ssh("rm -rf ~/" + dest)
except Exception:
pass # keep the copy error; an empty folder isn't listed as media
raise
return {"message": "Media sent. Choose its layout and press Play.",
"id": token + "/" + name}
# ---- Mac in the headset (frame_macview.py) ---------------------------------- # ---- Mac in the headset (frame_macview.py) ----------------------------------
# The tunnel gets its own connection: the shared master's options would win # The tunnel gets its own connection: the shared master's options would win
@@ -1675,14 +1950,14 @@ def agent_approval(body):
return frame_agent.approvals.decide(body.get("confirmation"), body.get("accept")) return frame_agent.approvals.decide(body.get("confirmation"), body.get("accept"))
POST = {"/api/agent/call": agent_call, "/api/agent/approval": agent_approval, POST = {"/api/vr": vr, "/api/comfort": comfort, "/api/media": media, "/api/agent/call": agent_call, "/api/agent/approval": agent_approval,
"/api/assistant/chat": assistant_chat, "/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard, "/api/assistant/chat": assistant_chat, "/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard,
"/api/input": remote_input, "/api/input": remote_input, "/api/touch": remote_touch,
"/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots, "/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots,
"/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start, "/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start,
"/api/webinstall/cancel": webinstall_cancel, "/api/webinstall/cancel": webinstall_cancel,
"/api/telemetry": frame_telemetry.update_settings, "/api/telemetry/event": frame_telemetry.page_event, "/api/telemetry": frame_telemetry.update_settings, "/api/telemetry/event": frame_telemetry.page_event,
"/api/report/preview": report_preview, "/api/report": report_send, "/api/macview": macview_action} "/api/report/preview": report_preview, "/api/report": report_send, "/api/macview": macview_action, "/api/panels": panels_action}
# ---- HTTP ------------------------------------------------------------------ # ---- HTTP ------------------------------------------------------------------
@@ -1811,6 +2086,10 @@ class Handler(BaseHTTPRequestHandler):
self.send_json(title_job(url.query)) self.send_json(title_job(url.query))
elif path == "/api/licenses": elif path == "/api/licenses":
self.send_json({"notices": licenses()}) self.send_json({"notices": licenses()})
elif path == "/api/panels":
self.send_json(panels())
elif path == "/api/touch":
self.send_json(_touch.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_touch.status))
elif path == "/api/input": elif path == "/api/input":
self.send_json(_input.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_input.status)) self.send_json(_input.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_input.status))
elif path == "/api/job": elif path == "/api/job":
@@ -1822,6 +2101,8 @@ class Handler(BaseHTTPRequestHandler):
"shared": frame_catalog.compat_db.shared()}) "shared": frame_catalog.compat_db.shared()})
elif path == "/api/android/catalog": elif path == "/api/android/catalog":
self.send_json({"apps": frame_catalog.catalog()}) self.send_json({"apps": frame_catalog.catalog()})
elif path == "/api/vr/utilities":
self.send_json(frame_utilities.catalogue(steam_frame("utilities")["utilities"], frame_compat_db.load()))
elif path == "/api/macview": elif path == "/api/macview":
self.send_json(macview_state(parse_qs(url.query))) self.send_json(macview_state(parse_qs(url.query)))
elif path == "/api/telemetry": elif path == "/api/telemetry":
@@ -1842,6 +2123,8 @@ class Handler(BaseHTTPRequestHandler):
self.send_bytes(*shot_image(url.query)) self.send_bytes(*shot_image(url.query))
elif path == "/api/stream": elif path == "/api/stream":
self.stream_video(url.query) self.stream_video(url.query)
elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["panel"]:
self.send_bytes(panel_capture(url.query), "image/png", headers=[("X-Capture-Source", "panel")])
elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["headset"]: elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["headset"]:
self.send_bytes(headset_view(), "image/png", headers=[("X-Capture-Source", "steamvr")]) self.send_bytes(headset_view(), "image/png", headers=[("X-Capture-Source", "steamvr")])
elif path == "/api/screenshot": elif path == "/api/screenshot":
@@ -1918,7 +2201,7 @@ class Handler(BaseHTTPRequestHandler):
proc.wait() proc.wait()
errors.seek(0) errors.seek(0)
err = strip_ansi(errors.read().decode(errors="replace")).strip() err = strip_ansi(errors.read().decode(errors="replace")).strip()
raise Failure(err or "The headset view sent no video for 20 s") raise Failure(err or "The Frame sent no video for 20 s")
chunk = first chunk = first
try: try:
self.send_response(200) self.send_response(200)
@@ -1972,6 +2255,8 @@ class Handler(BaseHTTPRequestHandler):
raise Failure("upload interrupted", 400) raise Failure("upload interrupted", 400)
f.write(chunk) f.write(chunk)
remaining -= len(chunk) remaining -= len(chunk)
if mode == "media":
return push_media(dest)
if mode == "apkinfo": if mode == "apkinfo":
# Read an APK for a report without installing it. # Read an APK for a report without installing it.
try: try: