Compare commits

..
Author SHA1 Message Date
saphidandClaude Opus 5.5 3ef7312654 Merge main into panel-workspaces (testing notes)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:50:17 +10:00
Alex Southwell 70e7f7f5e5 Merge pull request #39 from saphid/family-comfort
Family and comfort: safe timers, casting, alerts and breaks
2026-09-29 11:38:47 +10:00
saphidandClaude Opus 5.5 2bd86207c7 Merge main into panel-workspaces: the panel switcher alongside media, analytics and the Mac view
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:37:18 +10:00
saphidandClaude Opus 5.5 be1ab129c9 Tests: read the page as UTF-8 (Windows' default codec can't read its arrows)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:33:34 +10:00
saphid a38d0cda6e Document shared Frame test procedure and blocked recheck 2026-09-29 11:16:29 +10:00
Alex SouthwellandClaude Opus 5.5 e702adbd77 Live view: a Desktop view that stays still, and Control to tap on the Frame (#46)
* Live view: a Desktop view that stays still, and Control to tap on the Frame

The live view gets a second source and a way to use the Frame from it:

- Desktop: the app panel in use in the headset, streamed from its own window
  (x11grab of gamescope's redirected window), so it doesn't move as the
  wearer looks around. A picker shows any other panel, view only.
- Control: on the Desktop view a tap or click lands exactly where you put it;
  drag is a mouse drag, press and hold right-clicks, two fingers scroll, and
  on a computer the mouse, wheel and keyboard work directly. On the headset
  view the view is a trackpad. A text field and key row type from a phone.

Input goes through gamescope's own EIS socket (the way Steam feeds Remote
Play input) with the libei already on the image: ui/frame_touch.py, over
the same long-lived ssh machinery as the keyboard agent, nothing to
install. It reaches the panel that has focus on either X display, which
the KDE Connect route can't. Verified on the Frame and from the iPhone app
in the Simulator.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from review

- Keys held on the Frame are released with buttons when Control stops or
  the view loses focus; keys for the Frame no longer trigger Frame Control's
  own shortcuts.
- Taps only act when the picture on screen is the panel in use; positions,
  presses, keys, text and scrolls name their panel (display and window: ids
  repeat across :0 and :1, told apart by pid), and the Frame drops them if
  focus has moved on. Releases always go.
- While connecting, a tap keeps its position; on an error only releases wait
  and retries back off; trimming a long queue never drops a release.
- Lifting one of two scrolling fingers ends the scroll; a cancelled touch
  isn't a tap; clicks and holds on the bars around the picture do nothing.
- A capture loop from before a Live restart can't stop the new video.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: close the targeting gaps from the second review

- The focused panel's display comes from GAMESCOPE_FOCUS_DISPLAY (gamescope
  packs ":1" into the first value), so a window id repeated across :0 and :1
  can't be mistaken; the pid is only the fallback.
- Presses, keys, text and scrolls read focus afresh on the Frame; only moves
  use a reading up to a second old.
- A gesture remembers the panel it started on and does nothing more if that
  stops being the one in use; a press with no panel to aim at isn't sent.
- Opening a screenshot clears the panel Control would act on; switching to
  another app releases held keys and buttons.
- Trimming keeps a click with its position; the error backoff holds for new
  input too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from the SWE-2 Max review

- The Frame side tracks keys as well as buttons and lets go of both when the
  session ends.
- A stale tap tells the page, which re-reads the panels at once.
- A paused input device waits instead of ending the session; only a
  disconnect does. An OS error on one event skips it.
- Presses check focus with two property reads and do the full lookup only
  when it changed.
- Writes to an agent's stdin are serialized, so two devices sending at once
  can't tear a line (the keyboard agent too).
- Connecting gives up with a message after 15 s instead of hanging on
  "Connecting…"; text goes in 100-character pieces so releases don't wait
  behind a long paste; a cancelled mouse gesture releases what's held.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: stale clears, pauses reconverge, pastes split per request

- The Frame says it has caught up as soon as an aimed event lands after a
  stale one, so the page stops re-reading the panels.
- After a device pause it lets go of everything it holds (releases that
  arrived while paused were dropped), and waits for the device once per
  batch, not once per event.
- The quick focus check no longer freshens the panel geometry's age.
- Each request carries at most about 100 characters of text.
- Turning Control off while it connects doesn't report an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: build the socket path on the Frame, so Windows can import it for tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: any event that goes through clears the stale flag

A trackpad move names no panel, so waiting for an aimed event could leave
the page re-reading panels for the rest of the session; a release still
aimed at the old panel doesn't count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:13:21 +10:00
saphidandClaude Opus 5.5 ab1985b6b3 Comfort: a state missing 'started' can't crash status (timestamps of 0 still count)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:12:41 +10:00
saphidandClaude Opus 5.5 1203ec0a9e Merge main into family-comfort; a session state without 'started' can't crash status
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:09:35 +10:00
saphid 222d5eccc9 fix(comfort): harden timer recovery and notification UX after review 2026-09-29 08:31:00 +10:00
saphid 91c5853627 Refresh panel switcher screenshot from final device check 2026-09-28 22:34:53 +10:00
saphid 72352c5914 Add companion and headset panel switchers with feasibility evidence 2026-09-28 22:33:23 +10:00
saphid d3fa282377 docs(comfort): include verified desktop and iPhone notification evidence 2026-09-28 22:32:54 +10:00
saphid 0622afcae9 feat(ui): add family controls, casting and native notifications 2026-09-28 22:29:58 +10:00
saphid 522c46ed6f feat(comfort): run safe session timers and alerts on the Frame 2026-09-28 22:29:58 +10:00
54 changed files with 3602 additions and 3378 deletions

No files matched your search

-53
View File
@@ -1,53 +0,0 @@
name: PC host libraries
on:
pull_request:
paths: ["desktop/**", "ui/**", "tests/**", "mac/frame-mac-view/**", "scripts/macview-bench.py", ".github/workflows/pc-host.yml"]
workflow_dispatch:
jobs:
linux:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- name: Native libraries
run: |
sudo apt-get update -qq
sudo apt-get install -y libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libglib2.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-pipewire libpipewire-0.3-modules libspa-0.2-modules
python3 desktop/build.py
- name: Tests (real x264, fake capture consent/input)
env:
FRAME_PC_REQUIRE_NATIVE: "1"
run: python3 -m unittest discover -s tests -p 'test_pc*.py' -v
- uses: actions/upload-artifact@v4
with:
name: pc-host-${{ matrix.os }}
path: desktop/bundle
windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
- uses: msys2/setup-msys2@v2
with:
msystem: UCRT64
update: true
install: >-
mingw-w64-ucrt-x86_64-gcc
mingw-w64-ucrt-x86_64-pkgconf
mingw-w64-ucrt-x86_64-python
mingw-w64-ucrt-x86_64-gstreamer
mingw-w64-ucrt-x86_64-gst-plugins-base
mingw-w64-ucrt-x86_64-gst-plugins-good
mingw-w64-ucrt-x86_64-gst-plugins-bad
mingw-w64-ucrt-x86_64-gst-plugins-ugly
- name: Native build and tests (x264; WGC and MF need a desktop/GPU)
shell: msys2 {0}
run: |
python desktop/build.py
FRAME_PC_REQUIRE_NATIVE=1 python -m unittest discover -s tests -p 'test_pc*.py' -v
- uses: actions/upload-artifact@v4
with:
name: pc-host-windows
path: desktop/bundle
+1 -45
View File
@@ -7,7 +7,7 @@ on:
push: push:
tags: ["v*"] tags: ["v*"]
pull_request: pull_request:
paths: ["app/**", "ui/**", "desktop/**", "mac/**", "scripts/**", "frame/**", "apk-catalog/**", ".github/workflows/release.yml"] paths: ["app/**", "ui/**", "scripts/**", "frame/**", "apk-catalog/**", ".github/workflows/release.yml"]
workflow_dispatch: workflow_dispatch:
permissions: permissions:
@@ -28,62 +28,18 @@ jobs:
- os: ubuntu-latest - os: ubuntu-latest
script: dist:linux script: dist:linux
files: app/dist/*.AppImage app/dist/*.deb files: app/dist/*.AppImage app/dist/*.deb
- os: ubuntu-24.04-arm
script: dist:linux:arm64
files: app/dist/*.AppImage app/dist/*.deb
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- uses: actions/setup-node@v4 - uses: actions/setup-node@v4
with: with:
node-version: "24" node-version: "24"
- name: Linux streaming libraries
if: runner.os == 'Linux'
shell: bash
run: |
sudo apt-get update -qq
sudo apt-get install -y libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libglib2.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-pipewire libpipewire-0.3-modules libspa-0.2-modules
arch=x64; if [ "$(uname -m)" = aarch64 ]; then arch=arm64; fi
python3 desktop/build.py --out "app/build/desktop/linux-$arch"
- uses: msys2/setup-msys2@v2
if: runner.os == 'Windows'
with:
msystem: UCRT64
update: true
install: >-
mingw-w64-ucrt-x86_64-gcc
mingw-w64-ucrt-x86_64-pkgconf
mingw-w64-ucrt-x86_64-python
mingw-w64-ucrt-x86_64-gstreamer
mingw-w64-ucrt-x86_64-gst-plugins-base
mingw-w64-ucrt-x86_64-gst-plugins-good
mingw-w64-ucrt-x86_64-gst-plugins-bad
mingw-w64-ucrt-x86_64-gst-plugins-ugly
- name: Windows streaming libraries
if: runner.os == 'Windows'
shell: msys2 {0}
run: python desktop/build.py --out app/build/desktop/win-x64
- name: Build - name: Build
working-directory: app working-directory: app
shell: bash shell: bash
run: npm ci && npm run ${{ matrix.script }} run: npm ci && npm run ${{ matrix.script }}
env: env:
CSC_IDENTITY_AUTO_DISCOVERY: "false" CSC_IDENTITY_AUTO_DISCOVERY: "false"
- name: Test the bundled Python with the native PC libraries
if: runner.os != 'macOS'
shell: bash
run: |
if [ "$RUNNER_OS" = Windows ]; then
platform=win; arch=x64
python="$PWD/app/build/deps/win-x64/python/python.exe"
native="$(cygpath -w "$PWD/app/build/desktop/win-x64")"
else
platform=linux; arch=x64
if [ "$(uname -m)" = aarch64 ]; then arch=arm64; fi
python="$PWD/app/build/deps/linux-$arch/python/bin/python3"
native="$PWD/app/build/desktop/linux-$arch"
fi
FRAME_PC_NATIVE="$native" FRAME_PC_REQUIRE_NATIVE=1 "$python" -m unittest discover -s tests -p 'test_pc*.py' -v
- name: Upload to the release - name: Upload to the release
if: startsWith(github.ref, 'refs/tags/') if: startsWith(github.ref, 'refs/tags/')
shell: bash shell: bash
-10
View File
@@ -7,15 +7,5 @@ compat-db/.lakebed/
tests/smoke/results/ tests/smoke/results/
mac/bin/ mac/bin/
# Shared desktop streaming controller build products
desktop/controller.dll
desktop/controller.dylib
desktop/controller.so
desktop/*.obj
desktop/*.lib
desktop/*.exp
desktop/bundle/
app/build/desktop/
# Downloaded at build time (frame/kdeconnect/fetch.py, app/build/fetch-deps.js) # Downloaded at build time (frame/kdeconnect/fetch.py, app/build/fetch-deps.js)
frame/kdeconnect/packages/ frame/kdeconnect/packages/
+7 -4
View File
@@ -35,8 +35,8 @@ See what the headset sees, install games and Android apps, move files and text a
<tr> <tr>
<td width="50%" valign="top"> <td width="50%" valign="top">
**👓 Headset view**<br> **👓 Headset view and Desktop**<br>
Live video of what the lenses show (about 30 fps), or a still of both eyes. Zoom, pan, full screen, save as PNG. Live video of what the lenses show, or of the app panel in use, flat and still however the wearer looks around. Turn on Control and tap or click right on it to use the Frame from your phone or computer.
</td> </td>
<td width="50%" valign="top"> <td width="50%" valign="top">
@@ -93,7 +93,11 @@ SSH, SFTP, Steam Link, remote desktop, volume, sleep, restart and shut down.
</tr> </tr>
</table> </table>
Nothing is installed on the Frame for any of this: the app uses what SteamOS The optional [Family and comfort](docs/family-comfort.md) card adds session
limits, breaks, local alerts and one-click casting. A session copies a small
Frame Control worker into your headset user account.
For the other features, nothing is installed on the Frame: the app uses what SteamOS
already ships (sideloading a game copies Valve's own devkit scripts to already ships (sideloading a game copies Valve's own devkit scripts to
`~/devkit-utils`, as Valve's Devkit Client does). [How each feature works](docs/frame-control.md). `~/devkit-utils`, as Valve's Devkit Client does). [How each feature works](docs/frame-control.md).
@@ -210,7 +214,6 @@ Frame's software fits together, all checked against a real headset and labelled
| [Sideloading Linux and Windows games](docs/sideloading.md) | A .zip, folder or .exe as a Steam Devkit Game, runtime detection | | [Sideloading Linux and Windows games](docs/sideloading.md) | A .zip, folder or .exe as a Steam Devkit Game, runtime detection |
| [Install links for websites](docs/web-install.md) | `frame-control://install` links and manifests, the rules, a button to paste | | [Install links for websites](docs/web-install.md) | `frame-control://install` links and manifests, the rules, a button to paste |
| [Steam games](docs/steam-games.md) · [VR video](docs/vr-video.md) · [WebXR in Chromium](docs/webxr-chromium.md) | Installing and buying, watching VR180/360, the Chromium build | | [Steam games](docs/steam-games.md) · [VR video](docs/vr-video.md) · [WebXR in Chromium](docs/webxr-chromium.md) | Installing and buying, watching VR180/360, the Chromium build |
| [PC in the headset](docs/pc-in-headset.md) | Windows and Linux host implementation and test coverage |
| [Mac in the headset](docs/mac-in-headset.md) | Mac windows and screens as panels in the Frame, with laser and keyboard input | | [Mac in the headset](docs/mac-in-headset.md) | Mac windows and screens as panels in the Frame, with laser and keyboard input |
| [VR mods and custom songs](docs/mods.md) | Per-game feasibility, real-Frame results and blockers; no installer yet | | [VR mods and custom songs](docs/mods.md) | Per-game feasibility, real-Frame results and blockers; no installer yet |
| [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes | | [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes |
+17 -2
View File
@@ -1,7 +1,7 @@
// Frame Control as a desktop app (macOS, Windows, Linux): starts ui/server.py on // Frame Control as a desktop app (macOS, Windows, Linux): starts ui/server.py on
// a free loopback port and shows it in a native window. The server does all the // a free loopback port and shows it in a native window. The server does all the
// work over the `frame` SSH alias; this file only hosts it. // work over the `frame` SSH alias; this file only hosts it.
const { app, BrowserWindow, Menu, clipboard, dialog, ipcMain, shell } = require("electron"); const { app, BrowserWindow, Menu, Notification, clipboard, dialog, ipcMain, shell } = require("electron");
const { execFile, spawn } = require("child_process"); const { execFile, spawn } = require("child_process");
const { promisify } = require("util"); const { promisify } = require("util");
const fs = require("fs"); const fs = require("fs");
@@ -254,6 +254,21 @@ ipcMain.handle("update:get", (e) => fromUi(e) ? publicUpdate() : null);
ipcMain.handle("update:check", (e) => fromUi(e) ? checkForUpdate({ manual: true }).then(publicUpdate) : null); ipcMain.handle("update:check", (e) => fromUi(e) ? checkForUpdate({ manual: true }).then(publicUpdate) : null);
ipcMain.handle("update:install", (e) => { if (fromUi(e)) installUpdate(); }); ipcMain.handle("update:install", (e) => { if (fromUi(e)) installUpdate(); });
ipcMain.handle("comfort:notify", (e, message) => {
if (!fromUi(e) || typeof message !== "string" || message.length > 500) throw new Error("Invalid notification");
if (!Notification.isSupported()) throw new Error("System notifications are unavailable");
return new Promise((resolve, reject) => {
const notification = new Notification({title: "Frame Control", body: message});
const timer = setTimeout(() => reject(new Error("Notification delivery was not confirmed. Check system notification settings.")), 5000);
notification.once("show", () => { clearTimeout(timer); resolve(true); });
notification.once("failed", (_event, error) => {
clearTimeout(timer);
reject(new Error("Notification delivery failed. Check system notification settings: " + error));
});
notification.show();
});
});
// frame-control://install links from websites (docs/web-install.md). They can // frame-control://install links from websites (docs/web-install.md). They can
// arrive before the window or server exists (macOS open-url on a cold launch), // arrive before the window or server exists (macOS open-url on a cold launch),
// so they wait here until the page asks for them. The page checks the link with // so they wait here until the page asks for them. The page checks the link with
@@ -376,7 +391,7 @@ function createWindow() {
title: "Frame Control", backgroundColor: BG, show: false, title: "Frame Control", backgroundColor: BG, show: false,
...(IS_MAC ? { titleBarStyle: "hiddenInset", trafficLightPosition: { x: 18, y: 26 } } ...(IS_MAC ? { titleBarStyle: "hiddenInset", trafficLightPosition: { x: 18, y: 26 } }
: { icon: path.join(__dirname, "build", "icon.png") }), : { icon: path.join(__dirname, "build", "icon.png") }),
webPreferences: { contextIsolation: true, nodeIntegration: false, sandbox: true, webPreferences: { contextIsolation: true, nodeIntegration: false, sandbox: true, backgroundThrottling: false,
preload: path.join(__dirname, "preload.js") }, preload: path.join(__dirname, "preload.js") },
}); });
win.once("ready-to-show", () => win.show()); win.once("ready-to-show", () => win.show());
+4 -23
View File
@@ -11,9 +11,8 @@
"icon": "env -u ELECTRON_RUN_AS_NODE electron build/make-icon.js", "icon": "env -u ELECTRON_RUN_AS_NODE electron build/make-icon.js",
"dist": "sh ../mac/frame-mac-view/build.sh && node build/fetch-deps.js mac arm64 && electron-builder --mac --arm64 --publish never", "dist": "sh ../mac/frame-mac-view/build.sh && node build/fetch-deps.js mac arm64 && electron-builder --mac --arm64 --publish never",
"dist:dir": "sh ../mac/frame-mac-view/build.sh && node build/fetch-deps.js mac arm64 && electron-builder --mac --arm64 --dir", "dist:dir": "sh ../mac/frame-mac-view/build.sh && node build/fetch-deps.js mac arm64 && electron-builder --mac --arm64 --dir",
"dist:linux": "node build/fetch-deps.js linux x64 && electron-builder --linux --x64 --publish never", "dist:linux": "node build/fetch-deps.js linux x64 arm64 && electron-builder --linux --x64 --arm64 --publish never",
"dist:win": "node build/fetch-deps.js win x64 && electron-builder --win --x64 --publish never", "dist:win": "node build/fetch-deps.js win x64 && electron-builder --win --x64 --publish never"
"dist:linux:arm64": "node build/fetch-deps.js linux arm64 && electron-builder --linux --arm64 --publish never"
}, },
"devDependencies": { "devDependencies": {
"electron": "^44.4.5", "electron": "^44.4.5",
@@ -172,16 +171,7 @@
"entry": { "entry": {
"StartupWMClass": "frame-control" "StartupWMClass": "frame-control"
} }
}, }
"extraResources": [
{
"from": "build/desktop/${os}-${arch}",
"to": "desktop/bundle",
"filter": [
"**/*"
]
}
]
}, },
"deb": { "deb": {
"depends": [ "depends": [
@@ -194,16 +184,7 @@
"zip" "zip"
], ],
"icon": "build/icon.png", "icon": "build/icon.png",
"artifactName": "Frame-Control-win-${arch}.${ext}", "artifactName": "Frame-Control-win-${arch}.${ext}"
"extraResources": [
{
"from": "build/desktop/${os}-${arch}",
"to": "desktop/bundle",
"filter": [
"**/*"
]
}
]
}, },
"nsis": { "nsis": {
"oneClick": false, "oneClick": false,
+1
View File
@@ -9,6 +9,7 @@
const { contextBridge, ipcRenderer, webUtils } = require("electron"); const { contextBridge, ipcRenderer, webUtils } = require("electron");
contextBridge.exposeInMainWorld("frameApp", { contextBridge.exposeInMainWorld("frameApp", {
notify: (message, request) => ipcRenderer.invoke("comfort:notify", message, request),
readClipboard: () => ipcRenderer.invoke("clipboard:read"), readClipboard: () => ipcRenderer.invoke("clipboard:read"),
setUpConnection: () => ipcRenderer.invoke("connection:setup"), setUpConnection: () => ipcRenderer.invoke("connection:setup"),
// While the keyboard-and-trackpad panel holds the keyboard, ⌘W, ⌘R and the rest go to the Frame. // While the keyboard-and-trackpad panel holds the keyboard, ⌘W, ⌘R and the rest go to the Frame.
@@ -1,311 +0,0 @@
{
"label": "pc-agent-frame-arm64-hairpin",
"date": "2026-09-28T22:36:19",
"commit": "c5fc552+dirty",
"config": {
"quality": "balanced",
"mode": "separate",
"net": "none",
"delay_ms": 0,
"buffer_ms": 250,
"host": "frame",
"usb": false,
"ssh_opts": [],
"encoder": "",
"duration_s": 10.0,
"browser_flags": []
},
"frame_build": "20260925.6191901",
"mac": "26.5.2",
"host_platform": "Linux aarch64, SteamOS Frame, BUILD_ID 20260925.6191901 (synthetic host only)",
"source": "",
"pc_host": true,
"headset": "0:03.056203 [Info] - 1 - leaving standby",
"scenarios": [
{
"scenario": "test",
"frames_sent": 351,
"frames_drawn": 351,
"frames_shown": 324,
"duration_s": 10.1,
"stages_ms": {
"capture": {
"p50": 0.1,
"p95": 0.2,
"p99": 0.5,
"n": 351
},
"queue": {
"p50": 0.1,
"p95": 0.3,
"p99": 0.4,
"n": 351
},
"encode": {
"p50": 2.0,
"p95": 2.9,
"p99": 3.5,
"n": 351
},
"socket": {
"p50": 0.1,
"p95": 0.4,
"p99": 0.6,
"n": 351
},
"network": {
"p50": 11.6,
"p95": 75.7,
"p99": 130.1,
"n": 351
},
"decode": {
"p50": 0.7,
"p95": 2.1,
"p99": 4.1,
"n": 351
},
"draw": {
"p50": 0.4,
"p95": 0.8,
"p99": 1.1,
"n": 351
},
"present": {
"p50": 0.8,
"p95": 18.1,
"p99": 24.9,
"n": 324
},
"content": {
"p50": 15.1,
"p95": 80.3,
"p99": 134.2,
"n": 351
},
"content_shown": {
"p50": 21.0,
"p95": 71.8,
"p99": 146.3,
"n": 324
}
},
"fps": 34.8,
"fps_shown": 32.0,
"late_pct": 53.41,
"stall_max": 226.8,
"stalls_over_100ms": 7,
"mbps": 0.26,
"keyframes": 6,
"size": "1280x720",
"captured": 607,
"viewer_never_drawn": 0,
"input_ms": {
"p50": 43.3,
"p95": 98.4,
"p99": 110.1,
"n": 18
},
"input_shown_ms": {
"p50": 54.7,
"p95": 98.9,
"p99": 114.0,
"n": 18
},
"input_parts_ms": {
"uplink": {
"p50": 17.1,
"p95": 48.3,
"p99": 53.1,
"n": 18
},
"mac": {
"p50": 12.5,
"p95": 32.9,
"p99": 35.7,
"n": 18
},
"back": {
"p50": 16.8,
"p95": 50.0,
"p99": 55.9,
"n": 18
}
},
"inputs": {
"asked": 18,
"sent": 18,
"seen": 18
},
"timeline": [
{
"t": 0,
"fps": 38,
"mbps": 0.3,
"content_p50": 14.8,
"content_p95": 25.3,
"bitrate": 2764800,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 1,
"fps": 33,
"mbps": 0.24,
"content_p50": 15.6,
"content_p95": 103.5,
"bitrate": 3845448,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 2,
"fps": 39,
"mbps": 0.28,
"content_p50": 15.6,
"content_p95": 29.9,
"bitrate": 5283790,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 3,
"fps": 25,
"mbps": 0.21,
"content_p50": 17.8,
"content_p95": 175.9,
"bitrate": 2764800,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 4,
"fps": 34,
"mbps": 0.22,
"content_p50": 15.4,
"content_p95": 69.8,
"bitrate": 2764800,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 5,
"fps": 38,
"mbps": 0.29,
"content_p50": 17.7,
"content_p95": 68.7,
"bitrate": 2764800,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 6,
"fps": 38,
"mbps": 0.26,
"content_p50": 15.0,
"content_p95": 18.2,
"bitrate": 3091280,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 7,
"fps": 30,
"mbps": 0.22,
"content_p50": 19.2,
"content_p95": 111.0,
"bitrate": 1725204,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 8,
"fps": 35,
"mbps": 0.26,
"content_p50": 13.4,
"content_p95": 102.6,
"bitrate": 1725204,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 9,
"fps": 38,
"mbps": 0.28,
"content_p50": 12.3,
"content_p95": 20.9,
"bitrate": 1725204,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 10,
"fps": 3,
"mbps": 0.02,
"content_p50": 20.9,
"content_p95": 24.7,
"bitrate": 1725204,
"tier": 0,
"w": 1280,
"net": null
}
],
"adapt": [],
"content_p50": 15.1,
"content_p95": 80.3,
"input_p50": 43.3,
"input_p95": 98.4,
"grades": {
"input_replies": "local",
"content_p50": "local",
"content_p95": "acceptable",
"input_p50": "local",
"input_p95": "acceptable",
"fps": "bad",
"late_pct": "bad",
"stall_max": "acceptable"
},
"controller": {
"target": 3442081,
"ceiling": 5529600,
"tier": 0,
"fps": 60,
"scale": 1.0,
"baseRtt": 13.334,
"inFlight": 1,
"slack": 71.738,
"adapt": true
},
"events": [
{
"t": 3.77,
"e": "target 2764800 bit/s; tier 0"
},
{
"t": 7.42,
"e": "target 1725204 bit/s; tier 0"
}
],
"source_fps": 60.1,
"cpu": {
"host": "not sampled"
},
"viewer": "h264 software; ANGLE (Mesa, zink Vulkan 1.4(Turnip Adreno (TM) 750 (MESA_TURNIP)), OpenGL 4.6); raf 71 Hz",
"show_s": 1.88,
"panel": "valve.steam.desktopgame.2001932457",
"src": "test",
"route": "network"
}
],
"proof": "Synthetic x264 PC agent runs on the Frame itself, forwarded via Mac and back. Not Windows/Linux desktop capture or laser input."
}
@@ -1,302 +0,0 @@
{
"label": "pc-agent-frame-arm64-final",
"date": "2026-09-28T22:55:27",
"commit": "cd20243",
"config": {
"quality": "balanced",
"mode": "native",
"net": "none",
"delay_ms": 0,
"buffer_ms": 250,
"host": "frame",
"usb": false,
"ssh_opts": [],
"encoder": "",
"duration_s": 10.0,
"browser_flags": []
},
"frame_build": "20260925.6191901",
"mac": "26.5.2",
"host_platform": "Linux aarch64, SteamOS Frame, BUILD_ID 20260925.6191901 (synthetic host only)",
"source": "",
"pc_host": true,
"headset": "0:03.056203 [Info] - 1 - leaving standby",
"scenarios": [
{
"scenario": "test",
"frames_sent": 400,
"frames_drawn": 400,
"frames_shown": 339,
"duration_s": 10.2,
"stages_ms": {
"capture": {
"p50": 0.1,
"p95": 0.1,
"p99": 0.2,
"n": 400
},
"queue": {
"p50": 0.1,
"p95": 0.2,
"p99": 0.4,
"n": 400
},
"encode": {
"p50": 2.0,
"p95": 2.7,
"p99": 3.1,
"n": 400
},
"socket": {
"p50": 0.1,
"p95": 0.3,
"p99": 0.4,
"n": 400
},
"network": {
"p50": 11.8,
"p95": 25.2,
"p99": 37.8,
"n": 400
},
"decode": {
"p50": 0.7,
"p95": 1.5,
"p99": 2.0,
"n": 400
},
"draw": {
"p50": 0.3,
"p95": 0.7,
"p99": 1.0,
"n": 400
},
"present": {
"p50": 0.8,
"p95": 19.3,
"p99": 26.8,
"n": 339
},
"content": {
"p50": 15.1,
"p95": 28.6,
"p99": 41.3,
"n": 400
},
"content_shown": {
"p50": 17.6,
"p95": 39.0,
"p99": 47.2,
"n": 339
}
},
"fps": 39.3,
"fps_shown": 33.4,
"late_pct": 50.25,
"stall_max": 69.3,
"stalls_over_100ms": 0,
"mbps": 0.35,
"keyframes": 6,
"size": "1280x720",
"captured": 612,
"viewer_never_drawn": 0,
"input_ms": {
"p50": 76.6,
"p95": 98.2,
"p99": 127.8,
"n": 19
},
"input_shown_ms": {
"p50": 80.5,
"p95": 107.8,
"p99": 128.1,
"n": 16
},
"input_parts_ms": {
"uplink": {
"p50": 36.5,
"p95": 57.6,
"p99": 83.2,
"n": 19
},
"mac": {
"p50": 14.6,
"p95": 31.3,
"p99": 31.7,
"n": 19
},
"back": {
"p50": 15.1,
"p95": 32.8,
"p99": 36.2,
"n": 19
}
},
"inputs": {
"asked": 19,
"sent": 19,
"seen": 19
},
"timeline": [
{
"t": 0,
"fps": 40,
"mbps": 0.34,
"content_p50": 11.7,
"content_p95": 27.4,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 1,
"fps": 39,
"mbps": 0.37,
"content_p50": 11.9,
"content_p95": 28.2,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 2,
"fps": 41,
"mbps": 0.35,
"content_p50": 11.4,
"content_p95": 16.6,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 3,
"fps": 39,
"mbps": 0.37,
"content_p50": 12.7,
"content_p95": 28.0,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 4,
"fps": 38,
"mbps": 0.33,
"content_p50": 15.3,
"content_p95": 30.0,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 5,
"fps": 39,
"mbps": 0.37,
"content_p50": 16.7,
"content_p95": 28.5,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 6,
"fps": 41,
"mbps": 0.38,
"content_p50": 15.9,
"content_p95": 29.4,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 7,
"fps": 41,
"mbps": 0.31,
"content_p50": 15.6,
"content_p95": 28.2,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 8,
"fps": 38,
"mbps": 0.38,
"content_p50": 15.5,
"content_p95": 25.9,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 9,
"fps": 38,
"mbps": 0.29,
"content_p50": 16.7,
"content_p95": 28.8,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
},
{
"t": 10,
"fps": 6,
"mbps": 0.03,
"content_p50": 17.6,
"content_p95": 23.1,
"bitrate": 5529600,
"tier": 0,
"w": 1280,
"net": null
}
],
"adapt": [],
"content_p50": 15.1,
"content_p95": 28.6,
"input_p50": 76.6,
"input_p95": 98.2,
"grades": {
"input_replies": "local",
"content_p50": "local",
"content_p95": "local",
"input_p50": "acceptable",
"input_p95": "acceptable",
"fps": "bad",
"late_pct": "bad",
"stall_max": "local"
},
"controller": {
"target": 5529600,
"ceiling": 5529600,
"tier": 0,
"fps": 60,
"scale": 1.0,
"baseRtt": 12.332,
"inFlight": 0,
"slack": 74.12,
"adapt": true
},
"events": [],
"source_fps": 60.0,
"cpu": {
"host": "not sampled"
},
"viewer": "h264 software; ANGLE (Mesa, zink Vulkan 1.4(Turnip Adreno (TM) 750 (MESA_TURNIP)), OpenGL 4.6); raf 72 Hz",
"show_s": 1.75,
"panel": "valve.steam.desktopgame.2001932457",
"src": "test",
"route": "network"
}
],
"proof": "Synthetic x264 PC agent runs on the Frame itself, forwarded via Mac and back. Not Windows/Linux desktop capture or laser input."
}
-16
View File
@@ -1,16 +0,0 @@
#!/usr/bin/env python3
"""Build the common controller for the Python PC host (a C11 compiler needed)."""
import os
from pathlib import Path
import subprocess
import sys
root = Path(__file__).resolve().parent
suffix = '.dll' if sys.platform == 'win32' else '.dylib' if sys.platform == 'darwin' else '.so'
out = root / ('controller' + suffix)
if sys.platform == 'win32' and os.environ.get('CC', 'cl') == 'cl':
cmd = ['cl', '/nologo', '/O2', '/LD', '/std:c11', str(root / 'controller.c'), '/link', '/OUT:' + str(out)]
else:
cmd = [os.environ.get('CC', 'cc'), '-O2', '-std=c11', '-shared', '-fPIC', str(root / 'controller.c'), '-o', str(out)]
subprocess.run(cmd, cwd=root, check=True)
print(out)
-118
View File
@@ -1,118 +0,0 @@
#!/usr/bin/env python3
"""Build our native PC adapter and bundle its ordinary shared libraries.
Run on the target architecture after installing GStreamer development packages.
No GStreamer executable or desktop streaming application is shipped or invoked.
Linux libc, display/GPU drivers and the portal service remain platform pieces.
"""
import argparse
import json
import os
from pathlib import Path
import re
import shutil
import subprocess
import sys
ROOT = Path(__file__).resolve().parent
def output(*args):
return subprocess.check_output(args, text=True).strip()
def build(destination):
win = sys.platform == 'win32'
destination.mkdir(parents=True, exist_ok=True)
modules = ['gstreamer-1.0', 'gstreamer-app-1.0', 'gstreamer-video-1.0']
if not win:
modules += ['gio-unix-2.0']
import shlex
flags = shlex.split(output('pkg-config', '--cflags', '--libs', *modules))
lib = destination / ('pc-host.dll' if win else 'pc-host.so')
cmd = [os.environ.get('CC', 'gcc' if win else 'cc'), '-std=c11', '-O2', '-shared', '-Wall', '-Wextra']
if not win:
cmd += ['-fPIC', '-Wl,-rpath,$ORIGIN/lib']
subprocess.run(cmd + [str(ROOT / f) for f in ('controller.c', 'capture.c', 'portal.c')] + flags + ['-o', str(lib)], check=True)
prefix = Path(output('pkg-config', '--variable=prefix', 'gstreamer-1.0'))
plugin_dir = Path(output('pkg-config', '--variable=pluginsdir', 'gstreamer-1.0'))
plugins_out, libs_out = destination / 'lib' / 'gstreamer-1.0', destination / ('bin' if win else 'lib')
plugins_out.mkdir(parents=True, exist_ok=True)
libs_out.mkdir(parents=True, exist_ok=True)
names = ['coreelements', 'videotestsrc', 'videoconvertscale', 'app', 'jpeg', 'videoparsersbad', 'x264']
names += ['d3d11', 'mediafoundation'] if win else ['pipewire', 'va']
pending = [lib]
for name in names:
matches = list(plugin_dir.glob('*gst' + name + ('.dll' if win else '.so')))
if not matches:
raise SystemExit('Missing GStreamer library: ' + name)
for source in matches:
target = plugins_out / source.name
shutil.copy2(source, target)
pending.append(source)
if not win:
# PipeWire dynamically loads support and protocol modules. ldd cannot
# discover those; bundle them with a private client-only config instead
# of accidentally loading a different distro's client modules/config.
libdir = plugin_dir.parent
for relative in ('pipewire-0.3/libpipewire-module-protocol-native.so',
'pipewire-0.3/libpipewire-module-client-node.so',
'pipewire-0.3/libpipewire-module-adapter.so',
'spa-0.2/support/libspa-support.so',
'spa-0.2/support/libspa-dbus.so',
'spa-0.2/videoconvert/libspa-videoconvert.so'):
source = libdir / relative
if not source.is_file():
raise SystemExit('Missing PipeWire runtime library: ' + str(source))
target = destination / 'lib' / relative
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(source, target)
pending.append(source)
config = destination / 'share' / 'pipewire'
config.mkdir(parents=True, exist_ok=True)
(config / 'client.conf').write_text(
'context.spa-libs = { support.* = support/libspa-support video.* = videoconvert/libspa-videoconvert }\n'
'context.modules = [\n'
' { name = libpipewire-module-protocol-native }\n'
' { name = libpipewire-module-client-node }\n'
' { name = libpipewire-module-adapter }\n'
']\n')
copied = set()
while pending:
binary = pending.pop()
if win:
imported = re.findall(r'DLL Name:\s*(\S+)', output('objdump', '-p', str(binary)))
dependencies = [prefix / 'bin' / name for name in imported]
else:
dependencies = [Path(p) for p in re.findall(r'=>\s+(/\S+)', output('ldd', str(binary)))]
for dep in dependencies:
if not dep.is_file() or dep.name in copied:
continue
if not win and re.match(r'lib(c|m|dl|rt|pthread|resolv)\.so', dep.name):
continue
shutil.copy2(dep, libs_out / dep.name)
copied.add(dep.name)
pending.append(dep)
# License texts and package provenance accompany the dynamically linked
# libraries. Distribution builders keep the upstream package/source URLs.
licenses = destination / 'licenses'
licenses.mkdir(exist_ok=True)
if win:
source = prefix / 'share' / 'licenses'
if source.exists():
shutil.copytree(source, licenses, dirs_exist_ok=True)
packages = output('pacman', '-Q') if shutil.which('pacman') else 'See MSYS2 build log'
else:
packages = output('dpkg-query', '-W', '-f=${Package} ${Version}\n') if shutil.which('dpkg-query') else ''
# Debian copyright files contain licenses and source homepage details.
for path in Path('/usr/share/doc').glob('*/copyright'):
shutil.copy2(path, licenses / (path.parent.name + '.copyright'))
(destination / 'libraries.json').write_text(json.dumps(dict(gstreamer=output('pkg-config', '--modversion', 'gstreamer-1.0'),
libraries=sorted(copied), packages=packages), indent=2))
print(lib)
if __name__ == '__main__':
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--out', type=Path, default=ROOT / 'bundle')
build(parser.parse_args().out.resolve())
-125
View File
@@ -1,125 +0,0 @@
/* Frame Control's capture/encode adapter. GStreamer is a bundled library;
* capture uses WGC on Windows and the consented PipeWire fd on Linux. */
#include "controller.h"
#include <gst/gst.h>
#include <gst/app/gstappsink.h>
#include <gst/video/video-event.h>
#include <string.h>
typedef int (*Gate)(int stage, int64_t pts, int64_t capture, int64_t arrived);
typedef struct {
GstElement *pipeline, *encoder, *sink, *raw_queue;
Gate gate;
GstSample *sample;
GstMapInfo map;
int mapped;
char error[512];
} Capture;
typedef struct { const unsigned char *data; int size, key, width, height; int64_t pts; } Encoded;
FC_API int64_t fc_now(void) {return g_get_monotonic_time();}
FC_API void fc_gst_init(void) {gst_init(NULL,NULL);}
FC_API int fc_has_element(const char *name) {
GstElementFactory *f=gst_element_factory_find(name);
if(!f)return 0;gst_object_unref(f);return 1;
}
static GstPadProbeReturn probe(GstPad *pad,GstPadProbeInfo *info,gpointer data) {
Capture *c=data; GstBuffer *b=GST_PAD_PROBE_INFO_BUFFER(info);
if(!b)return GST_PAD_PROBE_OK;
int64_t now=fc_now(), cap=now;
GstClock *clock=gst_element_get_clock(c->pipeline);
if(clock && GST_BUFFER_PTS_IS_VALID(b)) {
GstClockTime current=gst_clock_get_time(clock),origin=gst_element_get_base_time(c->pipeline);
if(current>=origin && current-origin>=GST_BUFFER_PTS(b))
cap-= (int64_t)((current-origin-GST_BUFFER_PTS(b))/1000);
}
if(clock)gst_object_unref(clock);
int stage=GPOINTER_TO_INT(g_object_get_data(G_OBJECT(pad),"stage"));
if(stage==1) return c->gate(stage,(int64_t)GST_BUFFER_PTS(b),cap,now)>0 ? GST_PAD_PROBE_OK : GST_PAD_PROBE_DROP;
int phase=0;
for(;;) {
/* While congested, prefer a newer raw picture queued upstream. If
* nothing changed, retain this last picture until the gate opens;
* an idle window must not stay stale after a dropped final update. */
guint queued=0;
if(phase && c->raw_queue)g_object_get(c->raw_queue,"current-level-buffers",&queued,NULL);
if(queued)return GST_PAD_PROBE_DROP;
int decision=c->gate(phase,(int64_t)GST_BUFFER_PTS(b),cap,now);
if(decision)return decision>0 ? GST_PAD_PROBE_OK : GST_PAD_PROBE_DROP;
phase=2;g_usleep(2000);
}
}
FC_API Capture *fc_capture_open(const char *pipeline,Gate gate,char *error,int capacity) {
GError *e=NULL;Capture *c=g_new0(Capture,1);c->gate=gate;
c->pipeline=gst_parse_launch(pipeline,&e);
if(e || !c->pipeline) {
g_strlcpy(error,e?e->message:"No pipeline",capacity);
if(e)g_error_free(e);if(c->pipeline)gst_object_unref(c->pipeline);g_free(c);return NULL;
}
c->encoder=gst_bin_get_by_name(GST_BIN(c->pipeline),"enc");
c->sink=gst_bin_get_by_name(GST_BIN(c->pipeline),"out");
GstElement *raw=gst_bin_get_by_name(GST_BIN(c->pipeline),"gate");
if(!c->encoder || !c->sink || !raw) {
g_strlcpy(error,"Pipeline is missing enc, out or gate",capacity);
if(raw)gst_object_unref(raw);if(c->encoder)gst_object_unref(c->encoder);
if(c->sink)gst_object_unref(c->sink);gst_object_unref(c->pipeline);g_free(c);return NULL;
}
c->raw_queue=gst_bin_get_by_name(GST_BIN(c->pipeline),"raw_queue");
GstPad *p=gst_element_get_static_pad(raw,"src");
g_object_set_data(G_OBJECT(p),"stage",GINT_TO_POINTER(0));
gst_pad_add_probe(p,GST_PAD_PROBE_TYPE_BUFFER,probe,c,NULL);gst_object_unref(p);gst_object_unref(raw);
p=gst_element_get_static_pad(c->encoder,"sink");
g_object_set_data(G_OBJECT(p),"stage",GINT_TO_POINTER(1));
gst_pad_add_probe(p,GST_PAD_PROBE_TYPE_BUFFER,probe,c,NULL);gst_object_unref(p);
gst_element_set_state(c->pipeline,GST_STATE_PLAYING);
return c;
}
FC_API int fc_capture_pull(Capture *c,Encoded *out) {
if(c->mapped) {gst_buffer_unmap(gst_sample_get_buffer(c->sample),&c->map);c->mapped=0;}
if(c->sample){gst_sample_unref(c->sample);c->sample=NULL;}
GstBus *bus=gst_element_get_bus(c->pipeline);
GstMessage *m=gst_bus_pop_filtered(bus,GST_MESSAGE_ERROR);gst_object_unref(bus);
if(m) {
if(GST_MESSAGE_TYPE(m)==GST_MESSAGE_ERROR) {
GError *e=NULL;char *debug=NULL;gst_message_parse_error(m,&e,&debug);
g_strlcpy(c->error,e->message,sizeof(c->error));g_error_free(e);g_free(debug);
} else g_strlcpy(c->error,"The capture source closed",sizeof(c->error));
gst_message_unref(m);return -1;
}
c->sample=gst_app_sink_try_pull_sample(GST_APP_SINK(c->sink),100*GST_MSECOND);
if(!c->sample) {
if(gst_app_sink_is_eos(GST_APP_SINK(c->sink))) {
g_strlcpy(c->error,"The capture source closed",sizeof(c->error));return -1;
}
return 0;
}
GstBuffer *b=gst_sample_get_buffer(c->sample);
if(!gst_buffer_map(b,&c->map,GST_MAP_READ))return 0;
c->mapped=1;out->data=c->map.data;out->size=(int)c->map.size;
out->key=!GST_BUFFER_FLAG_IS_SET(b,GST_BUFFER_FLAG_DELTA_UNIT);out->pts=(int64_t)GST_BUFFER_PTS(b);
const GstStructure *s=gst_caps_get_structure(gst_sample_get_caps(c->sample),0);
gst_structure_get_int(s,"width",&out->width);gst_structure_get_int(s,"height",&out->height);return 1;
}
FC_API const char *fc_capture_error(Capture *c) {return c->error;}
FC_API void fc_capture_bitrate(Capture *c,int bps) {
g_object_set(c->encoder,"bitrate",(guint)MAX(1,bps/1000),NULL);
}
FC_API void fc_capture_test(Capture *c,uint32_t input) {
GstElement *source=gst_bin_get_by_name(GST_BIN(c->pipeline),"source");
if(source) {
/* Each benchmark click visibly changes the ball, before injection is
* timestamped. This is a response, not merely a protocol input echo. */
guint color=0xff000000u | ((input*2654435761u)&0x00ffffffu);
g_object_set(source,"foreground-color",color,NULL);gst_object_unref(source);
}
}
FC_API void fc_capture_key(Capture *c) {
GstPad *p=gst_element_get_static_pad(c->encoder,"src");
gst_pad_send_event(p,gst_video_event_new_upstream_force_key_unit(GST_CLOCK_TIME_NONE,TRUE,0));gst_object_unref(p);
}
FC_API void fc_capture_close(Capture *c) {
gst_element_set_state(c->pipeline,GST_STATE_NULL);
if(c->mapped)gst_buffer_unmap(gst_sample_get_buffer(c->sample),&c->map);
if(c->sample)gst_sample_unref(c->sample);
if(c->raw_queue)gst_object_unref(c->raw_queue);
gst_object_unref(c->sink);gst_object_unref(c->encoder);gst_object_unref(c->pipeline);g_free(c);
}
-127
View File
@@ -1,127 +0,0 @@
/* Extracted from the Mac agent's RateController. Same gate, demand protection,
* bitrate recovery and tier hysteresis for every host. Bounded sample buffers. */
#include "controller.h"
#include <stdlib.h>
#include <string.h>
#define CAP 1024
#define MIN(a,b) ((a)<(b)?(a):(b))
#define MAX(a,b) ((a)>(b)?(a):(b))
typedef struct { int64_t t, v; uint32_t seq; } Sample;
typedef struct { Sample a[CAP]; int n; } Samples;
struct FCController {
int max_fps, enabled, ceiling, target, tier, frame_bytes, held, signal, saw_ack;
int64_t decrease, increase, below, above, slack;
Samples flight, rtts, acked, sent, captures;
};
static const int fps[] = {60,45,30,30,30}, scale[] = {100,100,100,75,50};
static const double floors[] = {.45,.28,.16,.08,0};
static void remove_first(Samples *s, int n) {
s->n -= n; memmove(s->a, s->a+n, (size_t)s->n*sizeof(Sample));
}
static void add(Samples *s, Sample v, int cap) {
if (s->n >= cap) remove_first(s, 1);
s->a[s->n++] = v;
}
static void expire(Samples *s, int64_t oldest) {
int n=0; while(n<s->n && s->a[n].t<oldest) n++;
remove_first(s,n);
}
static int64_t base(FCController *c) {
int64_t v=0;
for(int i=0;i<c->rtts.n;i++) if(!i || c->rtts.a[i].v<v) v=c->rtts.a[i].v;
return v;
}
static int compare(const void *a, const void *b) {
int64_t x=*(const int64_t*)a, y=*(const int64_t*)b;
return (x>y)-(x<y);
}
static int64_t quantile(Samples *s,int64_t since,int numerator,int denominator) {
int64_t a[CAP]; int n=0;
for(int i=0;i<s->n;i++) if(s->a[i].t>since) a[n++]=s->a[i].v;
if(!n) return 0;
qsort(a,(size_t)n,sizeof(int64_t),compare);
return a[n*numerator/denominator];
}
static int rate(Samples *s) {
int64_t total=0; for(int i=0;i<s->n;i++) total+=s->a[i].v;
return (int)MIN(total*16,2147483647);
}
FCController *fc_new(int max_fps,int enabled) {
FCController *c=calloc(1,sizeof(*c));
if(c) {c->max_fps=MAX(1,max_fps);c->enabled=enabled;c->slack=40000;}
return c;
}
void fc_free(FCController *c) {free(c);}
void fc_ceiling(FCController *c,int bps) {
if(!c->target || c->target>bps) c->target=bps;
c->ceiling=bps;
}
int fc_gate(FCController *c,int64_t now,int counts) {
if(!c->enabled || !c->saw_ack) return 1;
expire(&c->flight,now-2000000);
if(!c->flight.n) return 1;
int64_t interval=1000000/MIN(c->max_fps,fps[c->tier]);
int window=MAX(3,(int)((base(c)+c->slack)/interval)+1);
if(c->flight.n<window && now-c->flight.a[0].t<=base(c)+c->slack) return 1;
if(counts) c->held++;
return 0;
}
void fc_capture(FCController *c,int64_t now) {add(&c->captures,(Sample){now,0,0},256);}
void fc_sent(FCController *c,uint32_t seq,int bytes,int64_t now) {
Sample s={now,bytes,seq};add(&c->flight,s,512);add(&c->sent,s,CAP);
}
int fc_ack(FCController *c,uint32_t seq,int64_t now) {
c->saw_ack=1;
for(int i=0;i<c->flight.n;i++) if(c->flight.a[i].seq==seq) {
Sample s=c->flight.a[i];remove_first(&c->flight,i+1);
add(&c->rtts,(Sample){now,now-s.t,0},CAP);
add(&c->acked,(Sample){now,s.v,0},CAP);return 1;
}
return 0;
}
int fc_update(FCController *c,int64_t now) {
expire(&c->rtts,now-10000000);expire(&c->acked,now-500000);
expire(&c->sent,now-500000);expire(&c->flight,now-2000000);
expire(&c->captures,now-1000000);
if(!c->enabled || c->ceiling<=0) return 0;
int64_t baseline=base(c), spread=quantile(&c->rtts,now-2000000,9,10);
int64_t jitter=spread ? spread-baseline : 0;
c->slack=1000000/MIN(c->max_fps,fps[c->tier])+MIN(MAX(jitter*3/2,25000),80000);
int64_t recent=quantile(&c->rtts,now-300000,1,2);
int64_t queue=recent ? recent-baseline : 0;
int64_t age=c->flight.n ? now-c->flight.a[0].t : 0;
int delivered=rate(&c->acked),sending=rate(&c->sent);
if(c->sent.n) c->frame_bytes=sending/16/c->sent.n;
int demand=(int)MIN((int64_t)MIN(c->captures.n,MIN(c->max_fps,fps[c->tier]))*c->frame_bytes*8,2147483647);
int signal=(sending>=c->target/2 && queue>40000)||c->held>=3||age>baseline+100000;
int congested=signal && c->signal;c->signal=signal;c->held=0;
if(congested && now-c->decrease>300000) {
int next=MAX(300000,MIN((int64_t)c->target*4/5,MAX((int64_t)delivered*9/10,c->target/2)));
if(demand>0 && (int64_t)demand*2<=(int64_t)c->target*5/4) next=MAX(next,MIN(c->target,(int64_t)demand*2));
c->target=next;c->decrease=now;
} else if(!congested && now-c->decrease>1000000 && now-c->increase>250000 && c->target<c->ceiling &&
(sending>(int64_t)c->target*6/10 || now-c->decrease>3000000)) {
c->target=MIN(c->ceiling,(int64_t)(c->target*1.1)+50000);c->increase=now;
}
double share=(double)c->target/MAX(c->ceiling,1);
if(c->tier<4 && share<floors[c->tier] && sending>=(int64_t)c->target*7/10) {
if(!c->below)c->below=now;
if(now-c->below>500000) {
for(c->tier=0;c->tier<4 && share<floors[c->tier];c->tier++) {}
c->below=0;
}
} else c->below=0;
if(c->tier>0 && share>floors[c->tier-1]*1.25) {
if(!c->above)c->above=now;
if(now-c->above>2000000) {c->tier--;c->above=0;}
} else c->above=0;
return c->target;
}
int64_t fc_value(FCController *c,int field) {
switch(field) {
case 0:return c->target;case 1:return c->ceiling;case 2:return c->tier;
case 3:return MIN(c->max_fps,fps[c->tier]);case 4:return scale[c->tier];
case 5:return base(c);case 6:return c->flight.n;case 7:return c->slack;
default:return 0;
}
}
-29
View File
@@ -1,29 +0,0 @@
/* Shared latency controller. Times are monotonic host microseconds.
* Callers serialize access. Capture is gated BEFORE encoding: dropping an
* encoded inter frame would break the decoder's reference chain. */
#ifndef FRAME_CONTROLLER_H
#define FRAME_CONTROLLER_H
#include <stdint.h>
#ifdef _WIN32
#define FC_API __declspec(dllexport)
#else
#define FC_API
#endif
#ifdef __cplusplus
extern "C" {
#endif
typedef struct FCController FCController;
FC_API FCController *fc_new(int fps, int enabled);
FC_API void fc_free(FCController *c);
FC_API void fc_ceiling(FCController *c, int bps);
FC_API int fc_gate(FCController *c, int64_t now, int counts);
FC_API void fc_capture(FCController *c, int64_t now);
FC_API void fc_sent(FCController *c, uint32_t seq, int bytes, int64_t now);
FC_API int fc_ack(FCController *c, uint32_t seq, int64_t now);
FC_API int fc_update(FCController *c, int64_t now);
/* target, ceiling, tier, fps, scale percent, baseline us, in flight, slack us */
FC_API int64_t fc_value(FCController *c, int field);
#ifdef __cplusplus
}
#endif
#endif
-147
View File
@@ -1,147 +0,0 @@
/* xdg-desktop-portal RemoteDesktop + ScreenCast, one consented session per
* panel. PipeWire receives the portal fd, never the unrestricted daemon fd.
* Input uses only the devices granted in Start's response. */
#ifndef _WIN32
#include "controller.h"
#include <gio/gio.h>
#include <gio/gunixfdlist.h>
#include <unistd.h>
#include <string.h>
#define BUS "org.freedesktop.portal.Desktop"
#define PATH "/org/freedesktop/portal/desktop"
#define RD "org.freedesktop.portal.RemoteDesktop"
#define SC "org.freedesktop.portal.ScreenCast"
typedef struct {
GDBusConnection *bus;
GMainContext *context;
char *session;
int fd;
unsigned node, devices;
int width,height;
} Portal;
typedef struct {GVariant *result;int done;unsigned code;} Response;
static void response(GDBusConnection *c,const char *sender,const char *path,const char *iface,
const char *signal,GVariant *params,gpointer data) {
(void)c;(void)sender;(void)path;(void)iface;(void)signal;
Response *r=data;g_variant_get(params,"(u@a{sv})",&r->code,&r->result);r->done=1;
}
static void option(GVariantBuilder *b,const char *key,GVariant *v) {g_variant_builder_add(b,"{sv}",key,v);}
static GVariant *request(Portal *p,const char *iface,const char *method,GVariant *args,
const char *token,char *error,int capacity) {
char *sender=g_strdup(g_dbus_connection_get_unique_name(p->bus)+1);
for(char *c=sender;*c;c++)if(*c=='.')*c='_';
char *path=g_strdup_printf(PATH "/request/%s/%s",sender,token);g_free(sender);
Response r={0};GError *e=NULL;
guint sub=g_dbus_connection_signal_subscribe(p->bus,BUS,"org.freedesktop.portal.Request","Response",
path,NULL,G_DBUS_SIGNAL_FLAGS_NONE,response,&r,NULL);
GVariant *reply=g_dbus_connection_call_sync(p->bus,BUS,PATH,iface,method,args,G_VARIANT_TYPE("(o)"),
G_DBUS_CALL_FLAGS_NONE,10000,NULL,&e);
if(reply)g_variant_unref(reply);
int64_t deadline=g_get_monotonic_time()+120000000;
while(!e && !r.done && g_get_monotonic_time()<deadline) {
while(g_main_context_iteration(p->context,FALSE)) {}
g_usleep(10000);
}
if(!r.done) {
GVariant *closed=g_dbus_connection_call_sync(p->bus,BUS,path,"org.freedesktop.portal.Request","Close",
NULL,NULL,G_DBUS_CALL_FLAGS_NONE,2000,NULL,NULL);
if(closed)g_variant_unref(closed);
}
g_dbus_connection_signal_unsubscribe(p->bus,sub);g_free(path);
if(e || !r.done || r.code) {
g_strlcpy(error,e?e->message:!r.done?"Screen sharing request timed out":"Screen sharing was cancelled or refused",capacity);
if(e)g_error_free(e);if(r.result)g_variant_unref(r.result);return NULL;
}
return r.result;
}
FC_API void fc_portal_close(Portal *p) {
if(!p)return;
if(p->session && p->bus) {
GVariant *r=g_dbus_connection_call_sync(p->bus,BUS,p->session,"org.freedesktop.portal.Session","Close",
NULL,NULL,G_DBUS_CALL_FLAGS_NONE,2000,NULL,NULL);
if(r)g_variant_unref(r);
}
if(p->fd>=0){close(p->fd);p->fd=-1;}
g_free(p->session);if(p->bus)g_object_unref(p->bus);
if(p->context)g_main_context_unref(p->context);g_free(p);
}
/* Each pipeline gets a fresh restricted PipeWire connection. A dup of a
* previously consumed protocol socket is not a new connection.
* Ownership: the Portal owns p->fd for its whole life and is its only closer.
* pipewiresrc never takes the fd it is given: its core connects with
* pw_context_connect_fd(ctx, fcntl(fd, F_DUPFD_CLOEXEC, 3), ...) and that
* duplicate is what PipeWire closes on teardown (src/gst/gstpipewirecore.c,
* unchanged from 0.3.19 through 1.x). So closing p->fd here is not a double
* close; not closing it would leak one socket per pipeline reopen. The
* caller closes the previous pipeline before asking for a new fd. */
FC_API int fc_portal_refresh(Portal *p,char *error,int capacity) {
GVariantBuilder b;g_variant_builder_init(&b,G_VARIANT_TYPE_VARDICT);
GUnixFDList *fds=NULL;GError *e=NULL;
GVariant *r=g_dbus_connection_call_with_unix_fd_list_sync(p->bus,BUS,PATH,SC,"OpenPipeWireRemote",
g_variant_new("(oa{sv})",p->session,&b),G_VARIANT_TYPE("(h)"),G_DBUS_CALL_FLAGS_NONE,10000,NULL,&fds,NULL,&e);
if(!r){g_strlcpy(error,e->message,capacity);g_error_free(e);return -1;}
int handle;g_variant_get(r,"(h)",&handle);g_variant_unref(r);
int fd=g_unix_fd_list_get(fds,handle,&e);g_object_unref(fds);
if(fd<0){g_strlcpy(error,e->message,capacity);g_error_free(e);return -1;}
if(p->fd>=0)close(p->fd);
p->fd=fd;return fd;
}
FC_API Portal *fc_portal_select(char *error,int capacity) {
Portal *p=g_new0(Portal,1);p->fd=-1;p->context=g_main_context_new();
g_main_context_push_thread_default(p->context);
GError *e=NULL;GVariant *r=NULL;
p->bus=g_bus_get_sync(G_BUS_TYPE_SESSION,NULL,&e);
if(!p->bus) {g_strlcpy(error,e->message,capacity);g_error_free(e);goto fail;}
GVariantBuilder b;char token[64],session[64];
g_snprintf(session,sizeof(session),"fc_%u",g_random_int());
g_snprintf(token,sizeof(token),"fc_%u",g_random_int());
g_variant_builder_init(&b,G_VARIANT_TYPE_VARDICT);
option(&b,"handle_token",g_variant_new_string(token));option(&b,"session_handle_token",g_variant_new_string(session));
r=request(p,RD,"CreateSession",g_variant_new("(a{sv})",&b),token,error,capacity);if(!r)goto fail;
g_variant_lookup(r,"session_handle","s",&p->session);g_variant_unref(r);r=NULL;
if(!p->session){g_strlcpy(error,"Portal did not return a session",capacity);goto fail;}
g_snprintf(token,sizeof(token),"fc_%u",g_random_int());g_variant_builder_init(&b,G_VARIANT_TYPE_VARDICT);
option(&b,"handle_token",g_variant_new_string(token));option(&b,"types",g_variant_new_uint32(3));
r=request(p,RD,"SelectDevices",g_variant_new("(oa{sv})",p->session,&b),token,error,capacity);if(!r)goto fail;g_variant_unref(r);
g_snprintf(token,sizeof(token),"fc_%u",g_random_int());g_variant_builder_init(&b,G_VARIANT_TYPE_VARDICT);
option(&b,"handle_token",g_variant_new_string(token));option(&b,"types",g_variant_new_uint32(3));
option(&b,"multiple",g_variant_new_boolean(FALSE));option(&b,"cursor_mode",g_variant_new_uint32(2));
r=request(p,SC,"SelectSources",g_variant_new("(oa{sv})",p->session,&b),token,error,capacity);if(!r)goto fail;g_variant_unref(r);
g_snprintf(token,sizeof(token),"fc_%u",g_random_int());g_variant_builder_init(&b,G_VARIANT_TYPE_VARDICT);
option(&b,"handle_token",g_variant_new_string(token));
r=request(p,RD,"Start",g_variant_new("(osa{sv})",p->session,"",&b),token,error,capacity);if(!r)goto fail;
g_variant_lookup(r,"devices","u",&p->devices);
GVariant *streams=g_variant_lookup_value(r,"streams",G_VARIANT_TYPE("a(ua{sv})"));
if(streams && g_variant_n_children(streams)>0) {
GVariant *entry=g_variant_get_child_value(streams,0),*props=NULL;
g_variant_get(entry,"(u@a{sv})",&p->node,&props);
g_variant_lookup(props,"size","(ii)",&p->width,&p->height);
g_variant_unref(props);g_variant_unref(entry);
}
if(streams)g_variant_unref(streams);g_variant_unref(r);r=NULL;
if(!p->node || p->width<=0 || p->height<=0) {g_strlcpy(error,"Portal returned no stream size",capacity);goto fail;}
if(fc_portal_refresh(p,error,capacity)<0)goto fail;
g_main_context_pop_thread_default(p->context);return p;
fail:
g_main_context_pop_thread_default(p->context);fc_portal_close(p);return NULL;
}
FC_API int fc_portal_value(Portal *p,int field) {
switch(field){case 0:return p->fd;case 1:return (int)p->node;case 2:return p->width;
case 3:return p->height;case 4:return (int)p->devices;default:return 0;}
}
FC_API int fc_portal_input(Portal *p,int type,double x,double y,int code,int down) {
const char *method=NULL;GVariant *args=NULL;GVariantBuilder b;g_variant_builder_init(&b,G_VARIANT_TYPE_VARDICT);
if(type==0 && (p->devices&2)) {
method="NotifyPointerMotionAbsolute";args=g_variant_new("(oa{sv}udd)",p->session,&b,p->node,x,y);
} else if(type==1 && (p->devices&2)) {
method="NotifyPointerButton";args=g_variant_new("(oa{sv}iu)",p->session,&b,code,(guint)down);
} else if(type==2 && (p->devices&2)) {
method="NotifyPointerAxis";args=g_variant_new("(oa{sv}dd)",p->session,&b,x,y);
} else if(type==3 && (p->devices&1)) {
method="NotifyKeyboardKeysym";args=g_variant_new("(oa{sv}iu)",p->session,&b,code,(guint)down);
} else return 0;
GError *e=NULL;GVariant *r=g_dbus_connection_call_sync(p->bus,BUS,PATH,RD,method,args,NULL,
G_DBUS_CALL_FLAGS_NONE,2000,NULL,&e);
if(e)g_error_free(e);if(r)g_variant_unref(r);return r!=NULL;
}
#endif
+122
View File
@@ -0,0 +1,122 @@
# Family and comfort
Frame Control's Home tab has a **Family and comfort** card, on desktop and
on iPhone. No third-party notification or parental-control app is needed.
This is Frame Control code using Python, Steam and SteamVR already on the Frame.
![Family and comfort controls in the desktop app](img/comfort-desktop.png)
## Sessions
Set a limit of 1–240 minutes, optional break and check-in intervals, then
**Start session**. Break and check-in intervals of 0 turn those reminders off.
**Cancel session** cancels the timer and monitoring without changing the game.
Cancel before starting a session with different settings.
The Frame shows a one-minute warning, then opens Steam Home in its dashboard.
**Games stay running**: save and pause before the limit. Some games pause when
the dashboard opens; others do not. There is no kill, power-off, Steam restart,
account restriction or parental lock. The wearer can return to the game.
**Documented implementation:** the timer is a single, opt-in Python worker in
the Frame user's account. Desktop and iPhone share its state. It keeps going
when the companion disconnects, closes or is suspended. It exits after
completion or cancellation (normally within five seconds). Cancellation waits
for any in-flight SteamVR action to finish within its timeout; it is not a boot
service. A Frame reboot invalidates the session. Suspend counts toward the
limit, using Linux's boot-time clock. If a warning was delayed by suspend or a
SteamVR failure, Home waits until at least a full minute after a successful
warning. A failed Home transition remains active and retries, with an error
shown in the companion. A stale worker is reported as unverified enforcement.
## Alerts and breaks
During a session, battery, overheating and check-in alerts go to connected
companions. Break reminders and session warnings also appear on the headset.
- **Low battery:** 15% or below while discharging. One alert until charging or
recovery to 20%, so values around 15% do not produce repeated notifications.
- **Overheating:** a thermal zone reaches its own kernel-reported hot/critical
trip, or the battery reports `Overheat`. Missing sensors mean unknown, not
safe. These are status alerts, not medical advice or an extra thermal governor.
- **Check in:** an alert after the chosen number of active minutes.
- **Breaks:** a SteamVR reminder and companion notification at the chosen interval.
**Inferred:** SteamVR activity levels 1 and 2 are a useful proxy for use, not
proof someone is wearing the headset. Inactive readings reset continuous use;
missing readings add no time. Long gaps count at most 30 seconds. Breaks and
check-ins are distinct from the elapsed-time session limit.
Click **Enable / test notifications** on each companion. iOS asks for permission;
macOS, Windows and Linux follow their notification settings. The page also shows
recent events and errors. Keep Frame Control open and connected for companion
alerts. **Phone alerts are local, not push notifications:** iOS suspension,
force-quit or a lost SSH connection prevents live delivery. Old alerts are not
replayed as a notification burst on reconnect. Headset warnings and the session
limit continue without the phone. A physical iPhone's background delivery has
not been verified and is not guaranteed.
## Casting
**Cast headset view** starts the existing headset Live view and requests full
screen where supported. Show that screen to people in the room, or use the
computer/phone's own screen mirroring. It creates no new stream transport,
public URL or LAN server. iPhone uses the inline viewer if full screen is not
available. The image includes private content visible to the wearer.
## What is installed
The shared authenticated `/api/comfort` endpoint copies three bundled Python
files to `~/.cache/frame-control/comfort/<content-hash>/`. Session state and
locks live in `~/.local/state/frame-control/comfort/`, with a private directory
and 0600 state file. There is no network listener or system service. Cancel a
session before removing these directories. The iPhone's normal server still
exits on disconnect; the explicitly started comfort worker is the exception.
## Verification
**Verified 2026-09-28**, SteamOS 0.4.1, build `20260925.6191901`: shipped
`/opt/steamvr/bin/linuxarm64/vrcmd --notify TEXT` reported success for a custom
reminder. Steam's CDP `SteamUIStore.Navigate('/library/home')` and
`SteamClient.OpenVR.VROverlay.ShowDashboard('valve.steam.gamepadui.main')`
opened Home while the running app ID stayed unchanged. Prior page and dashboard
visibility were restored. Kernel hot/critical trips and SteamVR activity were
read from the real device. No temperature or battery fault was induced.
**Verified locally:** deterministic fake-Frame tests cover late warnings,
failed warnings/Home actions, cancellation, activity gaps, thresholds, duplicate
suppression, reboot invalidation, shared session state and the exact Home
JavaScript. `python3 -m unittest discover -s tests` runs them. The iOS Simulator
build tests notification content and bounds. Physical iPhone delivery and
wearer-perceived headset notification visibility remain unverified.
**Verified end to end on the same Frame:** a two-minute session with no companion
connection for 135 seconds emitted its warning, break and check-in, then opened
Home. The running app ID was unchanged; the test restored the previous page and
dashboard visibility and confirmed the worker exited. Casting through the Home
shortcut decoded the existing headset stream at 30 fps.
**Verified on the iOS 26.5 Simulator:** connected to the real Frame, approved the
notification prompt, and saw the native Frame Control test banner. Seven iOS
tests passed.
![Native test notification in the iOS Simulator](img/comfort-notification-ios.png)
Desktop and 390-pixel phone layouts had no horizontal overflow.
On macOS the development Electron app's real notification attempt was denied
(`UNErrorDomain` 1); the bridge now returns that failure instead of reporting
success. Successful macOS/Windows/Linux notification display remains unverified.
**Verified on the real Frame:** its naturally discharging 15% battery produced
one low-battery event during a short session; the test then cancelled the
session. Overheating alerts use fake sensor samples in tests: the shared
headset was not deliberately overheated.
**Verified 2026-09-29 on the same Frame:** a fresh one-minute session opened
Home more than 60 seconds after the successful warning. The test restored the
previous page and dashboard visibility. Local regression coverage now includes
slow notification delivery, a total Home-action timeout, failed worker startup,
unreadable saved state, malformed activity samples and notification UX: 173
Python tests passed. Desktop and 390-pixel layouts were checked again; system
notification-denial guidance stayed visible across polls. Initial event history
did not replay notifications, and only the latest new event was announced.
+2
View File
@@ -40,6 +40,8 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600
| Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale` (installable in `~`, see below), `krfb`, `wayvnc`. | Script design | | Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale` (installable in `~`, see below), `krfb`, `wayvnc`. | Script design |
| **SteamOS updates arrive on their own.** The Frame went from 0.3.0 (build 20260922.6101926) to **0.4.1, build 20260925.6191901**, between 2026-09-27 and 2026-09-28 with no action from us; `~` (keys, user Flatpaks, `~/.local/share`) survived. **Verified 2026-09-28.** | Keep changes in `~` | | **SteamOS updates arrive on their own.** The Frame went from 0.3.0 (build 20260922.6101926) to **0.4.1, build 20260925.6191901**, between 2026-09-27 and 2026-09-28 with no action from us; `~` (keys, user Flatpaks, `~/.local/share`) survived. **Verified 2026-09-28.** | Keep changes in `~` |
| **Valve's package repository has more than the image.** `pacman -Si` / `pacman -Sp` work as `steamos` without root and list Valve's own builds, such as `kdeconnect` 24.02.2 and `python-evdev` 1.7.0 in `extra`. Unpacking those packages into `~` runs them without touching the read-only root. The repository URLs say not to share them, so never write them down; Valve also publishes each build's source package there (`sources/packages/`), which is how Frame Control got the complete source for the KDE Connect it ships. **Verified 2026-09-28**, SteamOS 0.4.1. | [streaming.md](streaming.md#input-type-and-point-in-the-frame-from-the-mac-or-iphone) | | **Valve's package repository has more than the image.** `pacman -Si` / `pacman -Sp` work as `steamos` without root and list Valve's own builds, such as `kdeconnect` 24.02.2 and `python-evdev` 1.7.0 in `extra`. Unpacking those packages into `~` runs them without touching the read-only root. The repository URLs say not to share them, so never write them down; Valve also publishes each build's source package there (`sources/packages/`), which is how Frame Control got the complete source for the KDE Connect it ships. **Verified 2026-09-28**, SteamOS 0.4.1. | [streaming.md](streaming.md#input-type-and-point-in-the-frame-from-the-mac-or-iphone) |
| **gamescope has its own input injection.** An EIS socket at `/run/user/1000/gamescope-0-ei` (libei 1.4.1 is on the image) offers "Gamescope Virtual Input": relative and absolute pointer, buttons, scroll, keyboard. It drives the panel that has focus in the headset, on either X display. Focus moves only with the controller's laser (or to a new panel when none has it); `gamescopectl focus_info` prints the focus state to the journal. **Verified 2026-09-29.** | [streaming.md](streaming.md#live-view-and-control-watch-a-panel-and-tap-on-it) |
| **A panel's own pixels:** `ffmpeg -f x11grab -window_id <window> -i :<display>` captures one window (x11grab of the root is black under gamescope). Panels live on `:0` (Steam's UI, windows tagged by `panel-on-frame.sh`) or `:1` (apps Steam starts). **Verified 2026-09-29.** | Frame Control's Desktop view |
| **gamescope runs two Xwayland displays.** `:0` holds Steam's VR bar and menus (`valve.steam.gamepadui.*`) and ignores XTest pointer motion; `:1` holds apps such as Chromium and takes it. There's also a libei socket, `/run/user/1000/gamescope-0-ei`. **Verified 2026-09-28**, SteamOS 0.4.1. | Keyboard and trackpad | | **gamescope runs two Xwayland displays.** `:0` holds Steam's VR bar and menus (`valve.steam.gamepadui.*`) and ignores XTest pointer motion; `:1` holds apps such as Chromium and takes it. There's also a libei socket, `/run/user/1000/gamescope-0-ei`. **Verified 2026-09-28**, SteamOS 0.4.1. | Keyboard and trackpad |
| Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` | | Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` |
| `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things | | `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things |
Binary file not shown.

After

Width:  |  Height:  |  Size: 192 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 71 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

+12 -1
View File
@@ -26,7 +26,10 @@ as its transport too), so the desktop and phone share one code path. Android
display settings use `podman exec` into each Lepton container instead of adb, display settings use `podman exec` into each Lepton container instead of adb,
which the Frame doesn't have. which the Frame doesn't have.
Nothing is left running on the Frame after the phone disconnects; the copied The app server stops after the phone disconnects. An explicitly started
[comfort session](family-comfort.md) keeps its timer and headset reminders running
until the session ends or is cancelled; phone notifications require the app to
remain connected and running. The copied
files stay in `~/.cache/frame-control` (delete it any time). files stay in `~/.cache/frame-control` (delete it any time).
## Pairing ## Pairing
@@ -108,3 +111,11 @@ running), a real sleep/restart/shut down on the Frame, and a physical iPhone.
Debug builds have Simulator test hooks (`FRAME_TEST_HOST`, `FRAME_TEST_PAGE`, Debug builds have Simulator test hooks (`FRAME_TEST_HOST`, `FRAME_TEST_PAGE`,
`FRAME_TEST_JS`, and the tunnel URL in the app's Caches folder); release builds `FRAME_TEST_JS`, and the tunnel URL in the app's Caches folder); release builds
don't. don't.
## Family and comfort
The shared Home card sets session limits, breaks and check-ins, and offers
**Cast headset view**. **Enable / test notifications** requests iOS notification
permission and sends a local test. These are local notifications, not APNs push;
iOS background suspension can interrupt phone alerts. The headset timer still
runs. See [the behavior and verification limits](family-comfort.md).
+15
View File
@@ -468,3 +468,18 @@ versus on, medians of the runs, ms):
window to the front first. window to the front first.
- Ctrl stays Ctrl. On the Mac, copy is ⌘C, so use Meta+C on a keyboard paired - Ctrl stays Ctrl. On the Mac, copy is ⌘C, so use Meta+C on a keyboard paired
with the Frame. with the Frame.
## Switching panels and workspace limits
**Tools → Panel switcher** lists open SteamVR panels, including Mac viewers.
Use **Show** to request focus or **Open in headset** for Frame Control's own
switcher panel. It uses SteamVR/gamescope and Chromium, with no third-party
overlay app. [Device checks and limits](panels.md#frame-controls-panel-switcher)
include the difference between a panel surviving a scene launch and staying
visible over it.
Saved spatial layouts are blocked on this build: the public OpenVR transform
setter denies access to gamescope-owned panels. Reconnecting an existing viewer
is supported; restoring its room position after a reboot is not. We do not
save short-lived Mac window IDs or viewer access keys as if they were a durable
workspace. See [the feasibility evidence](panels.md#saved-spatial-layouts-blocked-on-the-current-panel-route).
+152
View File
@@ -124,6 +124,158 @@ a limit on the number of floating panels.
- **Windows-only overlay tools** (Desktop+, OVR Toolkit, OVRdrop) do this for a - **Windows-only overlay tools** (Desktop+, OVR Toolkit, OVRdrop) do this for a
PC's desktop in SteamVR. They don't run on the Frame's standalone Linux. PC's desktop in SteamVR. They don't run on the Frame's standalone Linux.
## Frame Control's panel switcher
**Verified 2026-09-28**, SteamOS 0.4.1, BUILD_ID `20260925.6191901`,
SteamVR 2.18.1: **Tools → Panel switcher** lists SteamVR's open main panels,
including panels that are currently hidden. **Show** asks SteamVR to bring one
forward. **Open in headset** opens the same switcher as its own panel; choose
it again from Steam's dashboard after switching away. Refresh updates the list.
This is a list, not thumbnail Exposé.
![Frame Control's switcher rendered on the Frame](img/panel-switcher.png)
This is our own Python/HTML implementation (`ui/frame_panels.py`), using the
Frame's shipped `vrcmd` OpenVR client and gamescope. The headset page uses
Chromium (Chromium XR when present, then system Chromium, then the existing
Chromium Flatpak). No XSOverlay, OVR Toolkit, WayVR or other overlay application
is needed. This dependency boundary also applies to future layout and panel
persistence work: platform APIs and bundled libraries are fine; another app
must not implement the feature for us.
The companion runs the helper over SSH. Opening it in the headset installs a
copy under `~/.local/share/frame-control/panels/` and starts a loopback HTTP
server and an isolated Chromium profile. There is no startup service or global
setting change. Close the switcher to stop its server and browser. Other
Chromium profiles, Steam and SteamVR are left alone. If the window or runtime
closes, use **Open in headset** again.
The page carries a random, per-process access key in its URL fragment, removes
it from the address bar, keeps it in tab session storage for page reloads, and
sends it in a header. Panel lists and actions need
that key; Host and Origin checks reject other sites. The key permits only
listing panels, requesting focus and closing this switcher. Like Mac viewer
launch tickets, it is initially readable by another process running as the
same Frame user. Panel titles are rendered as text, never HTML. The companion
retains its existing request guards. No Mac capture credentials cross this API.
**Verified:** the real headset page rendered its panel list (image above), its
HTTP focus request changed `GAMESCOPE_FOCUSED_APP` to `2000999030`, a request
without the key returned HTTP 403, and Close stopped the helper and its browser.
Opening an already running switcher requests its focus rather than creating a
second one. The companion uses the same list/focus helper. **Unverified:** laser
selection while wearing the headset, physical placement, and non-XR Chromium.
The API reports that focus was *requested*: another action can take focus before
we observe the result. Closed panels are rejected after re-enumeration.
### Shared-device recheck, 2026-09-29
**Verified:** the follow-up's atomic `mkdir /tmp/frame-test.lock` attempts
failed because another thread held the lock. The existing lock was left alone;
no applications were installed, launched or stopped in this follow-up. The last
read-only battery check showed 62%, charging. The 180 Python and 8 website tests
passed again locally.
**Unverified in this follow-up:** the prepared browser-button test (Refresh,
selection, reload and Close) and repeated OpenXR transition could not run under
the shared lock. The device results elsewhere in this page are the earlier
2026-09-28 observations, not results from this blocked recheck. In particular,
HTTP focus is not evidence of worn-headset laser input. Follow the
[shared-device test procedure](testing.md#headset-smoke-test) for the next run.
## Saved spatial layouts: blocked on the current panel route
**Verified 2026-09-28**, same build, using a temporary xterm panel with
`STEAM_GAME=2000999031` and `FnTable:IVROverlay_028` from
`/opt/steamvr/bin/linuxarm64/libopenvr_api.so`:
| OpenVR call | Result |
|---|---|
| `FindOverlay("valve.steam.desktopgame.2000999031")` | Success |
| `GetOverlayWidthInMeters` | Success, 2.67 m |
| `SetOverlayWidthInMeters` (same width) | Success |
| `GetOverlayTransformType` | Success, type 5 (`VROverlayTransform_DashboardTab`) |
| `GetOverlayTransformAbsolute` | 18 (`WrongTransformType`) |
| `SetOverlayTransformAbsolute` (identity rotation, 1.2 m up, 1.5 m forward) | 12 (`PermissionDenied`); type remained 5 |
The public interface names type 5 **DashboardTab**; SteamVR's dashboard code
places these panels through its scene graph. It owns the frame/docking
transforms. A successful width setter does not grant permission to restore the
position. `vrcmd --dock-overlay world <key>` dispatched a docking request but
the dashboard logged `Failed to get SGTransform in setInitialTransformForLocation.
Invalid transform ID`. This does not establish working world placement.
**Inferred:** saving X11 pixel rectangles or Mac window IDs would not restore
this spatial arrangement. Mac window IDs also change when an application
reopens; viewer tickets and reconnect keys must not go into a layout file.
The base Mac stream reconnects after a network break, but that is different
from recreating windows and their room positions after a reboot.
There is consequently no Save/Restore control yet. A durable layout needs a
working transform restore path, stable source identity, and a fresh capture
permission/ticket flow. The tested gamescope-owned overlay route denies that
transform operation. A future Frame Control-owned overlay renderer, or a
supported platform API for dashboard frame transforms, needs its own device
proof before building layout UI. This is a blocker for the current approach,
not a claim that all possible implementations are impossible. Reboot recovery
was not tested: the shared headset was not rebooted.
## Panels during an immersive session
**Verified 2026-09-28**, same build: our Chromium switcher panel remained in
OpenVR's overlay list before, during and after the Frame's shipped `helloxr -g
Vulkan` sample. During the test `vrcmd --stats` identified
`system.generated.openxr.helloxr.helloxr`, with 242 frame submissions. The test
ended only its own sample process; no SteamVR, Steam, power or global settings
were changed. The switcher was still selectable afterwards.
This proves survival of that panel across an OpenXR scene session, **not** that
it stayed visibly composited over the scene: OpenVR reported it `not_visible`
before, during and after. **Verified:** calling `ShowOverlay` on our
*gamescope-owned* switcher overlay returns 12 (`PermissionDenied`). A helper
cannot force that panel visible using the public overlay call. Use the
switcher/dashboard to request access to it; we do not fight the runtime with a
repeated force-focus loop.
**Verified in a second controlled run:** a live H.264 test-pattern stream from
this checkout's Mac helper, through its own SSH tunnel and a temporary Chromium
profile, survived the same OpenXR sample (257 scene-frame submissions). Its
panel `2000999032` changed from `visible` before launch to `not_visible` during
and after the scene. The Mac helper still reported the same `test` stream;
captured frames increased from 35 to 232, with 29.5 decoded/drawn fps afterwards.
The test did not capture personal Mac windows or inject Mac input. The sample,
viewer, temporary profile, tunnel and Mac helper were cleaned up. This proves
stream survival, and also shows why it must not be advertised as always visible.
**Unverified:** persistent visible placement while playing a Steam-launched VR
game, Plasma desktop and real Mac-window behavior during that launch, and worn
headset input. Other threads were launching games and changing the runtime on
the shared device, so those transitions were not treated as controlled evidence.
A runtime/X-server restart can destroy the viewer windows; a network reconnect
cannot recreate them. No “always visible during games” guarantee is shipped.
## Keyboard passthrough feasibility
**Verified 2026-09-28**, same build, using `FnTable:IVRTrackedCamera_006`:
`HasCamera(0)` returned success and true. `GetCameraFrameSize` returned 100
(`OperationFailed`), with zero dimensions, for all three public frame types
(distorted, undistorted and maximum-undistorted), including after acquiring the
video service. Acquisition returned success and a handle; release returned 101
(`InvalidHandle`). The probe shut down its OpenVR client afterwards. No camera
frames were captured and no camera settings were changed.
**Documented:** the public OpenVR camera interface provides camera frame sizes,
intrinsics, projections and streaming handles; these are prerequisites for a
spatially aligned camera cutout. See Valve's
[OpenVR C API](https://github.com/ValveSoftware/openvr/blob/master/headers/openvr_capi.h).
**Inferred:** camera presence alone does not establish access to camera pixels.
The failed frame-size path blocks a keyboard cutout in our current panel
implementation. We have not established a keyboard detector or a calibrated
camera-to-panel mapping. Built-in full-room passthrough is not proof of a
public, selectively masked camera stream. No keyboard cutout is offered, and
no third-party camera/overlay app is substituted for it.
## Frame Control's media theatre ## Frame Control's media theatre
[The owned media player](vr-video.md) can show its video or stereo image on a [The owned media player](vr-video.md) can show its video or stereo image on a
-199
View File
@@ -1,199 +0,0 @@
# PC in the headset
Windows and Linux hosts use **Tools → PC in the headset**. The host shares a
window or screen, and the existing Frame viewer makes it a SteamVR panel.
Move it with the dashboard's Float in World, Move and Size controls.
**Inferred / not yet verified on a desktop host:** the Windows and Linux
capture and input paths below. This work was developed on a Mac with no
Windows or Linux desktop VM. A native build or test-pattern test in CI does
not establish that desktop capture, a permission dialog, hardware encoding
or laser input works. Keep this feature in the draft/testing stage until
those paths have been tried on real hosts.
## Own implementation, platform APIs and bundled libraries
Frame Control owns the host agent, input routing, authentication, streaming
protocol, panel launcher and adaptation. It does not launch or require
Sunshine, OBS or another desktop-streaming app. GStreamer and its codec
plugins are ordinary libraries bundled with the Windows and Linux app;
users do not install a GStreamer application. The shared library build keeps
license texts and package provenance alongside the libraries. Linux also
bundles the PipeWire client’s dynamically loaded SPA/protocol modules and a
private client configuration; it does not change the desktop’s configuration.
First-party alternatives considered (**documented**): Valve Remote Play
streams a game/desktop, rather than providing this per-window panel protocol;
Windows Remote Desktop opens a remote session; Linux's desktop portal is the
consent mechanism for sharing the current desktop. The chosen paths are:
| Host | Capture | Encoding | Input |
|---|---|---|---|
| Windows | Windows.Graphics.Capture, through `d3d11screencapturesrc capture-api=wgc`; HWND or HMONITOR | Hardware Media Foundation (`mfh264enc`), low latency, no B-frames | `SendInput`, with source bounds and per-monitor DPI awareness |
| Linux | RemoteDesktop + ScreenCast portal, then the returned PipeWire fd/node | VA-API (`vah264enc`) where registered; x264 otherwise | RemoteDesktop portal notifications, using only granted pointer/keyboard devices |
API choices are **documented**, not device verification:
[Windows capture](https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture),
[GStreamer WGC](https://gstreamer.freedesktop.org/documentation/d3d11/d3d11screencapturesrc.html),
[Media Foundation encoder](https://gstreamer.freedesktop.org/documentation/mediafoundation/mfh264enc.html),
[ScreenCast portal](https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html),
[RemoteDesktop portal](https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.RemoteDesktop.html).
Windows needs a WGC-capable Windows 10/11 desktop and an available hardware
Media Foundation H.264 encoder. Elevated windows and the secure desktop
cannot be driven by an ordinary Frame Control process. If Windows refuses
to focus a selected window, input is refused rather than sent to the app
covering it; bring the selected window forward, then Stop and Show again. Minimized/closed
windows may stop producing frames. Protected content is not supported.
On Linux, press **Choose a window or screen…** and approve the desktop's
sharing dialog. Choose another source to add another panel. Stop releases
that source's portal session; sharing it again asks for consent again.
A desktop must implement both ScreenCast and RemoteDesktop for this path;
a ScreenCast-only compositor cannot provide laser input through this API.
Cancelling or denying a dialog is reported on the card. No portal permission
is bypassed, and Frame Control does not open `/dev/uinput` or the unrestricted
PipeWire daemon on the host.
The host's own keyboard still works. Input from the viewer uses normalized
picture coordinates, maps through the selected source's bounds, and releases
held buttons/keys on blur, disconnect and Stop. Linux requires the pointer
and keyboard grants. **Untested:** desktop-specific consent, mixed-DPI
Windows input alignment, multi-monitor layouts, hardware encoder behavior,
window resize/minimize, and non-US keyboard layouts.
## Shared pieces
- `ui/frame_macview.py` owns the SSH tunnel, reconnect supervision, quality
presets and panel launch for all hosts. `ui/frame_pcview.py` selects the PC
helper; `/api/macview` remains the compatible endpoint.
- `ui/mac-view.html` is the one viewer. The 17-byte big-endian frame header,
Annex-B H.264/JPEG payloads, `hello`/`ack` reconnect handshake, clock sync,
`rx`/`fd` timing reports and input messages are unchanged.
- `desktop/controller.c` is the rate controller shared by the Mac Swift
binding and the PC Python binding. Capture is gated **before** encoding;
encoded reference frames are never discarded. A bounded raw-frame queue
keeps the newest picture, including the last update of an idle window,
until the gate opens. A native one-frame-source test covers that case.
It keeps the Mac's bitrate demand protection and tier hysteresis.
- PC records use the existing `Stats.swift` JSON schema, with bounded
4096-frame/512-input storage in `ui/frame_stream_stats.py`. The benchmark's
analysis, targets and network shaping are shared, not reimplemented.
Capture timestamps describe the native pipeline's source time; they do
not prove the time at which the host compositor displayed the pixels.
- Mac virtual-display separation remains Mac-only. Windows WGC and the
Linux portal share the selected window directly.
PC capture follows the shared frame-rate and resolution tiers. x264 updates
bitrate while running. Hardware encoders are drained and reopened when the
budget changes materially, at most once a second, because their live property
support varies. Reopening starts a new keyframe and retains the consented
portal session. **Untested:** hardware reconfiguration latency and whether a
particular desktop permits reconnecting its PipeWire stream this way.
## Build and measure
Packaged Windows/Linux builds include `desktop/bundle/pc-host` and its shared
libraries. Source checkouts build them with `python3 desktop/build.py` after
installing GStreamer development packages (see the `PC host libraries` CI
workflow). The feature reports a missing bundle; it does not download or
install a streaming app on first use.
The existing benchmark now accepts a PC host:
```sh
python3 scripts/macview-bench.py run --pc --scenario test --label pc-test
# Linux: select a real source in the desktop's sharing dialog
python3 scripts/macview-bench.py run --pc --scenario capture --source choose --label linux-window
# Windows: use the HWND/monitor source ID shown by the host's /windows or /displays
python3 scripts/macview-bench.py run --pc --scenario capture --source window:12345 --label windows-window
```
The synthetic PC pattern uses bundled x264 so headless CI can verify the
wire protocol without claiming that a GPU was exercised. The `capture`
scenario measures the selected real source without injecting input or
assuming that it animates at 60 fps. Mac-only Chrome/virtual-display typing
and scrolling automation is not run on PC hosts. Results retain the same
latency stages and record `host_platform`, `pc_host` and `source`. CPU sampling
on PC hosts is explicitly unavailable. `--net` and `--delay` still use the
same bounded shaping relay, without administrator privileges.
## Evidence
- **Verified, Mac, 2026-09-28:** the final full unit/integration suite ran
183 tests successfully, including the real Mac helper's H.264, ticket, timing
and input-echo tests. The native PC test class was skipped locally because
its libraries were absent; the PC adapter and shared-controller tests passed.
- **Verified, real Frame, 2026-09-28, BUILD_ID 20260925.6191901:** the base
helper's synthetic source created panel `valve.steam.desktopgame.2001639889`,
and the shared Chromium viewer decoded H.264. It recorded 286 frames over
the short probe, with a two-second summary of 19.5 fps shown and total
latency p50/p95 83/156.5 ms. This establishes the existing viewer/transport
route, not Windows/Linux capture, input or a latency target. The probe's
helper, tunnel and viewer were stopped afterward.
- **Verified in CI:** native library builds and real x264 protocol tests passed
on Windows, Ubuntu x64 and Ubuntu ARM64 in
[run 36422214445](https://github.com/saphid/frame-control/actions/runs/36422214445).
All four installer builds passed in
[run 36422214425](https://github.com/saphid/frame-control/actions/runs/36422214425).
These are build/synthetic tests, not desktop-host verification. No VM was used.
- **Verified, real Frame, same build/date:** the ARM64 PC agent and its
bundled libraries ran from a temporary user directory, using x264's moving
test pattern. Traffic travelled Frame → Mac SSH relay → Frame viewer.
The shared bench recorded 351 drawn frames, content p50/p95 15.1/80.3 ms,
and 34.8 fps. The frame-rate and late-frame targets failed. This checks the
new agent and real viewer together; it is not a representative PC link.
The helper exited 0, its viewer/tunnels stopped, and its directory was removed.
[Raw benchmark result](../bench/results/2026-09-28-c5fc552-dirty-pc-agent-frame-arm64-hairpin.json).
This first probe's input echo measured message receipt to the next capture,
not a visible pattern response; later builds make test clicks change its color.
- **Verified, Mac:** all 600 states in a 60-second congestion/recovery trace
matched the original Swift controller. `tests/test_pc_controller.py` retains
the original trace digest as a regression check.
- **Verified, real Frame, agent at `cd20243`:** the repeated synthetic
probe drew 400 frames at 39.3 fps, content p50/p95 15.1/28.6 ms, and synthetic
input-to-drawn p50 76.6 ms. Test clicks now change the pattern color before
injection is timestamped. The frame-rate/late-frame targets still failed;
this remains a Frame-hosted x264 test through a Mac relay, not a desktop or
physical-laser measurement. Helper exit 0 and cleanup succeeded.
[Latest device probe result](../bench/results/2026-09-28-cd20243-pc-agent-frame-arm64-final.json).
- **Untested:** real Windows WGC → Media Foundation → Frame; real Linux
portal → PipeWire → VA-API/x264 → Frame; physical laser input on either.
No benchmark numbers for those desktop paths are claimed.
## Independent review availability
A direct read-only review was attempted with
`devin -p --model swe-2-max --permission-mode auto --prompt-file …`. The first
attempt exited 0 after rejecting a tool that needed interactive permission;
it did not inspect the diff. A full inline-diff attempt returned no output
for 15 minutes and was terminated (shell exit 143). A smaller inline native
code review returned no output within 300 seconds (process exit -15).
SWE-2 Max was requested; no completed review or findings were received, so
independent review is **unverified**, not a passed check. The PR remains draft.
## Device follow-up, 2026-09-29
**Verified, read-only, SteamOS BUILD_ID 20260925.6191901:** the Frame was
reachable, charging (44% initially, 63% at the end), and SteamVR reported
activity level 3 (standby). No Plasma desktop was running. Introspection of
the active `org.freedesktop.portal.Desktop` service exposed ScreenCast with
source types 3, but no RemoteDesktop interface. **Inferred:** this gamescope
session cannot provide this feature's required consented input path; a
ScreenCast-only session is insufficient.
**Blocked, not a device execution result:** the shared `/tmp/frame-test.lock`
was held by another thread. Ten lock-acquisition attempts over five minutes
all failed, after earlier preparation-time attempts also found it occupied.
The current `a992f6d` ARM64 CI bundle was downloaded to the Mac, but nothing
was installed, launched or stopped on the Frame. No new screenshots or
current-revision device timings were obtained. The lock was not removed and
no global settings were changed. The earlier synthetic results above remain
valid only for their named revisions; standby visibility, physical laser
input and worn-headset performance remain unverified.
**Verified, Mac:** a fresh full suite ran 309 tests successfully, with one
native-PC class skipped. Current-revision CI passed
[native hosts](https://github.com/saphid/frame-control/actions/runs/36504546909),
[installers](https://github.com/saphid/frame-control/actions/runs/36504546800)
and [general checks](https://github.com/saphid/frame-control/actions/runs/36504546887).
+62 -2
View File
@@ -8,6 +8,7 @@ This covers three directions, plus input:
- **PC VR from Linux**: [feasibility and options](linux-vr-streaming.md), - **PC VR from Linux**: [feasibility and options](linux-vr-streaming.md),
including Valve's streaming and USB support. No Linux host tested yet. including Valve's streaming and USB support. No Linux host tested yet.
- **Input**: type and point in the Frame from the Mac or iPhone. - **Input**: type and point in the Frame from the Mac or iPhone.
- **Live view and Control**: watch a panel flat and tap on it to use it.
The confidence labels are the same as in [ssh.md](ssh.md). The confidence labels are the same as in [ssh.md](ssh.md).
@@ -184,13 +185,72 @@ on the Frame, which talks KDE Connect's own LAN protocol to the Frame's
and ignores injected pointer motion; `:1` holds apps such as Chromium and and ignores injected pointer motion; `:1` holds apps such as Chromium and
takes it. KDE Connect runs on `:1`, so it reaches apps, not Steam's own menus. takes it. KDE Connect runs on `:1`, so it reaches apps, not Steam's own menus.
There's also a `gamescope-0-ei` (libei) socket. There's also a `gamescope-0-ei` (libei) socket.
- **Not yet tested:** typing and clicking as seen in the headset, and whether - Typing through KDE Connect lands in a Chromium panel on `:1` (seen in the
it reaches the KDE desktop panel (Plasma is its own session). panel's own capture, 2026-09-29). It **can't reach panels on `:0`** (Frame
Control's own panels, Steam's UI) and, since XTest positions are clamped to
`:1`'s 1280×720 root, can't reach beyond that in a bigger window. Control on
the live view (below) has neither limit.
- **Not yet tested:** whether it reaches the KDE desktop panel (Plasma is its
own session).
- **Known limit:** keys and clicks typed while the link is reconnecting wait - **Known limit:** keys and clicks typed while the link is reconnecting wait
and are sent once it's back, but anything sent in the moment the Wi-Fi and are sent once it's back, but anything sent in the moment the Wi-Fi
drops, before SSH notices, can be lost. Confirming every event would add a drops, before SSH notices, can be lost. Confirming every event would add a
round trip to each pointer move. round trip to each pointer move.
## Live view and Control: watch a panel and tap on it
**Built: Home → Desktop / Headset view → Control.** The live view has two
sources:
- **Headset view**: what the lenses show (SteamVR's mirror, `/dev/video99`). It
moves with the wearer's head, so Control makes the view a trackpad: drag to
move the pointer, tap to click, press and hold to right-click, two fingers to
scroll. With a mouse, moving over the view moves the pointer.
- **Desktop**: the app panel in use in the headset, from its own window, so it
stays still however the wearer looks around. Control makes taps and clicks
land exactly where you put them. Dragging is a mouse drag, press and hold is a
right-click, two fingers scroll, and on a computer the mouse, wheel and
keyboard work directly on it (⌘ is sent as Ctrl on a Mac). A picker shows any
other panel, view only.
Below the view, a text field and key buttons type on the Frame from a phone.
How (**verified 2026-09-29**, SteamOS 0.4.1, build 20260925.6191901):
- **Input goes through gamescope's own injection.** gamescope serves an EIS
socket (`/run/user/1000/gamescope-0-ei`; Steam feeds Remote Play input through
it), and `libei` 1.4.1 is on the image. [`ui/frame_touch.py`](../ui/frame_touch.py)
talks to it with `ctypes`: nothing to install. gamescope offers one device,
"Gamescope Virtual Input", with relative and absolute pointer, buttons,
scroll and keyboard (Linux key codes; no text capability, so the text field
types printable ASCII on a US layout). Its absolute region is unbounded; the
pointer uses the focused panel's display coordinates, and gamescope fits each
window to its display, so a 1920×1080 window on the 1280×720 `:1` takes
positions at two thirds scale. Taps on a 1280×720 page landed on the exact
pixel.
- **It reaches the panel that has focus** (`GAMESCOPE_FOCUSED_WINDOW` on `:0`'s
root), on either X display. In the OpenVR backend focus moves only on SteamVR
overlay events (the controller's laser entering or clicking a panel), or to a
new panel when none holds it (read from gamescope's `OpenVRBackend.cpp`, seen
with `gamescopectl focus_info`, which writes to the journal). Neither
`GAMESCOPECTRL_BASELAYER_WINDOW`/`_APPID` nor X focus moves it, and no
gamescope command does. So Control follows the wearer: whatever they last
used is what your taps reach. A window without a Steam app id (`STEAM_GAME`)
gets a connector of its own and doesn't hold focus.
- **Keys in a burst can arrive out of order**, so the helper paces them (8 ms
apart).
- **Known limit:** if focus moves to another panel in the middle of a drag, the
release goes to the panel that has focus then. Whether gamescope hands it to
the window that got the press isn't known yet. When the session ends, the
helper lets go of every button and key it still holds.
- **The Desktop picture is the window's own pixels**: `ffmpeg -f x11grab
-window_id <window> -i :<display>` works on gamescope's redirected windows,
while grabbing the root gives black. It streams as H.264 like the headset view
(about 30 fps at 720p).
- Tested from the iPhone app (Simulator): a tap on the Desktop view focused a
text box in the panel and the text field typed into it; a trackpad move went
exactly (+40, +25).
Our own `uinput` keyboard and mouse would also work (`steamos` is in the Our own `uinput` keyboard and mouse would also work (`steamos` is in the
`input` group and `/dev/uinput` is group-writable, verified 2026-09-27), and `input` group and `/dev/uinput` is group-writable, verified 2026-09-27), and
remains the fallback if the bundled KDE Connect ever stops working on a new SteamOS. remains the fallback if the bundled KDE Connect ever stops working on a new SteamOS.
+40 -11
View File
@@ -104,6 +104,20 @@ switch while SSH is down:
## Headset smoke test ## Headset smoke test
**Documented shared-device procedure:** before a test installs, launches or
stops an application, acquire `ssh frame 'mkdir /tmp/frame-test.lock'`. If it
fails, leave that lock alone and continue offline work. Only the thread that
acquired it releases it with `ssh frame 'rmdir /tmp/frame-test.lock'`, after
cleanup. Keep each device session to a few minutes.
Check battery capacity and charging state under `/sys/class/power_supply`
before and after; keep capacity above 20%. Stop only processes started by the
test, remove temporary installs and profiles, and restore the prior dashboard
state. Leave Steam and SteamVR running. Do not reboot or change global settings.
Record the build, actual interaction results, cleanup and any unworn-headset
limits alongside screenshots or logs. These are caller responsibilities; the
smoke script below does not acquire this shared lock itself.
```sh ```sh
scripts/frame-smoke.sh # needs `ssh frame` to work without a password scripts/frame-smoke.sh # needs `ssh frame` to work without a password
scripts/frame-smoke.sh --pair # also pairs a throwaway key: approve it in the headset scripts/frame-smoke.sh --pair # also pairs a throwaway key: approve it in the headset
@@ -149,17 +163,6 @@ refuses ids with a hyphen (`missing/invalid arguments`), which the fake had
accepted. The fake now refuses them the same way, and Frame Control makes ids accepted. The fake now refuses them the same way, and Frame Control makes ids
Steam accepts. Steam accepts.
## PC panel host tests
`tests/test_pcview.py` checks source-bound tickets, reconnect/Stop revocation,
input release, source-coordinate mapping and use of the existing panel
launcher. The `PC host libraries` workflow builds the bundled native adapter
on Windows, Ubuntu x64 and Ubuntu ARM64, then runs a real x264 synthetic
stream through the HTTP/WebSocket agent. `FRAME_PC_REQUIRE_NATIVE=1` makes a
missing native bundle fail CI instead of skipping. These tests do not grant a
portal dialog, capture a desktop, exercise a hardware encoder or validate
laser alignment. See [PC host evidence](pc-in-headset.md#evidence).
## Owned media player ## Owned media player
`tests/test_media.py` covers layout evidence and overrides, OU eye ordering, `tests/test_media.py` covers layout evidence and overrides, OU eye ordering,
@@ -181,3 +184,29 @@ assistant against an in-process HTTP endpoint with canned responses (no keys or
external calls). `tests/e2e/test_agents.py` runs the MCP/HTTP/SSH path against the external calls). `tests/e2e/test_agents.py` runs the MCP/HTTP/SSH path against the
fake Frame for approved installs, clipboard and file transfer. Headset Chromium fake Frame for approved installs, clipboard and file transfer. Headset Chromium
rendering and real screenshots still need a device; see [agent evidence](agents.md#evidence-and-limits). rendering and real screenshots still need a device; see [agent evidence](agents.md#evidence-and-limits).
## Family and comfort
`tests/test_comfort.py` uses an injected clock, fake headset sensor readings and
actions, plus a Node fake of Steam's Home API. It covers warnings before Home,
late/suspended sessions, cancellation, failed actions, duplicate alerts, reboot
invalidation, per-zone thermal trips and shared on-headset state. The server
guards reject invalid session settings before SSH. See
[real-device evidence and limits](family-comfort.md#verification).
## Panel switcher
`tests/test_panels.py` supplies fake-Frame `vrcmd --overlays` output, checks
main-panel filtering (including hidden panels), revalidates closed panels before
focus, and drives the headset helper's real loopback HTTP server to test access
keys, Host/Origin guards, malformed requests, offline errors and Close. It runs
in the normal unit suite without OpenVR or a headset. The fixture format comes
from SteamVR 2.18.1, BUILD_ID `20260925.6191901`; it does not simulate rendering.
On the Frame, run `python3 -` over SSH with `ui/frame_panels.py` on stdin to
list panels. `--focus <key>` rechecks the list and requests focus. In Frame
Control, **Tools → Panel switcher → Open in headset** exercises installation,
Chromium rendering and the same helper through HTTP. Close the switcher after
testing. [The recorded device checks](panels.md#frame-controls-panel-switcher)
cover actual focus, HTTP guards and an OpenXR sample transition, and separately
identify the unverified Steam-game, spatial layout, reboot and laser behaviors.
@@ -17,6 +17,7 @@
78427FC66780623F31E7501E /* FrameControlApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */; }; 78427FC66780623F31E7501E /* FrameControlApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */; };
84423CB45629465420180A64 /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 8F2CB550FC81C01E6BDD5A71 /* Assets.xcassets */; }; 84423CB45629465420180A64 /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 8F2CB550FC81C01E6BDD5A71 /* Assets.xcassets */; };
9657F7BC23E3352E5AB30777 /* SetupView.swift in Sources */ = {isa = PBXBuildFile; fileRef = DB544223FC60A59CC3E8EF5F /* SetupView.swift */; }; 9657F7BC23E3352E5AB30777 /* SetupView.swift in Sources */ = {isa = PBXBuildFile; fileRef = DB544223FC60A59CC3E8EF5F /* SetupView.swift */; };
A0C5B00E257230A38DBD9E54 /* ComfortNotificationTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */; };
A8C7AED25A6280682FCE45DC /* Citadel in Frameworks */ = {isa = PBXBuildFile; productRef = 6BA549B6CC0A0CB847126456 /* Citadel */; }; A8C7AED25A6280682FCE45DC /* Citadel in Frameworks */ = {isa = PBXBuildFile; productRef = 6BA549B6CC0A0CB847126456 /* Citadel */; };
DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */; }; DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */; };
E6898C714A92D3979F73B6E1 /* FrameFinder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2F288DF6636A417F0CA3A6CD /* FrameFinder.swift */; }; E6898C714A92D3979F73B6E1 /* FrameFinder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2F288DF6636A417F0CA3A6CD /* FrameFinder.swift */; };
@@ -48,6 +49,7 @@
BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InstallLink.swift; sourceTree = "<group>"; }; BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InstallLink.swift; sourceTree = "<group>"; };
D6C4E6C28315CA8729FCAAEA /* WebShell.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WebShell.swift; sourceTree = "<group>"; }; D6C4E6C28315CA8729FCAAEA /* WebShell.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WebShell.swift; sourceTree = "<group>"; };
DB544223FC60A59CC3E8EF5F /* SetupView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SetupView.swift; sourceTree = "<group>"; }; DB544223FC60A59CC3E8EF5F /* SetupView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SetupView.swift; sourceTree = "<group>"; };
E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ComfortNotificationTests.swift; sourceTree = "<group>"; };
EDC7BA8014DBC302D08FD397 /* HeadsetServer.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = HeadsetServer.swift; sourceTree = "<group>"; }; EDC7BA8014DBC302D08FD397 /* HeadsetServer.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = HeadsetServer.swift; sourceTree = "<group>"; };
F3E2F5607DD877272483D64E /* FrameControl.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = FrameControl.app; sourceTree = BUILT_PRODUCTS_DIR; }; F3E2F5607DD877272483D64E /* FrameControl.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = FrameControl.app; sourceTree = BUILT_PRODUCTS_DIR; };
/* End PBXFileReference section */ /* End PBXFileReference section */
@@ -107,6 +109,7 @@
75A17B1C79C8C3C60FEABBA6 /* FrameControlTests */ = { 75A17B1C79C8C3C60FEABBA6 /* FrameControlTests */ = {
isa = PBXGroup; isa = PBXGroup;
children = ( children = (
E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */,
1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */, 1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */,
); );
path = FrameControlTests; path = FrameControlTests;
@@ -274,6 +277,7 @@
isa = PBXSourcesBuildPhase; isa = PBXSourcesBuildPhase;
buildActionMask = 2147483647; buildActionMask = 2147483647;
files = ( files = (
A0C5B00E257230A38DBD9E54 /* ComfortNotificationTests.swift in Sources */,
DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */, DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */,
); );
runOnlyForDeploymentPostprocessing = 0; runOnlyForDeploymentPostprocessing = 0;
+44 -2
View File
@@ -1,6 +1,7 @@
import SwiftUI import SwiftUI
import UIKit import UIKit
import WebKit import WebKit
import UserNotifications
/// The Frame Control page, served by the server on the headset, in a web view. /// The Frame Control page, served by the server on the headset, in a web view.
/// window.frameApp (the same bridge the desktop app's preload.js provides) lets /// window.frameApp (the same bridge the desktop app's preload.js provides) lets
@@ -48,6 +49,7 @@ struct WebShell: UIViewRepresentable {
let installCb = null; let installCb = null;
window.frameApp = { window.frameApp = {
platform: "ios", platform: "ios",
notify: (message, request) => call("notify", { message, request }),
readClipboard: () => call("readClipboard"), readClipboard: () => call("readClipboard"),
setUpConnection: () => call("setUpConnection"), setUpConnection: () => call("setUpConnection"),
open: (what) => call("open", what), open: (what) => call("open", what),
@@ -58,13 +60,31 @@ struct WebShell: UIViewRepresentable {
})(); })();
""" """
final class Coordinator: NSObject, WKScriptMessageHandlerWithReply, WKNavigationDelegate, WKUIDelegate { final class Coordinator: NSObject, WKScriptMessageHandlerWithReply, WKNavigationDelegate, WKUIDelegate, UNUserNotificationCenterDelegate {
let model: AppModel let model: AppModel
weak var web: WKWebView? weak var web: WKWebView?
var loaded: URL? var loaded: URL?
private var installReady = false private var installReady = false
init(model: AppModel) { self.model = model } init(model: AppModel) {
self.model = model
super.init()
UNUserNotificationCenter.current().delegate = self
}
func userNotificationCenter(_ center: UNUserNotificationCenter, willPresent notification: UNNotification,
withCompletionHandler completionHandler: @escaping (UNNotificationPresentationOptions) -> Void) {
completionHandler([.banner, .sound, .list])
}
static func notificationContent(_ message: String) -> UNMutableNotificationContent? {
guard !message.isEmpty, message.count <= 500 else { return nil }
let content = UNMutableNotificationContent()
content.title = "Frame Control"
content.body = message
content.sound = .default
return content
}
// MARK: bridge // MARK: bridge
@@ -76,6 +96,28 @@ struct WebShell: UIViewRepresentable {
} }
let arg = body["arg"] let arg = body["arg"]
switch name { switch name {
case "notify":
guard message.frameInfo.isMainFrame,
message.frameInfo.securityOrigin.host == "127.0.0.1",
let args = arg as? [String: Any], let text = args["message"] as? String,
let content = Self.notificationContent(text) else {
return replyHandler(nil, "Invalid notification")
}
let center = UNUserNotificationCenter.current()
let send: (Bool, Error?) -> Void = { allowed, error in
guard allowed else {
return replyHandler(nil, error?.localizedDescription ?? "Notifications are off. Enable them in iOS Settings.")
}
let request = UNNotificationRequest(identifier: UUID().uuidString, content: content, trigger: nil)
center.add(request) { error in replyHandler(error == nil, error?.localizedDescription) }
}
if args["request"] as? Bool == true {
center.requestAuthorization(options: [.alert, .sound], completionHandler: send)
} else {
center.getNotificationSettings { settings in
send(settings.authorizationStatus == .authorized || settings.authorizationStatus == .provisional, nil)
}
}
case "readClipboard": case "readClipboard":
replyHandler(UIPasteboard.general.string ?? "", nil) replyHandler(UIPasteboard.general.string ?? "", nil)
case "setUpConnection": case "setUpConnection":
@@ -0,0 +1,14 @@
import XCTest
import UserNotifications
@testable import Frame_Control
final class ComfortNotificationTests: XCTestCase {
func testNotificationContentAndBounds() {
let content = WebShell.Coordinator.notificationContent("Time for a break")
XCTAssertEqual(content?.title, "Frame Control")
XCTAssertEqual(content?.body, "Time for a break")
XCTAssertNotNil(content?.sound)
XCTAssertNil(WebShell.Coordinator.notificationContent(""))
XCTAssertNil(WebShell.Coordinator.notificationContent(String(repeating: "x", count: 501)))
}
}
@@ -26,5 +26,3 @@
- (instancetype)initWithDescriptor:(CGVirtualDisplayDescriptor *)descriptor; - (instancetype)initWithDescriptor:(CGVirtualDisplayDescriptor *)descriptor;
- (BOOL)applySettings:(CGVirtualDisplaySettings *)settings; - (BOOL)applySettings:(CGVirtualDisplaySettings *)settings;
@end @end
#include "../../../desktop/controller.h"
+222 -33
View File
@@ -1,45 +1,234 @@
// Thin, serialized binding to the same controller used by the PC host. // Adapts each stream to its network, latency first. The viewer acknowledges
// Algorithm and bounded state live in desktop/controller.c. // every frame as it arrives ("rx"); from those acks the controller knows how
// long frames take to get through and how fast the link delivers them.
//
// - The gate: a new frame is sent only while the oldest unacknowledged one is
// younger than the path's usual round trip plus a little slack. So frames
// never queue up in SSH, TCP or the Wi-Fi driver; while the link is stuck
// the newest picture waits and goes out as soon as it moves again.
// - The bitrate: when frames start queueing (the round trip grows) or the
// gate has to hold frames back, it drops to a bit under what the link
// actually delivered; once things are clear it probes up again slowly, never
// above the quality setting's bitrate (the ceiling).
// - The tier: as the bitrate falls, fewer frames per second (60, 45, 30), then
// a smaller picture (75%, then 50% of the panel's pixels).
// See docs/mac-in-headset.md ("Adapting to the network"). Thread-safe.
import Foundation import Foundation
final class RateController { final class RateController {
struct Tier: Equatable {
let fps: Int
let scale: Double // of the picture's long side
}
static let tiers = [Tier(fps: 60, scale: 1), Tier(fps: 45, scale: 1), Tier(fps: 30, scale: 1),
Tier(fps: 30, scale: 0.75), Tier(fps: 30, scale: 0.5)]
/// A tier is used while the target bitrate is at least this share of the ceiling.
static let floors = [0.45, 0.28, 0.16, 0.08, 0]
static let enabled = ProcessInfo.processInfo.environment["FRAME_MAC_VIEW_ADAPT"] != "0" static let enabled = ProcessInfo.processInfo.environment["FRAME_MAC_VIEW_ADAPT"] != "0"
let maxFps: Int let maxFps: Int
private let lock = NSLock() private let lock = NSLock()
private let core: OpaquePointer private var ceiling = 0 // bits/s at full size and frame rate
private var events: [[String: Any]] = [] private(set) var target = 0
init(maxFps: Int) { private(set) var tier = 0
self.maxFps = maxFps private var unacked: [(seq: UInt32, sent: Int64, bytes: Int)] = []
core = fc_new(Int32(maxFps), Self.enabled ? 1 : 0)! /// Round trips (send -> ack arrives here), for the baseline: the lowest
} /// in the last 10 s is the path without any queue.
deinit { fc_free(core) } private var rtts: [(t: Int64, v: Int64)] = []
private var acked: [(t: Int64, bytes: Int)] = [] // the last second
private var sentLog: [(t: Int64, bytes: Int)] = []
private var captures: [Int64] = [] // the last second
private var frameBytes = 0 // average recent frame, kept while the gate holds everything back
private var held = 0 // frames the gate held back since the last update
private var lastSignal = false
private var sawAck = false
private var lastDecrease: Int64 = 0
private var lastIncrease: Int64 = 0
private var belowSince: Int64 = 0, aboveSince: Int64 = 0
/// What changed, for the timeline: (time, event).
private(set) var events: [(Int64, String)] = []
init(maxFps: Int) { self.maxFps = maxFps }
private func locked<T>(_ f: () -> T) -> T { lock.lock(); defer { lock.unlock() }; return f() } private func locked<T>(_ f: () -> T) -> T { lock.lock(); defer { lock.unlock() }; return f() }
private func value(_ field: Int32) -> Int { Int(fc_value(core, field)) }
var target: Int { locked { value(0) } } /// The quality setting's bitrate at full size; the first call also starts there.
var tier: Int { locked { value(2) } } func setCeiling(_ bps: Int) {
var fps: Int { locked { value(3) } }
var scale: Double { locked { Double(value(4)) / 100 } }
var baseRtt: Int64 { locked { fc_value(core, 5) } }
func setCeiling(_ bps: Int) { locked { fc_ceiling(core, Int32(bps)) } }
func maySend(now: Int64, counts: Bool = true) -> Bool { locked { fc_gate(core, now, counts ? 1 : 0) != 0 } }
func captured(at t: Int64) { locked { fc_capture(core, t) } }
func sent(seq: UInt32, bytes: Int, at t: Int64) { locked { fc_sent(core, seq, Int32(bytes), t) } }
func acked(seq: UInt32, at t: Int64) -> Bool { locked { fc_ack(core, seq, t) != 0 } }
func update(now: Int64) -> Int? {
locked { locked {
let old = value(0), tier = value(2) if target == 0 || target > bps { target = bps }
let result = Int(fc_update(core, now)) ceiling = bps
if value(0) < old || value(2) != tier {
events.append(["t": now, "e": "target \(value(0)) bit/s; tier \(value(2))"])
if events.count > 200 { events.removeFirst() }
}
return result == 0 ? nil : result
} }
} }
func state() -> [String: Any] {
locked { ["target": value(0), "ceiling": value(1), "tier": value(2), "fps": value(3), var fps: Int { locked { fpsLocked } }
"scale": Double(value(4)) / 100, "baseRtt": Double(value(5)) / 1000, private var fpsLocked: Int { min(maxFps, RateController.tiers[tier].fps) }
"inFlight": value(6), "slack": Double(value(7)) / 1000, "adapt": Self.enabled] } var scale: Double { locked { RateController.tiers[tier].scale } }
var baseRtt: Int64 { locked { baseline() } }
private func baseline() -> Int64 { rtts.map(\.v).min() ?? 0 }
/// How late a frame may be before the gate holds the next one: one frame
/// interval, plus room for the jitter this link normally has (1.5 times
/// its recent spread), so ordinary Wi-Fi jitter doesn't cost frames but a
/// real queue does. Updated in update().
private var slack: Int64 = 40_000
/// Whether a frame may be sent now without queueing behind earlier ones.
/// `counts`: a held frame is a sign of congestion (not when merely
/// re-checking whether a held frame can go yet).
func maySend(now: Int64, counts: Bool = true) -> Bool {
locked {
guard RateController.enabled, sawAck else { return true } // not heard from the viewer yet
// Unacknowledged for 2 s: gone with a reconnection, not in a queue.
unacked.removeAll { now - $0.sent > 2_000_000 }
guard let oldest = unacked.first else { return true }
// Age is what bounds latency. The count only stops a burst, and it
// allows a full round trip of frames, so a long but clear path
// (100 ms away) still gets every frame.
let interval = Int64(1_000_000 / max(1, fpsLocked))
let window = max(3, Int((baseline() + slack) / interval) + 1)
if unacked.count < window, now - oldest.sent <= baseline() + slack { return true }
if counts { held += 1 }
return false
}
} }
func eventList() -> [[String: Any]] { locked { events } }
/// A picture was captured (sent or not): with the frame sizes, what this
/// stream would send if the link allowed.
func captured(at t: Int64) {
locked {
captures.append(t)
if captures.count > 256 { captures.removeFirst(captures.count - 256) }
}
}
func sent(seq: UInt32, bytes: Int, at t: Int64) {
locked {
// Bounded even if the viewer never acknowledges (an old viewer, or
// the controller is off).
unacked.append((seq, t, bytes))
if unacked.count > 512 { unacked.removeFirst(unacked.count - 512) }
sentLog.append((t, bytes))
if sentLog.count > 1024 { sentLog.removeFirst(sentLog.count - 1024) }
}
}
/// The viewer has frame `seq`. Returns true if that may let a held frame go.
func acked(seq: UInt32, at now: Int64) -> Bool {
locked {
sawAck = true
guard let i = unacked.firstIndex(where: { $0.seq == seq }) else { return false }
let f = unacked[i]
unacked.removeSubrange(0...i) // TCP delivers in order: earlier ones arrived too
rtts.append((now, now - f.sent))
acked.append((now, f.bytes))
return true
}
}
/// Called every 100 ms. Returns the new bitrate target, or nil if the
/// controller is off.
func update(now: Int64) -> Int? {
locked {
rtts.removeAll { now - $0.t > 10_000_000 }
acked.removeAll { now - $0.t > 500_000 }
sentLog.removeAll { now - $0.t > 500_000 }
unacked.removeAll { now - $0.sent > 2_000_000 }
captures.removeAll { now - $0 > 1_000_000 }
guard RateController.enabled, ceiling > 0 else { return nil }
let base = baseline()
let spread = rtts.filter { now - $0.t < 2_000_000 }.map(\.v).sorted()
let jitter = spread.isEmpty ? 0 : spread[spread.count * 9 / 10] - base
let interval = Int64(1_000_000 / max(1, fpsLocked))
slack = interval + min(max(jitter * 3 / 2, 25_000), 80_000)
let recent = rtts.filter { now - $0.t < 300_000 }.map(\.v).sorted()
let queueing = recent.isEmpty ? 0 : recent[recent.count / 2] - base
let oldestAge = unacked.first.map { now - $0.sent } ?? 0
let stuck = oldestAge > base + 100_000
let delivered = acked.reduce(0) { $0 + $1.bytes } * 16 // bits/s over the last half second
let sending = sentLog.reduce(0) { $0 + $1.bytes } * 16
// Demand: captures per second (up to the tier's rate) times the
// average frame. A test card or a mostly still window wants far
// less than its budget; when the link hiccups, cutting its bitrate
// can't help, and it would only look link-limited afterwards.
if !sentLog.isEmpty { frameBytes = sentLog.reduce(0) { $0 + $1.bytes } / sentLog.count }
let demand = min(captures.count, fpsLocked) * frameBytes * 8
// Delay alone isn't our queue: Wi-Fi jitters by itself. It only
// counts while this stream uses a good part of its budget (so its
// own data could be what's queueing). Frames the gate had to hold,
// or one stuck in flight, show demand the link isn't carrying
// whatever was sent (the gate itself keeps what's sent low).
let busy = sending >= target / 2
// Twice in a row (200 ms), so one late ack doesn't count.
let signal = (busy && queueing > 40_000) || held >= 3 || stuck
let congested = signal && lastSignal
lastSignal = signal
let heldNow = held
held = 0
let floorBps = 300_000
if congested, now - lastDecrease > 300_000 {
// Down to a bit under what got through: at least a fifth off, at
// most half (a stall delivers nothing, but the link is still there).
let measured = Int(Double(delivered) * 0.9)
var next = max(floorBps, min(target * 4 / 5, max(measured, target / 2)))
// App-limited (it wants about half its budget or less): never
// below twice what it wants, however many cuts in a row. The
// extra quarter is hysteresis, so frame sizes wobbling at the
// floor don't switch the protection off.
if demand > 0, demand * 2 <= target * 5 / 4 { next = max(next, min(target, demand * 2)) }
target = next
lastDecrease = now
events.append((now, "down to \(target / 1000) kbit/s: queue \(queueing / 1000) ms, held \(heldNow), "
+ "oldest \(oldestAge / 1000) ms, base \(base / 1000) ms, sent \(sending / 1000) got \(delivered / 1000) "
+ "wants \(demand / 1000)"))
} else if !congested, now - lastDecrease > 1_000_000, now - lastIncrease > 250_000, target < ceiling,
sending > target * 6 / 10 || now - lastDecrease > 3_000_000 {
// Clear for a second and using what it has: probe up.
target = min(ceiling, Int(Double(target) * 1.1) + 50_000)
lastIncrease = now
}
// Fewer frames or pixels only help a stream that fills its budget;
// a small one (a still window, a test card) keeps its tier.
retier(now: now, linkLimited: sending >= target * 7 / 10)
if events.count > 200 { events.removeFirst(events.count - 200) }
return target
}
}
/// Steps down quickly, straight to the tier the bitrate supports, and back
/// up one tier at a time only when there's clearly room (hysteresis).
private func retier(now: Int64, linkLimited: Bool) {
let share = Double(target) / Double(max(ceiling, 1))
if tier < RateController.tiers.count - 1, share < RateController.floors[tier], linkLimited {
if belowSince == 0 { belowSince = now }
if now - belowSince > 500_000 {
tier = RateController.floors.firstIndex { share >= $0 } ?? RateController.tiers.count - 1
belowSince = 0
events.append((now, "tier \(tier)"))
}
} else {
belowSince = 0
}
if tier > 0, share > RateController.floors[tier - 1] * 1.25 {
if aboveSince == 0 { aboveSince = now }
if now - aboveSince > 2_000_000 {
tier -= 1
aboveSince = 0
events.append((now, "tier \(tier)"))
}
} else {
aboveSince = 0
}
}
func state() -> [String: Any] {
locked {
["target": target, "ceiling": ceiling, "tier": tier, "fps": min(maxFps, RateController.tiers[tier].fps),
"scale": RateController.tiers[tier].scale, "baseRtt": Double(baseline()) / 1000,
"inFlight": unacked.count, "slack": Double(slack) / 1000, "adapt": RateController.enabled]
}
}
func eventList() -> [[String: Any]] { locked { events.map { ["t": $0.0, "e": $0.1] } } }
} }
+1 -4
View File
@@ -5,10 +5,7 @@ set -eu
here=$(cd "$(dirname "$0")" && pwd) here=$(cd "$(dirname "$0")" && pwd)
out=${1:-$here/../bin/frame-mac-view} out=${1:-$here/../bin/frame-mac-view}
mkdir -p "$(dirname "$out")" mkdir -p "$(dirname "$out")"
obj=$(mktemp /tmp/frame-controller.XXXXXX)
trap 'rm -f "$obj"' EXIT
xcrun clang -O2 -target arm64-apple-macos14.0 -c "$here/../../desktop/controller.c" -o "$obj"
xcrun swiftc -O -swift-version 5 -target arm64-apple-macos14.0 \ xcrun swiftc -O -swift-version 5 -target arm64-apple-macos14.0 \
-import-objc-header "$here/Sources/CGVirtualDisplay.h" \ -import-objc-header "$here/Sources/CGVirtualDisplay.h" \
-o "$out" "$here"/Sources/*.swift "$obj" -o "$out" "$here"/Sources/*.swift
echo "built $out" echo "built $out"
+13 -55
View File
@@ -55,7 +55,6 @@ from urllib.parse import quote, urlencode # %20, not +: the agent's URLComponen
ROOT = Path(__file__).resolve().parent.parent ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "ui")) sys.path.insert(0, str(ROOT / "ui"))
import frame_macview # noqa: E402 import frame_macview # noqa: E402
import frame_pcview # noqa: E402
RESULTS = ROOT / "bench" / "results" RESULTS = ROOT / "bench" / "results"
PAGES = ROOT / "bench" / "pages" PAGES = ROOT / "bench" / "pages"
@@ -255,10 +254,7 @@ class LabView(frame_macview.MacView):
def ssh_runner(base): def ssh_runner(base):
def run(remote, stdin=None, timeout=30): def run(remote, stdin=None, timeout=30):
r = subprocess.run([*base, remote], input=stdin.encode("utf-8") if stdin is not None else None, r = subprocess.run([*base, remote], input=stdin, capture_output=True, text=True, timeout=timeout)
capture_output=True, timeout=timeout)
r.stdout = r.stdout.decode("utf-8", errors="replace")
r.stderr = r.stderr.decode("utf-8", errors="replace")
if r.returncode: if r.returncode:
e = RuntimeError((r.stderr or r.stdout).strip()) e = RuntimeError((r.stderr or r.stdout).strip())
e.stdout = r.stdout e.stdout = r.stdout
@@ -385,11 +381,7 @@ def run_scenario(args, scenario, agent_port, token, frame_ssh):
if args.browser_flag is not None: if args.browser_flag is not None:
mv.browser_flags = [f for f in args.browser_flag if f] mv.browser_flags = [f for f in args.browser_flag if f]
chrome = None chrome = None
src = args.source or "test" src = "test"
cpu = None
if args.pc:
mv.host = "windows" if sys.platform == "win32" else "linux"
mv.viewer_profile = "pc-view"
try: try:
if scenario in ("scroll", "type"): if scenario in ("scroll", "type"):
chrome = chrome_window(f"{scenario}.html") chrome = chrome_window(f"{scenario}.html")
@@ -413,10 +405,9 @@ def run_scenario(args, scenario, agent_port, token, frame_ssh):
since = max([f["s"] for f in first["frames"]] or [0]) since = max([f["s"] for f in first["frames"]] or [0])
captured_before = first["captured"] captured_before = first["captured"]
start_mac_us = mv.call("/stats", id=stream["id"])["now"] start_mac_us = mv.call("/stats", id=stream["id"])["now"]
if not args.pc: agent_pid = int(subprocess.run(["pgrep", "-f", "Frame Mac View Lab.app/Contents/MacOS/frame-mac-view"],
agent_pid = int(subprocess.run(["pgrep", "-f", "Frame Mac View Lab.app/Contents/MacOS/frame-mac-view"], capture_output=True, text=True).stdout.split()[0])
capture_output=True, text=True).stdout.split()[0]) cpu = CpuSampler(frame_ssh, agent_pid)
cpu = CpuSampler(frame_ssh, agent_pid)
if relay: if relay:
relay.begin() relay.begin()
expected = 0 # input events the harness asked the viewer for expected = 0 # input events the harness asked the viewer for
@@ -452,8 +443,7 @@ def run_scenario(args, scenario, agent_port, token, frame_ssh):
if not streams: if not streams:
raise SystemExit(f"{scenario}: the stream ended during the run (did the Frame go to sleep?)") raise SystemExit(f"{scenario}: the stream ended during the run (did the Frame go to sleep?)")
data = streams[0] data = streams[0]
if cpu: cpu.stop()
cpu.stop()
if args.raw: if args.raw:
Path(f"{args.raw}-{scenario}.json").write_text(json.dumps(data)) Path(f"{args.raw}-{scenario}.json").write_text(json.dumps(data))
result = summarize(scenario, data, start_mac_us, end_mac_us, args, relay, schedule, expected) result = summarize(scenario, data, start_mac_us, end_mac_us, args, relay, schedule, expected)
@@ -462,7 +452,7 @@ def run_scenario(args, scenario, agent_port, token, frame_ssh):
for e in data.get("events", []) if e["t"] >= start_mac_us] for e in data.get("events", []) if e["t"] >= start_mac_us]
result["captured"] = (captured_end if captured_end is not None else data["captured"]) - captured_before result["captured"] = (captured_end if captured_end is not None else data["captured"]) - captured_before
result["source_fps"] = round(result["captured"] / max(result["duration_s"], 1), 1) result["source_fps"] = round(result["captured"] / max(result["duration_s"], 1), 1)
result["cpu"] = cpu.summary() if cpu else {"host": "not sampled"} result["cpu"] = cpu.summary()
result["viewer"] = data["summary"].get("decoder", "") result["viewer"] = data["summary"].get("decoder", "")
result["show_s"] = show_s result["show_s"] = show_s
result["panel"] = shown.get("panel") result["panel"] = shown.get("panel")
@@ -470,8 +460,6 @@ def run_scenario(args, scenario, agent_port, token, frame_ssh):
result["route"] = mv.route result["route"] = mv.route
return result return result
finally: finally:
if cpu:
cpu.stop()
try: try:
mv.stop(src) mv.stop(src)
except Exception: # noqa: BLE001 - best effort except Exception: # noqa: BLE001 - best effort
@@ -682,8 +670,6 @@ def fmt(v):
def add_run_args(r): def add_run_args(r):
r.add_argument("--scenario", default="test,scroll,type") r.add_argument("--scenario", default="test,scroll,type")
r.add_argument("--pc", action="store_true", help="run the bundled PC host; use --scenario test or capture")
r.add_argument("--source", default="", help="PC source: window:ID, display:ID, or choose (Linux portal)")
r.add_argument("--duration", type=float, default=20) r.add_argument("--duration", type=float, default=20)
r.add_argument("--warmup", type=float, default=3) r.add_argument("--warmup", type=float, default=3)
r.add_argument("--quality", default="balanced", choices=list(frame_macview.QUALITY)) r.add_argument("--quality", default="balanced", choices=list(frame_macview.QUALITY))
@@ -731,7 +717,7 @@ def frame_ssh_for(args):
def run_suite(args, frame_ssh, quiet=False): def run_suite(args, frame_ssh, quiet=False):
agent_port, token = (args.pc_view.port, args.pc_view.token) if args.pc else lab_agent() agent_port, token = lab_agent()
results = [] results = []
for sc in args.scenario.split(","): for sc in args.scenario.split(","):
if not quiet: if not quiet:
@@ -758,12 +744,11 @@ def document(args, frame_ssh, results, **extra):
commit = git("rev-parse", "--short", "HEAD") + ("+dirty" if git("status", "--porcelain", "--", "mac", "ui") else "") commit = git("rev-parse", "--short", "HEAD") + ("+dirty" if git("status", "--porcelain", "--", "mac", "ui") else "")
return { return {
"label": args.label or args.cmd, "date": datetime.datetime.now().isoformat(timespec="seconds"), "commit": commit, "label": args.label or args.cmd, "date": datetime.datetime.now().isoformat(timespec="seconds"), "commit": commit,
"config": {"quality": args.quality, "mode": "native" if args.pc else args.mode, "net": args.net or "none", "delay_ms": args.delay, "config": {"quality": args.quality, "mode": args.mode, "net": args.net or "none", "delay_ms": args.delay,
"buffer_ms": args.buffer, "host": args.host or args.frame, "usb": args.usb, "ssh_opts": args.ssh_opt, "buffer_ms": args.buffer, "host": args.host or args.frame, "usb": args.usb, "ssh_opts": args.ssh_opt,
"encoder": os.environ.get("FRAME_MAC_VIEW_ENCODER", ""), "duration_s": args.duration, "encoder": os.environ.get("FRAME_MAC_VIEW_ENCODER", ""), "duration_s": args.duration,
"browser_flags": args.browser_flag if args.browser_flag is not None else frame_macview.BROWSER_FLAGS}, "browser_flags": args.browser_flag if args.browser_flag is not None else frame_macview.BROWSER_FLAGS},
"frame_build": build.strip().partition("=")[2], "mac": platform.mac_ver()[0], "frame_build": build.strip().partition("=")[2], "mac": platform.mac_ver()[0], "headset": standby[-40:],
"host_platform": platform.platform(), "source": args.source, "pc_host": args.pc, "headset": standby[-40:],
"scenarios": results, **extra, "scenarios": results, **extra,
} }
@@ -834,46 +819,19 @@ def main():
print("\nworse:\n " + "\n ".join(regs)) print("\nworse:\n " + "\n ".join(regs))
sys.exit(1 if regs else 0) sys.exit(1 if regs else 0)
if args.pc and (args.cmd == "ab" or args.scenario not in ("test", "capture")): lab_agent()
p.error("PC runs use --scenario test or --scenario capture; Mac automation is not portable")
if args.pc and args.scenario == "capture" and not args.source:
p.error("capture needs --source window:ID, display:ID, or choose")
if not args.pc:
lab_agent()
frame_ssh = frame_ssh_for(args) frame_ssh = frame_ssh_for(args)
# Keep the Mac's screen awake: virtual displays aren't removed while it sleeps. # Keep the Mac's screen awake: virtual displays aren't removed while it sleeps.
runs = 1 if args.cmd == "run" else args.repeat * len(args.arm) runs = 1 if args.cmd == "run" else args.repeat * len(args.arm)
awake = subprocess.Popen(["caffeinate", "-d", "-u", "-t", str(int(runs * (args.duration * 4 + 60) + 120))]) if sys.platform == "darwin" else None awake = subprocess.Popen(["caffeinate", "-d", "-u", "-t", str(int(runs * (args.duration * 4 + 60) + 120))])
args.pc_view = None
try: try:
if args.pc:
args.pc_view = frame_pcview.PCView(frame_ssh[:-1], ssh_runner(frame_ssh), frame_ssh[-1])
args.pc_view.ensure_agent()
if args.source == "choose":
args.pc_view.call("/permissions", method="POST")
print("Choose a window or screen in your desktop's sharing dialog.", flush=True)
deadline = time.monotonic()+125
while time.monotonic() < deadline:
state = args.pc_view.call("/status")
if not state.get("selecting"):
sources = args.pc_view.call("/windows")["windows"]
if not sources:
raise SystemExit(state.get("selectionError") or "Nothing was shared")
args.source = sources[-1]["src"]
break
time.sleep(.5)
else:
raise SystemExit("Sharing dialog timed out")
if args.cmd == "ab": if args.cmd == "ab":
arm_runs, table = ab(args, frame_ssh) arm_runs, table = ab(args, frame_ssh)
write(document(args, frame_ssh, [], arms=args.arm, runs=arm_runs, table=table), args) write(document(args, frame_ssh, [], arms=args.arm, runs=arm_runs, table=table), args)
return return
results = run_suite(args, frame_ssh) results = run_suite(args, frame_ssh)
finally: finally:
if args.pc_view: awake.terminate()
args.pc_view.shutdown()
if awake:
awake.terminate()
doc = document(args, frame_ssh, results) doc = document(args, frame_ssh, results)
write(doc, args) write(doc, args)
bad = [f"{r['scenario']} {k}" for r in results for k, v in r["grades"].items() if v in ("bad", "missing")] bad = [f"{r['scenario']} {k}" for r in results for k, v in r["grades"].items() if v in ("bad", "missing")]
+244
View File
@@ -0,0 +1,244 @@
// Control on the live view (tap, drag, hold, scroll, type), run in node against the real
// functions from ui/index.html with a fake canvas and a fake server.
import { readFileSync } from "fs";
const src = readFileSync(new URL("../../ui/index.html", import.meta.url), "utf8");
const grab = name => {
const one = src.match(new RegExp(`\\n((?:async )?function ${name}\\(.*\\}\\n)`)); // one-line function
if (one) return one[1];
const m = src.match(new RegExp(`(?:\\nconst ${name} = [^\\n]*\\n)|((?:async )?function ${name}\\([\\s\\S]*?\\n}\\n)`));
if (!m) throw new Error("not found: " + name);
return m[0];
};
const NAMES = ["panelKey", "ctrlAimedAt", "ctrlSameTarget", "isMoveEvent", "isRelease", "ctrlKeepable", "TAP_MOVE", "ctrlAim", "ctrlKeyEvent", "ctrlTouchCancel", "ctrlSend", "ctrlFlush", "ctrlMoveTo", "ctrlMoveBy", "ctrlSchedule",
"ctrlFlushMoves", "ctrlButton", "ctrlClick", "ctrlRelease", "ctrlFraction", "ctrlTouchDown", "centroid",
"ctrlTouchMove", "ctrlTouchUp", "ctrlTap", "ctrlText"];
const code = NAMES.map(grab).join("");
const fail = msg => { console.log("FAIL " + msg); process.exit(1); };
const tick = (ms = 0) => new Promise(r => setTimeout(r, ms));
function page({ mode = "abs", rect = { left: 0, top: 0, width: 640, height: 360 }, api } = {}) {
const target = { panel: { window: 42, display: ":1" } };
const ctrl = { on: true, queue: [], sending: false, state: "ready", message: "", move: null, rel: [0, 0], raf: 0,
held: new Set(), keys: new Set(), pointers: new Map(), g: null, retry: null };
const sent = [];
const canvas = { width: 1280, height: 720, getBoundingClientRect: () => rect };
const env = {
ctrl, $: () => canvas, ctrlMode: () => mode, ctrlShow: () => {}, toast: () => {},
ctrlTarget: () => (mode !== "abs" ? { ok: true } : target.panel ? { ok: true, panel: target.panel } : { why: "gone" }),
api: api || (async (path, body) => { sent.push(...body.events); return { state: "ready", sent: true }; }),
requestAnimationFrame: cb => { setTimeout(cb, 0); return 1; },
navigator: {},
};
const fns = new Function(...Object.keys(env), `let ctrlWarned = false;\n${code}
return { ctrlTouchDown, ctrlTouchMove, ctrlTouchUp, ctrlTouchCancel, ctrlSend, ctrlText, ctrlButton, ctrlKeyEvent, ctrlRelease, ctrlFlushMoves };`)(...Object.values(env));
const at = (id, x, y) => ({ pointerId: id, clientX: x, clientY: y });
return { ctrl, sent, target, ...fns, at };
}
// A tap lands where it was tapped: pointer there first, then the click.
{
const p = page();
p.ctrlTouchDown(p.at(1, 320, 90)); p.ctrlTouchUp(p.at(1, 320, 90));
await tick(10);
const [move, down, up] = p.sent;
if (!(move.fx === 0.5 && Math.abs(move.fy - 0.25) < 1e-9 && move.window === 42 && move.display === ":1")) fail("tap position " + JSON.stringify(move));
if (!(down.window === 42 && down.display === ":1")) fail("a press names its panel " + JSON.stringify(down));
if (!(down.button === "left" && down.down && up.button === "left" && up.down === false && p.sent.length === 3))
fail("tap click " + JSON.stringify(p.sent));
}
// The panel is letterboxed in a taller view: taps map inside it, taps on the bars do nothing.
{
const p = page({ rect: { left: 0, top: 0, width: 640, height: 480 } }); // 640x360 picture, 60px bars
p.ctrlTouchDown(p.at(1, 320, 150)); p.ctrlTouchUp(p.at(1, 320, 150));
p.ctrlTouchDown(p.at(2, 320, 10)); p.ctrlTouchUp(p.at(2, 320, 10));
await tick(10);
if (!(p.sent.length === 3 && Math.abs(p.sent[0].fy - 0.25) < 1e-9)) fail("letterbox " + JSON.stringify(p.sent));
}
// A drag is a mouse drag: down where it started, moves, up at the end.
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 100));
p.ctrlTouchMove(p.at(1, 140, 100)); await tick(5);
p.ctrlTouchMove(p.at(1, 200, 120)); await tick(5);
p.ctrlTouchUp(p.at(1, 200, 120)); await tick(10);
const kinds = p.sent.map(e => "fx" in e ? "move" : `${e.button}-${e.down ? "down" : "up"}`);
if (!(kinds[0] === "move" && kinds[1] === "left-down" && kinds.at(-1) === "left-up" && kinds.includes("move", 2)))
fail("drag " + kinds.join(","));
if (Math.abs(p.sent[0].fx - 100 / 640) > 1e-9) fail("drag starts where the finger went down");
if (p.ctrl.held.size) fail("drag left the button held");
}
// Press and hold is a right-click where the finger is.
{
const p = page();
p.ctrlTouchDown(p.at(1, 64, 36));
await tick(620);
p.ctrlTouchUp(p.at(1, 64, 36)); await tick(10);
const buttons = p.sent.filter(e => "button" in e).map(e => `${e.button}-${e.down}`);
if (buttons.join() !== "right-true,right-false") fail("hold " + buttons.join());
if (!(p.sent[0].fx === 0.1)) fail("hold position " + JSON.stringify(p.sent[0]));
}
// Two fingers scroll, and the content follows them (fingers up scrolls down).
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 200)); p.ctrlTouchDown(p.at(2, 200, 200));
p.ctrlTouchMove(p.at(1, 100, 180)); p.ctrlTouchMove(p.at(2, 200, 180));
p.ctrlTouchUp(p.at(1, 100, 180)); p.ctrlTouchUp(p.at(2, 200, 180));
await tick(10);
const dy = p.sent.filter(e => e.scroll).reduce((a, e) => a + e.scroll[1], 0);
if (!(dy === 40 && !p.sent.some(e => "button" in e))) fail("scroll " + JSON.stringify(p.sent));
}
// On the headset view it's a trackpad: drags move the pointer, taps click where it is.
{
const p = page({ mode: "rel" });
p.ctrlTouchDown(p.at(1, 100, 100));
p.ctrlTouchMove(p.at(1, 110, 100)); p.ctrlTouchMove(p.at(1, 120, 105));
p.ctrlTouchUp(p.at(1, 120, 105)); await tick(10);
p.ctrlTouchDown(p.at(1, 50, 50)); p.ctrlTouchUp(p.at(1, 50, 50)); await tick(10);
const moved = p.sent.filter(e => "dx" in e).reduce((a, e) => [a[0] + e.dx, a[1] + e.dy], [0, 0]);
if (!(moved[0] === 32 && moved[1] === 8)) fail("trackpad move " + JSON.stringify(moved));
if (p.sent.some(e => "fx" in e)) fail("trackpad sent an absolute position");
if (p.sent.filter(e => e.button === "left").length !== 2) fail("trackpad tap " + JSON.stringify(p.sent));
}
// Text: plain ASCII only, in chunks the server takes.
{
const p = page();
p.ctrlText("héllo " + "x".repeat(600));
await tick(10);
const text = p.sent.map(e => e.text).join("");
if (!(text === "hllo " + "x".repeat(600) && p.sent.every(e => e.text.length <= 500))) fail("text " + text.length);
}
// Not connected yet: clicks and keys wait with their position. The request fails: only releases wait.
{
let calls = 0;
const p = page({ api: async () => { calls++; if (calls === 1) return { state: "starting", sent: false }; throw new Error("offline"); } });
p.ctrlSend([{ fx: 0.1, fy: 0.1, window: 42 }, { button: "left", down: true }, { key: 30, down: true }]);
await tick(5);
const q = p.ctrl.queue;
if (!(q.length === 3 && "fx" in q[0] && q[1].button === "left" && q[2].key === 30)) fail("kept while starting " + JSON.stringify(q));
clearTimeout(p.ctrl.retry);
p.ctrl.queue = []; p.ctrl.retryAt = 0;
p.ctrlSend([{ button: "left", down: false }, { key: 31, down: true }, { dx: 3, dy: 1 }]);
await tick(5);
if (!(p.ctrl.queue.length === 1 && p.ctrl.queue[0].button === "left" && p.ctrl.queue[0].down === false))
fail("release kept on failure " + JSON.stringify(p.ctrl.queue));
if (calls !== 2) fail("retried in a tight loop: " + calls + " requests");
clearTimeout(p.ctrl.retry);
}
// Turning Control off (or the view losing focus) lets go of anything held.
{
const p = page();
p.ctrlButton("left", true);
p.ctrlRelease(); await tick(10);
if (!(p.sent.at(-1).button === "left" && p.sent.at(-1).down === false && !p.ctrl.held.size)) fail("release " + JSON.stringify(p.sent));
}
// Keys held on the Frame are let go too.
{
const p = page();
p.ctrlKeyEvent(42, true);
p.ctrlRelease(); await tick(10);
if (!(p.sent.at(-1).key === 42 && p.sent.at(-1).down === false && !p.ctrl.keys.size)) fail("key release " + JSON.stringify(p.sent));
}
// A long queue sheds moves and old scrolls, never a release.
{
const p = page({ api: () => new Promise(() => {}) }); // stuck request
p.ctrlSend([{ dx: 1, dy: 1 }]);
p.ctrlSend([{ button: "left", down: false }]);
for (let i = 0; i < 320; i++) p.ctrlSend([{ scroll: [0, 1] }]);
if (!p.ctrl.queue.some(e => e.button === "left" && e.down === false)) fail("trim dropped a release");
if (p.ctrl.queue.length > 300) fail("trim kept " + p.ctrl.queue.length);
}
// Broken on the Frame side: no hammering, and only releases wait.
{
let calls = 0;
const p = page({ api: async () => { calls++; return { state: "error", sent: false }; } });
p.ctrlSend([{ button: "left", down: true }, { button: "left", down: false }]);
await tick(50);
if (calls !== 1) fail("error response reposted " + calls + " times");
if (!(p.ctrl.queue.length === 1 && p.ctrl.queue[0].down === false)) fail("error kept " + JSON.stringify(p.ctrl.queue));
clearTimeout(p.ctrl.retry);
}
// Connecting: a tap keeps its position, so it lands where it was made.
{
const p = page({ api: async () => ({ state: "starting", sent: false }) });
p.ctrlTouchDown(p.at(1, 320, 90)); p.ctrlTouchUp(p.at(1, 320, 90));
await tick(10);
const q = p.ctrl.queue;
if (!("fx" in q[0] && q[1].button === "left")) fail("connecting tap " + JSON.stringify(q));
clearTimeout(p.ctrl.retry);
}
// Lifting one of two scrolling fingers doesn't jump the scroll or start a drag.
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 200)); p.ctrlTouchDown(p.at(2, 300, 200));
p.ctrlTouchUp(p.at(1, 100, 200));
p.ctrlTouchMove(p.at(2, 300, 199));
p.ctrlTouchUp(p.at(2, 300, 199)); await tick(10);
if (p.sent.length) fail("one finger left after scrolling " + JSON.stringify(p.sent));
}
// A cancelled touch isn't a tap.
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 100)); p.ctrlTouchCancel(p.at(1, 100, 100)); await tick(10);
if (p.sent.length) fail("cancel clicked " + JSON.stringify(p.sent));
}
// Press and hold on the bars around the picture does nothing.
{
const p = page({ rect: { left: 0, top: 0, width: 640, height: 480 } });
p.ctrlTouchDown(p.at(1, 320, 10)); await tick(620); p.ctrlTouchUp(p.at(1, 320, 10)); await tick(10);
if (p.sent.length) fail("hold on the bars " + JSON.stringify(p.sent));
}
// Taps only reach the panel in use, and only once its picture is the one on screen.
{
const fns = ["panelKey", "deskPanel", "ctrlTarget"].map(grab).join("");
const check = (desk, live = true) => new Function("desk", "live", "ctrlMode", fns + "; return ctrlTarget();")(desk, live, () => "abs");
const a = { display: ":1", window: 5 }, b = { display: ":0", window: 5 };
const base = { panels: [a, b], loaded: true, pick: "", focus: ":1/5", shown: ":1/5" };
if (!check(base).ok) fail("target: shown and in use");
if (check({ ...base, shown: ":0/5" }).ok) fail("target: the picture is another panel with the same id");
if (!/Capture or Live/.test(check({ ...base, shown: null }, false).why)) fail("target: stale picture message");
if (check({ ...base, pick: ":0/5", shown: ":0/5" }).ok) fail("target: a watched panel that isn't in use");
if (check({ ...base, focus: null }).ok) fail("target: nothing in use");
}
// Focus moves to another panel mid-gesture: the tap or hold does nothing.
{
const p = page();
p.ctrlTouchDown(p.at(1, 100, 100));
p.target.panel = { window: 43, display: ":1" };
p.ctrlTouchUp(p.at(1, 100, 100)); await tick(10);
p.target.panel = { window: 42, display: ":1" };
p.ctrlTouchDown(p.at(1, 100, 100));
p.target.panel = null;
await tick(620); p.ctrlTouchUp(p.at(1, 100, 100)); await tick(10);
if (p.sent.some(e => "button" in e)) fail("gesture outlived its panel " + JSON.stringify(p.sent));
}
// Trimming keeps a click together with its position.
{
const p = page({ api: () => new Promise(() => {}) });
p.ctrlSend([{ dx: 1, dy: 0 }]); // in flight forever
for (let i = 0; i < 100; i++) p.ctrlSend([{ scroll: [0, 1] }]);
p.ctrlSend([{ fx: 0.5, fy: 0.5, window: 42, display: ":1" }, { button: "left", down: true }, { button: "left", down: false }]);
for (let i = 0; i < 198; i++) p.ctrlSend([{ scroll: [0, 1] }]);
const q = p.ctrl.queue, i = q.findIndex(e => e.button === "left" && e.down);
if (i < 1 || !("fx" in q[i - 1])) fail("trim split a click from its position");
}
// Backing off holds for new input too.
{
let calls = 0;
const p = page({ api: async () => { calls++; return { state: "error", sent: false }; } });
p.ctrlSend([{ button: "left", down: false }]);
await tick(5);
for (let i = 0; i < 10; i++) { p.ctrlSend([{ key: 30, down: false }]); await tick(2); }
if (calls !== 1) fail("new input bypassed the backoff: " + calls + " requests");
clearTimeout(p.ctrl.retry);
}
// A long paste goes in several requests, so a release never waits behind all of it.
{
const batches = [];
const p = page({ api: async (path, body) => { batches.push(body.events); return { state: "ready", sent: true }; } });
p.ctrlText("y".repeat(450));
p.ctrlButton("left", false);
await tick(30);
if (!batches.every(b => b.reduce((a, e) => a + (e.text?.length || 0), 0) <= 100)) fail("a batch carried too much text");
if (batches.length < 5) fail("paste went in " + batches.length + " requests");
}
console.log("control gestures ok");
+257
View File
@@ -0,0 +1,257 @@
"""Fake-Frame session clock/actions plus real helper serialization and sensor probes."""
import os
import shutil
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import Mock, patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'ui'))
import frame_comfort as comfort
import frame_status as status
OPTIONS = {'action': 'start', 'minutes': 3, 'breakMinutes': 1, 'stillMinutes': 1,
'batteryAlert': True, 'heatAlert': True}
class SessionTests(unittest.TestCase):
def setUp(self):
self.s = comfort.new_session(OPTIONS, 0, 'boot-one')
self.warn, self.home = Mock(), Mock()
def step(self, now, **sample):
comfort.tick(self.s, now, sample, self.warn, self.home, read_clock=lambda: now)
def test_warning_then_home_never_closes_a_game(self):
self.step(119)
self.warn.assert_not_called()
self.step(120)
self.warn.assert_called_once()
self.step(179)
self.home.assert_not_called()
self.step(180)
self.home.assert_called_once()
self.assertFalse(self.s['active'])
self.step(181)
self.home.assert_called_once()
def test_late_wakeup_always_gets_a_full_warning_minute(self):
self.step(400)
self.home.assert_not_called()
self.step(459)
self.home.assert_not_called()
self.step(460)
self.home.assert_called_once()
def test_slow_warning_still_leaves_a_full_minute(self):
comfort.tick(self.s, 120, {}, self.warn, self.home, read_clock=lambda: 140)
self.assertEqual(self.s['warned'], 140)
self.step(180)
self.home.assert_not_called()
self.step(199)
self.home.assert_not_called()
self.step(200)
self.home.assert_called_once()
def test_failed_warning_never_stops_session(self):
self.warn.side_effect = RuntimeError('offline')
with self.assertRaises(RuntimeError):
self.step(200)
self.assertIsNone(self.s['warned'])
self.home.assert_not_called()
self.warn.side_effect = None
self.step(300)
self.step(359)
self.home.assert_not_called()
self.step(360)
self.home.assert_called_once()
def test_failed_home_stays_active_and_retries(self):
self.step(120)
self.home.side_effect = RuntimeError('Steam offline')
with self.assertRaises(RuntimeError):
self.step(180)
self.assertTrue(self.s['active'])
self.home.side_effect = None
self.step(185)
self.assertFalse(self.s['active'])
def test_cancel_prevents_all_actions(self):
self.s['active'] = False
self.step(999, battery={'percent': 1, 'status': 'Discharging'})
self.warn.assert_not_called()
self.home.assert_not_called()
self.assertEqual(self.s['events'], [])
def test_breaks_and_checkin_require_measured_activity(self):
self.step(20, activity=1)
self.step(40, activity=2)
self.step(60, activity=1)
self.assertEqual([e['kind'] for e in self.s['events']], ['break', 'still'])
self.step(70, activity=1)
self.assertEqual(len(self.s['events']), 2)
self.step(75, activity=3)
self.assertEqual(self.s['used'], 0)
self.assertFalse(self.s['stillSent'])
def test_unknown_activity_and_gaps_do_not_count_as_wear(self):
self.step(25)
self.assertEqual(self.s['used'], 0)
self.assertEqual(self.s['unavailable'], ['battery', 'temperature', 'activity'])
self.step(100, activity=1)
self.assertEqual(self.s['used'], 30)
def test_alerts_latch_and_rearm_without_battery_chatter(self):
low = {'percent': 10, 'status': 'Discharging'}
self.step(1, battery=low, thermal=['cpu'])
self.step(2, battery=low, thermal=['cpu'])
self.step(3)
self.assertEqual(len(self.s['events']), 2)
self.step(4, battery={'percent': 16, 'status': 'Discharging'}, thermal=[])
self.step(5, battery=low, thermal=[])
self.assertEqual(len(self.s['events']), 2)
self.step(6, battery={'percent': 22, 'status': 'Discharging'}, thermal=[])
self.step(7, battery=low, thermal=['cpu'])
self.assertEqual([e['kind'] for e in self.s['events']], ['battery', 'heat', 'battery', 'heat'])
def test_disabled_alerts_and_charging(self):
self.s['options']['heatAlert'] = False
self.step(1, battery={'percent': 2, 'status': 'Charging'}, thermal=['cpu'])
self.assertEqual(self.s['events'], [])
def test_invalid_options(self):
for key, value in [('minutes', 0), ('minutes', 241), ('minutes', True), ('minutes', 2.5),
('breakMinutes', -1), ('stillMinutes', '1'), ('heatAlert', 1)]:
with self.subTest(key=key, value=value), self.assertRaises(ValueError):
comfort.validate({**OPTIONS, key: value})
for value in (None, [], {'action': 'shutdown'}):
with self.assertRaises(ValueError):
comfort.validate(value)
def test_restart_invalidates_session_and_stale_worker_is_explicit(self):
with patch.object(comfort, 'boot', return_value='boot-one'), patch.object(comfort.time, 'time', return_value=999):
current = comfort.current(self.s, 100)
self.assertIn('not responding', current['error'])
self.assertEqual(current['time'], 999)
with patch.object(comfort, 'boot', return_value='boot-two'):
result = comfort.current(self.s, 100)
self.assertFalse(result['active'])
self.assertIn('restarted', result['error'])
@unittest.skipUnless(os.name == "posix", "on-headset state uses POSIX flock")
def test_real_state_commands_share_one_session_and_cancel(self):
with tempfile.TemporaryDirectory() as tmp, patch.object(comfort, 'ROOT', Path(tmp)), \
patch.object(comfort, 'boot', return_value='boot-one'), \
patch.object(comfort, 'clock', return_value=0), patch.object(comfort.subprocess, 'Popen') as spawn:
started = comfort.command(OPTIONS)
self.assertEqual(comfort.command({'action': 'status'})['id'], started['id'])
with self.assertRaises(ValueError):
comfort.command(OPTIONS)
self.assertFalse(comfort.command({'action': 'cancel'})['active'])
spawn.assert_called_once()
self.assertEqual((Path(tmp) / 'session.json').stat().st_mode & 0o777, 0o600)
@unittest.skipUnless(os.name == "posix", "on-headset state uses POSIX flock")
def test_cancel_clears_stale_worker_error(self):
with tempfile.TemporaryDirectory() as tmp, patch.object(comfort, 'ROOT', Path(tmp)), \
patch.object(comfort, 'boot', return_value='boot-one'), \
patch.object(comfort, 'clock', return_value=200):
with comfort.locked():
comfort.save(self.s)
self.assertIn('not responding', comfort.command({'action': 'status'})['error'])
cancelled = comfort.command({'action': 'cancel'})
self.assertFalse(cancelled['active'])
self.assertIsNone(cancelled['error'])
self.assertIsNone(comfort.command({'action': 'status'})['error'])
@unittest.skipUnless(os.name == "posix", "on-headset state uses POSIX flock")
def test_failed_spawn_leaves_session_inactive_and_retryable(self):
with tempfile.TemporaryDirectory() as tmp, patch.object(comfort, 'ROOT', Path(tmp)), \
patch.object(comfort, 'boot', return_value='boot-one'), \
patch.object(comfort, 'clock', return_value=0), patch.object(comfort.subprocess, 'Popen') as spawn:
spawn.side_effect = OSError('process limit')
with self.assertRaises(OSError):
comfort.command(OPTIONS)
failed = comfort.command({'action': 'status'})
self.assertFalse(failed['active'])
self.assertIn('Could not start', failed['error'])
spawn.side_effect = None
self.assertTrue(comfort.command(OPTIONS)['active'])
@unittest.skipUnless(os.name == "posix", "on-headset state uses POSIX flock")
def test_unreadable_state_is_preserved_and_can_be_replaced(self):
for contents in (b'{broken', b'\xff', b'null', b'[]', b'42', b'"x"'):
with self.subTest(contents=contents):
with tempfile.TemporaryDirectory() as tmp, patch.object(comfort, 'ROOT', Path(tmp)), \
patch.object(comfort, 'boot', return_value='boot-one'), \
patch.object(comfort, 'clock', return_value=0), patch.object(comfort.subprocess, 'Popen'):
(Path(tmp) / 'session.json').write_bytes(contents)
failed = comfort.command({'action': 'status'})
self.assertFalse(failed['active'])
self.assertIn('unreadable', failed['error'])
backups = list(Path(tmp).glob('session-unreadable-*.json'))
self.assertEqual(len(backups), 1)
self.assertEqual(backups[0].read_bytes(), contents)
self.assertTrue(comfort.command(OPTIONS)['active'])
def test_home_has_total_process_deadline_and_propagates_timeout(self):
with patch.object(comfort.subprocess, 'run', side_effect=subprocess.TimeoutExpired('home', 15)) as run:
with self.assertRaises(subprocess.TimeoutExpired):
comfort.home()
self.assertEqual(run.call_args.kwargs['timeout'], 15)
self.assertEqual(run.call_args.args[0][-1], '--home')
def test_native_warning_reports_failures_and_quotes_as_one_argument(self):
with patch.object(comfort.subprocess, 'run') as run:
run.return_value = subprocess.CompletedProcess([], 0, 'Notification succeeded', '')
comfort.notify('Save "now"; $(nothing)')
args = run.call_args.args[0]
self.assertEqual(args, [comfort.VRCMD, '--notify', 'Frame Control: Save "now"; $(nothing)'])
run.return_value.stdout = 'Notification failed with error 1'
with self.assertRaises(RuntimeError):
comfort.notify('test')
@unittest.skipUnless(shutil.which("node"), "Node exercises the fake Steam JS context")
def test_home_javascript_against_fake_steam_preserves_game(self):
# Same JS runs in Steam CDP. This fake records navigation and refuses any
# unexpected API call; it offers no shutdown or terminate-game primitive.
js = '''let running = [123], path = '/routes/library/app/123', visible = false;
const location = {get pathname() {return path;}};
const SteamUIStore = {Navigate(p) {path = '/routes' + p;}};
const SteamClient = {OpenVR: {VROverlay: {
async ShowDashboard(key) {if (key !== 'valve.steam.gamepadui.main') throw Error(key); visible = true;},
async IsDashboardVisible() {return visible;}
}}};
'''
js += comfort.HOME_JS + '.then(result => console.log(JSON.stringify({result, running, visible})));'
r = subprocess.run(['node', '-e', js], capture_output=True, text=True, check=True)
result = json.loads(r.stdout)
self.assertEqual(result['running'], [123])
self.assertTrue(result['visible'])
self.assertEqual(result['result']['path'], '/routes/library/home')
class SensorTests(unittest.TestCase):
def test_hot_trip_uses_its_own_zone_not_hottest_unrelated_chip(self):
values = {'/z/a/temp': '90000', '/z/a/trip_point_0_type': 'hot', '/z/a/trip_point_0_temp': '110000',
'/z/b/temp': '45000', '/z/b/trip_point_0_type': 'hot', '/z/b/trip_point_0_temp': '44000', '/z/b/type': 'battery'}
def glob(pattern):
if pattern.endswith('thermal_zone*'):
return ['/z/a', '/z/b']
return [pattern.replace('*', '0')]
with patch.object(status.glob, 'glob', side_effect=glob), patch.object(status, 'read', side_effect=values.get):
self.assertEqual(status.thermal_alerts(), [{'zone': 'battery', 'tempC': 45, 'limitC': 44}])
def test_missing_thermal_and_activity_are_unknown(self):
with patch.object(status.glob, 'glob', return_value=[]):
self.assertIsNone(status.thermal_alerts())
with patch.object(status, 'run', return_value='unavailable'):
self.assertIsNone(status.activity_level())
for malformed in ('{}', '[null, 42, "bad"]'):
with patch.object(status, 'run', return_value=malformed):
self.assertIsNone(status.activity_level())
with patch.object(status, 'run', return_value='[{"operation":"status","activity_level":3}]'):
self.assertEqual(status.activity_level(), 3)
+60
View File
@@ -0,0 +1,60 @@
"""Run the actual shared page's comfort renderer against a minimal DOM/bridge."""
import pathlib
import shutil
import subprocess
import unittest
ROOT = pathlib.Path(__file__).resolve().parents[1]
@unittest.skipUnless(shutil.which('node'), 'Node exercises the shared page JS')
class ComfortUI(unittest.TestCase):
def test_notification_failure_survives_poll_until_success(self):
page = (ROOT / 'ui/index.html').read_text(encoding='utf-8')
code = page[page.index('let comfortBusy ='):page.index('async function pollComfort()')]
setup = r'''
const assert = require('node:assert/strict');
const elements = new Map();
const $ = id => {
if (!elements.has(id)) elements.set(id, {textContent:'', hidden:true, disabled:false, type: 'number'});
return elements.get(id);
};
let denied = 0;
const window = {frameApp:{notify:async()=>{denied++;throw Error('permission denied');}}};
const log = ()=>{}, toast = ()=>{};
'''
checks = r'''
(async()=>{
const active = {id:'session-one',active:true,time:100,remaining:120,
options:{minutes:2,breakMinutes:1,stillMinutes:1,batteryAlert:true,heatAlert:true},
events:[{id:'event-one',kind:'battery',time:99,message:'Low battery'}]};
renderComfort(active); // initial history must not replay even a fresh event
assert.equal(denied,0);
assert.equal($('comfortAnnouncement').textContent,'');
active.events.push({id:'event-two',kind:'break',time:100,message:'Take a break'});
renderComfort(active);
await new Promise(resolve=>setImmediate(resolve));
assert.equal(denied,1);
assert.equal($('comfortAnnouncement').textContent,'Take a break');
assert.equal($('comfortNotificationStatus').hidden,false);
assert.match($('comfortNotificationStatus').textContent,/notification settings/);
renderComfort({...active,time:105}); // the next normal poll must not erase failure
assert.equal($('comfortNotificationStatus').hidden,false);
assert.equal($('sessionStart').disabled,true);
assert.equal($('sessionMinutes').disabled,true);
assert.equal($('sessionCancel').disabled,false);
let requests=0;
window.frameApp.notify=async()=>{requests++;};
renderComfort({...active,time:106});
assert.equal($('comfortAnnouncement').textContent,'Take a break');
assert.equal(requests,0); // polling does not replay an already-seen event
await localNotification('test',true);
assert.equal($('comfortNotificationStatus').hidden,true);
renderComfort({...active,active:false});
assert.equal($('sessionStart').disabled,false);
assert.equal($('sessionMinutes').disabled,false);
assert.equal($('sessionCancel').disabled,true);
})().catch(e=>{console.error(e);process.exitCode=1;});
'''
result = subprocess.run(['node', '-e', setup + code + checks], capture_output=True, text=True)
self.assertEqual(result.returncode, 0, result.stderr)
+1 -4
View File
@@ -45,10 +45,7 @@ class Helpers(unittest.TestCase):
@unittest.skipUnless(shutil.which("bash"), "needs bash") @unittest.skipUnless(shutil.which("bash"), "needs bash")
@unittest.skipIf(os.name == "nt", "Windows' bash.exe is WSL's launcher, and runners have no distribution") @unittest.skipIf(os.name == "nt", "Windows' bash.exe is WSL's launcher, and runners have no distribution")
def test_launch_script_parses(self): def test_launch_script_parses(self):
bash = str(Path(os.environ.get("ProgramFiles", "C:/Program Files")) / "Git/bin/bash.exe") if sys.platform == "win32" else "bash" r = subprocess.run(["bash", "-n"], input=frame_macview.LAUNCH, text=True, capture_output=True)
if sys.platform == "win32" and not Path(bash).exists():
self.skipTest("Git Bash is not installed (WSL bash is not a local shell)")
r = subprocess.run([bash, "-n"], input=frame_macview.LAUNCH.encode("utf-8"), capture_output=True)
self.assertEqual(r.returncode, 0, r.stderr) self.assertEqual(r.returncode, 0, r.stderr)
def test_show_checks_the_source_before_anything_else(self): def test_show_checks_the_source_before_anything_else(self):
+102
View File
@@ -0,0 +1,102 @@
"""Fake-Frame panel responses and the headset page's real HTTP guards."""
import http.client
import json
from pathlib import Path
import sys
import threading
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / 'ui'))
import frame_panels as panels
# Shape verified with vrcmd on SteamVR 2.18.1 / BUILD_ID 20260925.6191901.
OVERLAYS = """---- OVERLAYS ----
'valve.steam.desktopgame.12' -- 'Alex's <notes>', 1920x1080 visible VROverlayType_Dashboard_Main
'valve.steam.desktopgame.12.thumb' -- 'Thumb', not_visible VROverlayType_Dashboard_Thumbnail
'valve.steam.desktopgame.12.layer1' -- 'Layer', visible VROverlayType_Subview
'system.pointer' -- 'Pointer', visible VROverlayType_Basic
'valve.steam.desktopgame.13' -- 'Other', not_visible VROverlayType_Dashboard_Main
"""
class Panels(unittest.TestCase):
def test_enumerates_only_main_panels_including_hidden(self):
rows = panels.parse_overlays(OVERLAYS)
self.assertEqual(len(rows), 2)
self.assertEqual(rows[0]['title'], "Alex's <notes>")
self.assertTrue(rows[0]['visible'])
self.assertFalse(rows[1]['visible'])
def test_unavailable_runtime_is_not_an_empty_workspace(self):
with self.assertRaises(panels.PanelError):
panels.parse_overlays('SteamVR not running')
self.assertEqual(panels.parse_overlays('---- OVERLAYS ----'), [])
@patch.object(panels, 'run', return_value=OVERLAYS)
def test_focus_revalidates_and_dispatches_without_shell(self, run):
key = 'valve.steam.desktopgame.12'
self.assertEqual(panels.focus(key)['requested'], key)
self.assertEqual(run.call_args.args[0], [panels.VRCMD, '--showdashboard', key])
@patch.object(panels, 'run', return_value=OVERLAYS)
def test_closed_or_injected_panel_never_dispatches(self, run):
for key in (None, 12, 'x;touch /tmp/bad', '../other', 'missing'):
with self.assertRaises(panels.PanelError):
panels.focus(key)
self.assertEqual(run.call_count, 1) # only valid-looking 'missing' enumerates
class HeadsetHTTP(unittest.TestCase):
def setUp(self):
self.server = panels.HTTPServer(('127.0.0.1', 0), panels.Handler)
self.server.key = 'test-key'
self.server.closing = False
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True)
self.thread.start()
def tearDown(self):
self.server.shutdown()
self.thread.join()
self.server.server_close()
def request(self, path, body=None, headers=None):
c = http.client.HTTPConnection('127.0.0.1', self.server.server_port, timeout=5)
try:
c.request('POST' if body is not None else 'GET', path, body=body, headers=headers or {})
r = c.getresponse()
return r.status, r.read().decode()
finally:
c.close()
def test_page_is_public_but_contains_no_key_or_private_titles(self):
status, page = self.request('/')
self.assertEqual(status, 200)
self.assertNotIn('test-key', page)
self.assertIn('textContent=p.title', page) # titles never become HTML
self.assertEqual(self.request('/panels')[0], 403)
@patch.object(panels, 'state', return_value={'panels': []})
def test_auth_host_and_origin_checks(self, state):
auth = {'X-Panel-Key': 'test-key'}
self.assertEqual(self.request('/panels', headers=auth)[0], 200)
for extra in ({'Host': 'evil.test'}, {'Origin': 'https://evil.test'}, {'X-Panel-Key': 'wrong'}):
self.assertEqual(self.request('/panels', headers={**auth, **extra})[0], 403)
self.assertEqual(state.call_count, 1)
@patch.object(panels, 'focus', return_value={'requested': 'panel'})
def test_post_validation_and_close(self, focus):
auth = {'X-Panel-Key': 'test-key'}
for body in ('[]', '{broken', '0', '"text"', 'x' * 1025):
self.assertEqual(self.request('/focus', body, auth)[0], 400)
self.assertEqual(focus.call_count, 0)
self.assertEqual(self.request('/focus', '{"key":"panel"}', auth)[0], 200)
self.assertEqual(focus.call_args.args, ('panel',))
self.assertEqual(self.request('/close', '{}', auth)[0], 200)
self.assertTrue(self.server.closing)
@patch.object(panels, 'state', side_effect=panels.PanelError('offline'))
def test_offline_is_an_error_not_a_successful_empty_list(self, state):
status, body = self.request('/panels', headers={'X-Panel-Key': 'test-key'})
self.assertEqual(status, 502)
self.assertEqual(json.loads(body), {'error': 'offline'})
-64
View File
@@ -1,64 +0,0 @@
"""Frozen regression trace from the original mac-in-headset Swift controller.
The 600-state reference was produced at 1b90c64, before the C extraction:
60 seconds with a slow/congested middle, then recovery. No generated behavior
from the new implementation is used to calculate the expected digest.
"""
import ctypes as C
import hashlib
import json
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
import unittest
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / 'ui'))
from frame_pc_capture import Controller
class ControllerTrace(unittest.TestCase):
def test_original_mac_congestion_and_recovery(self):
cc = shutil.which('cc') or shutil.which('gcc')
if not cc:
self.skipTest('C compiler not installed; native build jobs cover this trace')
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / ('trace.dll' if sys.platform == 'win32' else 'trace.so')
subprocess.run([cc, '-O2', '-shared', '-fPIC', str(ROOT/'desktop/controller.c'), '-o', str(path)], check=True, capture_output=True)
lib = C.CDLL(str(path))
sig = {'new': (C.c_void_p, [C.c_int, C.c_int]), 'free': (None, [C.c_void_p]),
'ceiling': (None, [C.c_void_p, C.c_int]), 'value': (C.c_int64, [C.c_void_p, C.c_int]),
'capture': (None, [C.c_void_p, C.c_int64]), 'gate': (C.c_int, [C.c_void_p, C.c_int64, C.c_int]),
'sent': (None, [C.c_void_p, C.c_uint32, C.c_int, C.c_int64]),
'ack': (C.c_int, [C.c_void_p, C.c_uint32, C.c_int64]), 'update': (C.c_int, [C.c_void_p, C.c_int64])}
for name, (ret, args) in sig.items():
fn = getattr(lib, 'fc_'+name)
fn.restype, fn.argtypes = ret, args
c = Controller(lib, 60, 8000000)
due, states, seq = [], [], 0
try:
for tick in range(6000):
now = 10000000+tick*10000
for t, s in due:
if t <= now:
c.call('ack', s, now)
due = [(t, s) for t, s in due if t > now]
if tick % 2 == 0:
c.call('capture', now)
if c.call('gate', now, 1):
seq += 1
c.call('sent', seq, 8000 if tick < 1000 or tick > 4000 else 22000, now)
due.append((now+(240000 if 1500 < tick < 3500 else 20000), seq))
if tick % 10 == 0:
c.update(now)
x = c.state()
states.append([x[k] for k in ('target', 'ceiling', 'tier', 'fps', 'inFlight')] +
[round(x['scale']*100), round(x['baseRtt']*1000), round(x['slack']*1000)])
digest = hashlib.sha256(json.dumps(states, separators=(',', ':')).encode()).hexdigest()
self.assertEqual(digest, '1932f16aafce5c9157bcd2aba5d90719a366f9999422920aaeae78a6cdec08f6')
finally:
c.close()
if sys.platform == 'win32':
import _ctypes
_ctypes.FreeLibrary(lib._handle)
-388
View File
@@ -1,388 +0,0 @@
"""PC adapters: grants, input cleanup, launcher reuse and native codec contract.
Native tests require the bundled libraries; CI sets REQUIRE_NATIVE so a missing
bundle is a failure, never a silent skip. No desktop consent or GPU is faked as
successful real-host capture.
"""
import ctypes
import http.client
import json
import os
from pathlib import Path
import shutil
import struct
import subprocess
import sys
import threading
import time
import unittest
from unittest import mock
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / 'ui'))
import frame_pc_agent as agent
import frame_pc_capture as capture
import frame_pcview
import frame_macview
from frame_stream_stats import Stats
from test_macview import WS
class GrantsTests(unittest.TestCase):
def test_source_bound_retry_ack_and_stop(self):
g = agent.Grants('master')
ticket = g.ticket('window:1')
self.assertIsNone(g.redeem('window:2', {'t': ticket}))
key = g.redeem('window:1', {'t': ticket})
self.assertEqual(key, g.redeem('window:1', {'t': ticket}))
g.ack(key)
self.assertIsNone(g.redeem('window:1', {'t': ticket}))
self.assertEqual(key, g.redeem('window:1', {'r': key}))
self.assertIsNone(g.redeem('window:2', {'r': key}))
g.revoke('window:1')
self.assertIsNone(g.redeem('window:1', {'r': key}))
def test_expiry_and_stop_before_redeem(self):
g = agent.Grants('master')
with mock.patch.object(agent.time, 'monotonic', return_value=1):
ticket = g.ticket('test')
with mock.patch.object(agent.time, 'monotonic', return_value=62):
self.assertIsNone(g.redeem('test', {'t': ticket}))
ticket = g.ticket('test')
g.revoke()
self.assertIsNone(g.redeem('test', {'t': ticket}))
class AdapterTests(unittest.TestCase):
def test_windows_uses_wgc_and_media_foundation(self):
for kind in ('window', 'display'):
pipeline = capture.pipeline(dict(src=kind+':42'), 'win32', 'mfh264enc', 640, 360, 30, 1000000)
self.assertIn('capture-api=wgc', pipeline)
self.assertIn('window-handle=42' if kind == 'window' else 'monitor-handle=42', pipeline)
self.assertIn('mfh264enc name=enc low-latency=true bframes=0', pipeline)
self.assertNotIn('drop=true', pipeline)
with self.assertRaises(ValueError):
capture.pipeline(dict(src='window:42 ! fakesink'), 'win32', 'mfh264enc', 640, 360, 30, 1000000)
def test_linux_uses_only_portal_fd_and_node(self):
p = capture.pipeline(dict(src='window:consented', fd=19, node=47), 'linux', 'x264enc', 640, 360, 30, 1000000)
self.assertIn('pipewiresrc fd=19 path=47', p)
self.assertIn('tune=zerolatency', p)
self.assertEqual(capture.dimensions(1920, 1080, 1280), (1280, 720))
def test_portal_input_clamps_and_releases_held_state(self):
native = mock.Mock()
native.lib.fc_portal_input.return_value = 1
inp = capture.PortalInput(native, dict(portal=123, w=100, h=50))
inp.handle(dict(t='m', e='down', b=0, x=2, y=-1))
inp.handle(dict(t='k', e='down', key='Control'))
inp.release()
calls = native.lib.fc_portal_input.call_args_list
self.assertEqual(calls[0].args, (123, 0, 99, 0, 0, 0))
self.assertIn(mock.call(123, 1, 0, 0, 0x110, 0), calls)
self.assertIn(mock.call(123, 3, 0, 0, 0xffe3, 0), calls)
self.assertFalse(inp.buttons or inp.keys)
def test_windows_release_and_negative_monitor_origin(self):
host = mock.Mock()
host.user.GetSystemMetrics.side_effect = [-1920, 0, 3840, 1080]
inp = capture.WindowsInput(host, dict(src='display:4', x=-1920, y=0, w=1920, h=1080))
inp.handle(dict(t='m', e='down', b=2, x=0, y=0))
self.assertEqual(host.send.call_args_list[0], mock.call(mouse=(0, 0, 0, 0xc001)))
inp.handle(dict(t='k', e='down', code='ControlLeft', key='Control'))
inp.release()
self.assertIn(mock.call(mouse=(0, 0, 0, 0x10)), host.send.call_args_list)
self.assertIn(mock.call(key=(162, 0, 2)), host.send.call_args_list)
def test_refused_windows_focus_never_clicks_the_covering_app(self):
host = mock.Mock()
host.rect.return_value = (0, 0, 800, 600)
host.user.GetForegroundWindow.return_value = 99
host.user.SetForegroundWindow.return_value = False
inp = capture.WindowsInput(host, dict(src='window:12', w=800, h=600))
with self.assertRaisesRegex(RuntimeError, 'could not focus'):
inp.handle(dict(t='m', e='down', b=0, x=.5, y=.5))
host.send.assert_not_called()
def test_pc_reuses_launcher_and_namespaces_panel_ids(self):
view = frame_pcview.PCView(['ssh'], mock.Mock(return_value='panel created'), 'frame')
view.call = mock.Mock(side_effect=lambda path, **kw: {'screen': True, 'ticket': 'one-use', 'streams': []})
view.ensure_tunnel = mock.Mock()
view.remote_port = 47900
view.show('window:12')
self.assertEqual(view.run.call_args.kwargs['stdin'], frame_macview.LAUNCH)
self.assertNotEqual(frame_macview.panel_id('window:12'), frame_macview.panel_id('window:12', 'windows'))
self.assertFalse(view.prefer_usb)
with self.assertRaises(frame_macview.MacViewError):
view.show('separate:12')
def test_session_end_releases_native_input_only_once(self):
session = object.__new__(agent.Session)
session.lock, session.input_lock = threading.RLock(), threading.RLock()
session.stop_event = threading.Event()
session.ws, session.input = mock.Mock(), mock.Mock()
session.released = False
session.end()
session.end()
session.input.release.assert_called_once()
self.assertTrue(session.stop_event.is_set())
def test_idle_source_is_not_a_stall(self):
# PipeWire/WGC deliver frames only on damage: a static desktop sends
# its first frame and then nothing. That must not end the session.
session = object.__new__(agent.Session)
session.lock, session.pending = threading.RLock(), {}
session.stats = Stats(lambda: 0)
session.stats.captured = 1
self.assertIsNone(session.stalled(60000000, 0, 0))
# A frame that entered the encoder and never came out is a fault.
session.pending[5] = dict(cap=0, arr=0, e0=1000000, tier=0, br=0)
self.assertIn('encoder', session.stalled(12000000, 0, 0))
self.assertIsNone(session.stalled(10000000, 0, 0))
# No initial frame at all after the pipeline opened is a fault.
session.pending.clear()
self.assertIn('capture', session.stalled(12000000, 0, 1))
self.assertIsNone(session.stalled(9000000, 0, 1))
def test_controller_is_inert_after_close(self):
lib = mock.Mock()
lib.fc_new.return_value = 1234
lib.fc_value.return_value = 100
controller = capture.Controller(lib, 60, 5000000)
controller.close()
lib.fc_free.assert_called_once_with(1234)
lib.reset_mock()
state = controller.state()
self.assertEqual((state['fps'], state['target'], state['scale']), (60, 0, 1))
self.assertEqual(controller.call('gate', 1, 1), 0)
self.assertEqual(controller.update(1), 0)
controller.close()
self.assertEqual(lib.method_calls, []) # nothing touched the freed pointer
def test_library_path_prepend_adds_no_empty_entry(self):
env = {}
frame_pcview.prepend(env, 'LD_LIBRARY_PATH', '/opt/fc/lib')
self.assertEqual(env['LD_LIBRARY_PATH'], '/opt/fc/lib')
frame_pcview.prepend(env, 'LD_LIBRARY_PATH', '/x')
self.assertEqual(env['LD_LIBRARY_PATH'], '/x' + os.pathsep + '/opt/fc/lib')
with mock.patch.dict(os.environ, clear=True):
env = frame_pcview.PCView([], lambda *a: None, 'frame').agent_environment()
for name in ('PATH', 'LD_LIBRARY_PATH'):
parts = env.get(name, 'x').split(os.pathsep)
self.assertNotIn('', parts, name)
def test_malformed_viewer_input_is_dropped_not_fatal(self):
session = object.__new__(agent.Session)
session.lock, session.input_lock = threading.RLock(), threading.RLock()
session.stop_event, session.key_event = threading.Event(), threading.Event()
session.src, session.codec, session.key, session.w, session.h = 'display:1', 'h264', 'r', 100, 100
session.source = dict(src='display:1', w=100, h=100, portal=1)
session.input_enabled, session.released = True, False
session.native = mock.Mock()
session.native.now.return_value = 0
session.controller, session.agent = mock.Mock(), mock.Mock()
session.stats = Stats(lambda: 0)
session.produce = lambda: None
lib = mock.Mock()
lib.fc_portal_input.return_value = 1
session.input = capture.PortalInput(mock.Mock(lib=lib), session.source)
session.ws = mock.Mock()
session.ws.receive.side_effect = [
dict(t='m', x='left', y=0), dict(t='wheel', dx=[1]), dict(t='k', e='down', key=None, i='x'),
dict(t='fd', drop='many'), dict(t='rx', s=[1]), dict(t='m', x=.5, y=.5), ConnectionError('closed')]
with self.assertRaises(ConnectionError):
session.start()
# Every malformed message was skipped; the good move after them still ran.
moves = [c.args for c in lib.fc_portal_input.call_args_list if c.args[1] == 0]
self.assertAlmostEqual(moves[-1][2], .5*99)
self.assertEqual(session.ws.receive.call_count, 7)
def test_stats_keep_capture_time_and_bound_records(self):
stats = Stats(lambda: 10000000)
stats.input(dict(t='m', i=1, tv=9999990))
f = stats.add(cap=10000001, arr=10000002, e0=10000003, e1=10000004)
self.assertEqual(f['echo'], 1)
stats.report(dict(t='rx', s=f['s'], r=10000010))
stats.report(dict(t='fd', f=[[f['s'], 10000011, 10000012, 10000013]]))
self.assertEqual(stats.snapshot(settle=0)['inputs'][0]['frame'], f['s'])
for _ in range(4100):
stats.add(cap=10000001)
self.assertEqual(len(stats.frames), 4096)
class NativeTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
if not capture.LIBRARY.exists():
if os.environ.get('FRAME_PC_REQUIRE_NATIVE') == '1':
raise AssertionError('CI required native PC libraries but they were not built')
raise unittest.SkipTest('native PC libraries not built on this host')
cls.token = 'test-' + os.urandom(8).hex()
env = frame_pcview.PCView([], lambda *a: None, 'frame').agent_environment()
env['FRAME_MAC_VIEW_TOKEN'] = cls.token
cls.proc = subprocess.Popen([sys.executable, str(ROOT/'ui/frame_pc_agent.py'), 'serve', '--port', '0',
'--page', str(ROOT/'ui/mac-view.html'), '--exit-on-eof'], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, text=True, env=env)
line = cls.proc.stdout.readline()
if 'listening' not in line:
raise AssertionError('Native agent failed: ' + cls.proc.stderr.read())
cls.port = int(line.rsplit(':', 1)[1])
@classmethod
def tearDownClass(cls):
cls.proc.stdin.close()
try:
rc = cls.proc.wait(10)
except subprocess.TimeoutExpired:
cls.proc.kill()
raise AssertionError('Native agent did not stop after EOF')
error = cls.proc.stderr.read()
cls.proc.stdout.close()
cls.proc.stderr.close()
if rc:
raise AssertionError('Native agent exited %s: %s' % (rc, error))
def request(self, path, method='GET'):
conn = http.client.HTTPConnection('127.0.0.1', self.port, timeout=10)
try:
conn.request(method, path)
response = conn.getresponse()
return response.status, response.read()
finally:
conn.close()
def tearDown(self):
self.request('/close?k='+self.token, 'POST')
@unittest.skipUnless(sys.platform.startswith('linux'), 'Linux PipeWire client libraries')
def test_bundled_pipewire_context_needs_no_system_modules(self):
# No portal request, remote connection or desktop capture: create and
# destroy a private client context to test the dynamically loaded SPA
# and protocol modules that ldd cannot discover.
code = """
import ctypes as C, os
lib = C.CDLL(os.path.join(os.environ['FRAME_PC_NATIVE'], 'lib', 'libpipewire-0.3.so.0'))
for name, ret, args in [
('pw_init', None, [C.c_void_p, C.c_void_p]),
('pw_main_loop_new', C.c_void_p, [C.c_void_p]),
('pw_main_loop_get_loop', C.c_void_p, [C.c_void_p]),
('pw_context_new', C.c_void_p, [C.c_void_p, C.c_void_p, C.c_size_t]),
('pw_context_find_factory', C.c_void_p, [C.c_void_p, C.c_char_p]),
('pw_context_find_spa_lib', C.c_char_p, [C.c_void_p, C.c_char_p]),
('pw_context_destroy', None, [C.c_void_p]),
('pw_main_loop_destroy', None, [C.c_void_p])]:
fn = getattr(lib, name)
fn.restype, fn.argtypes = ret, args
lib.pw_init(None, None)
loop = lib.pw_main_loop_new(None)
assert loop, 'bundled SPA loop support did not load'
context = lib.pw_context_new(lib.pw_main_loop_get_loop(loop), None, 0)
assert context, 'bundled PipeWire client modules did not load'
assert lib.pw_context_find_factory(context, b'adapter'), 'video adapter factory is missing'
# The exported factory is video.adapt, which needs a live follower node to
# instantiate. Check its configured library and exported factory without
# pretending a headless context is a consented video stream.
plugin_name = lib.pw_context_find_spa_lib(context, b'video.adapt')
assert plugin_name, 'video.adapt has no bundled library mapping'
plugin = C.CDLL(os.path.join(os.environ['SPA_PLUGIN_DIR'], plugin_name.decode() + '.so'))
class Factory(C.Structure):
_fields_ = [('version', C.c_uint32), ('name', C.c_char_p)]
plugin.spa_handle_factory_enum.argtypes = [C.POINTER(C.POINTER(Factory)), C.POINTER(C.c_uint32)]
plugin.spa_handle_factory_enum.restype = C.c_int
factory, index, names = C.POINTER(Factory)(), C.c_uint32(), []
while plugin.spa_handle_factory_enum(C.byref(factory), C.byref(index)) > 0:
names.append(factory.contents.name)
assert b'video.adapt' in names, 'bundled SPA video adapter factory is missing'
lib.pw_context_destroy(context)
lib.pw_main_loop_destroy(loop)
"""
env = frame_pcview.PCView([], lambda *a: None, 'frame').agent_environment()
env['FRAME_PC_NATIVE'] = str(capture.NATIVE)
result = subprocess.run([sys.executable, '-c', code], env=env, capture_output=True, text=True, timeout=15)
self.assertEqual(result.returncode, 0, result.stderr)
def test_last_raw_picture_survives_a_closed_gate(self):
# A source sends ONE changed frame, then becomes idle. It must reach
# the encoder after congestion clears, without needing another update.
code = """
import ctypes as C, sys, time
from frame_pc_capture import Native, GATE, Encoded, pipeline
native = Native()
start = None
closing = False
phases = []
def gate(stage, pts, capture, arrived):
global start
phases.append(stage)
if closing: return -1
if start is None: start = time.monotonic()
return int(stage == 1 or time.monotonic()-start >= .15)
callback = GATE(gate)
p = pipeline(dict(src='test'), sys.platform, 'x264enc', 320, 180, 30, 300000)
p = p.replace('is-live=true', 'is-live=true num-buffers=1')
error = C.create_string_buffer(1024)
handle = native.lib.fc_capture_open(p.encode(), callback, error, len(error))
assert handle, error.value
try:
frame = Encoded()
result = 0
deadline = time.monotonic()+5
while not result and time.monotonic() < deadline:
result = native.lib.fc_capture_pull(handle, C.byref(frame))
assert result == 1, native.lib.fc_capture_error(handle)
assert frame.key and frame.size > 17
assert phases.count(0) == 1 and 2 in phases, phases
finally:
closing = True
native.lib.fc_capture_close(handle)
"""
env = frame_pcview.PCView([], lambda *a: None, 'frame').agent_environment()
env['PYTHONPATH'] = str(ROOT / 'ui')
result = subprocess.run([sys.executable, '-c', code], env=env, capture_output=True, text=True, timeout=10)
self.assertEqual(result.returncode, 0, result.stderr)
def test_auth_and_real_h264_stats(self):
self.assertEqual(self.request('/status')[0], 403)
status, body = self.request('/ticket?src=test&k='+self.token, 'POST')
self.assertEqual(status, 200)
ticket = json.loads(body)['ticket']
ws = WS(self.port, '/stream?src=test&fps=30&max=640&t='+ticket)
try:
self.assertEqual(ws.status, 101)
key, got, echo = None, 0, False
ws.send_text(json.dumps(dict(t='m', e='down', i=7, tv=1, x=.2, y=.3)))
deadline = time.monotonic()+10
while time.monotonic() < deadline and (got < 5 or not echo):
op, data = ws.recv()
if op == 1:
m = json.loads(data)
self.assertNotEqual(m.get('t'), 'error', m)
if m.get('t') == 'hello':
key = m['r']
ws.send_text(json.dumps(dict(t='ack')))
elif op == 2:
got += 1
seq, echoed = struct.unpack('!II', data[9:17])
now = struct.unpack('!Q', data[1:9])[0]
echo |= echoed == 7
self.assertIn(b'\x00\x00\x00\x01', data[17:])
ws.send_text(json.dumps(dict(t='rx', s=seq, r=now+100)))
ws.send_text(json.dumps(dict(t='fd', f=[[seq, now+200, now+300, now+400]])))
self.assertGreaterEqual(got, 5)
self.assertTrue(echo)
time.sleep(.1)
stats = json.loads(self.request('/stats?settle=0&k='+self.token)[1])['streams'][0]
self.assertGreater(stats['frames'][0]['e1'], stats['frames'][0]['e0'])
self.assertIn('controller', stats)
used = WS(self.port, '/stream?src=test&t='+ticket)
self.assertEqual(used.status, 403)
used.close()
self.request('/close?src=test&k='+self.token, 'POST')
revoked = WS(self.port, '/stream?src=test&r='+key)
self.assertEqual(revoked.status, 403)
revoked.close()
finally:
ws.close()
if __name__ == '__main__':
unittest.main()
+3
View File
@@ -84,6 +84,7 @@ class ServerGuards(unittest.TestCase):
def test_api_needs_custom_header(self): def test_api_needs_custom_header(self):
# <img src> and plain form posts from other sites can't set it. # <img src> and plain form posts from other sites can't set it.
self.assertEqual(self.request("POST", "/api/comfort", {"action": "start"})[0], 403)
self.assertEqual(self.request("GET", "/api/status")[0], 403) self.assertEqual(self.request("GET", "/api/status")[0], 403)
self.assertEqual(self.request("GET", "/api/screenshot?view=headset")[0], 403) self.assertEqual(self.request("GET", "/api/screenshot?view=headset")[0], 403)
self.assertEqual(self.request("GET", "/api/shots")[0], 403) self.assertEqual(self.request("GET", "/api/shots")[0], 403)
@@ -99,6 +100,8 @@ class ServerGuards(unittest.TestCase):
def test_input_validation(self): def test_input_validation(self):
cases = [ cases = [
("/api/comfort", {"action": "poweroff"}),
("/api/comfort", {"action": "start", "minutes": 0}),
("/api/launch", {"appid": "620; rm -rf ~"}), ("/api/launch", {"appid": "620; rm -rf ~"}),
("/api/launch", {"appid": ""}), ("/api/launch", {"appid": ""}),
("/api/flatpak", {"id": "org.example.App;id", "action": "install"}), ("/api/flatpak", {"id": "org.example.App;id", "action": "install"}),
+283
View File
@@ -0,0 +1,283 @@
"""Touch: the live view's Control (ui/frame_touch.py on the Frame, and the server's checks).
Run: python3 -m unittest discover -s tests
"""
import shutil
import subprocess
import sys
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "ui"))
class Mapping(unittest.TestCase):
@classmethod
def setUpClass(cls):
import frame_touch
cls.t = frame_touch
def test_panel_bigger_than_its_display_is_scaled(self):
# Verified 2026-09-29: a 1920x1080 window on :1 (1280x720) took Accept at
# window (1828, 1020) as pointer (1219, 680).
panel = {"root": [1280, 720], "width": 1920, "height": 1080}
x, y = self.t.to_root(panel, 1828 / 1920, 1020 / 1080)
self.assertAlmostEqual(x, 1218.7, delta=1)
self.assertAlmostEqual(y, 679.4, delta=1)
def test_same_size_is_one_to_one_and_clamped(self):
panel = {"root": [1280, 720], "width": 1280, "height": 720}
self.assertEqual(self.t.to_root(panel, 0, 0), (0, 0))
self.assertEqual(self.t.to_root(panel, 1, 1), (1279, 719))
self.assertEqual(self.t.to_root(panel, -3, 7), (0, 719))
def test_other_shapes_are_letterboxed(self):
panel = {"root": [1280, 720], "width": 800, "height": 800} # square: bars left and right
x, y = self.t.to_root(panel, 0, 0.5)
self.assertAlmostEqual(x, 280, delta=0.5)
self.assertAlmostEqual(y, 359.5, delta=0.5)
def test_same_id_on_both_displays_is_told_apart_by_pid(self):
t = self.t
saved = t.displays, t.window_info, t.window_pid
self.addCleanup(lambda: (setattr(t, "displays", saved[0]), setattr(t, "window_info", saved[1]),
setattr(t, "window_pid", saved[2])))
t.displays = lambda: [":0", ":1"]
t.window_info = lambda d, w: {"name": f"on {d}", "width": 1280 if w != "root" else 1920, "height": 720}
t.window_pid = lambda d, w: {":0": 111, ":1": 222}[d]
self.assertEqual(t.locate(5, 222)["display"], ":1")
self.assertEqual(t.locate(5, 111)["display"], ":0")
self.assertEqual(t.locate(5, None)["display"], ":0")
def test_focus_display_is_decoded(self):
t = self.t
saved = t.xprop_root
self.addCleanup(lambda: setattr(t, "xprop_root", saved))
for values, want in (([12602, 0, 58], ":1"), ([12346], ":0"), ([], None), ([0x41], None)):
t.xprop_root = lambda d, n, v=values: v
self.assertEqual(t.focus_display(), want)
def test_ascii_table_covers_printable_characters(self):
for code in range(0x20, 0x7F):
self.assertIn(chr(code), self.t.ASCII, chr(code))
self.assertEqual(self.t.ASCII["a"], (30, False))
self.assertEqual(self.t.ASCII["A"], (30, True))
self.assertEqual(self.t.ASCII["?"], (53, True))
self.assertEqual(self.t.ASCII["1"], (2, False))
self.assertEqual(self.t.ASCII["0"], (11, False))
def test_events_parsing(self):
self.assertEqual(self.t.events(b'{"dx": 1}'), [{"dx": 1}])
self.assertEqual(self.t.events(b'[{"dx": 1}, 5]'), [{"dx": 1}])
self.assertEqual(self.t.events(b"nope"), [])
class FakeGamescope:
def __init__(self):
self.calls = []
def __getattr__(self, name):
return lambda *a: self.calls.append((name, *a))
class Apply(unittest.TestCase):
@classmethod
def setUpClass(cls):
import frame_touch
cls.t = frame_touch
def setUp(self):
self.focused = {"window": 7, "display": ":1", "root": [1280, 720], "width": 1280, "height": 720, "name": "x"}
saved = self.t.focus, self.t.say, self.t.focus_now
self.t.STALE[0] = False
self.said = []
self.t.focus = lambda: dict(self.focused)
self.t.focus_now = lambda: (self.focused["window"], self.focused["display"])
self.t.say = lambda state, **more: self.said.append((state, more))
self.addCleanup(lambda: (setattr(self.t, "focus", saved[0]), setattr(self.t, "say", saved[1]),
setattr(self.t, "focus_now", saved[2])))
def test_tap_moves_then_clicks_in_order(self):
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"}, {"button": "left", "down": True, "window": 7, "display": ":1"},
{"button": "left", "down": False}):
panel = self.t.apply(gs, e, panel)
self.assertEqual([c[0] for c in gs.calls], ["move_to", "button", "button"])
self.assertEqual(gs.calls[1][1:], ("left", True))
def test_a_tap_meant_for_another_panel_goes_nowhere(self):
# Focus moved on: the position and the press are dropped; the release still goes.
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, {"button": "left", "down": True, "window": 99, "display": ":1"},
{"key": 30, "down": True, "window": 7, "display": ":0"}, {"button": "left", "down": False, "window": 99, "display": ":1"}):
panel = self.t.apply(gs, e, panel)
self.assertEqual(gs.calls, [("button", "left", False)])
self.assertEqual(self.said[0], ("ready", {"focus": 7, "display": ":1", "stale": True}))
def test_presses_read_focus_afresh(self):
# A move may use a recent reading; a press checks again, so a panel that just
# took focus doesn't get a click meant for another.
gs, calls = FakeGamescope(), []
self.t.focus = lambda: calls.append(1) or dict(self.focused)
panel = self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"}, None)
panel = self.t.apply(gs, {"fx": 0.6, "fy": 0.5, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 1)
self.t.apply(gs, {"button": "left", "down": True, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 1) # same panel still: the quick check was enough
self.focused["window"] = 8
self.t.apply(gs, {"button": "left", "down": True, "window": 7, "display": ":1"}, panel)
self.assertEqual(len(calls), 2)
self.assertEqual([c[0] for c in gs.calls], ["move_to", "move_to", "button"])
def test_stale_is_said_once_and_cleared(self):
gs, panel = FakeGamescope(), None
for e in ({"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":1"},
{"fx": 0.6, "fy": 0.5, "window": 7, "display": ":1"}):
panel = self.t.apply(gs, e, panel)
self.assertEqual(self.said, [("ready", {"focus": 7, "display": ":1", "stale": True}),
("ready", {"focus": 7, "display": ":1"})])
def test_stale_clears_on_a_trackpad_move_but_not_on_a_stale_release(self):
gs = FakeGamescope()
panel = self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 99, "display": ":1"}, None)
panel = self.t.apply(gs, {"button": "left", "down": False, "window": 99, "display": ":1"}, panel)
self.assertTrue(self.t.STALE[0]) # that release was still aimed at the old panel
self.t.apply(gs, {"dx": 3, "dy": 0}, panel)
self.assertFalse(self.t.STALE[0])
self.assertEqual(self.said[-1][1].get("stale"), None)
def test_same_window_id_on_the_other_display_is_another_panel(self):
gs = FakeGamescope()
self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":0"}, None)
self.assertEqual(gs.calls, [])
def test_relative_scroll_keys_text(self):
gs = FakeGamescope()
for e in ({"dx": 5, "dy": -3}, {"scroll": [0, 120]}, {"key": 30, "down": True}, {"text": "hi"},
{"key": True}, {"button": "sideways"}):
self.t.apply(gs, e, None)
self.assertEqual([c[0] for c in gs.calls], ["move_by", "scroll", "key", "text"])
def test_everything_held_is_let_go(self):
class Held(self.t.Gamescope):
def __init__(self):
self.held, self.keys, self.log = set(), set(), []
def frame(self):
pass
g = Held()
g.L = type("L", (), {"ei_device_button_button": lambda *a: g.log.append(("button",) + a[2:]),
"ei_device_keyboard_key": lambda *a: g.log.append(("key",) + a[2:])})()
g.device = object()
g.button("left", True)
g.key(42, True)
g.release_all()
self.assertEqual(g.log[-2:], [("button", 0x110, False), ("key", 42, False)])
self.assertEqual((g.held, g.keys), (set(), set()))
def test_resuming_after_a_pause_lets_go_of_everything(self):
t = self.t
log, queue = [], [1]
class L:
def __getattr__(self, name):
if name == "ei_get_event":
return lambda ei: queue.pop(0) if queue else None
return {"ei_event_get_type": lambda ev: t.EV_DEVICE_RESUMED, "ei_event_get_device": lambda ev: "dev",
"ei_device_has_capability": lambda d, c: True,
"ei_device_button_button": lambda d, c, down: log.append(("button", c, down)),
"ei_device_keyboard_key": lambda d, c, down: log.append(("key", c, down))}.get(name, lambda *a: 0)
g = t.Gamescope.__new__(t.Gamescope)
g.L, g.ei, g.fd, g.device, g.sequence, g.alive = L(), None, None, None, 0, True
g.held, g.keys = {0x110}, {42}
g.frame = lambda: None
old = t.select.select
t.select.select = lambda *a: ([], [], [])
try:
g.pump()
finally:
t.select.select = old
self.assertEqual(sorted(log), [("button", 0x110, False), ("key", 42, False)])
self.assertEqual(g.device, "dev")
def test_text_uses_shift_for_capitals(self):
class Keys(self.t.Gamescope):
def __init__(self):
self.pressed = []
def key(self, code, down):
self.pressed.append((code, down))
k = Keys()
k.text("Hié") # the é has no key on a US layout and is left out
self.assertEqual(k.pressed, [(42, True), (35, True), (35, False), (42, False), (23, True), (23, False)])
class ServerChecks(unittest.TestCase):
@classmethod
def setUpClass(cls):
import server
cls.s = server
def test_touch_event_keeps_known_fields(self):
ev = self.s.touch_event
self.assertEqual(ev({"fx": 0.5, "fy": 2, "window": 5, "display": ":1"}), {"fx": 0.5, "fy": 1.0, "window": 5, "display": ":1"})
self.assertEqual(ev({"button": "left", "window": 5, "display": ":0"}), {"button": "left", "down": True, "window": 5, "display": ":0"})
self.assertEqual(ev({"button": "right"}), {"button": "right", "down": True})
self.assertEqual(ev({"key": 30, "down": False}), {"key": 30, "down": False})
self.assertEqual(ev({"scroll": [0, 1e9]}), {"scroll": [0.0, 5000.0]})
self.assertEqual(ev({"dx": 3, "other": 1}), {"dx": 3.0})
def test_touch_event_rejects_bad_ones(self):
for bad in (None, {}, {"fx": 0.5, "fy": 0.5}, {"fx": "1", "fy": 0, "window": 1, "display": ":1"}, {"button": "side"},
{"fx": 0.5, "fy": 0.5, "window": 1}, {"fx": 0.5, "fy": 0.5, "window": 1, "display": ":1;x"},
{"window": 1, "display": ":1"},
{"key": 0}, {"key": 999}, {"key": True}, {"scroll": [1]}, {"text": ""}, {"text": "x" * 501},
{"fx": 0.1, "fy": 0.1, "window": True, "display": ":1"}):
with self.assertRaises(self.s.Failure, msg=repr(bad)):
self.s.touch_event(bad)
def test_panel_stream_is_the_window_and_checked(self):
cmd = self.s.stream_command("src=panel&window=10485777&display=:1&h=720&fps=30")
self.assertIn("DISPLAY=:1 ffmpeg", cmd)
self.assertIn("-window_id 10485777 -i :1", cmd)
for bad in ("src=panel&window=1;rm&display=:1", "src=panel&window=1&display=:1;x", "src=panel&display=:1"):
with self.assertRaises(self.s.Failure):
self.s.stream_command(bad + "&h=720&fps=30")
def test_touch_agent_runs_the_helper_with_nothing_to_copy(self):
agent = self.s.TouchAgent()
self.assertEqual(agent.deliver(lambda m: None), "")
cmd = agent.command()
self.assertTrue(cmd.startswith("python3 -u -c '"))
self.assertIn("frame_touch", cmd)
def test_batch_limit(self):
with self.assertRaises(self.s.Failure):
self.s.remote_touch({"events": [{"dx": 1}] * (self.s.INPUT_BATCH_LIMIT + 1)})
@unittest.skipUnless(shutil.which("node"), "needs node")
class PageGestures(unittest.TestCase):
"""Control's gestures and queue (tests/page/ctrl_gestures.mjs runs the real functions from index.html)."""
def test_gestures(self):
r = subprocess.run(["node", str(ROOT / "tests" / "page" / "ctrl_gestures.mjs")], capture_output=True, text=True,
timeout=60)
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
class Script(unittest.TestCase):
def test_helper_compiles_on_the_frames_python(self):
# The Frame runs it with its own python3 (3.13 on SteamOS 0.4.1); stdlib and ctypes only.
src = (ROOT / "ui/frame_touch.py").read_text()
compile(src, "frame_touch.py", "exec")
for mod in ("import ctypes", "import json", "import select"):
self.assertIn(mod, src)
self.assertNotIn("import requests", src)
if __name__ == "__main__":
unittest.main()
+264
View File
@@ -0,0 +1,264 @@
"""Opt-in session worker ON the Frame; no root, extra apps, or power actions.
One worker per user, shared by desktop and phone. State survives companion
connections, not headset reboots. See docs/family-comfort.md for guarantees.
"""
import contextlib
import json
import os
from pathlib import Path
import subprocess
import sys
import time
import uuid
from frame_steam import Page
from frame_status import battery, thermal_alerts, activity_level
ROOT = Path.home() / '.local/state/frame-control/comfort'
VRCMD = '/opt/steamvr/bin/linuxarm64/vrcmd'
HOME_JS = """(async () => {
SteamUIStore.Navigate('/library/home');
await SteamClient.OpenVR.VROverlay.ShowDashboard('valve.steam.gamepadui.main');
if (!await SteamClient.OpenVR.VROverlay.IsDashboardVisible()) throw Error('Steam dashboard did not open');
return {path: location.pathname};
})()"""
def clock():
# CLOCK_BOOTTIME includes headset suspend; wall-clock corrections don't alter limits.
return time.clock_gettime(time.CLOCK_BOOTTIME)
def boot():
return Path('/proc/sys/kernel/random/boot_id').read_text().strip()
def validate(body):
if not isinstance(body, dict) or body.get('action') not in ('status', 'start', 'cancel'):
raise ValueError('Choose status, start or cancel')
if body['action'] == 'start':
for key, low, high in (('minutes', 1, 240), ('breakMinutes', 0, 120), ('stillMinutes', 0, 240)):
n = body.get(key)
if type(n) is not int or not low <= n <= high:
raise ValueError(f'{key} must be a whole number from {low} to {high}')
for key in ('batteryAlert', 'heatAlert'):
if type(body.get(key)) is not bool:
raise ValueError(f'{key} must be true or false')
return body
def new_session(body, now, boot_id):
return {'id': uuid.uuid4().hex, 'boot': boot_id, 'active': True,
'options': {k: body[k] for k in ('minutes', 'breakMinutes', 'stillMinutes', 'batteryAlert', 'heatAlert')},
'started': now, 'deadline': now + body['minutes'] * 60, 'lastSample': now,
'used': 0, 'nextBreak': body['breakMinutes'] * 60, 'stillSent': False,
'warned': None, 'events': [], 'seq': 0, 'latched': [], 'error': None}
def event(s, kind, message):
s['seq'] += 1
s['events'].append({'id': s['id'] + ':' + str(s['seq']), 'kind': kind,
'message': message, 'time': time.time()})
s['events'] = s['events'][-40:]
def notify(message):
r = subprocess.run([VRCMD, '--notify', 'Frame Control: ' + message],
capture_output=True, text=True, timeout=20)
if r.returncode or 'succeeded' not in r.stdout:
raise RuntimeError('SteamVR could not show the reminder: ' + (r.stderr or r.stdout)[-300:])
def home():
# A total process deadline also bounds a CDP peer that keeps sending events
# without completing the request. Keep cancellation ordered after this action.
r = subprocess.run([sys.executable, str(Path(__file__).resolve()), '--home'],
capture_output=True, text=True, timeout=15)
if r.returncode:
raise RuntimeError('Steam Home failed: ' + (r.stdout or r.stderr)[-300:])
def open_home():
page = Page()
try:
result = page.eval(HOME_JS)
if result.get('path') != '/routes/library/home':
raise RuntimeError('Steam did not navigate Home')
finally:
page.sock.close()
def tick(s, now, sample, warn=notify, go_home=home, read_clock=clock):
"""One deterministic step; injected actions/samples also exercise a fake Frame."""
if not s.get('active'):
return
o = s['options']
s['heartbeat'] = now
delta = max(0, min(30, now - s['lastSample']))
s['lastSample'] = now
level = sample.get('activity')
b = sample.get('battery') or {}
s['unavailable'] = []
if o['batteryAlert'] and b.get('percent') is None:
s['unavailable'].append('battery')
if o['heatAlert'] and sample.get('thermal') is None:
s['unavailable'].append('temperature')
if (o['breakMinutes'] or o['stillMinutes']) and level is None:
s['unavailable'].append('activity')
s['activity'] = level
if level in (1, 2):
s['used'] += delta
elif level is not None:
s['used'] = 0
s['nextBreak'] = o['breakMinutes'] * 60
s['stillSent'] = False
# Missing samples never count as time worn. No catch-up burst after a disconnect.
if now >= s['deadline'] - 60 and s['warned'] is None:
warn('One minute left. Save your progress; Steam Home will open.')
s['warned'] = max(now, read_clock())
event(s, 'warning', 'One minute left. Save your progress; Steam Home will open.')
if s['warned'] is not None and now >= max(s['deadline'], s['warned'] + 60):
go_home()
s['active'] = False
event(s, 'finished', 'Session ended: Steam Home opened. Your game is still running.')
return
if o['breakMinutes'] and s['used'] >= s['nextBreak']:
warn('Time for a break. Take off the headset and rest your eyes.')
event(s, 'break', 'Time for a break. Take off the headset and rest your eyes.')
s['nextBreak'] = s['used'] + o['breakMinutes'] * 60
if o['stillMinutes'] and not s['stillSent'] and s['used'] >= o['stillMinutes'] * 60:
event(s, 'still', f"Headset still active after {o['stillMinutes']} active minute(s). Check in with the wearer.")
s['stillSent'] = True
low = b.get('percent') is not None and b['percent'] <= 15 and b.get('status') == 'Discharging'
hot = sample.get('thermal')
for kind, enabled, value, message in (
('battery', o['batteryAlert'], low if b else None, 'Frame battery is low (15% or less).'),
('heat', o['heatAlert'], bool(hot) if hot is not None else None,
'Frame reports a hot/critical thermal trip or battery overheat. Ask the wearer to take a break.')):
if enabled and value and kind not in s['latched']:
event(s, kind, message)
s['latched'].append(kind)
elif value is False and kind in s['latched']:
# Battery hysteresis prevents repeated alerts around 15%.
if kind != 'battery' or b.get('status') == 'Charging' or (b.get('percent') or 0) >= 20:
s['latched'].remove(kind)
@contextlib.contextmanager
def locked(name='state.lock', nonblocking=False):
import fcntl # only needed ON the Linux headset, not by desktop validation/tests
ROOT.mkdir(parents=True, exist_ok=True, mode=0o700)
with (ROOT / name).open('a') as f:
fcntl.flock(f, fcntl.LOCK_EX | (fcntl.LOCK_NB if nonblocking else 0))
yield f
def read_state():
try:
state = json.loads((ROOT / 'session.json').read_text())
if not isinstance(state, dict):
raise ValueError('Saved session must be an object')
return state
except FileNotFoundError:
return {'active': False, 'events': []}
except (ValueError, UnicodeDecodeError):
# Preserve the unreadable state for diagnosis, then allow a new session.
(ROOT / 'session.json').replace(ROOT / ('session-unreadable-' + uuid.uuid4().hex + '.json'))
return {'active': False, 'events': [],
'error': 'Saved session was unreadable. Start a new session.'}
def save(s):
p = ROOT / 'session.tmp'
p.write_text(json.dumps(s))
p.chmod(0o600)
p.replace(ROOT / 'session.json')
def current(s, now):
if s.get('active') and s.get('boot') != boot():
s['active'] = False
s['error'] = 'Headset restarted. Start a new session.'
out = dict(s)
out['time'] = time.time() # event age uses the Frame's clock, not the phone's
out['remaining'] = max(0, max(s.get('deadline', now), (s.get('warned') or 0) + 60) - now) if s.get('active') else 0
beat = s.get('heartbeat', s.get('started', now)) # a hand-edited state may lack either
if s.get('active') and now - beat > 90:
out['error'] = 'Session worker is not responding. Timer enforcement is unverified; cancel and start again.'
return out
def watch():
try:
with locked('worker.lock', nonblocking=True) as worker:
while True:
with locked():
s = current(read_state(), clock())
if not s.get('active'):
save(s)
# Release ownership before state.lock: a concurrent start
# cannot miss the gap between an old worker and its exit.
import fcntl
fcntl.flock(worker, fcntl.LOCK_UN)
return
try:
b = battery()
hot = thermal_alerts()
if b and b.get('health') == 'Overheat':
hot = (hot or []) + ['battery']
tick(s, clock(), {'battery': b, 'thermal': hot, 'activity': activity_level()})
s['error'] = None
except Exception as e:
error = str(e)
if s.get('error') != error:
event(s, 'error', 'Session action failed: ' + error)
s['error'] = error
save(s)
time.sleep(5)
except BlockingIOError:
pass # another connection already started the single worker
def command(body):
validate(body)
with locked():
s = current(read_state(), clock())
if body['action'] == 'start':
if s.get('active'):
raise ValueError('A session is already running. Cancel it before starting another.')
s = new_session(body, clock(), boot())
event(s, 'started', 'Session started. Steam Home opens at the limit; games are not closed.')
elif body['action'] == 'cancel':
s['active'] = False
s['error'] = None
if s.get('id'):
event(s, 'cancelled', 'Session timer and monitoring cancelled.')
save(s)
if body['action'] == 'start':
try:
subprocess.Popen([sys.executable, str(Path(__file__).resolve()), '--watch'],
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
start_new_session=True, close_fds=True)
except OSError as e:
s['active'] = False
s['error'] = 'Could not start session worker: ' + str(e)
event(s, 'error', s['error'])
save(s)
raise
return current(s, clock())
if __name__ == '__main__':
if sys.argv[1:] == ['--watch']:
watch()
else:
try:
if sys.argv[1:] == ['--home']:
open_home()
print(json.dumps({'home': True}))
else:
print(json.dumps(command(json.loads(sys.argv[1]))))
except Exception as e:
print(json.dumps({'error': str(e)}))
sys.exit(1)
+17 -29
View File
@@ -56,12 +56,12 @@ BROWSER_FLAGS = []
PANEL_BOX = (1920, 1080) PANEL_BOX = (1920, 1080)
# Opens one viewer on gamescope's X display and gives its window its own panel # Opens one viewer on gamescope's X display and gives its window its own panel
# id. Args: appid url width height tag profile [browser flags...]. The page puts "[tag]" in its title at # id. Args: appid url width height tag [browser flags...]. The page puts "[tag]" in its title at
# once, which is how its X window is found (Chromium may hand the URL to an # once, which is how its X window is found (Chromium may hand the URL to an
# instance that's already running, so there's no process to follow). # instance that's already running, so there's no process to follow).
LAUNCH = r"""set -u LAUNCH = r"""set -u
appid=$1 url=$2 w=$3 h=$4 tag=$5 profile=$6 appid=$1 url=$2 w=$3 h=$4 tag=$5
shift 6 shift 5
export DISPLAY=:0 LC_ALL=C.UTF-8 export DISPLAY=:0 LC_ALL=C.UTF-8
unset WAYLAND_DISPLAY unset WAYLAND_DISPLAY
if ! xprop -root GAMESCOPE_FOCUSABLE_WINDOWS >/dev/null 2>&1; then if ! xprop -root GAMESCOPE_FOCUSABLE_WINDOWS >/dev/null 2>&1; then
@@ -73,15 +73,15 @@ common=(--ozone-platform=x11 --force-device-scale-factor=1 --no-first-run --no-d
--disable-features=Translate,MediaRouter --autoplay-policy=no-user-gesture-required --disable-features=Translate,MediaRouter --autoplay-policy=no-user-gesture-required
"--window-size=$w,$h" "$@" "--app=$url") "--window-size=$w,$h" "$@" "--app=$url")
if [ -x "$HOME/chromium-xr/chrome" ]; then if [ -x "$HOME/chromium-xr/chrome" ]; then
cmd=("$HOME/chromium-xr/chrome" "--user-data-dir=$HOME/.local/share/frame-control/$profile" "${common[@]}") cmd=("$HOME/chromium-xr/chrome" "--user-data-dir=$HOME/.local/share/frame-control/mac-view" "${common[@]}")
elif flatpak info org.chromium.Chromium >/dev/null 2>&1; then elif flatpak info org.chromium.Chromium >/dev/null 2>&1; then
cmd=(flatpak run org.chromium.Chromium cmd=(flatpak run org.chromium.Chromium
"--user-data-dir=$HOME/.var/app/org.chromium.Chromium/data/frame-$profile" "${common[@]}") "--user-data-dir=$HOME/.var/app/org.chromium.Chromium/data/frame-mac-view" "${common[@]}")
else else
echo "NO_BROWSER" echo "NO_BROWSER"
exit 3 exit 3
fi fi
log=/tmp/frame-$profile.log log=/tmp/frame-mac-view.log
setsid nohup "${cmd[@]}" >>"$log" 2>&1 </dev/null & setsid nohup "${cmd[@]}" >>"$log" 2>&1 </dev/null &
for _ in $(seq 1 60); do for _ in $(seq 1 60); do
sleep 0.5 sleep 0.5
@@ -105,9 +105,9 @@ class MacViewError(Exception):
pass pass
def panel_id(src, host="mac"): def panel_id(src):
"""A stable panel id per source, in the range panel-on-frame.sh uses.""" """A stable panel id per source, in the range panel-on-frame.sh uses."""
return 2_001_000_000 + zlib.crc32(f"{host}:{src}".encode()) % 1_000_000 return 2_001_000_000 + zlib.crc32(f"mac:{src}".encode()) % 1_000_000
def fit(w, h, box=PANEL_BOX): def fit(w, h, box=PANEL_BOX):
@@ -116,9 +116,6 @@ def fit(w, h, box=PANEL_BOX):
class MacView: class MacView:
host = "mac"
viewer_profile = "mac-view"
def __init__(self, tunnel_ssh, run, frame, track=None): def __init__(self, tunnel_ssh, run, frame, track=None):
self.tunnel_ssh = list(tunnel_ssh) self.tunnel_ssh = list(tunnel_ssh)
self.run = run self.run = run
@@ -169,17 +166,10 @@ class MacView:
def _stale(self): def _stale(self):
try: try:
built = AGENT.stat().st_mtime built = AGENT.stat().st_mtime
return any(p.stat().st_mtime > built for p in [*(SOURCES / "Sources").glob("*.swift"), return any(p.stat().st_mtime > built for p in (SOURCES / "Sources").glob("*.swift"))
ROOT / "desktop" / "controller.c", ROOT / "desktop" / "controller.h"])
except OSError: except OSError:
return False return False
def agent_command(self):
return [str(AGENT)]
def agent_environment(self):
return {**os.environ, "FRAME_MAC_VIEW_TOKEN": self.token}
def ensure_agent(self): def ensure_agent(self):
with self.lock: with self.lock:
if self.agent and self.agent.poll() is None: if self.agent and self.agent.poll() is None:
@@ -188,11 +178,11 @@ class MacView:
if reason: if reason:
raise MacViewError(reason) raise MacViewError(reason)
self.build() self.build()
env = self.agent_environment() env = {**os.environ, "FRAME_MAC_VIEW_TOKEN": self.token}
# The same port as before when restarting, so a running tunnel still # The same port as before when restarting, so a running tunnel still
# fits; otherwise (or if it's gone) whatever the system gives. # fits; otherwise (or if it's gone) whatever the system gives.
for port in dict.fromkeys([self.port or 0, 0]): for port in dict.fromkeys([self.port or 0, 0]):
self.agent = subprocess.Popen([*self.agent_command(), "serve", "--port", str(port), "--page", str(PAGE), self.agent = subprocess.Popen([str(AGENT), "serve", "--port", str(port), "--page", str(PAGE),
"--exit-on-eof"], env=env, stdin=subprocess.PIPE, "--exit-on-eof"], env=env, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
line = self.agent.stdout.readline() line = self.agent.stdout.readline()
@@ -201,7 +191,7 @@ class MacView:
break break
self.agent.kill() self.agent.kill()
else: else:
raise MacViewError(f"The streaming helper didn't start: {line.strip() or 'no output'}") raise MacViewError(f"The Mac streaming helper didn't start: {line.strip() or 'no output'}")
if self.port != int(m.group(1)): if self.port != int(m.group(1)):
self._drop_tunnel() self._drop_tunnel()
self.port = int(m.group(1)) self.port = int(m.group(1))
@@ -217,7 +207,7 @@ class MacView:
with urllib.request.urlopen(req, timeout=10) as r: with urllib.request.urlopen(req, timeout=10) as r:
return json.load(r) return json.load(r)
except (urllib.error.URLError, OSError, ValueError) as e: except (urllib.error.URLError, OSError, ValueError) as e:
raise MacViewError(f"The streaming helper didn't answer: {e}") raise MacViewError(f"The Mac streaming helper didn't answer: {e}")
# ---- the tunnel from the Frame ---- # ---- the tunnel from the Frame ----
@@ -372,7 +362,7 @@ class MacView:
# the agent no longer counts them, so only move when none are out there. # the agent no longer counts them, so only move when none are out there.
others = self.shown - {src} others = self.shown - {src}
self.ensure_tunnel(allow_new_port=not others) self.ensure_tunnel(allow_new_port=not others)
appid = panel_id(src, self.host) appid = panel_id(src)
# Unique per launch, so a new window is never confused with an old one. # Unique per launch, so a new window is never confused with an old one.
tag = "fc" + secrets.token_hex(4) tag = "fc" + secrets.token_hex(4)
# A single-use ticket for this source, not Frame Control's key: the URL # A single-use ticket for this source, not Frame Control's key: the URL
@@ -382,7 +372,7 @@ class MacView:
"bpp": q["bpp"]} "bpp": q["bpp"]}
url = f"http://127.0.0.1:{self.remote_port}/view?{urlencode(params)}" url = f"http://127.0.0.1:{self.remote_port}/view?{urlencode(params)}"
w, h = fit(width or 1280, height or 720) w, h = fit(width or 1280, height or 720)
args = " ".join(_quote(str(a)) for a in (appid, url, w, h, tag, self.viewer_profile, *self.browser_flags)) args = " ".join(_quote(str(a)) for a in (appid, url, w, h, tag, *self.browser_flags))
try: try:
out = self.run("bash -s -- " + args, stdin=LAUNCH, timeout=60) out = self.run("bash -s -- " + args, stdin=LAUNCH, timeout=60)
except Exception as e: # noqa: BLE001 - the server's Failure carries the Frame's words except Exception as e: # noqa: BLE001 - the server's Failure carries the Frame's words
@@ -414,8 +404,7 @@ class MacView:
if self.shown or self.shows != shows or self.launching: if self.shown or self.shows != shows or self.launching:
return return
try: try:
pattern = "[f]rame-control/" + self.viewer_profile + "|[d]ata/frame-" + self.viewer_profile self.run("pkill -f '[f]rame-control/mac-view|[d]ata/frame-mac-view' || true", timeout=10)
self.run("pkill -f " + _quote(pattern) + " || true", timeout=10)
except Exception: except Exception:
pass pass
@@ -426,8 +415,7 @@ class MacView:
status = self.call("/status") status = self.call("/status")
windows = self.call("/windows").get("windows", []) if status.get("screen") else [] windows = self.call("/windows").get("windows", []) if status.get("screen") else []
displays = self.call("/displays").get("displays", []) displays = self.call("/displays").get("displays", [])
return {"available": True, "host": self.host, "selecting": status.get("selecting", False), return {"available": True, "screen": status.get("screen", False),
"selectionError": status.get("selectionError", ""), "screen": status.get("screen", False),
"accessibility": status.get("accessibility", False), "streams": status.get("streams", []), "accessibility": status.get("accessibility", False), "streams": status.get("streams", []),
"windows": windows, "displays": displays, "tunnel": self.tunnel_up(), "windows": windows, "displays": displays, "tunnel": self.tunnel_up(),
"route": self.route} "route": self.route}
+261
View File
@@ -0,0 +1,261 @@
"""Panel switcher. Runs on the Frame, either piped over SSH or installed with
--open for its loopback-only headset page. Uses Valve's shipped vrcmd and
Chromium, not an overlay app. Spatial layout limitations: docs/panels.md.
"""
import argparse
import hmac
import json
import os
from pathlib import Path
import re
import secrets
import signal
import subprocess
import sys
import time
from http.server import BaseHTTPRequestHandler, HTTPServer
VRCMD = '/opt/steamvr/bin/linuxarm64/vrcmd'
PANEL_ID = 2000999030
PANEL_KEY = 'valve.steam.desktopgame.' + str(PANEL_ID)
KEY = re.compile(r'[A-Za-z0-9_.:-]{1,200}\Z')
class PanelError(Exception):
pass
def run(args):
try:
p = subprocess.run(args, capture_output=True, text=True, timeout=10,
env={**os.environ, 'DISPLAY': ':0', 'LC_ALL': 'C.UTF-8'})
except (OSError, subprocess.TimeoutExpired) as e:
raise PanelError('The panel service did not answer: ' + str(e))
if p.returncode:
raise PanelError((p.stderr or p.stdout).strip()[-400:] or 'Panel command failed')
return p.stdout
def parse_overlays(text):
"""Only main dashboard panels, never their thumbnails, layers or cursors.
vrcmd output verified on SteamVR 2.18.1, BUILD_ID 20260925.6191901.
"""
if '---- OVERLAYS ----' not in text:
raise PanelError('SteamVR did not return its panel list. Is the headset awake?')
panels = []
for line in text.splitlines():
m = re.fullmatch(r"'([^']+)' -- '(.*)', (.*?) VROverlayType_Dashboard_Main\s*", line)
if m and KEY.fullmatch(m[1]):
panels.append({'key': m[1], 'title': 'Panel switcher' if m[1] == PANEL_KEY else m[2] or m[1],
'visible': 'not_visible' not in m[3]})
return panels
def state():
return {'panels': parse_overlays(run([VRCMD, '--overlays']))}
def focus(key):
if not isinstance(key, str) or not KEY.fullmatch(key):
raise PanelError('Choose an open panel.')
if key not in {p['key'] for p in state()['panels']}:
raise PanelError('That panel has closed. Refresh the list.')
run([VRCMD, '--showdashboard', key])
# vrcmd acknowledges dispatch, not final focus (a game or the user can
# switch again). Do not report a focus success without observing it.
return {'requested': key, 'message': 'Asked SteamVR to show the panel.'}
PAGE = '''<!doctype html><html lang="en"><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1"><title>Panel switcher [fc-panels]</title>
<style>body{background:#101b27;color:#eee;font:24px system-ui;margin:36px;max-width:1000px}
h1{font-size:36px}button{font:inherit;padding:16px 24px;border:1px solid #546574;border-radius:10px;
background:#23384b;color:white;cursor:pointer}button:focus-visible{outline:4px solid #66c0f4}
#panels{display:grid;gap:14px;margin:24px 0}#panels button{text-align:left}p{color:#bac8d5}</style>
<h1>Panel switcher</h1><p>Choose a panel to show it. Open this panel again from Steam's dashboard.</p>
<button id="refresh">Refresh</button> <button id="close">Close switcher</button>
<p id="status" role="status"></p><div id="panels"></div>
<script>
const token=location.hash.slice(1)||sessionStorage.getItem('panelKey')||'';
if(token)sessionStorage.setItem('panelKey',token);history.replaceState(null,'',location.pathname);
async function api(path,body){const r=await fetch(path,{method:body?'POST':'GET',
headers:{'X-Panel-Key':token,'Content-Type':'application/json'},body:body?JSON.stringify(body):undefined});
const s=await r.json();if(!r.ok)throw Error(s.error||'Panel request failed');return s;}
const status=document.getElementById('status');
async function refresh(){try{const s=await api('/panels');const list=document.getElementById('panels');list.replaceChildren();
for(const p of s.panels){const b=document.createElement('button');b.textContent=p.title;
b.onclick=async()=>{b.disabled=true;try{const r=await api('/focus',{key:p.key});status.textContent=r.message;}
catch(e){status.textContent=e.message;}finally{b.disabled=false;}};list.append(b);}
status.textContent=s.panels.length?'':'No open panels.';}catch(e){status.textContent=e.message;}}
document.getElementById('refresh').onclick=refresh;
document.getElementById('close').onclick=async()=>{try{await api('/close',{});window.close();}catch(e){status.textContent=e.message;}};
refresh();
</script></html>'''
class Handler(BaseHTTPRequestHandler):
def setup(self):
super().setup()
self.connection.settimeout(5)
def log_message(self, *args):
pass # never log the page's access key
def reply(self, code, value, html=False):
data = value.encode() if html else json.dumps(value).encode()
self.send_response(code)
self.send_header('Content-Type', 'text/html; charset=utf-8' if html else 'application/json')
self.send_header('Content-Length', str(len(data)))
self.send_header('Cache-Control', 'no-store')
self.send_header('X-Content-Type-Options', 'nosniff')
self.send_header('Referrer-Policy', 'no-referrer')
self.send_header('Content-Security-Policy', "default-src 'none'; script-src 'unsafe-inline'; style-src 'unsafe-inline'; connect-src 'self'; frame-ancestors 'none'")
self.end_headers()
self.wfile.write(data)
def allowed(self):
host = '127.0.0.1:' + str(self.server.server_port)
origin = self.headers.get('Origin')
return (self.headers.get('Host') == host and
(origin is None or origin == 'http://' + host) and
hmac.compare_digest(self.headers.get('X-Panel-Key', '').encode(), self.server.key.encode()))
def do_GET(self):
if self.path == '/':
return self.reply(200, PAGE, html=True) # no data or access key in the page
if not self.allowed():
return self.reply(403, {'error': 'Open the switcher from Frame Control.'})
try:
if self.path == '/panels':
return self.reply(200, state())
self.reply(404, {'error': 'Not found'})
except PanelError as e:
self.reply(502, {'error': str(e)})
def do_POST(self):
if not self.allowed():
return self.reply(403, {'error': 'Forbidden'})
try:
size = int(self.headers.get('Content-Length', '0'))
if not 0 < size <= 1024:
raise ValueError('Invalid request size')
body = json.loads(self.rfile.read(size))
if not isinstance(body, dict):
raise ValueError('Expected an object')
if self.path == '/focus':
return self.reply(200, focus(body.get('key')))
if self.path == '/close':
self.server.closing = True
return self.reply(200, {'closed': True})
self.reply(404, {'error': 'Not found'})
except (ValueError, PanelError) as e:
self.reply(400, {'error': str(e)})
def serve():
"""Own only our Chromium profile and process group. No changes to Steam,
SteamVR, other Chromium sessions, or global power settings.
"""
import fcntl # only on the Frame; module/tests also import on Windows
folder = Path.home() / '.local/share/frame-control/panels'
folder.mkdir(parents=True, exist_ok=True, mode=0o700)
with (folder / 'lock').open('w') as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
print(json.dumps(focus(PANEL_KEY)), flush=True)
return
chrome = Path.home() / 'chromium-xr/chrome'
if chrome.is_file():
command = [str(chrome)]
profile = folder / 'chromium'
elif Path('/usr/bin/chromium').is_file():
command = ['/usr/bin/chromium']
profile = folder / 'chromium'
elif subprocess.run(['flatpak', 'info', 'org.chromium.Chromium'],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=10).returncode == 0:
command = ['flatpak', 'run', 'org.chromium.Chromium']
profile = Path.home() / '.var/app/org.chromium.Chromium/data/frame-panel-switcher'
else:
raise PanelError('The headset switcher needs Chromium. The companion switcher still works.')
server = HTTPServer(('127.0.0.1', 0), Handler)
server.key = secrets.token_urlsafe(32)
server.closing = False
server.timeout = .5
url = 'http://127.0.0.1:%d/#%s' % (server.server_port, server.key)
env = {**os.environ, 'DISPLAY': ':0'}
env.pop('WAYLAND_DISPLAY', None)
browser = subprocess.Popen([*command, '--ozone-platform=x11',
'--user-data-dir=' + str(profile),
'--no-first-run', '--no-default-browser-check',
'--password-store=basic', '--window-size=1200,800', '--app=' + url],
env=env, start_new_session=True, stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
def stop(signum, frame):
server.closing = True
signal.signal(signal.SIGTERM, stop)
win = None
try:
deadline = time.monotonic() + 30
while not win and time.monotonic() < deadline and browser.poll() is None:
server.handle_request() # Chromium must fetch the page before it has a title
for line in run(['xwininfo', '-root', '-children']).splitlines():
m = re.match(r'\s*(0x[0-9a-fA-F]+) .*\[fc-panels\]', line)
if m:
win = m[1]
break
if not win:
raise PanelError('The switcher window did not appear within 30 seconds.')
run(['xprop', '-id', win, '-f', 'STEAM_GAME', '32c', '-set', 'STEAM_GAME', str(PANEL_ID)])
print(json.dumps({'message': 'Opened the panel switcher in the headset.'}), flush=True)
# Caller reads exactly one line, then disconnects; no more stdout.
while not server.closing and browser.poll() is None:
server.handle_request()
# Closing the last app window need not exit Chromium.
if win not in run(['xwininfo', '-root', '-children']):
break
finally:
server.server_close()
if browser.poll() is None:
os.killpg(browser.pid, signal.SIGTERM)
try:
browser.wait(timeout=5)
except subprocess.TimeoutExpired:
os.killpg(browser.pid, signal.SIGKILL)
browser.wait()
def open_switcher():
# This command runs from an installed path, never from the SSH stdin copy.
proc = subprocess.Popen([sys.executable, str(Path(__file__).resolve()), '--serve'],
stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
text=True, start_new_session=True)
line = proc.stdout.readline()
proc.stdout.close()
if not line:
raise PanelError('The headset switcher could not start.')
result = json.loads(line)
if 'error' in result:
raise PanelError(result['error'])
return result
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--focus')
parser.add_argument('--open', action='store_true')
parser.add_argument('--serve', action='store_true')
args = parser.parse_args()
try:
if args.serve:
serve()
else:
print(json.dumps(open_switcher() if args.open else focus(args.focus) if args.focus else state()))
except (PanelError, OSError) as e:
print(json.dumps({'error': str(e)}), flush=True)
return 1
return 0
if __name__ == '__main__':
sys.exit(main())
-624
View File
@@ -1,624 +0,0 @@
#!/usr/bin/env python3
"""Frame Control's Windows/Linux host. Same ticket, H.264/JPEG, timing and
input protocol as frame-mac-view; serves the same ui/mac-view.html.
Only loopback is bound. Frame Control owns the master token, the Frame gets
one-source tickets and reconnect keys. Native libraries are bundled.
"""
import argparse
import base64
import ctypes as C
import hashlib
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import math
import os
import queue
from pathlib import Path
import secrets
import socket
import struct
import sys
import threading
import time
from urllib.parse import parse_qs, urlsplit
from frame_pc_capture import Native, Controller, Windows, WindowsInput, PortalInput, Encoded, GATE, dimensions, pipeline
from frame_stream_stats import Stats
# Viewer fields are untrusted JSON: float('x'), int(None), m['t'] missing.
MALFORMED = (ValueError, TypeError, KeyError, OverflowError)
class Grants:
"""Caller holds the agent lock, including replacement and Stop."""
def __init__(self, token):
self.token, self.tickets, self.keys = token, {}, {}
def master(self, key):
return isinstance(key, str) and secrets.compare_digest(key, self.token)
def ticket(self, src):
self.tickets = {k: v for k, v in self.tickets.items() if v[1] > time.monotonic()}
if len(self.tickets) >= 128:
raise ValueError('Too many pending viewers')
ticket = secrets.token_urlsafe(24)
self.tickets[ticket] = (src, time.monotonic()+60, None)
return ticket
def redeem(self, src, q):
if self.master(q.get('k')):
key = secrets.token_urlsafe(24)
self.keys[key] = src
return key
entry = self.tickets.get(q.get('t'))
if entry and entry[0] == src and entry[1] > time.monotonic():
key = entry[2] or secrets.token_urlsafe(24)
self.tickets[q['t']] = (src, entry[1], key)
self.keys[key] = src
return key
key = q.get('r')
return key if key and self.keys.get(key) == src else None
def ack(self, key):
self.tickets = {k: v for k, v in self.tickets.items() if v[2] != key}
def revoke(self, src=None):
self.tickets = {k: v for k, v in self.tickets.items() if src is not None and v[0] != src}
self.keys = {k: v for k, v in self.keys.items() if src is not None and v != src}
class WebSocket:
def __init__(self, handler):
self.sock, self.reader = handler.connection, handler.rfile
self.lock = threading.Lock()
self.closed = False
self.sock.settimeout(5)
self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
def send(self, data, opcode=1):
if not isinstance(data, bytes):
data = json.dumps(data, separators=(',', ':')).encode()
n = len(data)
head = bytes([0x80 | opcode, n]) if n < 126 else bytes([0x80 | opcode, 126]) + struct.pack('!H', n) if n <= 65535 else bytes([0x80 | opcode, 127]) + struct.pack('!Q', n)
with self.lock:
if self.closed:
raise ConnectionError('Viewer disconnected')
self.sock.sendall(head + data)
def exact(self, n):
data = self.reader.read(n)
if len(data) != n:
raise ConnectionError('Viewer disconnected')
return data
def receive(self):
a, b = self.exact(2)
opcode, size = a & 15, b & 127
if a & 0x70 or not a & 0x80 or not b & 0x80 or opcode not in (1, 8, 9, 10):
raise ValueError('Unsupported WebSocket frame')
if size == 126:
size = struct.unpack('!H', self.exact(2))[0]
elif size == 127:
size = struct.unpack('!Q', self.exact(8))[0]
if size > 65536 or (opcode >= 8 and size > 125):
raise ValueError('WebSocket message too large')
mask = self.exact(4)
data = bytes(v ^ mask[i % 4] for i, v in enumerate(self.exact(size)))
if opcode == 8:
raise ConnectionError('Viewer closed')
if opcode == 9:
self.send(data, 10)
if opcode != 1:
return {}
message = json.loads(data)
if not isinstance(message, dict):
raise ValueError('Expected an input object')
return message
def close(self):
self.closed = True
try:
self.sock.shutdown(socket.SHUT_RDWR)
except OSError:
pass
class Session:
def __init__(self, agent, ws, key, source, query):
self.agent, self.native, self.ws, self.key, self.source = agent, agent.native, ws, key, source
self.src = source['src']
self.codec = query.get('codec', 'h264')
if self.codec not in ('h264', 'jpeg'):
raise ValueError('Unsupported codec')
self.fps = min(120, max(5, int(query.get('fps', 60))))
bpp = float(query.get('bpp', .1))
if not math.isfinite(bpp):
raise ValueError('Invalid bitrate')
self.w, self.h = dimensions(source['w'], source['h'], min(3840, max(320, int(query.get('max', 1920)))))
self.bitrate = max(300000, int(self.w*self.h*self.fps*min(.5, max(.02, bpp))))
self.controller = Controller(self.native.lib, self.fps, self.bitrate)
self.stats = Stats(self.native.now)
self.stop_event, self.key_event = threading.Event(), threading.Event()
self.lock, self.input_lock = threading.RLock(), threading.RLock()
self.pending, self.last_submit = {}, 0
self.reconfiguring = False
self.test_inputs = queue.Queue(maxsize=128)
self.input = None if self.src == 'test' else WindowsInput(agent.windows, source) if agent.windows else PortalInput(self.native, source)
self.writer = None
self.released = False
self.input_enabled = self.src == 'test' or self.source.get('devices', 3) == 3
def gate(self, stage, pts, capture, arrived):
# Exceptions cannot cross a ctypes callback boundary.
try:
with self.lock:
if self.stop_event.is_set() or self.reconfiguring:
return -1
if stage == 1:
if pts in self.pending:
self.pending[pts]['e0'] = self.native.now()
return 1
if stage == 0:
self.stats.captured += 1
self.controller.call('capture', arrived)
now = self.native.now()
state = self.controller.state()
if len(self.pending) >= 3 or now-self.last_submit < 1000000/state['fps'] or not self.controller.call('gate', now, int(stage == 0)):
if stage == 0:
self.stats.skipped += 1
return 0
self.pending[pts] = dict(cap=capture, arr=arrived, e0=now, tier=state['tier'], br=state['target'])
self.last_submit = now
return 1
except Exception:
self.stop_event.set()
return -1
STALL = 10000000
def stalled(self, now, opened, captured_at_open):
"""PipeWire and WGC deliver frames only on screen damage, so an idle
desktop legitimately produces no output. Fail only on evidence of a
fault: a frame entered the encoder and never came out, or the capture
never produced its initial frame after the pipeline opened."""
with self.lock:
oldest = min((r['e0'] for r in self.pending.values()), default=None)
captured = self.stats.captured
if oldest is not None and now-oldest > self.STALL:
return 'The encoder stopped producing frames; check the video encoder'
if captured == captured_at_open and now-opened > self.STALL:
return 'No frames captured for 10 seconds; check capture permissions'
return None
def fresh_pipewire(self):
if 'portal' not in self.source:
return
error = C.create_string_buffer(1024)
fd = self.native.lib.fc_portal_refresh(self.source['portal'], error, len(error))
if fd < 0:
raise RuntimeError(error.value.decode(errors='replace'))
self.source['fd'] = fd
def produce(self):
native, capture = self.native.lib, None
try:
encoder = "x264enc" if self.src == "test" and self.native.has("x264enc") else self.agent.encoder
self.fresh_pipewire()
description = pipeline(self.source, sys.platform, encoder, self.w, self.h, self.fps, self.bitrate, self.codec)
self.callback = GATE(self.gate)
error = C.create_string_buffer(1024)
capture = native.fc_capture_open(description.encode(), self.callback, error, len(error))
if not capture:
raise RuntimeError(error.value.decode(errors='replace'))
last_update = last_stats = self.native.now()
last_config = opened = last_update
captured_at_open = self.stats.captured
applied = (self.w, self.h, self.bitrate)
wanted = applied
while not self.stop_event.is_set():
while not self.test_inputs.empty():
event = self.test_inputs.get_nowait()
try:
native.fc_capture_test(capture, int(event.get('i', 0)) & 0xffffffff)
self.stats.input(event)
except MALFORMED:
pass # a bad probe field drops the event, not the stream
output = Encoded()
result = native.fc_capture_pull(capture, C.byref(output))
now = self.native.now()
if result < 0:
raise RuntimeError(native.fc_capture_error(capture).decode(errors='replace'))
if result:
with self.lock:
# At most three raw frames are in flight; no B-frames.
# x264 offsets PTS, so match the FIFO encode order while
# retaining the actual pre-encode capture timestamp.
raw_pts = next(iter(self.pending), None)
record = self.pending.get(raw_pts)
if record is None:
raise RuntimeError('Encoder changed frame timestamps; timing cannot be matched')
data = C.string_at(output.data, output.size)
f = self.stats.add(**record, e1=now, snd=now, b=len(data), k=output.key,
w=output.width or self.w, h=output.height or self.h)
self.controller.call('sent', f['s'], len(data)+17, now)
self.ws.send(struct.pack('!BQII', output.key, max(0, f['cap']), f['s'], f['echo'])+data, 2)
with self.lock:
f['wire'] = self.native.now()
self.pending.pop(raw_pts, None)
stall = self.stalled(now, opened, captured_at_open)
if stall:
raise RuntimeError(stall)
if self.key_event.is_set():
self.key_event.clear()
if self.codec == 'h264':
native.fc_capture_key(capture)
if now-last_update >= 100000:
target = self.controller.update(now)
state = self.controller.state()
w, h = dimensions(self.w, self.h, max(320, int(max(self.w, self.h)*state['scale'])))
if target and self.codec == 'h264' and encoder == 'x264enc':
native.fc_capture_bitrate(capture, target)
self.bitrate = target
# Hardware properties are not uniformly mutable in PLAYING.
# Drain then reopen the pipeline at a keyframe when its
# budget changes materially. The portal fd/session stays
# alive, so this does not bypass or repeat user consent.
desired_bitrate = target or applied[2]
bitrate_change = self.codec == 'h264' and encoder != 'x264enc' and abs(desired_bitrate-applied[2]) > applied[2]*.2
if not self.reconfiguring and now-last_config >= 1000000 and ((w, h) != applied[:2] or bitrate_change):
wanted = (w, h, desired_bitrate)
with self.lock:
self.reconfiguring = True
last_update = now
if self.reconfiguring:
with self.lock:
drained = not self.pending
if drained:
native.fc_capture_close(capture)
capture = None
self.fresh_pipewire()
description = pipeline(self.source, sys.platform, encoder, wanted[0], wanted[1], self.fps, wanted[2], self.codec)
capture = native.fc_capture_open(description.encode(), self.callback, error, len(error))
if not capture:
raise RuntimeError(error.value.decode(errors='replace'))
applied, self.bitrate, last_config = wanted, wanted[2], now
opened, captured_at_open = now, self.stats.captured
with self.lock:
self.reconfiguring = False
if now-last_stats >= 1000000:
self.ws.send(dict(self.stats.summary(), t='stats', bitrate=self.bitrate,
size='%dx%d' % (self.w, self.h), tier=self.controller.state()['tier']))
last_stats = now
except Exception as e:
try:
self.ws.send({'t': 'error', 'message': str(e)})
except OSError:
pass
finally:
self.stop_event.set()
if capture:
native.fc_capture_close(capture)
self.ws.close()
def start(self):
if self.stop_event.is_set():
self.controller.close()
return
self.ws.send(dict(t='hello', r=self.key))
self.ws.send(dict(t='info', src=self.src, title=self.source.get('title', self.source.get('name', 'Test pattern')),
app='PC', codec=self.codec, input=self.input_enabled,
inputMessage='Allow pointer and keyboard control in the host sharing dialog.',
aspect=self.w/self.h, warm=0))
with self.lock:
if self.stop_event.is_set():
self.controller.close()
return
self.writer = threading.Thread(target=self.produce, daemon=True)
self.writer.start()
try:
while not self.stop_event.is_set():
m = self.ws.receive()
t = m.get('t')
if t == 'ping':
self.ws.send(dict(t='pong', c=m.get('c', 0), a=self.native.now()))
elif t == 'ack':
with self.agent.lock:
self.agent.grants.ack(self.key)
elif t == 'key-frame':
self.key_event.set()
elif t in ('rx', 'fd', 'clock'):
try:
self.stats.report(m)
except MALFORMED:
pass
if t == 'rx' and isinstance(m.get('s'), int):
self.controller.call('ack', m['s'] & 0xffffffff, self.native.now())
elif t in ('m', 'wheel', 'k', 'text', 'release'):
with self.input_lock:
if self.stop_event.is_set():
break
if self.input:
if not self.input_enabled and t != 'release':
continue
try:
self.input.handle(m)
self.stats.input(m)
except MALFORMED:
pass # drop a malformed viewer event; keep the session
except RuntimeError as e:
self.input_enabled = False
try:
self.input.release()
except RuntimeError:
pass
self.ws.send(dict(t='error', message=str(e)))
elif m.get('i'):
try:
self.test_inputs.put_nowait(m)
except queue.Full:
pass
finally:
self.end()
self.writer.join(6)
if not self.writer.is_alive():
self.controller.close()
def end(self):
with self.lock:
self.stop_event.set()
self.ws.close()
with self.input_lock:
if self.input and not self.released:
self.released = True
try:
self.input.release()
except (RuntimeError, OSError):
pass
class Agent:
def __init__(self, native, token, page):
self.native, self.page = native, page
self.lock = threading.RLock()
self.grants = Grants(token)
self.sessions, self.sources = {}, {}
self.next_id, self.selecting, self.selection_error = 1, False, ''
self.windows = Windows() if sys.platform == 'win32' else None
self.encoder = 'mfh264enc' if self.windows else 'vah264enc' if native.has('vah264enc') else 'x264enc'
self.shutting_down = False
self.selection_generation = 0
def lists(self):
if self.windows:
windows, displays = self.windows.sources()
self.sources = {s['src']: s for s in windows + displays}
return windows, displays
return [{k: v for k, v in s.items() if k not in ('portal', 'fd', 'node')} for s in self.sources.values()], []
def source(self, src):
if src == 'test':
return dict(src='test', title='Test pattern', w=1280, h=720)
self.lists()
if src not in self.sources:
raise ValueError('Choose a window or screen on this computer first')
return dict(self.sources[src])
def select(self):
if self.windows or self.selecting or self.shutting_down:
return
if len(self.sources) >= 8:
raise ValueError('Stop a panel before sharing another source')
self.selecting, self.selection_error = True, ''
generation = self.selection_generation
def choose():
error = C.create_string_buffer(1024)
portal = self.native.lib.fc_portal_select(error, len(error))
with self.lock:
self.selecting = False
if not portal:
self.selection_error = error.value.decode(errors='replace')
elif self.shutting_down or generation != self.selection_generation:
self.native.lib.fc_portal_close(portal)
else:
fd, node, w, h, devices = [self.native.lib.fc_portal_value(portal, i) for i in range(5)]
src = 'window:' + secrets.token_hex(8)
self.sources[src] = dict(src=src, title='Shared window or screen', app='Linux portal',
w=w, h=h, fd=fd, node=node, portal=portal, devices=devices)
threading.Thread(target=choose, daemon=True).start()
def stop(self, src=None):
with self.lock:
if src is None:
self.selection_generation += 1
self.grants.revoke(src)
sessions = [s for s in self.sessions.values() if src is None or s.src == src]
for session in sessions:
try:
session.ws.send(dict(t='close'))
except OSError:
pass
session.end()
for session in sessions:
if session.writer and session.writer is not threading.current_thread():
session.writer.join(6)
# The capture is stopped before releasing its PipeWire fd/session.
with self.lock:
if not self.windows:
for key, source in list(self.sources.items()):
if src is None or key == src:
if any(s.src == key and s.writer and s.writer.is_alive() for s in sessions):
continue
self.native.lib.fc_portal_close(source['portal'])
self.sources.pop(key, None)
return {'closed': len(sessions)}
class Handler(BaseHTTPRequestHandler):
protocol_version = 'HTTP/1.1'
def log_message(self, *args):
pass # URLs contain credentials
def reply(self, data, status=200, content='application/json'):
if not isinstance(data, bytes):
data = json.dumps(data).encode()
self.send_response(status)
self.send_header('Content-Type', content)
self.send_header('Content-Length', str(len(data)))
self.send_header('Cache-Control', 'no-store')
self.send_header('Connection', 'close')
self.end_headers()
self.wfile.write(data)
self.close_connection = True
def do_GET(self):
self.dispatch('GET')
def do_POST(self):
self.dispatch('POST')
def dispatch(self, method):
agent = self.server.agent
url = urlsplit(self.path)
q = {k: v[-1] for k, v in parse_qs(url.query).items()}
try:
if method == 'GET' and url.path == '/ping':
return self.reply(b'frame-mac-view', content='text/plain')
if method == 'GET' and url.path == '/view':
return self.reply(agent.page.read_bytes(), content='text/html; charset=utf-8')
if method == 'GET' and url.path == '/stream':
return self.stream(q)
if not agent.grants.master(q.get('k', self.headers.get('X-Token'))):
return self.reply({'error': 'forbidden'}, 403)
if method == 'POST' and url.path == '/close':
return self.reply(agent.stop(q.get('src')))
with agent.lock:
windows, displays = agent.lists()
if method == 'GET' and url.path == '/status':
data = dict(version=1, host='windows' if agent.windows else 'linux', screen=True,
accessibility=True, selecting=agent.selecting, selectionError=agent.selection_error,
encoder=agent.encoder, finished=[], streams=[dict(id=i, src=s.src,
title=s.source.get('title', ''), stats=s.stats.summary(), controller=s.controller.state())
for i, s in agent.sessions.items() if not s.stop_event.is_set()])
elif method == 'GET' and url.path in ('/windows', '/displays'):
data = dict(windows=windows, displays=displays, screen=True)
elif method == 'POST' and url.path == '/ticket':
agent.source(q.get('src'))
data = {'ticket': agent.grants.ticket(q['src'])}
elif method == 'POST' and url.path == '/permissions':
agent.select()
data = {'selecting': agent.selecting}
elif method == 'GET' and url.path == '/stats':
data = dict(now=agent.native.now(), streams=[dict(id=i, src=s.src, controller=s.controller.state(),
events=list(s.controller.events), **s.stats.snapshot(max(0, int(q.get('since', 0))), max(0, int(q.get('settle', 1500000)))))
for i, s in agent.sessions.items() if q.get('id', str(i)) == str(i) and not s.stop_event.is_set()])
elif method == 'POST' and url.path == '/bench':
data = {'sent': 0}
for s in agent.sessions.values():
if s.src == q.get('src'):
m = {k: v for k, v in q.items() if k not in ('k', 'src')}
for k in ('x', 'y', 'interval'):
if k in m:
m[k] = float(m[k])
s.ws.send(dict(m, t='bench'))
data['sent'] += 1
else:
return self.reply({'error': 'not found'}, 404)
self.reply(data)
except (ValueError, RuntimeError) as e:
self.reply({'error': str(e)}, 400)
except (OSError, ConnectionError):
self.close_connection = True
def stream(self, query):
agent = self.server.agent
with agent.lock:
key = agent.grants.redeem(query.get('src'), query)
if not key:
return self.reply({'error': 'forbidden'}, 403)
source = agent.source(query.get('src'))
if self.headers.get('Upgrade', '').lower() != 'websocket' or self.headers.get('Sec-WebSocket-Version') != '13':
return self.reply({'error': 'expected WebSocket'}, 400)
wskey = self.headers.get('Sec-WebSocket-Key', '')
if len(base64.b64decode(wskey, validate=True)) != 16:
raise ValueError('Bad WebSocket key')
if len(agent.sessions) >= 8:
raise ValueError('At most eight panels may be open')
# Stop can revoke and close only a registered session. Register
# under the same lock as redemption, before any capture starts.
session = Session(agent, WebSocket(self), key, source, query)
for old in list(agent.sessions.values()):
if old.key == key or old.src == source['src']:
if old.key != key:
agent.grants.keys.pop(old.key, None)
try:
old.ws.send(dict(t='close'))
except OSError:
pass
old.end()
if old.writer:
old.writer.join(6)
if old.writer.is_alive():
session.controller.close()
raise ValueError('The previous capture is still stopping; retry shortly')
ident = agent.next_id
agent.next_id += 1
agent.sessions[ident] = session
accept = base64.b64encode(hashlib.sha1((wskey+'258EAFA5-E914-47DA-95CA-C5AB0DC85B11').encode()).digest()).decode()
self.send_response(101)
self.send_header('Upgrade', 'websocket')
self.send_header('Connection', 'Upgrade')
self.send_header('Sec-WebSocket-Accept', accept)
self.end_headers()
try:
session.start()
except (OSError, ValueError, RuntimeError):
session.end()
finally:
session.end()
if session.writer:
session.writer.join(6)
if not session.writer or not session.writer.is_alive():
session.controller.close()
with agent.lock:
agent.sessions.pop(ident, None)
self.close_connection = True
def main():
parser = argparse.ArgumentParser()
parser.add_argument('command', choices=['serve'])
parser.add_argument('--port', type=int, default=0)
parser.add_argument('--page', type=Path, required=True)
parser.add_argument('--exit-on-eof', action='store_true')
args = parser.parse_args()
token = os.environ.get('FRAME_MAC_VIEW_TOKEN')
if not token:
raise SystemExit('Frame Control must provide a private token')
native = Native()
server = ThreadingHTTPServer(('127.0.0.1', args.port), Handler)
agent = server.agent = Agent(native, token, args.page)
if args.exit_on_eof:
def eof():
sys.stdin.buffer.read()
with agent.lock:
agent.shutting_down = True
agent.stop()
server.shutdown()
threading.Thread(target=eof, daemon=True).start()
print('listening on 127.0.0.1:%d' % server.server_port, flush=True)
try:
server.serve_forever()
finally:
agent.shutting_down = True
agent.stop()
server.server_close()
if __name__ == '__main__':
try:
main()
except (OSError, RuntimeError, AttributeError) as e:
raise SystemExit('PC streaming helper: ' + str(e))
-383
View File
@@ -1,383 +0,0 @@
"""Native PC capture and input bindings. No desktop-streaming app required.
The packaged native library contains our adapter and the shared rate controller;
GStreamer supplies WGC/Media Foundation and PipeWire/VA-API/x264 libraries.
"""
import ctypes as C
import os
from pathlib import Path
import re
import sys
import threading
ROOT = Path(__file__).resolve().parent.parent
NATIVE = Path(os.environ.get('FRAME_PC_NATIVE', ROOT / 'desktop' / 'bundle'))
LIBRARY = NATIVE / ('pc-host.dll' if sys.platform == 'win32' else 'pc-host.so')
class Encoded(C.Structure):
_fields_ = [('data', C.c_void_p), ('size', C.c_int), ('key', C.c_int),
('width', C.c_int), ('height', C.c_int), ('pts', C.c_int64)]
GATE = C.CFUNCTYPE(C.c_int, C.c_int, C.c_int64, C.c_int64, C.c_int64)
class Native:
def __init__(self, path=LIBRARY):
self.dll_dirs = []
if sys.platform == 'win32':
# Set process coordinates before loading GStreamer: a plugin may
# create a helper HWND during discovery, after which it is too late.
user = C.WinDLL('user32')
user.SetProcessDpiAwarenessContext.argtypes = [C.c_void_p]
user.SetProcessDpiAwarenessContext(C.c_void_p(-4))
for folder in (path.parent, path.parent / 'bin'):
self.dll_dirs.append(os.add_dll_directory(str(folder)))
self.lib = C.CDLL(str(path))
signatures = {
'fc_now': (C.c_int64, []), 'fc_gst_init': (None, []),
'fc_has_element': (C.c_int, [C.c_char_p]),
'fc_new': (C.c_void_p, [C.c_int, C.c_int]), 'fc_free': (None, [C.c_void_p]),
'fc_ceiling': (None, [C.c_void_p, C.c_int]),
'fc_gate': (C.c_int, [C.c_void_p, C.c_int64, C.c_int]),
'fc_capture': (None, [C.c_void_p, C.c_int64]),
'fc_sent': (None, [C.c_void_p, C.c_uint32, C.c_int, C.c_int64]),
'fc_ack': (C.c_int, [C.c_void_p, C.c_uint32, C.c_int64]),
'fc_update': (C.c_int, [C.c_void_p, C.c_int64]),
'fc_value': (C.c_int64, [C.c_void_p, C.c_int]),
'fc_capture_open': (C.c_void_p, [C.c_char_p, GATE, C.c_char_p, C.c_int]),
'fc_capture_pull': (C.c_int, [C.c_void_p, C.POINTER(Encoded)]),
'fc_capture_error': (C.c_char_p, [C.c_void_p]),
'fc_capture_bitrate': (None, [C.c_void_p, C.c_int]),
'fc_capture_key': (None, [C.c_void_p]),
'fc_capture_test': (None, [C.c_void_p, C.c_uint32]), 'fc_capture_close': (None, [C.c_void_p]),
}
if sys.platform.startswith('linux'):
signatures.update({
'fc_portal_select': (C.c_void_p, [C.c_char_p, C.c_int]),
'fc_portal_close': (None, [C.c_void_p]),
'fc_portal_refresh': (C.c_int, [C.c_void_p, C.c_char_p, C.c_int]),
'fc_portal_value': (C.c_int, [C.c_void_p, C.c_int]),
'fc_portal_input': (C.c_int, [C.c_void_p, C.c_int, C.c_double, C.c_double, C.c_int, C.c_int]),
})
for name, (result, args) in signatures.items():
fn = getattr(self.lib, name)
fn.restype, fn.argtypes = result, args
self.lib.fc_gst_init()
def has(self, name):
return bool(self.lib.fc_has_element(name.encode()))
def now(self):
return self.lib.fc_now()
class Controller:
def __init__(self, lib, fps, ceiling):
# close() may run on the stream thread while /status reads state();
# after close every entry point is a no-op, never a NULL dereference.
self.lib, self.lock, self.fps = lib, threading.RLock(), fps
self.enabled = os.environ.get('FRAME_MAC_VIEW_ADAPT') != '0'
self.ptr = lib.fc_new(fps, self.enabled)
if not self.ptr:
raise MemoryError('Unable to allocate the streaming controller')
lib.fc_ceiling(self.ptr, ceiling)
self.events = []
def state(self):
with self.lock:
names = ['target', 'ceiling', 'tier', 'fps', 'scale', 'baseRtt', 'inFlight', 'slack']
if not self.ptr:
out = dict.fromkeys(names, 0)
out.update(fps=self.fps, scale=100)
else:
out = {k: self.lib.fc_value(self.ptr, i) for i, k in enumerate(names)}
out.update(scale=out['scale'] / 100, baseRtt=out['baseRtt'] / 1000,
slack=out['slack'] / 1000, adapt=self.enabled)
return out
def call(self, name, *args):
with self.lock:
return getattr(self.lib, 'fc_' + name)(self.ptr, *args) if self.ptr else 0
def update(self, now):
with self.lock:
if not self.ptr:
return 0
old = self.state()
result = self.lib.fc_update(self.ptr, now)
state = self.state()
if state['target'] < old['target'] or state['tier'] != old['tier']:
self.events.append({'t': now, 'e': 'target %s bit/s; tier %s' % (state['target'], state['tier'])})
self.events = self.events[-200:]
return result
def close(self):
with self.lock:
if self.ptr:
self.lib.fc_free(self.ptr)
self.ptr = None
def dimensions(w, h, maximum):
scale = min(1, maximum / max(w, h))
return max(2, int(w * scale) // 2 * 2), max(2, int(h * scale) // 2 * 2)
def pipeline(source, platform, encoder, w, h, fps, bitrate, codec='h264'):
"""Only locally constructed numeric source IDs enter the pipeline parser."""
if source['src'] == 'test':
capture = 'videotestsrc name=source is-live=true pattern=ball'
elif platform == 'win32':
kind, ident = source['src'].split(':')
if kind not in ('window', 'display') or not re.fullmatch(r'[0-9]+', ident):
raise ValueError('Invalid Windows capture source')
prop = 'window-handle' if kind == 'window' else 'monitor-handle'
capture = 'd3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true %s=%d' % (prop, int(ident))
else:
capture = 'pipewiresrc fd=%d path=%d do-timestamp=true' % (source['fd'], source['node'])
# The source gate runs before conversion/encoding. There is no leaky queue
# of H.264 frames; every encoded reference frame reaches the socket.
raw = '%s ! video/x-raw,framerate=%d/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=%d,height=%d' % (capture, fps, w, h)
if codec == 'jpeg':
enc = 'jpegenc name=enc quality=80'
parse = ''
else:
choices = {
'mfh264enc': 'mfh264enc name=enc low-latency=true bframes=0 gop-size=60',
'vah264enc': 'vah264enc name=enc b-frames=0 key-int-max=60',
'x264enc': 'x264enc name=enc tune=zerolatency speed-preset=ultrafast bframes=0 key-int-max=60',
}
enc = choices[encoder] + ' bitrate=%d' % max(1, bitrate // 1000)
raw += ',format=NV12' if encoder != 'x264enc' else ',format=I420'
parse = ' ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au'
return raw + ' ! ' + enc + parse + ' ! appsink name=out sync=false max-buffers=2 drop=false'
# X11 keysyms work on Wayland through the RemoteDesktop portal too. The host's
# keyboard layout handles physical keys; Unicode text is sent as Unicode keysyms.
KEYSYMS = {'Enter': 0xff0d, 'Tab': 0xff09, 'Backspace': 0xff08, 'Escape': 0xff1b,
'Delete': 0xffff, 'ArrowLeft': 0xff51, 'ArrowUp': 0xff52, 'ArrowRight': 0xff53,
'ArrowDown': 0xff54, 'Home': 0xff50, 'End': 0xff57, 'PageUp': 0xff55,
'PageDown': 0xff56, 'Shift': 0xffe1, 'Control': 0xffe3, 'Alt': 0xffe9, 'Meta': 0xffeb}
class PortalInput:
def __init__(self, native, source):
self.lib, self.source = native.lib, source
self.buttons, self.keys = set(), set()
def emit(self, kind, x=0, y=0, code=0, down=0):
if not self.lib.fc_portal_input(self.source['portal'], kind, x, y, code, down):
raise RuntimeError('The desktop portal refused input; check the sharing permission')
def release(self):
error = None
for values, kind in ((self.buttons, 1), (self.keys, 3)):
for code in list(values):
try:
self.emit(kind, code=code, down=0)
values.discard(code)
except RuntimeError as e:
error = e
if error:
raise error
def handle(self, m):
t = m['t']
if t == 'release':
return self.release()
if t in ('m', 'wheel'):
x, y = (max(0, min(1, float(m.get(k, 0)))) for k in ('x', 'y'))
self.emit(0, x * (self.source['w'] - 1), y * (self.source['h'] - 1))
if t == 'm' and m.get('e') in ('up', 'down'):
b = {0: 0x110, 1: 0x112, 2: 0x111}.get(m.get('b', 0))
if b is not None:
down = m['e'] == 'down'
self.emit(1, code=b, down=down)
(self.buttons.add if down else self.buttons.discard)(b)
elif t == 'wheel':
self.emit(2, max(-4096, min(4096, float(m.get('dx', 0)))), max(-4096, min(4096, float(m.get('dy', 0)))))
elif t == 'k':
key = str(m.get('key', ''))
k = KEYSYMS.get(key) or (ord(key) if len(key) == 1 and ord(key) < 256 else
0x1000000 + ord(key) if len(key) == 1 else 0)
if k:
down = m.get('e') == 'down'
self.emit(3, code=k, down=down)
(self.keys.add if down else self.keys.discard)(k)
elif t == 'text':
for ch in str(m.get('s', ''))[:4096]:
k = ord(ch) if ord(ch) < 256 else 0x1000000 + ord(ch)
self.emit(3, code=k, down=1)
self.emit(3, code=k, down=0)
class Windows:
"""WGC source handles and SendInput. No elevation or global input hook."""
def __init__(self):
from ctypes import wintypes as W
self.W = W
self.user = C.WinDLL('user32', use_last_error=True)
self.dwm = C.WinDLL('dwmapi')
self.user.IsWindow.argtypes = [W.HWND]
self.user.IsWindowVisible.argtypes = [W.HWND]
self.user.IsIconic.argtypes = [W.HWND]
self.user.GetWindowTextLengthW.argtypes = [W.HWND]
self.user.GetWindowTextW.argtypes = [W.HWND, W.LPWSTR, C.c_int]
self.user.GetWindowRect.argtypes = [W.HWND, C.POINTER(W.RECT)]
self.user.SetForegroundWindow.argtypes = [W.HWND]
self.user.GetForegroundWindow.argtypes = []
self.user.GetForegroundWindow.restype = W.HWND
self.dwm.DwmGetWindowAttribute.argtypes = [W.HWND, W.DWORD, C.c_void_p, W.DWORD]
self.callback = C.WINFUNCTYPE(W.BOOL, W.HWND, W.LPARAM)
self.monitor_callback = C.WINFUNCTYPE(W.BOOL, W.HMONITOR, W.HDC, C.POINTER(W.RECT), W.LPARAM)
self.user.EnumWindows.argtypes = [self.callback, W.LPARAM]
self.user.EnumDisplayMonitors.argtypes = [W.HDC, C.c_void_p, self.monitor_callback, W.LPARAM]
class Mouse(C.Structure):
_fields_ = [('dx', W.LONG), ('dy', W.LONG), ('data', W.DWORD), ('flags', W.DWORD),
('time', W.DWORD), ('extra', C.c_size_t)]
class Key(C.Structure):
_fields_ = [('vk', W.WORD), ('scan', W.WORD), ('flags', W.DWORD), ('time', W.DWORD), ('extra', C.c_size_t)]
class Union(C.Union):
_fields_ = [('mouse', Mouse), ('key', Key)]
class Input(C.Structure):
_fields_ = [('type', W.DWORD), ('data', Union)]
self.Mouse, self.Key, self.Input = Mouse, Key, Input
self.user.SendInput.argtypes = [W.UINT, C.POINTER(Input), C.c_int]
self.user.SendInput.restype = W.UINT
def rect(self, hwnd):
r = self.W.RECT()
if not self.user.IsWindow(hwnd) or self.user.IsIconic(hwnd):
raise RuntimeError('The captured window closed or was minimized')
# WGC captures the visible extended frame, excluding invisible resize borders.
if self.dwm.DwmGetWindowAttribute(hwnd, 9, C.byref(r), C.sizeof(r)):
if not self.user.GetWindowRect(hwnd, C.byref(r)):
raise RuntimeError('Cannot locate the captured window')
return r.left, r.top, r.right - r.left, r.bottom - r.top
def sources(self):
windows, displays = [], []
def window(hwnd, _):
if not self.user.IsWindowVisible(hwnd) or self.user.IsIconic(hwnd):
return True
n = self.user.GetWindowTextLengthW(hwnd)
if n:
title = C.create_unicode_buffer(n + 1)
self.user.GetWindowTextW(hwnd, title, n + 1)
try:
x, y, w, h = self.rect(hwnd)
if w > 0 and h > 0:
windows.append(dict(src='window:%d' % hwnd, id=hwnd, title=title.value,
app='Windows', w=w, h=h, x=x, y=y))
except RuntimeError:
pass
return True
def monitor(handle, dc, rect, data):
r = rect.contents
displays.append(dict(src='display:%d' % handle, name='Display %d' % (len(displays) + 1),
x=r.left, y=r.top, w=r.right-r.left, h=r.bottom-r.top))
return True
self.user.EnumWindows(self.callback(window), 0)
self.user.EnumDisplayMonitors(None, None, self.monitor_callback(monitor), 0)
return windows, displays
def send(self, mouse=None, key=None):
inp = self.Input()
inp.type = 0 if mouse else 1
if mouse:
inp.data.mouse = self.Mouse(*mouse, 0, 0)
else:
inp.data.key = self.Key(*key, 0, 0)
if self.user.SendInput(1, C.byref(inp), C.sizeof(inp)) != 1:
raise RuntimeError('Windows refused input (elevated apps and the secure desktop cannot be controlled)')
class WindowsInput:
def __init__(self, host, source):
self.host, self.source = host, source
self.buttons, self.keys = set(), set()
def release(self):
error = None
for button in list(self.buttons):
try:
self.host.send(mouse=(0, 0, 0, {0: 4, 1: 0x40, 2: 0x10}[button]))
self.buttons.discard(button)
except RuntimeError as e:
error = e
for vk in list(self.keys):
try:
extended = 1 if vk in (33, 34, 35, 36, 37, 38, 39, 40, 45, 46, 91, 92, 163, 165) else 0
self.host.send(key=(vk, 0, 2 | extended))
self.keys.discard(vk)
except RuntimeError as e:
error = e
if error:
raise error
def focus(self):
if not self.source['src'].startswith('window:'):
return
hwnd = int(self.source['src'].split(':')[1])
user = self.host.user
if user.GetForegroundWindow() != hwnd and not user.SetForegroundWindow(hwnd):
raise RuntimeError('Windows could not focus this window. Bring it to the front on your PC, then press Stop and Show again.')
if user.GetForegroundWindow() != hwnd:
raise RuntimeError('The selected window is not in front. Reopen the view after bringing it to the front on your PC.')
def handle(self, m):
t, u = m['t'], self.host.user
if t == 'release':
return self.release()
if t in ('m', 'wheel'):
source = self.source
if source['src'].startswith('window:'):
hwnd = int(source['src'].split(':')[1])
x, y, w, h = self.host.rect(hwnd)
if m.get('e') == 'down' or t == 'wheel':
self.focus()
else:
x, y, w, h = (source[k] for k in ('x', 'y', 'w', 'h'))
x += max(0, min(1, float(m.get('x', 0)))) * (w - 1)
y += max(0, min(1, float(m.get('y', 0)))) * (h - 1)
left, top, width, height = (u.GetSystemMetrics(i) for i in (76, 77, 78, 79))
self.host.send(mouse=(round((x-left)*65535/max(1,width-1)), round((y-top)*65535/max(1,height-1)), 0, 0xc001))
if t == 'm' and m.get('e') in ('up', 'down'):
b = m.get('b', 0)
if b in (0, 1, 2):
down = m['e'] == 'down'
self.host.send(mouse=(0, 0, 0, {0: 2, 1: 0x20, 2: 8}[b] * (1 if down else 2)))
(self.buttons.add if down else self.buttons.discard)(b)
elif t == 'wheel':
for name, flag, direction in (('dy', 0x800, -1), ('dx', 0x1000, 1)):
delta = int(max(-4096, min(4096, float(m.get(name, 0))))) * direction
if delta:
self.host.send(mouse=(0, 0, delta & 0xffffffff, flag))
elif t == 'k':
code, down = str(m.get('code', '')), m.get('e') == 'down'
if down:
self.focus()
special = {'Enter': 13, 'Escape': 27, 'Tab': 9, 'Backspace': 8, 'Space': 32,
'ArrowLeft': 37, 'ArrowUp': 38, 'ArrowRight': 39, 'ArrowDown': 40,
'Delete': 46, 'Home': 36, 'End': 35, 'PageUp': 33, 'PageDown': 34,
'ShiftLeft': 160, 'ShiftRight': 161, 'ControlLeft': 162, 'ControlRight': 163,
'AltLeft': 164, 'AltRight': 165, 'MetaLeft': 91, 'MetaRight': 92}
vk = special.get(code, 0)
if re.fullmatch(r'Key[A-Z]', code) or re.fullmatch(r'Digit[0-9]', code):
vk = ord(code[-1])
if vk:
extended = 1 if vk in (33, 34, 35, 36, 37, 38, 39, 40, 45, 46, 91, 92, 163, 165) else 0
self.host.send(key=(vk, 0, extended | (0 if down else 2)))
(self.keys.add if down else self.keys.discard)(vk)
elif down and len(str(m.get('key', ''))) == 1:
self.handle({'t': 'text', 's': m['key']})
elif t == 'text':
self.focus()
raw = str(m.get('s', ''))[:4096].encode('utf-16-le')
for i in range(0, len(raw), 2):
unit = int.from_bytes(raw[i:i+2], 'little')
self.host.send(key=(0, unit, 4))
self.host.send(key=(0, unit, 6))
-90
View File
@@ -1,90 +0,0 @@
"""PC host selection behind the existing MacView tunnel/panel controller.
The public /api/macview name and viewer URL remain compatible with the Mac
base branch. Only helper launch, platform availability and source validation
are different. No second viewer, SSH supervisor or benchmark launcher.
"""
import os
import sys
import subprocess
from frame_macview import MacView, MacViewError, ROOT
from frame_pc_capture import LIBRARY, NATIVE
def prepend(env, name, path):
"""An empty entry means the current directory to the loader; never add one."""
rest = env.get(name, '')
env[name] = str(path) + (os.pathsep + rest if rest else '')
class PCView(MacView):
viewer_profile = 'pc-view'
host = 'windows' if sys.platform == 'win32' else 'linux'
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.prefer_usb = False # Mac networksetup probe is platform-specific
def unavailable(self):
if sys.platform not in ('win32', 'linux'):
return 'PC streaming needs Windows or a Linux desktop.'
if not LIBRARY.is_file():
return 'The PC streaming libraries are missing from this build. See docs/pc-in-headset.md.'
return None
def state(self):
result = super().state()
result["host"] = self.host
return result
def build(self):
pass # native libraries are built and bundled with the app
def agent_command(self):
return [sys.executable, str(ROOT / 'ui' / 'frame_pc_agent.py')]
def agent_environment(self):
env = super().agent_environment()
env['GST_PLUGIN_PATH_1_0'] = str(NATIVE / 'lib' / 'gstreamer-1.0')
env['GST_PLUGIN_SYSTEM_PATH_1_0'] = ''
cache = os.path.join(os.path.expanduser('~'), '.cache', 'frame-control')
os.makedirs(cache, exist_ok=True)
env['GST_REGISTRY_1_0'] = os.path.join(cache, 'gstreamer-registry.bin')
env['GST_REGISTRY_FORK'] = 'no'
if sys.platform == 'win32':
prepend(env, 'PATH', NATIVE / 'bin')
else:
prepend(env, 'LD_LIBRARY_PATH', NATIVE / 'lib')
env['PIPEWIRE_MODULE_DIR'] = str(NATIVE / 'lib' / 'pipewire-0.3')
env['SPA_PLUGIN_DIR'] = str(NATIVE / 'lib' / 'spa-0.2')
env['PIPEWIRE_CONFIG_DIR'] = str(NATIVE / 'share' / 'pipewire')
env['PIPEWIRE_CONFIG_NAME'] = 'client.conf'
return env
def shutdown(self):
self.closing = True
try:
self.stop()
except MacViewError:
pass
if self.tunnel and self.tunnel.poll() is None:
self.tunnel.terminate()
if self.agent and self.agent.poll() is None:
# EOF gives the host a chance to release held input and portal
# sessions, including on Windows where terminate is uncatchable.
if self.agent.stdin:
self.agent.stdin.close()
try:
self.agent.wait(10)
except subprocess.TimeoutExpired:
self.agent.kill()
self.agent.wait(5)
def _show(self, src, quality, width, height):
if src.startswith('separate:'):
raise MacViewError('Separate virtual displays are a Mac-only feature. Choose a window or screen.')
return super()._show(src, quality, width, height)
def host_view(*args, **kwargs):
return (MacView if sys.platform == 'darwin' else PCView)(*args, **kwargs)
+56 -24
View File
@@ -156,27 +156,59 @@ def flatpaks():
return out return out
uptime = read("/proc/uptime") def thermal_alerts():
procs = process_names() """Use the kernel's per-zone hot/critical trips, never a guessed chip limit."""
print(json.dumps({ alerts, known = [], False
"time": time.time(), for z in glob.glob("/sys/class/thermal/thermal_zone*"):
"hostname": socket.gethostname(), t = num(z + "/temp", 0.001)
"os": os_release(), for trip in glob.glob(z + "/trip_point_*_type"):
"uptime": float(uptime.split()[0]) if uptime else None, if read(trip) not in ("hot", "critical"):
"battery": battery(), continue
"power": power_source(), limit = num(trip[:-4] + "temp", 0.001)
"disk": {"root": disk("/"), "home": disk("/home")}, if t is not None and limit is not None and limit > 0:
"memory": memory(), known = True
"temp": max_temp(), if t >= limit:
"wifi": wifi(), alerts.append({"zone": read(z + "/type"), "tempC": t, "limitC": limit})
"ip": ip_addr(), return alerts if known else None
"volume": volume(),
"services": {
"steamvr": "vrserver" in procs, def activity_level():
"desktop": "plasmashell" in procs, try:
"lepton": port_listening(5555), rows = json.loads(run("/opt/steamvr/bin/linuxarm64/vrcmd", "--stats"))
"rdp": "xrdp" in procs, if not isinstance(rows, list):
}, return None
"games": games(), return next((r.get("activity_level") for r in rows
"flatpaks": flatpaks(), if isinstance(r, dict) and r.get("operation") == "status"), None)
})) except (ValueError, TypeError):
return None
def main():
uptime = read("/proc/uptime")
procs = process_names()
print(json.dumps({
"time": time.time(),
"hostname": socket.gethostname(),
"os": os_release(),
"uptime": float(uptime.split()[0]) if uptime else None,
"battery": battery(),
"power": power_source(),
"disk": {"root": disk("/"), "home": disk("/home")},
"memory": memory(),
"temp": max_temp(),
"wifi": wifi(),
"ip": ip_addr(),
"volume": volume(),
"services": {
"steamvr": "vrserver" in procs,
"desktop": "plasmashell" in procs,
"lepton": port_listening(5555),
"rdp": "xrdp" in procs,
},
"games": games(),
"flatpaks": flatpaks(),
}))
if __name__ == "__main__":
main()
-88
View File
@@ -1,88 +0,0 @@
"""The desktop stream's timing records, in the Mac viewer/bench schema.
Times are host monotonic microseconds. Zero means unknown, never a fabricated
capture/display timestamp. Storage is bounded to the same 4096/512 records as
Stats.swift. The existing macview-bench.py grades these records unchanged.
"""
from collections import OrderedDict
import threading
def distribution(values):
values = sorted(v for v in values if v is not None)
if not values:
return {}
return {key: round(values[min(len(values)-1, int((len(values)-1)*p + .5))], 1)
for key, p in (('p50', .5), ('p95', .95))}
class Stats:
def __init__(self, now):
self.now, self.lock = now, threading.RLock()
self.frames, self.inputs = OrderedDict(), OrderedDict()
self.captured = self.skipped = self.dropped = 0
self.rtt, self.decoder, self.synced = 0, '', False
self.sequence = 0
def add(self, **values):
with self.lock:
self.sequence = (self.sequence + 1) & 0xffffffff
record = dict.fromkeys(('s', 'k', 'b', 'w', 'h', 'cap', 'arr', 'e0', 'e1', 'snd',
'wire', 'rx', 'dec', 'drw', 'vs', 'echo', 'tier', 'br'), 0)
record.update(values, s=self.sequence)
self.frames[self.sequence] = record
while len(self.frames) > 4096:
self.frames.popitem(last=False)
for event in self.inputs.values():
if not event['frame'] and event['inj'] <= record['cap']:
event.update(frame=self.sequence, cap=record['cap'])
record['echo'] = event['id']
return record
def report(self, message):
with self.lock:
if message['t'] == 'rx':
frame = self.frames.get(message.get('s'))
if frame and isinstance(message.get('r'), (int, float)):
frame['rx'] = int(message['r'])
elif message['t'] == 'fd':
for item in message.get('f', [])[:4096]:
if isinstance(item, list) and len(item) >= 4:
frame = self.frames.get(item[0])
if frame:
for name, val in zip(('dec', 'drw', 'vs'), item[1:4]):
if isinstance(val, (int, float)):
frame[name] = int(val)
self.dropped += max(0, int(message.get('drop', 0)))
elif message['t'] == 'clock':
self.rtt = float(message.get('rtt', 0))
self.decoder = str(message.get('dec', ''))[:1024]
self.synced = True
def input(self, m):
if not isinstance(m.get('i'), int) or not m['i']:
return
with self.lock:
self.inputs[m['i']] = dict(id=m['i'], kind=m['t'], tv=m.get('tv') or 0,
inj=self.now(), frame=0, cap=0)
while len(self.inputs) > 512:
self.inputs.popitem(last=False)
def summary(self):
with self.lock:
now = self.now()
frames = [f for f in self.frames.values() if f['cap'] >= now-2000000]
out = dict(captured=self.captured, skipped=self.skipped, dropped=self.dropped,
rtt=self.rtt, decoder=self.decoder, synced=self.synced,
fps=sum(f['vs'] > 0 for f in frames)/2, sentFps=len(frames)/2,
mbps=round(sum(f['b'] for f in frames)*4/1e6, 2))
for key, start, end in (('capture', 'cap', 'arr'), ('queue', 'arr', 'e0'),
('encode', 'e0', 'e1'), ('network', 'e1', 'rx'),
('decode', 'rx', 'dec'), ('draw', 'dec', 'drw'), ('total', 'cap', 'drw')):
out[key] = distribution([(f[end]-f[start])/1000 for f in frames if f[start] and f[end]])
return out
def snapshot(self, since=0, settle=1500000):
with self.lock:
return dict(frames=[dict(f) for f in self.frames.values() if f['s'] > since and f['cap'] <= self.now()-settle],
inputs=[dict(i) for i in self.inputs.values()], captured=self.captured, summary=self.summary())
+446
View File
@@ -0,0 +1,446 @@
"""Touch and direct input for the Steam Frame's panels. Frame Control's server runs this ON the Frame.
gamescope, the Frame's compositor, serves Valve's own input injection: an EIS
socket (libei's server side), which Steam uses to feed it Remote Play input.
This connects to it with libei, which is on the SteamOS image, and points,
clicks, scrolls and types into the panel that has focus in the headset: the
one the wearer last used. No install, and it reaches every panel, on either
of gamescope's X displays (see docs/streaming.md).
python3 frame_touch.py focus print the focused panel as JSON
python3 frame_touch.py panels print every app panel as JSON, and which has focus
python3 frame_touch.py read events on stdin, one JSON object (or list) per line:
{"fx": 0.5, "fy": 0.2, "window": 123, "display": ":1"}
pointer to that fraction of that panel; any event can
name its panel, and goes nowhere if another has focus
{"dx": 4, "dy": -2} pointer by that much
{"button": "left", "down": true} left, right or middle; "down" false releases
{"scroll": [0, 120]} by pixels; positive y scrolls down
{"key": 30, "down": true} a Linux (evdev) key code, as the page maps KeyboardEvent.code
{"text": "hello"} printable ASCII, typed on a US layout
Status goes to stdout, one JSON object per line: {"state": "ready" | "error", ...}.
Standard library only (ctypes for libei), like the rest of what runs on the Frame.
"""
import ctypes
import json
import os
import select
import subprocess
import sys
import time
SOCKET = "/run/user/{uid}/gamescope-0-ei" # filled in on the Frame (Windows has no getuid; the tests import this)
BUTTONS = {"left": 0x110, "right": 0x111, "middle": 0x112} # BTN_LEFT, BTN_RIGHT, BTN_MIDDLE
SHIFT = 42 # KEY_LEFTSHIFT
# Printable ASCII on a US layout: character -> (evdev key code, shifted).
ROWS = [("1234567890-=", "!@#$%^&*()_+", 2), ("qwertyuiop[]", "QWERTYUIOP{}", 16),
("asdfghjkl;'`", 'ASDFGHJKL:"~', 30), ("\\zxcvbnm,./", "|ZXCVBNM<>?", 43)]
ASCII = {" ": (57, False), "\n": (28, False), "\t": (15, False)}
for plain, shifted, first in ROWS:
for i, (a, b) in enumerate(zip(plain, shifted)):
ASCII[a], ASCII[b] = (first + i, False), (first + i, True)
# libei's event types and device capabilities (libei.h, libei 1.4).
EV_CONNECT, EV_DISCONNECT, EV_SEAT_ADDED, EV_DEVICE_ADDED, EV_DEVICE_REMOVED = 1, 2, 3, 5, 6
EV_DEVICE_PAUSED, EV_DEVICE_RESUMED = 7, 8
CAP_POINTER, CAP_ABSOLUTE, CAP_KEYBOARD, CAP_SCROLL, CAP_BUTTON = 1, 2, 4, 16, 32
def say(state, **more):
print(json.dumps({"state": state, **more}), flush=True)
# ---- which panel has focus -----------------------------------------------------
def xprop_root(display, name):
try:
out = subprocess.run(["xprop", "-root", name], env=dict(os.environ, DISPLAY=display),
capture_output=True, text=True, timeout=5).stdout
except (OSError, subprocess.SubprocessError):
return []
values = out.split("=", 1)[1] if "=" in out else ""
return [int(v) for v in values.replace(",", " ").split() if v.isdigit()]
def window_info(display, window):
"""Name and geometry of a window on one X display, or None if it isn't there."""
try:
which = ["-root"] if window == "root" else ["-id", str(window)]
out = subprocess.run(["xwininfo", *which], env=dict(os.environ, DISPLAY=display),
capture_output=True, text=True, timeout=5).stdout
except (OSError, subprocess.SubprocessError):
return None
if "IsViewable" not in out:
return None
info = {}
for line in out.splitlines():
line = line.strip()
if line.startswith("xwininfo: Window id:"):
info["name"] = line.split('"', 1)[1].rsplit('"', 1)[0] if '"' in line else ""
for key, field in (("Absolute upper-left X:", "x"), ("Absolute upper-left Y:", "y"),
("Width:", "width"), ("Height:", "height")):
if line.startswith(key):
info[field] = int(line.split(":", 1)[1])
return info if "width" in info else None
def displays():
return sorted(f":{n[1:]}" for n in os.listdir("/tmp/.X11-unix") if n[1:].isdigit())
def window_pid(display, window):
try:
out = subprocess.run(["xprop", "-id", str(window), "_NET_WM_PID"], env=dict(os.environ, DISPLAY=display),
capture_output=True, text=True, timeout=5).stdout
except (OSError, subprocess.SubprocessError):
return None
value = out.rsplit("=", 1)[-1].strip() if "=" in out else ""
return int(value) if value.isdigit() else None
def locate(window, pid):
"""The display a focusable window is on, with its name and geometry.
Window ids are per X server, so :0 and :1 can both have one; the pid gamescope
lists with it (GAMESCOPE_FOCUSABLE_WINDOWS) tells them apart.
"""
found = []
for display in displays():
info = window_info(display, window)
if info:
found.append((display, info))
if len(found) > 1 and pid:
found = [f for f in found if window_pid(f[0], window) == pid] or found
if not found:
return None
display, info = found[0]
root = window_info(display, "root") or {}
return {"window": window, "display": display, **info,
"root": [root.get("width", info["width"]), root.get("height", info["height"])]}
def focusable():
"""gamescope's focusable windows as (window, app id, pid)."""
t = xprop_root(":0", "GAMESCOPE_FOCUSABLE_WINDOWS")
return [tuple(t[i:i + 3]) for i in range(0, len(t) - 2, 3)]
def focus_display():
"""The display of the focused window, from GAMESCOPE_FOCUS_DISPLAY on :0's root.
gamescope writes the name (":1") as 32-bit items, so its first four bytes land,
little-endian, in the first value: 12602 is 0x313A, ":1" (steamcompmgr.cpp;
seen 2026-09-29).
"""
values = xprop_root(":0", "GAMESCOPE_FOCUS_DISPLAY")
if not values:
return None
name = (values[0] & 0xFFFFFFFF).to_bytes(4, "little").split(b"\0", 1)[0].decode("ascii", "replace")
return name if name[:1] == ":" and name[1:].isdigit() else None
def focus_now():
"""Just which window and display have focus: two property reads, for checking a press."""
window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
return (window or None, focus_display() if window else None)
def focus():
"""The panel that has focus in the headset: window, display, name and sizes (gamescope
publishes the window and its display on :0's root)."""
window = (xprop_root(":0", "GAMESCOPE_FOCUSED_WINDOW") or [0])[0]
if not window:
return {"window": None}
app, pid = next(((a, p) for w, a, p in focusable() if w == window), (None, None))
display = focus_display()
info = window_info(display, window) if display else None
if info:
root = window_info(display, "root") or {}
panel = {"window": window, "display": display, **info,
"root": [root.get("width", info["width"]), root.get("height", info["height"])]}
else:
panel = locate(window, pid) # no display published: tell them apart by pid
return {**panel, "app": app} if panel else {"window": None}
def panels():
"""Every app panel (gamescope's focusable windows), for watching one that hasn't focus."""
now = focus()
found = []
for window, app, pid in focusable():
panel = locate(window, pid)
if panel and panel["width"] > 1 and panel["height"] > 1 and \
not any(f["window"] == window and f["display"] == panel["display"] for f in found):
panel.pop("root", None)
found.append({**panel, "app": app, "focused": (window, panel["display"]) ==
(now.get("window"), now.get("display"))})
return {"focus": now.get("window"), "focus_display": now.get("display"), "panels": found}
def to_root(panel, fx, fy):
"""A point given as a fraction of the panel, in the root coordinates gamescope's pointer uses.
gamescope fits each panel's window to its display, so a 1920x1080 window on a
1280x720 display takes pointer positions at two thirds scale (verified 2026-09-29).
"""
rw, rh = panel["root"]
w, h = panel["width"], panel["height"]
s = min(rw / w, rh / h)
ox, oy = (rw - w * s) / 2, (rh - h * s) / 2
fx, fy = min(max(fx, 0.0), 1.0), min(max(fy, 0.0), 1.0)
return ox + fx * (w * s - 1), oy + fy * (h * s - 1)
# ---- gamescope's input socket ----------------------------------------------------
def libei():
L = ctypes.CDLL("libei.so.1")
vp, c = ctypes.c_void_p, ctypes
sig = {
"ei_new_sender": (vp, [vp]), "ei_configure_name": (None, [vp, c.c_char_p]),
"ei_setup_backend_socket": (c.c_int, [vp, c.c_char_p]), "ei_get_fd": (c.c_int, [vp]),
"ei_dispatch": (None, [vp]), "ei_get_event": (vp, [vp]), "ei_event_get_type": (c.c_int, [vp]),
"ei_event_unref": (vp, [vp]), "ei_event_get_seat": (vp, [vp]), "ei_event_get_device": (vp, [vp]),
"ei_device_has_capability": (c.c_bool, [vp, c.c_int]), "ei_now": (c.c_uint64, [vp]),
"ei_device_start_emulating": (None, [vp, c.c_uint32]), "ei_device_stop_emulating": (None, [vp]),
"ei_device_frame": (None, [vp, c.c_uint64]),
"ei_device_pointer_motion": (None, [vp, c.c_double, c.c_double]),
"ei_device_pointer_motion_absolute": (None, [vp, c.c_double, c.c_double]),
"ei_device_button_button": (None, [vp, c.c_uint32, c.c_bool]),
"ei_device_scroll_delta": (None, [vp, c.c_double, c.c_double]),
"ei_device_keyboard_key": (None, [vp, c.c_uint32, c.c_bool]),
"ei_unref": (vp, [vp]),
}
for name, (res, args) in sig.items():
f = getattr(L, name)
f.restype, f.argtypes = res, args
return L
class Gamescope:
"""One connection to gamescope's EIS socket and its virtual input device."""
def __init__(self):
self.L = L = libei()
self.ei = L.ei_new_sender(None)
L.ei_configure_name(self.ei, b"Frame Control")
if L.ei_setup_backend_socket(self.ei, SOCKET.format(uid=os.getuid()).encode()) != 0:
raise RuntimeError("Couldn't reach gamescope's input socket. Is the headset on?")
self.fd = L.ei_get_fd(self.ei)
self.device, self.sequence, self.held, self.keys, self.alive = None, 0, set(), set(), True
def pump(self, wait=0.0):
"""Handle gamescope's events; False once it has disconnected."""
select.select([self.fd], [], [], wait)
self.L.ei_dispatch(self.ei)
alive = True
while True:
ev = self.L.ei_get_event(self.ei)
if not ev:
return alive
kind = self.L.ei_event_get_type(ev)
if kind == EV_SEAT_ADDED:
seat = self.L.ei_event_get_seat(ev)
# Variadic, ending in 0 (NULL): ask for everything we send.
self.L.ei_seat_bind_capabilities(ctypes.c_void_p(seat), *map(ctypes.c_int, (
CAP_POINTER, CAP_ABSOLUTE, CAP_BUTTON, CAP_SCROLL, CAP_KEYBOARD, 0)))
elif kind == EV_DEVICE_RESUMED:
device = self.L.ei_event_get_device(ev)
if self.L.ei_device_has_capability(device, CAP_ABSOLUTE):
self.sequence += 1
self.L.ei_device_start_emulating(device, self.sequence)
self.device = device
# Releases that arrived while it was paused were dropped: let go of
# everything now, so the headset and this agent agree nothing is held.
if self.held or self.keys:
self.release_all()
elif kind in (EV_DEVICE_PAUSED, EV_DEVICE_REMOVED):
if self.L.ei_event_get_device(ev) == self.device:
self.device = None
elif kind == EV_DISCONNECT:
self.device, alive, self.alive = None, False, False
self.L.ei_event_unref(ev)
def wait_ready(self, timeout=5):
end = time.time() + timeout
while self.device is None and time.time() < end:
if not self.pump(0.1):
break
if self.device is None:
raise RuntimeError("gamescope closed its input socket" if not self.alive
else "gamescope didn't offer an input device")
def frame(self):
self.L.ei_device_frame(self.device, self.L.ei_now(self.ei))
self.L.ei_dispatch(self.ei)
def move_to(self, x, y):
self.L.ei_device_pointer_motion_absolute(self.device, x, y)
self.frame()
def move_by(self, dx, dy):
self.L.ei_device_pointer_motion(self.device, dx, dy)
self.frame()
def button(self, name, down):
code = BUTTONS[name]
if down == (code in self.held):
return # already in that state
self.L.ei_device_button_button(self.device, code, down)
self.frame()
(self.held.add if down else self.held.discard)(code)
def scroll(self, dx, dy):
self.L.ei_device_scroll_delta(self.device, dx, dy)
self.frame()
def key(self, code, down):
self.L.ei_device_keyboard_key(self.device, code, down)
self.frame()
(self.keys.add if down else self.keys.discard)(code)
# Paced: a burst of keys can reach the app out of order (seen 2026-09-29).
time.sleep(0.008)
def text(self, text):
for ch in text:
if ch not in ASCII:
continue
code, shifted = ASCII[ch]
if shifted:
self.key(SHIFT, True)
self.key(code, True)
self.key(code, False)
if shifted:
self.key(SHIFT, False)
def release_all(self):
"""Let go of every button and key still down, so nothing stays held in the headset."""
for code in list(self.held):
name = next(n for n, c in BUTTONS.items() if c == code)
self.button(name, False)
for code in list(self.keys):
self.key(code, False)
# ---- events from the server --------------------------------------------------------
def events(line):
try:
data = json.loads(line)
except ValueError:
return []
return [e for e in (data if isinstance(data, list) else [data]) if isinstance(e, dict)]
def number(value, limit=100000.0):
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
raise ValueError("not a number")
return max(-limit, min(limit, float(value)))
STALE = [False] # whether the last status said a tap went nowhere
def aimed_elsewhere(event, panel):
"""Whether an event names a panel that isn't the one with focus now."""
if "window" not in event:
return False
return (panel.get("window"), panel.get("display")) != (event.get("window"), event.get("display"))
def apply(gs, event, panel):
"""Send one event; returns the focused panel it checked against (looked up at most once a second).
Positions and presses name the panel they were meant for. If focus has moved to
another panel since, they go nowhere, so a tap can't land on the wrong one;
releases always go, so nothing stays held.
"""
# Moves may use a focus reading up to a second old; anything that acts (a press, key,
# text or scroll) reads it afresh, so it can't land on a panel that took focus since.
acts = any(k in event for k in ("button", "key", "text", "scroll")) and event.get("down") is not False
if "window" in event:
if panel and acts and focus_now() == (panel.get("window"), panel.get("display")):
pass # still the same panel (its geometry is re-read on the usual one-second schedule)
elif acts or not panel or time.time() - panel.get("_at", 0) > 1 or aimed_elsewhere(event, panel):
panel = {**focus(), "_at": time.time()}
stale = aimed_elsewhere(event, panel) if "window" in event else False
if stale and not (event.get("down") is False and ("button" in event or "key" in event)):
say("ready", focus=panel.get("window"), display=panel.get("display"), stale=True) # the page re-syncs
STALE[0] = True
return panel
if STALE[0] and not stale:
# Anything that goes through (a trackpad move names no panel) means caught up: stop re-syncing.
STALE[0] = False
say("ready", focus=(panel or {}).get("window"), display=(panel or {}).get("display"))
if "fx" in event and panel and panel.get("window"):
gs.move_to(*to_root(panel, number(event["fx"], 1), number(event["fy"], 1)))
if "dx" in event or "dy" in event:
gs.move_by(number(event.get("dx", 0), 2000), number(event.get("dy", 0), 2000))
if event.get("button") in BUTTONS:
gs.button(event["button"], event.get("down") is not False)
if isinstance(event.get("scroll"), list) and len(event["scroll"]) == 2:
gs.scroll(number(event["scroll"][0], 5000), number(event["scroll"][1], 5000))
if isinstance(event.get("key"), int) and not isinstance(event["key"], bool) and 0 < event["key"] < 768:
gs.key(event["key"], event.get("down") is not False)
if isinstance(event.get("text"), str):
gs.text(event["text"][:500])
return panel
def main():
if sys.argv[1:] == ["focus"]:
print(json.dumps(focus()))
return 0
if sys.argv[1:] == ["panels"]:
print(json.dumps(panels()))
return 0
try:
gs = Gamescope()
gs.wait_ready()
except (OSError, RuntimeError) as e:
say("error", message=str(e))
return 1
say("ready", focus=focus().get("window"))
stdin, pending, panel = sys.stdin.fileno(), b"", None
try:
while True:
ready, _, _ = select.select([stdin, gs.fd], [], [], 30)
if gs.fd in ready and not gs.pump():
say("error", message="gamescope closed its input socket")
return 1
if stdin not in ready:
continue
chunk = os.read(stdin, 65536)
if not chunk:
return 0 # the server went away
*lines, pending = (pending + chunk).split(b"\n")
for line in lines:
waited = False
for event in events(line):
if gs.device is None and waited:
continue # still paused: don't wait again for each event of this batch
if gs.device is None:
waited = True
# Paused (gamescope can pause the device): wait a moment; drop this
# event if it doesn't come back. Only a disconnect ends the session.
try:
gs.wait_ready(2)
except RuntimeError:
if not gs.alive:
raise
continue
try:
panel = apply(gs, event, panel)
except (ValueError, KeyError, TypeError, OSError):
continue # the server checks events; skip anything odd
except RuntimeError as e:
say("error", message=str(e))
return 1
finally:
if gs.device is not None:
gs.release_all() # never leave a button held down in the headset
if __name__ == "__main__":
sys.exit(main())
+674 -47
View File
@@ -102,6 +102,7 @@
.shelf-head .count { color: var(--muted); font-size: 13px; } .shelf-head .count { color: var(--muted); font-size: 13px; }
.shelf-head .spacer { flex: 1; } .shelf-head .spacer { flex: 1; }
.sub { color: var(--muted); font-size: 12.5px; } .sub { color: var(--muted); font-size: 12.5px; }
.sr-only { position: absolute; width: 1px; height: 1px; overflow: hidden; clip-path: inset(50%); white-space: nowrap; }
.hint { color: var(--muted); font-size: 12.5px; margin-top: 11px; line-height: 1.5; } .hint { color: var(--muted); font-size: 12.5px; margin-top: 11px; line-height: 1.5; }
/* ---- buttons ---- */ /* ---- buttons ---- */
@@ -147,6 +148,18 @@
.viewer .badge { position: absolute; left: 12px; top: 12px; font-size: 11px; letter-spacing: 1.2px; text-transform: uppercase; .viewer .badge { position: absolute; left: 12px; top: 12px; font-size: 11px; letter-spacing: 1.2px; text-transform: uppercase;
background: rgba(0,0,0,.6); color: var(--text); padding: 4px 9px; border-radius: 2px; } background: rgba(0,0,0,.6); color: var(--text); padding: 4px 9px; border-radius: 2px; }
.viewer .badge.live { left: auto; right: 12px; background: var(--bad); color: #fff; font-weight: 700; } .viewer .badge.live { left: auto; right: 12px; background: var(--bad); color: #fff; font-weight: 700; }
.view-actions { display: flex; gap: 10px; }
.ctrl-status { margin-top: 10px; font-size: 13px; color: var(--blue); }
.ctrl-status::before { content: "● "; }
.ctrl-status.warn { color: var(--warn); }
/* Control: the view takes taps, clicks, drags and keys, and none of them zoom or pan it. */
.viewer.control { outline: 1px solid var(--blue); }
.viewer.control canvas { cursor: crosshair; touch-action: none; }
.viewer.control.rel canvas { cursor: move; }
.viewer.immersive { position: fixed; z-index: 40; inset: env(safe-area-inset-top) env(safe-area-inset-right) 0 env(safe-area-inset-left);
max-height: none; aspect-ratio: auto; border-radius: 0; }
.ctrl-keys { margin-top: 10px; }
#panelPick { max-width: 240px; height: 30px; }
.viewer .stamp { position: absolute; right: 12px; bottom: 12px; font-size: 11px; color: #c7d5e0; .viewer .stamp { position: absolute; right: 12px; bottom: 12px; font-size: 11px; color: #c7d5e0;
background: rgba(0,0,0,.6); padding: 3px 8px; border-radius: 2px; } background: rgba(0,0,0,.6); padding: 3px 8px; border-radius: 2px; }
.viewer .asleep { position: absolute; inset: auto 0 0 0; padding: 14px 18px; text-align: center; color: var(--text); .viewer .asleep { position: absolute; inset: auto 0 0 0; padding: 14px 18px; text-align: center; color: var(--text);
@@ -348,7 +361,7 @@
.disp .k2 { color: var(--muted); font-size: 11px; letter-spacing: 1px; text-transform: uppercase; margin: 12px 0 5px; } .disp .k2 { color: var(--muted); font-size: 11px; letter-spacing: 1px; text-transform: uppercase; margin: 12px 0 5px; }
.disp .seg { flex-wrap: wrap; } .disp .seg { flex-wrap: wrap; }
.disp .seg button { padding: 0 10px; } .disp .seg button { padding: 0 10px; }
.disp input[type=number] { width: 72px; background: rgba(0,0,0,.28); color: var(--text); border: 1px solid transparent; .disp input[type=number], #comfort input[type=number] { width: 72px; background: rgba(0,0,0,.28); color: var(--text); border: 1px solid transparent;
border-radius: 3px; height: 32px; padding: 0 8px; font: inherit; font-size: 13px; } border-radius: 3px; height: 32px; padding: 0 8px; font: inherit; font-size: 13px; }
/* Touch screens can't hover: keep the library's name and Play button showing. */ /* Touch screens can't hover: keep the library's name and Play button showing. */
@media (hover: none) { .capsule .over { opacity: 1; } .capsule:hover { transform: none; } } @media (hover: none) { .capsule .over { opacity: 1; } .capsule:hover { transform: none; } }
@@ -390,6 +403,9 @@
.grid-top, .grid-3, .and-grid, .and-col { gap: 14px; } .grid-top, .grid-3, .and-grid, .and-col { gap: 14px; }
.toolbar { gap: 8px; } .toolbar { gap: 8px; }
.toolbar .spacer { display: none; } .toolbar .spacer { display: none; }
.view-actions { flex: 1 1 100%; display: grid; grid-template-columns: repeat(4, 1fr); gap: 6px; }
.view-actions button { justify-content: center; padding: 0 4px; }
#panelPick { flex: 1 1 100%; max-width: none; }
.shelf-head { flex-wrap: wrap; row-gap: 8px; } .shelf-head { flex-wrap: wrap; row-gap: 8px; }
.shelf { grid-template-columns: repeat(auto-fill, minmax(100px, 1fr)); gap: 10px; } .shelf { grid-template-columns: repeat(auto-fill, minmax(100px, 1fr)); gap: 10px; }
.cat-grid { grid-template-columns: 1fr; } .cat-grid { grid-template-columns: 1fr; }
@@ -469,16 +485,20 @@
<div class="toolbar"> <div class="toolbar">
<div class="seg" aria-label="What to capture"> <div class="seg" aria-label="What to capture">
<button data-view="headset" class="on" title="What the headset shows, via the SteamVR compositor">Headset view</button> <button data-view="headset" class="on" title="What the headset shows, via the SteamVR compositor">Headset view</button>
<button data-view="flat" title="gamescope's 2D layer: the desktop panel and Steam's flat UI">Desktop panel</button> <button data-view="flat" title="The app panel in use, flat: it stays still however the wearer looks around">Desktop</button>
</div> </div>
<select id="panelPick" class="small" hidden aria-label="Which panel to show"></select>
<div class="seg" id="eyeSeg" aria-label="Crop"> <div class="seg" id="eyeSeg" aria-label="Crop">
<button data-eye="left" class="on" title="One eye, like pointing a camera into one lens">Left eye</button> <button data-eye="left" class="on" title="One eye, like pointing a camera into one lens">Left eye</button>
<button data-eye="both" title="Both eyes side by side">Both eyes</button> <button data-eye="both" title="Both eyes side by side">Both eyes</button>
</div> </div>
<div class="spacer"></div> <div class="spacer"></div>
<button class="action" id="shotBtn">Capture</button> <span class="view-actions">
<button id="liveBtn" title="Keep updating: video of the headset view, or repeated captures of the desktop panel">Live</button> <button class="action" id="shotBtn">Capture</button>
<button id="saveBtn" disabled>Save</button> <button id="liveBtn" title="Keep updating, as video">Live</button>
<button id="ctrlBtn" title="Control the Frame by tapping or clicking on the view (C)">Control</button>
<button id="saveBtn" disabled>Save</button>
</span>
</div> </div>
<div class="viewer" id="viewer"> <div class="viewer" id="viewer">
<div class="empty" id="viewerEmpty"> <div class="empty" id="viewerEmpty">
@@ -500,6 +520,25 @@
<div class="asleep" id="asleep" hidden>The headset is showing a blank frame. It's probably asleep or not being worn. Put it on and capture again.</div> <div class="asleep" id="asleep" hidden>The headset is showing a blank frame. It's probably asleep or not being worn. Put it on and capture again.</div>
<div class="loading"><div class="spinner"></div></div> <div class="loading"><div class="spinner"></div></div>
</div> </div>
<div class="ctrl-status" id="ctrlBadge" role="status" hidden></div>
<div class="pad-keys ctrl-keys" id="ctrlKeys" hidden aria-label="Keys and clicks for the Frame">
<input id="ctrlType" type="text" placeholder="Type here to type on the Frame" autocomplete="off" autocapitalize="off"
autocorrect="off" spellcheck="false" enterkeyhint="send">
<span class="seg pad-clicks">
<button data-ctrl-click="left" title="Left click where the pointer is">Click</button>
<button data-ctrl-click="right" title="Right click where the pointer is">Right-click</button>
</span>
<span class="seg pad-special">
<button data-ctrl-key="1" title="Escape">Esc</button>
<button data-ctrl-key="15" title="Tab">Tab</button>
<button data-ctrl-key="105" title="Left arrow" aria-label="Left arrow">←</button>
<button data-ctrl-key="103" title="Up arrow" aria-label="Up arrow">↑</button>
<button data-ctrl-key="108" title="Down arrow" aria-label="Down arrow">↓</button>
<button data-ctrl-key="106" title="Right arrow" aria-label="Right arrow">→</button>
<button data-ctrl-key="14" title="Backspace" aria-label="Backspace">⌫</button>
<button data-ctrl-key="28" title="Enter">Enter</button>
</span>
</div>
<div class="hint" id="viewHint"></div> <div class="hint" id="viewHint"></div>
</section> </section>
@@ -537,6 +576,36 @@
</section> </section>
</div> </div>
<section class="panel" id="comfort" style="margin:20px 0;padding:16px">
<div class="shelf-head"><h2>Family and comfort</h2><span class="spacer"></span>
<button class="action small" id="castBtn">Cast headset view</button>
</div>
<p class="hint">Share this screen with people in the room. Casting shows everything the wearer sees, including private content.</p>
<form id="comfortForm">
<div class="row" style="flex-wrap:wrap;gap:12px">
<label>Session limit (minutes) <input id="sessionMinutes" type="number" min="1" max="240" value="30" required style="width:75px"></label>
<label>Break every <input id="breakMinutes" type="number" min="0" max="120" value="20" required style="width:75px"> minutes</label>
<label>Check in after <input id="stillMinutes" type="number" min="0" max="240" value="30" required style="width:75px"> active minutes</label>
</div>
<div class="row" style="flex-wrap:wrap;margin:14px 0;gap:14px">
<label><input id="batteryAlert" type="checkbox" checked> Alert on low battery</label>
<label><input id="heatAlert" type="checkbox" checked> Alert on overheating</label>
<button class="action small" id="sessionStart" type="submit">Start session</button>
<button class="small" id="sessionCancel" type="button" disabled>Cancel session</button>
<button class="small" id="testNotification" type="button">Enable / test notifications</button>
</div>
</form>
<p id="comfortStatus" role="status">Checking session…</p>
<p id="comfortNotificationStatus" class="hint" role="status" hidden></p>
<p class="hint">A one-minute warning, then Steam Home. Games stay running: save and pause first. Keep this app open and connected for notifications.</p>
<details class="hint"><summary>How sessions and alerts work</summary>
<p>This is a reminder, not a parental lock. The timer continues on the Frame if you disconnect; a headset restart cancels it. Cancel before changing settings. Set break/check-in to 0 to turn them off.</p>
<p>Battery, heat and check-in alerts appear on connected companions during a session. Headset warnings and break reminders continue without a companion. iOS may suspend phone notifications in the background. Breaks and check-ins count SteamVR activity, not confirmed wear time.</p>
</details>
<div id="comfortEvents" class="hint"></div>
<span id="comfortAnnouncement" class="sr-only" aria-live="polite"></span>
</section>
<section class="panel" id="pad"> <section class="panel" id="pad">
<div class="shelf-head"><h2>Keyboard and trackpad</h2><span class="sub" id="padState"></span><span class="spacer"></span> <div class="shelf-head"><h2>Keyboard and trackpad</h2><span class="sub" id="padState"></span><span class="spacer"></span>
<button class="small" id="padOn">Turn on</button> <button class="small" id="padOn">Turn on</button>
@@ -773,8 +842,17 @@
</div> </div>
</section> </section>
<section class="panel" id="panels">
<div class="shelf-head"><h2>Panel switcher</h2><span class="spacer"></span>
<button class="small" id="panelsRefresh">Refresh</button>
<button class="small action" id="panelsOpen">Open in headset</button></div>
<div class="list" id="panelList"><div class="sub">Refresh to see open panels.</div></div>
<div class="hint">Choose a panel to show it in SteamVR. The headset switcher stays available from Steam's dashboard.
Saved spatial layouts are not available yet.</div>
</section>
<section class="panel" id="macview" hidden> <section class="panel" id="macview" hidden>
<div class="shelf-head"><h2 id="mvTitle">Mac in the headset</h2><span class="count" id="mvCount"></span><span class="spacer"></span> <div class="shelf-head"><h2>Mac in the headset</h2><span class="count" id="mvCount"></span><span class="spacer"></span>
<select id="mvQuality" title="Picture quality and bandwidth"> <select id="mvQuality" title="Picture quality and bandwidth">
<option value="sharp">Sharp</option><option value="balanced" selected>Balanced</option> <option value="sharp">Sharp</option><option value="balanced" selected>Balanced</option>
<option value="light">Light</option><option value="compatible">Compatible (JPEG)</option> <option value="light">Light</option><option value="compatible">Compatible (JPEG)</option>
@@ -791,7 +869,7 @@
<button class="small danger" data-mv="stopall">Stop all</button> <button class="small danger" data-mv="stopall">Stop all</button>
</div> </div>
<div class="hint">Each window or screen becomes its own panel in the headset. Place it with the SteamVR dashboard <div class="hint">Each window or screen becomes its own panel in the headset. Place it with the SteamVR dashboard
(Float in World, Move, Size). Point and click with the laser, scroll with the thumbstick, and type on your computer’s keyboard.</div> (Float in World, Move, Size). Point and click with the laser, scroll with the thumbstick, and type on the Mac's keyboard.</div>
</section> </section>
<section class="panel" id="privacy"> <section class="panel" id="privacy">
@@ -943,9 +1021,9 @@ const QUICK = [["Remmina", "org.remmina.Remmina"], ["Moonlight", "com.moonlight_
const CDN = "https://cdn.cloudflare.steamstatic.com/steam/apps"; const CDN = "https://cdn.cloudflare.steamstatic.com/steam/apps";
const HINTS = { const HINTS = {
headset: "What the lenses show, composited by SteamVR: the room, floating panels, dashboard and controllers. Live streams it as video (one eye, about 30 fps); Capture takes a still of both eyes. Captures show everything on screen, including anything private.", headset: "What the lenses show, composited by SteamVR: the room, floating panels, dashboard and controllers. Live streams it as video (one eye, about 30 fps); Capture takes a still of both eyes. Captures show everything on screen, including anything private.",
flat: "gamescope's 2D layer: the desktop panel and Steam's flat UI, without the room or VR scene.", flat: "The app panel in use in the headset, flat and still: it doesn't move as the wearer looks around. Pick another panel to watch it. Control lets you tap or click right where you want on it.",
}; };
const SOURCE_LABEL = { steamvr: "Headset view", gamescope: "Desktop panel", shot: "Screenshot" }; const SOURCE_LABEL = { steamvr: "Headset view", gamescope: "Desktop panel", panel: "Desktop", shot: "Screenshot" };
let state = null, view = "headset", eye = "left", live = false, liveTimer = null, volTimer = null; let state = null, view = "headset", eye = "left", live = false, liveTimer = null, volTimer = null;
let lastImg = null, lastSource = null, lastShot = null, viewGen = 0; let lastImg = null, lastSource = null, lastShot = null, viewGen = 0;
@@ -1097,6 +1175,101 @@ function refresh() {
return refreshing; return refreshing;
} }
// The Frame owns the clock. Poll independently of status and the selected tab.
let comfortBusy = false, comfortSeen = new Set(), comfortSession = null, comfortHydrated = false;
async function localNotification(message, request = false) {
try {
if (window.frameApp?.notify) await window.frameApp.notify(message, request);
else {
if (!("Notification" in window)) throw new Error("This browser has no notifications; use the Frame Control app.");
const permission = request ? await Notification.requestPermission() : Notification.permission;
if (permission !== "granted") throw new Error("Notifications are off. Enable them in system settings.");
new Notification("Frame Control", {body: message});
}
$("comfortNotificationStatus").hidden = true;
} catch (e) {
$("comfortNotificationStatus").hidden = false;
$("comfortNotificationStatus").textContent = "Notifications aren't available on this device. Check system notification settings, then use Enable / test notifications. Headset reminders continue during an active session.";
throw e;
}
}
function renderComfort(s) {
const firstSnapshot = !comfortHydrated;
comfortHydrated = true;
$("sessionStart").disabled = !!s.active;
for (const id of ["sessionMinutes", "breakMinutes", "stillMinutes", "batteryAlert", "heatAlert"])
$(id).disabled = !!s.active;
$("sessionCancel").disabled = !s.active;
if (s.id !== comfortSession) {
comfortSession = s.id;
comfortSeen.clear();
if (s.options) for (const [key, value] of Object.entries(s.options)) {
const el = $(key === "minutes" ? "sessionMinutes" : key);
if (!el) continue;
if (el.type === "checkbox") el.checked = value; else el.value = value;
}
}
let statusText = s.error || (s.active
? `${Math.ceil(s.remaining / 60)} min until Steam Home · ${s.activity == null ? "activity unknown" : s.activity === 3 ? "headset in standby" : "monitoring"}`
: "No session running.");
if (s.active && s.unavailable?.length)
statusText += " · No readings: " + s.unavailable.join(", ");
if ($("comfortStatus").textContent !== statusText) $("comfortStatus").textContent = statusText;
let latest = null;
for (const e of s.events || []) {
if (comfortSeen.has(e.id)) continue;
comfortSeen.add(e.id);
// Historical events remain visible but never produce a burst on reconnect.
if (!firstSnapshot && s.time - e.time >= 0 && s.time - e.time < 30 && !["started", "cancelled"].includes(e.kind)) {
latest = e.message;
log(e.message); toast(e.message, e.kind === "error");
localNotification(e.message).catch(() => {}); // the notification status keeps the failure visible
}
}
// Keep only the server's bounded history; a new page seeds it without replay.
comfortSeen = new Set((s.events || []).map(e => e.id));
if (latest !== null) $("comfortAnnouncement").textContent = latest;
const history = (s.events || []).slice(-3).map(e => e.message).join(" · ");
if ($("comfortEvents").textContent !== history) $("comfortEvents").textContent = history;
}
async function pollComfort() {
if (!comfortBusy) {
comfortBusy = true;
try { renderComfort(await api("/api/comfort", {action: "status"})); }
catch (e) {
const message = "Session status unavailable: " + e.message;
if ($("comfortStatus").textContent !== message) $("comfortStatus").textContent = message;
}
finally { comfortBusy = false; }
}
setTimeout(pollComfort, 5000);
}
$("comfortForm").onsubmit = async e => {
e.preventDefault();
const result = await act("Start session", () => api("/api/comfort", {
action: "start", minutes: Number($("sessionMinutes").value), breakMinutes: Number($("breakMinutes").value),
stillMinutes: Number($("stillMinutes").value), batteryAlert: $("batteryAlert").checked, heatAlert: $("heatAlert").checked,
}), $("sessionStart"));
if (result) renderComfort(result);
};
$("sessionCancel").onclick = async () => {
const result = await act("Cancel session", () => api("/api/comfort", {action: "cancel"}), $("sessionCancel"));
if (result) renderComfort(result);
};
$("testNotification").onclick = () => act("Test notification", async () => {
await localNotification("Comfort notifications are enabled on this device.", true);
return {message: "Test notification sent. Check your device's notification settings if it didn't appear."};
});
$("castBtn").onclick = () => {
setView("headset");
if (!live) toggleLive(true);
$("viewer").scrollIntoView({behavior: "smooth", block: "center"});
// Fullscreen is a direct user gesture; iPhone falls back to its inline viewer.
$("viewer").requestFullscreen?.().catch(() => {});
};
pollComfort();
// ---- battery ---- // ---- battery ----
function battery(b, power) { function battery(b, power) {
const pct = b?.percent; const pct = b?.percent;
@@ -1205,8 +1378,12 @@ function setView(v) {
view = v; view = v;
document.querySelectorAll("[data-view]").forEach(b => b.classList.toggle("on", b.dataset.view === v)); document.querySelectorAll("[data-view]").forEach(b => b.classList.toggle("on", b.dataset.view === v));
$("eyeSeg").style.visibility = v === "headset" && !video.ctl ? "visible" : "hidden"; $("eyeSeg").style.visibility = v === "headset" && !video.ctl ? "visible" : "hidden";
$("eyeSeg").style.display = v === "flat" ? "none" : ""; // the Desktop view has no eyes to pick
$("viewHint").textContent = HINTS[v]; $("viewHint").textContent = HINTS[v];
if (live && changed) { toggleLive(false); toggleLive(true); } // video for the headset, captures for the panel $("panelPick").hidden = v !== "flat";
if (v === "flat") watchPanels(); else clearTimeout(desk.timer);
if (live && changed) { toggleLive(false); toggleLive(true); }
ctrlShow();
} }
function setEye(e) { function setEye(e) {
eye = e; eye = e;
@@ -1233,20 +1410,27 @@ function setZoom(z, cx, cy) {
$("zoomIn").onclick = () => setZoom(zoom * 1.5); $("zoomIn").onclick = () => setZoom(zoom * 1.5);
$("zoomOut").onclick = () => setZoom(zoom / 1.5); $("zoomOut").onclick = () => setZoom(zoom / 1.5);
$("zoomFit").onclick = () => setZoom(1); $("zoomFit").onclick = () => setZoom(1);
$("fullBtn").onclick = () => document.fullscreenElement ? document.exitFullscreen() : $("viewer").requestFullscreen(); // Full screen, or on phones (no element full screen there) the viewer filling the window.
$("fullBtn").onclick = () => {
if (document.fullscreenElement) return document.exitFullscreen();
if ($("viewer").requestFullscreen && !TOUCH) return $("viewer").requestFullscreen().catch(() => $("viewer").classList.toggle("immersive"));
$("viewer").classList.toggle("immersive");
setZoom(1);
};
document.addEventListener("fullscreenchange", () => setZoom(1)); document.addEventListener("fullscreenchange", () => setZoom(1));
$("viewer").addEventListener("wheel", e => { $("viewer").addEventListener("wheel", e => {
if ($("canvas").hidden) return; if ($("canvas").hidden || ctrl.on) return;
e.preventDefault(); e.preventDefault();
const v = $("viewer").getBoundingClientRect(); const v = $("viewer").getBoundingClientRect();
setZoom(zoom * Math.exp(-e.deltaY * 0.002), e.clientX - v.left, e.clientY - v.top); setZoom(zoom * Math.exp(-e.deltaY * 0.002), e.clientX - v.left, e.clientY - v.top);
}, { passive: false }); }, { passive: false });
$("canvas").addEventListener("dblclick", e => { $("canvas").addEventListener("dblclick", e => {
if (ctrl.on) return;
const v = $("viewer").getBoundingClientRect(); const v = $("viewer").getBoundingClientRect();
setZoom(zoom > 1 ? 1 : 2.5, e.clientX - v.left, e.clientY - v.top); setZoom(zoom > 1 ? 1 : 2.5, e.clientX - v.left, e.clientY - v.top);
}); });
$("canvas").addEventListener("pointerdown", e => { $("canvas").addEventListener("pointerdown", e => {
if (zoom === 1) return; if (zoom === 1 || ctrl.on) return;
const start = { x: e.clientX - panX, y: e.clientY - panY }; const start = { x: e.clientX - panX, y: e.clientY - panY };
$("viewer").classList.add("panning"); $("viewer").classList.add("panning");
$("canvas").setPointerCapture(e.pointerId); $("canvas").setPointerCapture(e.pointerId);
@@ -1256,7 +1440,10 @@ $("canvas").addEventListener("pointerdown", e => {
$("canvas").addEventListener("pointerup", up, { once: true }); $("canvas").addEventListener("pointerup", up, { once: true });
}); });
document.addEventListener("keydown", e => { document.addEventListener("keydown", e => {
if (page !== "home" || $("canvas").hidden || /INPUT|TEXTAREA|SELECT/.test(document.activeElement.tagName) || e.metaKey || e.ctrlKey) return; if (page !== "home" || /INPUT|TEXTAREA|SELECT/.test(document.activeElement.tagName) || e.metaKey || e.ctrlKey) return;
if (ctrl.on && document.activeElement === $("viewer")) return; // keys are going to the Frame
if (e.key === "c") return $("ctrlBtn").click();
if ($("canvas").hidden) return;
if (e.key === "+" || e.key === "=") setZoom(zoom * 1.5); if (e.key === "+" || e.key === "=") setZoom(zoom * 1.5);
else if (e.key === "-") setZoom(zoom / 1.5); else if (e.key === "-") setZoom(zoom / 1.5);
else if (e.key === "0") setZoom(1); else if (e.key === "0") setZoom(1);
@@ -1298,8 +1485,14 @@ async function capture() {
if (!live) $("viewer").classList.add("busy"); // no spinner flashing over a live stream if (!live) $("viewer").classList.add("busy"); // no spinner flashing over a live stream
let url = null; let url = null;
try { try {
const r = await fetch(view === "headset" ? "/api/screenshot?view=headset" : "/api/screenshot", let url_ = "/api/screenshot?view=headset";
{ headers: {"X-Frame-UI": UI_KEY} }); var shotPanel = null;
if (view === "flat") {
shotPanel = deskPanel() || (await loadPanels(), deskPanel());
if (!shotPanel) throw new Error("no app panel is open in the headset");
url_ = `/api/screenshot?view=panel&window=${shotPanel.window}&display=${encodeURIComponent(shotPanel.display)}`;
}
const r = await fetch(url_, { headers: {"X-Frame-UI": UI_KEY} });
if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`); if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`);
const source = r.headers.get("X-Capture-Source") || "gamescope"; const source = r.headers.get("X-Capture-Source") || "gamescope";
url = URL.createObjectURL(await r.blob()); url = URL.createObjectURL(await r.blob());
@@ -1308,6 +1501,9 @@ async function capture() {
if (gen !== viewGen) return true; if (gen !== viewGen) return true;
lastImg = img; lastSource = source; lastShot = null; lastImg = img; lastSource = source; lastShot = null;
draw(); draw();
desk.shown = source === "panel" ? panelKey(shotPanel) : null;
if (source === "panel") $("srcBadge").textContent = `Desktop · ${shotPanel?.name || "panel"}`;
ctrlShow();
$("stamp").hidden = false; $("stamp").textContent = new Date().toLocaleTimeString(); $("stamp").hidden = false; $("stamp").textContent = new Date().toLocaleTimeString();
$("saveBtn").disabled = false; $("saveBtn").disabled = false;
return true; return true;
@@ -1320,26 +1516,29 @@ async function capture() {
$("viewer").classList.remove("busy"); $("viewer").classList.remove("busy");
} }
} }
let liveFailures = 0; let liveFailures = 0, liveGen = 0;
async function liveLoop() { async function liveLoop(gen = liveGen) {
if (!live) return; if (!live || gen !== liveGen) return;
liveFailures = (await capture()) ? 0 : liveFailures + 1; const ok = await capture();
if (gen !== liveGen) return; // Live was restarted (say, as video) while this capture ran
liveFailures = ok ? 0 : liveFailures + 1;
if (liveFailures >= 5) { if (liveFailures >= 5) {
toggleLive(false); toggleLive(false);
toast("Live stopped after 5 failed captures. Is the headset on?", true); toast(view === "flat" ? "Live stopped: couldn't capture the panel." : "Live stopped after 5 failed captures. Is the headset on?", true);
return; return;
} }
if (live) liveTimer = setTimeout(liveLoop, view === "headset" ? 300 : 700); if (live) liveTimer = setTimeout(() => liveLoop(gen), view === "headset" ? 300 : 700);
} }
function toggleLive(on) { function toggleLive(on) {
live = on; live = on;
liveGen++;
liveFailures = 0; liveFailures = 0;
$("liveBtn").classList.toggle("on", live); $("liveBtn").classList.toggle("on", live);
$("liveBadge").hidden = !live; $("liveBadge").hidden = !live;
clearTimeout(liveTimer); clearTimeout(liveTimer);
stopVideo(); stopVideo();
if (!live) return; if (!live) return;
if (view === "headset" && "VideoDecoder" in window) { if ("VideoDecoder" in window && (view === "headset" || deskPanel())) {
const started = startVideo(); const started = startVideo();
const ctl = video.ctl; const ctl = video.ctl;
started.catch(e => { started.catch(e => {
@@ -1384,9 +1583,10 @@ async function startVideo() {
if (!shown++) { if (!shown++) {
viewGen++; // a capture still in flight mustn't replace the video viewGen++; // a capture still in flight mustn't replace the video
lastImg = null; lastSource = "video"; lastShot = null; lastImg = null; lastSource = "video"; lastShot = null;
desk.shown = panelKey(panel); ctrlShow();
$("viewer").classList.remove("busy"); $("viewer").classList.remove("busy");
c.hidden = false; $("viewerEmpty").hidden = true; $("zoombar").hidden = false; $("asleep").hidden = true; c.hidden = false; $("viewerEmpty").hidden = true; $("zoombar").hidden = false; $("asleep").hidden = true;
$("srcBadge").hidden = false; $("srcBadge").textContent = "Headset view · video"; $("srcBadge").hidden = false; $("srcBadge").textContent = `${video.label} · video`;
$("stamp").hidden = false; $("saveBtn").disabled = false; $("stamp").hidden = false; $("saveBtn").disabled = false;
} }
frames++; frames++;
@@ -1398,7 +1598,11 @@ async function startVideo() {
}, },
error: e => log("Video decoder: " + e.message, "e"), error: e => log("Video decoder: " + e.message, "e"),
}); });
const r = await fetch(`/api/stream?${STREAM_QUERY}`, { headers: {"X-Frame-UI": UI_KEY}, signal: ctl.signal }); const panel = view === "flat" ? deskPanel() : null;
desk.shown = null; // until this stream's first picture
video.label = panel ? `Desktop · ${panel.name || "panel"}` : "Headset view";
const query = panel ? `src=panel&window=${panel.window}&display=${encodeURIComponent(panel.display)}&${STREAM_QUERY}` : STREAM_QUERY;
const r = await fetch(`/api/stream?${query}`, { headers: {"X-Frame-UI": UI_KEY}, signal: ctl.signal });
if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`); if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `HTTP ${r.status}`);
const reader = r.body.getReader(); const reader = r.body.getReader();
let buf = new Uint8Array(0), scan = 0, auStart = -1; let buf = new Uint8Array(0), scan = 0, auStart = -1;
@@ -1443,7 +1647,7 @@ function stopVideo() {
if (video.dec && video.dec.state !== "closed") video.dec.close(); if (video.dec && video.dec.state !== "closed") video.dec.close();
video.ctl = video.dec = null; video.ctl = video.dec = null;
$("viewer").classList.remove("busy"); $("viewer").classList.remove("busy");
if (lastSource === "video") { $("srcBadge").textContent = "Headset view · video (stopped)"; $("stamp").hidden = true; } if (lastSource === "video") { $("srcBadge").textContent = `${video.label || "Headset view"} · video (stopped)`; $("stamp").hidden = true; }
$("eyeSeg").style.visibility = view === "headset" ? "visible" : "hidden"; $("eyeSeg").style.visibility = view === "headset" ? "visible" : "hidden";
} }
$("shotBtn").onclick = () => capture(); $("shotBtn").onclick = () => capture();
@@ -1463,6 +1667,421 @@ function download(blob, name) {
document.querySelectorAll("[data-view]").forEach(b => b.onclick = () => setView(b.dataset.view)); document.querySelectorAll("[data-view]").forEach(b => b.onclick = () => setView(b.dataset.view));
document.querySelectorAll("[data-eye]").forEach(b => b.onclick = () => setEye(b.dataset.eye)); document.querySelectorAll("[data-eye]").forEach(b => b.onclick = () => setEye(b.dataset.eye));
// ---- Desktop view: the app panel in use in the headset (or one picked to watch) ----
// gamescope says which panel has focus: the one the wearer last used. That's the one
// Control reaches, so the view follows it unless you pick another to watch.
// Panels are "display/window": window ids are per X display, and the Frame has two.
// desk.shown is the panel whose picture is on screen now; Control only acts on that one.
const desk = { panels: [], focus: null, pick: "", timer: null, loaded: false, shown: null };
const panelKey = p => p ? `${p.display}/${p.window}` : null;
function deskPanel() {
const want = desk.pick || desk.focus;
// Nothing in use and nothing picked: show the first panel rather than nothing.
return desk.panels.find(p => panelKey(p) === want) || (!desk.pick && !desk.focus && desk.panels[0]) || null;
}
async function loadPanels() {
try {
const before = panelKey(deskPanel());
const r = await api("/api/panels");
desk.panels = r.panels || []; desk.loaded = true;
desk.focus = r.focus ? `${r.focus_display}/${r.focus}` : null;
if (desk.pick && !desk.panels.some(p => panelKey(p) === desk.pick)) desk.pick = "";
renderPanelPick();
const now = panelKey(deskPanel());
if (view === "flat" && live && now !== before) { toggleLive(false); toggleLive(true); }
ctrlShow();
} catch (e) {
log("Couldn't list the headset's panels: " + e.message, "e");
}
}
function watchPanels() {
clearTimeout(desk.timer);
if (view !== "flat") return;
loadPanels().finally(() => { if (view === "flat") desk.timer = setTimeout(watchPanels, document.hidden ? 10000 : 3000); });
}
function renderPanelPick() {
const sel = $("panelPick");
sel.innerHTML = `<option value="">${desk.focus ? "Panel in use" : "Panel in use (none yet)"}</option>` +
desk.panels.map(p => `<option value="${esc(panelKey(p))}">${esc(p.name || "Untitled")}${p.focused ? " (in use)" : ""}</option>`).join("");
sel.value = desk.pick;
}
$("panelPick").onchange = () => {
desk.pick = $("panelPick").value;
ctrlShow();
if (live) { toggleLive(false); toggleLive(true); } else if (lastImg || lastSource) capture();
};
// ---- Control: tap or click on the view to use the Frame ----
// Through gamescope's own input (frame_touch.py). On the Desktop view a tap lands
// exactly where you tapped; on the headset view, which moves with the wearer's head,
// the view is a trackpad for the pointer instead.
const ctrl = { on: false, queue: [], sending: false, state: "off", message: "", move: null, rel: [0, 0], raf: 0,
held: new Set(), keys: new Set(), pointers: new Map(), g: null, retry: null };
const EVDEV = { Escape: 1, Minus: 12, Equal: 13, Backspace: 14, Tab: 15, BracketLeft: 26, BracketRight: 27, Enter: 28,
ControlLeft: 29, Semicolon: 39, Quote: 40, Backquote: 41, ShiftLeft: 42, Backslash: 43, Comma: 51, Period: 52,
Slash: 53, ShiftRight: 54, NumpadMultiply: 55, AltLeft: 56, Space: 57, CapsLock: 58, F11: 87, F12: 88,
IntlBackslash: 86, NumpadEnter: 96, ControlRight: 97, NumpadDivide: 98, AltRight: 100, Home: 102, ArrowUp: 103,
PageUp: 104, ArrowLeft: 105, ArrowRight: 106, End: 107, ArrowDown: 108, PageDown: 109, Insert: 110, Delete: 111,
MetaLeft: 125, MetaRight: 126, ContextMenu: 127, NumpadSubtract: 74, NumpadAdd: 78, NumpadDecimal: 83 };
[..."qwertyuiop"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 16 + i);
[..."asdfghjkl"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 30 + i);
[..."zxcvbnm"].forEach((c, i) => EVDEV["Key" + c.toUpperCase()] = 44 + i);
[..."1234567890"].forEach((c, i) => EVDEV["Digit" + c] = 2 + i);
for (let i = 1; i <= 10; i++) EVDEV["F" + i] = 58 + i;
[[7, 71], [8, 72], [9, 73], [4, 75], [5, 76], [6, 77], [1, 79], [2, 80], [3, 81], [0, 82]].forEach(([n, k]) => EVDEV["Numpad" + n] = k);
const CTRL_BUTTONS = ["left", "middle", "right"]; // MouseEvent.button 0, 1, 2
function ctrlMode() { return view === "flat" ? "abs" : "rel"; }
// Where taps go on the Desktop view, or why they can't.
function ctrlTarget() {
if (ctrlMode() === "rel") return { ok: true };
const p = deskPanel();
if (!desk.loaded) return { why: "Finding the headset's panels…" };
if (!p) return { why: "No app panel is open in the headset." };
if (!desk.focus) return { why: "Nobody's using a panel in the headset yet. Taps work once one is in use." };
if (panelKey(p) !== desk.focus) return { why: "Only the panel in use takes taps. Pick “Panel in use” to control it." };
if (desk.shown !== desk.focus) return { why: live ? "Waiting for the picture of the panel in use…"
: "The panel in use has changed. Press Capture or Live to see it first." };
return { ok: true, panel: p };
}
function ctrlShow() {
const v = $("viewer"), b = $("ctrlBadge");
$("ctrlBtn").classList.toggle("on", ctrl.on);
v.classList.toggle("control", ctrl.on);
v.classList.toggle("rel", ctrl.on && ctrlMode() === "rel");
$("ctrlKeys").hidden = !ctrl.on;
b.hidden = !ctrl.on;
if (!ctrl.on) return;
const t = ctrlTarget();
let text = ctrlMode() === "abs" ? `Tap or click to use ${t.panel ? t.panel.name || "the panel" : "the panel"}`
: "Drag to move the pointer, tap to click, two fingers to scroll";
let warn = false;
if (!t.ok) { text = t.why; warn = true; }
else if (ctrl.state === "error") { text = ctrl.message || "Control isn't working"; warn = true; }
else if (ctrl.state === "starting") text = "Connecting…";
b.textContent = text;
b.classList.toggle("warn", warn);
}
function ctrlToggle(on) {
if (!on) ctrlRelease();
ctrl.on = on;
if (on) {
setZoom(1);
ctrlPoll(0);
if (view === "flat" && !desk.loaded) watchPanels();
if (!TOUCH) $("viewer").focus({ preventScroll: true });
}
if (window.frameApp && window.frameApp.captureKeys) window.frameApp.captureKeys(on && document.activeElement === $("viewer"));
ctrlShow();
}
$("ctrlBtn").onclick = () => ctrlToggle(!ctrl.on);
// Until it's ready, check on the Frame side every half second (for up to 15 s).
async function ctrlPoll(n) {
if (!ctrl.on) return;
try {
const r = await api(n ? "/api/touch" : "/api/touch?start=1");
ctrl.state = r.state; ctrl.message = r.message || "";
} catch (e) { ctrl.state = "error"; ctrl.message = e.message; }
if (!ctrl.on) return; // turned off while connecting
ctrlShow();
if (ctrl.state !== "ready" && ctrl.state !== "error" && n < 30) return setTimeout(() => ctrlPoll(n + 1), 500);
if (ctrl.state === "ready") return ctrlFlush();
if (ctrl.state !== "error") {
ctrl.state = "error"; ctrl.message = "The Frame didn't answer. Turn Control off and on to try again.";
ctrlShow();
}
}
$("viewer").tabIndex = 0;
$("viewer").addEventListener("focus", () => { if (ctrl.on && window.frameApp?.captureKeys) window.frameApp.captureKeys(true); });
$("viewer").addEventListener("blur", () => { ctrlRelease(); if (window.frameApp?.captureKeys) window.frameApp.captureKeys(false); });
document.addEventListener("visibilitychange", () => { if (document.hidden) ctrlRelease(); });
window.addEventListener("blur", () => ctrlRelease()); // another app took the keyboard
const isMoveEvent = e => ("fx" in e || "dx" in e) && !("button" in e) && !("key" in e);
const isRelease = e => ("button" in e || "key" in e) && e.down === false;
// While it connects, keep clicks and keys, each with the last position before it, so a
// tap still lands where it was made; lone moves are stale by then.
function ctrlKeepable(events) {
const out = [];
let move = null;
for (const e of events) {
if (isMoveEvent(e)) { if ("fx" in e) move = e; continue; }
if (move && "button" in e && e.down !== false) out.push(move);
move = null;
out.push(e);
}
if (move) out.push(move); // the last position: a click queued after it needs it
return out;
}
function ctrlSend(events) {
ctrl.queue.push(...events.map(e => isMoveEvent(e) ? e : ctrlAim(e)).filter(Boolean));
// Keep order; if the Frame is slow to answer, drop pointer moves first, then the oldest
// presses and scrolls, never a release.
if (ctrl.queue.length > 300) {
const q = ctrlKeepable(ctrl.queue), keep = new Set();
q.forEach((e, i) => { if (isRelease(e) || i >= q.length - 200) keep.add(i); });
// A kept press keeps the position before it, so it lands where it was made.
for (const i of [...keep]) if (i > 0 && "button" in q[i] && q[i].down !== false && "fx" in q[i - 1]) keep.add(i - 1);
ctrl.queue = q.filter((e, i) => keep.has(i));
}
ctrlFlush();
}
async function ctrlFlush() {
if (ctrl.sending || !ctrl.queue.length) return;
// Backing off after a failure: new input waits for the same retry.
const wait = (ctrl.retryAt || 0) - Date.now();
if (wait > 0) {
clearTimeout(ctrl.retry);
ctrl.retry = setTimeout(ctrlFlush, wait);
return;
}
ctrl.sending = true;
// Up to 200 events, and at most about 100 characters of text: the Frame types a batch
// key by key before reading the next, and a release mustn't wait behind a long paste.
let n = 0, chars = 0;
while (n < ctrl.queue.length && n < 200 && (n === 0 || chars + (ctrl.queue[n].text?.length || 0) <= 100)) {
chars += ctrl.queue[n].text?.length || 0;
n++;
}
const batch = ctrl.queue.splice(0, n);
let again = 0;
try {
const r = await api("/api/touch", { events: batch });
ctrl.state = r.state; ctrl.message = r.message || "";
if (r.stale && typeof loadPanels === "function") loadPanels(); // focus moved on: catch up now
if (!r.sent && r.state === "error") {
// Broken: presses would be stale by the time it's back; releases still matter.
ctrl.queue.unshift(...batch.filter(isRelease));
again = 2000;
} else if (!r.sent) {
ctrl.queue.unshift(...ctrlKeepable(batch)); // connecting: send them once it's ready
again = 300;
}
} catch (e) {
ctrl.state = "error"; ctrl.message = e.message;
ctrl.queue.unshift(...batch.filter(isRelease));
again = 1000; // try the releases again in a moment, not in a tight loop
} finally {
ctrl.sending = false;
ctrlShow();
clearTimeout(ctrl.retry);
ctrl.retryAt = again ? Date.now() + again : 0;
if (again) { if (ctrl.queue.length) ctrl.retry = setTimeout(ctrlFlush, again); }
else if (ctrl.queue.length) ctrlFlush();
}
}
// Pointer moves: at most one per animation frame.
function ctrlMoveTo(fx, fy) {
const p = ctrlTarget().panel;
if (!p) return;
ctrl.move = { fx, fy, window: p.window, display: p.display };
ctrlSchedule();
}
function ctrlMoveBy(dx, dy) { ctrl.rel[0] += dx; ctrl.rel[1] += dy; ctrlSchedule(); }
function ctrlSchedule() { if (!ctrl.raf) ctrl.raf = requestAnimationFrame(ctrlFlushMoves); }
function ctrlFlushMoves() {
ctrl.raf = 0;
const out = [];
if (ctrl.move) { out.push(ctrl.move); ctrl.move = null; }
if (ctrl.rel[0] || ctrl.rel[1]) { out.push({ dx: Math.round(ctrl.rel[0]), dy: Math.round(ctrl.rel[1]) }); ctrl.rel = [0, 0]; }
if (out.length) ctrlSend(out);
}
// On the Desktop view, presses name the panel they're for, so the Frame drops them if
// focus has moved on (releases always go).
// A press with no panel to aim at (the Desktop view lost its target) isn't sent at all.
function ctrlAim(e) {
if (ctrlMode() !== "abs" || e.down === false) return e;
const p = ctrlTarget().panel;
return p ? { ...e, window: p.window, display: p.display } : null;
}
function ctrlButton(name, down) {
ctrlFlushMoves(); // the click lands where the pointer was sent
if (down) ctrl.held.add(name); else ctrl.held.delete(name);
ctrlSend([ctrlAim({ button: name, down })]);
}
function ctrlKeyEvent(code, down) {
ctrlFlushMoves();
if (down) ctrl.keys.add(code); else ctrl.keys.delete(code);
ctrlSend([ctrlAim({ key: code, down })]);
}
function ctrlClick(name = "left") { ctrlButton(name, true); ctrlButton(name, false); }
function ctrlRelease() {
for (const b of [...ctrl.held]) ctrlButton(b, false);
for (const k of [...ctrl.keys]) ctrlKeyEvent(k, false);
if (ctrl.g && ctrl.g.timer) clearTimeout(ctrl.g.timer);
ctrl.pointers.clear(); ctrl.g = null;
}
// Where a pointer event falls on the panel, as fractions (the canvas box includes any letterboxing).
function ctrlFraction(e) {
const c = $("canvas"), r = c.getBoundingClientRect();
const scale = Math.min(r.width / c.width, r.height / c.height);
const w = c.width * scale, h = c.height * scale;
const x = (e.clientX - r.left - (r.width - w) / 2) / w, y = (e.clientY - r.top - (r.height - h) / 2) / h;
return x >= 0 && x <= 1 && y >= 0 && y <= 1 ? [x, y] : null;
}
// Mouse (and pen) on a computer: the view is the Frame's screen.
function ctrlMouse(e) {
if (ctrlMode() === "abs") {
const f = ctrlFraction(e);
if (f) ctrlMoveTo(...f);
} else if (e.type === "pointermove" && (e.movementX || e.movementY)) {
ctrlMoveBy(e.movementX * 1.5, e.movementY * 1.5);
}
}
// Touch: tap to click, press and hold to right-click, two fingers to scroll. Dragging is a
// mouse drag on the Desktop view (sliders, selecting text) and moves the pointer on the headset view.
const TAP_MOVE = 8, HOLD_MS = 550, TOUCH_SPEED = 1.6, SCROLL_SPEED = 2;
function ctrlTouchDown(e) {
ctrl.pointers.set(e.pointerId, { x: e.clientX, y: e.clientY });
if (ctrl.pointers.size === 1) {
const g = ctrl.g = { kind: "maybe", x: e.clientX, y: e.clientY, e, target: ctrlAimedAt() };
g.timer = setTimeout(() => {
if (ctrl.g !== g || g.kind !== "maybe" || !ctrlSameTarget(g)) return;
g.kind = "held";
if (ctrlMode() === "abs") {
const f = ctrlFraction(g.e);
if (!f) return; // on the bars around the picture: nothing to press
ctrlMoveTo(...f);
}
ctrlClick("right");
navigator.vibrate?.(10);
}, HOLD_MS);
} else if (ctrl.pointers.size === 2) {
if (ctrl.g?.timer) clearTimeout(ctrl.g.timer);
if (ctrl.held.has("left")) ctrlButton("left", false);
ctrl.g = { kind: "scroll", ...centroid() };
}
}
// The panel a gesture started on; if that stops being the one in use, the gesture
// does nothing more (except let go of what it holds).
function ctrlAimedAt() { return ctrlMode() === "abs" ? panelKey(ctrlTarget().panel) : "headset"; }
function ctrlSameTarget(g) { return g.target && g.target === ctrlAimedAt(); }
function centroid() {
const ps = [...ctrl.pointers.values()];
return { x: ps.reduce((a, p) => a + p.x, 0) / ps.length, y: ps.reduce((a, p) => a + p.y, 0) / ps.length };
}
function ctrlTouchMove(e) {
const prev = ctrl.pointers.get(e.pointerId);
if (!prev || !ctrl.g) return;
const g = ctrl.g, dx = e.clientX - prev.x, dy = e.clientY - prev.y;
ctrl.pointers.set(e.pointerId, { x: e.clientX, y: e.clientY });
if (g.kind === "scroll") {
const c = centroid();
// Content follows the fingers, as on the phone itself.
ctrlSend([{ scroll: [-(c.x - g.x) * SCROLL_SPEED, -(c.y - g.y) * SCROLL_SPEED] }]);
g.x = c.x; g.y = c.y;
return;
}
if (g.kind === "maybe" && Math.hypot(e.clientX - g.x, e.clientY - g.y) > TAP_MOVE) {
clearTimeout(g.timer);
if (!ctrlSameTarget(g)) { g.kind = "done"; return; }
if (ctrlMode() === "abs") {
const f = ctrlFraction(g.e);
if (f) { ctrlMoveTo(...f); ctrlButton("left", true); }
g.kind = "drag";
} else g.kind = "move";
}
if (g.kind === "drag") { const f = ctrlFraction(e); if (f) ctrlMoveTo(...f); }
else if (g.kind === "move") ctrlMoveBy(dx * TOUCH_SPEED, dy * TOUCH_SPEED);
}
function ctrlTouchUp(e) {
const g = ctrl.g;
ctrl.pointers.delete(e.pointerId);
if (!g) return;
if (g.kind === "scroll") g.kind = "done"; // the finger left behind doesn't jump the scroll or drag
if (g.kind === "maybe" && !ctrlSameTarget(g)) g.kind = "done";
if (g.kind === "maybe" && ctrl.pointers.size === 0) {
clearTimeout(g.timer);
if (ctrlMode() === "abs") { const f = ctrlFraction(g.e); if (!f) { ctrl.g = null; return; } ctrlMoveTo(...f); }
ctrlClick("left");
} else if (g.kind === "drag") ctrlButton("left", false);
if (ctrl.pointers.size === 0) ctrl.g = null;
}
// The system took the touch (a swipe home, a call): no tap, and let go of anything held.
function ctrlTouchCancel(e) {
ctrl.pointers.delete(e.pointerId);
if (ctrl.g?.timer) clearTimeout(ctrl.g.timer);
if (ctrl.held.has("left")) ctrlButton("left", false);
if (ctrl.g) ctrl.g.kind = "done";
if (ctrl.pointers.size === 0) ctrl.g = null;
}
const canvasEl = $("canvas");
canvasEl.addEventListener("pointerdown", e => {
if (!ctrl.on) return;
e.preventDefault();
$("viewer").focus({ preventScroll: true });
try { canvasEl.setPointerCapture(e.pointerId); } catch { /* a pointer that's already gone */ }
if (!ctrlTarget().ok) return ctrlShow();
if (e.pointerType === "touch") return ctrlTouchDown(e);
if (ctrlMode() === "abs" && !ctrlFraction(e)) return; // on the bars around the picture
ctrlMouse(e);
ctrlButton(CTRL_BUTTONS[e.button] || "left", true);
});
canvasEl.addEventListener("pointermove", e => {
if (!ctrl.on || !ctrlTarget().ok) return;
if (e.pointerType === "touch") return ctrlTouchMove(e);
ctrlMouse(e);
});
for (const type of ["pointerup", "pointercancel"]) canvasEl.addEventListener(type, e => {
if (!ctrl.on) return;
if (e.pointerType === "touch") return type === "pointercancel" ? ctrlTouchCancel(e) : ctrlTouchUp(e);
if (type === "pointercancel") return [...ctrl.held].forEach(b => ctrlButton(b, false)); // no button is named
const name = CTRL_BUTTONS[e.button] || "left";
if (ctrl.held.has(name)) ctrlButton(name, false);
});
canvasEl.addEventListener("contextmenu", e => { if (ctrl.on) e.preventDefault(); });
$("viewer").addEventListener("wheel", e => {
if (!ctrl.on || !ctrlTarget().ok) return;
e.preventDefault();
const unit = e.deltaMode === 1 ? 40 : e.deltaMode === 2 ? 800 : 1;
ctrlSend([{ scroll: [e.deltaX * unit, e.deltaY * unit] }]);
}, { passive: false });
// Keys while the view has focus go to the Frame (⌘ as Ctrl on a Mac); click elsewhere to stop.
function ctrlKey(e, down) {
if (!ctrl.on || document.activeElement !== $("viewer") || (down && !ctrlTarget().ok)) return;
let code = EVDEV[e.code];
if (IS_MAC && (e.code === "MetaLeft" || e.code === "MetaRight")) code = 29;
if (!code) return;
e.preventDefault();
e.stopPropagation(); // it's for the Frame, not Frame Control's own shortcuts
if (down && e.repeat && [29, 42, 54, 56, 97, 100, 125, 126].includes(code)) return;
if (!down && !ctrl.keys.has(code)) return;
ctrlKeyEvent(code, down);
}
$("viewer").addEventListener("keydown", e => ctrlKey(e, true));
$("viewer").addEventListener("keyup", e => ctrlKey(e, false));
// Phone keyboard: the field types plain text; the buttons send single keys and clicks.
function ctrlTap(code) { ctrlKeyEvent(code, true); ctrlKeyEvent(code, false); }
let ctrlWarned = false;
function ctrlText(text) {
const plain = text.replace(/[^\x20-\x7e\n\t]/g, "");
if (plain !== text && !ctrlWarned) {
ctrlWarned = true;
toast("Control types plain letters, numbers and symbols. For accents and emoji, use Keyboard and trackpad below.");
}
// Small pieces: the Frame types them key by key, and a release queued behind mustn't wait long.
for (const part of plain.match(/[\s\S]{1,100}/g) || []) ctrlSend([{ text: part }]);
}
$("ctrlType").addEventListener("beforeinput", e => {
if (e.isComposing) return;
e.preventDefault();
if (!ctrlTarget().ok) return ctrlShow();
if (e.inputType === "insertLineBreak" || e.inputType === "insertParagraph") return ctrlTap(28);
if (e.inputType === "deleteContentBackward") return ctrlTap(14);
if (e.inputType === "deleteContentForward") return ctrlTap(111);
const text = e.data ?? e.dataTransfer?.getData("text/plain") ?? "";
if (text) ctrlText(text);
});
$("ctrlType").addEventListener("compositionend", e => { if (e.data) ctrlText(e.data); $("ctrlType").value = ""; });
$("ctrlKeys").addEventListener("click", e => {
const k = e.target.closest("[data-ctrl-key]"), c = e.target.closest("[data-ctrl-click]");
if (!ctrlTarget().ok) return ctrlShow();
if (k) ctrlTap(+k.dataset.ctrlKey);
if (c) ctrlClick(c.dataset.ctrlClick);
});
// ---- controls ---- // ---- controls ----
$("refreshAll").onclick = () => { refresh(); loadTitles(); }; // titles too: the Frame may have been asleep at start $("refreshAll").onclick = () => { refresh(); loadTitles(); }; // titles too: the Frame may have been asleep at start
document.addEventListener("keydown", e => { document.addEventListener("keydown", e => {
@@ -2391,6 +3010,7 @@ async function openShot(s) {
await new Promise((ok, bad) => { img.onload = ok; img.onerror = () => bad(new Error("not an image")); img.src = url; }); await new Promise((ok, bad) => { img.onload = ok; img.onerror = () => bad(new Error("not an image")); img.src = url; });
if (gen !== viewGen) return; if (gen !== viewGen) return;
lastImg = img; lastSource = "shot"; lastShot = { blob, file: s.file }; lastImg = img; lastSource = "shot"; lastShot = { blob, file: s.file };
desk.shown = null; // a screenshot, not a panel: Control has nothing to act on here
draw(); draw();
$("srcBadge").textContent = `Screenshot · ${shotApp(s.appid)}`; $("srcBadge").textContent = `Screenshot · ${shotApp(s.appid)}`;
$("stamp").hidden = false; $("stamp").textContent = new Date(s.time * 1000).toLocaleString(); $("stamp").hidden = false; $("stamp").textContent = new Date(s.time * 1000).toLocaleString();
@@ -2423,6 +3043,27 @@ $("shotsRefresh").onclick = loadShots;
$("shotsSaveNew").onclick = e => saveShots(shots.list.filter(s => !s.saved), e.currentTarget); $("shotsSaveNew").onclick = e => saveShots(shots.list.filter(s => !s.saved), e.currentTarget);
$("shotsFolder").onclick = e => act($("shotsFolder").textContent, () => api("/api/open", { what: "shots" }), e.currentTarget); $("shotsFolder").onclick = e => act($("shotsFolder").textContent, () => api("/api/open", { what: "shots" }), e.currentTarget);
// ---- Panel switcher: our UI over SteamVR's panel API ----
let panelSeq = 0;
async function loadPanels() {
const seq = ++panelSeq;
try {
const s = await api("/api/panels", {action: "list"});
if (seq !== panelSeq) return;
$("panelList").innerHTML = s.panels.length ? s.panels.map(p =>
`<div class="item"><div class="grow"><div class="t">${esc(p.title)}</div></div>
<button class="small" data-panel="${esc(p.key)}">Show</button></div>`).join("")
: '<div class="sub">No open panels.</div>';
} catch (e) { if (seq === panelSeq) failed($("panelList"), e); }
}
$("panelsRefresh").onclick = loadPanels;
$("panelsOpen").onclick = e => act("Open panel switcher in headset",
() => api("/api/panels", {action: "open"}), e.currentTarget);
$("panelList").onclick = e => {
const b = e.target.closest("[data-panel]"); if (!b) return;
act("Show panel", () => api("/api/panels", {action: "focus", key: b.dataset.panel}), b);
};
// ---- Mac in the headset: windows and displays as panels (ui/frame_macview.py) ---- // ---- Mac in the headset: windows and displays as panels (ui/frame_macview.py) ----
let mvSeq = 0, mvTimer = 0; let mvSeq = 0, mvTimer = 0;
// streaming: the src it's shown as (a window may be "separate:<id>"), or "". // streaming: the src it's shown as (a window may be "separate:<id>"), or "".
@@ -2451,16 +3092,8 @@ async function loadMacView(restart) {
if (seq !== mvSeq) return; if (seq !== mvSeq) return;
const box = $("macview"); const box = $("macview");
if (err) { box.hidden = false; return failed($("mvWindows"), err); } if (err) { box.hidden = false; return failed($("mvWindows"), err); }
if (!s.available) { if (!s.available) { box.hidden = true; return; }
box.hidden = !s.host;
if (s.host) { $("mvTitle").textContent = "PC in the headset"; $("mvWindows").textContent = s.reason; }
return;
}
box.hidden = false; box.hidden = false;
const pc = s.host && s.host !== "mac";
$("mvTitle").textContent = pc ? "PC in the headset" : "Mac in the headset";
$("mvSeparate").closest("label").hidden = pc;
$("mvSeparate").disabled = pc;
const live = new Set((s.streams || []).map(x => x.src)); const live = new Set((s.streams || []).map(x => x.src));
const byStream = src => (s.streams || []).find(x => x.src === src); const byStream = src => (s.streams || []).find(x => x.src === src);
$("mvCount").textContent = live.size ? `${live.size} showing${s.route === "usb" ? " · over USB-C" : ""}` : ""; $("mvCount").textContent = live.size ? `${live.size} showing${s.route === "usb" ? " · over USB-C" : ""}` : "";
@@ -2470,12 +3103,6 @@ async function loadMacView(restart) {
$("mvPerm").hidden = !perm.length; $("mvPerm").hidden = !perm.length;
$("mvPerm").innerHTML = perm.join(" ") + `<div class="row"><button class="small action" data-mv="perm">Allow…</button> $("mvPerm").innerHTML = perm.join(" ") + `<div class="row"><button class="small action" data-mv="perm">Allow…</button>
<span class="sub">Then press Refresh. macOS may ask you to reopen Frame Control.</span></div>`; <span class="sub">Then press Refresh. macOS may ask you to reopen Frame Control.</span></div>`;
if (pc) {
$("mvPerm").hidden = s.host !== "linux";
$("mvPerm").innerHTML = `<button class="small action" data-mv="perm" ${s.selecting ? "disabled" : ""}>${s.selecting ? "Choose on your computer…" : "Choose a window or screen…"}</button>
<span class="sub">Your desktop asks what to share and whether to allow pointer and keyboard control.</span>` +
(s.selectionError ? `<div class="sub">${esc(s.selectionError)}</div>` : "");
}
const liveAs = src => live.has(src) ? src : src.startsWith("window:") && live.has("separate:" + src.slice(7)) ? "separate:" + src.slice(7) : ""; const liveAs = src => live.has(src) ? src : src.startsWith("window:") && live.has("separate:" + src.slice(7)) ? "separate:" + src.slice(7) : "";
$("mvDisplays").innerHTML = (s.displays || []).map(d => $("mvDisplays").innerHTML = (s.displays || []).map(d =>
mvRow(d.src, `Whole screen: ${d.name}`, `${d.w}×${d.h}${d.main ? " · main display" : ""}`, d.w, d.h, liveAs(d.src), byStream(liveAs(d.src)))).join(""); mvRow(d.src, `Whole screen: ${d.name}`, `${d.w}×${d.h}${d.main ? " · main display" : ""}`, d.w, d.h, liveAs(d.src), byStream(liveAs(d.src)))).join("");
@@ -2483,11 +3110,11 @@ async function loadMacView(restart) {
$("mvWinCount").textContent = wins.length ? `${wins.length}` : ""; $("mvWinCount").textContent = wins.length ? `${wins.length}` : "";
$("mvWindows").innerHTML = !s.screen ? `<div class="sub">Windows appear here once Screen Recording is allowed.</div>` $("mvWindows").innerHTML = !s.screen ? `<div class="sub">Windows appear here once Screen Recording is allowed.</div>`
: wins.length ? wins.map(w => mvRow(w.src, w.title || w.app, `${w.title ? w.app + " · " : ""}${w.w}×${w.h}`, w.w, w.h, liveAs(w.src), byStream(liveAs(w.src)))).join("") : wins.length ? wins.map(w => mvRow(w.src, w.title || w.app, `${w.title ? w.app + " · " : ""}${w.w}×${w.h}`, w.w, w.h, liveAs(w.src), byStream(liveAs(w.src)))).join("")
: `<div class="sub">No shared windows yet.</div>`; : `<div class="sub">No windows open on this Mac's current desktop.</div>`;
// Keep the numbers fresh while something is in the headset and the card is on screen. // Keep the numbers fresh while something is in the headset and the card is on screen.
clearTimeout(mvTimer); clearTimeout(mvTimer);
const tick = () => { mvTimer = document.hidden || !box.offsetParent ? setTimeout(tick, 3000) : (loadMacView(), 0); }; const tick = () => { mvTimer = document.hidden || !box.offsetParent ? setTimeout(tick, 3000) : (loadMacView(), 0); };
if (live.size || s.selecting) mvTimer = setTimeout(tick, 3000); if (live.size) mvTimer = setTimeout(tick, 3000);
} }
$("mvRefresh").onclick = () => loadMacView(true); $("mvRefresh").onclick = () => loadMacView(true);
$("mvSeparate").checked = localStorage.getItem("mvSeparate") !== "0"; $("mvSeparate").checked = localStorage.getItem("mvSeparate") !== "0";
@@ -2496,7 +3123,7 @@ $("macview").onclick = async e => {
const b = e.target.closest("[data-mv]"); if (!b) return; const b = e.target.closest("[data-mv]"); if (!b) return;
const src = b.dataset.src; const src = b.dataset.src;
if (b.dataset.mv === "show") { if (b.dataset.mv === "show") {
const as = !$("mvSeparate").disabled && $("mvSeparate").checked && src.startsWith("window:") ? "separate:" + src.slice(7) : src; const as = $("mvSeparate").checked && src.startsWith("window:") ? "separate:" + src.slice(7) : src;
await act(`Show ${b.dataset.title} in the headset`, async () => { await act(`Show ${b.dataset.title} in the headset`, async () => {
const r = await api("/api/macview", { action: "show", src: as, quality: $("mvQuality").value, const r = await api("/api/macview", { action: "show", src: as, quality: $("mvQuality").value,
w: +b.dataset.w || undefined, h: +b.dataset.h || undefined }); w: +b.dataset.w || undefined, h: +b.dataset.h || undefined });
@@ -2507,7 +3134,7 @@ $("macview").onclick = async e => {
} else if (b.dataset.mv === "stopall") { } else if (b.dataset.mv === "stopall") {
await act("Stop everything in the headset", () => api("/api/macview", { action: "stop" }), b); await act("Stop everything in the headset", () => api("/api/macview", { action: "stop" }), b);
} else if (b.dataset.mv === "perm") { } else if (b.dataset.mv === "perm") {
await act("Choose what to share", () => api("/api/macview", { action: "permissions" }), b); await act("Ask macOS for permission", () => api("/api/macview", { action: "permissions" }), b);
} }
setTimeout(loadMacView, 800); setTimeout(loadMacView, 800);
}; };
@@ -2515,7 +3142,7 @@ $("macview").onclick = async e => {
// ---- pages: #home, #games, #android, #tools (older section links still work) ---- // ---- pages: #home, #games, #android, #tools (older section links still work) ----
const PAGES = ["home", "games", "android", "tools"]; const PAGES = ["home", "games", "android", "tools"];
const SECTION_PAGE = { view: "home", device: "home", shots: "home", library: "games", sideloaded: "games", getgames: "games", const SECTION_PAGE = { view: "home", device: "home", shots: "home", library: "games", sideloaded: "games", getgames: "games",
display: "android", transfer: "tools", apps: "tools", power: "tools", macview: "tools", privacy: "tools" }; display: "android", transfer: "tools", apps: "tools", power: "tools", macview: "tools", privacy: "tools", panels: "tools" };
let page = "home"; let page = "home";
function showPage() { function showPage() {
const id = location.hash.slice(1); const id = location.hash.slice(1);
@@ -2527,7 +3154,7 @@ function showPage() {
document.title = page === "home" ? "Frame Control" : `${page[0].toUpperCase() + page.slice(1)} · Frame Control`; document.title = page === "home" ? "Frame Control" : `${page[0].toUpperCase() + page.slice(1)} · Frame Control`;
const section = !PAGES.includes(id) && id && $(id); const section = !PAGES.includes(id) && id && $(id);
if (section) section.scrollIntoView(); else window.scrollTo(0, 0); if (section) section.scrollIntoView(); else window.scrollTo(0, 0);
if (page === "tools") loadMacView(); if (page === "tools") { loadMacView(); loadPanels(); }
} }
window.addEventListener("hashchange", showPage); window.addEventListener("hashchange", showPage);
// While a text field has focus, phones hide the bottom tab bar (see body.typing in the CSS). // While a text field has focus, phones hide the bottom tab bar (see body.typing in the CSS).
+5 -5
View File
@@ -22,7 +22,7 @@
</head> </head>
<body> <body>
<canvas id="c" width="16" height="9"></canvas> <canvas id="c" width="16" height="9"></canvas>
<div id="note">Connecting to your computer…</div> <div id="note">Connecting to the Mac…</div>
<div id="hud" hidden></div> <div id="hud" hidden></div>
<script> <script>
"use strict"; "use strict";
@@ -243,14 +243,14 @@ async function connect() {
// Refused before ever getting a key: the ticket expired or was revoked, // Refused before ever getting a key: the ticket expired or was revoked,
// and retrying can't help. // and retrying can't help.
if (!reconnectKey && ++failedStarts >= 3) { if (!reconnectKey && ++failedStarts >= 3) {
show("This view has expired. Press Show in Frame Control on your computer to open it again."); show("This view has expired. Press Show in Frame Control on the Mac to open it again.");
return; return;
} }
hideNoteOnFrame = true; hideNoteOnFrame = true;
// The Mac may be asleep or the tunnel restarting: keep trying. // The Mac may be asleep or the tunnel restarting: keep trying.
retry = Math.min(retry + 1, 6); retry = Math.min(retry + 1, 6);
// Keep the Mac's reason (a missing permission, a closed window) on screen. // Keep the Mac's reason (a missing permission, a closed window) on screen.
show(lastError ? `${lastError} Retrying…` : "Lost the computer. Reconnecting…"); show(lastError ? `${lastError} Retrying…` : "Lost the Mac. Reconnecting…");
setTimeout(connect, 500 * 2 ** retry); setTimeout(connect, 500 * 2 ** retry);
}; };
} }
@@ -269,7 +269,7 @@ function control(m) {
document.title = tag ? `${name} [${tag}]` : name; document.title = tag ? `${name} [${tag}]` : name;
if (!m.input) { if (!m.input) {
hideNoteOnFrame = false; // a warning, not a connection notice: let it stay its 8 s hideNoteOnFrame = false; // a warning, not a connection notice: let it stay its 8 s
show(m.inputMessage || "Clicks and keys need Accessibility permission on the Mac (Frame Control asks for it).", 8000); show("Clicks and keys need Accessibility permission on the Mac (Frame Control asks for it).", 8000);
} }
} else if (m.t === "error") { } else if (m.t === "error") {
stats.errors.push(m.message); stats.errors.push(m.message);
@@ -391,7 +391,7 @@ addEventListener("keydown", e => onKey(e, true));
addEventListener("keyup", e => onKey(e, false)); addEventListener("keyup", e => onKey(e, false));
addEventListener("blur", () => send({ t: "release" })); addEventListener("blur", () => send({ t: "release" }));
show("Connecting to your computer…"); show("Connecting to the Mac…");
connect(); connect();
</script> </script>
</body> </body>
+196 -11
View File
@@ -44,10 +44,11 @@ import frame_assistant # noqa: E402
import frame_android # noqa: E402 import frame_android # noqa: E402
import frame_apk_versions # noqa: E402 import frame_apk_versions # noqa: E402
import frame_catalog # noqa: E402 import frame_catalog # noqa: E402
import frame_comfort # noqa: E402
import frame_host # noqa: E402 import frame_host # noqa: E402
import frame_macview # noqa: E402 import frame_macview # noqa: E402
import frame_pcview # noqa: E402
import frame_media # noqa: E402 import frame_media # noqa: E402
import frame_panels # noqa: E402
import frame_report # noqa: E402 import frame_report # noqa: E402
import frame_store # noqa: E402 import frame_store # noqa: E402
import frame_telemetry # noqa: E402 import frame_telemetry # noqa: E402
@@ -271,6 +272,37 @@ def status(_body):
return s return s
def comfort(body):
try:
frame_comfort.validate(body)
except ValueError as e:
raise Failure(str(e), 400)
# Content-addressed, user-only helper bundle. Desktop and phone use the same
# on-headset state/lock; no listener, service registration or third-party app.
import hashlib
files = {name: (HERE / name).read_text() for name in
("frame_comfort.py", "frame_status.py", "frame_steam.py")}
version = hashlib.sha256(json.dumps(files, sort_keys=True).encode()).hexdigest()[:16]
script = """import json, os, pathlib, subprocess, sys
os.umask(0o077)
files = %r
root = pathlib.Path.home() / '.cache/frame-control/comfort' / %r
root.mkdir(parents=True, exist_ok=True)
for name, source in files.items():
path = root / name
if not path.exists():
tmp = root / (name + '.' + str(os.getpid()))
tmp.write_text(source)
tmp.replace(path)
r = subprocess.run([sys.executable, str(root / 'frame_comfort.py'), %r], capture_output=True, text=True)
print(r.stdout, end='')
""" % (files, version, json.dumps(body))
out = json.loads(ssh("python3 -", stdin=script, timeout=65))
if out.get("error") and "active" not in out:
raise Failure(out["error"], 409)
return out
def headset_view(): def headset_view():
"""Both eyes as SteamVR composites them (see frame_vrshot.py); PNG bytes.""" """Both eyes as SteamVR composites them (see frame_vrshot.py); PNG bytes."""
# `timeout`: VR_Init can block if SteamVR is restarting. # `timeout`: VR_Init can block if SteamVR is restarting.
@@ -401,6 +433,7 @@ _stream_proc = None
def stream_command(query): def stream_command(query):
"""The ffmpeg that streams H.264: the headset view, or one panel's own window (src=panel)."""
q = parse_qs(query) q = parse_qs(query)
try: try:
height = int((q.get("h") or ["720"])[0]) height = int((q.get("h") or ["720"])[0])
@@ -410,6 +443,16 @@ def stream_command(query):
if height not in STREAM_HEIGHTS or fps not in STREAM_FPS: if height not in STREAM_HEIGHTS or fps not in STREAM_FPS:
raise Failure(f"h must be one of {STREAM_HEIGHTS} and fps one of {STREAM_FPS}", 400) raise Failure(f"h must be one of {STREAM_HEIGHTS} and fps one of {STREAM_FPS}", 400)
rate = 3 if height == 720 else 6 # Mbit/s rate = 3 if height == 720 else 6 # Mbit/s
if q.get("src") == ["panel"]:
# The panel's own pixels (x11grab of its window: gamescope keeps them), fitted
# to the height asked for. It stays still however the wearer moves their head.
window, display = panel_target(q)
return (f"DISPLAY={display} ffmpeg -hide_banner -loglevel error -nostdin -f x11grab -framerate {fps} "
f"-window_id {window} -i {display} "
f"-vf \"scale=-2:'trunc(min({height},ih)/2)*2',format=yuv420p\" -c:v libx264 -preset ultrafast "
f"-tune zerolatency -g {fps * 2} -bf 0 -b:v {rate}M -maxrate {rate}M -bufsize {rate // 2 or 1}M "
f"-x264-params aud=1:repeat-headers=1 -f h264 - & p=$!; "
f"exec >&-; cat >/dev/null; kill $p 2>/dev/null; wait $p")
return (f"[ -e {STREAM_DEVICE} ] || {{ echo 'No headset view device ({STREAM_DEVICE}). Is SteamVR running?' >&2; exit 3; }}; " return (f"[ -e {STREAM_DEVICE} ] || {{ echo 'No headset view device ({STREAM_DEVICE}). Is SteamVR running?' >&2; exit 3; }}; "
f"ffmpeg -hide_banner -loglevel error -nostdin -f v4l2 -video_size 1920x1080 -i {STREAM_DEVICE} " f"ffmpeg -hide_banner -loglevel error -nostdin -f v4l2 -video_size 1920x1080 -i {STREAM_DEVICE} "
f"-vf fps={fps},scale=-2:{height},format=yuv420p -c:v libx264 -preset ultrafast -tune zerolatency " f"-vf fps={fps},scale=-2:{height},format=yuv420p -c:v libx264 -preset ultrafast -tune zerolatency "
@@ -629,7 +672,7 @@ class InputAgent:
""" """
def __init__(self, source=HERE / "frame_input_agent.py", packages=None): def __init__(self, source=HERE / "frame_input_agent.py", packages=None):
self.source, self.proc, self.lock = source, None, threading.Lock() self.source, self.proc, self.lock, self.write_lock = source, None, threading.Lock(), threading.Lock()
self.packages = kdeconnect_packages() if packages is None else packages self.packages = kdeconnect_packages() if packages is None else packages
# generation counts stop()s; launching is the generation a launch is under way for. # generation counts stop()s; launching is the generation a launch is under way for.
self.status, self.launching, self.generation = {"state": "off"}, None, 0 self.status, self.launching, self.generation = {"state": "off"}, None, 0
@@ -802,8 +845,10 @@ class InputAgent:
self.start() self.start()
elif ready and events: elif ready and events:
try: try:
proc.stdin.write((json.dumps(events) + "\n").encode()) # One writer at a time: two devices sending at once mustn't tear a line.
proc.stdin.flush() with self.write_lock:
proc.stdin.write((json.dumps(events) + "\n").encode())
proc.stdin.flush()
sent = True sent = True
except (BrokenPipeError, OSError, ValueError): except (BrokenPipeError, OSError, ValueError):
pass # _watch reports how it ended pass # _watch reports how it ended
@@ -843,6 +888,115 @@ def remote_input(body):
return _input.send([input_event(e) for e in events]) return _input.send([input_event(e) for e in events])
# ---- touch: the headset's panels, through gamescope's own input (frame_touch.py) ----
PANEL_WINDOW = re.compile(r"^\d{1,10}$")
PANEL_DISPLAY = re.compile(r"^:\d{1,2}$")
TOUCH_BUTTONS = ("left", "right", "middle")
def panels():
"""The headset's app panels (window, display, name, size) and which has focus."""
return json.loads(ssh("python3 - panels", stdin=(HERE / "frame_touch.py").read_text(), timeout=20))
def panel_target(q):
window, display = (q.get("window") or [""])[0], (q.get("display") or [""])[0]
if not PANEL_WINDOW.match(window) or not PANEL_DISPLAY.match(display):
raise Failure("window must be a window id and display an X display such as :1", 400)
return window, display
def panel_capture(query):
"""One frame of a panel's own window, as PNG (its pixels, without the room around it)."""
window, display = panel_target(parse_qs(query))
return ssh(f"DISPLAY={display} timeout 10 ffmpeg -hide_banner -loglevel error -nostdin -f x11grab "
f"-window_id {window} -i {display} -frames:v 1 -f image2pipe -c:v png -", timeout=20, text=False)
def touch_event(event):
"""A frame_touch.py event with only the fields it knows, in range."""
if not isinstance(event, dict):
raise Failure("each touch event must be an object", 400)
def num(name, limit):
value = event.get(name)
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
raise Failure(f"{name} must be a number", 400)
return max(-limit, min(limit, round(float(value), 4)))
out = {}
if "fx" in event or "fy" in event:
if "window" not in event:
raise Failure("a position needs the panel's window and display", 400)
out.update(fx=num("fx", 1), fy=num("fy", 1))
# Any event can name the panel it's meant for; the Frame drops it if another has focus.
if "window" in event:
window, display = event.get("window"), event.get("display")
if isinstance(window, bool) or not isinstance(window, int) or window <= 0:
raise Failure("window must be the panel's window id", 400)
if not isinstance(display, str) or not PANEL_DISPLAY.match(display):
raise Failure("display must be an X display such as :1", 400)
out.update(window=window, display=display)
for name in ("dx", "dy"):
if name in event:
out[name] = num(name, INPUT_MOVE_LIMIT)
if "button" in event:
if event["button"] not in TOUCH_BUTTONS:
raise Failure(f"button must be one of {', '.join(TOUCH_BUTTONS)}", 400)
out["button"], out["down"] = event["button"], event.get("down") is not False
if "scroll" in event:
sc = event["scroll"]
if not isinstance(sc, list) or len(sc) != 2:
raise Failure("scroll must be [dx, dy]", 400)
out["scroll"] = [num_value(v, 5000) for v in sc]
if "key" in event:
key = event["key"]
if isinstance(key, bool) or not isinstance(key, int) or not 0 < key < 768:
raise Failure("key must be a Linux key code", 400)
out["key"], out["down"] = key, event.get("down") is not False
if "text" in event:
text = event["text"]
if not isinstance(text, str) or not 0 < len(text) <= INPUT_TEXT_LIMIT:
raise Failure(f"text must be 1 to {INPUT_TEXT_LIMIT} characters", 400)
out["text"] = text
if not set(out) - {"window", "display"}:
raise Failure("touch event has nothing to do", 400)
return out
def num_value(value, limit):
if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
raise Failure("scroll values must be numbers", 400)
return max(-limit, min(limit, round(float(value), 2)))
class TouchAgent(InputAgent):
"""frame_touch.py on the Frame, fed events over one long-lived ssh. Nothing to install:
it uses gamescope's own input socket and the libei that's on the image."""
def __init__(self):
super().__init__(source=HERE / "frame_touch.py", packages=[])
def deliver(self, report, force=False):
return ""
def command(self, folder=""):
code = base64.b64encode(self.source.read_bytes()).decode()
return "python3 -u -c " + shlex.quote(
f"import base64;exec(compile(base64.b64decode('{code}'),'frame_touch','exec'))")
_touch = TouchAgent()
def remote_touch(body):
"""{"events": [...]} points, clicks, scrolls and types into the focused panel."""
events = body.get("events", [])
if not isinstance(events, list) or len(events) > INPUT_BATCH_LIMIT:
raise Failure(f"events must be a list of at most {INPUT_BATCH_LIMIT}", 400)
return _touch.send([touch_event(e) for e in events])
def flatpak(body): def flatpak(body):
app, action = str(body.get("id", "")), body.get("action") app, action = str(body.get("id", "")), body.get("action")
if not FLATPAK_ID.match(app): if not FLATPAK_ID.match(app):
@@ -1611,6 +1765,32 @@ def _sweep_one(prefix, d):
pass pass
# ---- Panel switcher (same helper on the companion and in the headset) ----
def panels_action(body):
action = body.get("action", "list")
script = (HERE / "frame_panels.py").read_text()
if action == "open":
# Installed in the user account so the page can outlive this SSH call.
remote = ('umask 077; mkdir -p ~/.local/share/frame-control/panels && '
'tmp=$(mktemp ~/.local/share/frame-control/panels/install.XXXXXX) && '
'cat > "$tmp" && mv "$tmp" ~/.local/share/frame-control/panels/switcher.py && '
'python3 ~/.local/share/frame-control/panels/switcher.py --open')
elif action == "list":
remote = "python3 -"
elif action == "focus":
key = body.get("key")
if not isinstance(key, str) or not frame_panels.KEY.fullmatch(key):
raise Failure("Choose an open panel.", 400)
remote = "python3 - --focus " + shlex.quote(key)
else:
raise Failure("Unknown panel action", 400)
result = json.loads(ssh(remote, stdin=script, timeout=45))
if "error" in result:
raise Failure(result["error"], 502)
return result
# ---- Our Frame-side media player ----------------------------------------- # ---- Our Frame-side media player -----------------------------------------
_MEDIA_LOCK = threading.Lock() _MEDIA_LOCK = threading.Lock()
@@ -1684,9 +1864,8 @@ def push_media(path):
# The tunnel gets its own connection: the shared master's options would win # The tunnel gets its own connection: the shared master's options would win
# over anything added after them. # over anything added after them.
macview = frame_pcview.host_view(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=8"], macview = frame_macview.MacView(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=8"],
lambda remote, stdin=None, timeout=30: ssh(remote, stdin=stdin.encode("utf-8") if stdin is not None else None, lambda remote, stdin=None, timeout=30: ssh(remote, stdin=stdin, timeout=timeout),
timeout=timeout, text=False).decode("utf-8", errors="replace"),
FRAME, track=_live_tunnels.add) FRAME, track=_live_tunnels.add)
@@ -1747,14 +1926,14 @@ def agent_approval(body):
return frame_agent.approvals.decide(body.get("confirmation"), body.get("accept")) return frame_agent.approvals.decide(body.get("confirmation"), body.get("accept"))
POST = {"/api/media": media, "/api/agent/call": agent_call, "/api/agent/approval": agent_approval, POST = {"/api/comfort": comfort, "/api/media": media, "/api/agent/call": agent_call, "/api/agent/approval": agent_approval,
"/api/assistant/chat": assistant_chat, "/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard, "/api/assistant/chat": assistant_chat, "/api/android/display": android_display, "/api/android": android, "/api/titles": titles, "/api/launch": launch, "/api/steam": steam, "/api/volume": set_volume, "/api/clipboard": clipboard,
"/api/input": remote_input, "/api/input": remote_input, "/api/touch": remote_touch,
"/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots, "/api/flatpak": flatpak, "/api/open": open_thing, "/api/shots/save": save_shots,
"/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start, "/api/webinstall/check": webinstall_check, "/api/webinstall/start": webinstall_start,
"/api/webinstall/cancel": webinstall_cancel, "/api/webinstall/cancel": webinstall_cancel,
"/api/telemetry": frame_telemetry.update_settings, "/api/telemetry/event": frame_telemetry.page_event, "/api/telemetry": frame_telemetry.update_settings, "/api/telemetry/event": frame_telemetry.page_event,
"/api/report/preview": report_preview, "/api/report": report_send, "/api/macview": macview_action} "/api/report/preview": report_preview, "/api/report": report_send, "/api/macview": macview_action, "/api/panels": panels_action}
# ---- HTTP ------------------------------------------------------------------ # ---- HTTP ------------------------------------------------------------------
@@ -1883,6 +2062,10 @@ class Handler(BaseHTTPRequestHandler):
self.send_json(title_job(url.query)) self.send_json(title_job(url.query))
elif path == "/api/licenses": elif path == "/api/licenses":
self.send_json({"notices": licenses()}) self.send_json({"notices": licenses()})
elif path == "/api/panels":
self.send_json(panels())
elif path == "/api/touch":
self.send_json(_touch.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_touch.status))
elif path == "/api/input": elif path == "/api/input":
self.send_json(_input.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_input.status)) self.send_json(_input.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_input.status))
elif path == "/api/job": elif path == "/api/job":
@@ -1914,6 +2097,8 @@ class Handler(BaseHTTPRequestHandler):
self.send_bytes(*shot_image(url.query)) self.send_bytes(*shot_image(url.query))
elif path == "/api/stream": elif path == "/api/stream":
self.stream_video(url.query) self.stream_video(url.query)
elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["panel"]:
self.send_bytes(panel_capture(url.query), "image/png", headers=[("X-Capture-Source", "panel")])
elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["headset"]: elif path == "/api/screenshot" and parse_qs(url.query).get("view") == ["headset"]:
self.send_bytes(headset_view(), "image/png", headers=[("X-Capture-Source", "steamvr")]) self.send_bytes(headset_view(), "image/png", headers=[("X-Capture-Source", "steamvr")])
elif path == "/api/screenshot": elif path == "/api/screenshot":
@@ -1990,7 +2175,7 @@ class Handler(BaseHTTPRequestHandler):
proc.wait() proc.wait()
errors.seek(0) errors.seek(0)
err = strip_ansi(errors.read().decode(errors="replace")).strip() err = strip_ansi(errors.read().decode(errors="replace")).strip()
raise Failure(err or "The headset view sent no video for 20 s") raise Failure(err or "The Frame sent no video for 20 s")
chunk = first chunk = first
try: try:
self.send_response(200) self.send_response(200)