mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-06 05:02:50 +02:00
Merge main into mac-in-headset (MCP agent routes alongside the Mac view); skip the bash syntax test on Windows
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
saphidandClaude Opus 5.5 committed
commit
f7573e3565
26 files changed
+1862
-8
No files matched your search
+185
@@ -0,0 +1,185 @@
|
||||
# Frame Control for AI agents
|
||||
|
||||
**Documented interface:** Frame Control's own stdlib Python MCP adapter wraps
|
||||
its loopback HTTP API. No API key, hosted service, model SDK or third-party
|
||||
helper app is needed. The assistant is our HTML/Python implementation hosted
|
||||
in the platform Chromium browser. Its optional LLM endpoint is user configuration.
|
||||
Installing other apps is an optional management action, never a prerequisite.
|
||||
|
||||
## Connect an MCP client
|
||||
|
||||
The default MCP command starts a private HTTP backend on a free loopback port,
|
||||
with a fresh local access key. It stops that backend when the MCP client closes
|
||||
stdin or sends SIGTERM. It uses its own SSH control socket, so closing it does
|
||||
not close the desktop app's connection. No manually started server is needed.
|
||||
|
||||
Add this stdio server to your MCP client (use absolute paths):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"frame-control": {
|
||||
"command": "python3",
|
||||
"args": ["/absolute/path/frame-control/ui/frame_mcp.py"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For Codex, the equivalent registration is:
|
||||
|
||||
```sh
|
||||
codex mcp add frame-control -- python3 /absolute/path/frame-control/ui/frame_mcp.py
|
||||
```
|
||||
|
||||
New agent sessions load the entry. An already running session may need its MCP
|
||||
connections reloaded; registration does not retroactively add tools to its
|
||||
initial tool inventory. Keep the checkout at that path while it is registered.
|
||||
Use `codex mcp remove frame-control` to remove only this registration.
|
||||
|
||||
To reuse a running server instead, pass `--url http://127.0.0.1:47810`.
|
||||
The desktop app uses a random port; use that port with `--url`, or run the
|
||||
checkout server above. If the HTTP server uses `FRAME_UI_KEY`, pass the same
|
||||
value in the MCP process environment. This is local access control, not an LLM
|
||||
API key. The adapter only accepts loopback HTTP servers, refuses redirects and
|
||||
ignores environment proxies. Stdout contains newline-delimited JSON-RPC only.
|
||||
It supports MCP initialization, ping, tool listing and tool calls; no sampling,
|
||||
resources, prompts or streaming transport.
|
||||
|
||||
| Tool | Arguments | Effect |
|
||||
|---|---|---|
|
||||
| `computer_state` | none | Read-only gamescope window IDs/focus and bounded AT-SPI tree; reports incomplete observations |
|
||||
| `status` | none | Battery, services, installed games and Flatpaks |
|
||||
| `screenshot` | `view`: `headset` (default) or `desktop` | Returns PNG image content to the MCP client |
|
||||
| `job` | `id` | Background install status; poll until `done`, inspect `error` |
|
||||
| `launch` | `appid` | Launch an installed Steam app |
|
||||
| `install` / `uninstall` | `id` | Install from Flathub / remove a user Flatpak |
|
||||
| `send_text` | `text` | Frame desktop clipboard; desktop must be open |
|
||||
| `send_file` | `path` | File on the HTTP server computer, up to 16 MiB, copied to Frame `~/Downloads` |
|
||||
| `panel` | `id` | Launch an installed Flatpak as a panel using the existing launcher |
|
||||
| `power` | `action`: `suspend`, `reboot`, `poweroff` | Open a terminal for the user to enter the sudo password |
|
||||
| `keep_awake` | `action`: `on`, `off`, `status` | Optional keep-awake script interface |
|
||||
|
||||
Only install free software with its developer's consent. There is no purchase,
|
||||
entitlement bypass or arbitrary shell tool. `install` returns a background job
|
||||
ID; it does not claim the installation has finished. APK and sideloaded title
|
||||
installs remain in the main UI for now.
|
||||
|
||||
### Approval is a separate human action
|
||||
|
||||
Every mutation first returns an `approvalUrl`, exact action and `confirmation`
|
||||
token. Ask the user to open that URL and choose **Approve this action** or
|
||||
**Reject**. Then repeat the same tool and arguments with the token in
|
||||
`confirmation`. The server refuses execution before approval, changed arguments,
|
||||
expired tokens and reuse. A file approval binds the content hash as well as the
|
||||
path. Approvals last five minutes and disappear when the HTTP server restarts.
|
||||
A failed execution also consumes the approval; review a fresh request to retry.
|
||||
The panel does not execute an action merely because it was approved.
|
||||
|
||||
MCP has no approval tool. This is protection against accidental model tool
|
||||
calls, not a sandbox against a client with independent shell/HTTP access to your
|
||||
computer. Grant the MCP client only the access you intend. Status, captures and computer-state observations
|
||||
are returned directly to that client, which may forward them to its configured
|
||||
model. The assistant's separate opt-in does not govern an external MCP client.
|
||||
|
||||
Power still requires the existing password prompt in a local terminal. MCP
|
||||
never receives passwords. Power via `FRAME_LOCAL=1` is unsupported: use the main
|
||||
UI. The panel launcher and keep-awake adapter require zsh on the computer.
|
||||
|
||||
[PR #16](https://github.com/saphid/frame-control/pull/16) owns
|
||||
`scripts/keep-awake.sh on|off|status`. This branch does not copy or change it.
|
||||
Until that script is present, the tool reports it unavailable. Keep-awake is
|
||||
never automatic: `on` changes the shared idle timers; explicitly approve `off`
|
||||
to restore them after work. It is not a per-agent lease; coordinate with other
|
||||
users. No changes are made to the analytics/update interfaces in
|
||||
[PR #17](https://github.com/saphid/frame-control/pull/17). Prompts, keys, model
|
||||
replies, screenshots and approval payloads are not sent to analytics.
|
||||
|
||||
## Assistant panel
|
||||
|
||||
Open **Tools → Open assistant**, or `http://127.0.0.1:47810/assistant`.
|
||||
To put the same page in the headset, with the HTTP server still running:
|
||||
|
||||
```sh
|
||||
python3 scripts/assistant-on-frame.py --port 47810
|
||||
```
|
||||
|
||||
This starts an SSH reverse forward bound to Frame loopback (port 47812 by
|
||||
default), then a dedicated Chromium profile tagged as a SteamVR panel. Keep the
|
||||
command running. Ctrl-C closes this browser profile and the tunnel; it leaves
|
||||
other Chromium windows and the existing HTTP server alone. A failed cleanup
|
||||
prints the temporary profile path so it can be removed when the Frame returns.
|
||||
Use `--frame-port` if the default is busy. Chromium must already be available as
|
||||
`org.chromium.Chromium`; the launcher never installs anything automatically.
|
||||
Place the panel with SteamVR's normal docking controls.
|
||||
|
||||
Enter your full **chat-completions endpoint**, model name and optional key.
|
||||
An OpenAI-compatible local server works without a key; no OpenAI account is
|
||||
required. HTTP is allowed only on loopback; other endpoints require HTTPS.
|
||||
Loopback refers to the computer running the HTTP server, even in the headset.
|
||||
Endpoints with embedded credentials, query strings or redirects are refused.
|
||||
|
||||
Check the message consent box and press **Send message**. Screenshot context is
|
||||
a separate unchecked box and sends one fresh capture with that request. Both
|
||||
boxes reset after sending, and changing endpoint/model revokes consent. Nothing
|
||||
is sent when opening the page or entering configuration. There is no model
|
||||
list fetch, saved history, automatic screenshot capture or assistant telemetry.
|
||||
Each send is independent: previous messages and replies are not included.
|
||||
|
||||
Configuration, credentials and chat remain in page memory; close/reload the page
|
||||
or choose **Clear everything** to clear them. A request already sent cannot be
|
||||
recalled. Only the chosen endpoint gets the request; proxy environment variables
|
||||
and redirects are disabled. Its privacy and retention policy still applies.
|
||||
Replies are plain text and cannot call tools or operate the Frame. A model must
|
||||
support image inputs to accept screenshot context.
|
||||
|
||||
## Evidence and limits
|
||||
|
||||
**Verified 2026-09-28, SteamOS 0.4.1, BUILD_ID 20260925.6191901:** loopback HTTP
|
||||
status through an SSH reverse tunnel; platform Chromium created a separate
|
||||
SteamVR panel (confirmed in `GAMESCOPE_FOCUSABLE_APPS`); headset capture returned
|
||||
a PNG. These checks preceded the UI implementation. No power or global settings
|
||||
were changed.
|
||||
|
||||
**Inferred:** visual comfort and controller keyboard usability while wearing
|
||||
the headset; panel creation in gamescope alone does not establish these.
|
||||
Windows/Linux launcher support, live third-party model endpoints, installs,
|
||||
uninstalls, power and keep-awake changes are not covered by that feasibility
|
||||
check. See the PR for the final unit and end-to-end results.
|
||||
|
||||
**Verified end to end on the same Frame/build (2026-09-28):** a stdio MCP client
|
||||
initialized, read status, retrieved a headset PNG, and transferred a test file
|
||||
only after approval through the Chromium page. Remote file bytes matched;
|
||||
reusing the confirmation was rejected. The actual headset Chromium page sent
|
||||
text and then separately opted-in image context to a local test endpoint and
|
||||
displayed its replies. Without consent there were zero endpoint requests.
|
||||
The test endpoint returned canned replies: model inference and a live external
|
||||
provider remain **unverified**. The launcher’s Ctrl-C cleanup was checked;
|
||||
profiles, SSH tunnels and the test file were removed. No installs, removals,
|
||||
launches of user games, power operations or keep-awake changes were performed.
|
||||
|
||||
**Verified locally:** unit coverage includes the stdio subprocess, approval
|
||||
binding/expiry/replay/concurrency, file-change rejection, and a real local HTTP
|
||||
endpoint for opt-in, text/image payloads and redirect refusal. Fake-Frame
|
||||
regressions are in `tests/e2e/test_agents.py`; local Docker execution was blocked
|
||||
because the Docker daemon was unavailable. The ARM64 fake-Frame CI job passed
|
||||
on this branch (run 36421345682).
|
||||
|
||||
**Verified on the same Frame/build:** both Ctrl-C and SIGTERM close the dedicated
|
||||
browser profile and SSH tunnel and remove the profile and panel log.
|
||||
|
||||
|
||||
|
||||
## Computer-use coverage
|
||||
|
||||
MCP is the tool transport, not a limit on what an agent can do. A screenshot,
|
||||
accessibility snapshot, click or keystroke can all be MCP tools when we have a
|
||||
reliable underlying implementation. See [the investigation](computer-use.md)
|
||||
for the verified boundaries. `computer_state` adds observation, not an input
|
||||
channel: it cannot click an approval button or send keyboard/mouse events.
|
||||
|
||||
**Verified 2026-09-29, SteamOS 0.4.1, BUILD_ID 20260925.6191901:** the command saved
|
||||
by `codex mcp add` launched without a prestarted server, negotiated MCP, listed
|
||||
12 tools, read live Frame status and returned X11 window state plus AT-SPI
|
||||
observations. It exited 0 at EOF. Steam's accessibility tree had inaccessible
|
||||
children, reported as `incomplete: true`; this is not a complete actionable UI.
|
||||
@@ -0,0 +1,144 @@
|
||||
# Announcing changes
|
||||
|
||||
How we tell people about Frame Control features and fixes as they merge. The
|
||||
same few sentences feed the X post, the release notes and the website, so they
|
||||
are written once, in the pull request, while the change is fresh.
|
||||
|
||||
## What gets announced
|
||||
|
||||
| Kind | Announce? | Example |
|
||||
|---|---|---|
|
||||
| **New** — something you can now do | Yes, its own post | Stream Mac windows into the Frame as panels |
|
||||
| **Better** — something existing got noticeably easier, faster or wider | Yes, its own post or a roundup | APKs install without the Android SDK |
|
||||
| **Fixed** — something broken that users hit | Yes if people reported it or it blocked a flow; otherwise the next roundup | Mac mirror showed a zoomed-in corner |
|
||||
| **Release** — a tagged build | Always, one post linking the release | Frame Control 0.3.1 |
|
||||
| Tests, refactors, CI, docs-only, website polish | No | Fake Frame tests, screenshot crop |
|
||||
|
||||
If a change isn't worth a sentence to someone who owns a Frame, it isn't
|
||||
announced.
|
||||
|
||||
## The voice
|
||||
|
||||
Write it the way the README and release notes already read.
|
||||
|
||||
- **Lead with what the person can now do**, in their words: "Install older
|
||||
versions of an app when the newest won't run on the Frame", not "Add APK
|
||||
version fallback resolver".
|
||||
- **Plain and specific.** Name the thing, give the number: "about 30 fps",
|
||||
"4,500 apps", "up to 8 older versions". No "blazing", "game-changing",
|
||||
"excited to announce", "huge", or exclamation marks.
|
||||
- **Say where it works.** Platforms and what it was tested on, briefly:
|
||||
"Tested on a real Frame from macOS 27." Don't claim what wasn't tested.
|
||||
- **Say the catch.** If it needs a setup step, an unsigned build, or only works
|
||||
on one OS, say so in the same post.
|
||||
- **Sentence case**, full sentences, British spelling to match the docs.
|
||||
Contractions are fine.
|
||||
- **No emoji in the text.** One image, GIF or short clip carries the tone
|
||||
instead. The only symbol is the kind label below.
|
||||
- **Unofficial, always.** Never imply Valve made or endorses it. Say "Steam
|
||||
Frame" for the headset and "Frame Control" for the app.
|
||||
- **Credit people.** If a user reported the bug or suggested the feature and is
|
||||
happy to be named, thank them by handle.
|
||||
|
||||
## The formats
|
||||
|
||||
Every announceable PR ends with an `## Announcement` section holding these.
|
||||
The reviewer checks it like code.
|
||||
|
||||
### 1. The post (X, and any other social account)
|
||||
|
||||
```
|
||||
<Kind>: <what you can do now, one sentence>
|
||||
|
||||
<one or two sentences: how it works, the catch, or what it was tested on>
|
||||
|
||||
<link>
|
||||
```
|
||||
|
||||
- `<Kind>` is `New`, `Better` or `Fixed`.
|
||||
- 280 characters maximum including the link (X counts any link as 23).
|
||||
- One link: the release if it has shipped, otherwise the PR.
|
||||
- One visual when the change is visible: a screenshot from the app, a GIF, or a
|
||||
short clip from the headset. Alt text describes what it shows.
|
||||
- No hashtags, except `#SteamFrame` on releases and on posts about something
|
||||
new, because people search for it.
|
||||
|
||||
### 2. The release-note line
|
||||
|
||||
One bullet under **New in x.y.z**, same as the current release notes: the
|
||||
first half of the post's first sentence, no kind label, no link.
|
||||
|
||||
### 3. Release post
|
||||
|
||||
```
|
||||
Frame Control <version>: <the headline change>
|
||||
|
||||
<one sentence on the headline change>. Also: <two or three short items>.
|
||||
|
||||
Windows, macOS and Linux: <release link>
|
||||
#SteamFrame
|
||||
```
|
||||
|
||||
The release title on GitHub uses the same `Frame Control <version>: <headline>`
|
||||
line, as 0.3.0 and 0.3.1 already do.
|
||||
|
||||
### Roundups
|
||||
|
||||
Small fixes that don't earn their own post wait for a roundup, posted with the
|
||||
next release or when three or more have piled up:
|
||||
|
||||
```
|
||||
Fixed in Frame Control this week:
|
||||
- <fix>
|
||||
- <fix>
|
||||
- <fix>
|
||||
|
||||
<link>
|
||||
```
|
||||
|
||||
## Examples from what has already merged
|
||||
|
||||
**#11, older APK versions**
|
||||
|
||||
```
|
||||
New: when an Android app is too new for the Frame, Frame Control now offers
|
||||
older versions that will install.
|
||||
|
||||
It checks F-Droid, its archive and IzzyOnDroid, and verifies each download
|
||||
before it goes on the headset.
|
||||
|
||||
https://github.com/saphid/steam-frame/pull/11
|
||||
```
|
||||
|
||||
**#8, Mac mirror fixes**
|
||||
|
||||
```
|
||||
Fixed: mirroring your Mac into the Steam Frame now fits the whole desktop in
|
||||
the panel, asks for the right password, and shows the cursor.
|
||||
|
||||
Tested end to end on a real Frame from macOS 27.
|
||||
|
||||
https://github.com/saphid/steam-frame/pull/8
|
||||
```
|
||||
|
||||
**v0.3.1**
|
||||
|
||||
```
|
||||
Frame Control 0.3.1: install APKs without the Android SDK
|
||||
|
||||
Frame Control now reads APK files itself, so there's nothing extra to install.
|
||||
Also: Linux and Windows game sideloading, and one-click install links.
|
||||
|
||||
Windows, macOS and Linux: https://github.com/saphid/steam-frame/releases/tag/v0.3.1
|
||||
#SteamFrame
|
||||
```
|
||||
|
||||
## Posting
|
||||
|
||||
Nothing is posted without a person approving it. The flow is:
|
||||
|
||||
1. The PR carries its `## Announcement` section.
|
||||
2. On merge, the post is drafted from that section (manually for now).
|
||||
3. Alex approves or edits it, then it's posted from the project account.
|
||||
4. Replies and questions that turn out to be bugs become GitHub issues labelled
|
||||
`feedback`, same as the website form.
|
||||
@@ -5,6 +5,12 @@ in **Lepton**, Valve's Waydroid-based container. Lepton is built for games,
|
||||
not general Android use
|
||||
([GamingOnLinux](https://www.gamingonlinux.com/2026/09/lepton-from-valve-to-run-android-games-on-linux-is-now-open-source/)).
|
||||
|
||||
**VR streaming clients:** WiVRn 26.9 and ALVR 20.14.1 install, but both fail
|
||||
OpenXR instance creation on the checked Frame because its Android runtime lacks
|
||||
`XR_KHR_convert_timespec_time` (**verified** 2026-09-28, SteamOS 0.4.1,
|
||||
BUILD_ID 20260925.6191901). They are not Frame Control dependencies. See
|
||||
[the feasibility results and options](linux-vr-streaming.md).
|
||||
|
||||
## Install from the Mac: one app, one Lepton instance (verified 2026-09-25)
|
||||
|
||||
Use Frame Control's **Android apps** section (search, Install, Test, Report), drop
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# Computer use through Frame Control MCP
|
||||
|
||||
The MCP transport can carry semantic actions or visual computer-use actions.
|
||||
The limits are the Frame's underlying interfaces, permissions and whether an
|
||||
action can be targeted and verified. A stereoscopic headset screenshot alone
|
||||
is not a reliable coordinate system for clicking a particular app window.
|
||||
|
||||
## What exists, and the right route
|
||||
|
||||
| Surface | Evidence and route | Remaining work or boundary |
|
||||
|---|---|---|
|
||||
| Frame management | **Verified:** existing SSH/HTTP operations for status, capture and file transfer work through MCP. Typed install/launch/power tools wrap the existing API. | Extend typed operations before adding generic mouse automation. Preserve explicit approval for consequential changes. |
|
||||
| App/window observation | **Verified 2026-09-29:** `computer_state` reads gamescope X11 window/app/process triples, focused app and the installed AT-SPI library. | Bounded to 96 accessible nodes and six levels. Trees may be truncated, stale, hidden or incomplete. Snapshot paths and XIDs are observations, never durable action permissions. |
|
||||
| Chromium page content | **Verified previously:** the assistant rendered and could be exercised through CDP in an isolated Frame Chromium profile. | A shipped click/type surface needs exact owned browser/target binding, fresh element references, lifecycle cleanup, consent and post-action readback. Do not expose unrestricted JavaScript or attach to arbitrary existing profiles automatically. |
|
||||
| Steam UI | **Verified 2026-09-29:** the AT-SPI service listed the Steam client's Chromium process and frame nodes, but child traversal was incomplete. Existing `frame_steam.py` uses Steam's loopback CDP endpoint for specific operations. | Prefer those narrow Steam interfaces. Presence of AT-SPI does not prove controls are actionable, and generic pointer injection is not proved for VR menus. |
|
||||
| Other Linux apps | **Verified 2026-09-29:** Frame ships libX11, libXtst and libatspi; `/dev/uinput` is writable by the current user. | Library presence and access permissions do not prove that a game accepts input. Global virtual input can affect whichever app has focus. Do not ship a blind keyboard/mouse tool on this evidence alone. |
|
||||
| Panel focus and layouts | **Documented in [#41](https://github.com/saphid/frame-control/pull/41):** `POST /api/panels` accepts `list`, `focus` and `open`. Focus was verified there. | Reuse that owned interface after integration. Its tested gamescope-owned overlay transform setters return `PermissionDenied`; no reliable saved spatial-layout interface was established. Do not duplicate its implementation here. |
|
||||
| Shared keyboard/trackpad | **Documented in [#19](https://github.com/saphid/frame-control/pull/19):** `/api/input` supplies state/start and event submission, implemented with a bundled KDE Connect daemon. | This branch does not import, launch or depend on that daemon. The user's own-implementation rule remains authoritative. A first-party input implementation or permitted bundled-library route needs its own delivery evidence before MCP integration. |
|
||||
| Physical/device boundaries | **Documented:** an asleep Frame may be off the network; power authorization can require the user's password; physical pairing and headset fit/comfort require the user. | MCP cannot bypass offline hardware, consent, compositor permissions or physical verification. Keep explicit human handoffs. |
|
||||
|
||||
## Reusing the existing computer-use work
|
||||
|
||||
**Documented:** the installed `cua-driver` skill has the right control pattern:
|
||||
observe an exact window, use a semantic target if available, fall back to pixels
|
||||
from that same snapshot, then read back the result. Its browser route requires
|
||||
an exact process/window/target binding and session-scoped element references.
|
||||
Those are useful design rules for Frame tools.
|
||||
|
||||
**Verified locally 2026-09-29:** `cua-driver describe get_window_state` describes
|
||||
host-local process/window IDs and macOS AX inspection. It does not establish an
|
||||
SSH Frame target. The installed skill's advertised Linux companion file is
|
||||
missing. A native ARM64 Frame backend, its dependencies and remote transport
|
||||
have not been verified. We therefore do not claim that the existing Mac driver
|
||||
can control the Frame by passing it a Frame PID or screenshot, and we do not
|
||||
make the feature depend on installing that application.
|
||||
|
||||
Frame Control's `computer_state` is our own Python implementation over installed
|
||||
platform libraries. It sends the probe over SSH stdin, writes no helper to disk,
|
||||
and exits after one observation. Missing displays/libraries return explicit
|
||||
errors; a 15-second process deadline prevents a stalled accessibility call from
|
||||
leaving a probe behind. Window names and accessibility text are untrusted app
|
||||
content, never instructions to an agent.
|
||||
|
||||
**Recommended next implementation:** an isolated Chromium session with typed
|
||||
snapshot/click/type/scroll tools and exact fresh target binding, then individually
|
||||
verified native app actions. Use the headset capture to judge appearance, not to
|
||||
invent a screen-to-window coordinate transform. Direct tool calls must retain
|
||||
approval rules; a generic computer-use tool must not become a route around the
|
||||
MCP approval panel, install confirmation or power confirmation.
|
||||
|
||||
## Isolated browser input proof
|
||||
|
||||
**Verified 2026-09-29, SteamOS 0.4.1, BUILD_ID 20260925.6191901:** a temporary
|
||||
Frame Chromium profile loaded a local test page through an SSH reverse tunnel.
|
||||
CDP `Input.insertText` entered the test string in its own input. A CDP
|
||||
`Input.dispatchMouseEvent` press/release on its own button copied that string
|
||||
to the page's result; DOM readback matched exactly. The browser profile,
|
||||
loopback forwards and panel log were removed afterward. No user app was typed
|
||||
into, no global settings were changed and no third-party helper app was used.
|
||||
|
||||
AT-SPI did **not** expose the test page's controls in that same probe, even with
|
||||
Chromium's renderer-accessibility flag. It returned the partial Steam-client
|
||||
tree instead. The reason remains **unverified**; this is an evidence gap, not
|
||||
proof that Frame accessibility cannot work. For a first implementation,
|
||||
Chromium's proven page-specific CDP route is stronger than assuming complete
|
||||
AT-SPI coverage. This proof does not ship unrestricted click/type tools or
|
||||
establish input delivery to SteamVR's menus.
|
||||
@@ -0,0 +1,47 @@
|
||||
Frame client feasibility, 2026-09-28
|
||||
SteamOS VERSION_ID=0.4.1 BUILD_ID=20260925.6191901; uname -m=aarch64
|
||||
SteamVR: vrserver log reports 2.18.1; process 2256 remained alive across checks.
|
||||
Base: dcf9689f6459e576d35fc507eb702ca6b2bf4dad (main).
|
||||
Unmodified upstream release APKs; installed with ui/frame_android.py install APK --vr.
|
||||
No Linux host attached. No pairing, streamed video, input, audio or worn-headset checks.
|
||||
Selected logcat lines only; timestamps in Android logs are UTC.
|
||||
|
||||
WiVRn-release.apk
|
||||
https://github.com/WiVRn/WiVRn/releases/tag/v26.9
|
||||
sha256=1df6649ec77224fcc821af0ab4897222bdf3d7eb6ce6ad636461336724111331
|
||||
E/OpenXR-Loader( 1140): Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_KHR_convert_timespec_time
|
||||
I/WiVRn ( 1140): [2026-09-28 12:07:58.987] [WiVRn] [info] Failed to create OpenXR instance version 1.1.58: XR_ERROR_EXTENSION_NOT_PRESENT
|
||||
E/OpenXR-Loader( 1140): Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_KHR_convert_timespec_time
|
||||
I/WiVRn ( 1140): [2026-09-28 12:07:59.034] [WiVRn] [info] Failed to create OpenXR instance version 1.0.58: XR_ERROR_EXTENSION_NOT_PRESENT
|
||||
E/WiVRn ( 1140): [2026-09-28 12:07:59.035] [WiVRn] [error] Error during initialization: Failed to create OpenXR instance: XR_ERROR_EXTENSION_NOT_PRESENT
|
||||
|
||||
alvr_client_android.apk
|
||||
https://github.com/alvr-org/ALVR/releases/tag/v20.14.1
|
||||
sha256=be68feeb02665e3d69f1cdbcabf38ea4d15c42868a7ec6b5e698dbefee4e4e36
|
||||
E/OpenXR-Loader( 1139): Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_KHR_convert_timespec_time
|
||||
I/RustStdoutStderr( 1139): Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_KHR_convert_timespec_time
|
||||
E/[ALVR NATIVE-RUST]( 1139): ALVR panicked: What happened:
|
||||
E/[ALVR NATIVE-RUST]( 1139): panicked at alvr/client_openxr/src/lib.rs:220:10:
|
||||
E/[ALVR NATIVE-RUST]( 1139): called `Result::unwrap()` on an `Err` value: ERROR_EXTENSION_NOT_PRESENT
|
||||
|
||||
Cleanup verified: both app directories, compatdata directories and Steam shortcuts absent.
|
||||
Both test containers stopped and removed. No Steam/SteamVR restart or global setting changes.
|
||||
|
||||
Valve release notes fetched from ISteamNews/GetNewsForApp/v2 (appid=250820).
|
||||
|
||||
SteamVR Beta Updated - 2.18.1
|
||||
https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1844751498219787
|
||||
Added tethered Quest support over USB (must be used with Steam Link Beta)
|
||||
|
||||
Introducing SteamVR 2.17
|
||||
https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1843481262693486
|
||||
Adds initial support for USB streaming. Note: Requires new Steam Client Beta.
|
||||
Fix crash using Steam Link on Linux when games submit invalid textures.
|
||||
Improve streaming recovery when using Steam Link on Linux.
|
||||
|
||||
SteamVR Beta Updated - 2.17.8
|
||||
https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1842212951314598
|
||||
Fix crash using Steam Link on Linux when games submit invalid textures.
|
||||
Improve streaming recovery when using Steam Link on Linux.
|
||||
Adds initial support for USB streaming.
|
||||
USB streaming can be used without WiFi by opting into the Steam Client Beta.
|
||||
@@ -154,3 +154,13 @@ npm run dist:linux # Linux: AppImage and .deb, x64 and arm64
|
||||
|
||||
Pushing a `v*` tag builds all three in GitHub Actions and attaches them to the
|
||||
release (`.github/workflows/release.yml`).
|
||||
|
||||
## AI agents and assistant
|
||||
|
||||
**Documented:** [the MCP adapter and assistant panel](agents.md) are Frame
|
||||
Control implementations. MCP wraps this HTTP API without API keys. Changes
|
||||
require a separate user approval; power also retains its password prompt. The
|
||||
assistant uses a user-chosen endpoint and sends nothing until the user opts in
|
||||
for a message. Screenshot context is separately opt-in. Model replies cannot
|
||||
operate the headset. Tools → Open assistant opens the page; the linked guide
|
||||
covers putting it in a Chromium panel on the Frame.
|
||||
@@ -58,6 +58,7 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600
|
||||
| **Tools on the image:** Python 3.12.3, `ffmpeg`, `openssl`, `curl`, `rsync`, `zip`/`unzip`, `flatpak`, `wpctl`, `podman`. **No `adb`.** `steamos` is uid 1000, in `wheel`, and sudoers has `%wheel ALL=(ALL) ALL`, so `sudo -S` takes the Developer Mode password on stdin. **Verified 2026-09-27.** | Running Frame Control's server on the Frame (`FRAME_LOCAL=1`, [iphone.md](iphone.md)) |
|
||||
| **Each Lepton instance is a podman container** named `lepton-steamlaunch-<instance id>`, labelled with its ADB port (`podman ps --format '{{.Names}} {{.Labels.adb_port}}'`). `podman exec <container> /system/bin/sh -c '…'` runs Android's shell inside it with no adb at all (used for `pidof` and `logcat` by the app tester). Running `wm size`/`wm density` that way is untested. **Verified 2026-09-27.** | `ui/frame_android.py`, the iPhone app's display settings |
|
||||
| **Asleep means off the network.** In standby the Frame stops answering on its LAN address, `frame.local` and Tailscale alike (`Host is down`, `No route to host`, timeouts), and ping fails. It was unreachable for about 2.5 hours until woken. Nothing over SSH can wake it. **Verified 2026-09-27.** | Frame Control's offline banner and retries |
|
||||
| **What puts it to sleep is Steam's idle timer**, not logind. The journal shows `steamui_system: Switching to power state: [ k_ESystemPowerState_Sleep ] reason: 'ComputeNextPowerState: active: 3600 < 3600 (k_EACState_Connected)'`, then Steam suspends. SSH work doesn't count as activity. The timers are the client settings `system_idle_suspend_ac_sec` (3600) and `system_idle_suspend_battery_sec` (900); 0 means Never (Settings → Power → Sleep after inactivity). They can be written over DevTools the way the settings page does. logind refuses a `systemd-inhibit --mode=block` sleep lock from an SSH session (`Interactive authentication required`) but accepts one started with `systemd-run --user`. `scripts/keep-awake.sh on|off|status` does both and restores the old timers on `off`. **Verified 2026-09-28**, BUILD_ID 20260925.6191901. Whether Steam's suspend honours the inhibitor on its own is **inferred** (polkit gives `steamos` no `suspend-ignore-inhibit`), not tested. | Keeping the Frame awake for agent work |
|
||||
| **Battery at full on a charger** can read `Discharging` at about 0 W (for example 99 %, 0.0 W, USB-C PD 18 W). Treat under 0.5 W on a charger as "not charging", not "draining". **Verified 2026-09-27.** | Frame Control's battery card |
|
||||
| **The OS image is downloadable.** Valve's recovery images for the Frame are at `https://steamdeck-images.steamos.cloud/recovery/`. The root filesystem inside is btrfs, and it runs as an SSH test target on ARM64 Linux without the headset (`tests/frame-container/frame-image.sh`). **Verified 2026-09-27.** | [recovery-and-images.md](recovery-and-images.md) |
|
||||
| **Boot / recovery menu.** Hold Power ~10 s until the LED goes off, then power on while holding the **AUX button on top of the Power button** (not the volume keys) until a text menu appears. Entries: `Current` (SteamOS-A/B + build), `Previous` (the other A/B slot), `Boot from USB`, `Repair Steam Installation`, `Erase User Data` (factory reset), `ADB mode`, `Battery Ship Mode`. It auto-boots `Current` after a ~15 s countdown. **Volume Up/Down (left side) move, AUX (right side) selects.** For a boot loop, Valve says pick `Previous` (keeps user data); then `Repair Steam Installation`; `Erase User Data` wipes `~` (SSH keys, Tailscale, Flatpaks, T3 setup). Last resort is a full re-image, two ways: (1) USB: write `steamframe-oobe-repair-<build>.img.bz2` to an 8 GB+ USB-C stick (Balena Etcher on the Mac), pick `Boot from USB`, then use "Wipe Device & Install SteamOS" / "Repair SteamOS" (keeps games and personal content) from the recovery desktop; (2) cable/EDL: `steamframe-oobe-repair-qdl-<build>.tar.gz`, run `flash.sh` (Linux) or `flash.cmd` (Windows), then with the Frame off for 10 s hold Power + Vol Up + Vol Down for 10 s and plug it in; it reflashes and reboots. Both images: `https://steamdeck-images.steamos.cloud/recovery/` (build 20260922.5153644, 0.3.0, 3.8 GiB each, no published checksums); local copies in `~/Downloads/steam-frame-recovery/`. File names, checksums and what's inside: [recovery-and-images.md](recovery-and-images.md). Source: Valve's [SteamOS Recovery FAQ](https://help.steampowered.com/en/faqs/view/1B71-EDF2-EB6D-2BB3) and [Installation and Repair FAQ](https://help.steampowered.com/en/faqs/view/65B4-2AA3-5F37-4227), plus a menu photo in [EloiStree/HelloSteamFrame#9](https://github.com/EloiStree/HelloSteamFrame/issues/9). **Inferred** (Valve docs, 2026-09-26); not yet tried on our Frame. | Recovering from a boot loop |
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 142 KiB |
@@ -0,0 +1,195 @@
|
||||
# PC VR streaming from Linux
|
||||
|
||||
**Recommendation, 2026-09-28:** test Valve's current SteamVR/Steam Link path
|
||||
on a Linux gaming PC before building another streamer. Valve now documents
|
||||
Linux streaming fixes and USB support. We have no Linux host attached, so
|
||||
Linux-to-Frame VR streaming remains **unverified here**.
|
||||
|
||||
This is the feasibility and options report for
|
||||
[#24](https://github.com/saphid/frame-control/issues/24), not a shipped streaming
|
||||
feature. Frame Control's features must use our own implementation or standard
|
||||
platform components. WiVRn and ALVR are research comparisons, not dependencies.
|
||||
An optional install shortcut is the most we would offer for a third-party app.
|
||||
Our own streamer requires Alex's choice before implementation.
|
||||
|
||||
## What was checked on the Frame
|
||||
|
||||
**Verified** on 2026-09-28: aarch64, SteamOS **0.4.1**, BUILD_ID
|
||||
`20260925.6191901`, SteamVR **2.18.1**. Version and build are recorded separately;
|
||||
earlier docs associate this build with other SteamOS version labels.
|
||||
|
||||
| Client | Installation | Runtime result |
|
||||
|---|---|---|
|
||||
| WiVRn **26.9**, upstream `WiVRn-release.apk` | API 29, arm64-v8a; installed in its own immersive Lepton instance | OpenXR instance creation fails: missing `XR_KHR_convert_timespec_time`. Both 1.1.58 and 1.0.58 attempts return `XR_ERROR_EXTENSION_NOT_PRESENT` |
|
||||
| ALVR **20.14.1**, upstream `alvr_client_android.apk` | API 26, arm64-v8a; installed in its own immersive Lepton instance | Same missing extension. Client panics at `client_openxr/src/lib.rs:220` with `ERROR_EXTENSION_NOT_PRESENT` |
|
||||
|
||||
The [evidence excerpt](evidence/linux-vr/2026-09-28.txt) includes APK SHA-256s,
|
||||
upstream release links, loader errors and cleanup results. These are failures
|
||||
before an OpenXR session, not successful VR clients. WiVRn was launched twice;
|
||||
ALVR's container remained up despite its client panic. Container liveness alone
|
||||
does not establish VR compatibility.
|
||||
|
||||
Both APKs already declare `MAIN` and `LAUNCHER`. They were installed unmodified
|
||||
using this branch's existing `python3 ui/frame_android.py install APK --vr`,
|
||||
then launched through their Steam shortcuts. Logs came from the instance's
|
||||
`podman exec … /system/bin/logcat`; the user journal also retained WiVRn's errors
|
||||
after its container exited. No headset was worn and no host was connected.
|
||||
|
||||
All test app files, compatdata, shortcuts and containers were removed afterwards.
|
||||
SteamVR's original process remained running. No global settings changed.
|
||||
|
||||
### Relation to the VR APK branch
|
||||
|
||||
**Documented from source:** [PR #20](https://github.com/saphid/frame-control/pull/20)
|
||||
was read, not edited (branch inspected at
|
||||
[`038dcd4`](https://github.com/saphid/frame-control/commit/038dcd48cd75336f6a86c63c7878bfc9c52deec9)).
|
||||
Its compatibility layer handles OpenXR version negotiation, some controller
|
||||
profiles and refresh-rate requests. It does **not** implement
|
||||
`XR_KHR_convert_timespec_time`. Its launcher fix is unnecessary for these APKs.
|
||||
This report has **no unmerged code dependency** on that PR, and neither APK was
|
||||
tested with its layer injected.
|
||||
|
||||
**Documented from upstream source:** WiVRn requests the extension in
|
||||
[`application.cpp`](https://github.com/WiVRn/WiVRn/blob/bbc6e4cc36c355fa6180980abd231673dc15115d/client/application.cpp#L1286)
|
||||
and uses it to convert `CLOCK_MONOTONIC` into `XrTime` in
|
||||
[`instance::now()`](https://github.com/WiVRn/WiVRn/blob/bbc6e4cc36c355fa6180980abd231673dc15115d/client/xr/instance.cpp#L335).
|
||||
ALVR also [requests it unconditionally](https://github.com/alvr-org/ALVR/blob/a9f6542fa507a841f40ab4f3fcb531427cd02550/alvr/client_openxr/src/lib.rs#L188).
|
||||
Simply deleting the extension request or returning made-up timestamps would
|
||||
not prove correct tracking or timing. A real fix needs a valid clock mapping
|
||||
and further runtime tests. No such patch was made.
|
||||
|
||||
### Native SteamOS aarch64 clients
|
||||
|
||||
**Verified:** the Frame has a native OpenXR runtime manifest at
|
||||
`~/.config/openxr/1/active_runtime.json`, pointing to SteamVR's
|
||||
`bin/linuxarm64/vrclient.so`.
|
||||
|
||||
**Documented:** WiVRn's [26.9 README](https://github.com/WiVRn/WiVRn/blob/bbc6e4cc36c355fa6180980abd231673dc15115d/README.md)
|
||||
describes its Linux client as debugging-only, without audio or hardware decode.
|
||||
ALVR 20.14.1's [non-Android decoder](https://github.com/alvr-org/ALVR/blob/a9f6542fa507a841f40ab4f3fcb531427cd02550/alvr/client_core/src/video_decoder/mod.rs)
|
||||
returns no decoded frames. The inspected releases ship Android clients, not a
|
||||
ready-to-run native Frame client.
|
||||
|
||||
**Inferred:** a native port is possible research, but neither release offers a
|
||||
demonstrated native alternative to the blocked APKs. Native builds, native
|
||||
extension enumeration, hardware decoding and audio were **not tested**. The
|
||||
Android extension failure does not establish that the native runtime lacks it.
|
||||
|
||||
## (a) Valve's own path — recommended first
|
||||
|
||||
**Documented**, from Valve's release notes rather than launch-window reports:
|
||||
|
||||
- [SteamVR 2.17.8 beta](https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1842212951314598)
|
||||
says “Fix crash using Steam Link on Linux when games submit invalid textures”
|
||||
and “Improve streaming recovery when using Steam Link on Linux.” It also
|
||||
adds initial USB streaming, with Steam Client Beta required to use USB
|
||||
without Wi-Fi.
|
||||
- [SteamVR 2.17 release](https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1843481262693486)
|
||||
repeats Linux streaming fixes and initial USB support. USB is no longer
|
||||
solely a claim about an old beta, but version/channel requirements still
|
||||
need checking on the actual host.
|
||||
- [SteamVR 2.18.1 beta](https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1844751498219787)
|
||||
adds USB-tethered **Quest** support with Steam Link Beta. That entry is not
|
||||
proof of a Frame/Linux combination.
|
||||
- The [Steam Link page](https://store.steampowered.com/app/353380/Steam_Link/)
|
||||
lists Linux desktop clients, while its Quest VR requirements still say
|
||||
Windows 10 or newer. Desktop Steam Link support is not equivalent to VR host
|
||||
support, and the Quest requirements are not a Frame support matrix.
|
||||
|
||||
**Inferred:** Valve has a Linux VR streaming path worth testing. The old blanket
|
||||
claim “Linux cannot stream VR” is no longer justified by the evidence. These
|
||||
release notes do not establish which Linux GPU/driver/Frame combinations work.
|
||||
USB changes the transport; it does not by itself prove host encoder support.
|
||||
|
||||
**Not verified:** Linux host discovery, pairing, wireless or USB streaming,
|
||||
stereo rendering, controllers, haptics, audio, latency, or a game. Frame-only
|
||||
inspection cannot establish any of these. Flat Remote Play and a desktop shown
|
||||
on a panel are not substitutes for this test.
|
||||
|
||||
Next test, once a Linux gaming PC is available: record distro, GPU/driver,
|
||||
Steam client channel/version and SteamVR version; use the Frame's built-in
|
||||
Steam connection flow, first wirelessly and then over a data-capable USB cable.
|
||||
Launch a free OpenXR sample or developer-consented VR game. Verify stereo,
|
||||
head/controller tracking, haptics and audio while worn; retain both ends' logs
|
||||
and measure latency and recovery after a link interruption. Restore any test
|
||||
channel changes. Do not change the shared headset's channel just for this report.
|
||||
|
||||
If that works, Frame Control can provide our own host checks, setup guidance
|
||||
and session controls around Valve's existing platform. First establish which
|
||||
controls have a usable interface; no stable automated pairing API has been
|
||||
verified. **Estimate (inferred):** 2–5 engineer-days for the hardware feasibility
|
||||
pass; another 1–2 weeks for a small integration if those interfaces exist.
|
||||
|
||||
## (b) Our own streaming — proposal only
|
||||
|
||||
This is a new VR transport and device integration, not a desktop capture feature.
|
||||
A plausible first target is **one Linux GPU family, one host, one Frame**, using
|
||||
SteamVR on both ends. Our host driver would expose a remote HMD/controllers,
|
||||
receive poses and inputs, and obtain stereo textures for hardware encoding.
|
||||
Our Frame OpenXR app would decode, submit the correct eye views and render poses
|
||||
at predicted display times, and return tracking/input. SteamVR/OpenXR, bundled
|
||||
codec/transport libraries and platform GPU APIs fit the ownership rule; a
|
||||
WiVRn/ALVR/Monado server dependency would not.
|
||||
|
||||
**Inferred design risks:** Linux SteamVR texture-sharing/driver interfaces and
|
||||
Frame decode-to-GPU interoperability need a spike before committing to this
|
||||
architecture. Sending an already-composited desktop mirror loses the stereo,
|
||||
pose and timing information we need. Late reprojection, clock conversion,
|
||||
backpressure, controller bindings, audio sync and reconnects are substantial
|
||||
work. A runtime shim must not assume `XrTime` equals monotonic nanoseconds.
|
||||
|
||||
### Reuse from `mac-in-headset`
|
||||
|
||||
**Documented from our code**, read-only at
|
||||
[`1b90c64`](https://github.com/saphid/frame-control/commit/1b90c64b73bace54c63a3aae154c5d29a9448d72):
|
||||
|
||||
- Reuse the ideas for low-latency encoding without B-frames, dropping work
|
||||
before encoding, bounded queues, keyframe recovery, adaptive bitrate,
|
||||
per-frame timing and authenticated session setup.
|
||||
- Its VideoToolbox encoder and ScreenCaptureKit capture are macOS-specific.
|
||||
Linux needs a new GPU encoder path (for example VA-API or NVENC via bundled
|
||||
libraries) and VR texture capture, not a port of window capture.
|
||||
- Its WebSocket over SSH is useful for a first controlled transport experiment
|
||||
and control messages. Reliable TCP can stall behind lost packets; a VR media
|
||||
path needs measured deadline behaviour, likely datagrams with loss recovery
|
||||
using an ordinary bundled transport library. Do not invent cryptography.
|
||||
- Its Chromium/WebCodecs panel viewer is not a VR client. That branch reports
|
||||
software H.264 decoding and occasional long Wi-Fi stalls on the Frame.
|
||||
Its desktop latency measurements are not motion-to-photon measurements or
|
||||
evidence that a 90/120 Hz stereo stream will work.
|
||||
|
||||
**Size/effort estimate (inferred, one experienced full-time engineer, hardware
|
||||
available):**
|
||||
|
||||
| Phase | Deliverable / stop condition | Effort |
|
||||
|---|---|---|
|
||||
| Feasibility | Linux driver texture access, Frame hardware decode into OpenXR, pose/clock loop; stop if any cannot meet frame deadlines | 2–4 weeks |
|
||||
| First end-to-end prototype | One GPU/codec, stereo sample over a controlled LAN, head/controllers, logs and teardown | 4–8 additional weeks |
|
||||
| Usable limited beta | Audio/haptics, pairing, recovery, bitrate/loss handling, installer, worn testing and latency work | 6–12 additional weeks |
|
||||
| Wider support | Multiple GPU vendors/distros, USB and Wi-Fi variation, long-session stability | 2–4 additional months |
|
||||
|
||||
Planning range: **12–24 engineer-weeks for a limited beta**, roughly
|
||||
**10–25k lines of our code plus tests/tooling**, excluding bundled libraries.
|
||||
This is a low-confidence scope estimate, not a delivery promise; an unsupported
|
||||
driver or decode interface could block it entirely. Foveated streaming,
|
||||
eye tracking and parity with Valve are excluded. A Linux gaming PC and repeatable
|
||||
worn-headset testing are prerequisites. **Do not build this until Alex chooses.**
|
||||
|
||||
## (c) Optional “install WiVRn” shortcut only
|
||||
|
||||
Allowed as a clearly optional convenience, never a prerequisite for a Frame
|
||||
Control feature. **Documented:** WiVRn's server Flatpak ID is
|
||||
`io.github.wivrn.wivrn`; its client/server versions must match, and its Flatpak
|
||||
includes xrizer/OpenComposite. Those are properties of an independently
|
||||
installed third-party stack, not components of our implementation.
|
||||
|
||||
**Recommendation:** defer the shortcut while the current client fails before
|
||||
session creation. If offered later, label that compatibility result and let
|
||||
the user choose the install; do not present “install” as “streaming works.”
|
||||
**Estimate (inferred):** 1–2 engineer-days for an optional host-side shortcut
|
||||
with package/version detection and honest status, excluding third-party fixes.
|
||||
No shortcut, host install, pairing automation or streaming UI was built here.
|
||||
|
||||
Choose **(a)** for the next hardware test. Keep **(b)** as a separately approved
|
||||
project if Valve's path fails or lacks a required capability. **(c)** does not
|
||||
solve the verified client blocker and should not be the product's foundation.
|
||||
+7
-3
@@ -5,6 +5,8 @@ This covers three directions, plus input:
|
||||
- **A. Frame → Mac**: see and control the headset from the Mac.
|
||||
- **B. Mac → Frame**: use the Mac's desktop inside the headset.
|
||||
- **C. iPhone → Frame**: mirror the phone inside the headset.
|
||||
- **PC VR from Linux**: [feasibility and options](linux-vr-streaming.md),
|
||||
including Valve's streaming and USB support. No Linux host tested yet.
|
||||
- **Input**: type and point in the Frame from the Mac or iPhone.
|
||||
|
||||
The confidence labels are the same as in [ssh.md](ssh.md).
|
||||
@@ -24,11 +26,13 @@ Mac with keyboard, mouse, and clipboard.
|
||||
|
||||
## B. Show the Mac's desktop inside the Frame
|
||||
|
||||
The Frame's streaming features are built around a **Windows PC running
|
||||
SteamVR** plus the USB Wi-Fi 6E dongle. Even Linux hosts had VR-streaming
|
||||
problems at launch
|
||||
The Frame's VR streaming uses **SteamVR** on the host. Linux hosts had
|
||||
VR-streaming problems at launch
|
||||
([Steam discussion](https://steamcommunity.com/app/4165890/discussions/0/528765047224280796/),
|
||||
[gbl08ma](https://gbl08ma.com/posts/steam-frame-a-linux-machine-doesnt-support-linux/)).
|
||||
Valve's later 2.17.8 notes explicitly describe Steam Link fixes on Linux and
|
||||
initial USB streaming support (**documented**, not tested from a Linux host
|
||||
here). See [the current comparison](linux-vr-streaming.md#a-valves-own-path--recommended-first).
|
||||
**macOS isn't a supported SteamVR host**, so for the Mac we're only looking at
|
||||
flat 2D desktop streaming into a window on the Frame's Linux desktop.
|
||||
|
||||
|
||||
@@ -148,3 +148,11 @@ For example, on 2026-09-27 the smoke test found that Steam's `create-shortcut`
|
||||
refuses ids with a hyphen (`missing/invalid arguments`), which the fake had
|
||||
accepted. The fake now refuses them the same way, and Frame Control makes ids
|
||||
Steam accepts.
|
||||
|
||||
## Agent interfaces
|
||||
|
||||
`tests/test_agent.py` exercises MCP stdio, exact-action human approvals and the
|
||||
assistant against an in-process HTTP endpoint with canned responses (no keys or
|
||||
external calls). `tests/e2e/test_agents.py` runs the MCP/HTTP/SSH path against the
|
||||
fake Frame for approved installs, clipboard and file transfer. Headset Chromium
|
||||
rendering and real screenshots still need a device; see [agent evidence](agents.md#evidence-and-limits).
|
||||
Reference in new issue
Block a user