mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-06 06:00:33 +02:00
Merge main into devices: several headsets alongside the app store, comfort, panels and media
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
saphidandClaude Opus 5.5 committed
commit
02b9413f78
129 files changed
+12672
-148
No files matched your search
@@ -168,6 +168,43 @@ on this branch (run 36421345682).
|
||||
**Verified on the same Frame/build:** both Ctrl-C and SIGTERM close the dedicated
|
||||
browser profile and SSH tunnel and remove the profile and panel log.
|
||||
|
||||
**Verified end to end on the same Frame/build (2026-09-29), with mutations:**
|
||||
a stdio MCP client started `ui/frame_mcp.py` in its default mode (private
|
||||
backend, no API key, no prestarted server) and a human approved or rejected
|
||||
each change in the approval page in a real Chrome window:
|
||||
|
||||
| Tool | Result on the Frame |
|
||||
|---|---|
|
||||
| `send_file` | Approved; the file arrived in `~/Downloads` with identical contents |
|
||||
| `install` | Approved; `io.github.fizzyizzy05.binary` job finished in about 35 s |
|
||||
| `panel` | Approved; gamescope listed a new panel window, and `computer_state` reported the same window ID and PID. The app rendered in that window (below) |
|
||||
| `launch` | Approved; Keep Talking and Nobody Explodes (341800) started under Proton and `computer_state` reported it as the focused app |
|
||||
| `uninstall` | Approved; app and locale removed |
|
||||
| `power` | Rejected in the page. Unapproved retries, the same token used for `uninstall`, and a retry after rejection were all refused. Nothing was powered off |
|
||||
| `send_text` | Approved, then refused because the Plasma desktop was not open (documented requirement) |
|
||||
| `keep_awake` | `status` reports the script unavailable until PR #16 lands |
|
||||
|
||||
A separate Claude Code CLI session, with only this server configured, read
|
||||
status, `computer_state` and a headset capture, and requested an install. It
|
||||
received an approval URL and did not execute anything.
|
||||
|
||||
The assistant opened as a Frame panel through `scripts/assistant-on-frame.py`.
|
||||
Against a loopback stub model, a send without consent made zero requests. With
|
||||
consent it made exactly one, carrying the text and a fresh Frame screenshot.
|
||||
Consent unticked itself after sending. SIGTERM removed the panel, profile, log
|
||||
and tunnel.
|
||||
|
||||
**Not verified while unworn:** every headset capture was a uniform dark frame,
|
||||
so SteamVR's rendered view of panels and the game could not be checked; window
|
||||
captures (`xwd`) were used instead. MCP can launch a game or panel but has no
|
||||
tool to stop one: the tester stopped them over SSH. Removing an app leaves any
|
||||
runtime it pulled in; Flatpak may also remove related extensions when that
|
||||
runtime is removed by hand.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Computer-use coverage
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
# APK repositories
|
||||
|
||||
Frame Control supports **F-Droid-format repositories**, including F-Droid,
|
||||
F-Droid archive, IzzyOnDroid and user-provided HTTPS repositories. Repository
|
||||
indexes are authenticated before their apps appear. Search lists builds with
|
||||
Android API ≤30 and arm64-v8a or no native libraries, using the same streaming
|
||||
reducer as the existing catalogue. This does not guarantee an app works in Lepton.
|
||||
|
||||
## Formats considered
|
||||
|
||||
| Format | Users and purpose | Support in this source |
|
||||
|---|---|---|
|
||||
| F-Droid v2 | F-Droid, IzzyOnDroid, self-hosted fdroidserver repositories; consumed by F-Droid clients including Droid-ify and Neo Store | Preferred: signed `entry.jar` authenticates `entry.json`; its SHA-256 authenticates `index-v2.json`, which supplies APK SHA-256 hashes |
|
||||
| F-Droid v1 | Older F-Droid servers and clients | Fallback: verify `index-v1.jar`, then read its signed `index-v1.json` |
|
||||
| Obtainium configurations / exports | Obtainium users share app URLs plus source-specific filters and update settings; exports can contain a list of app configuration objects | Not imported here: configurations describe how to find releases, not one signed repository index |
|
||||
| SideQuest listings / custom feeds | SideQuest's own app discovery and installation service | No interoperable signed custom-repository specification was established from the public project documentation examined; SideQuest needs its own adapter |
|
||||
| GitHub release lists | Developers publish APK assets on release pages; community lists link to projects | Not a repository standard: asset naming, build selection and publisher verification vary; handled separately from this F-Droid source |
|
||||
| Minimal JSON list | A private list could contain package, title, APK URL and SHA-256 | Deliberately not introduced: unsigned hashes downloaded alongside files do not authenticate their publisher; another bespoke signing/update protocol would duplicate F-Droid |
|
||||
|
||||
Research references (checked 2026-09-28):
|
||||
|
||||
- [F-Droid APIs](https://f-droid.org/docs/All_our_APIs/) and
|
||||
[repository setup](https://f-droid.org/docs/Setup_an_F-Droid_App_Repo/).
|
||||
- [F-Droid signing keys](https://f-droid.org/docs/Release_Channels_and_Signing_Keys/)
|
||||
and [IzzyOnDroid's repository page and fingerprint](https://apt.izzysoft.de/fdroid/).
|
||||
- [Droid-ify](https://github.com/Droid-ify/client) and
|
||||
[Neo Store](https://github.com/NeoApplications/Neo-Store).
|
||||
- [Obtainium](https://github.com/ImranR98/Obtainium), its
|
||||
[configuration/deep-link format](https://wiki.obtainium.imranr.dev/deep_links/),
|
||||
and [community app configurations](https://apps.obtainium.imranr.dev/).
|
||||
- [SideQuest's public client](https://github.com/SideQuestVR/SideQuest).
|
||||
The absence of a specification in these materials is not proof that no
|
||||
historical or private custom-feed format exists.
|
||||
|
||||
## Add a repository in Frame Control
|
||||
|
||||
From the Frame Control checkout, use its source-management CLI:
|
||||
|
||||
```sh
|
||||
python3 ui/apk_sources/fdroid.py add 'https://example.org/fdroid/repo?fingerprint=YOUR_64_HEX_CERTIFICATE_FINGERPRINT' --name 'My apps'
|
||||
python3 ui/apk_sources/fdroid.py list
|
||||
python3 ui/apk_sources/fdroid.py search SOURCE_ID 'music'
|
||||
python3 ui/apk_sources/fdroid.py download SOURCE_ID org.example.app
|
||||
python3 ui/apk_sources/fdroid.py remove SOURCE_ID
|
||||
```
|
||||
|
||||
Replace `SOURCE_ID` with the `id` printed by `add` or `list`. `--fingerprint`
|
||||
can also supply the pin. `fdroidrepos://example.org/fdroid/repo?fingerprint=…`
|
||||
links are accepted and converted to HTTPS. Conflicting fingerprints are refused.
|
||||
A URL must identify the repository directory, not its website or an index file.
|
||||
|
||||
Adding fetches and validates the complete index **before saving** the source.
|
||||
Without a fingerprint, Frame Control verifies the JAR signature and remembers
|
||||
its signer: trust on first use (TOFU). This establishes continuity with the
|
||||
first server response, not independent publisher identity. Obtain the published
|
||||
fingerprint through a trusted channel when possible; the store's Add a source
|
||||
form shows the pinned one ("Trusted on first use: …") so you can compare it.
|
||||
Re-adding an existing URL preserves its pin; changing it requires deliberately
|
||||
removing and re-adding it.
|
||||
|
||||
The API for the search/server integration is in `ui/apk_sources/fdroid.py`:
|
||||
`add_repo(url, fingerprint=None, name=None)`, `remove_repo(source_id)`,
|
||||
`set_enabled(source_id, enabled)`, and `user_repos()`. The module also exposes
|
||||
`sources`, `search`, `details`, and `download` from the shared source contract.
|
||||
This change supplies the CLI and API; the integrated source-management UI is
|
||||
separate work. Built-in sources can be disabled but cannot be removed.
|
||||
|
||||
Settings and pins live in `frame_host.data_dir('apk-repos.json')`
|
||||
(`~/Library/Application Support/Frame Control/apk-repos.json` on macOS).
|
||||
Authenticated reduced indexes and APKs live under
|
||||
`frame_host.cache_dir('apk-sources')`; indexes refresh after 24 hours. An
|
||||
expired index is still served (marked stale in the store) while it refreshes in
|
||||
the background; a failed refresh is retried after 10 minutes.
|
||||
Only the running Frame Control app prunes cached APKs (at start and after store
|
||||
downloads); the command-line tools never do.
|
||||
The existing catalogue's unverified index cache is never treated as authenticated.
|
||||
|
||||
Rollback protection: each repository's newest accepted signed index timestamp
|
||||
is kept in `apk-repo-state.json` next to the settings, and an older index is
|
||||
refused. Once a repository has served a v2 `entry.jar`, a missing `entry.jar`
|
||||
is an error rather than a reason to fall back to `index-v1.jar`. `entry.jar`
|
||||
must be signed with SHA-2 (SHA-1 is still accepted for legacy `index-v1.jar`).
|
||||
Removing a repository clears its state.
|
||||
|
||||
## Publish your own repository
|
||||
|
||||
Only publish free APKs you own or have the developer's permission to distribute.
|
||||
Do not publish paid app mirrors or bypass store licences. Check distribution
|
||||
terms before adding someone else's repository; this module does not infer legal
|
||||
permission from a signature or automatically audit a repository's terms.
|
||||
|
||||
Install a current [fdroidserver](https://f-droid.org/docs/Installing_the_Server_and_Repo_Tools/)
|
||||
and its documented Android/Java dependencies on the publishing machine, then:
|
||||
|
||||
```sh
|
||||
mkdir my-fdroid
|
||||
cd my-fdroid
|
||||
fdroid init
|
||||
# Set repo_url in config.yml to https://example.org/fdroid/repo
|
||||
# Also set repo_name and repo_description; keep the generated signing key safe.
|
||||
cp /path/to/your-free-app.apk repo/
|
||||
fdroid update --create-metadata
|
||||
# Review the generated metadata (name, summary, licence, source and website).
|
||||
fdroid update
|
||||
```
|
||||
|
||||
Serve the generated **repo directory** at that HTTPS URL, including APKs,
|
||||
icons, `entry.jar`, `index-v2.json` and `index-v1.jar`. Do not publish the
|
||||
private signing keystore or configuration passwords. Configure fdroidserver's
|
||||
`serverwebroot` and run `fdroid deploy` for managed publication, or copy the
|
||||
public directory with your existing deployment tool. Publish the SHA-256
|
||||
repository certificate fingerprint displayed by fdroidserver in a link such as
|
||||
`https://example.org/fdroid/repo?fingerprint=…`.
|
||||
|
||||
Keep the repository signing key backed up: changing it breaks existing pins.
|
||||
For updates, add the new APK, edit metadata as needed, run `fdroid update` and
|
||||
publish again. Test the published URL with Frame Control's `add`, `search` and
|
||||
`download` commands. The above publisher setup is documented from fdroidserver;
|
||||
it was not executed as part of this implementation.
|
||||
|
||||
## Verification and limits
|
||||
|
||||
The stdlib verifier supports one RSA PKCS#1 v1.5 JAR/CMS signer with a key of
|
||||
2048–8192 bits; SHA-256/384/512 and legacy SHA-1 digest encodings are
|
||||
recognized. It checks the signer certificate pin, the signature over `.SF`,
|
||||
the whole-manifest digest, and the manifest's digest of the JSON member.
|
||||
ECDSA, DSA, RSA-PSS, multiple signers and section-only `.SF` manifests are
|
||||
rejected. Certificates are pinned identities, not validated as Web PKI chains.
|
||||
HTTPS certificates are separately checked by Python's normal TLS validation.
|
||||
|
||||
v1 fallback occurs only when `entry.jar` returns HTTP 404 or 410. Signature,
|
||||
fingerprint, index hash, TLS and server errors never trigger an unsigned
|
||||
fallback. APKs are cached by SHA-256 and checked again before reuse. Here,
|
||||
`verified: true` means the bytes match the signed repository's APK hash; it
|
||||
is not an independent APK publisher-signature or runtime compatibility verdict.
|
||||
There is no repository timestamp rollback/expiry policy or automated signing-key
|
||||
rotation yet. An old correctly signed index can still validate.
|
||||
|
||||
Offline fixtures exercise v2, v1, TOFU, pin changes, disabled sources, cache
|
||||
reuse, URL rejection and corruption of every signature/hash layer. On the Mac,
|
||||
the real IzzyOnDroid repository was added with its published pin, searched for
|
||||
Tiny Music Player, and its 16,520-byte APK downloaded with SHA-256
|
||||
`d7bcb24d101b04beb3394b695b24be4e2c3d6ed702f1d0e06bc4dd707f64d86a`.
|
||||
No headset connection or installation was performed.
|
||||
@@ -0,0 +1,103 @@
|
||||
# Developer-consented APK sources
|
||||
|
||||
Surveyed 2026-09-28. Free access is not proof of redistribution permission or
|
||||
Frame compatibility. These adapters fetch only public publisher releases or
|
||||
link to publisher pages. They do not acquire store entitlements, defeat access
|
||||
checks, install anything, or rehost APKs. See [VR compatibility](vr-apks.md).
|
||||
|
||||
| Source | Developer consent and automated-access position | API/feed; VR coverage | Decision |
|
||||
|---|---|---|---|
|
||||
| [itch.io](https://itch.io/docs/legal/terms) | Publishers warrant distribution rights (§4). Users may access content through the service; this is not blanket scraping permission. Main robots excludes `/game/download/`; author subdomains exclude `/*/download/`. No challenge bypass. | Public free Android RSS for `openxr` and `oculus-quest`; substantial indie VR. Server API is mostly authenticated publisher/account functionality, not a general anonymous store-download API. | Implement RSS search, artwork and page links; `downloadable: False`. The supplied free-download script follows keyed download pages excluded by robots, so it is not shipped. |
|
||||
| [GitHub releases](https://docs.github.com/en/rest/releases/releases) | Maintainers publish assets; curated repositories below establish provenance. Public hosting or an open-source topic alone does not establish rights to every uploaded binary. Use supported REST API under [API terms](https://docs.github.com/en/site-policy/github-terms/github-terms-of-service#h-api-terms), not HTML crawling. | Releases API includes APK assets and sometimes SHA-256. Topic search finds OpenXR/Quest projects. 60 unauthenticated requests/hour; authenticated user limits are generally 5,000/hour, with separate search/secondary limits. | Implement curated downloads and explicit topic discovery. Unreviewed topic results are page-only. |
|
||||
| [Uptodown](https://www.uptodown.com/aboutus) | Developer distribution program exists, but that does not prove publisher authorization for every catalog item. [Privacy policy](https://www.uptodown.com/aboutus/privacy) explicitly describes protection against automated access. General automation permission was not established. | Broad Android catalog, limited VR focus; no supported public consumer-download API established in this survey. | Page links only; no downloader. Do not infer consent from an unchanged APK signature. |
|
||||
| [APKPure](https://apkpure.com/terms) | Third-party APK catalog; individual publisher consent and automation rights were not established. Terms request returned HTTP 403; no bypass attempted. | Broad Android coverage, incidental VR; internal endpoints are not permission to automate. | Exclude automatic indexing/downloading; user may open site. |
|
||||
| [APKMirror](https://www.apkmirror.com/faq/) | Publisher-signed files and a free-app policy are not a blanket developer-consent or automation grant. FAQ request returned HTTP 403, so current terms could not be confirmed. | General Android/version archive; APK bundles often need another installer; little VR focus. No supported consumer-download API established. | Page links only, no scraping or bundle conversion. |
|
||||
| [Aptoide](https://en.aptoide.com/company/legal) | Terms define an app supplier as developer, owner or authorized distributor; user stores still require per-item provenance. API availability alone does not settle third-party access rights. | API ecosystem and general Android catalog; weak VR focus. | Defer until a publisher-owned store and its API terms can be approved. No blanket community-store downloader. |
|
||||
| [Amazon Appstore](https://developer.amazon.com/docs/app-submission/understanding-submission.html) | Official developer submissions; store account, device and license rules apply. Publisher submission APIs do not authorize public binary extraction. | Fire-device distribution; Android-device Appstore support ended in 2025; little Quest relevance. | Official product links only; no account or entitlement extraction. |
|
||||
| [PICO / ByteDance store](https://developer.picoxr.com/document/distribute) | Official publisher channel with store/device entitlements. No public unauthenticated binary-download grant established; documentation request encountered a redirect error. | Strong standalone VR; PICO builds may depend on PICO services/extensions. | Store links only. A developer's independently published GitHub/itch build can qualify separately. |
|
||||
| [Meta Horizon Store / former App Lab](https://www.meta.com/experiences/) | Official developer submissions. A free store entitlement is still an entitlement; no license bypass or authenticated store extraction. App Lab was folded into the main store in 2024. | Strongest Quest coverage; no supported anonymous APK-download API established. | Store links only; independently distributed free builds use their publisher source. |
|
||||
| [Khronos samples](https://github.com/KhronosGroup/OpenXR-SDK-Source) | Official upstream, Apache-2.0 sample; developer-published release APKs. GitHub API terms apply. | `hello_xr` Vulkan/OpenGL ES APKs; excellent OpenXR diagnostics. | Included in GitHub curated list, Vulkan variant selected. |
|
||||
| [Meta OpenXR samples](https://github.com/meta-quest/Meta-OpenXR-SDK) | Official upstream; check each sample's license. Source availability does not imply a published APK, and some samples require Meta extensions/services. | Source/build examples, inconsistent ready-made APK releases. | Link to upstream; add specific free APKs only after release/provenance review. |
|
||||
| [Godot XR demos](https://github.com/GodotVR/godot-xr-tools) | Official project source and publisher demo pages; licenses and dependencies vary by demo. | OpenXR examples on GitHub/itch. Older Godot builds can fail on Lepton's missing clipboard service. | Covered by source discovery; no compatibility promise from an OpenXR tag. |
|
||||
|
||||
The table distinguishes observed restrictions from unknown permission. An
|
||||
unverified policy is a reason to defer automation, not a claim that a site is
|
||||
unlawful. Only the two implemented source kinds are registered by their own
|
||||
`sources()` functions; the other rows are recommendations, not new UI entries.
|
||||
|
||||
## Adapters
|
||||
|
||||
`ui/apk_sources/github.py` uses `github_curated.json`: Khronos `hello_xr`,
|
||||
[Open Brush](https://github.com/icosa-foundation/open-brush), and
|
||||
[SuperTux 3D](https://github.com/SgtBilko76/SuperTux-3D). These have official
|
||||
OpenXR project/release evidence, not a blanket claim of headset compatibility.
|
||||
Open Brush's compatibility evidence is recorded in [vr-apks.md](vr-apks.md).
|
||||
Open Brush and SuperTux publish the selected builds as prereleases; curated
|
||||
opt-ins preserve that label in version records. Exact APK filename patterns
|
||||
avoid downloading desktop archives or alternate non-Quest builds.
|
||||
[OpenSaberPlus](https://github.com/arpruss/OpenSaberPlus) was examined but not
|
||||
curated: GitHub reports its license as `NOASSERTION`, and current OpenXR APK
|
||||
provenance was not established in this pass.
|
||||
|
||||
Default GitHub search is offline against this small list. Queries
|
||||
`topic:openxr`, `topic:oculus-quest`, and `topic:quest` explicitly call repository
|
||||
search. Results outside the curated list stay page-only, even if a repository
|
||||
claims an open-source license. This prevents an arbitrary tagged mirror from
|
||||
becoming a trusted downloader. Extend the curated JSON after provenance review.
|
||||
|
||||
Set optional `FRAME_GITHUB_TOKEN` in the process environment for a higher API
|
||||
quota. Tokens are sent only to `api.github.com`, never written to the cache,
|
||||
never sent to asset hosts, and removed on redirects. The adapter does not
|
||||
read `gh` credentials automatically. Metadata is cached for one hour under
|
||||
`frame_host.cache_dir('apk-sources', 'publisher')`. A cold details request
|
||||
fetches at most ten releases. Rate-limit errors are surfaced without retry
|
||||
loops. Asset IDs and release tags are not Android version codes: metadata
|
||||
leaves the latter unknown and rejects a requested `version_code` rather than
|
||||
silently fetching a different build.
|
||||
|
||||
`itch.py` exposes separate OpenXR and Quest feed sources, so one feed's failure
|
||||
does not suppress the other at the aggregator level. Queries filter the current
|
||||
feed window locally: this is not an exhaustive historical itch search. Only
|
||||
explicit zero-price Android entries are returned. Covers are exposed in
|
||||
`images`; absent screenshots, APK version, ABI and minimum SDK stay unknown.
|
||||
Curated GitHub entries include publisher artwork and plain-language summaries.
|
||||
Repository image URLs are pinned to inspected commits. Open Brush screenshots
|
||||
come from its README-linked Steam listing; SuperTux uses the upstream gameplay
|
||||
preview embedded in the port's README (not a headset capture). The hello_xr
|
||||
sample has a launcher icon and GitHub social banner; no published screenshot
|
||||
was found in the inspected repository/README, so its screenshot list is empty.
|
||||
Uncurated topic results use the owner's avatar and GitHub's repository social
|
||||
preview. These are repository placeholders, not app screenshots. Itch's recorded
|
||||
RSS includes only covers, so screenshot lists remain empty without page scraping. VR is
|
||||
based on curated evidence or a VR-specific feed/topic, not a compatibility claim.
|
||||
|
||||
Downloads stream to unique temporary files, require an APK manifest entry,
|
||||
restrict HTTPS origins and redirects, and enforce a 2 GiB ceiling. `verified`
|
||||
means the downloaded SHA-256 matches GitHub's published digest. Without such a
|
||||
digest, the computed SHA-256 is returned with `verified: False`; neither value
|
||||
claims publisher-signature validation. Installation must inspect the APK as
|
||||
usual. OBBs, split APKs, paid assets and external release-body download links
|
||||
are unsupported.
|
||||
|
||||
## Evidence and limits
|
||||
|
||||
On this Mac, Python 3.9 downloaded the real Khronos Vulkan 1.1.63 APK through
|
||||
the GitHub adapter, matched its published SHA-256
|
||||
`f24bbe8ba6f6339fca658628868ba8189cbc33390d6ac508f69d76fb67b5fa34`, and
|
||||
`python3 ui/frame_android.py info <apk>` exited 0: package
|
||||
`org.khronos.openxr.hello_xr.vulkan`, version code 1063, minimum API 24,
|
||||
arm64-v8a present, OpenXR detected. No Frame connection or installation occurred.
|
||||
|
||||
The itch OpenXR RSS was fetched successfully and recorded as a fixture.
|
||||
Subsequent live adapter search encountered HTTP 429; it is not claimed as a
|
||||
successful live end-to-end search. Fixture search finds Off Nominal and parses
|
||||
nine Android entries from the ten-item feed (one has only an HTML platform).
|
||||
A real itch download and APK inspection were deliberately not performed:
|
||||
robots restrictions take precedence over that requested proof. No current
|
||||
policy text is claimed verified where the table records failed access.
|
||||
|
||||
Tests use recorded, reduced API/RSS fixtures with network access blocked in
|
||||
the new test class. They cover selection, prereleases, unknown topic results,
|
||||
paid/non-Android exclusion, URL restrictions, redirect credential removal,
|
||||
caching, rate limits, checksum mismatch, non-APK rejection and partial-file
|
||||
cleanup. See `.claude/NOTES-more-sources.md` for commands and local evidence.
|
||||
@@ -331,3 +331,11 @@ because gamescope scales Lepton's surface to fit the same panel. Also unverified
|
||||
whether the settings survive the app or its Lepton instance relaunching.
|
||||
Lepton Development rebuilds its Android data on exit, so there they probably
|
||||
don't.
|
||||
|
||||
## Expansion files and save backups
|
||||
|
||||
SideQuest-inspired CLI helpers install local OBB files into an already-running
|
||||
app instance and back up/restore a stopped instance's private app data. See
|
||||
[SideQuest features and limits](sidequest.md) for commands, archive scope and
|
||||
verification status. These paths have offline coverage; real Frame storage and
|
||||
permissions remain unverified. They do not change APK install or launch behavior.
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"date": "2026-09-28",
|
||||
"os": {
|
||||
"version": "0.4.1",
|
||||
"build": "20260925.6191901",
|
||||
"variant": "vr"
|
||||
},
|
||||
"performance": {
|
||||
"compositorFps": 72.0,
|
||||
"frameMs": 13.89,
|
||||
"appFps": null,
|
||||
"gpuMs": 3.07,
|
||||
"compositorCpuMs": 0.61,
|
||||
"cpuPercent": 40.6,
|
||||
"gpuMHz": 903.0
|
||||
},
|
||||
"batteryPercent": 16,
|
||||
"maxTempC": 73.5,
|
||||
"ownership": [
|
||||
{
|
||||
"id": 1009850,
|
||||
"owned": false,
|
||||
"installed": false,
|
||||
"frame": 0
|
||||
},
|
||||
{
|
||||
"id": 1173510,
|
||||
"owned": false,
|
||||
"installed": false,
|
||||
"frame": 0
|
||||
},
|
||||
{
|
||||
"id": 1068820,
|
||||
"owned": false,
|
||||
"installed": false,
|
||||
"frame": 0
|
||||
},
|
||||
{
|
||||
"id": 908520,
|
||||
"owned": false,
|
||||
"installed": false,
|
||||
"frame": 0
|
||||
},
|
||||
{
|
||||
"id": 1494460,
|
||||
"owned": false,
|
||||
"installed": false,
|
||||
"frame": 0
|
||||
}
|
||||
],
|
||||
"hudLifecycle": "open, duplicate-open, close passed before control-test pause; overlay probe removal verified",
|
||||
"comfort": "Initial 1cm seated probe restored exactly. Later commits/readback disagreed while Frame in use; Alex paused control tests. No controls shipped."
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 269 KiB |
@@ -0,0 +1,58 @@
|
||||
# Independent review attempts — 2026-09-28
|
||||
|
||||
Target: the implementation and evidence in
|
||||
[3fb541d](https://github.com/saphid/frame-control/commit/3fb541d) and
|
||||
[6a63a5a](https://github.com/saphid/frame-control/commit/6a63a5a), supplied as a
|
||||
frozen diff before those commits were made. No executable code changed after
|
||||
the final review snapshot.
|
||||
|
||||
Requested model: **SWE-2 Max**, explicitly selected with `--model swe-2-max`.
|
||||
No completed verdict or model self-identification was returned. This is not a
|
||||
passed review, and there is no "no actionable findings" claim.
|
||||
|
||||
## First attempt
|
||||
|
||||
Launcher:
|
||||
|
||||
```sh
|
||||
devin -p --model swe-2-max --permission-mode auto --respect-workspace-trust false --prompt-file /tmp/frame-vr-review-prompt.txt
|
||||
```
|
||||
|
||||
The prompt required read-only review, no delegation, no edits and no Frame
|
||||
access. The reviewer inspected surrounding code, then stopped while checking
|
||||
the public OpenVR header. The tool runner reported:
|
||||
|
||||
> warning: rejected a tool call that requires confirmation. Running in non-interactive mode.
|
||||
|
||||
The real launcher exit status was **0**, but no verdict was returned. A zero
|
||||
process status here is not evidence that the review completed.
|
||||
|
||||
## Tool-free retry
|
||||
|
||||
Launcher:
|
||||
|
||||
```sh
|
||||
devin -p --model swe-2-max --permission-mode auto --respect-workspace-trust false --prompt-file /tmp/frame-vr-review-final-prompt.txt
|
||||
```
|
||||
|
||||
The self-contained prompt supplied the complete frozen changes, surrounding
|
||||
code, standards and locally fetched authoritative OpenVR header excerpts. It
|
||||
explicitly prohibited tools, edits, delegation and device access. This avoided
|
||||
the first attempt's permission boundary without escalating permissions.
|
||||
|
||||
No output or verdict arrived within the fifteen-minute review window. The
|
||||
process was sent SIGTERM at 918 seconds; the shell recorded real exit status
|
||||
**143**. Findings are unavailable. Review must be completed before considering
|
||||
this partial draft ready; playspace feasibility is also still paused.
|
||||
|
||||
## Other validation
|
||||
|
||||
- 166 Python unit tests passed locally.
|
||||
- 8 website tests and UI JavaScript syntax passed locally.
|
||||
- Desktop and phone-width attached-preview checks passed using live telemetry;
|
||||
paid optional install buttons were disabled, and unavailable metrics cleared.
|
||||
- [Fake-Frame CI](https://github.com/saphid/frame-control/actions/runs/36423548487/job/108931878908)
|
||||
passed, as did Windows/Linux server tests and the main checks job. Docker was
|
||||
unavailable locally. macOS/iOS jobs were still queued at this handoff.
|
||||
- Real-device evidence and the paused-control limitation are in
|
||||
[VR utilities](../../vr-utilities.md).
|
||||
@@ -0,0 +1,122 @@
|
||||
# Family and comfort
|
||||
|
||||
Frame Control's Home tab has a **Family and comfort** card, on desktop and
|
||||
on iPhone. No third-party notification or parental-control app is needed.
|
||||
This is Frame Control code using Python, Steam and SteamVR already on the Frame.
|
||||
|
||||
|
||||
|
||||
## Sessions
|
||||
|
||||
Set a limit of 1–240 minutes, optional break and check-in intervals, then
|
||||
**Start session**. Break and check-in intervals of 0 turn those reminders off.
|
||||
**Cancel session** cancels the timer and monitoring without changing the game.
|
||||
Cancel before starting a session with different settings.
|
||||
|
||||
The Frame shows a one-minute warning, then opens Steam Home in its dashboard.
|
||||
**Games stay running**: save and pause before the limit. Some games pause when
|
||||
the dashboard opens; others do not. There is no kill, power-off, Steam restart,
|
||||
account restriction or parental lock. The wearer can return to the game.
|
||||
|
||||
**Documented implementation:** the timer is a single, opt-in Python worker in
|
||||
the Frame user's account. Desktop and iPhone share its state. It keeps going
|
||||
when the companion disconnects, closes or is suspended. It exits after
|
||||
completion or cancellation (normally within five seconds). Cancellation waits
|
||||
for any in-flight SteamVR action to finish within its timeout; it is not a boot
|
||||
service. A Frame reboot invalidates the session. Suspend counts toward the
|
||||
limit, using Linux's boot-time clock. If a warning was delayed by suspend or a
|
||||
SteamVR failure, Home waits until at least a full minute after a successful
|
||||
warning. A failed Home transition remains active and retries, with an error
|
||||
shown in the companion. A stale worker is reported as unverified enforcement.
|
||||
|
||||
## Alerts and breaks
|
||||
|
||||
During a session, battery, overheating and check-in alerts go to connected
|
||||
companions. Break reminders and session warnings also appear on the headset.
|
||||
|
||||
- **Low battery:** 15% or below while discharging. One alert until charging or
|
||||
recovery to 20%, so values around 15% do not produce repeated notifications.
|
||||
- **Overheating:** a thermal zone reaches its own kernel-reported hot/critical
|
||||
trip, or the battery reports `Overheat`. Missing sensors mean unknown, not
|
||||
safe. These are status alerts, not medical advice or an extra thermal governor.
|
||||
- **Check in:** an alert after the chosen number of active minutes.
|
||||
- **Breaks:** a SteamVR reminder and companion notification at the chosen interval.
|
||||
|
||||
**Inferred:** SteamVR activity levels 1 and 2 are a useful proxy for use, not
|
||||
proof someone is wearing the headset. Inactive readings reset continuous use;
|
||||
missing readings add no time. Long gaps count at most 30 seconds. Breaks and
|
||||
check-ins are distinct from the elapsed-time session limit.
|
||||
|
||||
Click **Enable / test notifications** on each companion. iOS asks for permission;
|
||||
macOS, Windows and Linux follow their notification settings. The page also shows
|
||||
recent events and errors. Keep Frame Control open and connected for companion
|
||||
alerts. **Phone alerts are local, not push notifications:** iOS suspension,
|
||||
force-quit or a lost SSH connection prevents live delivery. Old alerts are not
|
||||
replayed as a notification burst on reconnect. Headset warnings and the session
|
||||
limit continue without the phone. A physical iPhone's background delivery has
|
||||
not been verified and is not guaranteed.
|
||||
|
||||
## Casting
|
||||
|
||||
**Cast headset view** starts the existing headset Live view and requests full
|
||||
screen where supported. Show that screen to people in the room, or use the
|
||||
computer/phone's own screen mirroring. It creates no new stream transport,
|
||||
public URL or LAN server. iPhone uses the inline viewer if full screen is not
|
||||
available. The image includes private content visible to the wearer.
|
||||
|
||||
## What is installed
|
||||
|
||||
The shared authenticated `/api/comfort` endpoint copies three bundled Python
|
||||
files to `~/.cache/frame-control/comfort/<content-hash>/`. Session state and
|
||||
locks live in `~/.local/state/frame-control/comfort/`, with a private directory
|
||||
and 0600 state file. There is no network listener or system service. Cancel a
|
||||
session before removing these directories. The iPhone's normal server still
|
||||
exits on disconnect; the explicitly started comfort worker is the exception.
|
||||
|
||||
## Verification
|
||||
|
||||
**Verified 2026-09-28**, SteamOS 0.4.1, build `20260925.6191901`: shipped
|
||||
`/opt/steamvr/bin/linuxarm64/vrcmd --notify TEXT` reported success for a custom
|
||||
reminder. Steam's CDP `SteamUIStore.Navigate('/library/home')` and
|
||||
`SteamClient.OpenVR.VROverlay.ShowDashboard('valve.steam.gamepadui.main')`
|
||||
opened Home while the running app ID stayed unchanged. Prior page and dashboard
|
||||
visibility were restored. Kernel hot/critical trips and SteamVR activity were
|
||||
read from the real device. No temperature or battery fault was induced.
|
||||
|
||||
**Verified locally:** deterministic fake-Frame tests cover late warnings,
|
||||
failed warnings/Home actions, cancellation, activity gaps, thresholds, duplicate
|
||||
suppression, reboot invalidation, shared session state and the exact Home
|
||||
JavaScript. `python3 -m unittest discover -s tests` runs them. The iOS Simulator
|
||||
build tests notification content and bounds. Physical iPhone delivery and
|
||||
wearer-perceived headset notification visibility remain unverified.
|
||||
|
||||
**Verified end to end on the same Frame:** a two-minute session with no companion
|
||||
connection for 135 seconds emitted its warning, break and check-in, then opened
|
||||
Home. The running app ID was unchanged; the test restored the previous page and
|
||||
dashboard visibility and confirmed the worker exited. Casting through the Home
|
||||
shortcut decoded the existing headset stream at 30 fps.
|
||||
|
||||
**Verified on the iOS 26.5 Simulator:** connected to the real Frame, approved the
|
||||
notification prompt, and saw the native Frame Control test banner. Seven iOS
|
||||
tests passed.
|
||||
|
||||
|
||||
|
||||
Desktop and 390-pixel phone layouts had no horizontal overflow.
|
||||
On macOS the development Electron app's real notification attempt was denied
|
||||
(`UNErrorDomain` 1); the bridge now returns that failure instead of reporting
|
||||
success. Successful macOS/Windows/Linux notification display remains unverified.
|
||||
|
||||
**Verified on the real Frame:** its naturally discharging 15% battery produced
|
||||
one low-battery event during a short session; the test then cancelled the
|
||||
session. Overheating alerts use fake sensor samples in tests: the shared
|
||||
headset was not deliberately overheated.
|
||||
|
||||
**Verified 2026-09-29 on the same Frame:** a fresh one-minute session opened
|
||||
Home more than 60 seconds after the successful warning. The test restored the
|
||||
previous page and dashboard visibility. Local regression coverage now includes
|
||||
slow notification delivery, a total Home-action timeout, failed worker startup,
|
||||
unreadable saved state, malformed activity samples and notification UX: 173
|
||||
Python tests passed. Desktop and 390-pixel layouts were checked again; system
|
||||
notification-denial guidance stayed visible across polls. Initial event history
|
||||
did not replay notifications, and only the latest new event was announced.
|
||||
@@ -40,6 +40,8 @@ Lepton (Android 11, podman container "lepton-dev") ← its own panel, app 305600
|
||||
| Present: `rsync`, `flatpak`, `python3`, `git`, `qdbus6`, `xrdp`, `xprop`, `xwininfo`, `xterm`, `konsole`, `dolphin`, `gamescopectl`. Missing: `wl-copy`, `xclip`, `xsel`, `kdeconnect-cli`, `tailscale` (installable in `~`, see below), `krfb`, `wayvnc`. | Script design |
|
||||
| **SteamOS updates arrive on their own.** The Frame went from 0.3.0 (build 20260922.6101926) to **0.4.1, build 20260925.6191901**, between 2026-09-27 and 2026-09-28 with no action from us; `~` (keys, user Flatpaks, `~/.local/share`) survived. **Verified 2026-09-28.** | Keep changes in `~` |
|
||||
| **Valve's package repository has more than the image.** `pacman -Si` / `pacman -Sp` work as `steamos` without root and list Valve's own builds, such as `kdeconnect` 24.02.2 and `python-evdev` 1.7.0 in `extra`. Unpacking those packages into `~` runs them without touching the read-only root. The repository URLs say not to share them, so never write them down; Valve also publishes each build's source package there (`sources/packages/`), which is how Frame Control got the complete source for the KDE Connect it ships. **Verified 2026-09-28**, SteamOS 0.4.1. | [streaming.md](streaming.md#input-type-and-point-in-the-frame-from-the-mac-or-iphone) |
|
||||
| **gamescope has its own input injection.** An EIS socket at `/run/user/1000/gamescope-0-ei` (libei 1.4.1 is on the image) offers "Gamescope Virtual Input": relative and absolute pointer, buttons, scroll, keyboard. It drives the panel that has focus in the headset, on either X display. Focus moves only with the controller's laser (or to a new panel when none has it); `gamescopectl focus_info` prints the focus state to the journal. **Verified 2026-09-29.** | [streaming.md](streaming.md#live-view-and-control-watch-a-panel-and-tap-on-it) |
|
||||
| **A panel's own pixels:** `ffmpeg -f x11grab -window_id <window> -i :<display>` captures one window (x11grab of the root is black under gamescope). Panels live on `:0` (Steam's UI, windows tagged by `panel-on-frame.sh`) or `:1` (apps Steam starts). **Verified 2026-09-29.** | Frame Control's Desktop view |
|
||||
| **gamescope runs two Xwayland displays.** `:0` holds Steam's VR bar and menus (`valve.steam.gamepadui.*`) and ignores XTest pointer motion; `:1` holds apps such as Chromium and takes it. There's also a libei socket, `/run/user/1000/gamescope-0-ei`. **Verified 2026-09-28**, SteamOS 0.4.1. | Keyboard and trackpad |
|
||||
| Flathub is a **system** remote. `--user` installs over SSH work and show up in the desktop menu. | `install-apps.sh` |
|
||||
| `/` is 10 GB and read-only. `/home` is 929 GB. | Where to put things |
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 192 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 71 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 81 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
+12
-1
@@ -26,7 +26,10 @@ as its transport too), so the desktop and phone share one code path. Android
|
||||
display settings use `podman exec` into each Lepton container instead of adb,
|
||||
which the Frame doesn't have.
|
||||
|
||||
Nothing is left running on the Frame after the phone disconnects; the copied
|
||||
The app server stops after the phone disconnects. An explicitly started
|
||||
[comfort session](family-comfort.md) keeps its timer and headset reminders running
|
||||
until the session ends or is cancelled; phone notifications require the app to
|
||||
remain connected and running. The copied
|
||||
files stay in `~/.cache/frame-control` (delete it any time).
|
||||
|
||||
## Pairing
|
||||
@@ -108,3 +111,11 @@ running), a real sleep/restart/shut down on the Frame, and a physical iPhone.
|
||||
Debug builds have Simulator test hooks (`FRAME_TEST_HOST`, `FRAME_TEST_PAGE`,
|
||||
`FRAME_TEST_JS`, and the tunnel URL in the app's Caches folder); release builds
|
||||
don't.
|
||||
|
||||
## Family and comfort
|
||||
|
||||
The shared Home card sets session limits, breaks and check-ins, and offers
|
||||
**Cast headset view**. **Enable / test notifications** requests iOS notification
|
||||
permission and sends a local test. These are local notifications, not APNs push;
|
||||
iOS background suspension can interrupt phone alerts. The headset timer still
|
||||
runs. See [the behavior and verification limits](family-comfort.md).
|
||||
@@ -468,3 +468,18 @@ versus on, medians of the runs, ms):
|
||||
window to the front first.
|
||||
- Ctrl stays Ctrl. On the Mac, copy is ⌘C, so use Meta+C on a keyboard paired
|
||||
with the Frame.
|
||||
|
||||
## Switching panels and workspace limits
|
||||
|
||||
**Tools → Panel switcher** lists open SteamVR panels, including Mac viewers.
|
||||
Use **Show** to request focus or **Open in headset** for Frame Control's own
|
||||
switcher panel. It uses SteamVR/gamescope and Chromium, with no third-party
|
||||
overlay app. [Device checks and limits](panels.md#frame-controls-panel-switcher)
|
||||
include the difference between a panel surviving a scene launch and staying
|
||||
visible over it.
|
||||
|
||||
Saved spatial layouts are blocked on this build: the public OpenVR transform
|
||||
setter denies access to gamescope-owned panels. Reconnecting an existing viewer
|
||||
is supported; restoring its room position after a reboot is not. We do not
|
||||
save short-lived Mac window IDs or viewer access keys as if they were a durable
|
||||
workspace. See [the feasibility evidence](panels.md#saved-spatial-layouts-blocked-on-the-current-panel-route).
|
||||
+159
-2
@@ -121,8 +121,165 @@ a limit on the number of floating panels.
|
||||
script needed.
|
||||
- **Inside the desktop panel**: KWin tiling (Meta+arrow keys with a Bluetooth
|
||||
keyboard) or virtual desktops arrange windows within the 1280×800 rectangle.
|
||||
- **Windows-only overlay tools** (Desktop+, OVR Toolkit, OVRdrop) do this for a
|
||||
PC's desktop in SteamVR. They don't run on the Frame's standalone Linux.
|
||||
- **Optional overlay tools:** Desktop+, OVR Toolkit and similar software are
|
||||
separate from Frame Control. Public reports describe some Proton support;
|
||||
Windows-only does not by itself prove a Frame app cannot run. Local status
|
||||
and sources are in [VR utilities](vr-utilities.md).
|
||||
- **Our performance HUD:** Home → VR comfort and performance → Open HUD in
|
||||
headset creates its own gamescope panel using built-in tools. It needs no
|
||||
third-party overlay app. [Metrics and verification](vr-utilities.md).
|
||||
|
||||
## Frame Control's panel switcher
|
||||
|
||||
**Verified 2026-09-28**, SteamOS 0.4.1, BUILD_ID `20260925.6191901`,
|
||||
SteamVR 2.18.1: **Tools → Panel switcher** lists SteamVR's open main panels,
|
||||
including panels that are currently hidden. **Show** asks SteamVR to bring one
|
||||
forward. **Open in headset** opens the same switcher as its own panel; choose
|
||||
it again from Steam's dashboard after switching away. Refresh updates the list.
|
||||
This is a list, not thumbnail Exposé.
|
||||
|
||||
|
||||
|
||||
This is our own Python/HTML implementation (`ui/frame_panels.py`), using the
|
||||
Frame's shipped `vrcmd` OpenVR client and gamescope. The headset page uses
|
||||
Chromium (Chromium XR when present, then system Chromium, then the existing
|
||||
Chromium Flatpak). No XSOverlay, OVR Toolkit, WayVR or other overlay application
|
||||
is needed. This dependency boundary also applies to future layout and panel
|
||||
persistence work: platform APIs and bundled libraries are fine; another app
|
||||
must not implement the feature for us.
|
||||
|
||||
The companion runs the helper over SSH. Opening it in the headset installs a
|
||||
copy under `~/.local/share/frame-control/panels/` and starts a loopback HTTP
|
||||
server and an isolated Chromium profile. There is no startup service or global
|
||||
setting change. Close the switcher to stop its server and browser. Other
|
||||
Chromium profiles, Steam and SteamVR are left alone. If the window or runtime
|
||||
closes, use **Open in headset** again.
|
||||
|
||||
The page carries a random, per-process access key in its URL fragment, removes
|
||||
it from the address bar, keeps it in tab session storage for page reloads, and
|
||||
sends it in a header. Panel lists and actions need
|
||||
that key; Host and Origin checks reject other sites. The key permits only
|
||||
listing panels, requesting focus and closing this switcher. Like Mac viewer
|
||||
launch tickets, it is initially readable by another process running as the
|
||||
same Frame user. Panel titles are rendered as text, never HTML. The companion
|
||||
retains its existing request guards. No Mac capture credentials cross this API.
|
||||
|
||||
**Verified:** the real headset page rendered its panel list (image above), its
|
||||
HTTP focus request changed `GAMESCOPE_FOCUSED_APP` to `2000999030`, a request
|
||||
without the key returned HTTP 403, and Close stopped the helper and its browser.
|
||||
Opening an already running switcher requests its focus rather than creating a
|
||||
second one. The companion uses the same list/focus helper. **Unverified:** laser
|
||||
selection while wearing the headset, physical placement, and non-XR Chromium.
|
||||
The API reports that focus was *requested*: another action can take focus before
|
||||
we observe the result. Closed panels are rejected after re-enumeration.
|
||||
|
||||
### Shared-device recheck, 2026-09-29
|
||||
|
||||
**Verified:** the follow-up's atomic `mkdir /tmp/frame-test.lock` attempts
|
||||
failed because another thread held the lock. The existing lock was left alone;
|
||||
no applications were installed, launched or stopped in this follow-up. The last
|
||||
read-only battery check showed 62%, charging. The 180 Python and 8 website tests
|
||||
passed again locally.
|
||||
|
||||
**Unverified in this follow-up:** the prepared browser-button test (Refresh,
|
||||
selection, reload and Close) and repeated OpenXR transition could not run under
|
||||
the shared lock. The device results elsewhere in this page are the earlier
|
||||
2026-09-28 observations, not results from this blocked recheck. In particular,
|
||||
HTTP focus is not evidence of worn-headset laser input. Follow the
|
||||
[shared-device test procedure](testing.md#headset-smoke-test) for the next run.
|
||||
|
||||
## Saved spatial layouts: blocked on the current panel route
|
||||
|
||||
**Verified 2026-09-28**, same build, using a temporary xterm panel with
|
||||
`STEAM_GAME=2000999031` and `FnTable:IVROverlay_028` from
|
||||
`/opt/steamvr/bin/linuxarm64/libopenvr_api.so`:
|
||||
|
||||
| OpenVR call | Result |
|
||||
|---|---|
|
||||
| `FindOverlay("valve.steam.desktopgame.2000999031")` | Success |
|
||||
| `GetOverlayWidthInMeters` | Success, 2.67 m |
|
||||
| `SetOverlayWidthInMeters` (same width) | Success |
|
||||
| `GetOverlayTransformType` | Success, type 5 (`VROverlayTransform_DashboardTab`) |
|
||||
| `GetOverlayTransformAbsolute` | 18 (`WrongTransformType`) |
|
||||
| `SetOverlayTransformAbsolute` (identity rotation, 1.2 m up, 1.5 m forward) | 12 (`PermissionDenied`); type remained 5 |
|
||||
|
||||
The public interface names type 5 **DashboardTab**; SteamVR's dashboard code
|
||||
places these panels through its scene graph. It owns the frame/docking
|
||||
transforms. A successful width setter does not grant permission to restore the
|
||||
position. `vrcmd --dock-overlay world <key>` dispatched a docking request but
|
||||
the dashboard logged `Failed to get SGTransform in setInitialTransformForLocation.
|
||||
Invalid transform ID`. This does not establish working world placement.
|
||||
|
||||
**Inferred:** saving X11 pixel rectangles or Mac window IDs would not restore
|
||||
this spatial arrangement. Mac window IDs also change when an application
|
||||
reopens; viewer tickets and reconnect keys must not go into a layout file.
|
||||
The base Mac stream reconnects after a network break, but that is different
|
||||
from recreating windows and their room positions after a reboot.
|
||||
|
||||
There is consequently no Save/Restore control yet. A durable layout needs a
|
||||
working transform restore path, stable source identity, and a fresh capture
|
||||
permission/ticket flow. The tested gamescope-owned overlay route denies that
|
||||
transform operation. A future Frame Control-owned overlay renderer, or a
|
||||
supported platform API for dashboard frame transforms, needs its own device
|
||||
proof before building layout UI. This is a blocker for the current approach,
|
||||
not a claim that all possible implementations are impossible. Reboot recovery
|
||||
was not tested: the shared headset was not rebooted.
|
||||
|
||||
## Panels during an immersive session
|
||||
|
||||
**Verified 2026-09-28**, same build: our Chromium switcher panel remained in
|
||||
OpenVR's overlay list before, during and after the Frame's shipped `helloxr -g
|
||||
Vulkan` sample. During the test `vrcmd --stats` identified
|
||||
`system.generated.openxr.helloxr.helloxr`, with 242 frame submissions. The test
|
||||
ended only its own sample process; no SteamVR, Steam, power or global settings
|
||||
were changed. The switcher was still selectable afterwards.
|
||||
|
||||
This proves survival of that panel across an OpenXR scene session, **not** that
|
||||
it stayed visibly composited over the scene: OpenVR reported it `not_visible`
|
||||
before, during and after. **Verified:** calling `ShowOverlay` on our
|
||||
*gamescope-owned* switcher overlay returns 12 (`PermissionDenied`). A helper
|
||||
cannot force that panel visible using the public overlay call. Use the
|
||||
switcher/dashboard to request access to it; we do not fight the runtime with a
|
||||
repeated force-focus loop.
|
||||
|
||||
**Verified in a second controlled run:** a live H.264 test-pattern stream from
|
||||
this checkout's Mac helper, through its own SSH tunnel and a temporary Chromium
|
||||
profile, survived the same OpenXR sample (257 scene-frame submissions). Its
|
||||
panel `2000999032` changed from `visible` before launch to `not_visible` during
|
||||
and after the scene. The Mac helper still reported the same `test` stream;
|
||||
captured frames increased from 35 to 232, with 29.5 decoded/drawn fps afterwards.
|
||||
The test did not capture personal Mac windows or inject Mac input. The sample,
|
||||
viewer, temporary profile, tunnel and Mac helper were cleaned up. This proves
|
||||
stream survival, and also shows why it must not be advertised as always visible.
|
||||
|
||||
**Unverified:** persistent visible placement while playing a Steam-launched VR
|
||||
game, Plasma desktop and real Mac-window behavior during that launch, and worn
|
||||
headset input. Other threads were launching games and changing the runtime on
|
||||
the shared device, so those transitions were not treated as controlled evidence.
|
||||
A runtime/X-server restart can destroy the viewer windows; a network reconnect
|
||||
cannot recreate them. No “always visible during games” guarantee is shipped.
|
||||
|
||||
## Keyboard passthrough feasibility
|
||||
|
||||
**Verified 2026-09-28**, same build, using `FnTable:IVRTrackedCamera_006`:
|
||||
`HasCamera(0)` returned success and true. `GetCameraFrameSize` returned 100
|
||||
(`OperationFailed`), with zero dimensions, for all three public frame types
|
||||
(distorted, undistorted and maximum-undistorted), including after acquiring the
|
||||
video service. Acquisition returned success and a handle; release returned 101
|
||||
(`InvalidHandle`). The probe shut down its OpenVR client afterwards. No camera
|
||||
frames were captured and no camera settings were changed.
|
||||
|
||||
**Documented:** the public OpenVR camera interface provides camera frame sizes,
|
||||
intrinsics, projections and streaming handles; these are prerequisites for a
|
||||
spatially aligned camera cutout. See Valve's
|
||||
[OpenVR C API](https://github.com/ValveSoftware/openvr/blob/master/headers/openvr_capi.h).
|
||||
|
||||
**Inferred:** camera presence alone does not establish access to camera pixels.
|
||||
The failed frame-size path blocks a keyboard cutout in our current panel
|
||||
implementation. We have not established a keyboard detector or a calibrated
|
||||
camera-to-panel mapping. Built-in full-room passthrough is not proof of a
|
||||
public, selectively masked camera stream. No keyboard cutout is offered, and
|
||||
no third-party camera/overlay app is substituted for it.
|
||||
|
||||
## Frame Control's media theatre
|
||||
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
# SideQuest and Frame Control
|
||||
|
||||
Researched 2026-09-28. SideQuest is both a Quest discovery website and a desktop
|
||||
sideloading/device-management app. Its Quest labels are **not** evidence that a
|
||||
game works on Lepton: inspect the APK for arm64/OpenXR, Android API requirements,
|
||||
VrApi and Meta services (see [VR APKs](vr-apks.md)).
|
||||
|
||||
## Features worth borrowing
|
||||
|
||||
Desktop evidence is the public [SideQuest source at af2ac70](https://github.com/SideQuestVR/SideQuest/tree/af2ac7043db122bca3c8db18f2b58f1660e9befb),
|
||||
especially [ADB operations](https://github.com/SideQuestVR/SideQuest/blob/af2ac7043db122bca3c8db18f2b58f1660e9befb/desktop-app/src/app/adb-client.service.ts),
|
||||
[drag and drop](https://github.com/SideQuestVR/SideQuest/blob/af2ac7043db122bca3c8db18f2b58f1660e9befb/desktop-app/src/app/drag-and-drop.service.ts),
|
||||
and the [legacy repository index](https://github.com/SideQuestVR/SideQuest/blob/af2ac7043db122bca3c8db18f2b58f1660e9befb/desktop-app/src/app/packages/package.service.ts).
|
||||
Website evidence: [SideQuest](https://sidequestvr.com/) and its public Angular
|
||||
bundle `main-4MMXZRXL.js`, inspected locally without browser automation.
|
||||
No SideQuest implementation code was copied.
|
||||
|
||||
| SideQuest feature | Frame Control before this change | Borrow? / effort |
|
||||
|---|---|---|
|
||||
| Store descriptions, screenshots, banners, trailers, ratings | F-Droid names, icons, compatibility verdicts and reports; no equivalent rich VR store | Yes, from authorised sources; medium. Search and library workers own presentation/artwork. |
|
||||
| OBB expansion-file install | APK-only install | **Implemented helper and CLI**, medium. Essential for games whose assets are separate from the APK. |
|
||||
| App-data backup/restore | Persistent instances and optional keep-data uninstall, no portable save archive | **Implemented private-data helper and CLI**, medium. Back up before updates or experiments. |
|
||||
| File manager (list, upload, download, remove) | General Send to Frame, no Android file browser | Useful later, medium; requires clear instance selection and scoped paths. |
|
||||
| Installed-app management (launch, uninstall, backup) | List, launch, stop, remove, probe | Already mostly covered. Backup added here. |
|
||||
| Update notices / account library | Compatible-version lookup; no source-aware installed update notices | Useful later, medium; needs original version code and source identity recorded on install. |
|
||||
| Custom repositories | Built-in F-Droid catalogue and compatible-version indexes | Separate user-repos worker. Legacy SideQuest source has a fixed SideQuestRepos index; arbitrary current custom-repo support was not verified. |
|
||||
| Drag-and-drop APK/OBB install | APK drag-and-drop already works | OBB backend added here; future UI can call it. UI drop wiring is not included. |
|
||||
| Tags, price, headset filters, reviews | Text search and Lepton verdicts, not Quest headset metadata | Useful, medium; search worker owns filters. Keep source headset claims distinct from tested Frame compatibility. |
|
||||
| Screenshot/video capture and streaming | Frame screenshots/VR capture already present | Reuse existing tools; do not port Quest capture commands. |
|
||||
| Device settings and ADB utilities | Frame/Android display settings, SSH and own-instance tools | Borrow selectively; Quest CPU/GPU presets and wireless-ADB setup do not map directly to Lepton. |
|
||||
|
||||
Priority: expansion files, then save backup/restore. Rich discovery and update
|
||||
notices follow once a permitted metadata source and source/version persistence
|
||||
are available. This patch deliberately exposes CLI/backend operations, leaving
|
||||
shared UI, install(), Steam artwork and launch behavior to sibling work.
|
||||
|
||||
## SideQuest as a source: page-only
|
||||
|
||||
[Terms](https://sidequestvr.com/terms), “Prohibited Activities”, (i) prohibits
|
||||
copying/distributing/disclosing the Service including automated or non-automated
|
||||
“scraping”; (xi) prohibits content access through means other than those provided
|
||||
or authorised by the Service; (xii) prohibits bypassing access restrictions.
|
||||
The terms describe downloading developer-posted games through the Service, but
|
||||
do not establish permission for this third-party API integration.
|
||||
|
||||
[robots.txt](https://sidequestvr.com/robots.txt) requests a three-second crawl
|
||||
delay and disallows `/search/`, `/user/*` and `/sideload/*`. Robots permission
|
||||
would not override the terms. The API host's robots request returned HTTP 403;
|
||||
a request for the first shared website JS chunk also returned 403. No bypass,
|
||||
account token, cookies, browser session or private endpoint was used.
|
||||
|
||||
The homepage publishes `https://api.sidequestvr.com` and
|
||||
`https://cdn.sidequestvr.com`. The website bundle calls `searchApps(...)` and
|
||||
`getApp(id, null)`; their actual HTTP search/detail routes could not be established
|
||||
from the retrieved bundle. Do not invent endpoints. The open-source desktop
|
||||
[install flow](https://github.com/SideQuestVR/SideQuest/blob/af2ac7043db122bca3c8db18f2b58f1660e9befb/electron/app.ts)
|
||||
POSTs `{token: ...}` to `/install-from-key`. It consumes
|
||||
`data.apps[].urls[]`, with `provider` values including `APK`, `OBB`,
|
||||
`Github Release` and `Mod`, and `link_url`. This is a website-issued install-key
|
||||
flow, not evidence of an anonymous download API. It is not implemented here.
|
||||
|
||||
`ui/apk_sources/sidequest.py` implements the shared interface conservatively:
|
||||
|
||||
- `sources()` marks SideQuest `page_only` and explains why.
|
||||
- `search()` raises a user-readable `SourceError` with the browse URL (zero
|
||||
limit returns no rows). It does not invent app results or report a false
|
||||
“no matching games”. The aggregate search UI should surface this source error.
|
||||
- `details()` accepts a numeric listing id and returns its canonical page link,
|
||||
`downloadable: False`, empty versions/tags/headsets and the `images` shape
|
||||
`{icon: None, banner: None, screenshots: []}`. Name is explicitly a listing id;
|
||||
unknown facts, including free/VR status, stay `None`.
|
||||
- `download()` refuses with that page link. Paid/external listings cannot be
|
||||
downloaded by this adapter either. No downloads means no verification claim.
|
||||
|
||||
The JSON fixture records policy evidence, **not a purported live app response**.
|
||||
No listing metadata, artwork URLs, or OBB download URLs were scraped.
|
||||
The requested real SideQuest → OpenXR APK → `frame_android.py info` test is
|
||||
**blocked by the terms**, and was not performed. No alternate source is silently
|
||||
substituted. A future integration needs SideQuest's permission or an expressly
|
||||
supported third-party API, plus recorded search/detail/download fixtures,
|
||||
free/direct-download classification, and size/hash verification. A calculated
|
||||
local SHA-256 alone must not be called publisher verification.
|
||||
|
||||
## OBB files
|
||||
|
||||
```sh
|
||||
python3 ui/frame_android.py install-obb org.example.game main.42.org.example.game.obb
|
||||
python3 ui/frame_android.py install-obb org.example.game main.42.org.example.game.obb patch.42.org.example.game.obb
|
||||
```
|
||||
|
||||
Install the APK first. The named instance must already be running; the helper
|
||||
never launches an app or uses Lepton Development. It requires standard
|
||||
`main|patch.<versionCode>.<package>.obb` filenames and nonempty files, validates
|
||||
the entire batch before transfer, streams each file through SSH into that
|
||||
instance, checks its SHA-256 **inside Android**, then renames it into
|
||||
`/sdcard/Android/obb/<package>/`. `verified: True` here means transfer integrity
|
||||
against the local input, not publisher authentication. Publication is atomic per
|
||||
file, not for the whole batch; retry after a partial batch failure. Existing OBBs
|
||||
with different version codes remain. The filename version must match the game;
|
||||
the current install metadata does not expose its version code for comparison.
|
||||
Restart the game yourself after the transfer if it cached missing expansion data.
|
||||
|
||||
Both read-only SSH attempts to the Frame timed out. Therefore the exact
|
||||
host-side `/sdcard` mapping and persistence of expansion data were **not verified**.
|
||||
`compatdata/<instance>/internal/<package>` is documented as `/data/data/<package>`;
|
||||
it must not be mistaken for `/sdcard`. Using Android's path avoids guessing a
|
||||
host layout, but device verification across restart/update is still required.
|
||||
No OBB file was installed on the Frame during this work.
|
||||
|
||||
## Private app-data backups
|
||||
|
||||
```sh
|
||||
python3 ui/frame_android.py stop org.example.game
|
||||
python3 ui/frame_android.py backup-data org.example.game ./game-save.tar.gz
|
||||
python3 ui/frame_android.py restore-data org.example.game ./game-save.tar.gz
|
||||
```
|
||||
|
||||
Keep the instance stopped throughout either operation; do not launch it from
|
||||
Steam concurrently. The remote guard fails if Podman cannot enumerate containers
|
||||
or reports that instance running. The helpers use `podman unshare` to read/write
|
||||
Android's mapped ownership without changing the live data's permissions.
|
||||
|
||||
The archive covers **only** `compatdata/<instance>/internal/<package>`, not the
|
||||
APK, external `/sdcard/Android/data`, OBBs, keystore, or the full Android snapshot.
|
||||
It contains a package/instance manifest and regular files/directories. Backups
|
||||
are private (0600), validated before publication, and never overwrite an existing
|
||||
backup. Keep them safe: app data can contain credentials and is not encrypted.
|
||||
|
||||
Restore checks the package and instance, rejects absolute/traversing/duplicate
|
||||
paths, links and devices, caps files at 100,000 and content at 20 GiB, and validates
|
||||
again on the Frame. It extracts into a separate directory, preserves numeric
|
||||
ownership, ordinary modes and timestamps, then swaps the private-data directory.
|
||||
Setuid/setgid bits are not restored. The previous directory remains beside it as
|
||||
`.<package>.before-restore-<timestamp>`; the returned `previous` path identifies
|
||||
it. This is an additional recovery copy, not an automatic deletion policy.
|
||||
|
||||
Locally verified: archive round trip including recovery copy, malformed archive
|
||||
rejection, transfer command construction and failure handling. Not verified:
|
||||
real Frame UID mappings/permissions, Android app-level recovery, live FUSE OBB
|
||||
writes or persistence. Backups reject symlinks/special files; an app requiring
|
||||
those needs a separately designed backup format. These CLI features still need
|
||||
a real-device acceptance pass before being exposed as a polished UI workflow.
|
||||
@@ -110,3 +110,11 @@ returns nothing without `cc`, so Frame Control takes the country from
|
||||
- Installing when there's more than one library folder, such as a microSD card.
|
||||
- Uninstalling. `steam://uninstall/<appid>` should open a confirmation in the
|
||||
headset.
|
||||
|
||||
## Optional VR software
|
||||
|
||||
The [VR utilities list](vr-utilities.md#optional-software) is separate from our
|
||||
controls and HUD. It checks software ownership as well as games; paid utilities
|
||||
are installable only when already in the loaded Frame account library. No
|
||||
purchase flow is added. Public reports, local results and ownership are shown
|
||||
separately, and untested tools remain untested.
|
||||
+62
-2
@@ -8,6 +8,7 @@ This covers three directions, plus input:
|
||||
- **PC VR from Linux**: [feasibility and options](linux-vr-streaming.md),
|
||||
including Valve's streaming and USB support. No Linux host tested yet.
|
||||
- **Input**: type and point in the Frame from the Mac or iPhone.
|
||||
- **Live view and Control**: watch a panel flat and tap on it to use it.
|
||||
|
||||
The confidence labels are the same as in [ssh.md](ssh.md).
|
||||
|
||||
@@ -184,13 +185,72 @@ on the Frame, which talks KDE Connect's own LAN protocol to the Frame's
|
||||
and ignores injected pointer motion; `:1` holds apps such as Chromium and
|
||||
takes it. KDE Connect runs on `:1`, so it reaches apps, not Steam's own menus.
|
||||
There's also a `gamescope-0-ei` (libei) socket.
|
||||
- **Not yet tested:** typing and clicking as seen in the headset, and whether
|
||||
it reaches the KDE desktop panel (Plasma is its own session).
|
||||
- Typing through KDE Connect lands in a Chromium panel on `:1` (seen in the
|
||||
panel's own capture, 2026-09-29). It **can't reach panels on `:0`** (Frame
|
||||
Control's own panels, Steam's UI) and, since XTest positions are clamped to
|
||||
`:1`'s 1280×720 root, can't reach beyond that in a bigger window. Control on
|
||||
the live view (below) has neither limit.
|
||||
- **Not yet tested:** whether it reaches the KDE desktop panel (Plasma is its
|
||||
own session).
|
||||
- **Known limit:** keys and clicks typed while the link is reconnecting wait
|
||||
and are sent once it's back, but anything sent in the moment the Wi-Fi
|
||||
drops, before SSH notices, can be lost. Confirming every event would add a
|
||||
round trip to each pointer move.
|
||||
|
||||
## Live view and Control: watch a panel and tap on it
|
||||
|
||||
**Built: Home → Desktop / Headset view → Control.** The live view has two
|
||||
sources:
|
||||
|
||||
- **Headset view**: what the lenses show (SteamVR's mirror, `/dev/video99`). It
|
||||
moves with the wearer's head, so Control makes the view a trackpad: drag to
|
||||
move the pointer, tap to click, press and hold to right-click, two fingers to
|
||||
scroll. With a mouse, moving over the view moves the pointer.
|
||||
- **Desktop**: the app panel in use in the headset, from its own window, so it
|
||||
stays still however the wearer looks around. Control makes taps and clicks
|
||||
land exactly where you put them. Dragging is a mouse drag, press and hold is a
|
||||
right-click, two fingers scroll, and on a computer the mouse, wheel and
|
||||
keyboard work directly on it (⌘ is sent as Ctrl on a Mac). A picker shows any
|
||||
other panel, view only.
|
||||
|
||||
Below the view, a text field and key buttons type on the Frame from a phone.
|
||||
|
||||
How (**verified 2026-09-29**, SteamOS 0.4.1, build 20260925.6191901):
|
||||
|
||||
- **Input goes through gamescope's own injection.** gamescope serves an EIS
|
||||
socket (`/run/user/1000/gamescope-0-ei`; Steam feeds Remote Play input through
|
||||
it), and `libei` 1.4.1 is on the image. [`ui/frame_touch.py`](../ui/frame_touch.py)
|
||||
talks to it with `ctypes`: nothing to install. gamescope offers one device,
|
||||
"Gamescope Virtual Input", with relative and absolute pointer, buttons,
|
||||
scroll and keyboard (Linux key codes; no text capability, so the text field
|
||||
types printable ASCII on a US layout). Its absolute region is unbounded; the
|
||||
pointer uses the focused panel's display coordinates, and gamescope fits each
|
||||
window to its display, so a 1920×1080 window on the 1280×720 `:1` takes
|
||||
positions at two thirds scale. Taps on a 1280×720 page landed on the exact
|
||||
pixel.
|
||||
- **It reaches the panel that has focus** (`GAMESCOPE_FOCUSED_WINDOW` on `:0`'s
|
||||
root), on either X display. In the OpenVR backend focus moves only on SteamVR
|
||||
overlay events (the controller's laser entering or clicking a panel), or to a
|
||||
new panel when none holds it (read from gamescope's `OpenVRBackend.cpp`, seen
|
||||
with `gamescopectl focus_info`, which writes to the journal). Neither
|
||||
`GAMESCOPECTRL_BASELAYER_WINDOW`/`_APPID` nor X focus moves it, and no
|
||||
gamescope command does. So Control follows the wearer: whatever they last
|
||||
used is what your taps reach. A window without a Steam app id (`STEAM_GAME`)
|
||||
gets a connector of its own and doesn't hold focus.
|
||||
- **Keys in a burst can arrive out of order**, so the helper paces them (8 ms
|
||||
apart).
|
||||
- **Known limit:** if focus moves to another panel in the middle of a drag, the
|
||||
release goes to the panel that has focus then. Whether gamescope hands it to
|
||||
the window that got the press isn't known yet. When the session ends, the
|
||||
helper lets go of every button and key it still holds.
|
||||
- **The Desktop picture is the window's own pixels**: `ffmpeg -f x11grab
|
||||
-window_id <window> -i :<display>` works on gamescope's redirected windows,
|
||||
while grabbing the root gives black. It streams as H.264 like the headset view
|
||||
(about 30 fps at 720p).
|
||||
- Tested from the iPhone app (Simulator): a tap on the Desktop view focused a
|
||||
text box in the panel and the text field typed into it; a trackpad move went
|
||||
exactly (+40, +25).
|
||||
|
||||
Our own `uinput` keyboard and mouse would also work (`steamos` is in the
|
||||
`input` group and `/dev/uinput` is group-writable, verified 2026-09-27), and
|
||||
remains the fallback if the bundled KDE Connect ever stops working on a new SteamOS.
|
||||
|
||||
@@ -104,6 +104,20 @@ switch while SSH is down:
|
||||
|
||||
## Headset smoke test
|
||||
|
||||
**Documented shared-device procedure:** before a test installs, launches or
|
||||
stops an application, acquire `ssh frame 'mkdir /tmp/frame-test.lock'`. If it
|
||||
fails, leave that lock alone and continue offline work. Only the thread that
|
||||
acquired it releases it with `ssh frame 'rmdir /tmp/frame-test.lock'`, after
|
||||
cleanup. Keep each device session to a few minutes.
|
||||
|
||||
Check battery capacity and charging state under `/sys/class/power_supply`
|
||||
before and after; keep capacity above 20%. Stop only processes started by the
|
||||
test, remove temporary installs and profiles, and restore the prior dashboard
|
||||
state. Leave Steam and SteamVR running. Do not reboot or change global settings.
|
||||
Record the build, actual interaction results, cleanup and any unworn-headset
|
||||
limits alongside screenshots or logs. These are caller responsibilities; the
|
||||
smoke script below does not acquire this shared lock itself.
|
||||
|
||||
```sh
|
||||
scripts/frame-smoke.sh # needs `ssh frame` to work without a password
|
||||
scripts/frame-smoke.sh --pair # also pairs a throwaway key: approve it in the headset
|
||||
@@ -170,3 +184,29 @@ assistant against an in-process HTTP endpoint with canned responses (no keys or
|
||||
external calls). `tests/e2e/test_agents.py` runs the MCP/HTTP/SSH path against the
|
||||
fake Frame for approved installs, clipboard and file transfer. Headset Chromium
|
||||
rendering and real screenshots still need a device; see [agent evidence](agents.md#evidence-and-limits).
|
||||
|
||||
## Family and comfort
|
||||
|
||||
`tests/test_comfort.py` uses an injected clock, fake headset sensor readings and
|
||||
actions, plus a Node fake of Steam's Home API. It covers warnings before Home,
|
||||
late/suspended sessions, cancellation, failed actions, duplicate alerts, reboot
|
||||
invalidation, per-zone thermal trips and shared on-headset state. The server
|
||||
guards reject invalid session settings before SSH. See
|
||||
[real-device evidence and limits](family-comfort.md#verification).
|
||||
|
||||
## Panel switcher
|
||||
|
||||
`tests/test_panels.py` supplies fake-Frame `vrcmd --overlays` output, checks
|
||||
main-panel filtering (including hidden panels), revalidates closed panels before
|
||||
focus, and drives the headset helper's real loopback HTTP server to test access
|
||||
keys, Host/Origin guards, malformed requests, offline errors and Close. It runs
|
||||
in the normal unit suite without OpenVR or a headset. The fixture format comes
|
||||
from SteamVR 2.18.1, BUILD_ID `20260925.6191901`; it does not simulate rendering.
|
||||
|
||||
On the Frame, run `python3 -` over SSH with `ui/frame_panels.py` on stdin to
|
||||
list panels. `--focus <key>` rechecks the list and requests focus. In Frame
|
||||
Control, **Tools → Panel switcher → Open in headset** exercises installation,
|
||||
Chromium rendering and the same helper through HTTP. Close the switcher after
|
||||
testing. [The recorded device checks](panels.md#frame-controls-panel-switcher)
|
||||
cover actual focus, HTTP guards and an OpenXR sample transition, and separately
|
||||
identify the unverified Steam-game, spatial layout, reboot and laser behaviors.
|
||||
+126
@@ -84,6 +84,132 @@ not being worn, so it did not reach `FOCUSED`).
|
||||
The loader was never the problem: Wolvic's Quest `libopenxr_loader.so` is a
|
||||
Khronos-style loader and found SteamVR through `/vendor`.
|
||||
|
||||
## In the Steam library
|
||||
|
||||
Every successful APK install goes through the same mandatory artwork writer:
|
||||
CLI (including `scripts/install-apk.sh`), upload, catalogue, version finder,
|
||||
web download and source modules calling `frame_android.install`. Native
|
||||
Linux/Windows sideloads also use it, preserving their devkit runtime wiring.
|
||||
A new shortcut is rolled back if artwork fails; failure is never reported as
|
||||
an installed app with a blank tile.
|
||||
|
||||
Artwork preference is **SteamGridDB → source images → generated fallback**.
|
||||
Set the optional free key in Frame Control's **Library artwork settings**, or
|
||||
`STEAMGRIDDB_API_KEY` (`FRAME_STEAMGRIDDB_API_KEY` also works). Environment
|
||||
settings override the saved key. Without a key there are no provider calls or
|
||||
warnings. Saved keys stay in host app data, mode 0600 on POSIX, and are never
|
||||
returned by the settings API or copied to the headset. Exact title matches
|
||||
(including a trailing “VR” variant) use the highest-scored returned static,
|
||||
non-NSFW image in each slot. Provider failures use the next source.
|
||||
|
||||
Sources pass `install(apk_path, artwork={...})`: keys are `grid`, `wide`,
|
||||
`hero`, `logo`, `icon`, `banner`, `feature_graphic`, `screenshot`, or a list
|
||||
`screenshots`. Values are PNG/JPEG bytes or HTTP(S) URLs (12 MiB and
|
||||
4096×4096 pixels maximum; any PNG depth or interlace, since the Frame's
|
||||
Chromium decodes them). URLs must resolve to public addresses, follow at most
|
||||
three redirects and share one deadline per install. Any source that fails,
|
||||
for any reason, becomes a warning and generated art. Banners and feature graphics supply hero/wide art;
|
||||
screenshots are the next fallback. Source images are cached for refresh.
|
||||
All images are fitted to 600×900 portrait, 920×430 wide, 3840×1240 hero,
|
||||
1280×480 logo and 256×256 icon. Explicit logos retain transparency.
|
||||
Photo-based portrait, wide and hero slots are JPEG: Steam takes at most
|
||||
12 MiB per slot, and on the Frame (2026-09-28) a noise-heavy 3840×1240 hero
|
||||
came to more than 12 MiB as PNG, 3.7 MB as JPEG (2.7 s to render); a
|
||||
landscape photo hero 5.6 MB as PNG, 0.76 MB as JPEG (0.75 s). A render that
|
||||
still fails is retried once with generated art. Steam keeps a slot's `.png`
|
||||
and `.jpg` side by side, so each slot is cleared before it is set.
|
||||
|
||||
Generated art uses the APK icon, a dominant-colour gradient, a blurred
|
||||
backdrop and large foreground icon with shadow. Steam's Chromium canvas and
|
||||
Motiva Sans render real text consistently regardless of the host OS; no
|
||||
Pillow, host font installation or bitmap font is needed. The hero has no
|
||||
title; the generated logo is a transparent title. APKs with no usable icon
|
||||
get a typographic monogram. The desktop package includes the renderer.
|
||||
|
||||
Backfill installed Android apps without reinstalling or stopping them:
|
||||
|
||||
```sh
|
||||
python3 ui/frame_android.py refresh-art org.godotengine.open_saber_plus
|
||||
python3 ui/frame_android.py refresh-art --all
|
||||
```
|
||||
|
||||
Devkit titles installed by Frame Control have the same command,
|
||||
`python3 ui/frame_titles.py refresh-art ID|--all`. The settings panel's
|
||||
refresh covers both. The API is `POST /api/android` with
|
||||
`{"action":"refresh-art","all":true}` (apps and titles) or a `package`, and
|
||||
`POST /api/titles` with `{"action":"refresh-art","id":…}`; each returns a
|
||||
background job. Batch results retain per-item errors, and the CLIs exit
|
||||
nonzero if any failed. Apps and titles without complete artwork show **Add
|
||||
artwork** and `list` prints the command. Only entries marked `art_pending` at
|
||||
install (a title Steam registered after an install made while it wasn't
|
||||
running) are backfilled automatically, when Frame Control lists them with
|
||||
Steam running (at most every five minutes), and that backfill only fills
|
||||
slots Steam has no art for: names, icons, flags and any art the user set are
|
||||
kept. Older installs without the flag are refreshed only on request.
|
||||
|
||||
Steam's app overviews carry no `devkit_gameid` (checked 2026-09-28, build
|
||||
20260925.6191901, on every non-Steam shortcut). A title's shortcut is found by
|
||||
its saved id, or by an executable or start folder inside
|
||||
`~/devkit-game/<id>/`, read from `appDetailsStore`; never by display name.
|
||||
That the devkit shortcut's exe/start folder sit inside the title folder is
|
||||
inferred from `docs/sideloading.md` (`proton waitforexitandrun
|
||||
"/home/steamos/devkit-game/<id>/<exe>"`), not yet seen in app details.
|
||||
Devkit titles keep the VR flag Steam gave them.
|
||||
|
||||
**Verified on build 20260925.6191901, SteamVR 2.18.1 (2026-09-28):** both
|
||||
Open Saber Plus and SuperTux were backfilled. Steam's cached portrait, wide,
|
||||
hero and logo PNGs have the dimensions above; each shortcut points at its
|
||||
256×256 icon. This Frame client mishandles custom-art type 4 (documented as
|
||||
Icon), overwriting the wide capsule; the implementation uses custom types
|
||||
0–3 and **SetShortcutIcon** separately.
|
||||
|
||||
Steam accepts display name, executable/start directory, icon, VR flag and
|
||||
sort-as name. Android apps join **Android**, immersive apps also **Android
|
||||
VR**; native sideloads join **Sideloaded**. Existing collection members and
|
||||
unrelated collections are preserved (both games retained **Played**).
|
||||
Dynamic/read-only collection conflicts produce warnings. The native notes
|
||||
API supports a managed **Installation details** note (package, version and
|
||||
source) while preserving other notes. Notes are keyed by sanitized shortcut
|
||||
name, so Steam itself cannot distinguish equal-name shortcut notes. No
|
||||
supported shortcut description/store-page, developer/publisher, release
|
||||
metadata or custom achievement API was found; these are not fabricated.
|
||||
|
||||
The launcher supervises Lepton and handles TERM/INT/HUP and normal exit by
|
||||
stopping its own container and child process group. A lock refuses duplicate launches;
|
||||
a container still running while the lock is free was orphaned by a killed
|
||||
launcher and is stopped before the new launch. Lepton doesn't inherit the
|
||||
lock. Orphan recovery only stops the app's own, deterministically named
|
||||
container; a Lepton host process whose launcher was killed before it created
|
||||
the container may linger briefly. Removing an app or title still deletes its files when Steam isn't
|
||||
running; tidying Steam's collections and artwork is best effort. Steam Stop uses `TerminateApp` with the exact
|
||||
64-bit game ID string. Frame Control's Stop additionally has a direct-container
|
||||
fallback. The stable instance ID and compatdata paths remain unchanged.
|
||||
|
||||
Lepton normally forwards the instance `SteamAppId` to Android, causing
|
||||
SteamVR to associate the scene with a different, artwork-less app. The
|
||||
launcher uses Lepton's supported `LEPTON_ENV_SteamAppId` passthrough to send
|
||||
the actual shortcut ID to Android while retaining the stable container ID.
|
||||
**Verified:** Open Saber was alive 22 seconds after Steam Play, SteamVR
|
||||
identified `steam.app.3346865537`, and its scene appeared in the headset
|
||||
capture without the previous blank Resume tile. Steam Stop then removed its
|
||||
tracked process and stopped the container. An earlier 32-second session was
|
||||
also tracked until Steam Stop. No global standby or dashboard overrides were
|
||||
installed; wear detection and other user-opened overlays still apply.
|
||||
|
||||
**SuperTux limitation:** Steam launched and tracked it, but SDL crashed during
|
||||
activity creation because Lepton lacks `ClipboardManager`. Its container
|
||||
cleaned up on exit after about 17 seconds. Consequently sustained SuperTux
|
||||
Play/Stop and its VR scene could not be verified. This is an APK/runtime
|
||||
compatibility failure, separate from library presentation.
|
||||
|
||||
Evidence is under `/tmp/vrlib-evidence/` on the development Mac: final artwork
|
||||
preview and three design passes, `steam-cache-final.log`,
|
||||
`steam-details-targets.json`, `opensaber-identity-session.log`,
|
||||
`opensaber-identity-headset.png`, and `supertux-lepton.log`. The preview is
|
||||
rendered artwork, not a Steam UI screenshot; CDP screenshot capture timed
|
||||
out. Authenticated SteamGridDB, Windows/Linux packaged builds and the sibling
|
||||
source-search endpoint remain unverified (the public install seam is tested).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Meta entitlement.** Apps that call the Oculus Platform SDK
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
# VR comfort and performance
|
||||
|
||||
Frame Control owns its HUD and telemetry. They use SteamVR/OpenVR, gamescope,
|
||||
Python and xterm already on the Frame, plus the Frame's sensors. No feature
|
||||
requires fpsVR, OVR Advanced Settings, XSOverlay or another third-party app.
|
||||
The software list is a separate, optional convenience.
|
||||
|
||||
This is **part of [#25](https://github.com/saphid/frame-control/issues/25)**,
|
||||
not completion of the issue. Playspace controls remain blocked on the device
|
||||
checks below. The PR stays draft.
|
||||
|
||||
## Our performance HUD
|
||||
|
||||
On **Home → VR comfort and performance**, the app shows a timestamped sample
|
||||
with each status refresh (30 seconds, or Refresh). **Open HUD in headset**
|
||||
starts our text HUD as a gamescope panel, refreshed every two seconds. In the
|
||||
SteamVR dashboard, select **Frame Control HUD**, then Float in World or dock
|
||||
it to a controller. **Close HUD**, or closing its terminal, ends it. Opening
|
||||
it twice reuses the existing process.
|
||||
|
||||
| Value | Meaning and source |
|
||||
|---|---|
|
||||
| Compositor FPS / period | Differences between two `IVRCompositor_029::GetFrameTiming` frame indices and monotonic compositor timestamps, sampled 200 ms apart. Output cadence, not game FPS or a long-term average. |
|
||||
| Application FPS | Reciprocal of OpenVR's client frame interval. Unavailable if there is no positive interval; not inferred from refresh rate. |
|
||||
| Render GPU time | OpenVR total render GPU milliseconds, not GPU utilisation. |
|
||||
| Compositor CPU | OpenVR compositor render CPU milliseconds, not game CPU time. |
|
||||
| System CPU | `/proc/stat` busy-time delta across the sample, with guest time counted once and iowait treated as idle. |
|
||||
| GPU clock | `3d00000.gpu/cur_freq`, converted from Hz to MHz; frequency is not load. |
|
||||
| Hottest sensor / battery | Existing thermal-zone and battery sysfs reads from `frame_status.py`. |
|
||||
|
||||
OpenVR uses background application mode, which does not start SteamVR or keep
|
||||
it running. This mode also returned live timing in a read-only device probe.
|
||||
|
||||
Missing sensors, a stopped or incompatible SteamVR runtime, and non-advancing
|
||||
frame indices display **Unavailable**, never invented zero FPS. Failed status
|
||||
refreshes clear the HUD card rather than keeping a stale live-looking sample.
|
||||
The HUD itself adds CPU/GPU work; it is a diagnostic, not a zero-overhead benchmark.
|
||||
|
||||
**Verified 2026-09-28**, SteamOS 0.4.1, build `20260925.6191901`, SteamVR
|
||||
2.18.1: the exact OpenVR interface and 192-byte timing layout returned advancing
|
||||
frame indices and live GPU/CPU timing. Sensor reads, creation of overlay
|
||||
`valve.steam.desktopgame.2000250025`, duplicate-open handling, and closing the
|
||||
HUD passed. The temporary probe overlay disappeared after its process closed.
|
||||
[Sanitized device sample](evidence/vr-utilities/device.json).
|
||||
|
||||
**Unverified:** visual placement while wearing the headset, controller docking,
|
||||
and overhead during gameplay. The companion card was checked in the attached
|
||||
preview using live Frame data. Creating an overlay does not establish that it
|
||||
was visible to the wearer.
|
||||
|
||||
The optional HUD copies only `frame_status.py` and `frame_vr.py` into
|
||||
`~/.local/share/frame-control/vr/`. It tags only the window whose X11 PID matches
|
||||
its own xterm, avoiding other threads' windows. Stop checks both PID and Linux
|
||||
process start time before sending SIGTERM. It does not stop Steam or SteamVR,
|
||||
edit their settings, install a service, or need sudo.
|
||||
|
||||
## Playspace, seated height and recenter: paused
|
||||
|
||||
**Verified 2026-09-28:** SteamVR exposes `IVRChaperoneSetup_006` and
|
||||
`IVRChaperone_004`. An initial 1 cm seated zero-pose translation committed,
|
||||
read back and restored numerically. A later trial, while the shared Frame was
|
||||
in use, changed universe IDs after commits and returned transforms that did
|
||||
not match the requested write or restore. Journal entries reported
|
||||
`CommitWorkingCopy`, `VREvent_ChaperoneUniverseHasChanged` and
|
||||
`VREvent_ChaperoneRoomSetupCommitted`. The collision-bound arrays and play-area
|
||||
size matched in the saved before/after records, but the origin matrices did not.
|
||||
|
||||
Alex confirmed the headset was in use and asked to pause control tests. No
|
||||
further playspace writes were made. The exploratory control implementation was
|
||||
removed from the shipping API; `recenter`, `adjust` and `restore` are rejected.
|
||||
This is an **unresolved feasibility check**, not evidence that OpenVR controls
|
||||
cannot work. Concurrent use and the Frame driver's coordinate-system handling
|
||||
still need to be separated.
|
||||
|
||||
The probe's original and last-read poses remain in the Frame's
|
||||
`~/.local/share/frame-control/vr/comfort.json` for investigation. This draft
|
||||
neither reads nor applies that baseline. Do not blindly replay it into a room
|
||||
that may have changed. No recenter test was reached in the later trial.
|
||||
|
||||
Before adding controls, on an idle Frame:
|
||||
|
||||
1. Establish current room and tracking state, and inspect the retained probe
|
||||
evidence before considering any restoration.
|
||||
2. Prove seated and standing height/move operations in the Frame driver's
|
||||
current coordinates, including delayed readback, coordinate rebasing and
|
||||
recovery. Show that the physical safety boundary stays correct.
|
||||
3. Verify recenter independently, and test a full apply/restore cycle plus a
|
||||
concurrent room-change refusal. Add a fake OpenVR test for those contracts.
|
||||
4. Check the apparent result in a seated and a standing app before exposing UI.
|
||||
|
||||
**Documented:** seated and standing are tracking origins selected by an app;
|
||||
changing a seated origin cannot force every game to support seated play.
|
||||
|
||||
**Inferred from the installed SteamVR defaults:** there is no generic snap-turn
|
||||
or locomotion-vignette setting. `dashboard.verticalOffsetCm_2` and
|
||||
`steamvr.panelMaskVignette` affect panels, not the player's height or game
|
||||
locomotion. The app gives game-setting hints for snap-turn, teleport movement
|
||||
and movement vignette instead of writing these unrelated settings.
|
||||
|
||||
## Optional software
|
||||
|
||||
**Verified 2026-09-28 (same build):** a read-only query of Steam's loaded
|
||||
`appStore.allApps` found none of these five apps on the Frame account. The query
|
||||
includes software, which the existing games-only library filter excludes.
|
||||
Steam's public app-details API listed only Desktop+ as free. No software was
|
||||
purchased, installed or launched during these ownership checks.
|
||||
|
||||
| Utility | Local Frame status | Public evidence / optional source |
|
||||
|---|---|---|
|
||||
| OVR Advanced Settings (1009850) | **Untested**, not owned | Steam edition is paid. Developer's [free source and releases](https://github.com/OpenVR-Advanced-Settings/OpenVR-AdvancedSettings). No verified Frame result in this work. |
|
||||
| XSOverlay (1173510) | **Untested**, not owned | Supplied research attributes Proton support with tweaks to [Road to VR](https://www.roadtovr.com/valve-steam-frame-review/). This is a public report, not our verification. |
|
||||
| OVR Toolkit (1068820) | **Untested**, not owned | Same [public Frame report](https://www.roadtovr.com/valve-steam-frame-review/); no local verification. |
|
||||
| fpsVR (908520) | **Untested**, not owned | [Steam listing](https://store.steampowered.com/app/908520/). No Frame-specific result established in the supplied research. General PC VR reviews do not verify Frame support. |
|
||||
| Desktop+ (1494460) | **Untested**, free | [Developer source](https://github.com/elvissteinjr/DesktopPlus). No verified Frame result in this work. |
|
||||
|
||||
**Documented (supplied research):** the XSOverlay/OVR Toolkit claims are kept as
|
||||
attributed leads. **Verified source check 2026-09-28:** fetching the linked
|
||||
review returned HTTP 200 and the expected review title, but neither utility name
|
||||
appeared in the fetched HTML or extracted text. The claims could not be
|
||||
corroborated from that page; this is not proof of incompatibility. They do not make those apps dependencies or mark them
|
||||
locally verified. No paid utility is auto-acquired. A server-side check blocks
|
||||
installation through the Steam endpoint if the paid utility is absent from the
|
||||
loaded Frame library; an empty/unavailable library fails closed. Desktop+ uses
|
||||
the existing free Steam install flow, which may need a license confirmation in
|
||||
the headset. None is advertised as known-good without local evidence.
|
||||
|
||||
Compatibility reports reuse the existing `compat-db` storage and validation
|
||||
with `package: "steam:<appid>"`, rather than colliding with Android package IDs.
|
||||
The optional list shows the latest `works`, `issues` or `broken` report with its
|
||||
date, build and notes, separately from public sources and ownership. With no
|
||||
report the status stays **untested**. No shared database schema change or
|
||||
production deployment is needed; no reports were published by this work.
|
||||
|
||||
## Validation and review
|
||||
|
||||
166 unit tests and 8 website tests passed locally. The new fake-Frame cases
|
||||
passed in GitHub CI (Docker was unavailable locally). Desktop and phone-width
|
||||
preview checks passed. The two independent review attempts did not produce a
|
||||
verdict; [commands, real exit statuses and limitations](evidence/vr-utilities/review.md).
|
||||
@@ -75,6 +75,34 @@ All test media were generated by us. No paid content or DRM was involved.
|
||||
|
||||
|
||||
|
||||
**Verified end to end, 2026-09-29** (same build; headset unworn): uploads
|
||||
through the HTTP API, the web UI and `scripts/push-vr-video.sh`, all played by
|
||||
the owned player. Our generated test files:
|
||||
|
||||
| Case | Result |
|
||||
|---|---|
|
||||
| H.264 half-SBS 1920×1080 with AAC, theatre | 240/240 frames in 8.09 s; audio stream "Frame Control Media" in PulseAudio |
|
||||
| H.265 half-OU 1920×1080 | 180/180 frames in 6.03 s; red left eye, cyan right |
|
||||
| H.264 full-SBS 3840×1080, `stereo_mode=left_right` only | Detected from metadata; 150/150 frames in 5.03 s |
|
||||
| H.264 1280×720, explicit 2D | 150/150 frames in 5.02 s |
|
||||
| SBS PNG, OU JPEG (theatre) | Correct eye in each capture |
|
||||
| 3,000-Gaussian `.splat` | Rendered in about 5 s, then held until Stop |
|
||||
| 2D file on Auto, `_SBS_OU` file, HEIC, VP9 | Refused with the documented message |
|
||||
| Second Play while one runs | Refused: "Stop the current media…" |
|
||||
|
||||
Stop always left the unit inactive, and no player process remained.
|
||||
|
||||
**Standby (verified):** an unworn Frame turns its displays off a few seconds
|
||||
after it wakes. `SetOverlayRaw` then returns `RequestFailed` (23). The
|
||||
first run's movie died there. The player now drops frames while the headset
|
||||
is in standby, keeps the audio and its clock going, and resumes the picture
|
||||
when the headset wakes. The 8 s movie above dropped 40 frames and finished.
|
||||
Stills and the theatre surround are re-sent after waking. Five minutes
|
||||
without an accepted frame is reported as an error. Headset-view captures
|
||||
taken during standby show a flat dark frame, not our screen.
|
||||
|
||||
|
||||
|
||||
**Verified failed route:** GStreamer 1.24.2's `playbin` selected
|
||||
`v4l2h264dec`, delivered the first RGBA sample and then segfaulted (exit 139)
|
||||
in the basic appsink probe and the OpenVR probe. We do not ship that route.
|
||||
|
||||
Reference in new issue
Block a user