From a6137351d959be72799f8ffe302ca235fed93a4c Mon Sep 17 00:00:00 2001 From: saphid <4596216+saphid@users.noreply.github.com> Date: Mon, 28 Sep 2026 22:23:19 +1000 Subject: [PATCH] docs: record Linux VR client blockers and streaming options --- docs/apks.md | 6 + docs/evidence/linux-vr/2026-09-28.txt | 47 +++++++ docs/linux-vr-streaming.md | 195 ++++++++++++++++++++++++++ docs/streaming.md | 10 +- 4 files changed, 255 insertions(+), 3 deletions(-) create mode 100644 docs/evidence/linux-vr/2026-09-28.txt create mode 100644 docs/linux-vr-streaming.md diff --git a/docs/apks.md b/docs/apks.md index 8bca60a..56b8c25 100644 --- a/docs/apks.md +++ b/docs/apks.md @@ -5,6 +5,12 @@ in **Lepton**, Valve's Waydroid-based container. Lepton is built for games, not general Android use ([GamingOnLinux](https://www.gamingonlinux.com/2026/09/lepton-from-valve-to-run-android-games-on-linux-is-now-open-source/)). +**VR streaming clients:** WiVRn 26.9 and ALVR 20.14.1 install, but both fail +OpenXR instance creation on the checked Frame because its Android runtime lacks +`XR_KHR_convert_timespec_time` (**verified** 2026-09-28, SteamOS 0.4.1, +BUILD_ID 20260925.6191901). They are not Frame Control dependencies. See +[the feasibility results and options](linux-vr-streaming.md). + ## Install from the Mac: one app, one Lepton instance (verified 2026-09-25) Use Frame Control's **Android apps** section (search, Install, Test, Report), drop diff --git a/docs/evidence/linux-vr/2026-09-28.txt b/docs/evidence/linux-vr/2026-09-28.txt new file mode 100644 index 0000000..ab71697 --- /dev/null +++ b/docs/evidence/linux-vr/2026-09-28.txt @@ -0,0 +1,47 @@ +Frame client feasibility, 2026-09-28 +SteamOS VERSION_ID=0.4.1 BUILD_ID=20260925.6191901; uname -m=aarch64 +SteamVR: vrserver log reports 2.18.1; process 2256 remained alive across checks. +Base: dcf9689f6459e576d35fc507eb702ca6b2bf4dad (main). +Unmodified upstream release APKs; installed with ui/frame_android.py install APK --vr. +No Linux host attached. No pairing, streamed video, input, audio or worn-headset checks. +Selected logcat lines only; timestamps in Android logs are UTC. + +WiVRn-release.apk +https://github.com/WiVRn/WiVRn/releases/tag/v26.9 +sha256=1df6649ec77224fcc821af0ab4897222bdf3d7eb6ce6ad636461336724111331 +E/OpenXR-Loader( 1140): Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_KHR_convert_timespec_time +I/WiVRn ( 1140): [2026-09-28 12:07:58.987] [WiVRn] [info] Failed to create OpenXR instance version 1.1.58: XR_ERROR_EXTENSION_NOT_PRESENT +E/OpenXR-Loader( 1140): Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_KHR_convert_timespec_time +I/WiVRn ( 1140): [2026-09-28 12:07:59.034] [WiVRn] [info] Failed to create OpenXR instance version 1.0.58: XR_ERROR_EXTENSION_NOT_PRESENT +E/WiVRn ( 1140): [2026-09-28 12:07:59.035] [WiVRn] [error] Error during initialization: Failed to create OpenXR instance: XR_ERROR_EXTENSION_NOT_PRESENT + +alvr_client_android.apk +https://github.com/alvr-org/ALVR/releases/tag/v20.14.1 +sha256=be68feeb02665e3d69f1cdbcabf38ea4d15c42868a7ec6b5e698dbefee4e4e36 +E/OpenXR-Loader( 1139): Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_KHR_convert_timespec_time +I/RustStdoutStderr( 1139): Error [GENERAL | xrCreateInstance | OpenXR-Loader] : LoaderInstance::CreateInstance, no support found for requested extension: XR_KHR_convert_timespec_time +E/[ALVR NATIVE-RUST]( 1139): ALVR panicked: What happened: +E/[ALVR NATIVE-RUST]( 1139): panicked at alvr/client_openxr/src/lib.rs:220:10: +E/[ALVR NATIVE-RUST]( 1139): called `Result::unwrap()` on an `Err` value: ERROR_EXTENSION_NOT_PRESENT + +Cleanup verified: both app directories, compatdata directories and Steam shortcuts absent. +Both test containers stopped and removed. No Steam/SteamVR restart or global setting changes. + +Valve release notes fetched from ISteamNews/GetNewsForApp/v2 (appid=250820). + +SteamVR Beta Updated - 2.18.1 +https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1844751498219787 +Added tethered Quest support over USB (must be used with Steam Link Beta) + +Introducing SteamVR 2.17 +https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1843481262693486 +Adds initial support for USB streaming. Note: Requires new Steam Client Beta. +Fix crash using Steam Link on Linux when games submit invalid textures. +Improve streaming recovery when using Steam Link on Linux. + +SteamVR Beta Updated - 2.17.8 +https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1842212951314598 +Fix crash using Steam Link on Linux when games submit invalid textures. +Improve streaming recovery when using Steam Link on Linux. +Adds initial support for USB streaming. +USB streaming can be used without WiFi by opting into the Steam Client Beta. diff --git a/docs/linux-vr-streaming.md b/docs/linux-vr-streaming.md new file mode 100644 index 0000000..af1789a --- /dev/null +++ b/docs/linux-vr-streaming.md @@ -0,0 +1,195 @@ +# PC VR streaming from Linux + +**Recommendation, 2026-09-28:** test Valve's current SteamVR/Steam Link path +on a Linux gaming PC before building another streamer. Valve now documents +Linux streaming fixes and USB support. We have no Linux host attached, so +Linux-to-Frame VR streaming remains **unverified here**. + +This is the feasibility and options report for +[#24](https://github.com/saphid/frame-control/issues/24), not a shipped streaming +feature. Frame Control's features must use our own implementation or standard +platform components. WiVRn and ALVR are research comparisons, not dependencies. +An optional install shortcut is the most we would offer for a third-party app. +Our own streamer requires Alex's choice before implementation. + +## What was checked on the Frame + +**Verified** on 2026-09-28: aarch64, SteamOS **0.4.1**, BUILD_ID +`20260925.6191901`, SteamVR **2.18.1**. Version and build are recorded separately; +earlier docs associate this build with other SteamOS version labels. + +| Client | Installation | Runtime result | +|---|---|---| +| WiVRn **26.9**, upstream `WiVRn-release.apk` | API 29, arm64-v8a; installed in its own immersive Lepton instance | OpenXR instance creation fails: missing `XR_KHR_convert_timespec_time`. Both 1.1.58 and 1.0.58 attempts return `XR_ERROR_EXTENSION_NOT_PRESENT` | +| ALVR **20.14.1**, upstream `alvr_client_android.apk` | API 26, arm64-v8a; installed in its own immersive Lepton instance | Same missing extension. Client panics at `client_openxr/src/lib.rs:220` with `ERROR_EXTENSION_NOT_PRESENT` | + +The [evidence excerpt](evidence/linux-vr/2026-09-28.txt) includes APK SHA-256s, +upstream release links, loader errors and cleanup results. These are failures +before an OpenXR session, not successful VR clients. WiVRn was launched twice; +ALVR's container remained up despite its client panic. Container liveness alone +does not establish VR compatibility. + +Both APKs already declare `MAIN` and `LAUNCHER`. They were installed unmodified +using this branch's existing `python3 ui/frame_android.py install APK --vr`, +then launched through their Steam shortcuts. Logs came from the instance's +`podman exec … /system/bin/logcat`; the user journal also retained WiVRn's errors +after its container exited. No headset was worn and no host was connected. + +All test app files, compatdata, shortcuts and containers were removed afterwards. +SteamVR's original process remained running. No global settings changed. + +### Relation to the VR APK branch + +**Documented from source:** [PR #20](https://github.com/saphid/frame-control/pull/20) +was read, not edited (branch inspected at +[`038dcd4`](https://github.com/saphid/frame-control/commit/038dcd48cd75336f6a86c63c7878bfc9c52deec9)). +Its compatibility layer handles OpenXR version negotiation, some controller +profiles and refresh-rate requests. It does **not** implement +`XR_KHR_convert_timespec_time`. Its launcher fix is unnecessary for these APKs. +This report has **no unmerged code dependency** on that PR, and neither APK was +tested with its layer injected. + +**Documented from upstream source:** WiVRn requests the extension in +[`application.cpp`](https://github.com/WiVRn/WiVRn/blob/bbc6e4cc36c355fa6180980abd231673dc15115d/client/application.cpp#L1286) +and uses it to convert `CLOCK_MONOTONIC` into `XrTime` in +[`instance::now()`](https://github.com/WiVRn/WiVRn/blob/bbc6e4cc36c355fa6180980abd231673dc15115d/client/xr/instance.cpp#L335). +ALVR also [requests it unconditionally](https://github.com/alvr-org/ALVR/blob/a9f6542fa507a841f40ab4f3fcb531427cd02550/alvr/client_openxr/src/lib.rs#L188). +Simply deleting the extension request or returning made-up timestamps would +not prove correct tracking or timing. A real fix needs a valid clock mapping +and further runtime tests. No such patch was made. + +### Native SteamOS aarch64 clients + +**Verified:** the Frame has a native OpenXR runtime manifest at +`~/.config/openxr/1/active_runtime.json`, pointing to SteamVR's +`bin/linuxarm64/vrclient.so`. + +**Documented:** WiVRn's [26.9 README](https://github.com/WiVRn/WiVRn/blob/bbc6e4cc36c355fa6180980abd231673dc15115d/README.md) +describes its Linux client as debugging-only, without audio or hardware decode. +ALVR 20.14.1's [non-Android decoder](https://github.com/alvr-org/ALVR/blob/a9f6542fa507a841f40ab4f3fcb531427cd02550/alvr/client_core/src/video_decoder/mod.rs) +returns no decoded frames. The inspected releases ship Android clients, not a +ready-to-run native Frame client. + +**Inferred:** a native port is possible research, but neither release offers a +demonstrated native alternative to the blocked APKs. Native builds, native +extension enumeration, hardware decoding and audio were **not tested**. The +Android extension failure does not establish that the native runtime lacks it. + +## (a) Valve's own path — recommended first + +**Documented**, from Valve's release notes rather than launch-window reports: + +- [SteamVR 2.17.8 beta](https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1842212951314598) + says “Fix crash using Steam Link on Linux when games submit invalid textures” + and “Improve streaming recovery when using Steam Link on Linux.” It also + adds initial USB streaming, with Steam Client Beta required to use USB + without Wi-Fi. +- [SteamVR 2.17 release](https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1843481262693486) + repeats Linux streaming fixes and initial USB support. USB is no longer + solely a claim about an old beta, but version/channel requirements still + need checking on the actual host. +- [SteamVR 2.18.1 beta](https://steamstore-a.akamaihd.net/news/externalpost/steam_community_announcements/1844751498219787) + adds USB-tethered **Quest** support with Steam Link Beta. That entry is not + proof of a Frame/Linux combination. +- The [Steam Link page](https://store.steampowered.com/app/353380/Steam_Link/) + lists Linux desktop clients, while its Quest VR requirements still say + Windows 10 or newer. Desktop Steam Link support is not equivalent to VR host + support, and the Quest requirements are not a Frame support matrix. + +**Inferred:** Valve has a Linux VR streaming path worth testing. The old blanket +claim “Linux cannot stream VR” is no longer justified by the evidence. These +release notes do not establish which Linux GPU/driver/Frame combinations work. +USB changes the transport; it does not by itself prove host encoder support. + +**Not verified:** Linux host discovery, pairing, wireless or USB streaming, +stereo rendering, controllers, haptics, audio, latency, or a game. Frame-only +inspection cannot establish any of these. Flat Remote Play and a desktop shown +on a panel are not substitutes for this test. + +Next test, once a Linux gaming PC is available: record distro, GPU/driver, +Steam client channel/version and SteamVR version; use the Frame's built-in +Steam connection flow, first wirelessly and then over a data-capable USB cable. +Launch a free OpenXR sample or developer-consented VR game. Verify stereo, +head/controller tracking, haptics and audio while worn; retain both ends' logs +and measure latency and recovery after a link interruption. Restore any test +channel changes. Do not change the shared headset's channel just for this report. + +If that works, Frame Control can provide our own host checks, setup guidance +and session controls around Valve's existing platform. First establish which +controls have a usable interface; no stable automated pairing API has been +verified. **Estimate (inferred):** 2–5 engineer-days for the hardware feasibility +pass; another 1–2 weeks for a small integration if those interfaces exist. + +## (b) Our own streaming — proposal only + +This is a new VR transport and device integration, not a desktop capture feature. +A plausible first target is **one Linux GPU family, one host, one Frame**, using +SteamVR on both ends. Our host driver would expose a remote HMD/controllers, +receive poses and inputs, and obtain stereo textures for hardware encoding. +Our Frame OpenXR app would decode, submit the correct eye views and render poses +at predicted display times, and return tracking/input. SteamVR/OpenXR, bundled +codec/transport libraries and platform GPU APIs fit the ownership rule; a +WiVRn/ALVR/Monado server dependency would not. + +**Inferred design risks:** Linux SteamVR texture-sharing/driver interfaces and +Frame decode-to-GPU interoperability need a spike before committing to this +architecture. Sending an already-composited desktop mirror loses the stereo, +pose and timing information we need. Late reprojection, clock conversion, +backpressure, controller bindings, audio sync and reconnects are substantial +work. A runtime shim must not assume `XrTime` equals monotonic nanoseconds. + +### Reuse from `mac-in-headset` + +**Documented from our code**, read-only at +[`1b90c64`](https://github.com/saphid/frame-control/commit/1b90c64b73bace54c63a3aae154c5d29a9448d72): + +- Reuse the ideas for low-latency encoding without B-frames, dropping work + before encoding, bounded queues, keyframe recovery, adaptive bitrate, + per-frame timing and authenticated session setup. +- Its VideoToolbox encoder and ScreenCaptureKit capture are macOS-specific. + Linux needs a new GPU encoder path (for example VA-API or NVENC via bundled + libraries) and VR texture capture, not a port of window capture. +- Its WebSocket over SSH is useful for a first controlled transport experiment + and control messages. Reliable TCP can stall behind lost packets; a VR media + path needs measured deadline behaviour, likely datagrams with loss recovery + using an ordinary bundled transport library. Do not invent cryptography. +- Its Chromium/WebCodecs panel viewer is not a VR client. That branch reports + software H.264 decoding and occasional long Wi-Fi stalls on the Frame. + Its desktop latency measurements are not motion-to-photon measurements or + evidence that a 90/120 Hz stereo stream will work. + +**Size/effort estimate (inferred, one experienced full-time engineer, hardware +available):** + +| Phase | Deliverable / stop condition | Effort | +|---|---|---| +| Feasibility | Linux driver texture access, Frame hardware decode into OpenXR, pose/clock loop; stop if any cannot meet frame deadlines | 2–4 weeks | +| First end-to-end prototype | One GPU/codec, stereo sample over a controlled LAN, head/controllers, logs and teardown | 4–8 additional weeks | +| Usable limited beta | Audio/haptics, pairing, recovery, bitrate/loss handling, installer, worn testing and latency work | 6–12 additional weeks | +| Wider support | Multiple GPU vendors/distros, USB and Wi-Fi variation, long-session stability | 2–4 additional months | + +Planning range: **12–24 engineer-weeks for a limited beta**, roughly +**10–25k lines of our code plus tests/tooling**, excluding bundled libraries. +This is a low-confidence scope estimate, not a delivery promise; an unsupported +driver or decode interface could block it entirely. Foveated streaming, +eye tracking and parity with Valve are excluded. A Linux gaming PC and repeatable +worn-headset testing are prerequisites. **Do not build this until Alex chooses.** + +## (c) Optional “install WiVRn” shortcut only + +Allowed as a clearly optional convenience, never a prerequisite for a Frame +Control feature. **Documented:** WiVRn's server Flatpak ID is +`io.github.wivrn.wivrn`; its client/server versions must match, and its Flatpak +includes xrizer/OpenComposite. Those are properties of an independently +installed third-party stack, not components of our implementation. + +**Recommendation:** defer the shortcut while the current client fails before +session creation. If offered later, label that compatibility result and let +the user choose the install; do not present “install” as “streaming works.” +**Estimate (inferred):** 1–2 engineer-days for an optional host-side shortcut +with package/version detection and honest status, excluding third-party fixes. +No shortcut, host install, pairing automation or streaming UI was built here. + +Choose **(a)** for the next hardware test. Keep **(b)** as a separately approved +project if Valve's path fails or lacks a required capability. **(c)** does not +solve the verified client blocker and should not be the product's foundation. diff --git a/docs/streaming.md b/docs/streaming.md index 707544e..04023e3 100644 --- a/docs/streaming.md +++ b/docs/streaming.md @@ -5,6 +5,8 @@ This covers three directions, plus input: - **A. Frame → Mac**: see and control the headset from the Mac. - **B. Mac → Frame**: use the Mac's desktop inside the headset. - **C. iPhone → Frame**: mirror the phone inside the headset. +- **PC VR from Linux**: [feasibility and options](linux-vr-streaming.md), + including Valve's streaming and USB support. No Linux host tested yet. - **Input**: type and point in the Frame from the Mac or iPhone. The confidence labels are the same as in [ssh.md](ssh.md). @@ -24,11 +26,13 @@ Mac with keyboard, mouse, and clipboard. ## B. Show the Mac's desktop inside the Frame -The Frame's streaming features are built around a **Windows PC running -SteamVR** plus the USB Wi-Fi 6E dongle. Even Linux hosts had VR-streaming -problems at launch +The Frame's VR streaming uses **SteamVR** on the host. Linux hosts had +VR-streaming problems at launch ([Steam discussion](https://steamcommunity.com/app/4165890/discussions/0/528765047224280796/), [gbl08ma](https://gbl08ma.com/posts/steam-frame-a-linux-machine-doesnt-support-linux/)). +Valve's later 2.17.8 notes explicitly describe Steam Link fixes on Linux and +initial USB streaming support (**documented**, not tested from a Linux host +here). See [the current comparison](linux-vr-streaming.md#a-valves-own-path--recommended-first). **macOS isn't a supported SteamVR host**, so for the Mac we're only looking at flat 2D desktop streaming into a window on the Frame's Linux desktop.