mitch030504--Wiicompiled_VR_Frame/docs/steam-frame.md

# WiiCompiled VR on the Steam Frame

Valve's Steam Frame runs SteamOS on a Snapdragon 8 Gen 3 (Cortex-X4, A720 and A520 cores, Adreno 750),
with 2160x2160 panels per eye at 72 to 144 Hz, eye tracking, and SteamVR as its OpenXR runtime. It runs
Android apps through Lepton, SteamOS's Android layer, where SteamVR provides an Android OpenXR runtime
(OpenXR 1.0, through the Khronos loader's runtime broker). The Steam Frame build is therefore a third
flavour of the Quest app, `steamFrame`: everything in `docs/quest-port.md` below the app shell (the
Vulkan backend, the game kit, `.wcgame` packages, the on-headset build) applies unchanged, and this
document covers what differs.

A native SteamOS ARM64 build is a separate, later piece of work: Linux has no OpenXR graphics backend
yet (see [A native SteamOS build](#a-native-steamos-build)).

**Status: not yet run on a Steam Frame, and the Android backend as it stands cannot present under
Lepton**: Lepton's Vulkan driver has no external memory or sync fd extensions, which the Quest
backend's two-device design needs (see [What the Frame reported](#what-the-frame-reported)). The
flavour, controller, refresh rate and foveation work below stays valid; the eye handoff has to move
to a single shared device first.

## What the flavour changes

| | Quest flavours | `steamFrame` |
| --- | --- | --- |
| CPU target (`kit.json` `androidCpu`) | `cortex-a77` (`kryo` on Quest 1) | `cortex-x4` |
| `MKW_ANDROID_HEADSET` | `quest` | `steam_frame` (defines `MKW_HEADSET_STEAM_FRAME`) |
| Library entry | `LauncherActivity` (Quest 1: `QuestActivity`) | `FrameEntryActivity` |
| Horizon OS manifest entries | present | removed |
| `[vr] refresh_rate` default | `0` (the headset's own) | `120` |
| `[vr] passthrough` | default on (`XR_FB_passthrough`) | not asked for, default off, setting hidden |
| `[vr] eye_tracked_foveation` default | off | on |

The application ID stays `org.wiicompiled.quest`, so the storage paths in `docs/quest-port.md` hold
as they are. The kit's CPU string differs from the Quest ones, which gives the Frame its own kit
fingerprint: a game built for a Quest is refused on the Frame and the other way round, by the same
checks that keep Quest 1 and modern Quest games apart.

**CPU.** Every core of the 8 Gen 3 implements ARMv9.2, so the products target the Cortex-X4 with
its whole feature set. That includes SVE and SVE2, which clang auto-vectorises with (a simple loop
compiled with `-O3` used SVE registers ten times). Phones with this chip do not expose SVE, but the
Frame's kernel does: `/proc/cpuinfo` lists `sve`, `sve2`, `svei8mm`, `svebf16` and the SVE2 crypto
extensions, on SteamOS and inside Lepton alike. A build for a device without SVE would need
`cortex-x4+nosve`. The flavour-to-CPU map lives once in
`android/app/build.gradle.kts` (`headsetCpus`), which the kit export also reads now instead of
guessing from the variant name.

**Launch under Lepton.** Lepton starts the one real activity that is both `MAIN` and `LAUNCHER`, and
runs the app in VR when that activity carries a VR category; it ignores `activity-alias` entries.
Quest builds put `LAUNCHER` on the 2D panel (or, on Quest 1, add an alias), so neither works there.
`src/steamFrame/AndroidManifest.xml` makes `FrameEntryActivity` the only `MAIN`/`LAUNCHER` activity,
with `org.khronos.openxr.intent.category.IMMERSIVE_HMD` and `com.oculus.intent.category.VR`. It
shows nothing: it always opens the setup panel (`LauncherActivity`), and when the selected game can
start as it is (game files, a game built for this kit, Retro Rewind's pack, and no enabled mods
still to copy into the pack), it opens `QuestActivity` on top of it. The headset therefore goes
straight into VR, and quitting the game returns to the panel for setup, imports and mods.
`adb logcat -s WiiCompiledLauncher` shows which way it went.

The manifest also removes Horizon OS's own entries (`com.oculus.supportedDevices`, `focusaware`,
`trade_cpu_for_gpu_amount`, the passthrough feature and the hand tracking permissions and feature)
and keeps the Khronos broker queries and the `OPENXR_SYSTEM` permission, which any Android OpenXR
runtime needs.

## Controllers

With `XR_VALVE_frame_controller_interaction` the runtime offers the Frame controller's own profile,
`/interaction_profiles/valve/frame_controller_valve`. Without it SteamVR presents the controllers as
Touch controllers, which loses the left D-pad. `openxr_input.cpp` suggests it after Touch and the
simple controller. Each hand has a thumbstick, trigger, grip and shoulder button. The right hand has
A, B, X, Y and a menu button; the left hand has a D-pad and a View button. Binding paths follow
DolphinXR's port (iChris4/dolphinXR#9). Windows asks for the same extension, so a Frame streaming
from a PC through SteamVR gets the D-pad too.

| Frame controller | Wii Remote mode | Gamepad mode |
| --- | --- | --- |
| Right A | A | South (A) |
| Right B | C (look behind) | East (B) |
| Right trigger | B | Right trigger |
| Right stick up / down | 1 / 2 | Right stick |
| Left View | + (pause) | Start |
| Left shoulder | Settings panel (Touch's left Y) | North (Y) |
| Left D-pad | Wii Remote D-pad | D-pad |
| Left stick, left trigger | Nunchuk stick, Z | Left stick, left trigger |
| Grips, stick clicks, motion, aim | as on Touch (`OPENXR.md`, Controllers) | as on Touch |
| Right X, Y, menu and shoulder | unbound | unbound |

The four D-pad actions are new and also reach the virtual gamepad's D-pad; Touch leaves them unbound,
so nothing changes on a Quest.

## Refresh rate

`[vr] refresh_rate` (Hz, `0` = the headset's own rate) is asked of the runtime through
`XR_FB_display_refresh_rate` each time the session starts running and whenever the setting changes.
The request uses the runtime's own value within half a hertz of the setting (runtimes report 119.98
for 120). Setting it back to `0` restores the rate the session started at. The game renders 60 frames
a second, so at 120 Hz each frame shows for exactly two refreshes. At 72 or 90 Hz some frames show
for one refresh and others for two, which judders. The Frame starts at 120. Render-first pacing
(`docs/quest-port.md`) already waits for each sealed game frame, so on the Frame the pacing summary
should read about 60 `skipped-slots` a second with no `late` cycles.

Lepton may decline the request (frame-control found SteamVR keeping its own rate there). The session
log then says `display refresh rate 120 Hz refused` with the rates it offers, and nothing else
changes. The setting is in the headset panel and in the launcher (VR → Headset); other headsets
offer it as well, at their own default of `0`.

## Eye-tracked foveation

`[vr] eye_tracked_foveation` (default on for the Frame) moves the foveation level's full-density
region to where the player looks:

1. At launch the runtime asks for `XR_EXT_eye_gaze_interaction`. If the system reports an eye tracker,
   `OpenXRInput` binds the gaze pose (`/user/eyes_ext/input/gaze_ext/pose`) and locates it for each
   packet's display time, in the space the eye views are located in.
2. `vr/eye_gaze.h` turns the gaze into tangents of each eye's own view, which may be canted.
   `AuroraStereoFrame` carries them as `gaze` and `gazeValid`, appended after its existing fields.
3. Aurora snaps the gaze to a cell of two map texels (64 pixels, about 3 degrees) and builds that
   cell's density map with the level's rings centred on the gaze (`gfx/foveation.hpp`).

Each eye keeps up to 32 maps, one per cell looked at, so a glance back reuses its map instead of
uploading a new one. A new map is bound once its upload has completed, and until then the eye keeps
the map it had. A blink, lost tracking or the setting turned off returns to the map centred on the
forward direction, which is byte-identical to the fixed foveation map. No change to the Dawn patch
was needed: maps stay immutable, and the patch already lets a view be rebound to another map.

The session log reports `OpenXR eye gaze: available` (or that the runtime has no eye tracker),
`OpenXR eye gaze: tracking` at the first tracked sample, and `eye foveation medium following the
gaze` for each eye's first gaze map. Foveation pays only when an eye is pixel-bound
(`OPENXR.md`, Foveated rendering), which the Frame's larger eyes make more likely.

If the Frame's driver offers `VK_QCOM_fragment_density_map_offset` or
`VK_EXT_fragment_density_map_offset`, shifting one map per pass would replace switching between
maps. That needs the Dawn patch to create the eye textures with the offset flag, so it waits for
the device's extension list.

## Building and installing

On the Windows build host described in `docs/quest-port.md`:

```powershell
powershell -ExecutionPolicy Bypass -File android/Build-Quest.ps1 -Headset frame               # the steamFrame APK; checks kit.json says cortex-x4
powershell -ExecutionPolicy Bypass -File android/Build-QuestGame.ps1 -Headset frame -Product base -Data <DATA>   # a .wcgame for the Frame's kit
```

The APK lands in `android/app/build/outputs/apk/steamFrame/<configuration>`. WheelWizard VR's
"Build for Quest" builds a Frame game unchanged once it is given the Frame APK (Setup takes the
headset from the APK's kit). WheelWizard itself still needs a Steam Frame choice that fetches that
APK from the release, published as `…-SteamFrame.apk`.

Lepton opens an adb port (5555 and up) for each running Android instance, reachable over the
network. With an Android app running on the Frame, `adb connect <frame-ip>:5555` reaches it from the
build PC. How the APK reaches the Steam library (adb into a Lepton instance, frame-control, or
Steam's own sideloading) is to be confirmed on the device.

## What the Frame reported

Read on 2026-10-04 from a Steam Frame running SteamOS (`holo`), kernel 6.18.0, with the commands
below:

| Reading | SteamOS | Lepton |
| --- | --- | --- |
| Page size | 4096 | 4096 |
| CPU | 8 cores; `sve sve2 svei8mm svebf16 sveaes svepmull svebitperm svesha3 svesm4 i8mm bf16 bti paca pacg ...` | the same |
| Android | — | 11 (API 30, LineageOS), `ro.product.model` Lepton, device `lepton_arm64_only`, platform `waydroid`, `ro.steam.running_in_app_container=true` |
| Vulkan driver | Turnip, Mesa 26.3.0-devel, Vulkan 1.4.362: `VK_EXT_fragment_density_map` (non-subsampled images, not dynamic), `VK_EXT_fragment_density_map_offset` and `VK_QCOM_fragment_density_map_offset`, `VK_KHR_external_{memory,semaphore,fence}_fd`, `VK_EXT_external_memory_dma_buf`, `VK_EXT_queue_family_foreign`, `VK_KHR_dynamic_rendering`; Valve's `fdm_injection` and `rpo` Vulkan layers | `ro.hardware.vulkan=freedreno`: the same Mesa 26.3.0-devel Turnip built for Android (`vulkan.pastel.so` is also present, not selected). `/dev/kgsl-3d0` is the DRM render node |
| OpenXR runtime | SteamVR, `bin/linuxarm64/vrclient.so` (`~/.config/openxr/1/active_runtime.json`); SteamOS ships the SDK headers, `libopenxr_loader.a` and `openxr.pc` | `/vendor/etc/openxr/1/active_runtime.json`, from the host's `/usr/share/guestos/android/vendor/etc/openxr`; no runtime broker package |
| Implicit OpenXR layers | `XrApiLayer_VALVE_fdm_injection` (also listed as explicit) | `XrApiLayer_VALVE_fdm_injection` |

What follows from them:

- **Fast memory path.** 4 KB pages keep the translated code's flat memory path. A 16 KB kernel
  would have sent it through the checked path.
- **CPU target.** The Frame exposes SVE, so the build targets the whole `cortex-x4` (above).
- **Android version.** API 30 meets the app's minimum of 29.
- **Driver workarounds.** Lepton is a Waydroid container, and its Vulkan driver is Turnip. The
  Adreno workarounds in `docs/quest-port.md` were found on Qualcomm's own driver. The vertex padding
  stays on (it is correct either way); `debug.wiicompiled.vtxpad 0` can check whether Turnip needs it.
- **Foveation.** The host's Turnip has density maps for non-subsampled images through dynamic
  rendering, what the Dawn patch needs, and both density map offset extensions, which would let
  eye-tracked foveation shift one map instead of switching maps.
- **No buffer sharing between devices in Lepton.** The Vulkan Hardware Capability Viewer (4.03,
  the last release for Android 11), run inside Lepton, reports Turnip `26.2.99` (Vulkan 1.4.362,
  display name "Valve Lepton") with `VK_EXT_fragment_density_map`, both density map offset
  extensions, `VK_VALVE_fragment_density_map_layered`, `VK_KHR_timeline_semaphore` and the
  maintenance extensions, but **no** `VK_ANDROID_external_memory_android_hardware_buffer`,
  `VK_KHR_external_memory_fd`, `VK_KHR_external_semaphore_fd` or `VK_KHR_external_fence_fd`. The
  Quest backend (`openxr_vulkan.cpp`) hands each eye from Dawn's device to its own OpenXR device
  through exactly those, so it cannot present under Lepton. What can: binding Dawn's own device to
  the session, as the Windows Vulkan backend (`openxr_vulkan_win32.cpp`) does, so the eyes are
  copied into the swapchain on Dawn's queue with no sharing at all. That backend is also the core
  of a native SteamOS build ([below](#a-native-steamos-build)).
- **Finding the runtime.** An app inside Lepton reaches SteamVR's OpenXR runtime through the system
  runtime file, not a broker. The Khronos loader the game links statically (`DYNAMIC_LOADER OFF`)
  tries the runtime brokers first, then reads `/{product,odm,oem,vendor,system}/etc/openxr/1/active_runtime.json`,
  so no app change is needed; the manifest's broker queries are simply unused here. Walkabout Mini
  Golf, an Android VR game, runs in the same Lepton.
- **Valve's foveation layer.** `XrApiLayer_VALVE_fdm_injection` is implicit, so it wraps every
  Android OpenXR app. By its name it adds fragment density maps to apps' own render passes. This
  game draws its eyes on Dawn's device and only copies them into the swapchain on the OpenXR
  device, so the layer has no render pass of the game's to change; the game's own maps
  (eye-tracked foveation, above) do that. If the layer gets in the way,
  `DISABLE_VULKAN_FDM_INJECTION_LAYER` turns it off (its manifest loads
  `libVkLayer_VALVE_fdm_injection.so`; the runtime itself is
  `/data/steamvr/runtime/bin/androidarm64/vrclient.so`).

### SteamVR's Android OpenXR extensions

Walkabout Mini Golf, a Unity game in the same Lepton, logs what the runtime offers (`adb logcat -d |
grep -F '[XR]'`). Its extensions:

```
XR_EXT_active_action_set_priority XR_EXT_debug_utils XR_EXT_dpad_binding XR_EXT_eye_gaze_interaction
XR_EXT_frame_composition_report XR_EXT_frame_synthesis XR_EXT_hand_interaction XR_EXT_hand_joints_motion_range
XR_EXT_hand_tracking XR_EXT_hand_tracking_data_source XR_EXT_hp_mixed_reality_controller
XR_EXT_interaction_profile_battery_state_display XR_EXT_interaction_render_model XR_EXT_local_floor
XR_EXT_palm_pose XR_EXT_performance_settings XR_EXT_render_model XR_EXT_user_presence XR_EXT_uuid
XR_EXT_view_configuration_views_change XR_FB_display_refresh_rate XR_FB_foveation
XR_FB_foveation_configuration XR_FB_foveation_vulkan XR_FB_space_warp XR_FB_swapchain_update_state
XR_HTC_vive_cosmos_controller_interaction XR_HTC_vive_focus3_controller_interaction
XR_HTC_vive_wrist_tracker_interaction XR_HTCX_vive_tracker_interaction XR_KHR_android_create_instance
XR_KHR_binding_modification XR_KHR_composition_layer_depth XR_KHR_generic_controller XR_KHR_locate_spaces
XR_KHR_opengl_enable XR_KHR_opengl_es_enable XR_KHR_visibility_mask XR_KHR_vulkan_enable
XR_KHR_vulkan_enable2 XR_META_foveation_eye_tracked XR_META_performance_metrics
XR_META_recommended_layer_resolution XR_META_vulkan_swapchain_create_info XR_MND_headless
XR_MNDX_egl_enable XR_VALVE_analog_threshold XR_VALVE_app_space_delta_pose
XR_VALVE_frame_controller_interaction XR_VALVE_timing_utils
```

Environment blend modes `OPAQUE` and `ALPHA_BLEND`; reference spaces `LOCAL`, `STAGE` and `VIEW`.
What this build asks for and gets:

| Extension | Offered | What it means here |
| --- | --- | --- |
| `XR_KHR_android_create_instance`, `XR_KHR_vulkan_enable2` | yes | The Android backend's instance and Vulkan binding |
| `XR_VALVE_frame_controller_interaction` | yes | The Frame controller profile and its D-pad |
| `XR_FB_display_refresh_rate` | yes | `[vr] refresh_rate` (120 Hz) can be requested |
| `XR_EXT_eye_gaze_interaction` | yes | Eye-tracked foveation |
| `XR_EXT_performance_settings` | yes | `performance_level` |
| `XR_EXT_hand_tracking`, `XR_EXT_hand_tracking_data_source` | yes | Tracked hands; `XR_FB_hand_tracking_mesh` and `_aim` are not offered, so the cockpit draws its procedural gloves and bare hands get no pinch gestures |
| `XR_KHR_convert_timespec_time` | **no** | VR frame interpolation is unavailable, and controller motion is sampled at the frame's display time rather than the current time |
| `XR_KHR_android_thread_settings` | no | Thread hints are skipped (logged as refused) |
| `XR_FB_passthrough` | no | As expected; the build does not ask for it |

Later candidates the runtime offers: `ALPHA_BLEND` could bring back the room around the menu screen
without `XR_FB_passthrough`, `XR_KHR_visibility_mask` would skip the pixels the lenses never show,
and `XR_FB_foveation` with `XR_META_foveation_eye_tracked` only shapes render passes into the
runtime's swapchain images, which this game's eyes reach by copy, so it does not apply.

## Device checklist

Information to collect first, from any PC over Lepton's adb port (the commands work in bash, zsh and
fish; in PowerShell only the adb lines do), with an Android app running on the Frame:

```sh
adb connect <frame-ip>:5555
adb -s <frame-ip>:5555 shell 'getprop ro.build.version.release; getprop ro.build.version.sdk; getprop ro.product.manufacturer; getprop ro.product.model; getprop ro.product.device; getprop ro.hardware.vulkan; getprop ro.board.platform'
adb -s <frame-ip>:5555 shell 'uname -a; getconf PAGE_SIZE; grep -m1 Features /proc/cpuinfo; grep -c processor /proc/cpuinfo'
adb -s <frame-ip>:5555 shell cmd gpu vkjson > frame-vkjson.json
adb -s <frame-ip>:5555 shell "pm list packages | grep -i -E 'xr|valve|steam|khronos|openxr'"
# The parts of frame-vkjson.json that matter:
grep -oE '"(deviceName|driverName|driverInfo|apiVersion|driverVersion)": *[^,]*' frame-vkjson.json | sort -u
grep -oE '"extensionName": *"[^"]*"' frame-vkjson.json | grep -iE 'hardware_buffer|external_semaphore|external_fence|external_memory|fragment_density|shading_rate|dynamic_rendering' | sort -u
```

and on the Frame itself (Desktop Mode, Konsole):

```bash
uname -a; getconf PAGESIZE; grep -m1 Features /proc/cpuinfo; head -5 /etc/os-release
cat ~/.config/openxr/1/active_runtime.json 2>/dev/null; ls /usr/share/openxr/1/ /etc/xdg/openxr/1/ 2>/dev/null
```

What each answers:

- `vkjson`: whether Lepton's Vulkan driver has what the Android bridge needs:
  - `VK_ANDROID_external_memory_android_hardware_buffer`;
  - `VK_KHR_external_semaphore_fd` and `VK_KHR_external_fence_fd` with sync fd handles.

  Without these the APK route cannot present at all, and the native route becomes the way. It also
  shows `VK_EXT_fragment_density_map` (foveation), the density map offset extensions, and which
  driver Lepton uses.
- The CPU features line: no `sve`, as the CPU target assumes.
- The page size: on a 16 KB-page kernel the runtime finds out at launch and routes translated memory
  accesses through the checked path (`guest_flat_memory.h`, `RequiresCheckedAccess`), which works but
  is slower. That is worth knowing before measuring.
- The Android version: the app needs API 29 (Android 10) or newer.

Then, on the first launch, the session log (`Logs/<product>_<stamp>_pid<pid>/console.log` next to
`DATA`) and `adb logcat -s SDL WiiCompiledQuest WiiCompiledLauncher` should show, in order:

1. `OpenXR Android loader initialized`;
2. `OpenXR runtime offers N extensions: ...`, the whole list SteamVR's Android runtime has;
3. `OpenXR initialized: runtime '...'` with SteamVR's name;
4. the negotiated Vulkan binding, then `OpenXR Vulkan swapchains ready`;
5. `display refresh rate 120 Hz requested` (or `refused`, with the rates on offer);
6. the session reaching `FOCUSED`;
7. `OpenXR interaction profiles: left /interaction_profiles/valve/frame_controller_valve, right ...`;
8. `OpenXR eye gaze: available`, then `tracking`;
9. `presentation=virtual-screen` for the menus, and `first immersive packet consumed` on race entry.

Then, in the game:

- the D-pad does tricks, the left shoulder opens the settings panel, and View pauses;
- `debug.wiicompiled.fpslog 1` on a race start shows the pacing summary and the GPU time per pass
  with foveation off, fixed and following the gaze;
- `render_scale` starts at 0.8, the Quest's value. The runtime's recommended eye size (logged at
  startup) and those measurements decide whether the Frame keeps it.

A black headset with a working mirror points at the AHardwareBuffer copy, as on the Quest
(`docs/quest-port.md`, Validation status).

Open questions only the device can answer:

- Whether Lepton's driver supports the AHardwareBuffer and sync fd bridge.
- Whether Lepton shows the 2D setup panel while the app runs in VR mode. If it does not, the game
  still starts directly once a `.wcgame` with the game files has been imported, but importing needs
  the panel.
- Whether SteamVR grants 120 Hz.
- Whether the gaze needs an Android permission under Lepton.
- Whether "Build on this headset" can run its toolchain through `/system/bin/linker64` inside
  Lepton. A game built on the PC does not depend on it.

## A native SteamOS build

Valve recommends native Linux ARM64 builds for the Frame, and one would avoid Lepton and the
two-device copy. The pieces:

- **Backend.** The Windows Vulkan backend (`openxr_vulkan_win32.cpp`), where OpenXR creates Dawn's
  own device and eyes are copied on Dawn's queue, ports almost as it is. Only its `_WIN32` guards are
  platform-specific.
- **Interop.** Aurora's `vulkan_win32_interop.cpp` finds the patched Dawn's exports with
  `GetModuleHandleW`. A static Linux Dawn would reference them directly, and `aurora_core.cmake`
  compiles that file on Windows only.
- **Dawn.** A linux-aarch64 Dawn built with `aurora-main/patches/dawn` (the hook ABI and density
  maps), as `android/Build-QuestDawn.ps1` already does for Android.
- **`openxr_integration.cpp`.** It needs a Linux branch asking for `XR_KHR_vulkan_enable2` and
  `XR_KHR_convert_timespec_time`. Today its `#else` is Android's and requires
  `XR_KHR_android_create_instance`.
- **Controller timing.** `openxr_input.cpp` needs a `__linux__` branch for the input clock.
- **CPU target.** An `MKW_LINUX_CPU` knob in place of `-mcpu=native`, for cross-builds.
- **Android-gated fixes.** The Adreno vertex padding, `headset_owns_display` and the
  `last_pass_feeding_replay` saving are gated on `__ANDROID__`. They would follow the GPU or the
  headset instead.
- **Packaging.** SteamOS packaging, and a way to build the player's game for it.