mitch030504--Wiicompiled_VR_Frame/docs/quest-port.md

WiiCompiled VR on Meta Quest (standalone Android)

This document is the design and build reference for the native Quest build. It complements OPENXR.md, which remains the specification for the presentation policy, the virtual screen, the first-person camera and frame interpolation: all of that is shared, unchanged, between the Windows D3D12 product and the Quest Vulkan product. What differs is everything below the stereo replay: the graphics binding, the app shell and the platform glue.

Sources of the design

• KartPad (kartpad-main/, the kartpad-android runtime branch of the WiiCompiled fork) proved that the translated game runs on Android arm64 with Aurora on Dawn/Vulkan under SDL3's SDLActivity. Its lessons carried over: the products are shared libraries SDL loads, the activity exports its directories through the environment before native code runs, the Windows-generated blob assembly needs its section syntax rewritten for ELF, and Dawn's android-aarch64 prebuilt package needs one path rewritten. KartPad's Android fiber/JNI split (never calling Java-backed SDL APIs from a guest fiber stack) is respected here by keeping every OpenXR call on the dedicated pacing thread, which is a real std::thread.
• DolphinXR (Dolphin-OpenXR-2/, quest flavour) supplied the Quest-side specifics: the loader must be initialised with the activity as its context or the session never leaves IDLE; XrInstanceCreateInfoAndroidKHR must be chained on instance creation; the manifest needs the Khronos runtime broker queries, the OPENXR_SYSTEM permission, the com.oculus.intent.category.VR intent category and the XR_ACTIVITY_START_MODE_FULL_SPACE_UNMANAGED property; XR_KHR_android_thread_settings may reject the renderer-worker type on some runtime builds.

Architecture

One runtime, two graphics bindings

runtime/src/vr/openxr_integration.cpp owns the pacing thread, policy evaluation, the retained-layer protocol and the head-pose maths. It is written against the backend-neutral vocabulary in runtime/include/vr/openxr_backend.h (OpenXRPresentation, OpenXRBackendFrame, OpenXRBeginStatus, OpenXRSubmissionStatus) and selects one backend class at compile time:

Platform	Backend	Binding
Windows	OpenXRD3D12Backend (openxr_d3d12.cpp)	Dawn's own D3D12 device and queue are bound to the session; eyes are copied on that queue.
Android	OpenXRVulkanBackend (openxr_vulkan.cpp)	The backend creates its own VkInstance/VkDevice through the OpenXR runtime; Dawn and that device meet on AHardwareBuffers.

The D3D12 types keep their old names through aliases, so openxr_d3d12_replay_tests and the desktop code did not change.

Why a second Vulkan device

The pinned Dawn package (v20260603.191052) exposes only VkInstance from its Vulkan backend, no VkDevice, VkQueue or queue family, and it will not enable the device extensions the OpenXR runtime demands. Binding Dawn's device to the session is therefore impossible without a patched Dawn. Instead:

1. xrCreateVulkanInstanceKHR / xrCreateVulkanDeviceKHR (XR_KHR_vulkan_enable2, with an XR_KHR_vulkan_enable fallback that queries the extension lists) create a small Vulkan device the runtime is happy with.
2. Per eye, two AHardwareBuffers (R8G8B8A8_UNORM, or RGBA16F when Aurora renders float) are allocated and imported on that device (VK_ANDROID_external_memory_android_hardware_buffer).
3. Aurora imports the same buffers as Dawn shared texture memory (SharedTextureMemoryAHardwareBuffer) and, inside the frame worker's command buffer, copies each replayed eye into the buffer (aurora-main/lib/webgpu/vulkan_interop.cpp, the twin of d3d12_interop.cpp and registered through the same stereo sink).
4. Ordering across the two devices uses Android sync file descriptors: Dawn's EndAccess exports a SharedFenceSyncFD the OpenXR device waits on before its vkCmdCopyImage into the acquired swapchain image, and the copy signals an exportable semaphore whose sync fd Dawn waits on before it writes that buffer again. Image layouts follow Vulkan's rule that a queue family ownership release and acquire must repeat the same old/new layout pair: Dawn reports its release layout, the OpenXR side acquires with it, transitions for the copy, and hands the buffer back in GENERAL with a transition-free release so Dawn's acquire can mirror it.
5. The copy runs on the queue bound to the session before xrReleaseSwapchainImage, so the compositor sees ordinary same-queue work.

The cost is one extra GPU copy per eye per frame, a few hundred microseconds at Quest eye resolutions; the benefit is that stock Dawn is used unchanged and the OpenXR device outlives Aurora's, which is exactly the failure DolphinXR hit on Vulkan when a game's device was destroyed under the compositor.

gpu.cpp steers Aurora to an RGBA8 surface format under xrInterop on Android (there is no BGRA AHardwareBuffer format) and requests the two Dawn features the bridge needs.

Controllers

Quest Touch controllers are not HID gamepads, so openxr_input.cpp syncs an OpenXR action set on the pacing thread and feeds a virtual SDL joystick (SDL_AttachVirtualJoystick, type gamepad). Aurora opens it like any pad and assigns it to player 1; every existing binding, dead zone and overlay setting applies. Mapping: A/B → South/East, X/Y → West/North, index triggers → trigger axes, grips → shoulders, thumbsticks → sticks (clicks → stick buttons), left menu → Start. Bindings are suggested for oculus/touch_controller and khr/simple_controller.

Android platform glue

• runtime/src/vr/openxr_android.cpp: xrInitializeLoaderKHR with the JavaVM and activity SDL already holds, the XrInstanceCreateInfoAndroidKHR chain (OpenXRConfig::instance_create_next), and the optional thread hint.
• runtime/src/platform/host_platform.cpp / runtime_config.h: the activity exports MKW_ANDROID_DATA_DIR (external files dir, user reachable) and MKW_ANDROID_RESOURCES_DIR (unpacked wii_bootstrap/, dsp_coef.bin, initial_pipeline_cache.db); the latter stands in for the executable directory so the existing adjacent-file lookups work unchanged.
• main.cpp includes SDL_main.h on Android so SDLActivity finds SDL_main in libmain.so, and passes the resources path to Aurora.
• Fibers use the vendored libco AArch64 backend (Bionic is Linux), guest memory uses the Linux mmap path, the MPRIS media monitor is compiled out.
• Surface readiness. Aurora presents only while g_surfaceReady is set, and on Android that flag starts false. Stock SDL3 exports neither its activity mutex (Android_LockActivityMutex) nor a readiness hook, so the app's QuestSurface subclass brackets SDL's surfaceChanged/surfaceDestroyed with aurora_android_begin/end_surface_mutation (aurora/android.h), and Aurora's SurfaceLock owns its own recursive mutex. This is KartPad's design.
• JNI only on the real thread stack. Guest threads run on libco stacks inside the SDL thread, and SDL's Android event pump can reach Java (joystick polling, HIDAPI). ART binds JNI transitions to the thread's real stack, so Aurora's pump_events is a no-op on Android and UpdateAuroraAndProcessEvents defers a poll made from a guest fiber until control is back on the scheduler context (GuestFiberManager::IsOnSchedulerFiber). The default guest thread shares the scheduler context, so most polls run immediately. Also KartPad's finding, from device crashes.
• Time conversion for frame interpolation uses XR_KHR_convert_timespec_time (CLOCK_MONOTONIC, the clock behind steady_clock on Bionic).

Build system

• runtime/CMakeLists.txt recognises CMAKE_SYSTEM_NAME=Android on arm64 as MKW_PLATFORM_ANDROID: OpenXR on by default, Dawn from the pinned android-aarch64 package (digest pinned in AuroraDawnProvider.cmake, which also rewrites the package's absolute liblog.so path and looks the package up with NO_CMAKE_FIND_ROOT_PATH so the NDK sysroot rule does not hide it), SDL3 built shared (or -DAURORA_SDL3_PROVIDER=system for the AAR prefab), tests off, products built as libmain.so / libmain_retro_rewind.so, -mcpu=cortex-a77 (Quest 2's XR2 Gen 1; Quest 3/Pro are supersets).
• runtime/cmake/PublicProducts.cmake gains MKW_GENERATED_DIR so a build configured from a checkout can name the translator output tree, and on Android rewrites the PE/COFF .section .rdata,"dr" of a Windows-generated blob .S into ELF .rodata plus a GNU-stack note. The translator itself also learned --target-os windows|macos|linux|android for generate-data-init and translate-mod, for pipelines that generate on another host.
• android/: the Gradle project. app/src/main/cpp/CMakeLists.txt adds the repository's runtime/ as a subdirectory with those Android choices; flavours base and retroRewind pick the product target and library name.

Building

Prerequisites on the Windows host (all already present on the machine this was developed on): JDK 17, Android SDK with platform 34+, NDK 29.0.14206865, SDK CMake 3.22.1, adb; a translated graph for your own disc (the installer's BuildWorkspace/generated, produced by the normal Windows pipeline).

powershell -ExecutionPolicy Bypass -File android/Prepare-QuestDependencies.ps1        # SDL3 3.4.4 AAR into android/app/libs
powershell -ExecutionPolicy Bypass -File android/Build-Quest.ps1 -Install             # base game, debug-signed
powershell -ExecutionPolicy Bypass -File android/Build-Quest.ps1 -Flavor retroRewind  # Retro Rewind (needs translate-mod output)
adb push DATA /sdcard/Android/data/org.wiicompiled.quest/files/WiiCompiledOpenXRVR/DATA

Config.toml, saves and per-run logs live next to DATA under WiiCompiledOpenXRVR; the activity writes a first Config.toml with [vr] enabled = true and paths.dvd_root set. Logs: adb logcat -s SDL WiiCompiledQuest plus the Logs/<product>_<stamp>_pid<pid>/console.log folder the runtime writes.

A CMake-only cross-compile of the native runtime (no game) is the quick compile check and needs no Gradle:

cmake -S runtime -B .scratch/android-audit-build -G Ninja `
  -DCMAKE_TOOLCHAIN_FILE=$env:LOCALAPPDATA/Android/Sdk/ndk/29.0.14206865/build/cmake/android.toolchain.cmake `
  -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29 -DANDROID_STL=c++_shared `
  -DCMAKE_BUILD_TYPE=Release -DMKW_BUILD_PRODUCTS=OFF
cmake --build .scratch/android-audit-build --target mkw_android_native_compile aurora_core aurora_gx

Validation status

What has been verified on the development machine (September 2026):

• Translator: dotnet test passes with the new --target-os tests (17/17 in the touched suites).
• Windows: mkw_openxr_replay_tests and mkw_vr_policy_tests pass; the mkw_runtime_common and aurora_core targets compile with the refactored integration; the workspace product rebuild links WiiCompiled.exe.
• Android: the cross-compile audit passes. mkw_android_native_compile, aurora_core and aurora_gx all build for aarch64-none-linux-android29 with NDK 29.0.14206865, which covers the whole native runtime including openxr_vulkan.cpp, openxr_android.cpp, openxr_input.cpp and the Aurora AHardwareBuffer bridge. Three Bionic portability fixes came out of it: the std::min call in hle/audio/audio.cpp needed an explicit type (int64_t is long on LP64 Android while the clock rep is long long), guest_flat_memory.cpp needs a __NR_memfd_create shim below API 30, and Crypto++'s cpu.cpp needs the NDK's cpu-features source compiled in.
• Device, 2026-09-16: running on a Quest 3 (HorizonOS 14, API 34). The runtime negotiates XR_KHR_vulkan_enable2 (Vulkan 1.0 to 1.2), creates 1680x1760 R8G8B8A8_SRGB (VkFormat 43) swapchains, attaches the controller actions, and the session reaches FOCUSED. The game boots through the title movies into the attract race, the policy switches to immersive-race with all 189 perspective draws replayed per eye, the first projection layer is submitted, and the menus return to the virtual screen. No WebGPU or OpenXR errors over a 90 second session.

Bring-up fixes that only a device could reveal:

Symptom	Cause	Fix
Activity crashed with EACCES on Config.toml	adb shell mkdir had created the app's data directory, so the shell user owned it	Let the app create its own directory; Run-Quest.ps1 launches once before placing DATA
DATA unreadable by the game	adb-placed files stay owned by the shell user and directories are 2770	chmod -R a+rX DATA as the owning shell user; run-as cannot reach shared storage (SELinux)
"Cannot persist NAND setting.txt"	FUSE storage has no hard links; link() fails with EACCES, not EPERM	Android falls back to exists-check plus rename in nand_settings.h
SharedFence ... signaled value (0) was not 1	A sync fd is binary; Dawn expects value 1	vulkan_interop.cpp passes 1
Link error on Android_LockActivityMutex	SDL's activity mutex is not exported	Aurora-owned mutex plus the QuestSurface bracket (above)
Crypto++ cpu-features.h not found	The NDK ships cpu-features as source	Compiled into mkw_cryptopp on Android
Exploded racers and menu characters; smeared movie panels in the menus; then, once those were fixed, damaged eyes and slightly misplaced detail on characters	The Adreno 740 driver reads the wrong bytes when the shader multiplies an index by a stride that is not a multiple of 4. That covers the vertex fetch (ubuf.vtx_start + vidx * stride + offset) and indexed array reads (array_start + index * stride, e.g. 6-byte S16 normals). GX packs both byte-tight, so skinned models (a 1-byte PNMTXIDX first, stride 7) broke everywhere	Android pads every uploaded vertex and every indexed-array element to a 4-byte stride (padded_upload_stride in lib/gx/gx.cpp). Offsets inside a vertex or element are unchanged, and desktop is unchanged. Fixed, headset-verified 2026-09-16 at character select and a Grand Prix start

How the explosion was isolated, so the next Adreno rendering bug starts further ahead:

• The CPU side was identical to Windows: a per-draw audit of palette indices and matrices matched byte for byte. Menus reach the headset as the mono desktop image, so stereo replay was not involved either.
• Shader-side rewrites did not help and were removed: constant-index palette matrix selection, shift-free sign extension, replacing extractBits, and byte helpers rewritten with constant shifts or integer division. The last two made menus worse, which is what pointed away from any one helper.
• Moving every attribute to a 4-byte boundary on the CPU fixed the explosion. Padding only the stride, with offsets still packed, fixed it just as well, which narrows the fault to the vidx * stride term. Mario's eyes stayed wrong under both, until indexed arrays got the same element padding. That combined padding is the shipped fix. It costs one copy per vertex and per array element. Peak uploads at a 12-racer race start were about 570 KB of the 3 MB vertex buffer and 710 KB of the 8 MB storage buffer.
• KartPad's Android reports of corrupted drivers on Adreno 750 match this symptom. That is plausible but not tested.

Diagnostics that stay in the build, all read once at launch from system properties. Set them with adb shell setprop <name> <value> before starting the app:

Property	Effect
debug.wiicompiled.vtxpad 0	Turns the stride padding off, to re-check a driver update
debug.wiicompiled.validation 1	Keeps WebGPU validation and robustness on in release builds
debug.wiicompiled.inject <n>:<button>	Presses a, b, x, y, start, up, down, left or right for 12 XR frames each time <n> changes

The injector makes headset tests possible with nobody wearing the headset. Keep the display awake, drive the menus, then take a compositor screenshot:

adb shell am broadcast -a com.oculus.vrpowermanager.prox_close
adb shell setprop debug.wiicompiled.inject 1:a    # title -> licence; bump the number per press
adb shell am startservice -n com.oculus.metacam/.capture.CaptureService -a TAKE_SCREENSHOT
adb pull /sdcard/Oculus/Screenshots/<newest>.jpg

From a cold start, five a presses about 5 s apart, starting once the title screen is up, reach Grand Prix character select. A value left over from an earlier run is ignored on the first read. Presses only land while the XR session is FOCUSED.

Open measurement: the attract race advanced 603 game frames in about 14 s, roughly 43 FPS against the game's 60, with an optimized build (-O3, translated code -O2, -mcpu=cortex-a77). Profiling on the XR2 Gen 2 is the first performance task.

Verified on device since: the menus on the virtual screen, controller input (the user has driven races), and an immersive Grand Prix start with all 12 racers rendering correctly. Not yet verified: stereo comfort and scale, lifecycle (Quest menu, guardian, sleep), and a full race to the finish. When diagnosing a new device, the session log should show, in order: the loader log line (OpenXR Android loader initialized), the requirements line (which binding extension was negotiated), OpenXR Vulkan swapchains ready, the session state reaching FOCUSED, presentation=virtual-screen for the menus, and first immersive packet consumed on race entry. A black headset with a working Android mirror points at the AHardwareBuffer copy (check for vkImportSemaphoreFdKHR or EndAccess errors); a black mirror too points at Aurora itself.

Known gaps and next steps

• Device bring-up. Run on a Quest 3, capture logcat, fix what the runtime rejects. Likely first candidates: the exact XR_KHR_vulkan_enable2 device extension negotiation, Dawn's begin/end layout reporting for AHardwareBuffer imports, and swapchain format choice (R8G8B8A8_SRGB is expected).
• Performance. The desktop product targets x86-64-v3; nothing has been profiled on the XR2. Expect shader compilation stalls on first run (Aurora's pipeline cache is bundled) and start with render_scale below 1.0 if the compositor reports missed frames. XR_FB_foveation is not used yet.
• Lifecycle. Backgrounding (the Quest menu, guardian) pauses the session through the ordinary STOPPING/READY events; SDL's Android surface loss is handled by Aurora's existing Android paths. Neither has been exercised.
• Input. D-pad (trick inputs) is not bound; remap in Config.toml or bind the thumbstick directions in a follow-up. Haptics are wired but nothing calls them yet.
• Retro Rewind on device needs the mod's translation and its extracted content pushed next to DATA, exactly like the desktop product.
• Release signing and store packaging are out of scope; Build-Quest.ps1 produces debug-signed APKs for sideloading.