From c990c595f06e8cac69f051141033d63427036237 Mon Sep 17 00:00:00 2001 From: iChris4 Date: Thu, 24 Sep 2026 22:29:28 +0200 Subject: [PATCH] Render only the window in the Quest's immersive window - The pacing thread aims each immersive window eye through the window itself (AimEyesThroughWindow): it keeps its position but looks square-on at the window's plane through an off-axis frustum just around it, so the eye image is the window, at the display's pixel density (about 680x380 per eye at render_scale 0.8 instead of 1344x1408), with a two-pixel border the mask leaves transparent. The frame's views carry that pose and field of view to the projection layer. - vulkan_interop.cpp copies an eye smaller than its AHardwareBuffer into the buffer's corner, and the Quest layer's imageRect is the rendered part of the swapchain image. - These eyes are not foveated: their field of view follows the head, which would rebuild the density map every frame. - Quest only (kWindowShapedEyesSupported); the PC backends copy whole eyes and keep masking them. debug.wiicompiled.window_eyes 0 renders them whole and masked again for A/B timing. - Quest 3, paused Retro Rewind race, render_scale 1.0: Immersive window runs the GPU at level 1 (456 MHz, app GPU 11.0 ms, eyes 8.4 ms) where Immersive needs level 3 (599-640 MHz, 13.4 ms, 10.3 ms), about 42% fewer GPU cycles. No edge artifacts, image as sharp. Docs: OPENXR.md. Co-Authored-By: Claude Opus 5.5 --- OPENXR.md | 53 +++++++-- aurora-main/lib/aurora.cpp | 4 +- aurora-main/lib/webgpu/vulkan_interop.cpp | 10 +- runtime/include/vr/openxr_backend.h | 4 + runtime/src/vr/openxr_integration.cpp | 134 +++++++++++++++++++++- runtime/src/vr/openxr_vulkan.cpp | 6 +- 6 files changed, 188 insertions(+), 23 deletions(-) diff --git a/OPENXR.md b/OPENXR.md index b279af4..7de57ec 100644 --- a/OPENXR.md +++ b/OPENXR.md @@ -651,13 +651,47 @@ that keeps the colour inside the window with alpha 1 and leaves transparent blac one-pixel ramp at the edge. The triangle carries, at each corner, where that pixel's ray meets the window's plane in homogeneous window coordinates (`stereo_replay::window_mask`), which interpolate exactly across the image. With `single_pass_eyes` it is drawn in the eye's own last render pass, so it -adds no pass and no tile load; otherwise it takes a pass of its own, as the cockpit overlay does. The -game still renders the whole eye: only its alpha changes. On the Quest the backend submits the -passthrough layer, then the projection layer with `XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT` -(premultiplied alpha), then the settings panel. The flag travels with the packet, so the eyes Aurora -masked and the layer that blends them always belong to the same frame, and switching the race view -mid-race needs no safety generation: presentation stays `ImmersiveRace`. The PC backends keep their -projection layer opaque, so the window is surrounded by black there. +adds no pass and no tile load; otherwise it takes a pass of its own, as the cockpit overlay does. On +the Quest the backend submits the passthrough layer, then the projection layer with +`XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT` (premultiplied alpha), then the settings panel. +The flag travels with the packet, so the eyes Aurora masked and the layer that blends them always +belong to the same frame, and switching the race view mid-race needs no safety generation: +presentation stays `ImmersiveRace`. The PC backends keep their projection layer opaque, so the window +is surrounded by black there. + +**Only the window is rendered on the Quest.** Rather than render the whole eye and mask most of it, +the pacing thread aims each eye through the window itself (`AimEyesThroughWindow` in +`openxr_integration.cpp`): the eye keeps its position but looks square-on at the window's plane, through +an off-axis frustum just around the window, so the image is the window. It keeps the display's pixel +density (the swapchain's pixels per unit of tangent as the eye is located) at the window's size seen +from the race origin, which is fixed while the window's geometry is: about 680 x 380 per eye with the +default window at `render_scale` 0.8, against 1344 x 1408 for a whole eye. A two-pixel border around +the window is left transparent by the mask. The frame's views carry that pose and field of view to the +projection layer, whose `imageRect` is the rendered part of the swapchain image, and the compositor +reprojects it like any other. Aurora copies the smaller eye into the corner of the shared buffer +(`vulkan_interop.cpp`), and does not foveate these eyes: their field of view follows the head, which +would rebuild the density map every frame, and they are small already. The PC backends copy whole eyes +into the swapchain, so there the window's eyes stay full size and masked. +`adb shell setprop debug.wiicompiled.window_eyes 0` renders them whole and masked on the Quest too, to +compare the two within one session. + +Measured on a Quest 3 with a Retro Rewind race paused (the same 439 draw calls every frame, +`render_scale` 1.0, 60 FPS throughout), switching the race view from the headset panel: + +| Race view | GPU level and clock | App GPU per frame | Both eyes | GPU load | Compositor | +| --- | --- | --- | --- | --- | --- | +| Immersive window | 1, 456 MHz | 11.0 ms | 8.4 ms | 82% | 1.6 ms | +| Immersive | 3, 599 to 640 MHz | 13.4 ms | 10.3 ms | 88% | 0.7 ms | + +The headset raised the GPU's level for the fully immersive race and it still took longer: in clock +cycles the window's frame is about 42% cheaper (5.0 against 8.6 million), which lets the Quest keep the +GPU at its lowest level. The compositor's extra time is the passthrough. During a race at `render_scale` +0.8, switching `debug.wiicompiled.window_eyes`, both eyes took 6.3 to 7.5 ms through the window against +9.1 to 9.5 ms whole and masked at similar draw counts (the compositor's `SF` field read 0.31 against +0.80), with the clock wandering between 350 and 600 MHz. The saving is smaller than the eye's pixels +(about 13% of a whole eye's) would suggest because much of an eye's cost is the geometry of every draw, +which each eye still processes; it grows with `render_scale`. Read the VrApi line's +`CPU4/GPU=,MHz` before comparing two timings. `gx_fifo_tests` covers the window's geometry (its corners through an asymmetric eye frustum, its agreement with the HUD's placement, an eye turned away or beyond the window, a sideways step), and @@ -936,9 +970,8 @@ ends, including mid-frame flushes, so live setting changes cannot invalidate pen analog grips (Touch); the simple controller profile cannot grab. - The headset settings panel has no laser beam, only the cursor on the panel itself, and text fields cannot be typed into without a keyboard. -- The immersive window renders the whole eye and only masks it, so it costs what a fully immersive - race costs, plus the passthrough's compositing. Hands and a separate VR wheel are masked with the - rest of the eye, so outside the window they are not seen. +- On the PC the immersive window renders the whole eye and only masks it, so it costs what a fully + immersive race costs. Hands and a separate VR wheel are seen only through the window. - The desktop window remains available as a mirror/fallback. OpenXR diagnostics are written to the normal run log under diff --git a/aurora-main/lib/aurora.cpp b/aurora-main/lib/aurora.cpp index 38fe083..d6cdcec 100644 --- a/aurora-main/lib/aurora.cpp +++ b/aurora-main/lib/aurora.cpp @@ -841,7 +841,9 @@ gfx::StereoReplayFrame make_stereo_replay_frame(const AuroraStereoFrame& input, .msaaSamples = webgpu::g_graphicsConfig.msaaSamples, .depthFormat = owned.depth.format, }; - if (input.mode == AURORA_STEREO_FRAME_IMMERSIVE_REPLAY) { + // Not the immersive window's eyes: the host may aim them through the window, whose field of + // view then changes with every head movement and would rebuild the density map each frame. + if (input.mode == AURORA_STEREO_FRAME_IMMERSIVE_REPLAY && !input.window) { view.target.foveatedColorView = foveated_eye_view(eye, input.eyes[eye]); } std::memcpy(&view.projection, input.eyes[eye].projection, sizeof(view.projection)); diff --git a/aurora-main/lib/webgpu/vulkan_interop.cpp b/aurora-main/lib/webgpu/vulkan_interop.cpp index 2ecd8b7..665aa6a 100644 --- a/aurora-main/lib/webgpu/vulkan_interop.cpp +++ b/aurora-main/lib/webgpu/vulkan_interop.cpp @@ -255,11 +255,13 @@ public: } private: + // An eye may be smaller than its buffer (the immersive window's eyes are the window only): it is + // copied into the buffer's top-left corner, and the OpenXR side shows just that rectangle. Import* EnsureImport(uint32_t eye, const stereo::EyeImage& source) noexcept { const auto& target = m_targets[eye]; - if (source.texture == nullptr || source.format != m_auroraFormat || - source.size.width != target.width || source.size.height != target.height) { - Log.error("Stereo image {} does not match its OpenXR Vulkan target ({}x{} vs {}x{})", eye, + if (source.texture == nullptr || source.format != m_auroraFormat || source.size.width == 0 || + source.size.height == 0 || source.size.width > target.width || source.size.height > target.height) { + Log.error("Stereo image {} does not fit its OpenXR Vulkan target ({}x{} in {}x{})", eye, source.size.width, source.size.height, target.width, target.height); return nullptr; } @@ -421,7 +423,7 @@ private: .origin = {}, .aspect = wgpu::TextureAspect::All, }; - const wgpu::Extent3D extent{import.width, import.height, 1}; + const wgpu::Extent3D extent{sources[eye].size.width, sources[eye].size.height, 1}; encoder.CopyTextureToTexture(&source, &destination, &extent); m_encodedImports[eye] = imports[eye]; } diff --git a/runtime/include/vr/openxr_backend.h b/runtime/include/vr/openxr_backend.h index db9e70d..a09b459 100644 --- a/runtime/include/vr/openxr_backend.h +++ b/runtime/include/vr/openxr_backend.h @@ -136,6 +136,10 @@ struct OpenXRPresentation { // the race's 2D-layer screen (AuroraStereoFrame::window), so the projection // layer is blended by its alpha over whatever is under it. bool immersive_window = false; + // The immersive window's eyes were aimed through the window itself, so each + // is only the window: its image is render_width x render_height, the + // top-left part of the eye's swapchain image the layer shows. + bool window_eyes = false; // Show the room through the headset's cameras around the virtual screen or // the immersive window (OpenXRPassthrough). Taken when the presentation is diff --git a/runtime/src/vr/openxr_integration.cpp b/runtime/src/vr/openxr_integration.cpp index c9f2cca..849d8b9 100644 --- a/runtime/src/vr/openxr_integration.cpp +++ b/runtime/src/vr/openxr_integration.cpp @@ -87,6 +87,16 @@ using GraphicsBackend = OpenXRVulkanBackend; inline constexpr const char* kGraphicsBackendName = "Vulkan"; #endif +// Whether the immersive window's eyes can be aimed through the window, so that only the window is +// rendered: the backend has to show just the part of each eye image they fill. The Quest's shared +// buffers and projection layer do; the PC backends copy whole eyes, so there the window's eyes stay +// full size and are only masked. +#if defined(__ANDROID__) +inline constexpr bool kWindowShapedEyesSupported = true; +#else +inline constexpr bool kWindowShapedEyesSupported = false; +#endif + struct Quaternion { float x = 0.0f; float y = 0.0f; @@ -866,6 +876,7 @@ private: aurora_set_stereo_panel_layer(panel_layer); PollEyePassesOverride(); PollFoveationOverride(); + PollWindowEyesOverride(); presentation.panel.requested = panel_layer && OpenXRSettingsPanelOpen(); // Pipeline caches are stored where their stall is least visible: once when a race @@ -968,8 +979,7 @@ private: // configured diorama scale. Head translation and IPD are the // only things this multiplies, so a one-frame disagreement with // the camera's own switch is not observable. - BuildPublishedFrame(frame, immersive, policy.EffectiveUnitsPerMeter(), - policy.content_tag); + BuildPublishedFrame(frame, immersive, policy); diagnostics::OnPacketPublished(); published_.store(&published_frame_, std::memory_order_release); } @@ -1131,7 +1141,7 @@ private: { const diagnostics::ScopedStage publish_timer(diagnostics::Stage::Publish); std::lock_guard lock(published_mutex_); - BuildPublishedFrame(packet, immersive, policy.EffectiveUnitsPerMeter(), policy.content_tag); + BuildPublishedFrame(packet, immersive, policy); diagnostics::OnPacketPublished(); published_.store(&published_frame_, std::memory_order_release); } @@ -1246,8 +1256,11 @@ private: return true; } - void BuildPublishedFrame(const OpenXRBackendFrame& source, bool immersive, - float units_per_meter, uint64_t content_tag) noexcept { + // Also aims the immersive window's eyes through the window, in `source` itself, so that the layer + // later built from it shows the eyes as they were rendered. + void BuildPublishedFrame(OpenXRBackendFrame& source, bool immersive, const MkwVRPolicySnapshot& policy) noexcept { + const float units_per_meter = policy.EffectiveUnitsPerMeter(); + const uint64_t content_tag = policy.content_tag; ApplyPendingReferenceSpaceChange(source.xr_frame); auto& destination = published_frame_.frame; destination = {}; @@ -1279,6 +1292,13 @@ private: base_position_valid_ = true; } last_immersive_ = true; + if (source.presentation.immersive_window && WindowShapedEyes()) { + source.presentation.window_eyes = AimEyesThroughWindow(source, policy); + for (uint32_t eye = 0; eye < kOpenXREyeCount; ++eye) { + destination.eyes[eye].width = source.render_width[eye]; + destination.eyes[eye].height = source.render_height[eye]; + } + } // Read once so both eyes are built from the same angle even if the // settings slider moves between them. const float lean_back_radians = @@ -1336,6 +1356,83 @@ private: } } + // The immersive window's eyes, aimed through the window itself: each keeps its position but looks + // square-on at the window's plane, through an off-axis frustum just around the window, so its + // image is the window and nothing outside it is rendered. The image keeps the display's pixel + // density at the window's size seen from the race origin (fixed while the window's geometry is), + // plus a two-pixel border that Aurora's mask leaves transparent, so the compositor finds nothing + // at the image's edge. The frame's views carry this pose and field of view to the projection + // layer, which the compositor reprojects like any other. False, with nothing changed, when the + // window cannot be placed or an eye is not in front of it. + bool AimEyesThroughWindow(OpenXRBackendFrame& frame, const MkwVRPolicySnapshot& policy) const noexcept { + const float distance = policy.config.hud_distance_meters; + const float half_width = 0.5f * policy.config.hud_width_meters; + XrPosef window{}; + if (!(half_width > 0.0f) || !(distance > 0.0f) || !RaceScreenPose(frame, policy, window)) { + return false; + } + float picture_aspect = 0.0f; + float snapshot_aspect = 0.0f; + if (!aurora_get_stereo_screen_aspects(&picture_aspect, &snapshot_aspect) || !(picture_aspect > 0.0f)) { + picture_aspect = 16.0f / 9.0f; // stereo_hud_screen's own fallback + } + const float half_height = half_width / picture_aspect; + const Quaternion to_window = + Conjugate(Normalize({window.orientation.x, window.orientation.y, window.orientation.z, window.orientation.w})); + constexpr uint32_t kBorder = 2; + std::array fov{}; + std::array width{}; + std::array height{}; + for (uint32_t eye = 0; eye < kOpenXREyeCount; ++eye) { + const XrView& view = frame.xr_frame.views[eye]; + // The eye in the window's frame, whose +Z faces the viewer. + const std::array at = Rotate( + to_window, {view.pose.position.x - window.position.x, view.pose.position.y - window.position.y, + view.pose.position.z - window.position.z}); + const float located_x = std::tan(view.fov.angleRight) - std::tan(view.fov.angleLeft); + const float located_y = std::tan(view.fov.angleUp) - std::tan(view.fov.angleDown); + if (!(at[2] > 0.05f) || !(located_x > 0.0f) || !(located_y > 0.0f) || frame.render_width[eye] <= 2 * kBorder || + frame.render_height[eye] <= 2 * kBorder) { + return false; + } + // The display's pixels per unit of tangent, as the eye was located, across the window's + // tangent extent seen straight on from the race origin. + const auto pixels = [&](uint32_t full, float located, float half_extent) { + const float content = std::floor(static_cast(full) / located * (2.0f * half_extent / distance)); + return std::clamp(static_cast(std::max(content, 1.0f)) + 2 * kBorder, 2 * kBorder + 1, + full); + }; + width[eye] = pixels(frame.render_width[eye], located_x, half_width); + height[eye] = pixels(frame.render_height[eye], located_y, half_height); + // This frame's frustum: the window's edges seen from where the eye is, widened by the border. + const float left = (-half_width - at[0]) / at[2]; + const float right = (half_width - at[0]) / at[2]; + const float down = (-half_height - at[1]) / at[2]; + const float up = (half_height - at[1]) / at[2]; + const float border_x = (right - left) * kBorder / static_cast(width[eye] - 2 * kBorder); + const float border_y = (up - down) * kBorder / static_cast(height[eye] - 2 * kBorder); + fov[eye].angleLeft = std::atan(left - border_x); + fov[eye].angleRight = std::atan(right + border_x); + fov[eye].angleUp = std::atan(up + border_y); + fov[eye].angleDown = std::atan(down - border_y); + } + for (uint32_t eye = 0; eye < kOpenXREyeCount; ++eye) { + frame.xr_frame.views[eye].pose.orientation = window.orientation; + frame.xr_frame.views[eye].fov = fov[eye]; + frame.render_width[eye] = width[eye]; + frame.render_height[eye] = height[eye]; + } + return true; + } + + bool WindowShapedEyes() const noexcept { +#if defined(__ANDROID__) + return kWindowShapedEyesSupported && !window_eyes_forced_off_; +#else + return kWindowShapedEyesSupported; +#endif + } + // The seated frame the controllers are located in for hand steering: the // immersive base the eye transforms use, from the previous frame (this // frame's is latched after input). @@ -1532,6 +1629,27 @@ private: #endif } + // Android: `adb shell setprop debug.wiicompiled.window_eyes 0` renders the immersive window's eyes + // whole and only masks them, as the PC does, to compare the cost within one session; an empty + // value aims them through the window again. Read about once a second. + void PollWindowEyesOverride() noexcept { +#if defined(__ANDROID__) + if (window_eyes_poll_ != 0) { + --window_eyes_poll_; + return; + } + window_eyes_poll_ = 72; + char value[PROP_VALUE_MAX] = {}; + const bool off = __system_property_get("debug.wiicompiled.window_eyes", value) > 0 && value[0] == '0'; + if (off != window_eyes_forced_off_) { + window_eyes_forced_off_ = off; + RT_LOG(RT_TAG_RUNTIME) << "OpenXR: immersive window eyes " + << (off ? "rendered whole and masked" : "aimed through the window") + << " (debug.wiicompiled.window_eyes)" << std::endl; + } +#endif + } + // The settings panel's layer hangs exactly where its pointer hits are // tested, the rectangle it used to cover in the eyes. static void PlacePanelLayer(OpenXRBackendFrame& frame, const OpenXRPointerScreen& screen) noexcept { @@ -1645,7 +1763,9 @@ private: if (diagnostics::ConsumeSessionInfoRequest()) { LogDiagnosticSession(frame); } - if (frame.xr_frame.should_render && frame.xr_frame.views_valid) { + // The immersive window's eyes are aimed through the window, so their fields of view follow + // the head and are no longer the headset's. + if (frame.xr_frame.should_render && frame.xr_frame.views_valid && !frame.presentation.window_eyes) { diagnostics::OnViewGeometry(DiagnosticViewGeometry(frame)); } } @@ -1759,6 +1879,8 @@ private: int eye_passes_override_ = -1; uint32_t foveation_poll_ = 0; int foveation_override_ = -1; + uint32_t window_eyes_poll_ = 0; + bool window_eyes_forced_off_ = false; #endif std::unique_ptr input_; std::thread pacing_thread_; diff --git a/runtime/src/vr/openxr_vulkan.cpp b/runtime/src/vr/openxr_vulkan.cpp index 8c1f146..6de16f4 100644 --- a/runtime/src/vr/openxr_vulkan.cpp +++ b/runtime/src/vr/openxr_vulkan.cpp @@ -935,10 +935,12 @@ public: views[eye].pose.position = frame.xr_frame.views[eye].pose.position; views[eye].fov = frame.xr_frame.views[eye].fov; views[eye].subImage.swapchain = retained_swapchains_[eye].handle; + // The part of the image the eye was rendered into: all of it, except for the immersive + // window's eyes, which are only the window (OpenXRPresentation::window_eyes). views[eye].subImage.imageRect = { {0, 0}, - {static_cast(retained_swapchains_[eye].width), - static_cast(retained_swapchains_[eye].height)}}; + {static_cast(std::min(frame.render_width[eye], retained_swapchains_[eye].width)), + static_cast(std::min(frame.render_height[eye], retained_swapchains_[eye].height))}}; views[eye].subImage.imageArrayIndex = 0; } XrCompositionLayerProjection projection{XR_TYPE_COMPOSITION_LAYER_PROJECTION};