Render only the window in the Quest's immersive window

- The pacing thread aims each immersive window eye through the window itself (AimEyesThroughWindow):
  it keeps its position but looks square-on at the window's plane through an off-axis frustum just
  around it, so the eye image is the window, at the display's pixel density (about 680x380 per eye
  at render_scale 0.8 instead of 1344x1408), with a two-pixel border the mask leaves transparent.
  The frame's views carry that pose and field of view to the projection layer.
- vulkan_interop.cpp copies an eye smaller than its AHardwareBuffer into the buffer's corner, and
  the Quest layer's imageRect is the rendered part of the swapchain image.
- These eyes are not foveated: their field of view follows the head, which would rebuild the
  density map every frame.
- Quest only (kWindowShapedEyesSupported); the PC backends copy whole eyes and keep masking them.
  debug.wiicompiled.window_eyes 0 renders them whole and masked again for A/B timing.
- Quest 3, paused Retro Rewind race, render_scale 1.0: Immersive window runs the GPU at level 1
  (456 MHz, app GPU 11.0 ms, eyes 8.4 ms) where Immersive needs level 3 (599-640 MHz, 13.4 ms,
  10.3 ms), about 42% fewer GPU cycles. No edge artifacts, image as sharp. Docs: OPENXR.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

OPENXR.md

@@ -651,13 +651,47 @@ that keeps the colour inside the window with alpha 1 and leaves transparent blac
 one-pixel ramp at the edge. The triangle carries, at each corner, where that pixel's ray meets the
 window's plane in homogeneous window coordinates (`stereo_replay::window_mask`), which interpolate
 exactly across the image. With `single_pass_eyes` it is drawn in the eye's own last render pass, so it
-adds no pass and no tile load; otherwise it takes a pass of its own, as the cockpit overlay does. The
+adds no pass and no tile load; otherwise it takes a pass of its own, as the cockpit overlay does. On
-game still renders the whole eye: only its alpha changes. On the Quest the backend submits the
+the Quest the backend submits the passthrough layer, then the projection layer with
-passthrough layer, then the projection layer with `XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT`
+`XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT` (premultiplied alpha), then the settings panel.
-(premultiplied alpha), then the settings panel. The flag travels with the packet, so the eyes Aurora
+The flag travels with the packet, so the eyes Aurora masked and the layer that blends them always
-masked and the layer that blends them always belong to the same frame, and switching the race view
+belong to the same frame, and switching the race view mid-race needs no safety generation:
-mid-race needs no safety generation: presentation stays `ImmersiveRace`. The PC backends keep their
+presentation stays `ImmersiveRace`. The PC backends keep their projection layer opaque, so the window
-projection layer opaque, so the window is surrounded by black there.
+is surrounded by black there.
+
+**Only the window is rendered on the Quest.** Rather than render the whole eye and mask most of it,
+the pacing thread aims each eye through the window itself (`AimEyesThroughWindow` in
+`openxr_integration.cpp`): the eye keeps its position but looks square-on at the window's plane, through
+an off-axis frustum just around the window, so the image is the window. It keeps the display's pixel
+density (the swapchain's pixels per unit of tangent as the eye is located) at the window's size seen
+from the race origin, which is fixed while the window's geometry is: about 680 x 380 per eye with the
+default window at `render_scale` 0.8, against 1344 x 1408 for a whole eye. A two-pixel border around
+the window is left transparent by the mask. The frame's views carry that pose and field of view to the
+projection layer, whose `imageRect` is the rendered part of the swapchain image, and the compositor
+reprojects it like any other. Aurora copies the smaller eye into the corner of the shared buffer
+(`vulkan_interop.cpp`), and does not foveate these eyes: their field of view follows the head, which
+would rebuild the density map every frame, and they are small already. The PC backends copy whole eyes
+into the swapchain, so there the window's eyes stay full size and masked.
+`adb shell setprop debug.wiicompiled.window_eyes 0` renders them whole and masked on the Quest too, to
+compare the two within one session.
+
+Measured on a Quest 3 with a Retro Rewind race paused (the same 439 draw calls every frame,
+`render_scale` 1.0, 60 FPS throughout), switching the race view from the headset panel:
+
+| Race view | GPU level and clock | App GPU per frame | Both eyes | GPU load | Compositor |
+| --- | --- | --- | --- | --- | --- |
+| Immersive window | 1, 456 MHz | 11.0 ms | 8.4 ms | 82% | 1.6 ms |
+| Immersive | 3, 599 to 640 MHz | 13.4 ms | 10.3 ms | 88% | 0.7 ms |
+
+The headset raised the GPU's level for the fully immersive race and it still took longer: in clock
+cycles the window's frame is about 42% cheaper (5.0 against 8.6 million), which lets the Quest keep the
+GPU at its lowest level. The compositor's extra time is the passthrough. During a race at `render_scale`
+0.8, switching `debug.wiicompiled.window_eyes`, both eyes took 6.3 to 7.5 ms through the window against
+9.1 to 9.5 ms whole and masked at similar draw counts (the compositor's `SF` field read 0.31 against
+0.80), with the clock wandering between 350 and 600 MHz. The saving is smaller than the eye's pixels
+(about 13% of a whole eye's) would suggest because much of an eye's cost is the geometry of every draw,
+which each eye still processes; it grows with `render_scale`. Read the VrApi line's
+`CPU4/GPU=<levels>,<clocks>MHz` before comparing two timings.
 
 `gx_fifo_tests` covers the window's geometry (its corners through an asymmetric eye frustum, its
 agreement with the HUD's placement, an eye turned away or beyond the window, a sideways step), and
@@ -936,9 +970,8 @@ ends, including mid-frame flushes, so live setting changes cannot invalidate pen
   analog grips (Touch); the simple controller profile cannot grab.
 - The headset settings panel has no laser beam, only the cursor on the panel itself, and text fields
   cannot be typed into without a keyboard.
-- The immersive window renders the whole eye and only masks it, so it costs what a fully immersive
+- On the PC the immersive window renders the whole eye and only masks it, so it costs what a fully
-  race costs, plus the passthrough's compositing. Hands and a separate VR wheel are masked with the
+  immersive race costs. Hands and a separate VR wheel are seen only through the window.
-  rest of the eye, so outside the window they are not seen.
 - The desktop window remains available as a mirror/fallback.
 
 OpenXR diagnostics are written to the normal run log under

aurora-main/lib/aurora.cpp

@@ -841,7 +841,9 @@ gfx::StereoReplayFrame make_stereo_replay_frame(const AuroraStereoFrame& input,
         .msaaSamples = webgpu::g_graphicsConfig.msaaSamples,
         .depthFormat = owned.depth.format,
     };
-    if (input.mode == AURORA_STEREO_FRAME_IMMERSIVE_REPLAY) {
+    // Not the immersive window's eyes: the host may aim them through the window, whose field of
+    // view then changes with every head movement and would rebuild the density map each frame.
+    if (input.mode == AURORA_STEREO_FRAME_IMMERSIVE_REPLAY && !input.window) {
       view.target.foveatedColorView = foveated_eye_view(eye, input.eyes[eye]);
     }
     std::memcpy(&view.projection, input.eyes[eye].projection, sizeof(view.projection));

aurora-main/lib/webgpu/vulkan_interop.cpp

@@ -255,11 +255,13 @@ public:
   }
 
 private:
+  // An eye may be smaller than its buffer (the immersive window's eyes are the window only): it is
+  // copied into the buffer's top-left corner, and the OpenXR side shows just that rectangle.
   Import* EnsureImport(uint32_t eye, const stereo::EyeImage& source) noexcept {
     const auto& target = m_targets[eye];
-    if (source.texture == nullptr || source.format != m_auroraFormat ||
+    if (source.texture == nullptr || source.format != m_auroraFormat || source.size.width == 0 ||
-        source.size.width != target.width || source.size.height != target.height) {
+        source.size.height == 0 || source.size.width > target.width || source.size.height > target.height) {
-      Log.error("Stereo image {} does not match its OpenXR Vulkan target ({}x{} vs {}x{})", eye,
+      Log.error("Stereo image {} does not fit its OpenXR Vulkan target ({}x{} in {}x{})", eye,
                 source.size.width, source.size.height, target.width, target.height);
       return nullptr;
     }
@@ -421,7 +423,7 @@ private:
           .origin = {},
           .aspect = wgpu::TextureAspect::All,
       };
-      const wgpu::Extent3D extent{import.width, import.height, 1};
+      const wgpu::Extent3D extent{sources[eye].size.width, sources[eye].size.height, 1};
       encoder.CopyTextureToTexture(&source, &destination, &extent);
       m_encodedImports[eye] = imports[eye];
     }

runtime/include/vr/openxr_backend.h

@@ -136,6 +136,10 @@ struct OpenXRPresentation {
     // the race's 2D-layer screen (AuroraStereoFrame::window), so the projection
     // layer is blended by its alpha over whatever is under it.
     bool immersive_window = false;
+    // The immersive window's eyes were aimed through the window itself, so each
+    // is only the window: its image is render_width x render_height, the
+    // top-left part of the eye's swapchain image the layer shows.
+    bool window_eyes = false;
 
     // Show the room through the headset's cameras around the virtual screen or
     // the immersive window (OpenXRPassthrough). Taken when the presentation is

runtime/src/vr/openxr_integration.cpp

@@ -87,6 +87,16 @@ using GraphicsBackend = OpenXRVulkanBackend;
 inline constexpr const char* kGraphicsBackendName = "Vulkan";
 #endif
 
+// Whether the immersive window's eyes can be aimed through the window, so that only the window is
+// rendered: the backend has to show just the part of each eye image they fill. The Quest's shared
+// buffers and projection layer do; the PC backends copy whole eyes, so there the window's eyes stay
+// full size and are only masked.
+#if defined(__ANDROID__)
+inline constexpr bool kWindowShapedEyesSupported = true;
+#else
+inline constexpr bool kWindowShapedEyesSupported = false;
+#endif
+
 struct Quaternion {
     float x = 0.0f;
     float y = 0.0f;
@@ -866,6 +876,7 @@ private:
             aurora_set_stereo_panel_layer(panel_layer);
             PollEyePassesOverride();
             PollFoveationOverride();
+            PollWindowEyesOverride();
             presentation.panel.requested = panel_layer && OpenXRSettingsPanelOpen();
 
             // Pipeline caches are stored where their stall is least visible: once when a race
@@ -968,8 +979,7 @@ private:
                 // configured diorama scale. Head translation and IPD are the
                 // only things this multiplies, so a one-frame disagreement with
                 // the camera's own switch is not observable.
-                BuildPublishedFrame(frame, immersive, policy.EffectiveUnitsPerMeter(),
+                BuildPublishedFrame(frame, immersive, policy);
-                                    policy.content_tag);
                 diagnostics::OnPacketPublished();
                 published_.store(&published_frame_, std::memory_order_release);
             }
@@ -1131,7 +1141,7 @@ private:
         {
             const diagnostics::ScopedStage publish_timer(diagnostics::Stage::Publish);
             std::lock_guard lock(published_mutex_);
-            BuildPublishedFrame(packet, immersive, policy.EffectiveUnitsPerMeter(), policy.content_tag);
+            BuildPublishedFrame(packet, immersive, policy);
             diagnostics::OnPacketPublished();
             published_.store(&published_frame_, std::memory_order_release);
         }
@@ -1246,8 +1256,11 @@ private:
         return true;
     }
 
-    void BuildPublishedFrame(const OpenXRBackendFrame& source, bool immersive,
+    // Also aims the immersive window's eyes through the window, in `source` itself, so that the layer
-                             float units_per_meter, uint64_t content_tag) noexcept {
+    // later built from it shows the eyes as they were rendered.
+    void BuildPublishedFrame(OpenXRBackendFrame& source, bool immersive, const MkwVRPolicySnapshot& policy) noexcept {
+        const float units_per_meter = policy.EffectiveUnitsPerMeter();
+        const uint64_t content_tag = policy.content_tag;
         ApplyPendingReferenceSpaceChange(source.xr_frame);
         auto& destination = published_frame_.frame;
         destination = {};
@@ -1279,6 +1292,13 @@ private:
             base_position_valid_ = true;
         }
         last_immersive_ = true;
+        if (source.presentation.immersive_window && WindowShapedEyes()) {
+            source.presentation.window_eyes = AimEyesThroughWindow(source, policy);
+            for (uint32_t eye = 0; eye < kOpenXREyeCount; ++eye) {
+                destination.eyes[eye].width = source.render_width[eye];
+                destination.eyes[eye].height = source.render_height[eye];
+            }
+        }
         // Read once so both eyes are built from the same angle even if the
         // settings slider moves between them.
         const float lean_back_radians =
@@ -1336,6 +1356,83 @@ private:
         }
     }
 
+    // The immersive window's eyes, aimed through the window itself: each keeps its position but looks
+    // square-on at the window's plane, through an off-axis frustum just around the window, so its
+    // image is the window and nothing outside it is rendered. The image keeps the display's pixel
+    // density at the window's size seen from the race origin (fixed while the window's geometry is),
+    // plus a two-pixel border that Aurora's mask leaves transparent, so the compositor finds nothing
+    // at the image's edge. The frame's views carry this pose and field of view to the projection
+    // layer, which the compositor reprojects like any other. False, with nothing changed, when the
+    // window cannot be placed or an eye is not in front of it.
+    bool AimEyesThroughWindow(OpenXRBackendFrame& frame, const MkwVRPolicySnapshot& policy) const noexcept {
+        const float distance = policy.config.hud_distance_meters;
+        const float half_width = 0.5f * policy.config.hud_width_meters;
+        XrPosef window{};
+        if (!(half_width > 0.0f) || !(distance > 0.0f) || !RaceScreenPose(frame, policy, window)) {
+            return false;
+        }
+        float picture_aspect = 0.0f;
+        float snapshot_aspect = 0.0f;
+        if (!aurora_get_stereo_screen_aspects(&picture_aspect, &snapshot_aspect) || !(picture_aspect > 0.0f)) {
+            picture_aspect = 16.0f / 9.0f; // stereo_hud_screen's own fallback
+        }
+        const float half_height = half_width / picture_aspect;
+        const Quaternion to_window =
+            Conjugate(Normalize({window.orientation.x, window.orientation.y, window.orientation.z, window.orientation.w}));
+        constexpr uint32_t kBorder = 2;
+        std::array<XrFovf, kOpenXREyeCount> fov{};
+        std::array<uint32_t, kOpenXREyeCount> width{};
+        std::array<uint32_t, kOpenXREyeCount> height{};
+        for (uint32_t eye = 0; eye < kOpenXREyeCount; ++eye) {
+            const XrView& view = frame.xr_frame.views[eye];
+            // The eye in the window's frame, whose +Z faces the viewer.
+            const std::array<float, 3> at = Rotate(
+                to_window, {view.pose.position.x - window.position.x, view.pose.position.y - window.position.y,
+                            view.pose.position.z - window.position.z});
+            const float located_x = std::tan(view.fov.angleRight) - std::tan(view.fov.angleLeft);
+            const float located_y = std::tan(view.fov.angleUp) - std::tan(view.fov.angleDown);
+            if (!(at[2] > 0.05f) || !(located_x > 0.0f) || !(located_y > 0.0f) || frame.render_width[eye] <= 2 * kBorder ||
+                frame.render_height[eye] <= 2 * kBorder) {
+                return false;
+            }
+            // The display's pixels per unit of tangent, as the eye was located, across the window's
+            // tangent extent seen straight on from the race origin.
+            const auto pixels = [&](uint32_t full, float located, float half_extent) {
+                const float content = std::floor(static_cast<float>(full) / located * (2.0f * half_extent / distance));
+                return std::clamp<uint32_t>(static_cast<uint32_t>(std::max(content, 1.0f)) + 2 * kBorder, 2 * kBorder + 1,
+                                            full);
+            };
+            width[eye] = pixels(frame.render_width[eye], located_x, half_width);
+            height[eye] = pixels(frame.render_height[eye], located_y, half_height);
+            // This frame's frustum: the window's edges seen from where the eye is, widened by the border.
+            const float left = (-half_width - at[0]) / at[2];
+            const float right = (half_width - at[0]) / at[2];
+            const float down = (-half_height - at[1]) / at[2];
+            const float up = (half_height - at[1]) / at[2];
+            const float border_x = (right - left) * kBorder / static_cast<float>(width[eye] - 2 * kBorder);
+            const float border_y = (up - down) * kBorder / static_cast<float>(height[eye] - 2 * kBorder);
+            fov[eye].angleLeft = std::atan(left - border_x);
+            fov[eye].angleRight = std::atan(right + border_x);
+            fov[eye].angleUp = std::atan(up + border_y);
+            fov[eye].angleDown = std::atan(down - border_y);
+        }
+        for (uint32_t eye = 0; eye < kOpenXREyeCount; ++eye) {
+            frame.xr_frame.views[eye].pose.orientation = window.orientation;
+            frame.xr_frame.views[eye].fov = fov[eye];
+            frame.render_width[eye] = width[eye];
+            frame.render_height[eye] = height[eye];
+        }
+        return true;
+    }
+
+    bool WindowShapedEyes() const noexcept {
+#if defined(__ANDROID__)
+        return kWindowShapedEyesSupported && !window_eyes_forced_off_;
+#else
+        return kWindowShapedEyesSupported;
+#endif
+    }
+
     // The seated frame the controllers are located in for hand steering: the
     // immersive base the eye transforms use, from the previous frame (this
     // frame's is latched after input).
@@ -1532,6 +1629,27 @@ private:
 #endif
     }
 
+    // Android: `adb shell setprop debug.wiicompiled.window_eyes 0` renders the immersive window's eyes
+    // whole and only masks them, as the PC does, to compare the cost within one session; an empty
+    // value aims them through the window again. Read about once a second.
+    void PollWindowEyesOverride() noexcept {
+#if defined(__ANDROID__)
+        if (window_eyes_poll_ != 0) {
+            --window_eyes_poll_;
+            return;
+        }
+        window_eyes_poll_ = 72;
+        char value[PROP_VALUE_MAX] = {};
+        const bool off = __system_property_get("debug.wiicompiled.window_eyes", value) > 0 && value[0] == '0';
+        if (off != window_eyes_forced_off_) {
+            window_eyes_forced_off_ = off;
+            RT_LOG(RT_TAG_RUNTIME) << "OpenXR: immersive window eyes "
+                                   << (off ? "rendered whole and masked" : "aimed through the window")
+                                   << " (debug.wiicompiled.window_eyes)" << std::endl;
+        }
+#endif
+    }
+
     // The settings panel's layer hangs exactly where its pointer hits are
     // tested, the rectangle it used to cover in the eyes.
     static void PlacePanelLayer(OpenXRBackendFrame& frame, const OpenXRPointerScreen& screen) noexcept {
@@ -1645,7 +1763,9 @@ private:
         if (diagnostics::ConsumeSessionInfoRequest()) {
             LogDiagnosticSession(frame);
         }
-        if (frame.xr_frame.should_render && frame.xr_frame.views_valid) {
+        // The immersive window's eyes are aimed through the window, so their fields of view follow
+        // the head and are no longer the headset's.
+        if (frame.xr_frame.should_render && frame.xr_frame.views_valid && !frame.presentation.window_eyes) {
             diagnostics::OnViewGeometry(DiagnosticViewGeometry(frame));
         }
     }
@@ -1759,6 +1879,8 @@ private:
     int eye_passes_override_ = -1;
     uint32_t foveation_poll_ = 0;
     int foveation_override_ = -1;
+    uint32_t window_eyes_poll_ = 0;
+    bool window_eyes_forced_off_ = false;
 #endif
     std::unique_ptr<OpenXRInput> input_;
     std::thread pacing_thread_;

runtime/src/vr/openxr_vulkan.cpp

@@ -935,10 +935,12 @@ public:
             views[eye].pose.position = frame.xr_frame.views[eye].pose.position;
             views[eye].fov = frame.xr_frame.views[eye].fov;
             views[eye].subImage.swapchain = retained_swapchains_[eye].handle;
+            // The part of the image the eye was rendered into: all of it, except for the immersive
+            // window's eyes, which are only the window (OpenXRPresentation::window_eyes).
             views[eye].subImage.imageRect = {
                 {0, 0},
-                {static_cast<int32_t>(retained_swapchains_[eye].width),
+                {static_cast<int32_t>(std::min(frame.render_width[eye], retained_swapchains_[eye].width)),
-                 static_cast<int32_t>(retained_swapchains_[eye].height)}};
+                 static_cast<int32_t>(std::min(frame.render_height[eye], retained_swapchains_[eye].height))}};
             views[eye].subImage.imageArrayIndex = 0;
         }
         XrCompositionLayerProjection projection{XR_TYPE_COMPOSITION_LAYER_PROJECTION};