diff --git a/OPENXR.md b/OPENXR.md
index 16c6a9a..650e796 100644
--- a/OPENXR.md
+++ b/OPENXR.md
@@ -139,6 +139,15 @@ and 0.8 on the Quest, whose mobile GPU needs the headroom.
launch and govern both the menu screen and the in-race 2D screen, so 2D content keeps its place
across the transition. `hud_virtual_screen` decides whether the race's 2D layer uses that screen;
it is live and can be flipped from the F10 settings bar.
+`passthrough` (Quest only, default on) shows the room through the headset's cameras around the
+menu screen and every other virtual screen, instead of black: an `XR_FB_passthrough`
+reconstruction layer submitted under the screen's quad, as PPSSPP VR does, with the blend mode
+left `OPAQUE`. An immersive race never shows it, and the cameras are paused for the race. It is
+live, from the headset panel's VR tab or the launcher's Settings page. The app declares
+`com.oculus.feature.PASSTHROUGH`, without which Horizon OS composites nothing for that layer.
+So that the room frames the picture rather than black bands, the Quest's menu quad shows only the
+part of its eye-sized image Aurora draws into (the desktop snapshot, and the in-eye settings
+panel's rectangle), at the same size per pixel, so nothing moves.
`stop_at_display_copy` ends eye replay at the final `GXCopyDisp`, matching the frame shown on the
desktop. `skip_copy_clears` independently suppresses the EFB reset performed after a copy. Both
default on and can be changed live from the F10 settings bar for diagnostics.
@@ -197,7 +206,7 @@ with the screen the renderer is showing, and the point it meets is where the cur
is nothing to recenter. On the menu screen that is the quad layer, `hud_width_meters` across with
the eye texture's aspect, and the pointer spans the game picture inside it (Aurora letterboxes the
desktop image into the quad and the picture into the desktop image, so a 4:3 picture keeps its
-pillarboxes). During a race it is the 2D layer's screen, `hud_distance_meters` ahead of the latched
+pillarboxes; the Quest crops the quad to the desktop image without changing where it is). During a race it is the 2D layer's screen, `hud_distance_meters` ahead of the latched
race origin and turned by the lean-back angle, with the picture's aspect. With
`hud_virtual_screen = false` the race's 2D layer has no fixed place and the pointer is off. The
game's own pointer switch (`KPADEnableDpd` / `KPADDisableDpd`) is honoured as well.
@@ -512,7 +521,8 @@ the pacing thread continues submitting the last completed layer. A stall alone n
desktop fallback after 250 ms.
Before the first valid image, when OpenXR requests no rendering, or after a session/reference-space
-change invalidates the retained content, frames can still have no layers. Actual runtime or GPU
+change invalidates the retained content, frames can still have no layers (on the Quest outside a
+race, only the passthrough layer while `passthrough` is on, so a recenter does not flash black). Actual runtime or GPU
submission failures retain the safe teardown path. This does not detect black images rendered by
the game itself, and cannot keep submitting if the entire process or XR runtime is suspended.
All OpenXR session and swapchain calls remain on their owning thread.
diff --git a/android/app/src/main/AndroidManifest.xml b/android/app/src/main/AndroidManifest.xml
index 11536b6..a9e4f2c 100644
--- a/android/app/src/main/AndroidManifest.xml
+++ b/android/app/src/main/AndroidManifest.xml
@@ -26,6 +26,9 @@
+
+
diff --git a/android/app/src/main/java/org/wiicompiled/quest/launcher/SettingsPage.kt b/android/app/src/main/java/org/wiicompiled/quest/launcher/SettingsPage.kt
index 523f346..17ce290 100644
--- a/android/app/src/main/java/org/wiicompiled/quest/launcher/SettingsPage.kt
+++ b/android/app/src/main/java/org/wiicompiled/quest/launcher/SettingsPage.kt
@@ -200,6 +200,11 @@ class SettingsPage(
format = { "%.1f m".format(it) },
write = { c, value -> c.setFloat("vr", "hud_width_meters", value) },
)
+ toggle(
+ R.string.vr_passthrough, R.string.vr_passthrough_helper,
+ read = { it.bool("vr", "passthrough") ?: true },
+ write = { c, value -> c.setBool("vr", "passthrough", value) },
+ )
}
}
diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xml
index 9cca959..5b85355 100644
--- a/android/app/src/main/res/values/strings.xml
+++ b/android/app/src/main/res/values/strings.xml
@@ -209,6 +209,8 @@
How far the menu screen and race HUD sit in front of you.
Screen width
How wide the menu screen and race HUD are.
+ Passthrough around the menu screen
+ Shows your room through the headset\'s cameras around the menus instead of black. Races stay fully virtual.
Rendering
diff --git a/docs/quest-port.md b/docs/quest-port.md
index 192afc8..990ceb2 100644
--- a/docs/quest-port.md
+++ b/docs/quest-port.md
@@ -159,6 +159,22 @@ suggested for `oculus/touch_controller` and `khr/simple_controller`.
finding, from device crashes.
- Time conversion for frame interpolation uses `XR_KHR_convert_timespec_time`
(CLOCK_MONOTONIC, the clock behind `steady_clock` on Bionic).
+- **Passthrough around the menus** (`[vr] passthrough`, default on, live):
+ `runtime/src/vr/openxr_passthrough.cpp` owns one `XR_FB_passthrough`
+ reconstruction layer, the PPSSPP VR design. The Vulkan backend starts it
+ (created on first use) or pauses it as each presentation arrives, and submits
+ it first, under the virtual screen's quad, or alone while there is no image
+ yet (startup, a recenter). An immersive race never submits it and pauses the
+ cameras. The quad is cropped to the snapshot Aurora letterboxes into the
+ nearly square eye image (`OpenXRVirtualScreenContentRect`), or its black
+ bands would frame the picture against the room. The manifest's `com.oculus.feature.PASSTHROUGH` is what lets Horizon
+ OS composite it; DolphinXR found that without it every call succeeds and the
+ layer stays empty. The log says `OpenXR passthrough started`, `paused` and
+ `resumed`; when it runs, logcat also shows `Starting camera streams for
+ purpose: passthrough` and `is_displaying_passthrough_content` going to true.
+ ClientMgrFocus logs `[App Enabled for PT: 0]` at every launch, flag or not
+ (it is about the launch transition), so it proves nothing. Checked on a
+ Quest 3 on 2026-09-22: the room shows around the title screen.
### Launcher and game process
diff --git a/runtime/include/runtime_config.h b/runtime/include/runtime_config.h
index 724f260..91ab03d 100644
--- a/runtime/include/runtime_config.h
+++ b/runtime/include/runtime_config.h
@@ -57,6 +57,7 @@ struct RuntimeUserConfig {
std::optional vrHudDistanceMeters;
std::optional vrHudWidthMeters;
std::optional vrHudVirtualScreen;
+ std::optional vrPassthrough;
std::optional vrStopAtDisplayCopy;
std::optional vrSkipCopyClears;
std::optional vrMirrorView;
@@ -743,6 +744,7 @@ inline RuntimeUserConfig ParseConfigDocument(const toml::value& document) {
config.vrHudWidthMeters = *value;
}
config.vrHudVirtualScreen = FindConfigValue(document, "vr", "hud_virtual_screen");
+ config.vrPassthrough = FindConfigValue(document, "vr", "passthrough");
config.vrStopAtDisplayCopy = FindConfigValue(document, "vr", "stop_at_display_copy");
config.vrSkipCopyClears = FindConfigValue(document, "vr", "skip_copy_clears");
config.vrFirstPerson = FindConfigValue(document, "vr", "first_person");
@@ -1056,6 +1058,11 @@ inline bool SetVrHudVirtualScreen(bool value) {
return WriteSetting("vr", "hud_virtual_screen", value ? "true" : "false");
}
+inline bool SetVrPassthrough(bool value) {
+ Mutable().vrPassthrough = value;
+ return WriteSetting("vr", "passthrough", value ? "true" : "false");
+}
+
inline bool SetVrStopAtDisplayCopy(bool value) {
Mutable().vrStopAtDisplayCopy = value;
return WriteSetting("vr", "stop_at_display_copy", value ? "true" : "false");
@@ -1508,6 +1515,13 @@ inline bool VrHudVirtualScreen(bool fallback = true) {
return Get().vrHudVirtualScreen.value_or(fallback);
}
+// The room, through the headset's cameras, around the menu screen and every
+// other virtual screen (never in a race). Only the Quest offers it; the
+// launcher's Settings page shows the same default.
+inline bool VrPassthrough(bool fallback = true) {
+ return Get().vrPassthrough.value_or(fallback);
+}
+
inline bool VrStopAtDisplayCopy(bool fallback = true) {
return Get().vrStopAtDisplayCopy.value_or(fallback);
}
diff --git a/runtime/include/vr/openxr_backend.h b/runtime/include/vr/openxr_backend.h
index ff900d4..6946055 100644
--- a/runtime/include/vr/openxr_backend.h
+++ b/runtime/include/vr/openxr_backend.h
@@ -7,7 +7,9 @@
#include "vr/openxr_runtime.h"
#include "vr/openxr_settings_panel.h"
+#include
#include
+#include
#include
namespace mkw::vr {
@@ -81,12 +83,46 @@ inline XrCompositionLayerQuad OpenXRPanelQuadLayer(const OpenXRPanelLayer& panel
return quad;
}
+// The part of the virtual screen's image that holds anything. Aurora
+// letterboxes the desktop snapshot into that eye-sized image exactly like this
+// (webgpu::calculate_present_viewport_for_aspect) and, when the settings panel
+// is drawn into the eyes, centres it at kSettingsPanelWidthFraction of the
+// width; the rest is black. An unknown content_aspect (0) keeps the whole image.
+inline XrRect2Di OpenXRVirtualScreenContentRect(uint32_t width, uint32_t height, float content_aspect) noexcept {
+ XrRect2Di rect{{0, 0}, {static_cast(width), static_cast(height)}};
+ if (width == 0 || height == 0 || !(content_aspect > 0.0f)) {
+ return rect;
+ }
+ uint32_t content_width = width;
+ uint32_t content_height = std::min(
+ height, std::max(1u, static_cast(std::lround(
+ static_cast(width) * static_cast(1.0f / content_aspect)))));
+ if (content_height == height) {
+ content_width = std::min(
+ width, std::max(1u, static_cast(std::lround(static_cast(height) *
+ static_cast(content_aspect)))));
+ }
+ const uint32_t panel_width = std::min(
+ width, static_cast(std::ceil(static_cast(width) * kSettingsPanelWidthFraction)));
+ const uint32_t panel_height = std::min(
+ height, static_cast(std::ceil(static_cast(panel_width) * kSettingsPanelHeightPixels /
+ kSettingsPanelWidthPixels)));
+ const uint32_t shown_width = std::max(content_width, panel_width);
+ const uint32_t shown_height = std::max(content_height, panel_height);
+ rect.offset = {static_cast((width - shown_width) / 2), static_cast((height - shown_height) / 2)};
+ rect.extent = {static_cast(shown_width), static_cast(shown_height)};
+ return rect;
+}
+
struct OpenXRPresentation {
OpenXRFrameMode mode = OpenXRFrameMode::ImmersiveProjection;
// Used only by VirtualScreen.
float quad_distance_meters = 2.0f;
float quad_width_meters = 2.4f;
+ // The desktop snapshot's width over height, which Aurora letterboxes into
+ // the screen's image (see OpenXRVirtualScreenContentRect); 0 while unknown.
+ float quad_content_aspect = 0.0f;
// When quad_anchored is set, the quad is placed at quad_pose in the
// application reference space and stays put as the player looks around.
@@ -96,6 +132,11 @@ struct OpenXRPresentation {
bool quad_anchored = false;
XrPosef quad_pose{{0.0f, 0.0f, 0.0f, 1.0f}, {0.0f, 0.0f, 0.0f}};
+ // Show the room through the headset's cameras around the virtual screen
+ // (OpenXRPassthrough). Taken when the presentation is handed to the backend,
+ // which starts or pauses the view then; a backend without one ignores it.
+ bool passthrough = false;
+
OpenXRPanelLayer panel;
};
diff --git a/runtime/include/vr/openxr_integration.h b/runtime/include/vr/openxr_integration.h
index d21b752..2e9be00 100644
--- a/runtime/include/vr/openxr_integration.h
+++ b/runtime/include/vr/openxr_integration.h
@@ -53,6 +53,12 @@ void OpenXRRequestRecenter() noexcept;
// once per published frame.
void OpenXRSetLeanBackDegrees(float degrees) noexcept;
+// Shows the room through the headset's cameras around the menu screen and every
+// other virtual screen, never during an immersive race. Only the standalone
+// (Quest) backend offers it; elsewhere this changes nothing. Callable from any
+// thread; applied on the XR pacing thread's next frame.
+void OpenXRSetPassthrough(bool enabled) noexcept;
+
// Live scene interpolation at the headset's own display deadlines.
// 0 = Off, 1 = Auto, otherwise 72/90/120 as a rendering-rate ceiling.
void OpenXRSetFrameInterpolationFps(uint32_t target) noexcept;
diff --git a/runtime/include/vr/openxr_passthrough.h b/runtime/include/vr/openxr_passthrough.h
new file mode 100644
index 0000000..34eb630
--- /dev/null
+++ b/runtime/include/vr/openxr_passthrough.h
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+
+#pragma once
+
+#if defined(MKW_ENABLE_OPENXR)
+
+#include "vr/openxr_runtime.h"
+
+#include
+
+namespace mkw::vr {
+
+// The headset's camera view of the room (XR_FB_passthrough) around the virtual
+// screen, as PPSSPP VR and DolphinXR show it: one reconstruction layer,
+// submitted before every other layer so they all cover it, with the frame's
+// blend mode left OPAQUE. Its objects are created the first time the view is
+// wanted and paused whenever it is not, so a race never keeps the cameras
+// running. On a Quest the app must also declare com.oculus.feature.PASSTHROUGH
+// in its manifest, or the runtime accepts every call and composites nothing.
+//
+// Not synchronized: every call belongs to the thread that ends the session's
+// frames, while the session exists. The handles are the session's children:
+// Destroy() before xrDestroySession, which otherwise frees them itself.
+class OpenXRPassthrough final {
+public:
+ explicit OpenXRPassthrough(OpenXRLogCallback logger = {});
+
+ // Starts (creating it on first use) or pauses the view. A runtime without
+ // XR_FB_passthrough, or one that refuses it, is logged once and leaves the
+ // view off until Destroy().
+ void SetRunning(OpenXRRuntime& runtime, bool running);
+
+ // The layer to submit first while the view runs, otherwise null.
+ const XrCompositionLayerBaseHeader* Layer() const noexcept;
+
+ void Destroy() noexcept;
+
+private:
+ bool Create(OpenXRRuntime& runtime);
+ // gives_up: the failure turns the view off until Destroy().
+ void LogResult(const OpenXRRuntime& runtime, std::string_view operation, XrResult result, bool gives_up) const;
+ void Log(OpenXRLogLevel level, std::string_view message) const noexcept;
+
+ OpenXRLogCallback m_logger;
+ PFN_xrCreatePassthroughFB m_create_passthrough = nullptr;
+ PFN_xrDestroyPassthroughFB m_destroy_passthrough = nullptr;
+ PFN_xrPassthroughStartFB m_start_passthrough = nullptr;
+ PFN_xrPassthroughPauseFB m_pause_passthrough = nullptr;
+ PFN_xrCreatePassthroughLayerFB m_create_layer = nullptr;
+ PFN_xrDestroyPassthroughLayerFB m_destroy_layer = nullptr;
+ PFN_xrPassthroughLayerResumeFB m_resume_layer = nullptr;
+ PFN_xrPassthroughLayerPauseFB m_pause_layer = nullptr;
+ XrPassthroughFB m_passthrough = XR_NULL_HANDLE;
+ XrPassthroughLayerFB m_layer = XR_NULL_HANDLE;
+ XrCompositionLayerPassthroughFB m_composition{XR_TYPE_COMPOSITION_LAYER_PASSTHROUGH_FB};
+ bool m_running = false;
+ bool m_failed = false;
+};
+
+} // namespace mkw::vr
+
+#endif // defined(MKW_ENABLE_OPENXR)
diff --git a/runtime/src/settings_overlay.cpp b/runtime/src/settings_overlay.cpp
index dbd932a..7690e51 100644
--- a/runtime/src/settings_overlay.cpp
+++ b/runtime/src/settings_overlay.cpp
@@ -141,6 +141,9 @@ bool g_vrEnabled = RuntimeConfigFile::VrEnabled(true);
bool g_vrStopAtDisplayCopy = RuntimeConfigFile::VrStopAtDisplayCopy(true);
bool g_vrSkipCopyClears = RuntimeConfigFile::VrSkipCopyClears(true);
bool g_vrHudVirtualScreen = RuntimeConfigFile::VrHudVirtualScreen(true);
+#if defined(__ANDROID__)
+bool g_vrPassthrough = RuntimeConfigFile::VrPassthrough();
+#endif
bool g_vrFirstPerson = RuntimeConfigFile::VrFirstPerson(false);
bool g_vrFirstPersonToggleClick = RuntimeConfigFile::VrFirstPersonToggleClick();
// Set from any thread by the right-thumbstick click, applied on the game thread.
@@ -1478,6 +1481,18 @@ void DrawVrSettings() {
"does pitch the view, and looking sideways while it is set will roll the "
"horizon the way a real recline would.");
}
+#if defined(__ANDROID__)
+ if (ImGui::Checkbox("Passthrough around the menu screen", &g_vrPassthrough)) {
+ mkw::vr::OpenXRSetPassthrough(g_vrPassthrough);
+ RuntimeConfigFile::SetVrPassthrough(g_vrPassthrough);
+ }
+ if (ImGui::IsItemHovered()) {
+ ImGui::SetTooltip(
+ "Shows your room through the headset's cameras around the menu screen and every "
+ "other screen outside a race, instead of black. Races stay fully virtual. "
+ "Applies immediately.");
+ }
+#endif
ImGui::Separator();
ImGui::Text("VR camera");
if (ImGui::Checkbox("First-person camera", &g_vrFirstPerson)) {
diff --git a/runtime/src/vr/openxr_integration.cpp b/runtime/src/vr/openxr_integration.cpp
index 6954112..8c50d3d 100644
--- a/runtime/src/vr/openxr_integration.cpp
+++ b/runtime/src/vr/openxr_integration.cpp
@@ -392,10 +392,13 @@ public:
// Either Vulkan binding extension is acceptable; the backend picks
// whichever the runtime enabled, preferring enable2.
config.required_extensions = {"XR_KHR_android_create_instance"};
+ // XR_FB_passthrough: the room around the virtual screen (OpenXRPassthrough), asked for
+ // whatever [vr] passthrough says, since the setting is live.
config.optional_extensions = {"XR_KHR_vulkan_enable2", "XR_KHR_vulkan_enable",
"XR_KHR_convert_timespec_time",
"XR_KHR_android_thread_settings",
- "XR_FB_display_refresh_rate", "XR_EXT_performance_settings"};
+ "XR_FB_display_refresh_rate", "XR_EXT_performance_settings",
+ "XR_FB_passthrough"};
AddHandMeshExtensions(config);
config.instance_create_next = OpenXRAndroidInstanceCreateNext();
#endif
@@ -561,6 +564,10 @@ public:
return interpolation_available_.load(std::memory_order_acquire);
}
+ void SetPassthrough(bool enabled) noexcept {
+ passthrough_.store(enabled, std::memory_order_relaxed);
+ }
+
void SetLeanBackDegrees(float degrees) noexcept {
lean_back_degrees_.store(
std::clamp(degrees, -RuntimeConfigFile::kVrLeanBackDegreesLimit,
@@ -829,6 +836,13 @@ private:
: OpenXRFrameMode::VirtualScreen;
presentation.quad_distance_meters = policy.config.hud_distance_meters;
presentation.quad_width_meters = policy.config.hud_width_meters;
+ if (float picture_aspect = 0.0f, snapshot_aspect = 0.0f;
+ aurora_get_stereo_screen_aspects(&picture_aspect, &snapshot_aspect)) {
+ presentation.quad_content_aspect = snapshot_aspect;
+ }
+ // The room around the menu screen and every other virtual screen; a race is
+ // fully virtual, and the cameras are paused for it.
+ presentation.passthrough = !immersive && passthrough_.load(std::memory_order_relaxed);
// The settings panel gets a compositor layer of its own while it is
// open, and Aurora leaves it out of the eyes. A backend that could
// not make that layer has the panel drawn into the eyes instead.
@@ -1660,6 +1674,7 @@ private:
std::atomic_bool teardown_requested_{false};
std::atomic_bool recenter_requested_{false};
std::atomic lean_back_degrees_{RuntimeConfigFile::VrLeanBackDegrees()};
+ std::atomic_bool passthrough_{RuntimeConfigFile::VrPassthrough()};
std::atomic_uint32_t frame_interpolation_fps_{RuntimeConfigFile::VrFrameInterpolationFps()};
std::atomic_bool interpolation_available_{false};
std::mutex interpolation_mutex_;
@@ -1767,6 +1782,14 @@ void OpenXRSetLeanBackDegrees(float degrees) noexcept {
#endif
}
+void OpenXRSetPassthrough(bool enabled) noexcept {
+#if MKW_OPENXR_GRAPHICS_BACKEND
+ OpenXRIntegration::Get().SetPassthrough(enabled);
+#else
+ (void)enabled;
+#endif
+}
+
void OpenXRSetFrameInterpolationFps(uint32_t target) noexcept {
#if MKW_OPENXR_GRAPHICS_BACKEND
OpenXRIntegration::Get().SetFrameInterpolationFps(target);
diff --git a/runtime/src/vr/openxr_passthrough.cpp b/runtime/src/vr/openxr_passthrough.cpp
new file mode 100644
index 0000000..31123cd
--- /dev/null
+++ b/runtime/src/vr/openxr_passthrough.cpp
@@ -0,0 +1,154 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+
+#if defined(MKW_ENABLE_OPENXR)
+
+#include "vr/openxr_passthrough.h"
+
+#include
+#include
+#include
+#include
+
+namespace mkw::vr {
+
+OpenXRPassthrough::OpenXRPassthrough(OpenXRLogCallback logger) : m_logger(std::move(logger)) {}
+
+void OpenXRPassthrough::SetRunning(OpenXRRuntime& runtime, bool running) {
+ if (running == m_running || m_failed || !runtime.HasSession()) {
+ return;
+ }
+ if (m_passthrough == XR_NULL_HANDLE) {
+ // Only a start reaches here: the view and its layer are created running.
+ m_failed = !Create(runtime);
+ m_running = !m_failed;
+ return;
+ }
+ if (running) {
+ XrResult result = m_start_passthrough(m_passthrough);
+ runtime.ObserveResult(result);
+ if (XR_SUCCEEDED(result)) {
+ result = m_resume_layer(m_layer);
+ runtime.ObserveResult(result);
+ }
+ if (XR_FAILED(result)) {
+ LogResult(runtime, "resuming the passthrough view", result, true);
+ m_failed = true;
+ return;
+ }
+ m_running = true;
+ Log(OpenXRLogLevel::Info, "OpenXR passthrough resumed");
+ return;
+ }
+ // The layer is no longer submitted from here on, whatever the runtime answers.
+ m_running = false;
+ const XrResult layer_result = m_pause_layer(m_layer);
+ runtime.ObserveResult(layer_result);
+ const XrResult result = m_pause_passthrough(m_passthrough);
+ runtime.ObserveResult(result);
+ if (XR_FAILED(layer_result) || XR_FAILED(result)) {
+ LogResult(runtime, "pausing the passthrough view", XR_FAILED(layer_result) ? layer_result : result,
+ false);
+ return;
+ }
+ Log(OpenXRLogLevel::Info, "OpenXR passthrough paused");
+}
+
+const XrCompositionLayerBaseHeader* OpenXRPassthrough::Layer() const noexcept {
+ return m_running ? reinterpret_cast(&m_composition) : nullptr;
+}
+
+void OpenXRPassthrough::Destroy() noexcept {
+ if (m_layer != XR_NULL_HANDLE) {
+ m_destroy_layer(m_layer);
+ }
+ if (m_passthrough != XR_NULL_HANDLE) {
+ m_destroy_passthrough(m_passthrough);
+ }
+ m_layer = XR_NULL_HANDLE;
+ m_passthrough = XR_NULL_HANDLE;
+ m_running = false;
+ m_failed = false;
+}
+
+bool OpenXRPassthrough::Create(OpenXRRuntime& runtime) {
+ const auto& extensions = runtime.EnabledExtensions();
+ if (std::find(extensions.begin(), extensions.end(), XR_FB_PASSTHROUGH_EXTENSION_NAME) == extensions.end()) {
+ Log(OpenXRLogLevel::Warning,
+ "OpenXR passthrough unavailable: the runtime does not offer " XR_FB_PASSTHROUGH_EXTENSION_NAME);
+ return false;
+ }
+ if (!runtime.LoadFunction("xrCreatePassthroughFB", &m_create_passthrough) ||
+ !runtime.LoadFunction("xrDestroyPassthroughFB", &m_destroy_passthrough) ||
+ !runtime.LoadFunction("xrPassthroughStartFB", &m_start_passthrough) ||
+ !runtime.LoadFunction("xrPassthroughPauseFB", &m_pause_passthrough) ||
+ !runtime.LoadFunction("xrCreatePassthroughLayerFB", &m_create_layer) ||
+ !runtime.LoadFunction("xrDestroyPassthroughLayerFB", &m_destroy_layer) ||
+ !runtime.LoadFunction("xrPassthroughLayerResumeFB", &m_resume_layer) ||
+ !runtime.LoadFunction("xrPassthroughLayerPauseFB", &m_pause_layer)) {
+ Log(OpenXRLogLevel::Warning, "OpenXR passthrough unavailable: " + runtime.LastError().message);
+ return false;
+ }
+
+ XrPassthroughCreateInfoFB create_info{XR_TYPE_PASSTHROUGH_CREATE_INFO_FB};
+ create_info.flags = XR_PASSTHROUGH_IS_RUNNING_AT_CREATION_BIT_FB;
+ XrResult result = m_create_passthrough(runtime.Session(), &create_info, &m_passthrough);
+ runtime.ObserveResult(result);
+ if (XR_FAILED(result)) {
+ m_passthrough = XR_NULL_HANDLE;
+ LogResult(runtime, "xrCreatePassthroughFB", result, true);
+ return false;
+ }
+
+ XrPassthroughLayerCreateInfoFB layer_info{XR_TYPE_PASSTHROUGH_LAYER_CREATE_INFO_FB};
+ layer_info.passthrough = m_passthrough;
+ layer_info.flags = XR_PASSTHROUGH_IS_RUNNING_AT_CREATION_BIT_FB;
+ layer_info.purpose = XR_PASSTHROUGH_LAYER_PURPOSE_RECONSTRUCTION_FB;
+ result = m_create_layer(runtime.Session(), &layer_info, &m_layer);
+ runtime.ObserveResult(result);
+ if (XR_FAILED(result)) {
+ m_layer = XR_NULL_HANDLE;
+ m_destroy_passthrough(m_passthrough);
+ m_passthrough = XR_NULL_HANDLE;
+ LogResult(runtime, "xrCreatePassthroughLayerFB", result, true);
+ return false;
+ }
+
+ // A reconstruction layer has no space of its own; the layers after it cover it.
+ m_composition = {XR_TYPE_COMPOSITION_LAYER_PASSTHROUGH_FB};
+ m_composition.flags = XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT;
+ m_composition.space = XR_NULL_HANDLE;
+ m_composition.layerHandle = m_layer;
+ Log(OpenXRLogLevel::Info, "OpenXR passthrough started");
+ return true;
+}
+
+void OpenXRPassthrough::LogResult(const OpenXRRuntime& runtime, std::string_view operation, XrResult result,
+ bool gives_up) const {
+ char name[XR_MAX_RESULT_STRING_SIZE]{};
+ std::ostringstream message;
+ message << "OpenXR passthrough: " << operation << " failed: ";
+ if (XR_SUCCEEDED(xrResultToString(runtime.Instance(), result, name))) {
+ message << name;
+ } else {
+ message << static_cast(result);
+ }
+ if (gives_up) {
+ message << "; the view stays off for this session";
+ }
+ Log(OpenXRLogLevel::Warning, message.str());
+}
+
+void OpenXRPassthrough::Log(OpenXRLogLevel level, std::string_view message) const noexcept {
+ if (!m_logger) {
+ return;
+ }
+ try {
+ m_logger(level, message);
+ } catch (...) {
+ // A diagnostic callback must never break frame submission.
+ }
+}
+
+} // namespace mkw::vr
+
+#endif // defined(MKW_ENABLE_OPENXR)
diff --git a/runtime/src/vr/openxr_vulkan.cpp b/runtime/src/vr/openxr_vulkan.cpp
index bbb41ba..44adc22 100644
--- a/runtime/src/vr/openxr_vulkan.cpp
+++ b/runtime/src/vr/openxr_vulkan.cpp
@@ -9,6 +9,7 @@
#include "vr/openxr_vulkan.h"
#include "vr/openxr_diagnostics.h"
+#include "vr/openxr_passthrough.h"
#include
@@ -356,6 +357,7 @@ public:
break;
}
}
+ passthrough_.SetRunning(*runtime_, presentation.passthrough);
if (!runtime_->BeginFrame(frame.xr_frame)) {
Fail("xrBeginFrame failed");
return OpenXRBeginStatus::Error;
@@ -532,6 +534,8 @@ public:
if (!runtime_->IsSessionRunning()) {
return OpenXRBeginStatus::SessionNotRunning;
}
+ // Before any frame ends on this presentation, the priming cycle below included.
+ passthrough_.SetRunning(*runtime_, presentation.passthrough);
if (last_display_period_ <= 0) {
// No display timing yet: one compositor cycle learns it.
const OpenXRBeginStatus primed = KeepAliveCycle();
@@ -887,6 +891,11 @@ public:
if (!have_retained_frame_ || !active_frame_.should_render) {
diagnostics::OnEmptyFrame(!active_frame_.should_render ? diagnostics::EmptyFrameReason::ShouldRenderOff
: diagnostics::EmptyFrameReason::NoRetainedLayer);
+ // Outside a race the room stays in view until there is an image to show (at start
+ // and after a recenter), rather than flashing black.
+ if (const XrCompositionLayerBaseHeader* passthrough = passthrough_.Layer()) {
+ return runtime_->EndFrame(active_frame_, &passthrough, 1);
+ }
return runtime_->EndFrameWithoutLayers(active_frame_);
}
diagnostics::OnLayer(fresh);
@@ -896,9 +905,11 @@ public:
quad.layerFlags = 0;
quad.eyeVisibility = XR_EYE_VISIBILITY_BOTH;
quad.subImage.swapchain = retained_swapchains_[0].handle;
- quad.subImage.imageRect = {{0, 0},
- {static_cast(retained_swapchains_[0].width),
- static_cast(retained_swapchains_[0].height)}};
+ // Only the picture, not the black bands letterboxing it into the eye-sized image: they
+ // would frame it against the passthrough view.
+ const uint32_t image_width = retained_swapchains_[0].width;
+ quad.subImage.imageRect = OpenXRVirtualScreenContentRect(
+ image_width, retained_swapchains_[0].height, frame.presentation.quad_content_aspect);
quad.subImage.imageArrayIndex = 0;
if (frame.presentation.quad_anchored) {
quad.space = runtime_->AppSpace();
@@ -909,9 +920,12 @@ public:
quad.pose.position = {
0.0f, 0.0f, -std::max(0.25f, frame.presentation.quad_distance_meters)};
}
- quad.size.width = std::max(0.25f, frame.presentation.quad_width_meters);
- quad.size.height = quad.size.width * static_cast(retained_swapchains_[0].height) /
- static_cast(retained_swapchains_[0].width);
+ // The whole image would be quad_width_meters across: the crop keeps that size per pixel,
+ // so the picture stays exactly where the pointer and the settings panel expect it.
+ const float meters_per_pixel =
+ std::max(0.25f, frame.presentation.quad_width_meters) / static_cast(image_width);
+ quad.size.width = meters_per_pixel * static_cast(quad.subImage.imageRect.extent.width);
+ quad.size.height = meters_per_pixel * static_cast(quad.subImage.imageRect.extent.height);
return EndFrameWithPanel(frame, reinterpret_cast(&quad));
}
std::array views{};
@@ -935,13 +949,20 @@ public:
return EndFrameWithPanel(frame, reinterpret_cast(&projection));
}
- // Ends the compositor frame with the scene's layer and, while the retained
- // frame rendered it, the settings panel's layer over it.
+ // Ends the compositor frame with the scene's layer: over the room's camera
+ // view while that runs and the scene is the virtual screen (never under the
+ // race's projection, which covers the whole view), and, while the retained
+ // frame rendered it, under the settings panel's layer.
bool EndFrameWithPanel(const OpenXRBackendFrame& frame, const XrCompositionLayerBaseHeader* scene) {
const auto& panel = frame.presentation.panel;
XrCompositionLayerQuad panel_quad{};
- const XrCompositionLayerBaseHeader* layers[2] = {scene, nullptr};
- uint32_t count = 1;
+ const XrCompositionLayerBaseHeader* layers[3] = {};
+ uint32_t count = 0;
+ if (const XrCompositionLayerBaseHeader* passthrough = passthrough_.Layer();
+ passthrough != nullptr && frame.presentation.mode == OpenXRFrameMode::VirtualScreen) {
+ layers[count++] = passthrough;
+ }
+ layers[count++] = scene;
if (retained_panel_valid_ && panel.requested && panel.placed) {
panel_quad = OpenXRPanelQuadLayer(panel, runtime_->AppSpace(), retained_panel_swapchain_.handle);
layers[count++] = reinterpret_cast(&panel_quad);
@@ -988,6 +1009,8 @@ public:
}
DestroySwapchains();
DestroySlots();
+ // Its handles belong to the session.
+ passthrough_.Destroy();
if (owns_session_ && runtime_ != nullptr) {
runtime_->DestroySession();
owns_session_ = false;
@@ -1995,6 +2018,8 @@ private:
OpenXRRuntime* runtime_ = nullptr;
OpenXRLogCallback logger_;
+ // The room around the virtual screen, started and paused as each presentation arrives.
+ OpenXRPassthrough passthrough_{logger_};
OpenXRVulkanGraphicsRequirements requirements_{};
std::array eye_swapchains_{};
std::array retained_swapchains_{};
diff --git a/runtime/tests/openxr_d3d12_replay_tests.cpp b/runtime/tests/openxr_d3d12_replay_tests.cpp
index f0a4842..b912edd 100644
--- a/runtime/tests/openxr_d3d12_replay_tests.cpp
+++ b/runtime/tests/openxr_d3d12_replay_tests.cpp
@@ -30,6 +30,9 @@
#include "vr/openxr_d3d12.h"
#endif
+#include "vr/openxr_wii_remote.h"
+
+#include
#include
#include
#include
@@ -581,9 +584,33 @@ void TestPanelLayer() {
display_time = 0;
}
+bool SameRect(const XrRect2Di& rect, int32_t x, int32_t y, int32_t width, int32_t height) {
+ return rect.offset.x == x && rect.offset.y == y && rect.extent.width == width && rect.extent.height == height;
+}
+
+// The menu quad shows only the part of its eye-sized image that Aurora draws into.
+void TestVirtualScreenContentRect() {
+ // A Quest 3 eye at render_scale 0.8 holding the 1280x720 snapshot: the bands above and below
+ // go, and the in-eye settings panel (3/4 of the width, 4:3) is exactly as tall as what is left.
+ const XrRect2Di quest = OpenXRVirtualScreenContentRect(1344, 1408, 16.0f / 9.0f);
+ Require(SameRect(quest, 0, 326, 1344, 756));
+ // At the whole image's size per pixel, the cropped quad is as tall as the snapshot the pointer
+ // maps onto (MenuPictureHalfExtents, 2.4 m across).
+ const float quad_height = 2.4f * static_cast(quest.extent.height) / 1344.0f;
+ const auto picture = wii_remote::MenuPictureHalfExtents(2.4f, 1344.0f / 1408.0f, 16.0f / 9.0f, 16.0f / 9.0f);
+ Require(std::fabs(quad_height - 2.0f * picture[1]) < 1e-4f);
+ // A snapshot narrower than the image is pillarboxed; the panel still has to fit.
+ Require(SameRect(OpenXRVirtualScreenContentRect(1000, 1000, 0.5f), 125, 0, 750, 1000));
+ // A wide snapshot leaves the panel taller than the picture: keep the panel whole.
+ Require(SameRect(OpenXRVirtualScreenContentRect(1344, 1408, 2.4f), 0, 326, 1344, 756));
+ // Before Aurora has published an aspect, the whole image.
+ Require(SameRect(OpenXRVirtualScreenContentRect(1344, 1408, 0.0f), 0, 0, 1344, 1408));
+}
+
int main() {
TestRenderFirst();
TestPanelLayer();
+ TestVirtualScreenContentRect();
OpenXRRuntime runtime;
OpenXRD3D12Backend backend;
Require(backend.QueryGraphicsRequirements(runtime) && backend.BindAurora(runtime));