daniel-lynch--ovrplugin-openxr-shim/docs/research/render-submit-sync-design.md

Render-submit race — fix design (#2)

Pairs with render-submit-sync-RE.md (subagent RE of UE/OVRPlugin submit timing). Status: design draft; final approach (A vs B) gated on the RE findings.

Confirmed mechanism

• Our ovrp_EndFrame4 → xrEndFrame presents synchronously (composites immediately).
• UE 4.25's RHI thread vkQueueSubmits the eye-render command buffer after ovrp_EndFrame4 returns. Proof: debug.re4vr.qwait (a vkQueueWaitIdle on UE's queue before we release/present) is a no-op — if UE's render were already on the queue, draining it would turn the black image correct; it doesn't, so the submit isn't on the queue yet when end_frame runs.
• The original VrApi path tolerates this because VrApi's EndFrame also defers the present onto the RHI submit (render flush + present ride the same queue submission / RHI flush), so render is naturally ordered before present. We broke that by presenting eagerly inside xrEndFrame.

Why the current sync can't fix it (vk_session.c)

xrr_vk_flush_submit_ex records a COLOR_ATTACHMENT_WRITE→MEMORY_READ barrier and submits it on UE's queue (s_queue), then xrr_vk_flush_wait waits its fence. Vulkan queue execution is in-order, so this is correct iff UE already submitted the eye render. Under load UE hasn't, so the barrier resolves an un-rendered image and the fence signals against empty content → we release+present black. Load-gated exactly as observed (sparse bridge = render makes the deadline = no black; dense geo = late = black).

Prior attempts and why they failed

• Barrier-only (current default): ordered before UE's later submit → black under load.
• qwait (vkQueueWaitIdle): no-op (UE hasn't submitted). Diagnostic only.
• Deferred-flush pipeline (pipeline=1, present N-1): conceptually right (gives UE a full frame to land its submit) but holds the OpenXR swapchain image acquired across xrEndFrame → Meta runtime mis-composites + crashes. Unstable; shelved.

Fix options

A. Observe UE's submit (hook/interpose vkQueueSubmit) — most robust. Wrap vkQueueSubmit so the shim sees exactly when UE flushes the eye render; record that submit's fence (or a timeline value). In end_frame, wait that fence before releasing + presenting. No swapchain-lifecycle hacks, minimal added latency (only the necessary wait). Open question (→ RE): is UE's vkQueueSubmit interceptable from our in-process .so (symbol interposition / a thin Vulkan layer), and which submit carries the eye render?

B. Deferred present via a shim-owned copy (robust fallback, +1 frame latency). Decouple the deferral from the OpenXR swapchain lifecycle (the cause of pipeline=1's instability): keep acquire→release within a single frame, but present one frame late. At frame N+1, UE's render-N submit has landed; copy UE's frame-N eye image into a shim-owned VkImage (barrier-ordered after UE's submit, on the same queue), then present the shim copy via a normal same-frame acquire/release. Never holds an OpenXR image across xrEndFrame. Costs 1 frame of latency + one image copy.

C. Bounded spin-wait for the submit in end_frame — fragile (no clean way to detect the submit without a hook), adds latency/stalls. Not recommended.

D. Use an ovrp call UE makes around submit as the sync point — only viable if the RE finds UE invokes an OVRPlugin entry point right after the render submit. Unlikely; → RE.

RE outcome (render-submit-sync-RE.md) → Option A is feasible

• Original = zero Vulkan sync, definitively: libOVRPlugin imports no vk* (only vrapi_*); libvrapi imports no vk* either. The compositor is a separate system process; swapchains are cross-process system-owned (vrapi_CreateTextureSwapChainCrossProcess). EndFrame4 only hands over swapchain handle + image index + pose; ordering is implicit via system swapchain ownership — the exact analogue of OpenXR acquire/wait/release.
• EndFrame4 runs on the RHI thread: FCustomPresent::FinishRendering_RHIThread → FOculusHMD::FinishRHIFrame_RHIThread → ovrp_EndFrame4 (via PluginWrapper dispatch).
• UE's eye-render submit is interceptable: OculusHMD never calls vkQueueSubmit directly; it goes through the FVulkan RHI's global dispatch pointer VulkanDynamicAPI::vkQueueSubmit (a global PFN resolvable by symbol at runtime), batched/deferred on the RHI thread. A global PFN we can patch → trampoline. This makes Option A viable and portable (pure Vulkan + a UE symbol; no Quest/VrApi dependency, so it carries to Steam Frame).

DECISION: Option A (hook VulkanDynamicAPI::vkQueueSubmit), but resolve the

threading interleave FIRST (instrument before we sync)

Critical open question the RE could not pin from statics: on the RHI thread, does the eye-render vkQueueSubmit happen BEFORE or AFTER the EndFrame4 call?

• If submit-before-EndFrame: by EndFrame the render is on the queue; we just vkWaitForFences on the recorded submit fence before release/present. No deadlock, no added latency. (But the qwait no-op argues against this — nothing was on the queue.)
• If EndFrame-before-submit (what qwait implies, same thread): we must NOT block in EndFrame (that thread does the later submit → self-deadlock). Instead present-on-submit: EndFrame stores the pending composition; the vkQueueSubmit trampoline, on seeing the eye-render submit, triggers the barrier+release+present. This mimics the original (present rides the RHI submit) with no fixed frame of latency and no cross-frame swapchain hold.

The interleave decides wait-in-EndFrame vs present-on-submit, so step 1 is the hook as pure instrumentation (no behavior change): patch the global PFN, log every submit with tid + timestamp + a monotonic seq, and log EndFrame with the same clock. One device run off the bridge confirms the order (and validates the patch offset + that s_queue is the queue UE submits eyes on — the RE's two stated uncertainties). Then implement the matching variant behind a debug.re4vr.* toggle.

Device test throughout: Standing, walk off the bridge toward the house (reliable black trigger), trace=1.