daniel-lynch--ovrplugin-openxr-shim/docs/research/lever2-detect-skip-black-handoff.md

Lever 2 handoff — detect dropped/black frames and reproject instead of presenting black

Entry point for implementing the elegant fix to the in-game black. Read alongside the auto-memory: black-is-not-submit-timing-ue-renders-empty, ingame-black-render-content, gameplay-eye-image-is-pure-black-confirmed, standing-vs-sitting-blackflash.

The settled root cause (do not re-litigate)

The in-game black is a UE-internal frame-drop under GPU load, NOT a shim bug:

• RenderDoc (~/renderdoc-captures/RE4/work_frame.rdc): under load UE renders only ~28 draws into its eye target vs ~198 in a normal frame (a truncated frame), and the resolved eye = pure black (MEAN [0,0,0], reliable ms=1 read).
• Ruled out conclusively: submit-timing (present-on-submit incl. full one-frame defer still blacks — debug.re4vr.submithook 2/defern 99), image targeting (LAYER MISMATCH count = 0, stage==acquiredIndex), empty-frame submission (SUBMIT-BLACK/COMPOSE-EMPTY = 0).
• Load-gated: bridge (sparse) never blacks; house/dense geometry blacks; "especially while casting" (extra load). The game does NOT self-scale from our GPU-time feed.

The idea

When UE hands us a truncated/black frame, don't present it — re-present the last good eye image with the current frame's pose, so the OpenXR compositor timewarps/ reprojects the last good content to the new head pose. A reprojected (slightly stale) frame is far better than a black flash. This is exactly the "app didn't produce a new frame" path that compositors are built for; our black frames currently defeat it.

Two hard parts

A. A cheap "this frame is bad" signal (the crux)

Per-frame GPU readback to detect black is too costly + unreliable (observer effect; see texture-dump-is-unreliable-probe). Candidate signals, cheapest first:

1. Post-hitch heuristic: the truncated frame is the RECOVERY frame after a stall (RenderDoc: "recovery frame was 26 draws"). We already detect HITCH (dt>20ms in the FRAME trace). Try: skip+repeat the 1–2 frames following a detected hitch. Coarse but zero new cost; test first.
2. Submit/command-buffer count via the vkQueueSubmit hook (already built, debug.re4vr.submithook 1): a truncated frame issues fewer submits / command buffers. Instrument submits-per-frame (between END markers) and correlate with perceived black. Under load we saw ~6 submits/frame normal — a dropped frame may show fewer. Needs a black ground-truth to calibrate (hard without readback; use the post-hitch frames as a proxy, or a one-off RenderDoc cross-check).
3. GPU frame time over budget: g_gpuFrameMs already tracked; but it's the PREVIOUS frame's dt (lagging), so use it to predict the next frame is at-risk, not to gate the current one. Recommendation: start with (1) post-hitch repeat — simplest, no new signal — and measure. If it helps but is too coarse, add (2) via the hook.

B. Re-presenting the last good frame

OpenXR requires xrEndFrame every frame after xrBeginFrame; you can't simply skip present. To reproject, present the LAST GOOD eye image again with the current predictedDisplayTime

• located views (the runtime timewarps it). Mechanics:

• Keep a reference to the last-good eye image. Two options: (a) DON'T release frame N-1's swapchain image and re-submit it (risky — holding across frames is what made pipeline=1 unstable; see deferred-flush-unstable-abandoned), or (b) copy the last good eye image into a shim-owned VkImage (the copy-ring infra already exists: xrr_vk_alloc_images, shimImages[], the copy path in end_frame) and present a normal fresh swapchain image blitted from the held copy. (b) avoids the cross-frame-hold instability.
• On a bad frame: skip UE's (black) image, blit last-good copy → the acquired swapchain image, submit the previous composition's layer with the CURRENT pose/displayTime.
• Gate the whole thing behind a new debug.re4vr.skipblack prop (default 0), like the other levers, so the known-good path is untouched.

Key code locations (shim/src/xr_runtime.c unless noted)

• xrr_end_frame sync branch (~1053): where present happens; add the skip+repeat here.
• FRAME/HITCH trace (~1123): the hitch signal (dt>20ms) for heuristic (1).
• build_composition (~620): the composition we'd re-submit with updated pose.
• Copy-ring infra: xrr_vk_alloc_images / shimImages[] (vk_session.c) + the copy-ring path in end_frame (~922) — reuse for holding/blitting the last-good image.
• vkQueueSubmit hook + xrr_on_ue_submit (~820): submits-per-frame instrumentation for signal (2). debug.re4vr.submithook 1 = instrument.
• Pose update for reprojection: xrr_eye_fov_tangents + the located views in g_xr.views.

Validation

Device: Quest 2 <redacted-serial> (USB). Build/deploy: ./shim/build_android.sh && ./packaging/repack.sh && adb -s <redacted-serial> install -r packaging/out/re4vr-shim.apk (NOTE: repack silently bundles the LAST successful build — always confirm the build had no errors and check shim/build/arm64/libOVRPlugin.so timestamp before repack.) Test: Standing, walk off the bridge toward the house (reliable black trigger), trace=1. Success = black flashes replaced by (at worst) brief reprojection judder, not black.

Prerequisite vs Lever 1

If Lever 1 (force lower GPU load so UE completes frames: ffr=3 + debug.re4vr.resscale < 100 + sscap=1) sufficiently stops the drops at acceptable quality, Lever 2 may be unnecessary or only needed for the worst spikes. Decide after the Lever 1 result.

debug.re4vr.resscale = percent of eye size (default 100; e.g. 75, 50). New this session.