Keep the cockpit wheel with the view and let the native wheel recover

Found on a Quest 3 through Virtual Desktop (VirtualDesktopXR, Vulkan):
the race camera's view matched the scene camera exactly and the kart's
own wheel animated (228 draws a frame took the copy), but during the
race's opening pan no draw took it, the 30-frame fallback latched the VR
wheel for that vehicle, and it stayed until the scene changed.

The fallback now heals: copies keep being published, the VR wheel stands
in only while draws are not taking them, and the vehicle's own wheel
returns as soon as they are. Both switches are logged, a few per race.

With first_person_rotation "yaw_pitch" or "full" the wheel copy and the
hands' wheel geometry now ride in the view's own frame (the kart's
orientation around the seat) instead of the level seat, so the wheel no
longer tilts against the view on slopes.

Aurora logs, after about half a second, five seconds and a minute of
copies, how many draws bound a copied array (inside and outside its
window), how many matched, and the closest position matrix against the
expected one, so a mismatch says whether the array or the matrix differs.

OPENXR.md

@@ -352,15 +352,22 @@ the copy to the GX thread; Aurora substitutes it into the draws that bind that a
 player's own model-view matrix (`aurora_set_native_wheel_vertices`), checking each changed vertex's
 matrix slot, so an opponent sharing the asset and other joints of the same draw are untouched. The
 guest's own vertices are never written, and the copies are dropped after the frame's draws. Bikes
-turn their handle part in the game already; its copy is only re-seated on the level cockpit frame so
-the bars stay with your hands while the bike banks. If no draw takes the copy for 30 frames the log
-says so and the vehicle falls back to a separate VR wheel, which is also what
-`native_steering_wheel = false` draws.
+turn their handle part in the game already; its copy is only re-seated on the cockpit frame so the
+bars stay with your hands while the bike banks. The wheel rides in the same frame as the view: the
+level seat for `"yaw"`, the kart's own orientation for `"yaw_pitch"` and `"full"`. While no draw takes
+the copy (for 30 frames running; the race's opening pan does this) a separate VR wheel stands in,
+which is also what `native_steering_wheel = false` draws. The copy keeps being published, so the
+vehicle's own wheel returns as soon as draws take it again, and the log notes both switches.
 
 The copy is matched against the race camera's view (`RaceCamera::GetViewMtx` with no dolly offset),
 because the scene camera is only set once the draws run. The log reports, once a second, how far
 that view is from the scene camera at the seal (`[mkw-vr] cockpit: race camera view vs scene view`)
 and how many draws took the copy; the F10 bar shows the same under the steering-wheel settings.
+Aurora adds a line after about half a second, five seconds and a minute of copies
+(`Native steering wheel: N sets; draws binding a replaced array ...`) counting the draws that bound a
+copied array, those that bound one outside the window it was set for, the matches, and how far the
+closest position matrix was from the expected one; the first such line with a bound draw also prints
+both matrices.
 
 **Hand steering.** `hand_steering = true` (off by default; also in WheelWizard's OpenXR VR settings)
 lets you take hold of the wheel or handlebar with the tracked controllers. Squeeze a grip near it:

aurora-main/lib/dolphin/gx/GXAurora.cpp

@@ -21,6 +21,10 @@ extern "C" void aurora_clear_native_wheel_vertices() {
   aurora::gx::fifo::drain();
   aurora::gx::nativeWheelLastMatches.store(aurora::gx::nativeWheelMatches);
   aurora::gx::nativeWheelMatches = 0;
+  aurora::gx::nativeWheelPreviousSources.clear();
+  for (const auto& array : aurora::gx::nativeWheelArrays)
+    aurora::gx::nativeWheelPreviousSources.push_back(array.source);
+  aurora::gx::native_wheel_report();
   aurora::gx::nativeWheelArrays.clear();
 }
 extern "C" uint32_t aurora_native_wheel_draw_count() { return aurora::gx::nativeWheelLastMatches.load(); }
@@ -38,6 +42,30 @@ extern "C" void aurora_set_native_wheel_vertices(const void* source, const void*
 // Single definition for the `Log` that gx.hpp declares for this directory.
 aurora::Module Log("aurora::gx");
 
+namespace aurora::gx {
+// Called as a set is cleared: a line at about half a second, five seconds and
+// a minute of sets, with the matrices on the first report that saw a draw.
+void native_wheel_report() {
+    auto& diagnostics=nativeWheelDiagnostics;
+    ++diagnostics.sets;
+    ++nativeWheelClears;
+    if(nativeWheelClears!=30 && nativeWheelClears!=300 && nativeWheelClears!=3600) return;
+    ::Log.info("Native steering wheel: {} sets; draws binding a replaced array {} ({} with a larger range, {} outside a "
+             "set); matched {}; closest position matrix off by {} ({} matrices)",
+             diagnostics.sets,diagnostics.boundDraws,diagnostics.oversizeDraws,diagnostics.outsideDraws,
+             diagnostics.matchedDraws,diagnostics.bestError,diagnostics.bestIndexed?"indexed":"current");
+    if(diagnostics.boundDraws!=0 && nativeWheelReports++==0) {
+        const auto& m=diagnostics.bestMatrix;
+        const auto& e=diagnostics.expected;
+        ::Log.info("Native steering wheel: closest [{} {} {} {} | {} {} {} {} | {} {} {} {}] expected [{} {} {} {} | {} {} "
+                 "{} {} | {} {} {} {}]",
+                 m[0],m[1],m[2],m[3],m[4],m[5],m[6],m[7],m[8],m[9],m[10],m[11],
+                 e[0],e[1],e[2],e[3],e[4],e[5],e[6],e[7],e[8],e[9],e[10],e[11]);
+    }
+    diagnostics={};
+}
+} // namespace aurora::gx
+
 static void GXWriteString(const char* label) {
   auto length = strlen(label);
 

aurora-main/lib/gx/command_processor.cpp

@@ -2422,6 +2422,8 @@ static void handle_draw_unmerged(GXPrimitive prim, GXVtxFmt fmt, u16 vtxCount, g
     }
     auto& array = g_gxState.arrays[i];
     const u32 uploadStride = padded_upload_stride(array.stride);
+    if (i == GX_VA_POS && nativeWheelArrays.empty() && !nativeWheelPreviousSources.empty())
+      UNLIKELY { native_wheel_note_outside(array.data); }
     if (i == GX_VA_POS && !nativeWheelArrays.empty())
       UNLIKELY {
         if (auto* nativeWheel = native_wheel_array(array, vertices, static_cast<u32>(vtxCount) * vtxStride, vtxStride,

aurora-main/lib/gx/native_wheel.hpp

@@ -9,6 +9,7 @@
 // thread; the set/clear entry points are posted there by the runtime.
 #pragma once
 #include "gx.hpp"
+#include <algorithm>
 #include <vector>
 #include <cstring>
 #include <cmath>
@@ -26,6 +27,49 @@ struct NativeWheelArray {
 inline std::vector<NativeWheelArray> nativeWheelArrays;
 inline uint32_t nativeWheelMatches=0;
 inline std::atomic<uint32_t> nativeWheelLastMatches{0};
+
+// For the host log: why draws of the replaced arrays did or did not take them.
+struct NativeWheelDiagnostics {
+    uint32_t sets=0;          // replacement sets cleared since the last report
+    uint32_t boundDraws=0;    // draws that bound a replacement's source array
+    uint32_t oversizeDraws=0; // ... whose bound range was larger than the replacement
+    uint32_t outsideDraws=0;  // draws that bound a cleared set's source while no set was active
+    uint32_t matchedDraws=0;
+    float bestError=INFINITY; // smallest largest-element difference of a position matrix
+    bool bestIndexed=false;
+    std::array<float,12> bestMatrix{};
+    std::array<float,12> expected{};
+};
+inline NativeWheelDiagnostics nativeWheelDiagnostics;
+inline std::vector<const void*> nativeWheelPreviousSources;
+inline uint32_t nativeWheelClears=0;
+inline uint32_t nativeWheelReports=0;
+
+inline void native_wheel_note_outside(const void* source) {
+    for(const void* previous:nativeWheelPreviousSources)
+        if(previous==source) { ++nativeWheelDiagnostics.outsideDraws;return; }
+}
+
+inline void native_wheel_note_bound(const NativeWheelArray& replacement,bool indexedMatrix) {
+    auto& diagnostics=nativeWheelDiagnostics;
+    ++diagnostics.boundDraws;
+    for(uint32_t slot=0;slot<MaxPnMtx;++slot) {
+        if(!indexedMatrix && slot!=g_gxState.currentPnMtx) continue;
+        const auto* matrix=reinterpret_cast<const float*>(&g_gxState.pnMtx[slot].pos);
+        float error=0;
+        for(int i=0;i<12;++i) error=std::max(error,std::abs(matrix[i]-replacement.modelView[i]));
+        if(error<diagnostics.bestError) {
+            diagnostics.bestError=error;
+            diagnostics.bestIndexed=indexedMatrix;
+            std::memcpy(diagnostics.bestMatrix.data(),matrix,sizeof(float)*12);
+            diagnostics.expected=replacement.modelView;
+        }
+    }
+}
+
+// Called as a set is cleared (GXAurora.cpp): one host log line at about half a
+// second, five seconds and a minute of sets.
+void native_wheel_report();
 inline bool native_wheel_source(const void* source) {
     for(const auto& replacement:nativeWheelArrays) if(source==replacement.source) return true;
     return false;
@@ -35,7 +79,9 @@ inline NativeWheelArray* native_wheel_array(const AttrArray& array,const uint8_t
     const bool indexedMatrix=g_gxState.vtxDesc[GX_VA_PNMTXIDX]==GX_DIRECT;
     if(!indexedMatrix && g_gxState.currentPnMtx>=MaxPnMtx) return nullptr;
     for(auto& replacement:nativeWheelArrays) {
-        if(array.data!=replacement.source || array.size>replacement.bytes.size()) continue;
+        if(array.data!=replacement.source) continue;
+        if(array.size>replacement.bytes.size()) { ++nativeWheelDiagnostics.oversizeDraws;continue; }
+        native_wheel_note_bound(replacement,indexedMatrix);
         // A matching asset alone would also animate an opponent. Multi-joint
         // models need a per-position ownership check, not a blanket exclusion.
         uint16_t matching=0;
@@ -56,7 +102,7 @@ inline NativeWheelArray* native_wheel_array(const AttrArray& array,const uint8_t
             {replacement.bytes.data(),array.size},array.stride,
             {vertices,vertexBytes},vertexStride,positionOffset,
             g_gxState.vtxDesc[GX_VA_POS]==GX_INDEX8?1:2,matching)) continue;
-        ++nativeWheelMatches;return &replacement;
+        ++nativeWheelMatches;++nativeWheelDiagnostics.matchedDraws;return &replacement;
     }
     return nullptr;
 }

runtime/src/vr/mkw_vr_first_person.cpp

@@ -166,9 +166,11 @@ constexpr uint32_t kScnMdlWorldMtxArrayOffset = 0xECu;
 // ResVtxPosData: data offset +8, component count +0x14 (1 = XYZ), type +0x18,
 // fraction bits +0x1C, stride +0x1D, count +0x1E, bounds min +0x20, max +0x2C.
 constexpr uint32_t kMdl0Magic = 0x4D444C30u;
-// Frames of published wheel copies that no draw took before falling back to
-// the separate VR wheel for this vehicle.
+// Frames of published wheel copies that no draw took before the separate VR
+// wheel stands in; it steps aside again as soon as a draw takes one.
 constexpr uint32_t kNativeWheelUnmatchedFrames = 30;
+// Fallback switches logged per race.
+constexpr uint32_t kNativeWheelSwitchLogs = 8;
 
 // Frames the last good anchor survives a failed read before the camera returns
 // to the game's own. Rides out a transient null during a respawn or transition
@@ -328,6 +330,10 @@ struct FirstPersonState {
         uint32_t body = 0;
         // The level seat frame: simulation position and driving direction.
         Mtx34 stable_body = kIdentityMtx34;
+        // The frame the seat rides in, and the wheel with it: the level frame
+        // for "yaw", otherwise the kart's own orientation around the same seat
+        // position, so the wheel always stays where the view puts it.
+        Mtx34 seat_body = kIdentityMtx34;
         std::array<float, 3> player_scale{1.0f, 1.0f, 1.0f};
         Mtx34 body_pose = kIdentityMtx34;
         bool grips_valid = false;
@@ -352,7 +358,9 @@ struct FirstPersonState {
     bool wheel_arrays_posted = false;
     uint32_t native_wheel_body = 0;
     uint32_t native_wheel_unmatched = 0;
-    bool native_wheel_disabled = false;
+    // The VR wheel is standing in: recent copies have not been taken.
+    bool native_wheel_fallback = false;
+    uint32_t native_wheel_switch_logs = 0;
 };
 
 std::mutex g_mutex;
@@ -874,6 +882,13 @@ void LatchCockpitLocked(uint64_t guest_frame_index) noexcept {
                          : 1.0f / 60.0f;
     g_state.stabilized_frame = guest_frame_index;
     latch.stable_body = g_state.stabilizer.Update(simulation, damage_type != UINT32_MAX, dt);
+    latch.seat_body = latch.stable_body;
+    if (g_state.rotation != FirstPersonRotation::YawOnly) {
+        latch.seat_body = pose;
+        latch.seat_body[3] = latch.stable_body[3];
+        latch.seat_body[7] = latch.stable_body[7];
+        latch.seat_body[11] = latch.stable_body[11];
+    }
     latch.player_scale = ReadPlayerScale(kart.accessor);
     latch.body = kart.body;
     latch.grips_valid = ReadGuestMtx34(kart.body + kKartBodyLeftGripOffset, latch.left_grip) &&
@@ -898,10 +913,10 @@ void LatchCockpitLocked(uint64_t guest_frame_index) noexcept {
     if (g_state.native_wheel_body != kart.body) {
         g_state.native_wheel_body = kart.body;
         g_state.native_wheel_unmatched = 0;
-        g_state.native_wheel_disabled = false;
+        g_state.native_wheel_fallback = false;
     }
     if (!latch.grips_valid || !latch.predicted_view_valid || !g_state.steering_wheel ||
-        !g_state.native_steering_wheel || g_state.native_wheel_disabled) {
+        !g_state.native_steering_wheel) {
         return;
     }
     const DrivingSnapshot driving = OpenXRReadDriving();
@@ -918,7 +933,7 @@ void LatchCockpitLocked(uint64_t guest_frame_index) noexcept {
             return;
         }
         const auto stable_handle = ScaleModelBasis(
-            ComposeMtx(ComposeMtx(latch.stable_body, inverse_body), latch.handle_pose), latch.player_scale);
+            ComposeMtx(ComposeMtx(latch.seat_body, inverse_body), latch.handle_pose), latch.player_scale);
         const auto rendered_handle = ScaleModelBasis(latch.handle_pose, latch.player_scale);
         if (!InvertMtx(rendered_handle, inverse_handle)) {
             return;
@@ -938,7 +953,7 @@ void LatchCockpitLocked(uint64_t guest_frame_index) noexcept {
         latch.mesh_published = PublishNativeWheelMesh(
             kart.body, ComposeMtx(latch.predicted_view, rendered_body), latch.left_grip, latch.right_grip,
             driving.visual_angle, false,
-            ComposeMtx(inverse_rendered, ScaleModelBasis(latch.stable_body, latch.player_scale)));
+            ComposeMtx(inverse_rendered, ScaleModelBasis(latch.seat_body, latch.player_scale)));
     }
 }
 
@@ -1001,20 +1016,14 @@ bool ComputeCockpitAnchorLocked(const Mtx34& view_from_world, const KartPoseRead
     }
     render_units *= scale[1];
 
-    // "yaw" takes the kart's own driving direction, level, rather than the
-    // chase camera's lagging heading; the other modes keep the kart's
-    // orientation around the same seat.
-    Mtx34 kart_frame = latch.stable_body;
-    FirstPersonRotation rotation = FirstPersonRotation::YawPitch;
-    if (g_state.rotation != FirstPersonRotation::YawOnly) {
-        kart_frame = pose;
-        kart_frame[3] = latch.stable_body[3];
-        kart_frame[7] = latch.stable_body[7];
-        kart_frame[11] = latch.stable_body[11];
-        rotation = g_state.rotation;
-    }
+    // "yaw" takes the kart's own driving direction from the level seat frame,
+    // rather than the chase camera's lagging heading; the other modes take the
+    // kart's orientation around the same seat (LatchCockpitLocked).
+    (void)pose;
+    const FirstPersonRotation rotation =
+        g_state.rotation == FirstPersonRotation::YawOnly ? FirstPersonRotation::YawPitch : g_state.rotation;
     Mtx34 anchor{};
-    if (!ComputeFirstPersonAnchor(view_from_world, kart_frame, eye[0], eye[1], eye[2], rotation, anchor)) {
+    if (!ComputeFirstPersonAnchor(view_from_world, latch.seat_body, eye[0], eye[1], eye[2], rotation, anchor)) {
         failed_step = "cockpit anchor math (degenerate camera or kart frame)";
         return false;
     }
@@ -1029,14 +1038,14 @@ bool ComputeCockpitAnchorLocked(const Mtx34& view_from_world, const KartPoseRead
         const detail::Vec3 left{latch.left_grip[3], latch.left_grip[7], latch.left_grip[11]};
         const detail::Vec3 right{latch.right_grip[3], latch.right_grip[7], latch.right_grip[11]};
         const auto seat_from_world = ComposeMtx(anchor, view_from_world);
-        const auto seat_from_body = ComposeMtx(seat_from_world, ScaleModelBasis(latch.stable_body, scale));
+        const auto seat_from_body = ComposeMtx(seat_from_world, ScaleModelBasis(latch.seat_body, scale));
         if (!latch.bike) {
             out.native_wheel = ComputeNativeWheelGeometry(seat_from_body, left, right, render_units);
         } else if (latch.handle_valid) {
             Mtx34 inverse_body{};
             if (InvertMtx(latch.body_pose, inverse_body)) {
                 const auto stable_handle = ScaleModelBasis(
-                    ComposeMtx(ComposeMtx(latch.stable_body, inverse_body), latch.handle_pose), scale);
+                    ComposeMtx(ComposeMtx(latch.seat_body, inverse_body), latch.handle_pose), scale);
                 out.native_wheel = ComputeNativeHandlebarGeometry(ComposeMtx(seat_from_world, stable_handle),
                                                                   seat_from_body, left, right, render_units);
             }
@@ -1045,31 +1054,43 @@ bool ComputeCockpitAnchorLocked(const Mtx34& view_from_world, const KartPoseRead
     // Waiting counts as prepared, so the separate VR wheel does not flash up for
     // the frame or two the XR side takes to engage.
     out.native_mesh_prepared =
-        (latch.mesh_published || latch.waiting_for_driving) && !g_state.native_wheel_disabled;
+        (latch.mesh_published || latch.waiting_for_driving) && !g_state.native_wheel_fallback;
     return true;
 }
 
-// After the frame's draws: drop the wheel copies, and give up on the vehicle's
-// own wheel if no draw has been taking them.
+// After the frame's draws: drop the wheel copies, and let the VR wheel stand
+// in while no draw takes them (the race's opening pan, for one). Copies keep
+// being published, so the vehicle's own wheel returns as soon as they match.
 void FinishNativeWheelFrameLocked() noexcept {
     if (!g_state.wheel_arrays_posted) {
         return;
     }
     DropNativeWheelLocked();
-    if (!g_state.cockpit.mesh_published || g_state.native_wheel_disabled) {
+    if (!g_state.cockpit.mesh_published) {
         return;
     }
+    const bool log = g_state.native_wheel_switch_logs < kNativeWheelSwitchLogs;
     // Lags a frame or two with the GX thread on; the threshold allows for it.
     if (GxNativeWheel::LastDrawCount() > 0) {
+        if (g_state.native_wheel_fallback && log) {
+            ++g_state.native_wheel_switch_logs;
+            RT_LOG(RT_TAG_RUNTIME) << "[mkw-vr] native steering wheel: draws take the animated copy of vehicle 0x"
+                                   << std::hex << g_state.cockpit.body << std::dec
+                                   << " again; back to the vehicle's own wheel" << std::endl;
+        }
+        g_state.native_wheel_fallback = false;
         g_state.native_wheel_unmatched = 0;
         return;
     }
-    if (++g_state.native_wheel_unmatched >= kNativeWheelUnmatchedFrames) {
-        g_state.native_wheel_disabled = true;
-        RT_LOG(RT_TAG_RUNTIME) << "[mkw-vr] native steering wheel: no draw took the animated copy of vehicle 0x"
-                               << std::hex << g_state.cockpit.body << std::dec << " in "
-                               << kNativeWheelUnmatchedFrames
-                               << " frames; drawing the VR steering wheel instead" << std::endl;
+    if (!g_state.native_wheel_fallback && ++g_state.native_wheel_unmatched >= kNativeWheelUnmatchedFrames) {
+        g_state.native_wheel_fallback = true;
+        if (log) {
+            ++g_state.native_wheel_switch_logs;
+            RT_LOG(RT_TAG_RUNTIME) << "[mkw-vr] native steering wheel: no draw took the animated copy of vehicle 0x"
+                                   << std::hex << g_state.cockpit.body << std::dec << " in "
+                                   << kNativeWheelUnmatchedFrames
+                                   << " frames; drawing the VR steering wheel until one does" << std::endl;
+        }
     }
 }
 
@@ -1212,7 +1233,7 @@ void MkwVRFirstPersonApplyConfiguredSettings() noexcept {
         g_state.cockpit_units_per_meter = cockpit_units;
         g_state.steering_wheel = RuntimeConfigFile::VrSteeringWheel();
         g_state.native_steering_wheel = RuntimeConfigFile::VrNativeSteeringWheel();
-        g_state.native_wheel_disabled = false;
+        g_state.native_wheel_fallback = false;
         g_state.native_wheel_unmatched = 0;
     }
     // The cockpit publishes its exact per-frame scale with each anchor; this
@@ -1246,7 +1267,8 @@ void MkwVRFirstPersonReset() noexcept {
     g_state.cockpit_forward.reset();
     g_state.native_wheel_body = 0;
     g_state.native_wheel_unmatched = 0;
-    g_state.native_wheel_disabled = false;
+    g_state.native_wheel_fallback = false;
+    g_state.native_wheel_switch_logs = 0;
 }
 
 void MkwVRFirstPersonUpdate(uint64_t guest_frame_index, uint32_t race_camera_address) noexcept {