Drive with bare hands in the cockpit

- With tracked hands on, a hand driving khr/simple_controller with camera-tracked joints is bare:
  latched through wheel_tracking_grace (Meta drops the select action while a hand is lost),
  cleared as soon as a controller's squeeze is back. Its palm joint stands in for the grip and a
  grasp from the middle, ring and little fingers' flexion (0 below 1.2 rad, 1 from 3.0) for the
  squeeze, so closing a hand on the rim takes hold under the wheel's own press and release. The
  grasp never reaches the game's buttons.
- In a cockpit race a bare hand holding the wheel holds the gas (A / South) and a pinch from a free
  bare hand uses an item (Z / L) once the hand has been off the wheel for 0.15 s. Only while the
  game has the remote's pointer off: KPAD publishes the game's pointer switch for the VR remote,
  so the pause menu and the results keep a right pinch as A. The pacing thread logs each change,
  to confirm on the headset that MKW turns the pointer off while driving.
- A held bare hand keeps its last joints drawn through a short loss; bare hands get no haptics.
  The headset panel's readout shows grasp, hold and pinch. Docs: OPENXR.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

OPENXR.md

@@ -275,6 +275,13 @@ see [Steering wheel and hand
 steering](#steering-wheel-and-hand-steering). Turning the wheel moves the controllers, and the game's
 own motion detection still reads them, so a sharp enough turn can read as a shake.
 
+**Bare hands.** On the Quest, with `hand_tracking` on and the controllers put down, the hands drive
+`khr/simple_controller`: a right pinch is A with the pointer on the hand's aim ray, the left
+palm-up pinch is + (pause), and in a cockpit race a hand on the wheel holds A and a free hand's
+pinch is Z; see "Tracked hands" under [Steering wheel and hand
+steering](#steering-wheel-and-hand-steering). A bare hand feeds no motion. With `hand_tracking` off
+a bare hand presses nothing but +.
+
 `"gamepad"` keeps the controllers one ordinary gamepad read through PAD as a GameCube controller:
 A/B → South/East, X/Y → West/North, index triggers → trigger axes, grips → shoulders, thumbsticks
 → sticks (clicks → stick buttons), left menu → Start. Every binding in the F10 controller menu
@@ -528,6 +535,27 @@ no Wii Remote motion, so the permission changes nothing for players who leave th
 PC none of this applies: a PC runtime can drive real controllers through `khr/simple_controller`
 and synthesize joints for them, so a hand-edited `hand_tracking = true` only changes the drawing.
 
+With tracked hands on, bare hands drive. A hand is bare while it drives `khr/simple_controller`
+with camera-tracked joints, latched through `wheel_tracking_grace` (Meta also drops the select
+action while a hand is lost) and cleared as soon as a controller's squeeze is back. Its palm joint
+stands in for the grip and a grasp for the squeeze: the middle, ring and little fingers' flexion,
+summed over each finger's three joints, read as 0 below 1.2 radians (a relaxed hand) and 1 from
+3.0 (a hand closed on a rim), so the wheel's own 55 % press and 15 % release apply, and closing a
+hand on the rim takes hold. The grasp never reaches the game's buttons (as a squeeze it would press
+the gamepad's shoulders). In a cockpit race a bare hand holding the wheel holds the gas (A, the
+gamepad's South) until both hands let go, and a pinch from a free bare hand uses an item (Z, the
+gamepad's L) if the hand has been off the wheel for 0.15 s, so opening a hand off the rim does not
+fire one; holding the pinch holds the button. The left palm-up pinch pauses (a pinch made in that
+gesture is not an item: `XR_FB_hand_tracking_aim` reports the system gesture). Outside a race, and
+in a race whenever the game has the remote's pointer on (the pause menu, the results), a right
+pinch is A and the pointer follows the hand's aim ray; a left pinch does nothing, so the headset
+panel cannot be opened with bare hands (an open one takes a right pinch, and the palm-up pinch
+closes it). The game's pointer switch is read where KPAD polls the VR remote, and the log notes
+each change (`OpenXR bare hands: the game's pointer is off (driving)`); with the gamepad
+presentation there is no pointer, so the race controls apply throughout the cockpit. Manual drift
+has no gesture: choose Automatic drift. The headset panel reads out each hand's source, grasp,
+hold and pinch under the checkbox, for tuning.
+
 **Hands and the separate wheel.** Hands are drawn while hand steering is on: the runtime's own hand
 mesh where it offers one (`XR_EXT_hand_tracking` and `XR_FB_hand_tracking_mesh`, requested when
 hand steering or tracked hands are on at launch), otherwise procedural gloves that curl with the

android/app/src/main/res/values/strings.xml

@@ -534,7 +534,7 @@
     <string name="vr_hand_steering">Hand steering</string>
     <string name="vr_hand_steering_helper">In the cockpit, squeeze a grip near the steering wheel or handlebar to grab it, and turn it to steer. Releasing both grips gives steering back to the stick. Hand steering by heurazy.</string>
     <string name="vr_hand_tracking">Tracked hands</string>
-    <string name="vr_hand_tracking_helper">The cockpit hands follow your own. Holding the controllers, the fingers follow their touch sensors; put the controllers down and the headset\'s cameras track your hands. Needs hand tracking on in the headset\'s settings.</string>
+    <string name="vr_hand_tracking_helper">The cockpit hands follow your own. Holding the controllers, the fingers follow their touch sensors. Put the controllers down to drive with bare hands: close a hand on the wheel to hold it, which also holds the gas; pinch with a free hand to use an item; pinch with your left palm facing you to pause. In menus, a right pinch is A. Choose Automatic drift. Needs hand tracking on in the headset\'s settings.</string>
     <string name="vr_lean_back">Lean back angle</string>
     <string name="vr_lean_back_helper">Tilts the race view back for playing reclined. 0 applies no tilt.</string>
 

runtime/include/vr/openxr_input.h

@@ -237,6 +237,11 @@ private:
     std::array<bool, kHands> m_squeeze_active{};
     std::array<bool, kHands> m_hand_driven{};
     std::array<bool, kHands> m_pinch{};
+    // Bare-hand driving: each hand's bare latch (camera joints, last grasp) and
+    // item pinch gate, and the game's pointer switch as last logged.
+    std::array<hand_tracking::BareLatch, kHands> m_bare_latch{};
+    std::array<hand_tracking::PinchGate, kHands> m_pinch_gate{};
+    int m_logged_game_pointer = -2;
     uint64_t m_profile_serial = 0;
 
     bool m_created = false;

runtime/include/vr/openxr_wii_remote.h

@@ -56,10 +56,18 @@ bool OpenXRReadWiiRemote(OpenXRWiiRemoteSample& sample) noexcept;
 // WPADControlMotor for the emulated remote.
 void OpenXRSetWiiRemoteRumble(bool active) noexcept;
 
+// Guest side: whether the game has the remote's pointer switched on, as KPAD
+// reads it each time the game polls the VR remote (Input::WiiController::
+// TogglePointer flips it). Tracked bare hands use it to tell driving from a
+// menu inside a race; OpenXRInput logs every change.
+void OpenXRPublishGamePointer(bool enabled) noexcept;
+
 // XR side.
 void OpenXRPublishWiiRemote(uint32_t sdl_joystick_id, const OpenXRWiiRemoteSample& sample) noexcept;
 void OpenXRWithdrawWiiRemote() noexcept;
 bool OpenXRWiiRemoteRumbleRequested() noexcept;
+// The game's pointer switch: 1 on, 0 off, -1 not known (no remote polled yet).
+int OpenXRGamePointerState() noexcept;
 
 // The geometry and signal conditioning behind a sample, kept free of OpenXR
 // types so it can be checked headlessly (tests/vr_wii_remote_tests.cpp).

runtime/src/hle/input/kpad.cpp

@@ -1,5 +1,6 @@
 #include "hle_stubs.h"
 #include "memory.h"
+#include "vr/openxr_wii_remote.h"
 #include "wii_remote_input.h"
 
 #include <algorithm>
@@ -276,6 +277,11 @@ extern "C" int32_t KPAD__Read_HLE(uint32_t chan, uint32_t statusPtr, uint32_t co
     WiiRemoteInput::KpadSample sample;
     const bool have = WiiRemoteInput::ReadKpadSample(chan, sample);
     try {
+        if (have && WiiRemoteInput::IsVrControllerChannel(chan)) {
+            // Tracked bare hands drive with the pointer off and point at menus
+            // with it on (openxr_hand_tracking.h).
+            mkw::vr::OpenXRPublishGamePointer(DpdEnabled(chan));
+        }
         return WriteStatus(chan, statusPtr, have ? &sample : nullptr);
     } catch (const Memory::AccessViolation&) {
         return 0;

runtime/src/settings_overlay.cpp

@@ -1192,8 +1192,11 @@ void DrawVrSteeringWheelSettings() {
         ImGui::SetTooltip(
             "%s", mkw::vr::OpenXRHandTrackingAvailable()
                       ? "The cockpit hands follow your own. Holding the controllers, the fingers follow "
-                        "their touch sensors; put the controllers down and the headset's cameras track "
-                        "your hands. Needs hand tracking on in the headset's settings."
+                        "their touch sensors. Put the controllers down to drive with bare hands: close a "
+                        "hand on the wheel to hold it, which also holds the gas; pinch with a free hand "
+                        "to use an item; pinch with your left palm facing you to pause. In menus, a "
+                        "right pinch is A. Choose Automatic drift. Needs hand tracking on in the "
+                        "headset's settings."
                       : "The cockpit hands follow your own, from the controllers' touch sensors or, "
                         "with the controllers put down, the headset's cameras. This session started "
                         "with hand steering and tracked hands off, so it applies after a restart.");

runtime/src/vr/openxr_input.cpp

@@ -744,6 +744,7 @@ void OpenXRInput::Destroy() {
     m_hand_trackers_failed = m_logged_hand_restart = false;
     m_logged_sources = {};
     m_squeeze_active = m_hand_driven = m_pinch = {};
+    m_logged_game_pointer = -2;
     m_profile_serial = 0;
     DestroyPoseSpaces();
     if (m_action_set != XR_NULL_HANDLE) {
@@ -970,6 +971,35 @@ void OpenXRInput::Sync(XrTime predicted_display_time, const OpenXRPointerScreen&
     // steers through the left stick and keeps its grips from the game.
     UpdateDriving(predicted_display_time, seat, hands, panel.withheld);
 
+#if defined(__ANDROID__)
+    // Bare hands in a cockpit race: a hand on the wheel holds the gas and a free
+    // hand's pinch uses an item. Only while the game's pointer is off: the
+    // pause menu and the results keep the cockpit but need A from a pinch.
+    const bool wii_remote = OpenXRGetControllerMode() == OpenXRControllerMode::WiiRemote;
+    const int game_pointer = wii_remote ? OpenXRGamePointerState() : -1;
+    if (hand_tracking_on && game_pointer != m_logged_game_pointer) {
+        m_logged_game_pointer = game_pointer;
+        Log(OpenXRLogLevel::Info, game_pointer > 0    ? "OpenXR bare hands: the game's pointer is on (menu)"
+                                  : game_pointer == 0 ? "OpenXR bare hands: the game's pointer is off (driving)"
+                                                      : "OpenXR bare hands: the game's pointer is unknown");
+    }
+    const bool bare_race = hand_tracking_on && m_driving.cockpit_active && m_driving.hand_steering &&
+                           !panel.withheld && game_pointer <= 0;
+    if (bare_race) {
+        std::array<bool, kHands> bare_held{}, item_pinch{};
+        for (uint32_t hand = 0; hand < kHands; ++hand) {
+            bare_held[hand] = m_bare_latch[hand].Bare() && m_wheel_held[hand];
+            item_pinch[hand] = m_hand_driven[hand] && m_pinch_gate[hand].Update(m_pinch[hand], m_wheel_held[hand],
+                                                                                 dt_seconds);
+        }
+        hand_tracking::ApplyBareHandRace(hands, m_hand_driven, bare_held, item_pinch);
+    } else {
+        for (auto& gate : m_pinch_gate) {
+            gate.Reset();
+        }
+    }
+#endif
+
     // While the panel has the controllers, the game sees them idle.
     static const std::array<wii_remote::HandInputs, kHands> kIdleHands{};
     const auto& game_hands = panel.withheld ? kIdleHands : hands;
@@ -1139,6 +1169,10 @@ void OpenXRInput::ResetDriving() {
     m_wheel_time = 0;
     m_driving = {};
     m_joint_frame.valid = {};
+    for (uint32_t hand = 0; hand < kHands; ++hand) {
+        m_bare_latch[hand].Reset();
+        m_pinch_gate[hand].Reset();
+    }
     OpenXRPublishDriving(m_driving);
 }
 
@@ -1197,6 +1231,9 @@ void OpenXRInput::UpdateDriving(XrTime display_time, const driving::SeatFrame& s
     constexpr XrSpaceLocationFlags kPoseValid =
         XR_SPACE_LOCATION_POSITION_VALID_BIT | XR_SPACE_LOCATION_ORIENTATION_VALID_BIT;
     const bool hand_tracking_on = RuntimeConfigFile::VrHandTracking();
+    // The wheel's own tracking grace (SteeringWheel::Update clamps it the same way).
+    const float grace = driving::IsFinite(tuning.trackingGrace) ? std::clamp(tuning.trackingGrace, 0.05f, 0.5f)
+                                                                : 0.2f;
     std::array<WheelHand, kHands> wheel_hands{};
     for (uint32_t hand = 0; hand < kHands; ++hand) {
         bool tracked = false;
@@ -1214,12 +1251,24 @@ void OpenXRInput::UpdateDriving(XrTime display_time, const driving::SeatFrame& s
         }
         const float squeeze = hands[hand].squeeze;
         const TrackedHand& located = m_tracked_hands[hand];
+        const bool fresh_joints = hand_tracking_on && located.active && located.seated;
+        // A bare hand: camera-tracked joints on a hand driving simple_controller
+        // (Android), latched through the wheel's grace with its last grasp.
+        const bool camera_joints = fresh_joints && located.source != hand_tracking::Source::Controller;
+        const float grasp = camera_joints ? hand_tracking::GraspFromJoints(located.positions) : 0.0f;
+        const bool bare = hand_tracking_on &&
+                          m_bare_latch[hand].Update(m_hand_driven[hand], m_squeeze_active[hand], camera_joints,
+                                                    grasp, dt, grace);
+        if (!hand_tracking_on) {
+            m_bare_latch[hand].Reset();
+        }
         // With tracked hands on, a hand whose joints were located is drawn from
-        // them (the controller's touch sensors, or the cameras). A bare hand is
-        // never drawn or steered from its grip pose, which would show an open
-        // hand wherever it rests.
-        const bool joints = hand_tracking_on && located.active && located.seated;
-        if (m_hand_driven[hand]) {
+        // them (the controller's touch sensors, or the cameras), and a bare hand
+        // holding the wheel keeps its last joints through a short loss. A bare
+        // hand is never drawn or steered from its grip pose, which would show
+        // an open hand wherever it rests.
+        const bool joints = fresh_joints || (bare && m_wheel_held[hand]);
+        if (m_hand_driven[hand] || bare) {
             tracked = false;
         }
         DrivingHand& out = snapshot.hands[hand];
@@ -1229,10 +1278,20 @@ void OpenXRInput::UpdateDriving(XrTime display_time, const driving::SeatFrame& s
         out.squeeze = squeeze;
         out.seat_from_grip = seat_from_grip;
         out.joints_valid = joints;
+        out.bare = bare;
         out.source = located.source;
+        out.grasp = bare ? m_bare_latch[hand].Grasp() : grasp;
         out.pinch = m_pinch[hand];
         m_joint_frame.valid[hand] = joints;
-        wheel_hands[hand] = {seat_from_grip[3], seat_from_grip[7], seat_from_grip[11], squeeze, tracked};
+        if (bare) {
+            // The palm stands in for the grip, and the fingers' grasp for the
+            // squeeze; the grasp never reaches the game's buttons.
+            const auto& palm = m_joint_frame.seat_from_joint[hand][hand_tracking::kPalm];
+            wheel_hands[hand] = {palm[3], palm[7], palm[11], m_bare_latch[hand].Grasp(),
+                                 m_bare_latch[hand].Tracked()};
+        } else {
+            wheel_hands[hand] = {seat_from_grip[3], seat_from_grip[7], seat_from_grip[11], squeeze, tracked};
+        }
         if (uses_geometry) {
             wheel_hands[hand] = geometry.ToWheel(wheel_hands[hand]);
         }

runtime/src/vr/openxr_wii_remote.cpp

@@ -25,6 +25,8 @@ std::atomic<OpenXRControllerMode> g_mode{OpenXRControllerMode::WiiRemote};
 // SDL_JoystickID of the virtual gamepad the samples belong to; 0 when none.
 std::atomic<uint32_t> g_joystick_id{0};
 std::atomic<bool> g_rumble{false};
+// -1 until the game polls the VR remote, then its pointer switch.
+std::atomic<int> g_game_pointer{-1};
 
 } // namespace wii_remote_bridge
 
@@ -73,9 +75,18 @@ void OpenXRPublishWiiRemote(uint32_t sdl_joystick_id, const OpenXRWiiRemoteSampl
     wii_remote_bridge::g_joystick_id.store(sdl_joystick_id, std::memory_order_relaxed);
 }
 
+void OpenXRPublishGamePointer(bool enabled) noexcept {
+    wii_remote_bridge::g_game_pointer.store(enabled ? 1 : 0, std::memory_order_relaxed);
+}
+
+int OpenXRGamePointerState() noexcept {
+    return wii_remote_bridge::g_game_pointer.load(std::memory_order_relaxed);
+}
+
 void OpenXRWithdrawWiiRemote() noexcept {
     wii_remote_bridge::g_joystick_id.store(0, std::memory_order_relaxed);
     wii_remote_bridge::g_rumble.store(false, std::memory_order_relaxed);
+    wii_remote_bridge::g_game_pointer.store(-1, std::memory_order_relaxed);
     auto& published = wii_remote_bridge::Published();
     std::lock_guard lock(published.mutex);
     published.available = false;