Hand a put-down controller's side to the cameras

- Started with the controllers connected, a race stayed on them: Horizon OS switches all input
  between controllers and hands, and back to the controllers as soon as one lying on a table moves
  (a Quest 3 log went touch_controller, simple_controller, touch_controller within seconds). An
  app cannot disconnect the controllers.
- With tracked hands on, the input now resumes XR_META_simultaneous_hands_and_controllers (Meta's
  multimodal), which overrides that switching: a controller not in a hand no longer owns it, so the
  cameras track that hand at once, and a held controller keeps working. Paused again when the
  option goes off; a refused resume is logged and not retried until the option is toggled.
- A hand is bare when its squeeze action is inactive and it either drives khr/simple_controller or
  has camera-tracked joints, since a free hand under simultaneous tracking may get a profile our
  actions are not bound in. Docs: OPENXR.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

OPENXR.md

@@ -535,8 +535,21 @@ no Wii Remote motion, so the permission changes nothing for players who leave th
 PC none of this applies: a PC runtime can drive real controllers through `khr/simple_controller`
 and synthesize joints for them, so a hand-edited `hand_tracking = true` only changes the drawing.
 
-With tracked hands on, bare hands drive. A hand is bare while it drives `khr/simple_controller`
-with camera-tracked joints, latched through `wheel_tracking_grace` (Meta also drops the select
+Left to itself, Horizon OS switches all input between the controllers and the hands, and it
+switches back to the controllers as soon as one lying on a table moves, so a race started with the
+controllers connected tended to stay on them (a Quest 3 log went controllers, hands, controllers
+within seconds). An app cannot disconnect them. Instead, with tracked hands on, the input resumes
+simultaneous hands and controllers (`XR_META_simultaneous_hands_and_controllers`, Meta's
+"multimodal"), which overrides that switching: a controller that is not in a hand no longer owns
+it, so the cameras track that hand at once, while a held controller keeps working (its fingers
+from its touch sensors). The log says `OpenXR simultaneous hands and controllers on` (and `off`
+when the option goes off). Meta documents that it cannot run together with passthrough and wide
+motion mode both on, and not while body tracking is; this app uses neither of the last two.
+
+With tracked hands on, bare hands drive. A hand is bare while it has no controller in it (its
+squeeze action inactive) and either drives `khr/simple_controller` or has camera-tracked joints (a
+free hand under simultaneous tracking may get no profile our actions are bound in),
+latched through `wheel_tracking_grace` (Meta also drops the select
 action while a hand is lost) and cleared as soon as a controller's squeeze is back. Its palm joint
 stands in for the grip and a grasp for the squeeze: the middle, ring and little fingers' flexion,
 summed over each finger's three joints, read as 0 below 1.2 radians (a relaxed hand) and 1 from

android/app/src/main/res/values/strings.xml

@@ -534,7 +534,7 @@
     <string name="vr_hand_steering">Hand steering</string>
     <string name="vr_hand_steering_helper">In the cockpit, squeeze a grip near the steering wheel or handlebar to grab it, and turn it to steer. Releasing both grips gives steering back to the stick. Hand steering by heurazy.</string>
     <string name="vr_hand_tracking">Tracked hands</string>
-    <string name="vr_hand_tracking_helper">The cockpit hands follow your own. Holding the controllers, the fingers follow their touch sensors. Put the controllers down to drive with bare hands: close a hand on the wheel to hold it, which also holds the gas; pinch with a free hand to use an item; flick your hands up for a trick; pinch with your left palm facing you to pause. In menus, a right pinch is A. Choose Automatic drift. Needs hand tracking on in the headset\'s settings.</string>
+    <string name="vr_hand_tracking_helper">The cockpit hands follow your own. Holding the controllers, the fingers follow their touch sensors; put one down and the cameras track that hand at once. Put the controllers down to drive with bare hands: close a hand on the wheel to hold it, which also holds the gas; pinch with a free hand to use an item; flick your hands up for a trick; pinch with your left palm facing you to pause. In menus, a right pinch is A. Choose Automatic drift. Needs hand tracking on in the headset\'s settings.</string>
     <string name="vr_lean_back">Lean back angle</string>
     <string name="vr_lean_back_helper">Tilts the race view back for playing reclined. 0 applies no tilt.</string>
 

runtime/include/vr/openxr_hand_tracking.h

@@ -241,6 +241,15 @@ inline Gestures GesturesOf(bool aim_valid, bool aim_pinching, bool aim_menu, boo
     return {action_select, action_menu};
 }
 
+// Whether a hand is a bare hand this frame, for its buttons: no controller in
+// it (its squeeze action, which only the Touch profile binds, is inactive), and
+// either the runtime drives khr/simple_controller from it (select active) or
+// the cameras track it. The second covers a hand the runtime gives no profile
+// our actions are bound in, as simultaneous hands and controllers may do.
+inline bool HandDriven(bool squeeze_active, bool select_active, bool camera_joints) noexcept {
+    return !squeeze_active && (select_active || camera_joints);
+}
+
 // The select action as simple_controller delivers it: right select is bound to
 // the primary action, left select to the secondary one (openxr_input.cpp).
 inline bool SelectOf(const wii_remote::HandInputs& inputs, size_t hand) noexcept {

runtime/include/vr/openxr_input.h

@@ -161,6 +161,7 @@ private:
     // settings ask for them, and both hands located for `time`, their joints
     // in `seat` when it is valid.
     void LoadHandTracking();
+    void UpdateSimultaneousHandsAndControllers(bool wanted);
     void UpdateHandTrackers();
     void DestroyHandTrackers();
     void LocateHands(XrTime time, const driving::SeatFrame& seat);
@@ -225,6 +226,12 @@ private:
     PFN_xrLocateHandJointsEXT m_locate_hand_joints = nullptr;
     bool m_hand_data_source = false; // XR_EXT_hand_tracking_data_source
     bool m_hand_aim = false;         // XR_FB_hand_tracking_aim
+    // XR_META_simultaneous_hands_and_controllers: resumed while tracked hands
+    // are on, so a controller put down gives its hand to the cameras at once.
+    PFN_xrResumeSimultaneousHandsAndControllersTrackingMETA m_resume_simultaneous = nullptr;
+    PFN_xrPauseSimultaneousHandsAndControllersTrackingMETA m_pause_simultaneous = nullptr;
+    bool m_simultaneous = false;
+    bool m_simultaneous_failed = false;
     XrHandTrackerEXT m_hand_trackers[kHands]{};
     bool m_hand_trackers_failed = false;
     bool m_logged_hand_restart = false;

runtime/src/settings_overlay.cpp

@@ -1192,7 +1192,8 @@ void DrawVrSteeringWheelSettings() {
         ImGui::SetTooltip(
             "%s", mkw::vr::OpenXRHandTrackingAvailable()
                       ? "The cockpit hands follow your own. Holding the controllers, the fingers follow "
-                        "their touch sensors. Put the controllers down to drive with bare hands: close a "
+                        "their touch sensors; put one down and the cameras track that hand at once. Put "
+                        "the controllers down to drive with bare hands: close a "
                         "hand on the wheel to hold it, which also holds the gas; pinch with a free hand "
                         "to use an item; flick your hands up for a trick; pinch with your left palm "
                         "facing you to pause. In menus, a right pinch is A. Choose Automatic drift. "

runtime/src/vr/openxr_input.cpp

@@ -508,6 +508,57 @@ void OpenXRInput::LoadHandTracking() {
     m_locate_hand_joints = reinterpret_cast<PFN_xrLocateHandJointsEXT>(locate);
     m_hand_data_source = enabled(XR_EXT_HAND_TRACKING_DATA_SOURCE_EXTENSION_NAME);
     m_hand_aim = enabled(XR_FB_HAND_TRACKING_AIM_EXTENSION_NAME);
+    if (enabled(XR_META_SIMULTANEOUS_HANDS_AND_CONTROLLERS_EXTENSION_NAME)) {
+        PFN_xrVoidFunction resume = nullptr, pause = nullptr;
+        if (m_runtime->GetInstanceProcAddress("xrResumeSimultaneousHandsAndControllersTrackingMETA", &resume) &&
+            m_runtime->GetInstanceProcAddress("xrPauseSimultaneousHandsAndControllersTrackingMETA", &pause) &&
+            resume != nullptr && pause != nullptr) {
+            m_resume_simultaneous = reinterpret_cast<PFN_xrResumeSimultaneousHandsAndControllersTrackingMETA>(resume);
+            m_pause_simultaneous = reinterpret_cast<PFN_xrPauseSimultaneousHandsAndControllersTrackingMETA>(pause);
+        }
+    }
+}
+
+// Simultaneous hands and controllers (Meta's "multimodal"): a controller that
+// is not in a hand no longer owns it, so the cameras track that hand straight
+// away and the system stops switching back to the controllers whenever one
+// lying on a table moves. A held controller keeps working, its fingers from its
+// touch sensors. Only while tracked hands are on; off, the system's own
+// switching between hands and controllers applies as before.
+void OpenXRInput::UpdateSimultaneousHandsAndControllers(bool wanted) {
+    if (m_resume_simultaneous == nullptr || m_pause_simultaneous == nullptr) {
+        return;
+    }
+    if (!wanted) {
+        m_simultaneous_failed = false;
+        if (m_simultaneous) {
+            XrSimultaneousHandsAndControllersTrackingPauseInfoMETA info{
+                XR_TYPE_SIMULTANEOUS_HANDS_AND_CONTROLLERS_TRACKING_PAUSE_INFO_META};
+            m_runtime->ObserveResult(m_pause_simultaneous(m_runtime->Session(), &info));
+            m_simultaneous = false;
+            Log(OpenXRLogLevel::Info, "OpenXR simultaneous hands and controllers off");
+        }
+        return;
+    }
+    if (m_simultaneous || m_simultaneous_failed) {
+        return;
+    }
+    XrSimultaneousHandsAndControllersTrackingResumeInfoMETA info{
+        XR_TYPE_SIMULTANEOUS_HANDS_AND_CONTROLLERS_TRACKING_RESUME_INFO_META};
+    const XrResult result = m_resume_simultaneous(m_runtime->Session(), &info);
+    m_runtime->ObserveResult(result);
+    if (XR_FAILED(result)) {
+        // Not asked again until tracked hands are turned off and on.
+        m_simultaneous_failed = true;
+        std::ostringstream message;
+        message << "xrResumeSimultaneousHandsAndControllersTrackingMETA failed (" << result
+                << "); putting a controller down hands over to the cameras only when the system switches";
+        Log(OpenXRLogLevel::Warning, message.str());
+        return;
+    }
+    m_simultaneous = true;
+    Log(OpenXRLogLevel::Info, "OpenXR simultaneous hands and controllers on: a controller put down hands its "
+                              "side to the cameras");
 }
 
 // Trackers only exist while tracked hands and hand steering are both on: the
@@ -515,7 +566,9 @@ void OpenXRInput::LoadHandTracking() {
 // otherwise (Idle and the cockpit's reset keep them); a runtime that refuses
 // them is not asked again until the option is turned off and on.
 void OpenXRInput::UpdateHandTrackers() {
-    if (!RuntimeConfigFile::VrHandTracking() || !RuntimeConfigFile::VrHandSteering()) {
+    const bool wanted = RuntimeConfigFile::VrHandTracking() && RuntimeConfigFile::VrHandSteering();
+    UpdateSimultaneousHandsAndControllers(wanted);
+    if (!wanted) {
         DestroyHandTrackers();
         m_hand_trackers_failed = false;
         m_logged_hand_restart = false;
@@ -747,6 +800,10 @@ void OpenXRInput::Destroy() {
     m_destroy_hand_tracker = nullptr;
     m_locate_hand_joints = nullptr;
     m_hand_data_source = m_hand_aim = false;
+    // The session, and simultaneous tracking with it, goes away after this.
+    m_resume_simultaneous = nullptr;
+    m_pause_simultaneous = nullptr;
+    m_simultaneous = m_simultaneous_failed = false;
     m_hand_trackers_failed = m_logged_hand_restart = false;
     m_logged_sources = {};
     m_sources_logged_at = 0;
@@ -909,18 +966,21 @@ void OpenXRInput::Sync(XrTime predicted_display_time, const OpenXRPointerScreen&
     UpdateHandTrackers();
     LocateHands(predicted_display_time, seat);
 #if defined(__ANDROID__)
-    // A hand driving simple_controller is a bare hand: its select is a pinch
-    // and its menu the palm-up gesture, which pauses from either hand.
+    // A bare hand (no controller in it: it drives simple_controller, or the
+    // cameras track it): its pinch and menu gesture come from the runtime's
+    // recognition, and the menu gesture pauses from either hand.
     bool menu_gesture = false;
     for (uint32_t hand = 0; hand < kHands; ++hand) {
-        m_hand_driven[hand] = !m_squeeze_active[hand] && select_active[hand];
+        const TrackedHand& located = m_tracked_hands[hand];
+        m_hand_driven[hand] = hand_tracking::HandDriven(
+            m_squeeze_active[hand], select_active[hand],
+            hand_tracking_on && located.active && located.source == hand_tracking::Source::Camera);
         if (!m_hand_driven[hand]) {
             m_pinch[hand] = false;
             continue;
         }
-        const TrackedHand& tracked = m_tracked_hands[hand];
         const hand_tracking::Gestures gestures = hand_tracking::GesturesOf(
-            tracked.aim_valid, tracked.aim_pinching, tracked.aim_menu, tracked.aim_system_gesture,
+            located.aim_valid, located.aim_pinching, located.aim_menu, located.aim_system_gesture,
             hand_tracking::SelectOf(hands[hand], hand), hands[hand].menu);
         m_pinch[hand] = gestures.pinch;
         menu_gesture = menu_gesture || gestures.menu;

runtime/src/vr/openxr_integration.cpp

@@ -196,16 +196,19 @@ XrPosef ScreenPoseAhead(const OpenXRFrame& frame, float distance) noexcept {
 // it offers one (XR_FB_hand_tracking_mesh), and tracked hands pose them from
 // the hand trackers: the controllers' touch sensors while they are held
 // (XR_EXT_hand_tracking_data_source) and the cameras once they are put down,
-// with the runtime's own pinch and menu gesture (XR_FB_hand_tracking_aim).
-// Asked for when either is on at launch, since none costs anything until a
-// tracker exists; turning both on later needs a restart for the mesh and the
-// tracked hands (until then the procedural gloves are drawn).
+// with the runtime's own pinch and menu gesture (XR_FB_hand_tracking_aim), and
+// with a put-down controller handing its side to the cameras at once
+// (XR_META_simultaneous_hands_and_controllers). Asked for when either is on at
+// launch, since none costs anything until it is used; turning both on later
+// needs a restart for the mesh and the tracked hands (until then the
+// procedural gloves are drawn).
 void AddHandMeshExtensions(OpenXRConfig& config) {
     if (RuntimeConfigFile::VrHandSteering() || RuntimeConfigFile::VrHandTracking()) {
         config.optional_extensions.push_back("XR_EXT_hand_tracking");
         config.optional_extensions.push_back("XR_FB_hand_tracking_mesh");
         config.optional_extensions.push_back("XR_EXT_hand_tracking_data_source");
         config.optional_extensions.push_back("XR_FB_hand_tracking_aim");
+        config.optional_extensions.push_back("XR_META_simultaneous_hands_and_controllers");
     }
 }
 

runtime/tests/vr_hand_tracking_tests.cpp

@@ -143,6 +143,11 @@ void TestGestures() {
     left.secondary = true;
     right.primary = true;
     Check(SelectOf(left, 0) && SelectOf(right, 1), "simple_controller's select per hand");
+
+    Check(!HandDriven(true, true, true), "a hand holding a controller is not bare");
+    Check(HandDriven(false, true, false), "a hand driving simple_controller is bare");
+    Check(HandDriven(false, false, true), "a camera-tracked hand with no profile of ours is bare");
+    Check(!HandDriven(false, false, false), "a hand with nothing active is not");
 }
 
 std::array<wii_remote::HandInputs, 2> PinchingHands() {