diff --git a/OPENXR.md b/OPENXR.md
index af6a022..609ef06 100644
--- a/OPENXR.md
+++ b/OPENXR.md
@@ -535,8 +535,21 @@ no Wii Remote motion, so the permission changes nothing for players who leave th
PC none of this applies: a PC runtime can drive real controllers through `khr/simple_controller`
and synthesize joints for them, so a hand-edited `hand_tracking = true` only changes the drawing.
-With tracked hands on, bare hands drive. A hand is bare while it drives `khr/simple_controller`
-with camera-tracked joints, latched through `wheel_tracking_grace` (Meta also drops the select
+Left to itself, Horizon OS switches all input between the controllers and the hands, and it
+switches back to the controllers as soon as one lying on a table moves, so a race started with the
+controllers connected tended to stay on them (a Quest 3 log went controllers, hands, controllers
+within seconds). An app cannot disconnect them. Instead, with tracked hands on, the input resumes
+simultaneous hands and controllers (`XR_META_simultaneous_hands_and_controllers`, Meta's
+"multimodal"), which overrides that switching: a controller that is not in a hand no longer owns
+it, so the cameras track that hand at once, while a held controller keeps working (its fingers
+from its touch sensors). The log says `OpenXR simultaneous hands and controllers on` (and `off`
+when the option goes off). Meta documents that it cannot run together with passthrough and wide
+motion mode both on, and not while body tracking is; this app uses neither of the last two.
+
+With tracked hands on, bare hands drive. A hand is bare while it has no controller in it (its
+squeeze action inactive) and either drives `khr/simple_controller` or has camera-tracked joints (a
+free hand under simultaneous tracking may get no profile our actions are bound in),
+latched through `wheel_tracking_grace` (Meta also drops the select
action while a hand is lost) and cleared as soon as a controller's squeeze is back. Its palm joint
stands in for the grip and a grasp for the squeeze: the middle, ring and little fingers' flexion,
summed over each finger's three joints, read as 0 below 1.2 radians (a relaxed hand) and 1 from
diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xml
index ee47aa8..8c4c07a 100644
--- a/android/app/src/main/res/values/strings.xml
+++ b/android/app/src/main/res/values/strings.xml
@@ -534,7 +534,7 @@
Hand steering
In the cockpit, squeeze a grip near the steering wheel or handlebar to grab it, and turn it to steer. Releasing both grips gives steering back to the stick. Hand steering by heurazy.
Tracked hands
- The cockpit hands follow your own. Holding the controllers, the fingers follow their touch sensors. Put the controllers down to drive with bare hands: close a hand on the wheel to hold it, which also holds the gas; pinch with a free hand to use an item; flick your hands up for a trick; pinch with your left palm facing you to pause. In menus, a right pinch is A. Choose Automatic drift. Needs hand tracking on in the headset\'s settings.
+ The cockpit hands follow your own. Holding the controllers, the fingers follow their touch sensors; put one down and the cameras track that hand at once. Put the controllers down to drive with bare hands: close a hand on the wheel to hold it, which also holds the gas; pinch with a free hand to use an item; flick your hands up for a trick; pinch with your left palm facing you to pause. In menus, a right pinch is A. Choose Automatic drift. Needs hand tracking on in the headset\'s settings.
Lean back angle
Tilts the race view back for playing reclined. 0 applies no tilt.
diff --git a/runtime/include/vr/openxr_hand_tracking.h b/runtime/include/vr/openxr_hand_tracking.h
index 573501d..a1813c6 100644
--- a/runtime/include/vr/openxr_hand_tracking.h
+++ b/runtime/include/vr/openxr_hand_tracking.h
@@ -241,6 +241,15 @@ inline Gestures GesturesOf(bool aim_valid, bool aim_pinching, bool aim_menu, boo
return {action_select, action_menu};
}
+// Whether a hand is a bare hand this frame, for its buttons: no controller in
+// it (its squeeze action, which only the Touch profile binds, is inactive), and
+// either the runtime drives khr/simple_controller from it (select active) or
+// the cameras track it. The second covers a hand the runtime gives no profile
+// our actions are bound in, as simultaneous hands and controllers may do.
+inline bool HandDriven(bool squeeze_active, bool select_active, bool camera_joints) noexcept {
+ return !squeeze_active && (select_active || camera_joints);
+}
+
// The select action as simple_controller delivers it: right select is bound to
// the primary action, left select to the secondary one (openxr_input.cpp).
inline bool SelectOf(const wii_remote::HandInputs& inputs, size_t hand) noexcept {
diff --git a/runtime/include/vr/openxr_input.h b/runtime/include/vr/openxr_input.h
index ebb757d..c29c076 100644
--- a/runtime/include/vr/openxr_input.h
+++ b/runtime/include/vr/openxr_input.h
@@ -161,6 +161,7 @@ private:
// settings ask for them, and both hands located for `time`, their joints
// in `seat` when it is valid.
void LoadHandTracking();
+ void UpdateSimultaneousHandsAndControllers(bool wanted);
void UpdateHandTrackers();
void DestroyHandTrackers();
void LocateHands(XrTime time, const driving::SeatFrame& seat);
@@ -225,6 +226,12 @@ private:
PFN_xrLocateHandJointsEXT m_locate_hand_joints = nullptr;
bool m_hand_data_source = false; // XR_EXT_hand_tracking_data_source
bool m_hand_aim = false; // XR_FB_hand_tracking_aim
+ // XR_META_simultaneous_hands_and_controllers: resumed while tracked hands
+ // are on, so a controller put down gives its hand to the cameras at once.
+ PFN_xrResumeSimultaneousHandsAndControllersTrackingMETA m_resume_simultaneous = nullptr;
+ PFN_xrPauseSimultaneousHandsAndControllersTrackingMETA m_pause_simultaneous = nullptr;
+ bool m_simultaneous = false;
+ bool m_simultaneous_failed = false;
XrHandTrackerEXT m_hand_trackers[kHands]{};
bool m_hand_trackers_failed = false;
bool m_logged_hand_restart = false;
diff --git a/runtime/src/settings_overlay.cpp b/runtime/src/settings_overlay.cpp
index 200e532..3e26c24 100644
--- a/runtime/src/settings_overlay.cpp
+++ b/runtime/src/settings_overlay.cpp
@@ -1192,7 +1192,8 @@ void DrawVrSteeringWheelSettings() {
ImGui::SetTooltip(
"%s", mkw::vr::OpenXRHandTrackingAvailable()
? "The cockpit hands follow your own. Holding the controllers, the fingers follow "
- "their touch sensors. Put the controllers down to drive with bare hands: close a "
+ "their touch sensors; put one down and the cameras track that hand at once. Put "
+ "the controllers down to drive with bare hands: close a "
"hand on the wheel to hold it, which also holds the gas; pinch with a free hand "
"to use an item; flick your hands up for a trick; pinch with your left palm "
"facing you to pause. In menus, a right pinch is A. Choose Automatic drift. "
diff --git a/runtime/src/vr/openxr_input.cpp b/runtime/src/vr/openxr_input.cpp
index bb467f9..a1213c3 100644
--- a/runtime/src/vr/openxr_input.cpp
+++ b/runtime/src/vr/openxr_input.cpp
@@ -508,6 +508,57 @@ void OpenXRInput::LoadHandTracking() {
m_locate_hand_joints = reinterpret_cast(locate);
m_hand_data_source = enabled(XR_EXT_HAND_TRACKING_DATA_SOURCE_EXTENSION_NAME);
m_hand_aim = enabled(XR_FB_HAND_TRACKING_AIM_EXTENSION_NAME);
+ if (enabled(XR_META_SIMULTANEOUS_HANDS_AND_CONTROLLERS_EXTENSION_NAME)) {
+ PFN_xrVoidFunction resume = nullptr, pause = nullptr;
+ if (m_runtime->GetInstanceProcAddress("xrResumeSimultaneousHandsAndControllersTrackingMETA", &resume) &&
+ m_runtime->GetInstanceProcAddress("xrPauseSimultaneousHandsAndControllersTrackingMETA", &pause) &&
+ resume != nullptr && pause != nullptr) {
+ m_resume_simultaneous = reinterpret_cast(resume);
+ m_pause_simultaneous = reinterpret_cast(pause);
+ }
+ }
+}
+
+// Simultaneous hands and controllers (Meta's "multimodal"): a controller that
+// is not in a hand no longer owns it, so the cameras track that hand straight
+// away and the system stops switching back to the controllers whenever one
+// lying on a table moves. A held controller keeps working, its fingers from its
+// touch sensors. Only while tracked hands are on; off, the system's own
+// switching between hands and controllers applies as before.
+void OpenXRInput::UpdateSimultaneousHandsAndControllers(bool wanted) {
+ if (m_resume_simultaneous == nullptr || m_pause_simultaneous == nullptr) {
+ return;
+ }
+ if (!wanted) {
+ m_simultaneous_failed = false;
+ if (m_simultaneous) {
+ XrSimultaneousHandsAndControllersTrackingPauseInfoMETA info{
+ XR_TYPE_SIMULTANEOUS_HANDS_AND_CONTROLLERS_TRACKING_PAUSE_INFO_META};
+ m_runtime->ObserveResult(m_pause_simultaneous(m_runtime->Session(), &info));
+ m_simultaneous = false;
+ Log(OpenXRLogLevel::Info, "OpenXR simultaneous hands and controllers off");
+ }
+ return;
+ }
+ if (m_simultaneous || m_simultaneous_failed) {
+ return;
+ }
+ XrSimultaneousHandsAndControllersTrackingResumeInfoMETA info{
+ XR_TYPE_SIMULTANEOUS_HANDS_AND_CONTROLLERS_TRACKING_RESUME_INFO_META};
+ const XrResult result = m_resume_simultaneous(m_runtime->Session(), &info);
+ m_runtime->ObserveResult(result);
+ if (XR_FAILED(result)) {
+ // Not asked again until tracked hands are turned off and on.
+ m_simultaneous_failed = true;
+ std::ostringstream message;
+ message << "xrResumeSimultaneousHandsAndControllersTrackingMETA failed (" << result
+ << "); putting a controller down hands over to the cameras only when the system switches";
+ Log(OpenXRLogLevel::Warning, message.str());
+ return;
+ }
+ m_simultaneous = true;
+ Log(OpenXRLogLevel::Info, "OpenXR simultaneous hands and controllers on: a controller put down hands its "
+ "side to the cameras");
}
// Trackers only exist while tracked hands and hand steering are both on: the
@@ -515,7 +566,9 @@ void OpenXRInput::LoadHandTracking() {
// otherwise (Idle and the cockpit's reset keep them); a runtime that refuses
// them is not asked again until the option is turned off and on.
void OpenXRInput::UpdateHandTrackers() {
- if (!RuntimeConfigFile::VrHandTracking() || !RuntimeConfigFile::VrHandSteering()) {
+ const bool wanted = RuntimeConfigFile::VrHandTracking() && RuntimeConfigFile::VrHandSteering();
+ UpdateSimultaneousHandsAndControllers(wanted);
+ if (!wanted) {
DestroyHandTrackers();
m_hand_trackers_failed = false;
m_logged_hand_restart = false;
@@ -747,6 +800,10 @@ void OpenXRInput::Destroy() {
m_destroy_hand_tracker = nullptr;
m_locate_hand_joints = nullptr;
m_hand_data_source = m_hand_aim = false;
+ // The session, and simultaneous tracking with it, goes away after this.
+ m_resume_simultaneous = nullptr;
+ m_pause_simultaneous = nullptr;
+ m_simultaneous = m_simultaneous_failed = false;
m_hand_trackers_failed = m_logged_hand_restart = false;
m_logged_sources = {};
m_sources_logged_at = 0;
@@ -909,18 +966,21 @@ void OpenXRInput::Sync(XrTime predicted_display_time, const OpenXRPointerScreen&
UpdateHandTrackers();
LocateHands(predicted_display_time, seat);
#if defined(__ANDROID__)
- // A hand driving simple_controller is a bare hand: its select is a pinch
- // and its menu the palm-up gesture, which pauses from either hand.
+ // A bare hand (no controller in it: it drives simple_controller, or the
+ // cameras track it): its pinch and menu gesture come from the runtime's
+ // recognition, and the menu gesture pauses from either hand.
bool menu_gesture = false;
for (uint32_t hand = 0; hand < kHands; ++hand) {
- m_hand_driven[hand] = !m_squeeze_active[hand] && select_active[hand];
+ const TrackedHand& located = m_tracked_hands[hand];
+ m_hand_driven[hand] = hand_tracking::HandDriven(
+ m_squeeze_active[hand], select_active[hand],
+ hand_tracking_on && located.active && located.source == hand_tracking::Source::Camera);
if (!m_hand_driven[hand]) {
m_pinch[hand] = false;
continue;
}
- const TrackedHand& tracked = m_tracked_hands[hand];
const hand_tracking::Gestures gestures = hand_tracking::GesturesOf(
- tracked.aim_valid, tracked.aim_pinching, tracked.aim_menu, tracked.aim_system_gesture,
+ located.aim_valid, located.aim_pinching, located.aim_menu, located.aim_system_gesture,
hand_tracking::SelectOf(hands[hand], hand), hands[hand].menu);
m_pinch[hand] = gestures.pinch;
menu_gesture = menu_gesture || gestures.menu;
diff --git a/runtime/src/vr/openxr_integration.cpp b/runtime/src/vr/openxr_integration.cpp
index 295c31d..431f08d 100644
--- a/runtime/src/vr/openxr_integration.cpp
+++ b/runtime/src/vr/openxr_integration.cpp
@@ -196,16 +196,19 @@ XrPosef ScreenPoseAhead(const OpenXRFrame& frame, float distance) noexcept {
// it offers one (XR_FB_hand_tracking_mesh), and tracked hands pose them from
// the hand trackers: the controllers' touch sensors while they are held
// (XR_EXT_hand_tracking_data_source) and the cameras once they are put down,
-// with the runtime's own pinch and menu gesture (XR_FB_hand_tracking_aim).
-// Asked for when either is on at launch, since none costs anything until a
-// tracker exists; turning both on later needs a restart for the mesh and the
-// tracked hands (until then the procedural gloves are drawn).
+// with the runtime's own pinch and menu gesture (XR_FB_hand_tracking_aim), and
+// with a put-down controller handing its side to the cameras at once
+// (XR_META_simultaneous_hands_and_controllers). Asked for when either is on at
+// launch, since none costs anything until it is used; turning both on later
+// needs a restart for the mesh and the tracked hands (until then the
+// procedural gloves are drawn).
void AddHandMeshExtensions(OpenXRConfig& config) {
if (RuntimeConfigFile::VrHandSteering() || RuntimeConfigFile::VrHandTracking()) {
config.optional_extensions.push_back("XR_EXT_hand_tracking");
config.optional_extensions.push_back("XR_FB_hand_tracking_mesh");
config.optional_extensions.push_back("XR_EXT_hand_tracking_data_source");
config.optional_extensions.push_back("XR_FB_hand_tracking_aim");
+ config.optional_extensions.push_back("XR_META_simultaneous_hands_and_controllers");
}
}
diff --git a/runtime/tests/vr_hand_tracking_tests.cpp b/runtime/tests/vr_hand_tracking_tests.cpp
index 075a724..9dc8e94 100644
--- a/runtime/tests/vr_hand_tracking_tests.cpp
+++ b/runtime/tests/vr_hand_tracking_tests.cpp
@@ -143,6 +143,11 @@ void TestGestures() {
left.secondary = true;
right.primary = true;
Check(SelectOf(left, 0) && SelectOf(right, 1), "simple_controller's select per hand");
+
+ Check(!HandDriven(true, true, true), "a hand holding a controller is not bare");
+ Check(HandDriven(false, true, false), "a hand driving simple_controller is bare");
+ Check(HandDriven(false, false, true), "a camera-tracked hand with no profile of ours is bare");
+ Check(!HandDriven(false, false, false), "a hand with nothing active is not");
}
std::array PinchingHands() {