diff --git a/OPENXR.md b/OPENXR.md index c3a6186..93aa79b 100644 --- a/OPENXR.md +++ b/OPENXR.md @@ -275,6 +275,13 @@ see [Steering wheel and hand steering](#steering-wheel-and-hand-steering). Turning the wheel moves the controllers, and the game's own motion detection still reads them, so a sharp enough turn can read as a shake. +**Bare hands.** On the Quest, with `hand_tracking` on and the controllers put down, the hands drive +`khr/simple_controller`: a right pinch is A with the pointer on the hand's aim ray, the left +palm-up pinch is + (pause), and in a cockpit race a hand on the wheel holds A and a free hand's +pinch is Z; see "Tracked hands" under [Steering wheel and hand +steering](#steering-wheel-and-hand-steering). A bare hand feeds no motion. With `hand_tracking` off +a bare hand presses nothing but +. + `"gamepad"` keeps the controllers one ordinary gamepad read through PAD as a GameCube controller: A/B → South/East, X/Y → West/North, index triggers → trigger axes, grips → shoulders, thumbsticks → sticks (clicks → stick buttons), left menu → Start. Every binding in the F10 controller menu @@ -528,6 +535,27 @@ no Wii Remote motion, so the permission changes nothing for players who leave th PC none of this applies: a PC runtime can drive real controllers through `khr/simple_controller` and synthesize joints for them, so a hand-edited `hand_tracking = true` only changes the drawing. +With tracked hands on, bare hands drive. A hand is bare while it drives `khr/simple_controller` +with camera-tracked joints, latched through `wheel_tracking_grace` (Meta also drops the select +action while a hand is lost) and cleared as soon as a controller's squeeze is back. Its palm joint +stands in for the grip and a grasp for the squeeze: the middle, ring and little fingers' flexion, +summed over each finger's three joints, read as 0 below 1.2 radians (a relaxed hand) and 1 from +3.0 (a hand closed on a rim), so the wheel's own 55 % press and 15 % release apply, and closing a +hand on the rim takes hold. The grasp never reaches the game's buttons (as a squeeze it would press +the gamepad's shoulders). In a cockpit race a bare hand holding the wheel holds the gas (A, the +gamepad's South) until both hands let go, and a pinch from a free bare hand uses an item (Z, the +gamepad's L) if the hand has been off the wheel for 0.15 s, so opening a hand off the rim does not +fire one; holding the pinch holds the button. The left palm-up pinch pauses (a pinch made in that +gesture is not an item: `XR_FB_hand_tracking_aim` reports the system gesture). Outside a race, and +in a race whenever the game has the remote's pointer on (the pause menu, the results), a right +pinch is A and the pointer follows the hand's aim ray; a left pinch does nothing, so the headset +panel cannot be opened with bare hands (an open one takes a right pinch, and the palm-up pinch +closes it). The game's pointer switch is read where KPAD polls the VR remote, and the log notes +each change (`OpenXR bare hands: the game's pointer is off (driving)`); with the gamepad +presentation there is no pointer, so the race controls apply throughout the cockpit. Manual drift +has no gesture: choose Automatic drift. The headset panel reads out each hand's source, grasp, +hold and pinch under the checkbox, for tuning. + **Hands and the separate wheel.** Hands are drawn while hand steering is on: the runtime's own hand mesh where it offers one (`XR_EXT_hand_tracking` and `XR_FB_hand_tracking_mesh`, requested when hand steering or tracked hands are on at launch), otherwise procedural gloves that curl with the diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xml index c91a937..f923ab6 100644 --- a/android/app/src/main/res/values/strings.xml +++ b/android/app/src/main/res/values/strings.xml @@ -534,7 +534,7 @@ Hand steering In the cockpit, squeeze a grip near the steering wheel or handlebar to grab it, and turn it to steer. Releasing both grips gives steering back to the stick. Hand steering by heurazy. Tracked hands - The cockpit hands follow your own. Holding the controllers, the fingers follow their touch sensors; put the controllers down and the headset\'s cameras track your hands. Needs hand tracking on in the headset\'s settings. + The cockpit hands follow your own. Holding the controllers, the fingers follow their touch sensors. Put the controllers down to drive with bare hands: close a hand on the wheel to hold it, which also holds the gas; pinch with a free hand to use an item; pinch with your left palm facing you to pause. In menus, a right pinch is A. Choose Automatic drift. Needs hand tracking on in the headset\'s settings. Lean back angle Tilts the race view back for playing reclined. 0 applies no tilt. diff --git a/runtime/include/vr/openxr_input.h b/runtime/include/vr/openxr_input.h index f14a35f..43b9915 100644 --- a/runtime/include/vr/openxr_input.h +++ b/runtime/include/vr/openxr_input.h @@ -237,6 +237,11 @@ private: std::array m_squeeze_active{}; std::array m_hand_driven{}; std::array m_pinch{}; + // Bare-hand driving: each hand's bare latch (camera joints, last grasp) and + // item pinch gate, and the game's pointer switch as last logged. + std::array m_bare_latch{}; + std::array m_pinch_gate{}; + int m_logged_game_pointer = -2; uint64_t m_profile_serial = 0; bool m_created = false; diff --git a/runtime/include/vr/openxr_wii_remote.h b/runtime/include/vr/openxr_wii_remote.h index ffd28d3..0b3804c 100644 --- a/runtime/include/vr/openxr_wii_remote.h +++ b/runtime/include/vr/openxr_wii_remote.h @@ -56,10 +56,18 @@ bool OpenXRReadWiiRemote(OpenXRWiiRemoteSample& sample) noexcept; // WPADControlMotor for the emulated remote. void OpenXRSetWiiRemoteRumble(bool active) noexcept; +// Guest side: whether the game has the remote's pointer switched on, as KPAD +// reads it each time the game polls the VR remote (Input::WiiController:: +// TogglePointer flips it). Tracked bare hands use it to tell driving from a +// menu inside a race; OpenXRInput logs every change. +void OpenXRPublishGamePointer(bool enabled) noexcept; + // XR side. void OpenXRPublishWiiRemote(uint32_t sdl_joystick_id, const OpenXRWiiRemoteSample& sample) noexcept; void OpenXRWithdrawWiiRemote() noexcept; bool OpenXRWiiRemoteRumbleRequested() noexcept; +// The game's pointer switch: 1 on, 0 off, -1 not known (no remote polled yet). +int OpenXRGamePointerState() noexcept; // The geometry and signal conditioning behind a sample, kept free of OpenXR // types so it can be checked headlessly (tests/vr_wii_remote_tests.cpp). diff --git a/runtime/src/hle/input/kpad.cpp b/runtime/src/hle/input/kpad.cpp index dd3fc3b..1c2165f 100644 --- a/runtime/src/hle/input/kpad.cpp +++ b/runtime/src/hle/input/kpad.cpp @@ -1,5 +1,6 @@ #include "hle_stubs.h" #include "memory.h" +#include "vr/openxr_wii_remote.h" #include "wii_remote_input.h" #include @@ -276,6 +277,11 @@ extern "C" int32_t KPAD__Read_HLE(uint32_t chan, uint32_t statusPtr, uint32_t co WiiRemoteInput::KpadSample sample; const bool have = WiiRemoteInput::ReadKpadSample(chan, sample); try { + if (have && WiiRemoteInput::IsVrControllerChannel(chan)) { + // Tracked bare hands drive with the pointer off and point at menus + // with it on (openxr_hand_tracking.h). + mkw::vr::OpenXRPublishGamePointer(DpdEnabled(chan)); + } return WriteStatus(chan, statusPtr, have ? &sample : nullptr); } catch (const Memory::AccessViolation&) { return 0; diff --git a/runtime/src/settings_overlay.cpp b/runtime/src/settings_overlay.cpp index f29b6af..cc54206 100644 --- a/runtime/src/settings_overlay.cpp +++ b/runtime/src/settings_overlay.cpp @@ -1192,8 +1192,11 @@ void DrawVrSteeringWheelSettings() { ImGui::SetTooltip( "%s", mkw::vr::OpenXRHandTrackingAvailable() ? "The cockpit hands follow your own. Holding the controllers, the fingers follow " - "their touch sensors; put the controllers down and the headset's cameras track " - "your hands. Needs hand tracking on in the headset's settings." + "their touch sensors. Put the controllers down to drive with bare hands: close a " + "hand on the wheel to hold it, which also holds the gas; pinch with a free hand " + "to use an item; pinch with your left palm facing you to pause. In menus, a " + "right pinch is A. Choose Automatic drift. Needs hand tracking on in the " + "headset's settings." : "The cockpit hands follow your own, from the controllers' touch sensors or, " "with the controllers put down, the headset's cameras. This session started " "with hand steering and tracked hands off, so it applies after a restart."); diff --git a/runtime/src/vr/openxr_input.cpp b/runtime/src/vr/openxr_input.cpp index 83872c1..26d9303 100644 --- a/runtime/src/vr/openxr_input.cpp +++ b/runtime/src/vr/openxr_input.cpp @@ -744,6 +744,7 @@ void OpenXRInput::Destroy() { m_hand_trackers_failed = m_logged_hand_restart = false; m_logged_sources = {}; m_squeeze_active = m_hand_driven = m_pinch = {}; + m_logged_game_pointer = -2; m_profile_serial = 0; DestroyPoseSpaces(); if (m_action_set != XR_NULL_HANDLE) { @@ -970,6 +971,35 @@ void OpenXRInput::Sync(XrTime predicted_display_time, const OpenXRPointerScreen& // steers through the left stick and keeps its grips from the game. UpdateDriving(predicted_display_time, seat, hands, panel.withheld); +#if defined(__ANDROID__) + // Bare hands in a cockpit race: a hand on the wheel holds the gas and a free + // hand's pinch uses an item. Only while the game's pointer is off: the + // pause menu and the results keep the cockpit but need A from a pinch. + const bool wii_remote = OpenXRGetControllerMode() == OpenXRControllerMode::WiiRemote; + const int game_pointer = wii_remote ? OpenXRGamePointerState() : -1; + if (hand_tracking_on && game_pointer != m_logged_game_pointer) { + m_logged_game_pointer = game_pointer; + Log(OpenXRLogLevel::Info, game_pointer > 0 ? "OpenXR bare hands: the game's pointer is on (menu)" + : game_pointer == 0 ? "OpenXR bare hands: the game's pointer is off (driving)" + : "OpenXR bare hands: the game's pointer is unknown"); + } + const bool bare_race = hand_tracking_on && m_driving.cockpit_active && m_driving.hand_steering && + !panel.withheld && game_pointer <= 0; + if (bare_race) { + std::array bare_held{}, item_pinch{}; + for (uint32_t hand = 0; hand < kHands; ++hand) { + bare_held[hand] = m_bare_latch[hand].Bare() && m_wheel_held[hand]; + item_pinch[hand] = m_hand_driven[hand] && m_pinch_gate[hand].Update(m_pinch[hand], m_wheel_held[hand], + dt_seconds); + } + hand_tracking::ApplyBareHandRace(hands, m_hand_driven, bare_held, item_pinch); + } else { + for (auto& gate : m_pinch_gate) { + gate.Reset(); + } + } +#endif + // While the panel has the controllers, the game sees them idle. static const std::array kIdleHands{}; const auto& game_hands = panel.withheld ? kIdleHands : hands; @@ -1139,6 +1169,10 @@ void OpenXRInput::ResetDriving() { m_wheel_time = 0; m_driving = {}; m_joint_frame.valid = {}; + for (uint32_t hand = 0; hand < kHands; ++hand) { + m_bare_latch[hand].Reset(); + m_pinch_gate[hand].Reset(); + } OpenXRPublishDriving(m_driving); } @@ -1197,6 +1231,9 @@ void OpenXRInput::UpdateDriving(XrTime display_time, const driving::SeatFrame& s constexpr XrSpaceLocationFlags kPoseValid = XR_SPACE_LOCATION_POSITION_VALID_BIT | XR_SPACE_LOCATION_ORIENTATION_VALID_BIT; const bool hand_tracking_on = RuntimeConfigFile::VrHandTracking(); + // The wheel's own tracking grace (SteeringWheel::Update clamps it the same way). + const float grace = driving::IsFinite(tuning.trackingGrace) ? std::clamp(tuning.trackingGrace, 0.05f, 0.5f) + : 0.2f; std::array wheel_hands{}; for (uint32_t hand = 0; hand < kHands; ++hand) { bool tracked = false; @@ -1214,12 +1251,24 @@ void OpenXRInput::UpdateDriving(XrTime display_time, const driving::SeatFrame& s } const float squeeze = hands[hand].squeeze; const TrackedHand& located = m_tracked_hands[hand]; + const bool fresh_joints = hand_tracking_on && located.active && located.seated; + // A bare hand: camera-tracked joints on a hand driving simple_controller + // (Android), latched through the wheel's grace with its last grasp. + const bool camera_joints = fresh_joints && located.source != hand_tracking::Source::Controller; + const float grasp = camera_joints ? hand_tracking::GraspFromJoints(located.positions) : 0.0f; + const bool bare = hand_tracking_on && + m_bare_latch[hand].Update(m_hand_driven[hand], m_squeeze_active[hand], camera_joints, + grasp, dt, grace); + if (!hand_tracking_on) { + m_bare_latch[hand].Reset(); + } // With tracked hands on, a hand whose joints were located is drawn from - // them (the controller's touch sensors, or the cameras). A bare hand is - // never drawn or steered from its grip pose, which would show an open - // hand wherever it rests. - const bool joints = hand_tracking_on && located.active && located.seated; - if (m_hand_driven[hand]) { + // them (the controller's touch sensors, or the cameras), and a bare hand + // holding the wheel keeps its last joints through a short loss. A bare + // hand is never drawn or steered from its grip pose, which would show + // an open hand wherever it rests. + const bool joints = fresh_joints || (bare && m_wheel_held[hand]); + if (m_hand_driven[hand] || bare) { tracked = false; } DrivingHand& out = snapshot.hands[hand]; @@ -1229,10 +1278,20 @@ void OpenXRInput::UpdateDriving(XrTime display_time, const driving::SeatFrame& s out.squeeze = squeeze; out.seat_from_grip = seat_from_grip; out.joints_valid = joints; + out.bare = bare; out.source = located.source; + out.grasp = bare ? m_bare_latch[hand].Grasp() : grasp; out.pinch = m_pinch[hand]; m_joint_frame.valid[hand] = joints; - wheel_hands[hand] = {seat_from_grip[3], seat_from_grip[7], seat_from_grip[11], squeeze, tracked}; + if (bare) { + // The palm stands in for the grip, and the fingers' grasp for the + // squeeze; the grasp never reaches the game's buttons. + const auto& palm = m_joint_frame.seat_from_joint[hand][hand_tracking::kPalm]; + wheel_hands[hand] = {palm[3], palm[7], palm[11], m_bare_latch[hand].Grasp(), + m_bare_latch[hand].Tracked()}; + } else { + wheel_hands[hand] = {seat_from_grip[3], seat_from_grip[7], seat_from_grip[11], squeeze, tracked}; + } if (uses_geometry) { wheel_hands[hand] = geometry.ToWheel(wheel_hands[hand]); } diff --git a/runtime/src/vr/openxr_wii_remote.cpp b/runtime/src/vr/openxr_wii_remote.cpp index ebe14de..24cbec2 100644 --- a/runtime/src/vr/openxr_wii_remote.cpp +++ b/runtime/src/vr/openxr_wii_remote.cpp @@ -25,6 +25,8 @@ std::atomic g_mode{OpenXRControllerMode::WiiRemote}; // SDL_JoystickID of the virtual gamepad the samples belong to; 0 when none. std::atomic g_joystick_id{0}; std::atomic g_rumble{false}; +// -1 until the game polls the VR remote, then its pointer switch. +std::atomic g_game_pointer{-1}; } // namespace wii_remote_bridge @@ -73,9 +75,18 @@ void OpenXRPublishWiiRemote(uint32_t sdl_joystick_id, const OpenXRWiiRemoteSampl wii_remote_bridge::g_joystick_id.store(sdl_joystick_id, std::memory_order_relaxed); } +void OpenXRPublishGamePointer(bool enabled) noexcept { + wii_remote_bridge::g_game_pointer.store(enabled ? 1 : 0, std::memory_order_relaxed); +} + +int OpenXRGamePointerState() noexcept { + return wii_remote_bridge::g_game_pointer.load(std::memory_order_relaxed); +} + void OpenXRWithdrawWiiRemote() noexcept { wii_remote_bridge::g_joystick_id.store(0, std::memory_order_relaxed); wii_remote_bridge::g_rumble.store(false, std::memory_order_relaxed); + wii_remote_bridge::g_game_pointer.store(-1, std::memory_order_relaxed); auto& published = wii_remote_bridge::Published(); std::lock_guard lock(published.mutex); published.available = false;