Pose the cockpit hands from the headset's hand tracking

- New [vr] hand_tracking (default off, Quest only for now): the Quest launcher's Settings > VR and
  the headset panel's VR tab, under hand steering. Two XR_EXT_hand_tracking trackers are located
  every XR frame: with the controllers held the Quest builds the joints from their touch sensors
  (XR_EXT_hand_tracking_data_source's controller source), once they are put down from its
  cameras. The trackers exist only while the option and hand steering are on and also serve the
  runtime hand mesh; the extensions (plus XR_FB_hand_tracking_aim) are asked for when either is
  on at launch.
- Aurora skins the runtime mesh with the joints themselves (tracked pose times inverse bind pose,
  no curl, no grip); runtimes with joints but no mesh get a skeleton; non-finite joints put only
  that hand back on its grip curl. AuroraCockpitHand carries the 26 seated-frame joints and radii.
- The manifest declares horizonos.permission.HAND_TRACKING (and the deprecated
  com.oculus.permission.HAND_TRACKING), both normal permissions with no prompt on a Quest 3, and
  oculus.software.handtracking as optional; without it Horizon OS keeps the app controllers-only.
  Bare hands then drive khr/simple_controller, so on Android a hand whose squeeze action is
  inactive and select active is treated as bare: with the option off it presses nothing but the
  menu gesture, is not drawn and feeds no Wii Remote motion.
- Interaction-profile changes and tracker sources are logged. The pure rules (grasp from finger
  flexion, bare latch, pinch gate, bare-hand buttons, flick) live in the OpenXR-free
  vr/openxr_hand_tracking.h with mkw_vr_hand_tracking_tests; the driving and flick parts are
  wired in the next commits. Docs: OPENXR.md, docs/quest-port.md, README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
iChris4andClaude Opus 5.5 committed 2026-09-25 00:05:48 +02:00
1 parent c990c595f0
commit a21e22b605
24 files changed
+1549 -49

No files matched your search

+36 -4
View File
@@ -166,6 +166,9 @@ live, from the headset panel's VR tab or the launcher's Settings page. The app d
So that the room frames the picture rather than black bands, the Quest's menu quad shows only the
part of its eye-sized image Aurora draws into (the desktop snapshot, and the in-eye settings
panel's rectangle), at the same size per pixel, so nothing moves.
`hand_tracking` (Quest only, default off) makes the first-person cockpit's hands follow the
headset's hand tracking; see "Tracked hands" under
[Steering wheel and hand steering](#steering-wheel-and-hand-steering).
`stop_at_display_copy` ends eye replay at the final `GXCopyDisp`, matching the frame shown on the
desktop. `skip_copy_clears` independently suppresses the EFB reset performed after a copy. Both
default on and can be changed live from the F10 settings bar for diagnostics.
@@ -496,11 +499,40 @@ player 1's GameCube controller. The cockpit's wheel follows its calibrated steer
full-lock angle as the stick (`wheel_kart_degrees`, `wheel_bike_degrees`), and hand steering steps
aside while it drives.
**Tracked hands.** `hand_tracking` (Quest only for now, default off; the Quest launcher's
Settings > VR and the headset panel's VR tab, under hand steering, which it needs) poses the
cockpit hands from the headset's hand tracking instead of curling them with the grip. Two hand
trackers (`XR_EXT_hand_tracking`) are located every XR frame at the display time. While the
controllers are held the Quest builds the joints from their touch sensors
(`XR_EXT_hand_tracking_data_source`'s controller source: the trigger finger, the thumb on its rest or
a button, the grip); once they are put down, from its cameras. The runtime's hand mesh is then
skinned with the joints themselves, each joint's tracked pose times its inverse bind pose (the bind
poses are in the mesh's space, as `xrLocateHandJointsEXT` reports poses), with no curl and no grip;
a runtime with joints but no mesh (SteamVR, Virtual Desktop) gets a skeleton along the joints. A
hand whose joints are not located, or not finite, falls back to the curl at its grip. The trackers
exist only while the option and hand steering are both on, and serve the mesh too; the extensions
(with `XR_FB_hand_tracking_aim`) are asked for when either is on at launch, otherwise turning the
option on applies after a restart. The log says `OpenXR tracked hands ready (controller-driven
hands: yes|no)` and, at each change, `OpenXR tracked hands: left camera, right controller`; the
headset panel shows each hand's source under the checkbox.
The Quest app declares `horizonos.permission.HAND_TRACKING` (and the older
`com.oculus.permission.HAND_TRACKING`), normal permissions granted at install with no prompt, and
`oculus.software.handtracking` as optional: without that flag Horizon OS keeps the app
controllers-only. With it, putting the controllers down drives `khr/simple_controller` from the
hands (an index pinch is select, the left palm-up pinch the menu), and every change of interaction
profile is logged (`OpenXR interaction profiles: left ..., right ...`). A hand driving that profile
instead of the Touch one (its squeeze action inactive, its select active) is a bare hand; with
tracked hands off it presses nothing but the menu gesture, which pauses, is not drawn, and feeds
no Wii Remote motion, so the permission changes nothing for players who leave the option off. On
PC none of this applies: a PC runtime can drive real controllers through `khr/simple_controller`
and synthesize joints for them, so a hand-edited `hand_tracking = true` only changes the drawing.
**Hands and the separate wheel.** Hands are drawn while hand steering is on: the runtime's own hand
mesh where it offers one (`XR_EXT_hand_tracking` and `XR_FB_hand_tracking_mesh`, requested only when
hand steering is on at launch), otherwise procedural gloves that curl with the squeeze. A Quest 3
offers that mesh without the app declaring hand tracking, and the log says which is drawn
(`[mkw-vr] cockpit hands:`). Both close their fingers towards the palm: the mesh's joints point
mesh where it offers one (`XR_EXT_hand_tracking` and `XR_FB_hand_tracking_mesh`, requested when
hand steering or tracked hands are on at launch), otherwise procedural gloves that curl with the
squeeze. A Quest 3 offers that mesh even without the app declaring hand tracking, and the log says
which is drawn (`[mkw-vr] cockpit hands:`). Both close their fingers towards the palm: the mesh's joints point
-Z towards the fingertip and +Y out of the back of the hand, so flexion is negative about the
joint's own X, on both hands. They and the
separate VR wheel or handlebar travel with the stereo packet in metres in the seated frame, and each