VR keyboard swipe with two controllers: the path from the controller bridge

With both lasers on the keyboard SteamVR forwards per poll only one laser's
movement, so the pressing one may get no touchmoves and its swipe became a
tap. keyboard.vr.swipe.twoHanded (default on; turns on the controller
bridge): a press is attributed to the hand whose laser hit is nearest to it
(within 30 px, the trigger as a tiebreak; without recent frames the first
frame after the press decides, within 60 px). The path comes from runs of
one source: the touch's own moves while they flow, that hand's laser hit
per bridge frame once they pause for 60 ms, touchmoves again if frames
pause. A touchend disagreeing with the laser during a bridge run ends at
the laser. Touchmoves on the other hand's laser move the attribution.
Without fresh frames: touch events only, as before. A second press during
a swipe stays Steam's tap (one text field, one word at a time).

The bridge sends full rate only on demand now: hub.js demand(ms) goes
back through the relay (CDP binding __sfuiCtlIn) to systemui's
__sfuiCtl.demand; the swipe patch asks from the press to the release.

Tests: gesture-input with the two-controller recording and synthetic bridge
frames (a pressing hand without touchmoves, attribution with both lasers on
the keyboard, a stale bridge, frames stopping mid-swipe, re-attribution,
interleaved hover and presses); hub demand.
This commit is contained in:
Pierre Kisters committed 2026-10-01 03:45:47 +02:00
1 parent 0699d0f532
commit af32cf72ce
13 files changed
+423 -53

No files matched your search

+2 -2
View File
@@ -104,7 +104,7 @@ modules/
swipe-decoder.js Steam (argument of patch.js): swipe path -> words
textmodel.js Steam (argument of patch.js): what the keyboard typed
corrector.js Steam (argument of patch.js): corrections, completions
gesture-input.js Steam (argument of patch.js): a gesture's own events
gesture-input.js Steam (argument of patch.js): a gesture's own events, its hand's bridge path
suggestions-panel/
patch.js, unpatch.js SteamVR: the strip as a panel above/below the keyboard
relay.mjs service vr-keyboard-relay: strip state Steam <-> SteamVR
@@ -116,7 +116,7 @@ modules/
bridge-patch.js, unpatch.js SteamVR: keyboard pose, tips, laser hits, triggers
geometry.js SteamVR (argument of bridge-patch.js): tip/laser -> keyboard
hub.js Steam (argument of consumer patches): __sfuiControllers
relay.mjs service vr-keyboard-controllers-relay: frames SteamVR -> Steam
relay.mjs service vr-keyboard-controllers-relay: frames SteamVR -> Steam, demand back
check.nix, tests/ test: geometry, hub
vr-keyboard-touch.nix touch typing (keyboard.vr.touchTyping)
vr-keyboard-touch/
+26 -9
View File
@@ -103,7 +103,8 @@ default):
not swiped.
With both controllers on the keyboard, a swipe follows only the
controller that pressed; the other one's laser doesn't enter its path,
and its presses meanwhile are plain taps.
and its presses meanwhile are plain taps. Both hands can swipe, one word
at a time (`swipe.twoHanded`, below).
- **Suggestions** never change text by themselves: a finished tapped word
that isn't in the dictionary gets corrections (itself first;
`autocorrect`), a word being tapped gets completions (the typed letters
@@ -144,9 +145,7 @@ via its xdotool helper).
changes them the keyboard stays stock (see
[after a Steam update](ui-patches.md#after-a-steam-update)). Accented words
of other languages are in the dictionary but only swipable where the layout
has the letters. With two controllers on the keyboard, SteamVR sometimes
sends no laser movement for a press (seen for the controller that didn't
own the keyboard's cursor just before): that press stays a tap.
has the letters.
**Remove when** Steam's VR keyboard gets swipe typing and suggestions.
@@ -164,15 +163,31 @@ own the keyboard's cursor just before): that press stays a tap.
port 8087), fed by the `vr-keyboard-relay` user service
(`suggestions-panel/relay.mjs`) between the two pages.
With `inside` neither the panel nor the relay runs.
- **Two controllers** (`swipe.twoHanded`, default on): with both lasers on
the keyboard SteamVR forwards per poll only one laser's movement, so the
pressing one may get none and its swipe would become a tap. The swipe
then follows that controller through the
[controller bridge](#how-it-works-2) (turned on by this option): a press
is attributed to the hand whose laser hit is nearest to it (within
30 px; the trigger as a tiebreak), and while its laser events pause
(60 ms) the path comes from that laser's hit per frame. The bridge
samples at full rate from the press to the release (asked for by the
patch) and costs nothing while the keyboard is hidden. Without fresh
frames (option off, relay or SteamVR gone) only laser events count, as
before. A second press during a swipe is Steam's (a tap): one text
field, one word at a time.
- Found by signature (entries `vr-keyboard`, `vr-keyboard-panel`).
**Tests:** `nix flake check` (checks `vr-keyboard`: text model, corrector,
decoder accuracy on German + English,
gesture input from two controllers) and `keyboard.vr.checks` for the
gesture input from two controllers, also with synthetic bridge frames:
a pressing hand without laser events, attribution, a stale bridge) and
`keyboard.vr.checks` for the
configured dictionary.
**Debugging:** `window.__sfuiSwipeLog` and `__sfuiSwipePaths` in Steam's
SharedJSContext (replay swipes with `scripts/vr-keyboard-replay.mjs`).
SharedJSContext (replay swipes with `scripts/vr-keyboard-replay.mjs`;
`__sfuiSwipePaths[i].src`: the hand, touch and bridge points).
## Touch typing
@@ -207,7 +222,7 @@ to the laser only. Depends on Steam and SteamVR UI internals (see
### How it works
- **Controller bridge** (module `vr-keyboard-controllers`, turned on by
touch typing; meant for other keyboard features too): a patch of SteamVR's
touch typing and `swipe.twoHanded`): a patch of SteamVR's
`systemui` page (port 8087, [SteamVR debugger](steamvr-debugger.md),
turned on automatically). It reads the keyboard's pose with SteamVR's own
scene graph query (`SGQueryService.requestSGTransform` on an empty
@@ -219,8 +234,10 @@ to the laser only. Depends on Steam and SteamVR UI internals (see
animated `trigger` component (not yet verified). The
`vr-keyboard-controllers-relay` user service carries these frames to
Steam's `SharedJSContext`: up to ~90 Hz (45 Hz and more under load)
while a hand's tip is within 10 cm of the keyboard or its trigger is
pulled, else every second; nothing while the keyboard is hidden. The
while a hand's tip is within 10 cm of the keyboard, its trigger is
pulled or Steam asks for it (`__sfuiControllers.demand(ms)`, back through
the relay: the swipe patch during a press), else every second; nothing
while the keyboard is hidden. The
laser hit matches SteamVR's laser to a few px.
- **Touch typing** (patch `vr-keyboard-touch`, `SharedJSContext`): per hand
`tracker.js` turns the tip's path into press and release, then calls the