mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-06 07:00:14 +02:00
Gaze first: plan, and tools for the headset tests
docs/gaze-first.md is the plan agreed on 2026-09-30: with gaze on and no game, the gaze drives the 3D pointer everywhere, either trigger clicks where you look (tap, precision by hand movement, hold to drag), and the controllers keep everything but their lasers. For its four tests: - The driver takes "role treadmill", and its compositor bindings repeat the right hand's under /user/treadmill. They're additive, so nothing changes while the device is a hand. POINTER_ROLE accepts treadmill. - pointer/probe/lasertest follows who has the dashboard laser and the controllers' roles, and with --snapback takes the laser back. - input/vrws.py reads vrserver's web socket with the standard library (the host's Python has no websockets module) and prints component changes and update rates. Checked: it connects, and lists the controllers' bumper, thumbstick axes, and Steam button, and the headset's /proximity, which flickers off for 0.3-0.5 s while worn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DeeJanuzandClaude Opus 5.5 committed
1 parent
d5c2ec65d7
commit
bd06207db9
7 files changed
+575
-10
No files matched your search
@@ -0,0 +1,143 @@
|
||||
# Gaze first: plan
|
||||
|
||||
Gaze first makes the eyes the pointer for everything flat in VR, and the controllers its buttons. With gaze on and no game running, the gaze drives Frametop's 3D pointer (the `ft_pointer` driver) everywhere the mouse can go, SteamVR's dashboard included. Either controller's trigger clicks where you look, and the controllers stop showing lasers. The work happens on branch `gaze-first` (worktree `~/frametop/.worktrees/gaze-first`, from `experimental`) and is merged into `experimental` after it's been tested in the headset.
|
||||
|
||||
The decisions below were made with the user on 2026-09-30. Nothing is built yet: the first step is a headset session with four tests (see "Tests before building").
|
||||
|
||||
## Decisions
|
||||
|
||||
Input, with gaze on and outside games:
|
||||
- The gaze drives `ft_pointer`, so anything the mouse does today works with the gaze, on Frametop's screens, floating windows, the keyboard, and SteamVR's and Steam's panels. The mouse keeps working as it does today in gaze mode.
|
||||
- The gaze wakes the pointer by itself: the headset is worn and the tracker sees your eyes. No mouse movement is needed.
|
||||
- Trigger, on either controller:
|
||||
- A tap clicks where you look.
|
||||
- Moving the controller within `POINTER_GAZE_HOLD` (0.5 s) of the press is precision: the pointer stops where you looked, and your hand's movement steers it, relative, like tap and drag on the Apple Vision Pro (the controller's position seen from the eye, not where it points). The release clicks there, and the correction goes to the gaze service as a lesson.
|
||||
- Holding still for 0.5 s makes a real press; then the hand's movement drags 1:1.
|
||||
- Bumper, on either controller: the same, with the right button.
|
||||
- Right thumbstick: scrolls at the pointer (vertical, and horizontal when pushed sideways).
|
||||
|
||||
Controllers:
|
||||
- They keep their hand roles and their stock SteamVR bindings: the Steam button (tap for the dashboard, double tap for the room view, hold to recenter, the screenshot chord), gamepad mode (both grips), locomotion, room setup, and every per-hand feature.
|
||||
- Only the trigger and the bumper are muted from SteamVR while gaze first is on, so they never click or take the laser.
|
||||
- When something still moves the laser to a controller (a Steam button summoning the dashboard, a grip), the helper moves it straight back to our device.
|
||||
- Later: mute the grips too, and maybe have turning gaze off switch the controllers to gamepad mode.
|
||||
|
||||
Turning gaze on and off:
|
||||
- Gaze is opt-in and off by default. On or off is remembered across restarts (`POINTER_GAZE`), whatever turned it on or off.
|
||||
- Any game gets the controllers: a VR game (a scene application) or a flatscreen Steam game. Gaze first is off during one.
|
||||
- The toggle macro, both thumbstick clicks held 1 s by default, turns gaze on or off. In a game it turns gaze first on for that game only, until the game exits.
|
||||
- You can record your own toggle macro on the Gaze page of Frametop Input Settings: a chord of buttons on one or both controllers, held together for 0.5 to 2 s. The Steam button can't be part of it. It needs 2 or more buttons, or 1 button held at least 1.5 s. Firing it cancels a gaze click in progress. "Reset to default" brings back the thumbsticks. The one-button "Gaze pointer on/off" mapping and key combinations keep working.
|
||||
- Gaze can't turn on without a calibration for the tracker in use, or without a working tracker service. Turning it on then opens the calibrator, or points to "Repair eye tracker".
|
||||
|
||||
Eye tracker:
|
||||
- Our own tracker (`gaze/tracker/`) is the default. Choosing SteamVR's tracker on the Gaze page turns ours off. The same calibration rule applies to SteamVR's.
|
||||
- Its root service, `frametop-eyegrab`, is installed by Frametop's installer (a step that defaults to yes) and stays enabled. Turning our tracker off means no longer asking it for frames: it then holds none of SteamVR's buffers, and nothing needs root.
|
||||
- The tracker waits for a calibration before tracking, and idles whenever gaze input is off (tracking costs roughly 7 to 36 % of a core).
|
||||
- If the service is missing or broken, gaze can't turn on, and the Gaze page offers "Repair eye tracker". There's no silent switch to SteamVR's tracker.
|
||||
- Updates: the gaze service compares the installed `ft-eyegrab` with the build, and when they differ the Gaze page offers "Update eye tracker". Both repair and update ask for the password each time, through polkit (`pkexec`). There's no passwordless sudoers or polkit rule: the build output is writable by the user, so such a rule would let any program running as the user get root.
|
||||
|
||||
Calibration, in one head-locked panel:
|
||||
- Quick check: one centre dot. It's captured by a dwell (about 0.6 s of steady fixation; steadiness, not position, so it works however far off the tracker is), the trigger accepts early, and it closes itself after about 4 s if ignored. It opens when the headset is put on, when our tracker notices the headset slipping (at most once every 2 minutes), and from a mappable action. If the first 3 nudges after it are still more than 2 degrees off, it asks for 5 dots.
|
||||
- Full calibration: the same panel, about 60 degrees across, running the probe's calibration (21 dots in dark, medium, and bright rounds). Frametop's screens hide while it runs. It opens when turning gaze on finds no calibration. Quitting it leaves gaze off; turning gaze on again reopens it. Resetting the calibration while gaze is on turns gaze off and opens it.
|
||||
- The dots move with your head, so there's no "keep your head still", and the calibration doesn't depend on where the screens are.
|
||||
- The gaze probe stays as the lab tool.
|
||||
|
||||
## What exists
|
||||
|
||||
- Gaze mode (`POINTER_GAZE`, `pointer/helper/ft-pointer.cpp`): the gaze aims the pointer; the mouse's held-back press, precision, hold to drag, and nudge lessons are the model for the trigger.
|
||||
- Holds in the helper (`struct Hold`): pinches and grips steer by the hand's movement seen from the eye, in the room, from where the eye was when the gesture began (`PoseHistory`). The trigger's steering is the same with the controller's position instead of the hand's.
|
||||
- `gaze_precision` and `gaze_drag` (relay actions, "precision|gazedrag <source> 1|0" to the helper) steer by the controller's aim. Controller holds switch to position steering.
|
||||
- Controller buttons through the helper's global action sets (`pointer/helper/vrbuttons.h`), one action set per button, active only for mapped buttons and only outside games.
|
||||
- ft-screens' `controllers always|outside_games|dashboard` and its `hide`/`show` switch.
|
||||
- Our tracker's headset-moved detection (`gaze/tracker/eyes_model.py`: a glint slip change over 3 px held 1 s) and its reseat after the frames stop.
|
||||
- `gaze/tracker/install.sh` already installs `ft-eyegrab` as a system service with sudo.
|
||||
|
||||
## What SteamVR does (found 2026-09-30)
|
||||
|
||||
- The Frame controller's compositor bindings (`/opt/steamvr/drivers/frame_controller/resources/input/vrcompositor_bindings_frame_controller.json`) put the laser on a controller in only two ways: a button that clicks or switches the laser (trigger and bumper click; trigger, bumper, and grip move the laser to that hand with `switchlaserhand`), or summoning the dashboard with its Steam button. Everything else there doesn't touch the laser.
|
||||
- The gamepad and laser modes are the `/actions/dualanalog` action set (`ModeSwitch1` and `ModeSwitch2` on the grips), with `dashboard.modalGamepadAndLaser`. Those actions are application-scoped: the mode lives in Steam's UI, and we can't set it from outside.
|
||||
- The laser doesn't need a hand. The headset's own binding (`/opt/steamvr/drivers/frame_hmd/resources/input/vrcompositor_bindings_frame_hmd.json`) runs the laser from `/user/head/pose/raw`. Our device has only tried the right, left, and stylus roles; the stylus attempt got no role at all.
|
||||
- A Frame controller in the hand takes its hand's role back through its touch sensors, so a device that needs a hand role loses it whenever both controllers are held. That's why the laser has to live somewhere else.
|
||||
- vrserver's web socket (`/input/getstate.json` and `request_input_state_updates`, as frame-voice uses) lists each controller's `/input/trigger/click`, `/input/bumper/click`, `/input/grip/click`, `/input/thumbstick/click`, `/input/thumbstick/x` and `y`, `/input/system/click`, and each device's `side`. The headset has `/proximity`, but it flickers off for 0.3 to 0.5 s at a time while worn, so the quick check follows SteamVR's activity level (as the helper already does) rather than the raw sensor. Reading the socket takes nothing from anyone. A controller's path is `/devices/cv/<serial>` instead of `/user/hand/<side>` while our device holds that hand.
|
||||
|
||||
## Tests before building
|
||||
|
||||
One headset session, about 30 minutes, with the user wearing the headset. Installing the test driver needs a SteamVR restart, which closes everything in VR, so it's done at the start of the session and only when the user says so.
|
||||
|
||||
1. **Laser on the treadmill role.** The driver takes a `role treadmill` command, and its compositor bindings repeat the right hand's under `/user/treadmill`. Pass: after its `switchlaserhand` (`/input/a`), `GetPrimaryDashboardDevice()` is our device, and the pointer clicks Frametop's screens and the dashboard while both controllers are held. Fail: the fallback (below).
|
||||
2. **Muting.** The helper activates its trigger and bumper action sets on both sides. Pass: a real trigger or bumper, pointed at a panel, neither clicks nor moves the laser, with the dashboard open and closed, and the helper sees the press. Fail: we need our own compositor binding for the Frame controller, switched when gaze first turns on and off.
|
||||
3. **Snap back.** A Steam button tap on either controller, and a grip. Pass: the helper sees the laser move to a controller and moves it back within about 100 ms, with no stray click.
|
||||
4. **Web socket.** Update rates for the thumbstick axes and clicks while held, and `/proximity` at don and doff.
|
||||
|
||||
Fallback if test 1 fails: gaze input goes straight into ft-screens for Frametop's own panels (the helper already knows which panel it hits and where), with both controllers held. SteamVR's and Steam's panels then take the gaze only while one hand is empty.
|
||||
|
||||
## Design by component
|
||||
|
||||
### Driver (`pointer/driver`)
|
||||
|
||||
- `role treadmill` joins `right`, `left`, and `stylus`. The hint stays OptOut while disconnected, as now.
|
||||
- `ft_pointer_vrcompositor.json` and `ft_pointer_steam.json` get the same bindings under `/user/treadmill`. They're additive, so nothing changes while the device is a hand.
|
||||
|
||||
### Pointer helper (`pointer/helper/ft-pointer.cpp`)
|
||||
|
||||
- Gaze first = gaze mode on, the gaze service ready, no game, or a game with the macro's override.
|
||||
- Waking: in gaze first, fresh gaze with the headset worn wakes the pointer (connect, treadmill role, `switchlaserhand`), with no mouse counts. The "no hand role" release doesn't apply to the treadmill role.
|
||||
- The laser: while awake in gaze first, if the dashboard's primary device becomes a controller, press `/input/a` on ours again, at most every 100 ms. Last used wins stays off in gaze mode, as now.
|
||||
- Muting: the trigger and bumper action sets on both sides are active whenever gaze first is on, whatever the relay's mappings say. Their presses go to the built-in state machine, not to the relay.
|
||||
- Trigger and bumper: a hold with a new source, a controller's position. It starts as a held-back press at the gaze; moving past `POINTER_TRIGGER_DEADZONE` (about 1 degree, seen from the eye; the pull jolts the controller) within `POINTER_GAZE_HOLD` is precision at `POINTER_TRIGGER_GAIN` (0.5); still until `POINTER_GAZE_HOLD` is a real press, and dragging at `POINTER_GAZE_DRAG_GAIN` (1). Release: click, lesson (as with the mouse, up to `POINTER_GAZE_NUDGE_MAX`), or release the press. The existing controller holds (`gaze_precision`, `gaze_drag`) move to position steering too.
|
||||
- Scroll: "scroll <dx> <dy>" from the relay goes to the driver, as the mouse's wheel does.
|
||||
- Calibration: while ft-gazed says its panel is up ("calpanel 1|0"), the pointer hides, and a trigger press goes to ft-gazed as "calaccept" instead of clicking.
|
||||
- The macro's "cancel": drops a held-back press without clicking.
|
||||
- Headset worn or not goes to ft-gazed ("headset 1|0"), for the quick check.
|
||||
|
||||
### Input relay (`input/input-relay.py`)
|
||||
|
||||
- A web socket reader for vrserver, in the standard library (the host's Python has no `websockets` module, and the relay needs nothing but Python). It follows the controllers by `side`, since their paths change with the roles.
|
||||
- The toggle macro: `"gaze_macro": {"buttons": ["left/thumbstick", "right/thumbstick"], "hold": 1.0}` in the rules file. Outside games it writes `POINTER_GAZE` and tells the helper; in a game it toggles the override for that game. It sends the helper "cancel" when it fires.
|
||||
- Recording: Input Settings asks for "macro record" and gets back the chord and how long it was held, after the checks in "Decisions".
|
||||
- Scroll: the right thumbstick's axes, while gaze first is on, as "scroll" lines to the helper.
|
||||
- Games: a Steam game running (a `reaper` process with `SteamLaunch AppId=`, checked every 2 s; to confirm with a flatscreen game) goes to the helper, which already knows scene applications.
|
||||
- `gaze_toggle` writes `POINTER_GAZE` instead of lasting until a restart.
|
||||
|
||||
### Gaze service (`gaze/ft-gazed`)
|
||||
|
||||
- `GAZE_TRACKER` defaults to `own`.
|
||||
- Readiness goes to the helper ("gazeready 1|0"): the tracker's service works (for ours: `frametop-eyegrab` active, and its binary the same as the build) and the tracker in use has a calibration. When gaze is on but not ready, it opens the calibrator, or asks for the repair.
|
||||
- Our tracker runs only while gaze is on and ready, while a calibration runs, or for the probe's lease.
|
||||
- The quick check opens on "headset 1" from the helper, on our tracker's jump (at most once every 2 minutes), and on "quickcal" (the mappable action). For our tracker, the dot is a click on `@ft_eyes`, like the probe's one-dot check; for SteamVR's, it's a lesson.
|
||||
- The full calibration's logic (the dots, rounds, sample rejection, and fits) moves out of the probe into `gazecal.py`, so the probe and the service share it. Quitting writes `POINTER_GAZE=0`.
|
||||
- Hiding Frametop's screens during a full calibration goes through ft-screens' `hide` and `show`, keeping the user's own switch as it was.
|
||||
|
||||
### Calibration panel (`gaze/panel/ft-gazepanel`, new)
|
||||
|
||||
A small C++ OpenVR overlay program in the dev container: a head-locked overlay (placed relative to the headset) of a fixed angular size, drawn on the CPU and uploaded with `SetOverlayRaw`, like ft-screens' keyboard, with labels from stb_truetype. ft-gazed drives it over `@ft_gazepanel` (show quick or full, dot at yaw and pitch with its state, background brightness, a line of text, hide). It takes no input: the trigger comes through the helper, and the dwell is ft-gazed's. It's a separate program because the helper is already 2,000 lines, and the panel has nothing to do with pointing.
|
||||
|
||||
### Frametop Input Settings, Gaze page
|
||||
|
||||
- The gaze switch, blocked with the reason while not ready ("Calibrate first", "Repair eye tracker").
|
||||
- Eye tracker: ours (default) or SteamVR's.
|
||||
- Status: the service, the calibration, and when it was made. Buttons for Calibrate, Quick check, and Repair or Update eye tracker (`pkexec gaze/tracker/install.sh`, which then runs without sudo inside).
|
||||
- The toggle macro: what it is, Record (a 3 s countdown, hold the chord, confirm), and Reset to default.
|
||||
|
||||
### Installer (`install.sh`)
|
||||
|
||||
- A step for our eye tracker ("It needs your password (sudo)"), defaulting to yes, and the gaze service, which needs no root. Gaze itself stays off.
|
||||
|
||||
## Order of work
|
||||
|
||||
1. The tests above, then this plan updated with the results.
|
||||
2. Driver and helper: the treadmill role, muting, snap back, waking by gaze, and the trigger and bumper holds. Gaze still turns on the old way.
|
||||
3. Relay: the web socket reader, scroll, the macro, games and the override, `POINTER_GAZE` remembered.
|
||||
4. Gaze service: readiness, idling, the default tracker, the service check, and the quick check's triggers.
|
||||
5. The calibration panel and calibrator: the quick check, then the full calibration and the move to 5 dots.
|
||||
6. Input Settings, the installer, and the docs (`README.md`, `docs/design.md`, `docs/reference.md`, `gaze/README.md`).
|
||||
|
||||
Each step is tested in the headset before it's merged into `experimental`.
|
||||
|
||||
## Risks
|
||||
|
||||
- SteamVR may refuse the laser on a treadmill device, or a SteamVR update may change the Frame's compositor bindings.
|
||||
- The muting relies on "Enable global input from overlays" (`steamvr/globalActionSetPriority`), which SteamVR calls experimental.
|
||||
- `pkexec` runs a script the user can write. That's acceptable only because every run asks for the password.
|
||||
- Our tracker's CPU cost while gaze is on, with SteamVR and a busy desktop; it idles otherwise.
|
||||
- Head-locked panels can be uncomfortable; the panel stays small and short-lived, except for the full calibration.
|
||||
Executable
+195
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/python3
|
||||
"""vrws: vrserver's local web socket, with nothing but the standard library.
|
||||
|
||||
vrserver (SteamVR) serves its controller binding page on 127.0.0.1:27062, and that page
|
||||
follows the controllers' raw input through a web socket there. Reading it takes nothing
|
||||
from anyone: whatever the buttons are bound to still happens, and it works whatever app
|
||||
has focus, in games too. frame-voice reads its push-to-talk button the same way. The host's
|
||||
Python has no `websockets` module, so this is a small RFC 6455 client of its own.
|
||||
|
||||
getstate() the devices SteamVR has now, from /input/getstate.json: each with its
|
||||
root path (/user/hand/left, /devices/cv/<serial> while our pointer holds
|
||||
that hand, /user/head), controller type, side, and input components
|
||||
VrSocket() a connection: open(mailbox), subscribe(device), recv() -> dict or None
|
||||
|
||||
Run as a program, it prints component changes as they come, for the gaze-first tests
|
||||
(docs/gaze-first.md, test 4):
|
||||
|
||||
input/vrws.py [--all] [--seconds N] [component ...]
|
||||
|
||||
By default it follows the buttons (…/click), the thumbsticks' axes, the trigger's value,
|
||||
and the headset's proximity; --all shows every component. At the end it prints how often
|
||||
each one updated.
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import select
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
HOST, PORT = "127.0.0.1", 27062
|
||||
ORIGIN = f"http://{HOST}:{PORT}"
|
||||
HEADERS = {"Referer": f"{ORIGIN}/dashboard/controllerbinding.html"}
|
||||
|
||||
|
||||
def getstate(timeout=3):
|
||||
"""The devices SteamVR has now (only the ones with a root path)."""
|
||||
req = urllib.request.Request(f"{ORIGIN}/input/getstate.json", headers=HEADERS)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
return [d for d in json.load(resp).get("devices", []) if d.get("root_path")]
|
||||
|
||||
|
||||
class VrSocket:
|
||||
def __init__(self, timeout=3):
|
||||
self.sock = socket.create_connection((HOST, PORT), timeout=timeout)
|
||||
key = base64.b64encode(os.urandom(16)).decode()
|
||||
self.sock.sendall((f"GET / HTTP/1.1\r\nHost: {HOST}:{PORT}\r\nUpgrade: websocket\r\n"
|
||||
f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n"
|
||||
f"Origin: {ORIGIN}\r\nReferer: {HEADERS['Referer']}\r\n\r\n").encode())
|
||||
head = b""
|
||||
while b"\r\n\r\n" not in head:
|
||||
chunk = self.sock.recv(4096)
|
||||
if not chunk:
|
||||
raise OSError("vrserver closed the connection during the handshake")
|
||||
head += chunk
|
||||
head, self.buf = head.split(b"\r\n\r\n", 1)
|
||||
status = head.split(b"\r\n", 1)[0]
|
||||
if b" 101 " not in status + b" ":
|
||||
raise OSError(f"vrserver refused the web socket: {status.decode(errors='replace')}")
|
||||
self.sock.settimeout(None)
|
||||
self.parts = b""
|
||||
|
||||
def fileno(self):
|
||||
return self.sock.fileno()
|
||||
|
||||
def close(self):
|
||||
try:
|
||||
self._frame(0x8, b"")
|
||||
except OSError:
|
||||
pass
|
||||
self.sock.close()
|
||||
|
||||
def _frame(self, opcode, payload):
|
||||
n = len(payload)
|
||||
head = bytes([0x80 | opcode])
|
||||
if n < 126:
|
||||
head += bytes([0x80 | n])
|
||||
elif n < 65536:
|
||||
head += bytes([0x80 | 126]) + struct.pack(">H", n)
|
||||
else:
|
||||
head += bytes([0x80 | 127]) + struct.pack(">Q", n)
|
||||
mask = os.urandom(4)
|
||||
self.sock.sendall(head + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(payload)))
|
||||
|
||||
def send(self, text):
|
||||
self._frame(0x1, text.encode())
|
||||
|
||||
def open(self, mailbox):
|
||||
self.send(f"mailbox_open {mailbox}")
|
||||
self.mailbox = mailbox
|
||||
|
||||
def subscribe(self, device, on=True):
|
||||
kind = "request_input_state_updates" if on else "cancel_input_state_updates"
|
||||
self.send("mailbox_send input_server " +
|
||||
json.dumps({"type": kind, "device_path": device, "returnAddress": self.mailbox}))
|
||||
|
||||
def _read(self, n):
|
||||
while len(self.buf) < n:
|
||||
chunk = self.sock.recv(65536)
|
||||
if not chunk:
|
||||
raise OSError("vrserver closed the web socket")
|
||||
self.buf += chunk
|
||||
out, self.buf = self.buf[:n], self.buf[n:]
|
||||
return out
|
||||
|
||||
def recv(self, timeout=None):
|
||||
"""The next text message as parsed JSON (None for one that isn't JSON), or None at
|
||||
the timeout. Raises OSError when the connection ends."""
|
||||
if not self.buf and timeout is not None:
|
||||
if not select.select([self.sock], [], [], timeout)[0]:
|
||||
return None
|
||||
while True:
|
||||
b0, b1 = self._read(2)
|
||||
opcode, n = b0 & 0x0F, b1 & 0x7F
|
||||
if n == 126:
|
||||
n = struct.unpack(">H", self._read(2))[0]
|
||||
elif n == 127:
|
||||
n = struct.unpack(">Q", self._read(8))[0]
|
||||
mask = self._read(4) if b1 & 0x80 else None
|
||||
data = self._read(n)
|
||||
if mask:
|
||||
data = bytes(b ^ mask[i % 4] for i, b in enumerate(data))
|
||||
if opcode == 0x8:
|
||||
raise OSError("vrserver closed the web socket")
|
||||
if opcode == 0x9:
|
||||
self._frame(0xA, data)
|
||||
continue
|
||||
if opcode in (0x1, 0x2, 0x0):
|
||||
self.parts += data
|
||||
if not b0 & 0x80:
|
||||
continue
|
||||
text, self.parts = self.parts, b""
|
||||
try:
|
||||
return json.loads(text)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
args = [a for a in sys.argv[1:]]
|
||||
show_all = "--all" in args
|
||||
seconds = 0.0
|
||||
if "--seconds" in args:
|
||||
i = args.index("--seconds")
|
||||
seconds = float(args[i + 1])
|
||||
del args[i:i + 2]
|
||||
wanted = [a for a in args if not a.startswith("--")]
|
||||
|
||||
def followed(name):
|
||||
if show_all:
|
||||
return True
|
||||
if wanted:
|
||||
return name in wanted
|
||||
return (name.endswith("/click") or name in ("/input/thumbstick/x", "/input/thumbstick/y",
|
||||
"/input/trigger/value", "/proximity"))
|
||||
|
||||
devices = getstate()
|
||||
names = {}
|
||||
for d in devices:
|
||||
names[d["root_path"]] = f"{d['root_path']} ({d.get('controller_type', '?')}{', ' + d['side'] if d.get('side') else ''})"
|
||||
print("device", names[d["root_path"]])
|
||||
ws = VrSocket()
|
||||
ws.open(f"frametop_vrws_{os.getpid()}")
|
||||
for d in devices:
|
||||
ws.subscribe(d["root_path"])
|
||||
start = time.monotonic()
|
||||
last, counts = {}, {}
|
||||
try:
|
||||
while not seconds or time.monotonic() - start < seconds:
|
||||
msg = ws.recv(timeout=0.5)
|
||||
if not isinstance(msg, dict) or msg.get("type") != "update_component_states":
|
||||
continue
|
||||
dev = msg.get("device")
|
||||
for name, value in (msg.get("components") or {}).items():
|
||||
if not followed(name):
|
||||
continue
|
||||
key = (dev, name)
|
||||
counts[key] = counts.get(key, 0) + 1
|
||||
if last.get(key) != value:
|
||||
last[key] = value
|
||||
print(f"{time.monotonic() - start:9.3f} {names.get(dev, dev)} {name} = {value}", flush=True)
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
ws.close()
|
||||
took = max(time.monotonic() - start, 1e-6)
|
||||
for (dev, name), n in sorted(counts.items()):
|
||||
print(f"updates: {names.get(dev, dev)} {name}: {n} ({n / took:.1f}/s)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -17,11 +17,12 @@
|
||||
// btn <name> <0|1> name: trigger, b, x, system, joystick, a (a = claim the laser, no click)
|
||||
// scroll <x> <y> joystick deflection -1..1
|
||||
// show | hide connect (take the hand role) or disconnect (give it back)
|
||||
// role right|left|stylus which role to hint while connected, from now on (the helper
|
||||
// sends POINTER_ROLE). A Frame controller in your hand takes
|
||||
// its hand's role back (it counts as used while held), so a
|
||||
// role right|left|stylus|treadmill which role to hint while connected, from now on (the
|
||||
// helper sends POINTER_ROLE). A Frame controller in your hand
|
||||
// takes its hand's role back (it counts as used while held), so a
|
||||
// controller used beside the pointer needs the pointer on the
|
||||
// other hand, or on no hand at all
|
||||
// other hand, or on no hand at all. Treadmill is no hand: its
|
||||
// bindings are under /user/treadmill (docs/gaze-first.md, test 1)
|
||||
//
|
||||
// The device starts disconnected, so it never holds a hand role at boot (holding
|
||||
// the right hand while SteamVR started left the Steam UI stuck loading). It
|
||||
@@ -30,7 +31,8 @@
|
||||
// SteamVR keeps a hand role reserved for a disconnected device that still hints
|
||||
// that hand, so the real controller would never get it back otherwise.
|
||||
//
|
||||
// Settings (steamvr.vrsettings section "driver_ft_pointer"): role (int, 2 = right hand, 5 = stylus).
|
||||
// Settings (steamvr.vrsettings section "driver_ft_pointer"): role (int, 2 = right hand, 4 = treadmill,
|
||||
// 5 = stylus).
|
||||
#include <openvr_driver.h>
|
||||
|
||||
#include <atomic>
|
||||
@@ -343,7 +345,8 @@ private:
|
||||
state_.role = !std::strcmp(name, "left") ? TrackedControllerRole_LeftHand
|
||||
: !std::strcmp(name, "right") ? TrackedControllerRole_RightHand
|
||||
: !std::strcmp(name, "stylus") ? TrackedControllerRole_Stylus
|
||||
: state_.role;
|
||||
: !std::strcmp(name, "treadmill") ? TrackedControllerRole_Treadmill
|
||||
: state_.role;
|
||||
} else if (std::sscanf(cmd, "btn %31s %d", name, &v) == 2) {
|
||||
for (int i = 0; i < kButtons; ++i)
|
||||
if (std::strcmp(name, kButtonNames[i]) == 0) state_.buttons[i] = v != 0;
|
||||
|
||||
@@ -82,6 +82,42 @@
|
||||
"output": "/actions/lasermouse/in/back"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/trigger",
|
||||
"mode": "button",
|
||||
"inputs": {
|
||||
"click": {
|
||||
"output": "/actions/lasermouse/in/leftclick"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/b",
|
||||
"mode": "button",
|
||||
"inputs": {
|
||||
"click": {
|
||||
"output": "/actions/lasermouse/in/rightclick"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/x",
|
||||
"mode": "button",
|
||||
"inputs": {
|
||||
"click": {
|
||||
"output": "/actions/lasermouse/in/middleclick"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/joystick",
|
||||
"mode": "button",
|
||||
"inputs": {
|
||||
"click": {
|
||||
"output": "/actions/lasermouse/in/back"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"poses": [
|
||||
@@ -92,6 +128,10 @@
|
||||
{
|
||||
"output": "/actions/lasermouse/in/Pointer",
|
||||
"path": "/user/hand/right/pose/raw"
|
||||
},
|
||||
{
|
||||
"output": "/actions/lasermouse/in/Pointer",
|
||||
"path": "/user/treadmill/pose/raw"
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -114,6 +154,15 @@
|
||||
"output": "/actions/scroll_discrete/in/scroll"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/joystick",
|
||||
"mode": "scroll",
|
||||
"inputs": {
|
||||
"scroll": {
|
||||
"output": "/actions/scroll_discrete/in/scroll"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -136,6 +185,15 @@
|
||||
"output": "/actions/scroll_smooth/in/scroll"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/joystick",
|
||||
"mode": "scroll",
|
||||
"inputs": {
|
||||
"scroll": {
|
||||
"output": "/actions/scroll_smooth/in/scroll"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -158,6 +216,15 @@
|
||||
"output": "/actions/system/in/ToggleDashboard"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/system",
|
||||
"mode": "button",
|
||||
"inputs": {
|
||||
"click": {
|
||||
"output": "/actions/system/in/ToggleDashboard"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -198,6 +265,24 @@
|
||||
"output": "/actions/lasermouse_secondary/in/switchlaserhand"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/trigger",
|
||||
"mode": "button",
|
||||
"inputs": {
|
||||
"click": {
|
||||
"output": "/actions/lasermouse_secondary/in/switchlaserhand"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "/user/treadmill/input/a",
|
||||
"mode": "button",
|
||||
"inputs": {
|
||||
"click": {
|
||||
"output": "/actions/lasermouse_secondary/in/switchlaserhand"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -177,10 +177,11 @@
|
||||
// precision, the default), or clicks at once where the pointer is (direct). And in gaze mode a
|
||||
// moving controller doesn't take the pointer away (last used wins is off): the gaze points,
|
||||
// and the controllers are its tools.
|
||||
// POINTER_ROLE (right, left, or stylus): the hand role our device takes while connected. A
|
||||
// POINTER_ROLE (right, left, stylus, or treadmill): the role our device takes while connected. A
|
||||
// Frame controller in your hand counts as used through its touch sensors and takes its hand's
|
||||
// role back, and then no click lands (see "no hand role" in the main loop): with a controller in
|
||||
// the right hand as the precision tool, the pointer needs the left hand, or the stylus role.
|
||||
// the right hand as the precision tool, the pointer needs the left hand, or a role that's no
|
||||
// hand. Treadmill is being tested for that (docs/gaze-first.md, test 1).
|
||||
//
|
||||
// Hands (POINTER_HANDS, off by default; needs hand tracking, hands/): ft-hands publishes
|
||||
// pinches and grips (hands/include/fh_gestures.h), read here every frame.
|
||||
@@ -713,7 +714,7 @@ int main() {
|
||||
const auto gm = conf.find("POINTER_GAZE_MOUSE");
|
||||
gazeMousePrecision = gm == conf.end() || gm->second != "direct";
|
||||
const auto ro = conf.find("POINTER_ROLE");
|
||||
role = ro != conf.end() && (ro->second == "left" || ro->second == "stylus") ? ro->second : "right";
|
||||
role = ro != conf.end() && (ro->second == "left" || ro->second == "stylus" || ro->second == "treadmill") ? ro->second : "right";
|
||||
pickupScale = std::clamp(ConfDouble(conf, "POINTER_CONTROLLER_PICKUP", 1), 0.5, 5.0);
|
||||
const auto ig = conf.find("POINTER_IGNORE");
|
||||
ignore = ParseIgnore(ig == conf.end() ? "" : ig->second);
|
||||
|
||||
@@ -6,4 +6,6 @@ root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
|
||||
exec "$root/scripts/frame.sh" -C pointer/probe 'set -e; mkdir -p build
|
||||
g++ -std=c++17 -O2 -Wall -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers -I../common \
|
||||
-o build/vrprobe vrprobe.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64
|
||||
echo "built build/vrprobe"'
|
||||
g++ -std=c++17 -O2 -Wall -I/opt/steamvr/tools/hellovr_vulkan_linux/src/openvr/headers \
|
||||
-o build/lasertest lasertest.cpp -L/opt/steamvr/bin/linuxarm64 -lopenvr_api -Wl,-rpath,/opt/steamvr/bin/linuxarm64
|
||||
echo "built build/vrprobe build/lasertest"'
|
||||
@@ -0,0 +1,136 @@
|
||||
// Watch who has SteamVR's laser mouse, for the gaze-first tests (docs/gaze-first.md, tests 1-3).
|
||||
// Prints the dashboard's primary device and every controller's role whenever either changes,
|
||||
// and the dashboard and role events. With --snapback, when the laser goes to any device but
|
||||
// ours (the ft_pointer driver's), it presses our /input/a (switchlaserhand) to take it back,
|
||||
// at most every 100 ms, and prints how long that took.
|
||||
// Runs as a background OpenVR client (in the dev container).
|
||||
// Usage: lasertest [--snapback] [--seconds N]
|
||||
#include <openvr.h>
|
||||
|
||||
#include <sys/socket.h>
|
||||
#include <sys/un.h>
|
||||
|
||||
#include <chrono>
|
||||
#include <csignal>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
|
||||
using Clock = std::chrono::steady_clock;
|
||||
|
||||
static volatile std::sig_atomic_t g_stop = 0;
|
||||
|
||||
static void Send(const char *msg) {
|
||||
static int fd = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0);
|
||||
sockaddr_un a{};
|
||||
a.sun_family = AF_UNIX;
|
||||
const char name[] = "ft_pointer"; // the driver's abstract socket
|
||||
std::memcpy(a.sun_path + 1, name, sizeof name - 1);
|
||||
sendto(fd, msg, std::strlen(msg), 0, reinterpret_cast<sockaddr *>(&a),
|
||||
socklen_t(offsetof(sockaddr_un, sun_path) + 1 + sizeof name - 1));
|
||||
}
|
||||
|
||||
static std::string Describe(vr::IVRSystem *sys, vr::TrackedDeviceIndex_t i) {
|
||||
if (i == vr::k_unTrackedDeviceIndexInvalid) return "none";
|
||||
static const char *roles[] = {"none", "left", "right", "optout", "treadmill", "stylus"};
|
||||
char type[64] = "", serial[64] = "";
|
||||
sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type);
|
||||
sys->GetStringTrackedDeviceProperty(i, vr::Prop_SerialNumber_String, serial, sizeof serial);
|
||||
const auto role = sys->GetControllerRoleForTrackedDeviceIndex(i);
|
||||
const int hint = sys->GetInt32TrackedDeviceProperty(i, vr::Prop_ControllerRoleHint_Int32);
|
||||
char out[200];
|
||||
std::snprintf(out, sizeof out, "%u %s %s role=%s hint=%d", i, type, serial, role < 6 ? roles[role] : "?", hint);
|
||||
return out;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
bool snapback = false;
|
||||
double seconds = 0;
|
||||
for (int i = 1; i < argc; ++i) {
|
||||
if (!std::strcmp(argv[i], "--snapback")) snapback = true;
|
||||
else if (!std::strcmp(argv[i], "--seconds") && i + 1 < argc) seconds = std::atof(argv[++i]);
|
||||
else {
|
||||
std::fprintf(stderr, "usage: %s [--snapback] [--seconds N]\n", argv[0]);
|
||||
return 2;
|
||||
}
|
||||
}
|
||||
std::signal(SIGINT, [](int) { g_stop = 1; });
|
||||
std::signal(SIGTERM, [](int) { g_stop = 1; });
|
||||
vr::EVRInitError err = vr::VRInitError_None;
|
||||
vr::IVRSystem *sys = vr::VR_Init(&err, vr::VRApplication_Background);
|
||||
if (err != vr::VRInitError_None) {
|
||||
std::printf("VR_Init failed: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
|
||||
return 1;
|
||||
}
|
||||
const auto start = Clock::now();
|
||||
auto ms = [&] { return std::chrono::duration<double, std::milli>(Clock::now() - start).count(); };
|
||||
std::string lastPrimary, lastRoles;
|
||||
Clock::time_point lostAt{}, pressedAt{};
|
||||
bool pressed = false;
|
||||
while (!g_stop && (seconds <= 0 || ms() < seconds * 1000)) {
|
||||
vr::VREvent_t ev;
|
||||
while (sys->PollNextEvent(&ev, sizeof ev)) {
|
||||
const char *name = sys->GetEventTypeNameFromEnum(vr::EVREventType(ev.eventType));
|
||||
switch (ev.eventType) {
|
||||
case vr::VREvent_DashboardActivated:
|
||||
case vr::VREvent_DashboardDeactivated:
|
||||
case vr::VREvent_TrackedDeviceRoleChanged:
|
||||
case vr::VREvent_TrackedDeviceActivated:
|
||||
case vr::VREvent_TrackedDeviceDeactivated:
|
||||
case vr::VREvent_ButtonPress:
|
||||
case vr::VREvent_ButtonUnpress:
|
||||
std::printf("%9.1f ms event %s device %u button %u\n", ms(), name, ev.trackedDeviceIndex,
|
||||
ev.data.controller.button);
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
}
|
||||
vr::TrackedDeviceIndex_t ours = vr::k_unTrackedDeviceIndexInvalid;
|
||||
std::string roles;
|
||||
for (vr::TrackedDeviceIndex_t i = 1; i < vr::k_unMaxTrackedDeviceCount; ++i) {
|
||||
if (sys->GetTrackedDeviceClass(i) != vr::TrackedDeviceClass_Controller) continue;
|
||||
char type[64] = "";
|
||||
sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type);
|
||||
if (!std::strcmp(type, "ft_pointer")) ours = i;
|
||||
roles += " [" + Describe(sys, i) + " connected=" + std::to_string(sys->IsTrackedDeviceConnected(i)) + "]";
|
||||
}
|
||||
if (roles != lastRoles) {
|
||||
std::printf("%9.1f ms controllers:%s\n", ms(), roles.c_str());
|
||||
lastRoles = roles;
|
||||
}
|
||||
const auto primary = vr::VROverlay()->GetPrimaryDashboardDevice();
|
||||
const std::string p = Describe(sys, primary);
|
||||
if (p != lastPrimary) {
|
||||
std::printf("%9.1f ms laser: %s%s\n", ms(), p.c_str(), primary == ours ? " (ours)" : "");
|
||||
if (primary == ours && lostAt != Clock::time_point{}) {
|
||||
std::printf("%9.1f ms back on ours after %.0f ms\n", ms(),
|
||||
std::chrono::duration<double, std::milli>(Clock::now() - lostAt).count());
|
||||
lostAt = {};
|
||||
}
|
||||
if (primary != ours && ours != vr::k_unTrackedDeviceIndexInvalid && primary != vr::k_unTrackedDeviceIndexInvalid)
|
||||
lostAt = Clock::now();
|
||||
lastPrimary = p;
|
||||
}
|
||||
if (snapback) {
|
||||
const auto now = Clock::now();
|
||||
if (pressed && now - pressedAt > std::chrono::milliseconds(40)) {
|
||||
Send("btn a 0");
|
||||
pressed = false;
|
||||
}
|
||||
if (!pressed && lostAt != Clock::time_point{} && now - pressedAt > std::chrono::milliseconds(100)) {
|
||||
Send("btn a 1");
|
||||
pressed = true;
|
||||
pressedAt = now;
|
||||
std::printf("%9.1f ms snapback: pressed our a\n", ms());
|
||||
}
|
||||
}
|
||||
std::fflush(stdout);
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(10));
|
||||
}
|
||||
if (pressed) Send("btn a 0");
|
||||
vr::VR_Shutdown();
|
||||
return 0;
|
||||
}
|
||||
Reference in new issue
Block a user