mirror of
https://github.com/DeeJanuz/frametop.git
synced 2026-10-06 01:00:06 +02:00
* Input relay: typing with the pointer helper down no longer ends the relay Typing on a pass-through keyboard tells the helper "typing". With the helper not running (SteamVR off), that send raised ConnectionRefusedError and the relay exited, dropping every grab until systemd restarted it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ft-steam: open Steam's menu through Steam's own UI steam/ft-steam menu opens the SteamVR dashboard on Steam's menu, or closes the dashboard if it's up, without pointer mode: it asks Steam's UI over its debugging port to show its dashboard overlay (ShowVROverlay, what Steam calls itself) and focus the Steam frame's left menu (MenuStore.OpenMainMenu). ft-steam check says whether those calls still exist, and update-check.py runs it, since a Steam client update can rename them. The CDP client moves from display-settings/steam_settings.py to steam/steamui.py, so both use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Shortcuts: Steam menu, commands, and modifier taps Key combinations (and mouse and controller buttons) get two new actions: - steam_menu: Open Steam menu / close dashboard (steam/ft-steam menu). - command:CMD: run CMD with sh -c, as the relay's service, with layout/, float/ and steam/ on its PATH. Input Settings offers it for key combinations as Run a command... Both work without pointer mode. A modifier on its own is now a key combination too: a tap, pressed and released with no other key, mouse button, or scroll in between. A bound tap sends the desktop F24 before the release, so Plasma's launcher stays shut. The defaults gain a Meta tap for the Steam menu; this replaces META_DASHBOARD, which only worked in pointer mode. input/test/keys-test.py runs the relay against fake devices with every outgoing socket renamed, so it's safe next to the live relay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Relay: share a key combination's Meta release with frame-voice A Meta+key combination (Meta+J for gaze_left, say) hides Meta's release from the desktop, and the relay skipped share_key for it too. frame-voice saw Meta go down on @frametop_keys and never come up, so it held all dictated text back, waiting for that release. Keys of grabbed keyboards are now shared as pressed, before key_binding() decides what the desktop gets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: ft-cutouts, the hand cutouts without pinches and grips hands/ft-cutouts on|off|status starts ft-camd and ft-hands as transient user units with ft-hands' new --no-gestures: hands are published for ft-screens' cutouts, but no pinch or grip is detected, so nothing clicks or drags and a closing hand doesn't raise the tracking rate. It needs a build and ft-camd's capabilities, not hands/run.sh install. Its units conflict with ft-handsctl's, so each stops the other, and they stop with SteamVR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ft-cutouts status: only the current run's tracker lines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: say why gaze mode can't work yet, and open the calibration whenever it's missing Turning gaze mode on without a calibration opened Calibrate only on the off-to-on change, and only if it could open right then. With the headset off, the eye tracker silent, or the panel not built, or with gaze mode already on when the gaze service started, nothing opened and nothing said why: the pointer just stayed a mouse. - The gaze service now checks every second: gaze mode on, no calibration for the tracker in use, eyes seen -> the full calibration opens. One that closes unfinished opens again only after the headset comes off and on, gaze mode off and on, or Calibrate, so it doesn't loop. A start that fails retries every 10 s. - Its status says why gaze mode can't work yet (checks.problem): not calibrated and opening, open, closed unfinished, or can't open and why. - Input Settings shows that under the Gaze pointer switch, along with the gaze service not installed or not running and our tracker missing its frame grabber. - ft-gazectl on notes a missing calibration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: install our eye tracker, prefer it, and say why a calibration dot wasn't taken A user on a fresh install got "Calibration failed: only 0 of 21 dots" with no reason. The installer never installed our tracker, so gaze used SteamVR's, and the only way SteamVR's tracker rejects a dot is losing an eye for most of the look. The panel just showed a red ring. - install.sh: step 9/10 installs our tracker (gaze/tracker/install.sh) after gaze mode, yes by default; it needs sudo, so --yes runs it only when sudo won't prompt. If it fails, gaze keeps SteamVR's tracker. Configs that still say GAZE_TRACKER=steam (the old template) are asked whether to switch. - GAZE_TRACKER=auto, the new default: ours when it's installed (the frame grabber, its unit, and ft-eyes' Python), else SteamVR's. ft-gazed rechecks every second, so installing it switches over. Input Settings lists Own tracker first as recommended, and says how to install it when it's missing (checking the host's /etc through /run/host from the dev container). - The calibration panel has a note line, orange over the instructions: why a dot wasn't taken (an eye lost, a blink, the eyes disagreeing for SteamVR's tracker, from steady_samples' new drop counts; ft-eyes' reply for ours), what a click is still waiting for after 1.5 s, and a failed calibration's most common reason, which the Gaze page shows too. steady_samples keeps the same samples as before (checked on 2037 windows of recordings); a lost eye is named before a blink, since its openness reads 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * README: link the Frametop Discord Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wait for a new container to finish setting up before entering it container-up.sh starts the dev container in a scope of its own, so distrobox enter finds it running and skips its wait for distrobox-init. On a fresh install, init was still setting up passwordless sudo when dev-container.sh ran sudo dnf install, and sudo asked for a password with no terminal to read it from. container-up.sh now waits for container_setup_done itself, and the container's sudo calls use -n, so a password prompt fails at once with a clear message. Fixes #9 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Prevent small desktop overlay pointer movements from starting a drag * Clear reported drag state on controller release * Click stability: only a hand controller's press starts it The 3D mouse drives SteamVR's laser through the ft_pointer virtual controller, so its events reach the screens the same way a controller's do. The filter held every press, which turned the mouse's short drags (selecting a character or two, nudging a slider) into clicks. Mark button events from hand controllers and start the filter only on those. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Input relay: retry a new device until udev gives it to the input group A new /dev/input node is root:root 0600 until udev applies GROUP=input. The scan probed each new node once and marked it seen even when the open failed, so a node caught in that gap was never opened. Behind a KVM, a switch brings back a hub of devices at once: on the Frame, four nodes failed with EACCES in one switch, the keyboard was never grabbed, and its keys went to gamescope instead of the desktop screens. A node that isn't readable yet now waits for the next scan. * Screens: take a screen's overlays from one copy in the catcher While a button pressed on a screen is held, UpdateCatcher checks every tick whether the laser is still on one of the screen's overlays. It built that list from s.All().begin() and s.All().end(), but All() returns a std::array by value: iterators into two different temporaries, which is undefined behaviour. A clang build of ft-screens got a garbage length, threw std::length_error, and aborted on the first click, taking KWin and the desktop with it. * Gaze: leave SteamVR's gaze action alone during VR games From curiousjtuber's PR #13: with the gaze service running, SteamVR restarted its eye tracker every 10 to 13 s of Beat Saber, as if the headset came off, and each restart took input focus from the game. The PR stopped every read in a game. Only the action path reaches SteamVR (UpdateActionState on the gaze set at overlay-global priority, then GetEyeTrackingDataRelativeToNow); the mmap and our tracker are read-only files. So only the action is skipped while a scene app runs, and gaze keeps moving the pointer over the dashboard in a game. The action source is only used with --source action. Co-Authored-By: CuriousJ <curious.j.tuber@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: --record-hz, and the hand recorder's design (hands/rec/DESIGN.md) ft-hands --record-hz N records at most N frame sets a second, for the hand recorder (10). DESIGN.md lays out the recorder: the headset panel, the session runner and its script, the files, review and export, consent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: ft-handpanel, the hand recorder's headset panel A head-locked SteamVR overlay for the hand recorder (hands/rec/DESIGN.md): 1.2 m ahead, 12 degrees up, 36 degrees wide, drawn with stb_truetype into three shared DMA-BUFs as ft-gazepanel does. It shows the title, step, wrapped instruction, note, countdown, hand chips, near/far bar and a "Paused" cover, driven over @ft_handpanel. It also places the touch target, a 2 cm dot in its own overlay fixed in the room where the head was at the first command for that point, and logs head and controller poses to poses.jsonl at 250 Hz from a thread of its own. Both threads take one lock around OpenVR calls. --no-vr prints each picture's state to stdout (and --dump writes the pictures), for testing without a headset. hands/rec/build.sh builds it in the dev container. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the recorder's worn check goes by the panel's backlight, as frame-job does Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the hand recorder's session runner and guided script hands/rec/session.py runs a recording session from script.json: it starts ft-camd and a tracking ft-hands as transient units only if they aren't running, records each section as one take (ft-hands --record-only at 10 sets/s, a new sets-N.bin after each pause), drives ft-handpanel, and writes session.json, calibration.json (identifying fields removed), prompts.jsonl and take.json. Feedback comes from the live hands file and, in the controller sections, from the panel's device poll. It also runs from the command line (--dry-run, --speed, --ring, --no-start). hands/rec/script.json: 11 sections, about 9 minutes without the object and controller sections. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the hand recorder's window, review and export hands/rec/ft_handrec.py + main.qml (Kirigami, dev container; host launcher hands/rec/ft-handrec): consent (CONSENT.md, asked again when its version changes; profile.json with a random contributor id), the before-you-start checklist with the lighting and free-space checks, the session controls (Space pauses, Esc stops), review with a frame-set viewer that deletes ranges, takes and sessions, export with progress and cancel (warns while the headset is worn), and the upload page (UPLOAD.md, the huggingface-cli command; HF_DATASET is a placeholder). --dry-run runs sessions without processes, for testing. hands/rec/takes.py (standard library): indexes sets.bin and sets-N.bin without reading pixels, reads one set's cameras, keeps deleted ranges in take.json, and exports: deleted sets left out, zstd -10 -T2 at nice 19, manifest.json and SHA256SUMS, nothing left behind on cancel. CONSENT.md and UPLOAD.md are drafts pending a legal review; the window says contributions aren't open yet. The dev container gains zstd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: upload from the hand recorder's window, export checks, a rehearsal hands/rec/validate.py (standard library; Linux and Windows, Python 3.12+) checks an export before upload and when it's received: SHA256SUMS, an allow-list of files, the manifest's schema and keys, the consent version, a uuid4 contributor, no identifying fields in calibration.json or device.json, every sets.bin.zst decompressed to its end as a stream with each FHSET01 header checked against the manifest, jsonl lines, the total size. It decompresses with compression.zstd, zstandard or the zstd program. validate.py DIR [--json]. hands/rec/hub.py uploads an export with huggingface_hub, as a pull request to contributions/<contributor>/<session>: validate first, refuse a repeat of the same export, check the login (whoami) and access (auth_check), upload_folder(create_pr=True) with the manifest summary as the description, then record the PR under "uploads" in session.json. Errors are explained (terms not accepted, not found, 401/403, network). --dry-run makes no network calls. FT_HANDREC_DATASET overrides HF_DATASET (DeeJanuz/frametop-hands); while the texts are drafts a real upload needs FT_HANDREC_ALLOW_UPLOAD=1. The Upload page shows the login with "Check again" and how to run hf auth login in a terminal (the token never enters the window), then Upload with a phase, progress and Cancel (hub.py as a child process), the PR link, and a warning for an export uploaded before. The manual command stays as the fallback. ft-handrec --hub-dry-run. session.py also saves device.json: cv.cad_from_cal and head from /persist/device_config.json, the labeller's shape, nothing identifying; export copies it. Session ids with a -N suffix are accepted everywhere. hands/rec/rehearse.sh runs it all without the headset: ft-ringplay plays 30 s of a capture into a ring, session.py records a short test script with ft-handpanel --no-vr and a tracker, then export, validate and a dry-run upload (--repo ID uploads for real). It runs in one frame-job scope, deletes its data and stops its processes, also on Ctrl+C. hands/rec/tests/test_validate.py covers good and broken exports and hub.py without the network. The dev container gains python3-huggingface-hub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * README: Frametop doesn't work on the SteamOS beta yet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit072a294941) * Hands: step mode and pose pictures for the hand recorder The first real session moved on every 5 s with text only, too fast to follow. Each step now waits for Next (Space or the window's button), counts down 3-2-1 while recording, then holds. P pauses, R redoes a step, S skips a section; "Advance by itself" (--auto) keeps the old timed flow. Nothing records while a step waits: each step is its own recording part. The panel and the window show a picture of each pose (hands/rec/poses, generated by make_poses.py from a parametric hand, MIT) and a diagram of where to hold the hands and how far out. prompts.jsonl gains ready, wait and redo events; session.json gains mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the headset button as Next, clearer push steps The headset's right-side click button (KEY_SELECT on gpio-keys, read without a grab) now works the session: Next while a step waits, pause during a hold, resume while paused. With no mouse connected the hints lead with it. The push sections say plainly to push straight out from the headset and pull back, with a side-view picture of the head, the headset and the arrow, and the bar's ends read "At your chest" and "Arm out". The bar labels are sent as one field, so labels with spaces no longer split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ft-floatd: a launch for a missing app no longer kills the control socket Gio.DesktopAppInfo.new() returns NULL for a desktop file that doesn't exist, and PyGObject raises TypeError ("constructor returned NULL") rather than returning None, so launch()'s `if info is None` never ran. The exception escaped the control socket's GLib callback, GLib dropped the watch, and ft-floatd stopped answering everything: the float key, dock, Launch as Standalone, and profiles, until the desktop restarted. Found on the Frame (2026-10-02): a profile saved with RustDesk's Flatpak open records its window's app id, com.carriez.flutter_hbb, which has no desktop file (the Flatpak's is com.rustdesk.RustDesk). `ft-layout use` on that profile asked ft-floatd to launch it, and ft-floatd went silent. With this, that launch replies "error no app com.carriez.flutter_hbb" and the profile's other apps open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ft-floatd: profiles keep and relaunch Flatpak apps whose window names another app id An X11 window in a Flatpak can give KWin an app id with no desktop file: RustDesk's says com.carriez.flutter_hbb (its GTK application id), and the Flatpak's desktop file is com.rustdesk.RustDesk. A profile recorded that id, so it couldn't relaunch the app (PR #16 keeps that from killing ft-floatd's socket). And the window's pid is the sandbox's own, so a launched window matched neither by process nor by app id, and didn't float. desktop_name() finds the desktop file whose StartupWMClass names the window's class (or app id) when the app id has none. Capture records that name, a profile claims open windows by it, and a launch's window matches by it. Profiles saved before this keep the old id; save them again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: check the tracking cameras before recording, and watch for losing them After the headset wakes, XRService sometimes fails to load the colour module's VCINT FPGA image; then only the two side cameras run, without the IR light, and the tracker finds no hands. hands/camcheck.py reads XRService's log, the video nodes it holds and ft-camd's ring, and says ok, degraded or unknown. The recorder won't start while degraded (--ignore-cameras overrides it), offers a confirmed SteamVR restart, and stops the first hand-size step when the tracker sees no hand at all. ft-camwatch (unit file only, not enabled) follows the log, notifies, and with CAMWATCH_AUTO_RESTART=1 restarts SteamVR when the headset isn't worn and nothing else uses VR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: shorter recording sessions with pose sweeps The second real session took 16 minutes, half of it 36 still poses. Labels come from the auto-labeller, so what matters is variety, not clean holds. A sweep step shows a strip of pose pictures and lights one every 4 s while the hands move slowly near and far; each cue is a prompt event with "cue": true. The core session is now 15 steps, about 5 minutes recorded. The pose groups, the one-hand sweeps' groups and the cue order are shuffled per session, seeded from its id and saved in session.json. A quick round (--quick, or the checklist's choice) is about 2 minutes for extra lighting. Touch the dot has 6 dots, the push sections two heights. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Input relay test: let the fake devices past the udev permission check Since the relay leaves a node it can't read yet for the next scan (12f2e84, PR #12), it checks os.access first, and the test's fake /dev/input paths don't exist, so the relay never opened them and every key check failed. The fake os now says they're readable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pause Frametop for VR games Frametop kept using the headset during games: with gaze mode off, our eye tracker still took about 60% of a core, remote desktop about 2 cores while on, and KWin kept drawing hidden screens because ft-screens sent their frame callbacks at 90 Hz. Pausing gives that back, and resuming brings back only what pausing stopped. It's also a way to keep the gaze service and our eye tracker off during games, which PR #13 asked for. Paused (input/game_pause.py, run by the input relay): - frametop-gaze stops (ft-eyegrab then idles by itself), and hand tracking and remote desktop stop if they run - the desktop hides and slows down: ft-screens "pause on" hides every panel and sends KWin a frame callback once a second; or, with pause_desktop "close", the desktop closes and starts again on resume - the relay lets go of the 3D mouse, typing goes to Steam, and mapped buttons and key combinations do only pause_toggle, steam_menu and commands Toggled by both thumbsticks clicked together twice (configurable), read passively from vrserver's web socket (input/vrws.py) so it works in games and takes nothing from them; by the new pause_toggle action; by input/ft-pause; and, with pause_auto (default on), by a VR game starting and ending, which the pointer helper now reports ("vrgame 1|0"). Frametop Input Settings has a Games page for it. update-check.py checks the web socket, and doesn't count a paused gaze service as failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: ft-hands works out which side camera is which ft-camd tells the side cameras apart by XRService's buffer allocation order, which some XRService starts reverse; both of 2026-10-02's starts did, so the cutouts missed the hands. HANDS_SWAP_SIDES=auto (the default) has ft-hands vote from hands seen in both side cameras: the landmark rays meet in front of both cameras only under the right naming. While undecided it probes the exchanged naming with the landmark model. It decides in about 2 s of hands (right on all 7 recordings replayed), swaps the views in place, and publishes sides.json. 0 and 1 still force it, with a warning when the hands disagree. Recordings carry each part's naming and the session's decision; review, export, validate and ft-handreplay put the names right, and takes.py sides records a decision by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: idle while the gaze isn't used The gaze service ran ft-gaze and our own eye tracker all the time: with gaze mode off, ft-eyes still took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. Now ft-gaze and our tracker run only while gaze mode is on and someone wears the headset, while a check or the calibration is open or asked for, or under a "wake" lease, which the Gaze page of Frametop Input Settings renews while it's open. 30 s after the last use they stop, and the frame grabber idles with our tracker. - The pointer helper answers "gaze ? headset" with worn|away (SteamVR's activity level for the headset); an older helper answers it as before, and the service then goes by gaze mode alone. - A quick check, calibration, or fit check asked for while idle wakes the tracker and opens once it sends; the automatic calibration waits quietly while it starts. - Status has "awake" and "idle" (why), and the Gaze page shows it. - gaze/test/idle-test.py runs the service with a fake helper and ft-gaze, offline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * update-check: a still controller isn't a broken web socket vrserver sends a controller's state only when something on it changes, and one lying still or asleep may not even send its first one. The check subscribed to one controller and failed after 3 s of silence. It now subscribes to all, and silence after a good handshake is a skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the recorder measures the light itself The checklist page measures the light when it opens, starting ft-camd if nothing runs it (and stopping it on quit), instead of saying the cameras aren't running. The round's lighting defaults to what the cameras measure: daylight or indoor, from the mono cameras' ambient infrared. Lamps give off little infrared, so dim and normal rooms read alike; picking dim, room or daylight still overrides it. session.json gets source, measured and ambient_ir. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: the recorder's host commands run from the home folder host-spawn starts a host command in the caller's folder. Started from /tmp, the app's folder in the container is /run/host/tmp, which the host doesn't have, so starting ft-camd (and every other host command) exited 127. host_command now runs from home (env -C), and the launcher cds there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Input Settings: the Games page is Game optimization Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: push steps say to follow the hollow circle, not the blue dot The dot is the current tracker's distance guess, often wrong; the ring is where the hands should be. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Eye tracker: one thread for OpenCV and numpy Nothing called cv2.setNumThreads, so OpenCV kept a pool of one worker per core for pupil windows of 140 to 240 px. Live, its idle workers spun and yielded about 14,000 times a second each, about a quarter of a core, next to SteamVR's compositor. numpy's OpenBLAS also started 8 threads that never had work. eyes_pupil.py now sets OpenCV to one thread, and ft-eyes sets OPENBLAS_NUM_THREADS and OMP_NUM_THREADS to 1 before numpy loads (a value already in the environment wins). Replaying fit1 into a scratch share (ft-eyes-replay, 14 s measured, capped at one core with the replay): threads 13 -> 1, involuntary context switches 3,812/s -> 430/s, system time 6.9% -> 1.6% of a core. Under that cap the frames it kept up with went from 21-36 to 57-69 a second per eye. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pointer: read the overlay list every 20 s, not every second The helper ran `vrcmd --overlays` once a second while the pointer was awake, and in gaze mode the pointer never sleeps. Each run is a shell plus vrcmd, a new SteamVR client, about 26 to 30 ms of CPU, so about 3% of a core all the time. The list is now read every 20 seconds, and at once (at most once a second) when it may have changed: the pointer waking, the dashboard opening or closing or creating an overlay, the scene app changing, an "overlays" request, and a left click that hit nothing, which may be on a panel that came up since. The thread waits on a condition variable instead of waking every 100 ms, so it sleeps while paused. The main loop looks the keys up again as soon as a new list is in, rather than at its next 1 s tick. Overlays already on the list still show and hide within 50 ms, from the IsOverlayVisible poll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: ft-eyes and ft-gaze below SteamVR's priority ft-eyes and ft-gaze run in the dev container through distrobox, so they live in podman's libpod scope: frametop-gaze.service's limits never reach them, and they ran at nice 0 next to vrcompositor and vrserver, also at nice 0. - ft-eyes sets itself to nice 10 and SCHED_BATCH at start, before its threads. Batch turns off wakeup preemption, so a frame ft-eyes wakes up for can wait out a running compositor's turn; a few ms late costs the gaze little. - ft-gaze sets nice 5 before its threads start, but stays SCHED_OTHER: each sample goes on to the pointer, and batch would add the same wait to every one of them. - Both only ever lower their priority (a higher nice already set wins), and a failure is logged and ignored. Checked in the dev container: nice 0 -> 10, policy 0 -> 3 (SCHED_BATCH) without any capability. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pointer: sleep on the command socket while the pointer is off The main loop slept a fixed 8 ms, about 116 wakeups a second, whether the pointer was awake or not, and every second it looked up every overlay's handle and read a string property from all 64 device slots to find its own device. With the pointer off and hand gestures off, the loop now waits in poll() on its command socket for up to 250 ms, or 20 ms while mapped Frame controller buttons are being read (SteamVR input has no event to wait for). A mouse command ends the wait at once. The headset's activity level, the game check, and the "vrgame" and "gazeawake" repeats keep going at that pace. The 50 ms visibility poll and the 1 s handle lookups run only while the pointer is awake, and waking forces both. The device index is looked for only while it's unknown, and again after SteamVR activates or deactivates a device. The HMD pose history is kept only with hand gestures on, its one user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remote desktop: connect FreeRDP only while a VNC viewer is connected vnc-bridge.sh kept FreeRDP connected to krdpserver from the moment remote desktop started, so krdp captured and H.264-encoded every KWin redraw in software (openh264) with nobody watching: krdpserver 55-78% of a core, xfreerdp 16-27%, Xvnc 6-11%, with 0 clients on :5900. krdp 6.7 creates its screencast session per RDP connection and drops it when the connection closes, so krdpserver itself idles without one and stays up. The bridge now counts established connections to Xvnc's port with ss, starts FreeRDP when a viewer appears (the desktop shows about 3 s later; the VNC screen is black until then) and stops it 45 s after the last one leaves (VNC_IDLE_SEC). Xvnc has no client hook, so its log output, which it writes for every connection, wakes the bridge early; otherwise it looks every 5 s while idle (0.1% of a core measured, against 0.9% for ss once a second) and every second while FreeRDP runs. The layout check runs only while FreeRDP runs. While a viewer is connected the bridge sends "watch 15" to ft-screens (@ft_screens) at once and every 5 s, so screens at a reduced frame rate (out of view, headset on a stand) stream at full rate; it lapses by itself if the bridge dies, and an ft-screens without the command just answers an error. The window search after starting FreeRDP now ends when FreeRDP exits instead of polling for 30 s. krdp on 127.0.0.1 with a fresh password, VNC on the tailnet address with VncAuth, and remote-ctl.sh start/stop (pause and resume) are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remote desktop: read the layout only after it changes While FreeRDP ran, vnc-bridge.sh called ft-layout remote-view every 5 s, which scans all of /proc for plasmashell and runs kscreen-doctor -j: about 4.4% of a core for a layout that rarely changes. It now stats the two files the answer depends on, the nested KWin's ~/.config/frametop/kwinoutputconfig.json (positions, scales, primary) and ~/.config/frametop-layout.json (screen sizes), once a second while FreeRDP runs. After either changes it reads the layout every 2 s for 10 s, since KWin's outputs follow the file a few seconds later; otherwise once a minute, in case a change touched neither. With no VNC viewer connected nothing runs (previous commit). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Eye tracker: ft-eyes sleeps until the next frame is due ft-eyes looked for new frames about 1,000 times a second: each pass of its loop asked the control socket with a non-blocking recvfrom (a BlockingIOError nearly every time), read both cameras' counters, and slept 1 ms. The frames come every 11.1 ms per camera, and only as counters in ft-eyegrab's shared memory, so there's no fd to wait on. Now each pass ends in select() on the control socket, with a timeout until 2 ms before the next frame of either camera is due (from when its last one was seen), then every 1 ms until it comes. A command wakes it at once. A camera with no frame for 0.1 s (headset off, grabber idle) isn't waited for, and with both stopped it looks every 20 ms. Waiting for the frame grabber's file uses the same select, 0.2 s at a time, instead of sleeping through commands. A new frame is still seen within about 1 ms of when it lands. On a synthetic share at 90 Hz per camera, on a heavily loaded headset (load average 23, so ft-eyes rarely sat idle), its waits went from 178 to 81 a second; unloaded, the old loop's 1 ms sleeps add up to about 1,000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Screens: frame rates by attention, ticks in step with the display ft-screens gave KWin a frame callback for every committed screen on each tick, and the tick was an 11 ms timer set again after each run, so it slid through the display's frame and came about 85 times a second at 90 Hz: the desktop repeated a frame several times a second (judder in scrolling and video), and KWin drew every screen in one burst at a random point of vrcompositor's frame. Hidden screens got the same 90 Hz unless Frametop was paused for a game. - Ticks run on a timerfd at absolute times, once per display frame, 1 ms after the vsync (IVRSystem::GetTimeSinceLastVsync and the HMD's display frequency, read once a second), so KWin gets its callbacks early in the frame. Measured with --no-vr: 91 wakeups a second instead of about 85. On the Frame the vsync times SteamVR reports lie on a 90 Hz grid. - Each screen's callbacks come at a rate for how much of it you see (vr.cpp, UpdateAttention): every frame while focused (within 12 degrees of where your head points, a laser or the mouse on it in the last 1.5 s, carried, or typed on), 15 a second for the rest of what you see (within 60 degrees), and 1 a second when hidden, behind you, or paused. Levels rise at once and fall after 1.5 s (focused) or 0.5 s (in view). KWin draws a screen only after its callback and its apps wait for theirs, so this throttles the apps too. A screen where nothing changes costs nothing at any rate, as before. - A video in view keeps every frame: 8 commits in a row that each redraw 6% or more of the screen, at 10 a second or more, count as one (from the surface's buffer damage). - "rates F V H" / --rates set the three rates (default 0 15 1, 0 = every frame), "rates?" shows them and each screen's level, "watch S" gives everything full rate for S seconds for a remote viewer (vnc-bridge.sh renews it), and "phase MS" moves the ticks for tuning. - ft-screens' main thread runs at nice -5 after the session starts: SteamOS allows down to -8 once the soft RLIMIT_NICE is raised, and KWin waits on these ticks. It had spent nearly 3 times as long waiting to run as running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pointer: skip unchanged work while the pointer is awake Every frame (about 116 a second) the helper tested the cursor ray against every visible overlay twice with ComputeOverlayIntersection, set the dot's alpha, width, transform and visibility (five calls into SteamVR), and sent the driver a pose datagram, even with the mouse and the head still. Now a frame reuses the last collision result when the mouse, the anchor (1 mm) and the eye (5 mm) haven't moved and no overlay showed, hid, or changed handle. The passes still run at least every 100 ms, since overlays move on their own (a floating window's controls follow it), and always while dragging. The dots' setters go to SteamVR only when their value changes: the placement when the dot moved 0.2 mm or the eye 5 mm, which turns or resizes it by well under 1%, and the width on a 0.5% change. The plain pose goes to the driver only when the laser's origin moved 0.2 mm or its direction 0.04 deg (0.1 mm where it lands, 15 cm on), and at least every 100 ms; the driver keeps the last pose and reports it every frame. A tilt's pose, a placement, or waking sends the next one regardless. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Session: blur, background contrast and animations off by default The nested kwinrc had no [Plugins] group, so KWin ran its default blur and background contrast effects, and kdeglobals had no AnimationDurationFactor, so animations ran at full length. KWin renders through zink on Turnip, on the GPU vrcompositor needs, and blur re-renders what's behind every translucent panel and menu; each animation frame is another frame for KWin and ft-screens. Before KWin starts, the session script now writes [Plugins] blurEnabled=false and contrastEnabled=false to $XDG_CONFIG_HOME/kwinrc and [KDE] AnimationDurationFactor=0 to its kdeglobals, each only if the desktop's own file has no value for it. It does this once and records that in $XDG_CONFIG_HOME/frametoprc ([Defaults] effects=1), because System Settings deletes a key put back to its default: without the marker, turning blur back on wouldn't survive a restart. The ids blur and contrast are the built-in effects of KWin 6.2.5 on SteamOS (both enabled by default in its plugin metadata). Tested against a temporary XDG_CONFIG_HOME: fresh config, an existing blurEnabled=true kept, and a deleted key not rewritten. README and docs/reference.md say how to turn them back on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Session: don't autostart Discover's notifier or IBus in the desktop The nested Plasma session runs the system's XDG autostart entries. Discover's update notifier (/etc/xdg/autostart/org.kde.discover.notifier.desktop) started plasma-discover --mode update inside it, 520-620 MB resident and about 9% of a core, with flatpak-system-helper and AppStream downloads behind it. IBus started a nested ibus-daemon with kimpanel and ibus-extension-gtk3, which no app in the desktop can use: KWin's input method is ft-textinput (zwp_input_method_v1, focus reports only; the VR keyboard types through ft-screens' seat), and the session already drops QT_IM_MODULE, GTK_IM_MODULE and XMODIFIERS. Nothing in Frametop talks to IBus. Before Plasma starts, the session script copies both entries into $XDG_CONFIG_HOME/autostart with Hidden=true, which plasma-session honours for that desktop only. It does this once ([Defaults] autostart=1 in frametoprc) and skips a name the user already has a file for, so deleting the copy brings the program back. The geoclue demo agent stays (it answers apps' location requests outside GNOME and idles at 0%), and orca's entry is OnlyShowIn GNOME-family desktops, so it never ran. Tested against a temporary XDG_CONFIG_HOME, including an existing user ibus.desktop left alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Input relay: never block on the pointer helper's socket The relay sent to @ft_pointer_helper on a blocking socket. When the helper stalled, a layout placement or grabprobe holds it for seconds while ft-gazed keeps filling its socket at 90 Hz, the relay's one loop blocked with it: keyboards, the volume keys (which must never reach gamescope), and pausing all stopped until the helper read again. The socket is non-blocking now. A command the helper doesn't take (EAGAIN) waits in a queue, and everything after it queues behind it so the order holds; tick() sends what it can on each loop, and the select timeout drops to 20 ms while anything waits. Mouse moves add up into one queued move. A scroll notch is dropped rather than queued, since scrolling seconds late is no use; its release still goes. Presses, releases, show, hide, and the rest are kept, so no button stays down. The queue holds at most 512 commands. While paused, the configured pointer's queue still drains, so the releases and "hide" from standing down arrive. A "vrbind" that hits a full socket is sent again on the next loop instead of being lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pointer driver: parse outside the lock, report only changes Handle() held the state lock through a chain of up to a dozen sscanf calls per command, and RunFrame, which vrserver calls every frame, takes the same lock, so a burst of commands (about 116 poses a second, plus moves and buttons) could hold up vrserver's frame. Commands are now parsed into locals first, and the lock is held only to store the result. RunFrame also called UpdateBooleanComponent six times and UpdateScalarComponent twice every frame, and TrackedDevicePoseUpdated every frame even while disconnected. Components now go to SteamVR only when they change (all of them on the first frame). The pose still goes out every frame while the device is connected, as a tracked device's should; the disconnected pose goes out once. The helper now sends a pose only when it changes, so the comment says the driver keeps the last one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ft-powerd: ask SteamVR every 100 ms, not on every input event The loop polled the input devices with a 100 ms timeout and then, on every wake, did SteamVR's part: PollNextEvent, the headset's activity level and every device's pose, all IPC calls to vrserver. Input wakes it at once so the displays come on with the first key or motion, but a moving mouse sends hundreds of events a second, so moving the mouse meant hundreds of rounds of IPC a second instead of 10. Every wake still drains the input devices and the control socket and counts input as use straight away; SteamVR's part, and the backlight read that goes with it, now run only when 100 ms have passed since the last time, and poll sleeps until then. Built in the dev container (power/build.sh, no warnings); not run, since the live ft-powerd holds @ft_powerd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Eye tracker: ft-eyegrab checks only the slot each camera writes next While copying, ft-eyegrab woke every 300 us (about 1,500 to 3,000 times a second) and fingerprinted all eight slots each time: 8 x 256 strided reads from DMA-BUF memory. - Each look now checks only the slot each camera writes next. The order is known (camera 0 3,0,1,2; camera 1 7,5,4,6,5,7,6,4), and the next slot follows from the last two; the table starts from those orders and learns from every frame, so a SteamVR update that changes them costs a few seconds of full scans, not frames. - A camera with nothing in its expected slot 1.5 frames after its last one, or with no order yet, gets all four slots checked, as before. A frame that turns up in an unexpected slot means full scans for that camera for 2 s. - A slot's fingerprint is taken again when it stops being one of the two in use, so a later check sees only a new frame. A frame is still passed on when its camera starts the frame after next. - Between frames it sleeps until 2.5 ms before the next is due, then looks every 1 ms, with 0.5 ms of timer slack (PR_SET_TIMERSLACK, --share only). With no frames from either camera for 0.5 s (headset off) it looks every 4 ms. - --rec keeps its 0.3 ms polls (and the expected-slot checks), for its timestamps. Tested offline by building poll_frames against simulated cameras that write each frame in four bursts, the last after the next frame starts (6 s, both cameras): 1,076 frames passed on, none torn or skipped, with the known orders and with camera 1 in a different order. Wakeups 1,486/s -> 207/s, the poller's CPU 3.6% -> 0.8% of a core (in plain memory; the real DMA-BUF reads cost more), and a frame's start is seen 1.35 ms after it begins on average instead of 0.76. With a camera stalling 15 ms every 2 s, the old poller passed on 22 torn frames and the new one 8 or fewer. Built (glibc 2.38 symbols at most, the host has 2.39), not installed: it runs as root from /etc/frametop, so it takes effect only after gaze/tracker/install.sh (sudo). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: upload from the headset, then plug in and leave it Export and upload are done in the headset now: the export page only notes that VR may stutter a little. Upload opens the pull request first (a draft) and shows its link, telling the person to plug in the headset and leave it until it says Uploaded; the files then go to refs/pr/N, and the pull request is marked open at the end. A retry of the same export goes on in the same pull request. While an export or upload runs, a host unit holds a logind sleep inhibitor so the Frame stays awake with the headset off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remote Access: check the status every 5 s instead of every 2 s While its window was open, Frametop Remote Access ran remote-ctl.sh status every 2 s, and each run spawns bash, curl (the tailnet name from tailscaled) and python3 to parse it. It now checks every 5 s, plus when the window comes to the front and once more 2 s after turning remote access on or off or changing the password, so a change still shows within a couple of seconds. A check doesn't start while one is still running. Doing the check in-process would duplicate remote-ctl.sh's idea of "running", which the session and the pause code share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * update-check: KWin's blur and contrast effect ids, retest hints The session now turns KWin's blur and contrast effects off by id (blurEnabled and contrastEnabled in the desktop's kwinrc), and a KWin that renamed them would quietly leave them on. The check looks for their built-in factories (KWin::blur_factory, KWin::contrast_factory) in kwin_wayland, which it already reads for --output-count, and warns if one is gone. The kwin and plasma-workspace retest hints gain the blur and the hidden autostart entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pause gesture: look the controllers up every 30 s, not every 3 s The gesture reader fetched vrserver's /input/getstate.json over HTTP every 3 seconds, the whole time the relay runs, to notice a controller's root path changing when the 3D mouse takes or gives back its hand role. It now looks them up when it connects, when a message comes from a device path it doesn't know (at most every 3 s; the device is read from the message with two string searches, not a JSON parse of all 160 a second), 1.5 s after the relay's 3D mouse connects or lets go (the relay tells it through GamePause.controllers_changed), and otherwise every 30 s. The keys test's pause stub gets the new method. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Input relay: send mouse motion at most every 4 ms The relay sent the helper one "move" per SYN_REPORT, so a 1000 Hz mouse sent 1000 datagrams a second to a helper whose loop runs every 8 ms, and each one went through a dozen sscanf and strncmp tests in the helper before reaching the move handler. In a 200 ms test at 1000 Hz, 149 reports now make 45 moves with the same total. flush() on a report now sends only once 4 ms have passed since the last move; tick() sends the rest when due, and the select timeout shrinks to match. Buttons and the gaze keys still flush first, unconditionally, so a click lands where the pointer was. In the helper, "move" is now tested first in the command dispatch, and its handling is one lambda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: ft-gaze prints and reads only the sources in use ft-gaze computed and printed all six sources for every sample: about 1.3 KB of JSON a line with our tracker (practice2), 120 KB a second through podman's stdio relay for ft-gazed to json.loads 90 times a second. It also read SteamVR's gaze action for every sample outside games (UpdateActionState and GetEyeTrackingDataRelativeToNow, two calls into vrserver, 180 a second), though with our tracker ft-gazed only uses own and mmap1. - ft-gaze takes --sources LIST (action, mmap1, mmap2, left, right, own, and eye for the EYE object; all by default, so the probe and ft-eyes-session are unchanged), and with --watch-stdin a line "sources LIST" on stdin switches them. A source left out isn't read and prints as {"ok":0} ("eye" as null), so every line keeps the same keys. An older ft-gaze ignores both, and prints everything as before. - ft-gazed asks for what it reads: own,mmap1 with our tracker; left,right,mmap1 with SteamVR's eyes; the source plus mmap1 and mmap2 on the older one-source path. While a check or the calibration runs or waits to open, all of them, since checks record every source (the calibration fits the action's correction too) and the fit check reads "eye". It switches as soon as that changes, well inside the check's 0.45 s settle. - So the action is read only during checks, or with --source action. On recorded samples, a line with own and mmap1 is about 700 bytes instead of 1,300 (practice2), and one with left, right and mmap1 about 550 instead of 940 (test1). gaze/test/idle-test.py now checks that ft-gaze starts with every source for a check and is then switched to those in use, without the action or own; all its checks pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pointer: don't put a vanished panel back in the visibility map The panel-edge test read visible[edgeKey], and when the last panel the cursor touched was gone from the overlay list, that added it back as hidden. The map then had more entries than there are handles, which made the 50 ms visibility poll run every frame, and since the last commit it also counted as a visibility change each time, so unchanged frames were never reused. The edge test now looks the key up without adding it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: ft-gaze's loop runs every 4 ms instead of 2 ft-gaze's loop slept 2 ms, so 500 times a second it read the head pose (GetDeviceToAbsoluteTrackingPose), checked the eye tracker's counter, and drained SteamVR's events, for samples that come 90 times a second. It now sleeps 4 ms. A new sample is printed within 4 ms of appearing, 2 on average (was 1), and the pose history still has a pose within 2 ms of any sample's time, which keeps the head-pose error under 0.2 degrees for a head turning 100 degrees a second. Sleeping until the next sample is due would have thinned the pose history to 11 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gaze: the hidden panel waits for a command instead of waking every 50 ms The calibration panel runs for as long as the gaze service does, hidden nearly all the time, and it woke 20 to 30 times a second to look at its socket and SteamVR's events: about 0.9% of a core, the main cost left with gaze idle. It now waits in poll() on its command socket: up to a second while hidden, and up to 10 ms while shown, as before (it still drains SteamVR's events each pass, so a quit is acknowledged within a second while hidden). A command wakes it at once, so "show" draws sooner than before. With --watch-stdin, its stdin closing wakes it as well, so stopping it doesn't wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pointer: ft-screens announces new panels to the helper The helper now reads SteamVR's list of panels every 20 s instead of every second, so a panel made in between (a floating window's menu, frametop.float.N.sub.K, or the Frametop keyboard the first time it opens) couldn't be clicked with the mouse until the next read. ft-screens now sends "overlay <key>" to @ft_pointer_helper right after it makes one, and the helper adds it to its list at once (only frametop.* keys). An older helper ignores it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hands: fix Export doing nothing, and show it's busy at onceaf2ea7cput _stay_awake between exportSession and its @Slot, so the window's Export button called a method QML couldn't see. A new test checks every backend call in main.qml against Backend's slots and properties. Export and Upload now say Exporting…/Uploading… with a spinner the moment they're pressed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Lazy susan: Meta+Alt+Tab spins the panels around you - ft-screens "spin next|prev|<degrees>": every unpinned screen and floating window turns together about a vertical axis through your head (0.3 s, eased), so the next panel on the right or left comes to straight ahead; the arrangement stays as it is. Taps during a spin add to it, from where the panels are headed; grabbing a panel or placing it (ft-layout, ft-floatd) takes it out of the spin - when a spin settles, the panel in front gets the pointer (recenter), typing (as after a click), and KWin's active window: its floating window, or the top window on a screen (ft-floatd "front N", the KWin script's activate-output). KWin's outputs follow the screens' new places (ft-layout scale), as after a move - the input relay: spin_next and spin_prev actions, Meta+Alt+Tab and Meta+Alt+Shift+Tab by default; Frametop Input Settings lists them. Not Meta+Tab: that's Cmd+Tab on a Mac reached through a remote desktop like RustDesk, and the relay would take the Mac's app switcher. Meta+Alt+Tab (Cmd+Option+Tab) is unused on macOS, Windows, and KDE Used on the Frame (SteamOS 0.3.0 build 20260922) with one screen and three or four floating windows, through RustDesk to a Mac. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hand recorder: login command works from Frametop's Konsole Frametop's Konsole sets XDG_RUNTIME_DIR=/run/user/UID/frametop, where podman finds no container state, so 'distrobox enter dev -- hf auth login' failed with a crun error. The command the Upload page shows now sets the real runtime folder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hand recorder: log in from the Upload page, three-step page, no terminal The Upload page is now three numbered steps: choose the export, log in to Hugging Face, upload. Log in runs hub.py login, huggingface_hub's browser login (OAuth device code, as hf auth login does): the link opens in the browser and the page shows the code to enter, with Copy code and Cancel. hub.py saves the token; the window never sees one, and nobody pastes one. The terminal upload and the login command are gone from the page, and UPLOAD.md is now a short 'About uploading' under the steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hand recorder: take.json keeps camera clock samples sets.bin's capture_ns is CLOCK_MONOTONIC_RAW; poses.jsonl and prompts.jsonl are CLOCK_MONOTONIC. On 2026-10-03 the two were 0.80 s apart during a session and 1.11 s apart five hours later, so images can't be paired with poses by capture_ns. take.json now samples RAW minus MONOTONIC as each recording part starts and stops (as ft-hands' raw_minus_mono_ns), so readers can put each exposure on the poses' clock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hand recorder export: no controller poses without controllers, nothing in deleted ranges When the checklist says no controllers, exported poses.jsonl has left and right null and feedback lines carry no controller state: controllers left switched on still get tracked (one wandered 2 m in a real session) and would read as the hands' ground truth. Poses and live-tracker feedback inside deleted ranges are left out too, as the images there are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * hand recorder: final consent text (2026-10-03), residency check, installer Consent 2026-10-03, after a non-lawyer review: who runs this and how to reach them, the dataset is public (Hugging Face, possibly abroad), the Hugging Face username shows next to the contributor id, purposes (no identification), safety, the maintainer grant passes to whoever maintains Frametop next, withdrawal before and after merge, rights such as the GDPR's, and what a new version means. Residents of Illinois, Texas and Washington can't take part for now (biometric privacy laws): a third checkbox, profile consent.region_ok, checked by validate.py from this consent version on. The DRAFT banners are gone, so uploads no longer need FT_HANDREC_ALLOW_UPLOAD. hands/rec/install.sh installs the recorder on a Frame with Frametop: container packages, hand tracking and panel builds, ft-camd's capabilities, menu entry. test_qml_backend also checks each call's argument count against the slots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Screens: a reset button next to the grab bar, clickable in VR games Each desktop screen gets a reset button left of its bar (a reticle). It puts every screen back in its layout around where you are now, like Meta+Shift+R (ft-layout apply). In a VR game the screens leave the controllers to the game (the outside_games and dashboard modes), so a controller couldn't click any of their controls. Aiming a hand controller at the reset button now sets MakeOverlaysInteractiveIfVisible on that button's overlay alone, so the trigger clicks it; the flag clears half a second after the aim leaves a zone twice as wide, and the game gets the controllers back. The aim comes from the laser poses ft-screens already reads to show the controls. The ft-layout spawn is now RunLayout(cmd), shared with the arrange. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Click stability: 32 logical pixels by default, not 8 8 is about 0.2 degrees on a 3.4 m wide 3440-pixel screen 2 m away, so a trigger press turned into a drag unless the hand was very still. 32 (about 0.9 degrees) felt much better in the headset (2026-10-03). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Screens: in games, pointing a controller at a panel turns its laser on SteamVR's own floating windows take the laser while a controller points at them in a game and give it back when it points away. Frametop's panels didn't: with the controllers left to the game (outside_games, the default, or dashboard), they couldn't be clicked without the dashboard. ft-screens now sets MakeOverlaysInteractiveIfVisible on a screen or floating window while a hand controller's laser pose meets it, its controls, or its popups (UpdateAim; curved screens hit on their cylinder), and clears it 0.3 s after the aim leaves a wider margin. A drag or a held button keeps it on. The keyboard, one overlay, uses ComputeOverlayIntersection and now follows the mode when a game starts or ends while it's open. This replaces the reset button's own aim zone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Screens: find the controllers' laser tip during VR games too GetComponentStateForDevicePath with no input source handle fails for every render model component while a VR game runs (checked 2026-10-03 with a game up: all 21 components of frame_controller_right). TipOffset then fell back to the controller's pose, which aims 40 degrees above the Frame controller's laser. In games, pointing at a screen's middle missed it and pointing below it hit, so the new aim-to-laser only worked from the bottom; the controls' reveal and pin/roll aim were off the same way. GetComponentState still answers then, with the same tip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * input-relay: Add mute key as volume key Add KEY_MUTE as volume key. It will be mapped to KEY_MACRO28 and use wpctl to toggle the mute of the default audio sink. The toggle of the mute state will be done once when the key is pressed instead of continuously toggling it when it is held down. This allows the mute button on keyboards to work properly. Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com> * Session: bring back a taskbar saved on a screen the desktop doesn't have Plasma 6.2.5 keeps a panel on a screen number and never moves one whose number is past the screen count, so a taskbar saved on a spare output (#18, lastScreen=8 with three screens) or on a screen a smaller layout dropped stayed hidden. Before Plasma starts, session/fix-panels.py moves such a panel and its tray's containment to screen 0 (the primary), keeping its widgets, unless screen 0 already has a panel on that edge. doctor.sh checks the panels' screens, and report.sh lists them with the live outputs and panels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * keys-test: the spin bindings (Meta+Alt+Tab, Meta+Alt+Shift+Tab), not while paused Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Menu entries: own programs for Reset Screen Layout and Hide/Show Screens Reset Screen Layout and Hide/Show Screens both ran ft-layout. Steam lists entries by program, so Hide/Show launched Reset. Each gets a wrapper. From PR #17 (only this part of 9618be8; its host_command change is for the Nix packages). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Screens: release a held button that can't come up on a screen Pausing, or hiding the screen a button went down on, took the laser off it mid-click; the pause gesture's second thumbstick click does that. SteamVR's laser mouse then forgot the button ("Mouse down count is 1 but states are all false"), no release came, and the catcher kept showing whenever the pressing laser was off the panels, even while paused. Being interactive, it kept the VR game's controllers from it until the desktop restarted (2026-10-04, Beat Saber). ft-screens now releases a held button when it's paused, when the screen it went down on is hidden, or, during VR games, when the pressing hand controller has held nothing for a second. GetControllerState answers overlay apps only while a game runs; outside games a hold is never cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Codex <codex@localhost> Co-authored-by: CuriousJ <curious.j.tuber@gmail.com> Co-authored-by: Patrick McDavid <fusionjunky@gmail.com> Co-authored-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com> Co-authored-by: John Murray <5672686+JRMurr@users.noreply.github.com>
2466 lines
136 KiB
C++
2466 lines
136 KiB
C++
// ft-pointer: the universal 3D mouse's brain (OpenVR overlay client, runs in the dev container).
|
|
//
|
|
// input-relay.py (pointer mode) sends mouse commands here; this program keeps the
|
|
// cursor, does collision against SteamVR's overlays, draws the free-space dot, and
|
|
// sends the ft_pointer driver the exact pose of its virtual controller.
|
|
//
|
|
// relay -> @ft_pointer_helper -> ft-pointer -> @ft_pointer -> ft_pointer driver (inside vrserver)
|
|
//
|
|
// Cursor model:
|
|
// - anchor: head position at the last recenter; yaw/pitch: direction from it (mouse-driven).
|
|
// - Every frame a ray from the anchor is tested against every visible overlay
|
|
// (ComputeOverlayIntersection). On a hit the cursor sits on that surface; otherwise
|
|
// it floats `distance` metres out and a small dot overlay is shown there, which the
|
|
// laser can hit, so SteamVR never draws a free-flying laser.
|
|
// - Then the line of sight from the eye (not the anchor) to that point is tested too:
|
|
// after the head moves, something nearer can cover the point, and the cursor goes on
|
|
// whatever you see under it (panels close together in view, at different depths).
|
|
// - Looks: the compositor ignores live changes to dashboard.laserRayWidthScale (only
|
|
// the dashboard's own Settings screen reloads it), so the beam can't be switched
|
|
// off per device. Instead the laser starts POINTER_ORIGIN_FRACTION (0.95) of the way
|
|
// from the eye to the cursor, along the line of sight: what's left of the beam is a
|
|
// few centimetres long and effectively invisible, and SteamVR's hit dot (sized by
|
|
// distance from the origin) becomes tiny. Our own white dot is the visible cursor
|
|
// everywhere: a non-interactive dot on panels (the laser passes through it), and an
|
|
// interactive one in free space (the laser lands on it instead of flying off).
|
|
// - The controller ray starts at the eye and aims at the cursor point. Everything here
|
|
// is computed in the standing universe; the pose sent to the driver is converted to
|
|
// SteamVR's raw tracking space (drivers report raw poses; on the Frame the standing
|
|
// origin is ~1.6 m above the raw one, so sending standing coordinates put the laser
|
|
// origin 1.6 m above the head). While our device
|
|
// owns the dashboard pointer, dashboard.laserRayWidthScale is 0 so only the dot shows.
|
|
// It's restored when a controller takes the pointer back.
|
|
//
|
|
// Headset off: when SteamVR says nobody is wearing the headset, the pointer is released and
|
|
// stays off until it's worn again, so the displays can sleep (see the main loop). While the
|
|
// pointer is off the helper also stops listing overlays with vrcmd, whose connection every
|
|
// second kept SteamVR from going to standby.
|
|
//
|
|
// Idle: while the pointer is off (and hand gestures are off), the main loop waits for a command
|
|
// on its socket for up to 250 ms instead of running every 8 ms, or 20 ms while it reads mapped
|
|
// Frame controller buttons (vrbuttons.h), which have no event to wait for. What has to go on
|
|
// meanwhile still does: the headset's activity level, the game check and the relay's "vrgame"
|
|
// and "gazeawake" repeats, and a mouse command wakes it at once. The overlay lookups (the 50 ms
|
|
// visibility poll, the once-a-second handles) wait for the pointer to wake, and run right then.
|
|
//
|
|
// Unchanged frames: while the pointer is awake, a frame where nothing moved reuses the last one's
|
|
// result. The collision passes (ComputeOverlayIntersection on every visible overlay, twice) run
|
|
// again when the mouse or the anchor moved, the eye moved 5 mm, an overlay showed or hid, or
|
|
// 100 ms have passed (overlays also move by themselves), and always while dragging. The dots'
|
|
// setters go to SteamVR only when their value changes, and a dot's placement only when it moved
|
|
// 0.2 mm or the eye 5 mm (its facing and size then change by well under 1%). The pose goes to the
|
|
// driver, which keeps the last one, when the laser's origin moved 0.2 mm or its direction 0.04
|
|
// deg (0.1 mm where it lands, 15 cm on), and at least every 100 ms.
|
|
//
|
|
// Last used wins: when a real controller moves (picked up), the pointer is released
|
|
// (driver "hide", which also drops its hand role hint), so the controller gets
|
|
// its role and laser back. The next mouse input reconnects and claims the laser again.
|
|
// Moving means faster than 0.35 m/s or 2 rad/s, both times POINTER_CONTROLLER_PICKUP,
|
|
// for 100 ms in a row with the controller tracked normally: a single sample over the
|
|
// limit was enough before, and controllers resting on a desk released the pointer on a
|
|
// knock or a tracking jump.
|
|
// SteamVR gives a contested hand role to the most recently used device, and a held
|
|
// Frame controller counts as used (touch sensors). If our device hasn't got the hand
|
|
// role within a second of waking, the pointer is released (no orphan white dot) and
|
|
// mouse input can't wake it again for 2 s.
|
|
//
|
|
// Laser mode: with the dashboard closed, SteamVR keeps its laser mouse off until a
|
|
// click (the first click on a panel only turned it on, the second one clicked), and a
|
|
// laser that leaves every panel turns it off again. While the pointer is awake, the
|
|
// helper shows frametop.pointer.lasermode: a transparent 1 mm overlay 50 m below the
|
|
// head with VROverlayFlags_MakeOverlaysInteractiveIfVisible, which keeps SteamVR's
|
|
// laser mouse mode on as long as it's visible. It's hidden whenever the pointer is
|
|
// released, so controllers and VR games get the normal behaviour back.
|
|
//
|
|
// Tilt: while the left button is held (dragging a panel by its grab bar, which SteamVR
|
|
// moves rigidly with the controller), pressing the right button enters tilt mode. The
|
|
// right press is not forwarded; mouse motion then rotates the virtual controller around
|
|
// the grab point (horizontal: about the vertical axis, vertical: about the view's
|
|
// horizontal axis), so the panel turns around that pivot. The tilt accumulates for the
|
|
// whole drag: after the right button is released, the rotation stays applied (about the
|
|
// moving cursor point) so the grabbed panel keeps its new orientation, and pressing right
|
|
// again continues from it. Releasing the left button drops the panel; the tilted pose (and
|
|
// the drag lock) are held 0.5 s longer, because SteamVR's dashboard finishes a floating
|
|
// move up to 150 ms after the release (UndockedOverlay.endFloatingWindowMove measures the
|
|
// push distance first) and reads the controller pose again then.
|
|
//
|
|
// Scene-graph overlays: the dashboard's dock (valve.steam.gamepadui.bar) and the controls
|
|
// under floating windows (valve.steam.gamepadui.floatingfooter, undock and friends) have
|
|
// no texture (0x0) and a placeholder width, so ComputeOverlayIntersection never hits
|
|
// them. For those the ray is tested against the overlay's plane, within
|
|
// POINTER_SCENE_RADIUS (0.5 m) of its origin; the laser-catching dot sits 5 cm behind the
|
|
// plane, so the laser reaches the buttons and still lands on the dot between them.
|
|
// Only absolutely placed 0x0 overlays count as scene-graph: gamescope's app panels (the
|
|
// desktops) also report 0x0, but they're placed as dashboard tabs, stay up when the
|
|
// dashboard closes, and ComputeOverlayIntersection hits them normally.
|
|
//
|
|
// SteamVR Settings (a workaround for that page only): Steam's pages (Library and the rest)
|
|
// are drawn in valve.steam.gamepadui.main, a dashboard overlay ComputeOverlayIntersection hits
|
|
// exactly. SteamVR's Settings page isn't: the main overlay is hidden, and the page is drawn by
|
|
// the scene-graph panel (valve.steam.gamepadui.frame.menu.N), whose shape OpenVR doesn't give
|
|
// out. Its transform's plane isn't the page's surface, which is nearer, so the laser, starting
|
|
// a few cm in front of where we thought the page was, started behind it: most of the page
|
|
// took no clicks, which went through to a desktop screen behind, and the page covered our
|
|
// dot. So while the cursor is on that page (OnSettingsPage), the laser starts near the eye
|
|
// (SETTINGS_ORIGIN) and SteamVR's own hit test finds the page; our dot is drawn close in front
|
|
// (SETTINGS_DOT), and the laser-catching dot sits far behind everything (SETTINGS_CATCHER),
|
|
// invisible and with SteamVR's hit dot hidden, so it can't cover the page. The beam and
|
|
// SteamVR's hit dot on the page then look like a controller's. Everywhere else nothing
|
|
// changes.
|
|
//
|
|
// Panel edges: off a panel, the cursor stays on that panel's plane while it's within
|
|
// POINTER_EDGE_REACH (0.3 m) of the last point it touched, instead of jumping to
|
|
// POINTER_DISTANCE. A floating panel's resize margins and the window controls under it
|
|
// sit just outside the panel, and the laser has to start in front of that plane to reach
|
|
// them (a controller's laser always does: it starts at the hand). Like on scene-graph
|
|
// planes, the laser-catching dot sits 5 cm behind the plane.
|
|
//
|
|
// Drag lock: while the left button is held, the cursor keeps the distance it had at the
|
|
// press and collision is frozen, so dragging past a panel's edge (resizing, moving)
|
|
// doesn't jump the cursor to free space or swap in the laser-catching dot, which made
|
|
// SteamVR's resize snap back.
|
|
// A left release also goes to ft-screens ("up"), which releases a button held on its
|
|
// screens in KWin if SteamVR gave the release to some other overlay.
|
|
// Across Frametop's panels (a drag and drop, or a window moved from one screen or floating
|
|
// window to another), the lock gives way: while the left button is held on one of ft-screens'
|
|
// panels, the ray is still tested against ft-screens' panels, and the cursor goes onto
|
|
// whichever it meets first. Not while that panel is being carried (its title bar or its bar):
|
|
// then the ray would find what's behind it.
|
|
//
|
|
// Head follow (experimental, off by default; POINTER_FOLLOW=1, or the relay's "follow toggle"): the cursor
|
|
// is carried by a reference direction, where the head faced when it last settled, and turns
|
|
// with it, keeping its offset (mouse movement changes the offset, up to POINTER_FOLLOW_REACH,
|
|
// 70 deg, so the cursor can sit in a corner of the view). While the head stays within
|
|
// POINTER_LEASH_DEG (10) of the reference, nothing moves on its own: the cursor stays put in
|
|
// the room. Once the head has been past the leash for POINTER_LEASH_DELAY (0.2 s; a glance
|
|
// out and back doesn't count), the reference follows: it eases toward the head's facing with a
|
|
// time constant of POINTER_LEASH_RETURN (0.2 s), never falling further behind than the leash
|
|
// (or than it already was), until it lands on the facing, and the cursor is back where it was
|
|
// in the view. Then it waits for the leash again. Earlier tries: dragging the reference only at
|
|
// the leash's end left it up to the leash off after turning back (getting it centred took an
|
|
// overshoot), and easing it all the time moved the cursor on every small head movement. At 0
|
|
// the reference is the head's facing, so the cursor is head-locked. Head roll is ignored (the
|
|
// frames have no roll), so tilting the head doesn't swing the cursor. The ray origin (the
|
|
// anchor) moves to the eye with the reference, so leaning inside the leash doesn't move the
|
|
// cursor either. While the left
|
|
// button is held (and the drop hold after it), the leash still moves the reference but the
|
|
// cursor stays put in the room, so a click or a drag can't be nudged by the head; the offset
|
|
// is taken up from where the cursor is when the hold ends, so it doesn't jump.
|
|
//
|
|
// Gaze mode (experimental, off by default; POINTER_GAZE=1, "gaze on|off|toggle", or the
|
|
// relay's gaze_toggle): the pointer goes where you look, and the mouse does the last bit
|
|
// (MAGIC pointing: Zhai, Morimoto and Ihde, CHI 1999). The gaze service (gaze/ft-gazed)
|
|
// sends the corrected gaze 90 times a second, "gz <yaw> <pitch> <raw yaw> <raw pitch>"
|
|
// (head-relative degrees), and while the gaze has the pointer, the cursor ray is simply
|
|
// that gaze from the eye: nothing is steered, so nothing can pile up. Moving the mouse takes
|
|
// the pointer from the gaze, and it moves from where the gaze left it, as usual. Looking
|
|
// well away from it (more than POINTER_GAZE_RETAKE, 5 deg, for 120 ms, with the mouse still
|
|
// for 300 ms) gives it back to the gaze; small eye movements around the pointer don't.
|
|
// A left press while the gaze has the pointer isn't sent yet: the pointer stops where the
|
|
// gaze put it, and if the gaze is off, you drag it onto what you meant with the mouse
|
|
// (still holding the button; panels only see it hover). The release clicks there, a press
|
|
// and a release 40 ms apart. Held still for POINTER_GAZE_HOLD (0.5 s) instead, it becomes
|
|
// a real press where the pointer is, so drags work: hold, then move. After a click that
|
|
// didn't need correcting, and after a drag, the gaze has the pointer again.
|
|
// Outside games (no scene application), gaze mode keeps the pointer: the relay doesn't
|
|
// release it when the mouse is idle ("gazeawake 1|0" tells it). A controller that moves
|
|
// still releases it, as without gaze (last used wins), and in games the mouse wakes it and
|
|
// idling releases it, as without gaze. Gaze mode is a mouse feature: the controllers aren't
|
|
// part of it. Steam reads the Frame controllers itself, outside SteamVR's bindings, so
|
|
// controller clicks at the gaze can't be done cleanly (docs/gaze-controllers.md).
|
|
// The dot shows all the time in gaze mode (POINTER_GAZE_DOT=always, the default). With
|
|
// POINTER_GAZE_DOT=moving it only shows while the mouse moves it (within POINTER_GAZE_SHOW,
|
|
// 1 s), while a press is held, and briefly for each click (a pulse); otherwise it's
|
|
// transparent (still there for the laser to land on), since you know where you're looking.
|
|
// When the mouse took the pointer and you then click, the nudge was probably onto what
|
|
// you were looking at: from the raw gaze when the mouse took over to where you clicked is
|
|
// the tracker's error there. The helper sends it to ft-gazed as a lesson ("lesson <raw yaw>
|
|
// <raw pitch> <true yaw> <true pitch>", the true direction relative to the head as it was
|
|
// when the mouse took over) if the mouse moved at least 0.2 deg, the click came within 10 s,
|
|
// and the correction (raw gaze to click) is within POINTER_GAZE_NUDGE_MAX (55 deg, half what
|
|
// the headset shows across: the Frame's eyes see 109 deg each). Past that, the tracker is far
|
|
// off (or you went somewhere else with the mouse): it isn't learned, and ft-gazed is asked
|
|
// for its quick check instead ("recheck <deg>"; it waits out its cooldown). Our tracker's
|
|
// clicks since its calibration put a one-dot check within 15 deg for the next 2 minutes and
|
|
// 25 for the next 10, so the check gets back under it (2026-10-01). A held press dragged onto
|
|
// the target is the same: from the raw gaze at the press to the release. With no
|
|
// fresh gaze (a blink, the service stopped, the headset off), the pointer stays put.
|
|
//
|
|
// Gaze precision (the relay's gaze_precision and gaze_drag actions, bound to a mouse button or
|
|
// a key combination; "precision|gazedrag mouse|keyboard 1|0" here): gaze mode aims, the button
|
|
// makes it exact. Pressing gaze precision stops the pointer where you look, as gaze mode's
|
|
// mouse press does; while it's held, the mouse steers the pointer by its moves. Releasing
|
|
// clicks where the pointer is. A correction is a lesson for the gaze tracker, as with the
|
|
// mouse. Gaze drag is the same with a real press at once, dragging until the release, for
|
|
// title bars, grab bars, and selections. Without gaze mode both still work from wherever the
|
|
// pointer is.
|
|
// In gaze mode the mouse's left button is a gaze precision button (POINTER_GAZE_MOUSE =
|
|
// precision, the default), or clicks at once where the pointer is (direct). With precision the
|
|
// mouse's buttons work like the keyboard clicks: the right button's press is held back the same
|
|
// way, and the right click comes on the release, where the pointer is by then (gaze_right).
|
|
// Pressing the right button while the left one's press is held back presses the left button
|
|
// where the pointer is now, and the mouse drags (gaze_left then gaze_right): the drag lasts
|
|
// while either button is held. So once you've moved the pointer, the left button alone only
|
|
// clicks; to drag from there, press the right one. Pressing the right one again (a double right
|
|
// click, with the left still held) tilts, as a right press does during any drag (see Tilt).
|
|
// Held still for POINTER_GAZE_HOLD, either press is a real one.
|
|
// POINTER_GAZE_MOUSE_MOVE: held (the default) or free. Held: while the gaze has the pointer,
|
|
// moving the mouse does nothing; it moves the pointer only during a press (as a correction,
|
|
// like a keyboard click's head). So a bumped or drifting mouse can't pull the pointer off what
|
|
// you're looking at, and every mouse move is a correction worth learning. With the gaze stale
|
|
// for a second (the tracker stopped, eyes closed), in a game, or the headset off, the mouse
|
|
// moves the pointer as usual. Free: the mouse takes the pointer whenever it moves.
|
|
// Keyboard clicks (the relay's gaze_left and gaze_right, Meta+J and Meta+K by default;
|
|
// "gazekey left|right 1|0" here) work like gaze mode's mouse press, steered by the head: the
|
|
// press stops the pointer where you look, and while the keys are held the pointer stays put in
|
|
// your view, so turning your head carries it onto what you meant (past
|
|
// POINTER_HEAD_DEADZONE, 0.5 deg, so a still head doesn't wobble it). The release clicks there
|
|
// (left, or right for gaze_right), and a correction is a lesson for the gaze tracker, as with
|
|
// the mouse. Held still for POINTER_GAZE_HOLD instead, it's a real press, and the head drags.
|
|
// A quick tap (let go within POINTER_KEY_TAP, 0.25 s) clicks where the dot was at the press,
|
|
// whatever the head did meanwhile, and tells the gaze tracker it was right there (a lesson
|
|
// with no correction). Pressing gaze_right while gaze_left aims (Meta+K with Meta+J held)
|
|
// presses the left button where the dot is now, so you can correct first and then drag; the
|
|
// drag lasts while either key is held. gaze_right pressed during a gaze_left drag (held still
|
|
// into one, or again after starting one with it: a double Meta+K) tilts while it's held (see
|
|
// Tilt): turning the head turns the panel, and the mouse can too; let go of it and the head
|
|
// drags again from there. Without gaze mode they work from wherever the pointer is.
|
|
// The gaze calibration panel (gaze/panel/ft-gazepanel, run by the gaze service): while
|
|
// ft-gazed says it's up ("calpanel 1", renewed every second; it lapses 3 s after the last),
|
|
// the dot hides and a press answers the panel instead of clicking: a left click or gaze_left
|
|
// sends "calaccept" to @ft_gazed (take this dot now), a right click or gaze_right "calquit".
|
|
// POINTER_ROLE (right, left, or stylus): the hand role our device takes while connected. A
|
|
// Frame controller in your hand counts as used through its touch sensors and takes its hand's
|
|
// role back, and then no click lands (see "no hand role" in the main loop): with a controller
|
|
// held in the right hand, the pointer needs the left hand, or the stylus role.
|
|
//
|
|
// Hands (POINTER_HANDS, off by default; needs hand tracking, hands/): ft-hands publishes
|
|
// pinches and grips (hands/include/fh_gestures.h), read here every frame.
|
|
// A pinch works like gaze mode's mouse press: the pointer stops (where the gaze put it), and
|
|
// the click comes when the pinch opens, where the pointer is then. A quick tap clicks where
|
|
// you looked. Held, the pinch's hand moves the pointer, for correcting the gaze: past
|
|
// POINTER_PINCH_DEADZONE (1.5 deg of hand movement, seen from the eye; a tap's jitter and the
|
|
// pinch point shifting as the fingers close stay inside it), at POINTER_PINCH_GAIN (0.5: half
|
|
// the hand's angle, for precision). A correction is a lesson for the gaze tracker, as with the
|
|
// mouse. A pinch ended by losing the hand (or by a grip taking over) doesn't click.
|
|
// Without gaze mode, a pinch is a real press instead, like the mouse's button: pressed when
|
|
// it closes and released when it opens, and held, its hand drags the pointer (at the same gain
|
|
// and past the same dead zone). A tap is still a click where the pointer is. That's what the
|
|
// gaze probe's Click practice wants: it has its own gaze dot, frozen by the press and dragged
|
|
// by the pointer's movement until the release.
|
|
// A grip (closing the hand) is a press and drag: the press where the pointer is, then the
|
|
// hand moves the pointer at POINTER_GRIP_GAIN (1: as far as it moves, seen from the eye), and
|
|
// opening the hand releases. So it drags whatever the pointer is on: a title bar moves the
|
|
// window, a panel's grab bar carries the panel, text is selected.
|
|
// Typing touches thumb to index like a pinch: no pinch begins within POINTER_PINCH_TYPING
|
|
// (1 s) of a key (the relay says "typing"). A grip begins only with the hand held up, at most
|
|
// POINTER_GRIP_BELOW (0.35 m) below the eyes: hands on a desk curl like a loose fist. (The
|
|
// user's own pinches sat 0.35-0.45 m below the eyes, elbow resting, so pinches have no such limit.)
|
|
// The hand's movement is taken in the room, from where the eye was when the gesture began,
|
|
// with the head pose at each frame's capture time, so turning your head doesn't move it.
|
|
// The first gesture while the pointer is off only wakes it. Gestures are ignored in a VR game
|
|
// (unless the dashboard is up) and with the headset off, and while the mouse's button is held.
|
|
// Hand use keeps the pointer from the relay's idle release for two minutes, as gaze mode does.
|
|
//
|
|
// Placement (for layout): SteamVR keeps a floating panel's position inside the
|
|
// dashboard, where nothing outside can set it, so the helper carries panels like a user
|
|
// would. It measures the panel (md::ScanPanel), aims the device at its grab bar
|
|
// (LAYOUT_GRAB_OFFSET, 7.5 cm below the bottom edge; the bands at 2-4 and 14-26 cm are
|
|
// other controls), presses, moves, and releases. While grabbed, the panel follows the
|
|
// device rigidly, except that the dashboard accelerates fast translations (0.1 m in 0.3 s
|
|
// moved it 0.19 m and turned it 8.5 deg, in jerky 25 ms steps right after the press). So
|
|
// the device hovers first, and the move is split into a rotation about the device origin
|
|
// (the eye) at 60 deg/s and a smooth 60 Hz slide at LAYOUT_SLIDE_SPEED (0.5 m/s; tested
|
|
// exact from 0.07 to 1 m/s). Scroll pushes along the panel normal, but only in whole notches of
|
|
// about 7 cm, so it isn't used. The result is measured again, and the move repeated up to
|
|
// twice while it's more than 1.5 cm or 1 deg off.
|
|
//
|
|
// Ignored panels: overlays matching POINTER_IGNORE are left out of the collision, so the cursor
|
|
// passes through them to what's behind. For display-only panels in the way, such as a
|
|
// head-locked performance overlay, which ComputeOverlayIntersection hits like any other. The
|
|
// laser starts just before the cursor point (see Looks), so a panel nearer to you doesn't
|
|
// catch it either.
|
|
//
|
|
// Commands (datagrams on @ft_pointer_helper): show, hide, recenter, move <dyaw> <dpitch>,
|
|
// follow on|off|toggle (head follow, until the next restart or a change to POINTER_FOLLOW),
|
|
// gaze on|off|toggle|? (gaze mode, likewise with POINTER_GAZE; ? only asks, and "? headset" adds
|
|
// worn|away to the reply), gz ... (the gaze, from ft-gazed),
|
|
// reload (re-read the settings below), debug (toggle a twice-a-second state log),
|
|
// overlays (replies with the overlay list as JSON, see OverlayList),
|
|
// vrbind/vrglobal/vrstatus (Frame controller buttons, see vrbuttons.h),
|
|
// and btn/scroll lines, which are forwarded to the driver unchanged. For layouts, with a
|
|
// reply datagram to the sender's (abstract) address:
|
|
// place <overlay> <x> <y> <z> <yaw> <pitch> [roll [grab]]: centre in the standing
|
|
// universe; the front faces back along the direction (yaw, pitch), turned by roll
|
|
// (counterclockwise as seen, degrees) -> "ok ..." | "error ..."
|
|
// measure <overlay> -> "ok cx cy cz width height xx xy xz yx yy yz zx zy zz" (centre,
|
|
// size, and the panel's right, up, and front vectors)
|
|
// head -> "ok x y z yaw pitch"
|
|
// grabprobe <overlay>: log where below the panel SteamVR's laser hits something (to
|
|
// find the grab bar again if a SteamVR update moves it)
|
|
//
|
|
// Settings (~/.config/frametop.conf): POINTER_DISTANCE (m, 1.5), POINTER_CURSOR_DEG
|
|
// (angular size of the dot, 0.4), POINTER_LASER_WIDTH (controller beam width to restore, 0.8),
|
|
// POINTER_ORIGIN_FRACTION (0.95): the laser starts this far along the eye-to-cursor line,
|
|
// but never closer than POINTER_ORIGIN_MARGIN (0.15 m) to the cursor point: SteamVR's
|
|
// small controls (undock, frame buttons) float a few centimetres in front of their
|
|
// panel, and a laser that starts behind them can't hit them. POINTER_FOLLOW (0) and
|
|
// POINTER_LEASH_DEG (10), POINTER_LEASH_DELAY (0.2 s), POINTER_LEASH_RETURN (0.2 s),
|
|
// POINTER_FOLLOW_REACH (70 deg): head follow, above. POINTER_GAZE (0), POINTER_GAZE_RETAKE
|
|
// (5 deg), POINTER_GAZE_NUDGE_MAX (55 deg), POINTER_GAZE_HOLD (0.5 s), POINTER_GAZE_DOT
|
|
// (always), POINTER_GAZE_SHOW (1 s): gaze mode, above. POINTER_CONTROLLER_PICKUP (1, 0.5 to 5): how hard a controller must
|
|
// move to take the laser back, above. POINTER_IGNORE (empty): ignored panels, above.
|
|
// POINTER_HANDS (0), POINTER_PINCH_GAIN (0.5), POINTER_PINCH_DEADZONE (1.5 deg),
|
|
// POINTER_GRIP_GAIN (1), POINTER_GRIP_BELOW (0.35 m), POINTER_PINCH_TYPING (1 s): hands, above.
|
|
// POINTER_GAZE_MOUSE (precision), POINTER_HEAD_DEADZONE (0.5 deg), POINTER_KEY_TAP (0.25 s),
|
|
// POINTER_ROLE (right): gaze precision and keyboard clicks, above.
|
|
#include <openvr.h>
|
|
|
|
#include "vrbuttons.h"
|
|
#include "vrmath.h"
|
|
|
|
extern "C" {
|
|
#include "../../hands/include/fh_gestures.h"
|
|
}
|
|
|
|
#include <algorithm>
|
|
#include <atomic>
|
|
#include <chrono>
|
|
#include <cmath>
|
|
#include <condition_variable>
|
|
#include <cstdio>
|
|
#include <cstdlib>
|
|
#include <cstring>
|
|
#include <deque>
|
|
#include <fstream>
|
|
#include <map>
|
|
#include <mutex>
|
|
#include <string>
|
|
#include <thread>
|
|
#include <tuple>
|
|
#include <vector>
|
|
|
|
#include <climits>
|
|
|
|
#include <fcntl.h>
|
|
#include <fnmatch.h>
|
|
|
|
#include <poll.h>
|
|
#include <sys/mman.h>
|
|
#include <sys/socket.h>
|
|
#include <sys/stat.h>
|
|
#include <sys/un.h>
|
|
#include <unistd.h>
|
|
|
|
namespace {
|
|
|
|
using namespace md;
|
|
|
|
std::map<std::string, std::string> ReadConfig() {
|
|
std::map<std::string, std::string> conf;
|
|
const char *home = std::getenv("HOME");
|
|
std::ifstream in(std::string(home ? home : "") + "/.config/frametop.conf");
|
|
std::string line;
|
|
while (std::getline(in, line)) {
|
|
line = line.substr(0, line.find('#'));
|
|
const auto eq = line.find('=');
|
|
if (eq == std::string::npos) continue;
|
|
auto trim = [](std::string s) {
|
|
s.erase(0, s.find_first_not_of(" \t"));
|
|
s.erase(s.find_last_not_of(" \t") + 1);
|
|
return s;
|
|
};
|
|
conf[trim(line.substr(0, eq))] = trim(line.substr(eq + 1));
|
|
}
|
|
return conf;
|
|
}
|
|
|
|
double ConfDouble(const std::map<std::string, std::string> &c, const char *key, double fallback) {
|
|
auto it = c.find(key);
|
|
return it == c.end() ? fallback : std::atof(it->second.c_str());
|
|
}
|
|
|
|
// Hand gestures from ft-hands (see "Hands" at the top): /run/user/UID/frametop-hands/gestures,
|
|
// mapped read-only. Version 1 files have pinches only; their grips read as zero.
|
|
class HandGestures {
|
|
public:
|
|
~HandGestures() { Close(); }
|
|
// A consistent copy of the file (its sequence lock), if it's there. Opens it, and
|
|
// checks it's still the same file, at most once a second.
|
|
bool Read(fh_gestures_t &out) {
|
|
const auto now = std::chrono::steady_clock::now();
|
|
if (now - checked_ > std::chrono::seconds(1)) {
|
|
checked_ = now;
|
|
const std::string path = "/run/user/" + std::to_string(getuid()) + "/frametop-hands/gestures";
|
|
struct stat st;
|
|
if (stat(path.c_str(), &st) != 0 || size_t(st.st_size) != len_ || st.st_ino != ino_) {
|
|
Close();
|
|
const int fd = open(path.c_str(), O_RDONLY | O_CLOEXEC | O_NOFOLLOW);
|
|
if (fd >= 0 && fstat(fd, &st) == 0 && size_t(st.st_size) >= offsetof(fh_gestures_t, grip)) {
|
|
void *m = mmap(nullptr, size_t(st.st_size), PROT_READ, MAP_SHARED, fd, 0);
|
|
if (m != MAP_FAILED) map_ = m, len_ = size_t(st.st_size), ino_ = st.st_ino, ++opens;
|
|
}
|
|
if (fd >= 0) close(fd);
|
|
}
|
|
}
|
|
if (!map_) return false;
|
|
const auto *g = static_cast<const fh_gestures_t *>(map_);
|
|
if (std::memcmp(g->magic, FH_GESTURES_MAGIC, 8) != 0) return false;
|
|
const size_t n = std::min(len_, sizeof out);
|
|
for (int tries = 0; tries < 3; ++tries) {
|
|
const uint64_t seq = __atomic_load_n(&g->seq, __ATOMIC_ACQUIRE);
|
|
if (seq & 1) continue;
|
|
std::memset(&out, 0, sizeof out);
|
|
std::memcpy(&out, map_, n);
|
|
__atomic_thread_fence(__ATOMIC_ACQUIRE);
|
|
if (__atomic_load_n(&g->seq, __ATOMIC_RELAXED) != seq) continue;
|
|
if (out.version < 2) std::memset(out.grip, 0, sizeof out.grip);
|
|
return true;
|
|
}
|
|
return false;
|
|
}
|
|
int opens = 0; // a new file: the counters start over
|
|
|
|
private:
|
|
void Close() {
|
|
if (map_) munmap(map_, len_);
|
|
map_ = nullptr, len_ = 0, ino_ = 0;
|
|
}
|
|
void *map_ = nullptr;
|
|
size_t len_ = 0;
|
|
ino_t ino_ = 0;
|
|
std::chrono::steady_clock::time_point checked_{};
|
|
};
|
|
|
|
// The HMD's recent poses (standing universe), to turn the gestures' head-frame points into
|
|
// the room as the head was when the cameras took them.
|
|
class PoseHistory {
|
|
public:
|
|
void Add(std::chrono::steady_clock::time_point t, const vr::HmdMatrix34_t &m) {
|
|
poses_.push_back({t, m});
|
|
while (poses_.size() > 128) poses_.pop_front(); // about a second
|
|
}
|
|
// The pose at CLOCK_MONOTONIC time t_ns (steady_clock's), or the nearest kept.
|
|
bool At(uint64_t t_ns, vr::HmdMatrix34_t &out) const {
|
|
if (poses_.empty()) return false;
|
|
const std::chrono::steady_clock::time_point t{std::chrono::nanoseconds(t_ns)};
|
|
const auto *best = &poses_.front();
|
|
for (const auto &p : poses_)
|
|
if (std::chrono::abs(p.first - t) < std::chrono::abs(best->first - t)) best = &p;
|
|
out = best->second;
|
|
return true;
|
|
}
|
|
|
|
private:
|
|
std::deque<std::pair<std::chrono::steady_clock::time_point, vr::HmdMatrix34_t>> poses_;
|
|
};
|
|
|
|
int AbstractSocket(const char *name, bool bindIt) {
|
|
const int fd = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0);
|
|
if (bindIt) {
|
|
sockaddr_un addr{};
|
|
addr.sun_family = AF_UNIX;
|
|
std::memcpy(addr.sun_path + 1, name, std::strlen(name));
|
|
const socklen_t len = offsetof(sockaddr_un, sun_path) + 1 + std::strlen(name);
|
|
if (bind(fd, reinterpret_cast<sockaddr *>(&addr), len) != 0) {
|
|
std::perror("bind @ft_pointer_helper (already running?)");
|
|
std::exit(1);
|
|
}
|
|
}
|
|
return fd;
|
|
}
|
|
|
|
std::string ExeDir() {
|
|
char buf[PATH_MAX];
|
|
const ssize_t n = readlink("/proc/self/exe", buf, sizeof buf - 1);
|
|
if (n <= 0) return ".";
|
|
buf[n] = 0;
|
|
std::string p(buf);
|
|
return p.substr(0, p.rfind('/'));
|
|
}
|
|
|
|
// One of ft-screens' panels showing a desktop: a screen (frametop.screen.N), a floating
|
|
// window (frametop.float.N), or a floating window's popup (frametop.float.N.sub.K), not a
|
|
// control of theirs.
|
|
bool FramePanel(const std::string &key) {
|
|
for (const char *prefix : {"frametop.screen.", "frametop.float."}) {
|
|
if (key.rfind(prefix, 0) != 0) continue;
|
|
const std::string rest = key.substr(std::strlen(prefix));
|
|
const size_t dot = rest.find('.');
|
|
return dot == std::string::npos || rest.compare(dot, 5, ".sub.") == 0;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
void SendTo(int fd, const char *name, const std::string &msg) {
|
|
sockaddr_un addr{};
|
|
addr.sun_family = AF_UNIX;
|
|
std::memcpy(addr.sun_path + 1, name, std::strlen(name));
|
|
const socklen_t len = offsetof(sockaddr_un, sun_path) + 1 + std::strlen(name);
|
|
sendto(fd, msg.data(), msg.size(), 0, reinterpret_cast<sockaddr *>(&addr), len);
|
|
}
|
|
|
|
// JSON string literal (names come from other apps).
|
|
std::string JsonQuote(const std::string &s) {
|
|
std::string out = "\"";
|
|
for (const unsigned char c : s) {
|
|
if (c == '"' || c == '\\') out += '\\', out += char(c);
|
|
else if (c < 0x20) {
|
|
char esc[8];
|
|
std::snprintf(esc, sizeof esc, "\\u%04x", c);
|
|
out += esc;
|
|
} else out += char(c);
|
|
}
|
|
return out + "\"";
|
|
}
|
|
|
|
// Overlay keys, refreshed in the background from `vrcmd --overlays` (OpenVR has no
|
|
// public call to enumerate other apps' overlays). Hidden ones are listed too: the
|
|
// window controls under a floating panel only appear while something hovers the
|
|
// panel, and the cursor has to find them the moment they do, not a second later.
|
|
// Each vrcmd run is a shell and a new SteamVR client, about 30 ms of CPU, so the list is read
|
|
// every 20 seconds (kEvery) while the pointer is awake, and at once when something says it may
|
|
// have changed (Kick: the pointer waking, the dashboard opening or closing, a click that hit
|
|
// nothing), at most once a second. Showing and hiding the overlays it knows doesn't need a new
|
|
// list: the main loop polls their visibility (IsOverlayVisible) every 50 ms.
|
|
// Paused while the pointer is off: each vrcmd run connects to SteamVR as a new app, and a new
|
|
// app every second kept SteamVR (and the headset's displays) from going to standby.
|
|
// "overlays" requests (Frametop Input Settings' Ignored panels page) refresh the list even
|
|
// while paused, and are answered from this thread once it's fresh.
|
|
class OverlayList {
|
|
public:
|
|
static constexpr auto kEvery = std::chrono::seconds(20), kGap = std::chrono::seconds(1);
|
|
void Start() {
|
|
out_ = socket(AF_UNIX, SOCK_DGRAM | SOCK_CLOEXEC, 0);
|
|
thread_ = std::thread([this] {
|
|
std::unique_lock<std::mutex> lk(lock_);
|
|
while (running_) {
|
|
// Sleeps while paused (until a request or the wake); otherwise until it's time, or
|
|
// a kick once kGap has passed since the last read.
|
|
const auto due = std::max(last_ + kGap, kicked_ ? last_ : last_ + kEvery);
|
|
if (!waiting_.empty()) {
|
|
} else if (paused_) {
|
|
cv_.wait(lk);
|
|
continue;
|
|
} else if (std::chrono::steady_clock::now() < due) {
|
|
cv_.wait_until(lk, due);
|
|
continue;
|
|
}
|
|
kicked_ = false;
|
|
lk.unlock();
|
|
Refresh();
|
|
lk.lock();
|
|
last_ = std::chrono::steady_clock::now();
|
|
}
|
|
});
|
|
}
|
|
// Unpausing (the pointer woke) reads the list again at once.
|
|
void SetPaused(bool paused) {
|
|
std::lock_guard<std::mutex> guard(lock_);
|
|
if (paused == paused_) return;
|
|
if (!paused) kicked_ = true, last_ = {};
|
|
paused_ = paused;
|
|
cv_.notify_one();
|
|
}
|
|
// The list may have changed: read it again soon (not while paused).
|
|
void Kick() {
|
|
std::lock_guard<std::mutex> guard(lock_);
|
|
kicked_ = true;
|
|
cv_.notify_one();
|
|
}
|
|
void Stop() {
|
|
{
|
|
std::lock_guard<std::mutex> guard(lock_);
|
|
running_ = false;
|
|
cv_.notify_one();
|
|
}
|
|
if (thread_.joinable()) thread_.join();
|
|
}
|
|
std::vector<std::string> Keys() {
|
|
std::lock_guard<std::mutex> guard(lock_);
|
|
std::vector<std::string> keys;
|
|
for (const auto &e : entries_) keys.push_back(e.key);
|
|
return keys;
|
|
}
|
|
// A panel just made by another Frametop program (ft-screens: a floating window's menu, the
|
|
// keyboard): known at once, without waiting for the next read ("overlay <key>").
|
|
void Add(const std::string &key) {
|
|
{
|
|
std::lock_guard<std::mutex> guard(lock_);
|
|
for (const auto &e : entries_)
|
|
if (e.key == key) return;
|
|
entries_.push_back({key, key, true});
|
|
}
|
|
++generation_;
|
|
}
|
|
// Goes up by one each time the list is read, so the main loop knows to look the keys up again.
|
|
unsigned Generation() const { return generation_; }
|
|
// Answer `to` with {"t":"overlays","list":[{"key","name","visible"}...]} after the next refresh.
|
|
void Request(const sockaddr_un &to, socklen_t len) {
|
|
if (len <= offsetof(sockaddr_un, sun_path)) return;
|
|
std::lock_guard<std::mutex> guard(lock_);
|
|
if (waiting_.size() < 8) waiting_.push_back({to, len});
|
|
cv_.notify_one();
|
|
}
|
|
|
|
private:
|
|
struct Entry {
|
|
std::string key, name;
|
|
bool visible;
|
|
};
|
|
void Refresh() {
|
|
FILE *p = popen("LD_LIBRARY_PATH=/opt/steamvr/bin/linuxarm64 /opt/steamvr/bin/linuxarm64/vrcmd --overlays 2>/dev/null", "r");
|
|
if (!p) return;
|
|
std::vector<Entry> entries;
|
|
char line[1024];
|
|
while (std::fgets(line, sizeof line, p)) {
|
|
// 'key' -- 'name', WxH visible VROverlayType_...
|
|
if (line[0] != '\'') continue;
|
|
const char *end = std::strchr(line + 1, '\'');
|
|
if (!end) continue;
|
|
const std::string key(line + 1, size_t(end - (line + 1)));
|
|
const std::string rest(end);
|
|
if (rest.find("Thumbnail") != std::string::npos || rest.find("Subview") != std::string::npos) continue;
|
|
if (key.rfind("system.pointer", 0) == 0 || key.rfind("system.cursor", 0) == 0 ||
|
|
key.rfind("frametop.pointer", 0) == 0 || key.rfind("frametop.guide", 0) == 0 ||
|
|
key == "frametop.gazepanel" || // the gaze calibration panel, fixed to the headset
|
|
key == "frametop.catcher" || // ft-screens' release catcher: only on a laser mid-drag
|
|
key == "system.HeadsetView" || key == "system.toast")
|
|
continue;
|
|
// The name can hold quotes; it ends at the last "', " (the size and state follow).
|
|
const auto nameAt = rest.find("-- '"), nameEnd = rest.rfind("', ");
|
|
const std::string name =
|
|
nameAt != std::string::npos && nameEnd > nameAt + 3 ? rest.substr(nameAt + 4, nameEnd - nameAt - 4) : key;
|
|
entries.push_back({key, name, rest.find(" not_visible ") == std::string::npos});
|
|
}
|
|
pclose(p);
|
|
std::vector<std::pair<sockaddr_un, socklen_t>> waiting;
|
|
{
|
|
std::lock_guard<std::mutex> guard(lock_);
|
|
entries_ = std::move(entries);
|
|
waiting.swap(waiting_);
|
|
}
|
|
++generation_;
|
|
if (waiting.empty()) return;
|
|
std::string msg = "{\"t\":\"overlays\",\"list\":[";
|
|
for (size_t i = 0; i < entries_.size(); ++i)
|
|
msg += std::string(i ? "," : "") + "{\"key\":" + JsonQuote(entries_[i].key) + ",\"name\":" +
|
|
JsonQuote(entries_[i].name) + ",\"visible\":" + (entries_[i].visible ? "true" : "false") + "}";
|
|
msg += "]}";
|
|
for (const auto &[to, len] : waiting)
|
|
sendto(out_, msg.data(), msg.size(), MSG_DONTWAIT, reinterpret_cast<const sockaddr *>(&to), len);
|
|
}
|
|
|
|
std::thread thread_;
|
|
int out_ = -1;
|
|
std::mutex lock_; // guards everything below
|
|
std::condition_variable cv_;
|
|
bool paused_ = false, running_ = true, kicked_ = false;
|
|
std::chrono::steady_clock::time_point last_{}; // the last read
|
|
std::atomic<unsigned> generation_{0};
|
|
std::vector<Entry> entries_; // written only by the thread; the lock guards readers
|
|
std::vector<std::pair<sockaddr_un, socklen_t>> waiting_;
|
|
};
|
|
|
|
// POINTER_IGNORE: overlay keys the pointer passes through, as if they weren't there
|
|
// (display-only panels such as a performance overlay). Comma-separated shell patterns
|
|
// (fnmatch, no escapes), so "vendor.app*" covers an app's overlays.
|
|
std::vector<std::string> ParseIgnore(const std::string &list) {
|
|
std::vector<std::string> out;
|
|
size_t at = 0;
|
|
while (at <= list.size()) {
|
|
const size_t comma = std::min(list.find(',', at), list.size());
|
|
std::string item = list.substr(at, comma - at);
|
|
item.erase(0, item.find_first_not_of(" \t"));
|
|
item.erase(item.find_last_not_of(" \t") + 1);
|
|
if (!item.empty()) out.push_back(item);
|
|
at = comma + 1;
|
|
}
|
|
return out;
|
|
}
|
|
|
|
bool Ignored(const std::vector<std::string> &patterns, const std::string &key) {
|
|
for (const auto &p : patterns)
|
|
if (fnmatch(p.c_str(), key.c_str(), FNM_NOESCAPE) == 0) return true;
|
|
return false;
|
|
}
|
|
|
|
vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye);
|
|
|
|
// One of our dots, with what was last set on it: each setter goes to SteamVR only when its value
|
|
// changes (see "Unchanged frames" at the top).
|
|
struct DotOverlay {
|
|
vr::VROverlayHandle_t h = vr::k_ulOverlayHandleInvalid;
|
|
int shown = -1; // unknown at first
|
|
float alpha = -1, width = -1;
|
|
bool placed = false;
|
|
Vec3 at, eye;
|
|
void Show(bool on) {
|
|
if (shown == int(on)) return;
|
|
shown = on;
|
|
if (on) vr::VROverlay()->ShowOverlay(h);
|
|
else vr::VROverlay()->HideOverlay(h);
|
|
}
|
|
void Alpha(float a) {
|
|
if (a != alpha) vr::VROverlay()->SetOverlayAlpha(h, alpha = a);
|
|
}
|
|
// Width w, at `to`, facing `from`: left as it is for a change under 0.5% in width, and while
|
|
// it moved less than 0.2 mm and the eye less than 5 mm.
|
|
void Place(Vec3 to, Vec3 from, float w) {
|
|
if (std::fabs(w - width) > 0.005f * width) vr::VROverlay()->SetOverlayWidthInMeters(h, width = w);
|
|
if (placed && Length(to - at) < 0.0002 && Length(from - eye) < 0.005) return;
|
|
placed = true, at = to, eye = from;
|
|
const auto m = Billboard(to, from);
|
|
vr::VROverlay()->SetOverlayTransformAbsolute(h, vr::TrackingUniverseStanding, &m);
|
|
}
|
|
};
|
|
|
|
vr::HmdMatrix34_t Billboard(Vec3 at, Vec3 eye) {
|
|
// Overlay faces +Z; point +Z at the eye, keep +Y roughly up.
|
|
const Vec3 z = Normalize(eye - at);
|
|
const Vec3 x = Normalize(Cross({0, 1, 0}, z));
|
|
const Vec3 y = Cross(z, x);
|
|
vr::HmdMatrix34_t m{};
|
|
const Vec3 cols[3] = {x, y, z};
|
|
for (int c = 0; c < 3; ++c) {
|
|
m.m[0][c] = float(cols[c].x);
|
|
m.m[1][c] = float(cols[c].y);
|
|
m.m[2][c] = float(cols[c].z);
|
|
}
|
|
m.m[0][3] = float(at.x);
|
|
m.m[1][3] = float(at.y);
|
|
m.m[2][3] = float(at.z);
|
|
return m;
|
|
}
|
|
|
|
std::vector<uint8_t> DotTexture(int size) {
|
|
// White dot with a dark rim, soft edge, transparent outside.
|
|
std::vector<uint8_t> px(size * size * 4, 0);
|
|
const double c = (size - 1) / 2.0, r = size * 0.42, rim = size * 0.10;
|
|
for (int y = 0; y < size; ++y)
|
|
for (int x = 0; x < size; ++x) {
|
|
const double d = std::hypot(x - c, y - c);
|
|
const double a = std::clamp(r - d + 0.5, 0.0, 1.0);
|
|
const bool inner = d < r - rim;
|
|
uint8_t *p = &px[(y * size + x) * 4];
|
|
const uint8_t v = inner ? 255 : 40;
|
|
p[0] = p[1] = p[2] = v;
|
|
p[3] = uint8_t(a * 235);
|
|
}
|
|
return px;
|
|
}
|
|
|
|
// Unit vector v, turned toward unit vector `center` until it's at most maxRad from it.
|
|
Vec3 PullWithin(Vec3 v, Vec3 center, double maxRad) {
|
|
if (std::acos(std::clamp(Dot(v, center), -1.0, 1.0)) <= maxRad) return v;
|
|
Vec3 axis = Cross(center, v);
|
|
if (Length(axis) < 1e-9) axis = Cross(center, {0, 1, 0}); // opposite: any perpendicular
|
|
return RotateAbout(center, Normalize(axis), maxRad);
|
|
}
|
|
|
|
// Unit direction with its pitch limited to +-maxDeg (AimBasis needs it off vertical).
|
|
Vec3 LimitPitch(Vec3 d, double maxDeg) {
|
|
const double pitch = std::clamp(std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI, -maxDeg, maxDeg);
|
|
return Direction(std::atan2(-d.x, -d.z) * 180 / M_PI, pitch);
|
|
}
|
|
|
|
// SteamVR Settings page (see the top): the laser starts this far from the eye, the dot is
|
|
// drawn this far out, and the laser-catching dot sits this far out (metres).
|
|
constexpr double SETTINGS_ORIGIN = 0.25, SETTINGS_DOT = 0.6, SETTINGS_CATCHER = 8.0;
|
|
|
|
// Whether the line of sight from `eye` along `d` crosses the SteamVR Settings page, drawn by
|
|
// the dashboard's scene-graph panel (valve.steam.gamepadui.frame.menu.N, transform t); see
|
|
// "SteamVR Settings" at the top. The page has no size in OpenVR, so this is its area as
|
|
// measured on the Frame, generously: in metres from the panel's origin, which is near the
|
|
// page's left edge, it ran from about -0.35 (the sidebar) to 1.15 across and +-0.4 up and
|
|
// down, with the transform scaled 0.369. Kept in the transform's units so it scales with it.
|
|
bool OnSettingsPage(const vr::HmdMatrix34_t &t, Vec3 eye, Vec3 d) {
|
|
const Vec3 c = Position(t), x{t.m[0][0], t.m[1][0], t.m[2][0]}, y{t.m[0][1], t.m[1][1], t.m[2][1]},
|
|
z{t.m[0][2], t.m[1][2], t.m[2][2]};
|
|
const double sx = Dot(x, x), sy = Dot(y, y), denom = Dot(d, z);
|
|
if (sx < 1e-9 || sy < 1e-9 || std::fabs(denom) < 1e-6) return false;
|
|
const double along = Dot(c - eye, z) / denom;
|
|
if (along <= 0) return false;
|
|
const Vec3 off = eye + d * along - c;
|
|
const double u = Dot(off, x) / sx, v = Dot(off, y) / sy; // in the transform's units
|
|
return u >= -1.35 && u <= 3.4 && std::fabs(v) <= 1.25;
|
|
}
|
|
|
|
} // namespace
|
|
|
|
// Placement speeds (see "Placement" at the top).
|
|
constexpr double kPlaceDegPerSec = 60; // tested: 40 deg/s is applied exactly
|
|
|
|
int main() {
|
|
double freeDistance = 1.5, cursorDeg = 0.4, originFraction = 0.95, originMargin = 0.15, sceneRadius = 0.5,
|
|
edgeReach = 0.3, grabOffset = 0.075,
|
|
slideSpeed = 0.5, leashDeg = 10, leashReturn = 0.2, leashDelay = 0.2, followReach = 70;
|
|
// Head follow (see the top). followConf is POINTER_FOLLOW as last read: a reload only
|
|
// overrides a "follow" command when the setting itself changed.
|
|
bool follow = false, followConf = false, followReset = true;
|
|
// Gaze mode (see the top); gazeConf is POINTER_GAZE as last read, like followConf.
|
|
bool gazeOn = false, gazeConf = false;
|
|
double gazeRetake = 5, gazeNudgeMax = 55, gazeHold = 0.5, gazeShow = 1;
|
|
bool gazeDotAlways = true; // POINTER_GAZE_DOT (see the top)
|
|
double headDeadzone = 0.5, keyTap = 0.25; // POINTER_HEAD_DEADZONE, POINTER_KEY_TAP (keyboard clicks, see the top)
|
|
// Hands (see the top): POINTER_HANDS, POINTER_PINCH_GAIN, POINTER_PINCH_DEADZONE, POINTER_GRIP_GAIN.
|
|
bool handsOn = false;
|
|
double pinchGain = 0.5, pinchDeadzone = 1.5, gripGain = 1.0, handBelow = 0.35, typingHold = 1.0;
|
|
// Gaze precision (see the top): POINTER_GAZE_MOUSE (precision: the mouse's left button
|
|
// holds back its press in gaze mode; direct: it clicks at once), POINTER_ROLE.
|
|
bool gazeMousePrecision = true;
|
|
bool gazeMouseHeld = true; // POINTER_GAZE_MOUSE_MOVE (see the top)
|
|
std::string role = "right";
|
|
double pickupScale = 1; // POINTER_CONTROLLER_PICKUP: scales the controller-moved limits
|
|
std::vector<std::string> ignore; // POINTER_IGNORE (see ParseIgnore)
|
|
auto loadConfig = [&] {
|
|
const auto conf = ReadConfig();
|
|
freeDistance = std::clamp(ConfDouble(conf, "POINTER_DISTANCE", 1.5), 0.3, 10.0);
|
|
cursorDeg = std::clamp(ConfDouble(conf, "POINTER_CURSOR_DEG", 0.4), 0.05, 5.0);
|
|
originFraction = std::clamp(ConfDouble(conf, "POINTER_ORIGIN_FRACTION", 0.95), 0.0, 0.98);
|
|
originMargin = std::clamp(ConfDouble(conf, "POINTER_ORIGIN_MARGIN", 0.15), 0.0, 1.0);
|
|
sceneRadius = std::clamp(ConfDouble(conf, "POINTER_SCENE_RADIUS", 0.5), 0.05, 2.0);
|
|
edgeReach = std::clamp(ConfDouble(conf, "POINTER_EDGE_REACH", 0.3), 0.0, 2.0);
|
|
grabOffset = std::clamp(ConfDouble(conf, "LAYOUT_GRAB_OFFSET", 0.075), 0.0, 1.0);
|
|
slideSpeed = std::clamp(ConfDouble(conf, "LAYOUT_SLIDE_SPEED", 0.5), 0.02, 2.0);
|
|
leashDeg = std::clamp(ConfDouble(conf, "POINTER_LEASH_DEG", 10), 0.0, 90.0);
|
|
leashReturn = std::clamp(ConfDouble(conf, "POINTER_LEASH_RETURN", 0.2), 0.0, 5.0);
|
|
leashDelay = std::clamp(ConfDouble(conf, "POINTER_LEASH_DELAY", 0.2), 0.0, 5.0);
|
|
followReach = std::clamp(ConfDouble(conf, "POINTER_FOLLOW_REACH", 70), 10.0, 89.0);
|
|
const bool wantFollow = ConfDouble(conf, "POINTER_FOLLOW", 0) != 0;
|
|
if (wantFollow != followConf) follow = followConf = wantFollow, followReset = true;
|
|
gazeRetake = std::clamp(ConfDouble(conf, "POINTER_GAZE_RETAKE", 5), 1.0, 45.0);
|
|
gazeNudgeMax = std::clamp(ConfDouble(conf, "POINTER_GAZE_NUDGE_MAX", 55), 1.0, 110.0);
|
|
gazeHold = std::clamp(ConfDouble(conf, "POINTER_GAZE_HOLD", 0.5), 0.1, 5.0);
|
|
gazeShow = std::clamp(ConfDouble(conf, "POINTER_GAZE_SHOW", 1), 0.0, 30.0);
|
|
headDeadzone = std::clamp(ConfDouble(conf, "POINTER_HEAD_DEADZONE", 0.5), 0.0, 5.0);
|
|
keyTap = std::clamp(ConfDouble(conf, "POINTER_KEY_TAP", 0.25), 0.0, 1.0);
|
|
const auto gd = conf.find("POINTER_GAZE_DOT");
|
|
gazeDotAlways = gd == conf.end() || gd->second != "moving";
|
|
const bool wantGaze = ConfDouble(conf, "POINTER_GAZE", 0) != 0;
|
|
if (wantGaze != gazeConf) gazeOn = gazeConf = wantGaze;
|
|
handsOn = ConfDouble(conf, "POINTER_HANDS", 0) != 0;
|
|
pinchGain = std::clamp(ConfDouble(conf, "POINTER_PINCH_GAIN", 0.5), 0.05, 3.0);
|
|
pinchDeadzone = std::clamp(ConfDouble(conf, "POINTER_PINCH_DEADZONE", 1.5), 0.0, 10.0);
|
|
gripGain = std::clamp(ConfDouble(conf, "POINTER_GRIP_GAIN", 1.0), 0.05, 3.0);
|
|
handBelow = std::clamp(ConfDouble(conf, "POINTER_GRIP_BELOW", 0.35), 0.05, 1.0);
|
|
typingHold = std::clamp(ConfDouble(conf, "POINTER_PINCH_TYPING", 1.0), 0.0, 5.0);
|
|
const auto gm = conf.find("POINTER_GAZE_MOUSE");
|
|
gazeMousePrecision = gm == conf.end() || gm->second != "direct";
|
|
const auto mm = conf.find("POINTER_GAZE_MOUSE_MOVE");
|
|
gazeMouseHeld = mm == conf.end() || mm->second != "free";
|
|
const auto ro = conf.find("POINTER_ROLE");
|
|
role = ro != conf.end() && (ro->second == "left" || ro->second == "stylus") ? ro->second : "right";
|
|
pickupScale = std::clamp(ConfDouble(conf, "POINTER_CONTROLLER_PICKUP", 1), 0.5, 5.0);
|
|
const auto ig = conf.find("POINTER_IGNORE");
|
|
ignore = ParseIgnore(ig == conf.end() ? "" : ig->second);
|
|
};
|
|
loadConfig();
|
|
const float laserWidth = float(ConfDouble(ReadConfig(), "POINTER_LASER_WIDTH", 0.8));
|
|
|
|
vr::EVRInitError err = vr::VRInitError_None;
|
|
while (true) {
|
|
vr::VR_Init(&err, vr::VRApplication_Background);
|
|
if (err == vr::VRInitError_None) {
|
|
vr::VR_Shutdown();
|
|
vr::VR_Init(&err, vr::VRApplication_Overlay);
|
|
}
|
|
if (err == vr::VRInitError_None) break;
|
|
std::fprintf(stderr, "waiting for SteamVR: %s\n", vr::VR_GetVRInitErrorAsEnglishDescription(err));
|
|
std::this_thread::sleep_for(std::chrono::seconds(2));
|
|
}
|
|
auto *sys = vr::VRSystem();
|
|
auto *overlay = vr::VROverlay();
|
|
|
|
vr::VROverlayHandle_t cursor = vr::k_ulOverlayHandleInvalid;
|
|
overlay->CreateOverlay("frametop.pointer.cursor", "Frametop pointer", &cursor);
|
|
const int texSize = 64;
|
|
auto tex = DotTexture(texSize);
|
|
overlay->SetOverlayRaw(cursor, tex.data(), texSize, texSize, 4);
|
|
overlay->SetOverlayInputMethod(cursor, vr::VROverlayInputMethod_Mouse); // the laser can land on it
|
|
overlay->SetOverlaySortOrder(cursor, 200);
|
|
// Same dot, not interactive, drawn on panels at the hit point; the laser passes through.
|
|
vr::VROverlayHandle_t marker = vr::k_ulOverlayHandleInvalid;
|
|
overlay->CreateOverlay("frametop.pointer.marker", "Frametop pointer marker", &marker);
|
|
overlay->SetOverlayRaw(marker, tex.data(), texSize, texSize, 4);
|
|
overlay->SetOverlayInputMethod(marker, vr::VROverlayInputMethod_None);
|
|
overlay->SetOverlaySortOrder(marker, 201);
|
|
// Laser mode (see the top of the file).
|
|
vr::VROverlayHandle_t laserMode = vr::k_ulOverlayHandleInvalid;
|
|
overlay->CreateOverlay("frametop.pointer.lasermode", "Frametop pointer laser mode", &laserMode);
|
|
std::vector<uint8_t> clear(4 * 4 * 4, 0);
|
|
overlay->SetOverlayRaw(laserMode, clear.data(), 4, 4, 4);
|
|
overlay->SetOverlayWidthInMeters(laserMode, 0.001f);
|
|
overlay->SetOverlayInputMethod(laserMode, vr::VROverlayInputMethod_Mouse); // the flag needs an input method
|
|
overlay->SetOverlayFlag(laserMode, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, true);
|
|
vr::HmdMatrix34_t below{};
|
|
below.m[0][0] = below.m[1][1] = below.m[2][2] = 1;
|
|
below.m[1][3] = -50;
|
|
overlay->SetOverlayTransformTrackedDeviceRelative(laserMode, vr::k_unTrackedDeviceIndex_Hmd, &below);
|
|
bool laserModeShown = false;
|
|
DotOverlay cursorDot, markerDot; // the setters for `cursor` and `marker` (see "Unchanged frames" at the top)
|
|
cursorDot.h = cursor, markerDot.h = marker;
|
|
// Controller beams keep the user's width; nothing here changes it any more.
|
|
vr::VRSettings()->SetFloat("dashboard", "laserRayWidthScale", laserWidth);
|
|
|
|
const int in = AbstractSocket("ft_pointer_helper", true);
|
|
const int out = AbstractSocket(nullptr, false);
|
|
// Frame controller buttons (vrbuttons.h). The build puts the binary in pointer/helper/build.
|
|
ControllerButtons controllerButtons;
|
|
{
|
|
const std::string manifest = ExeDir() + "/../actions/ft_pointer_actions.json";
|
|
char real[PATH_MAX];
|
|
controllerButtons.Init(realpath(manifest.c_str(), real) ? real : manifest);
|
|
}
|
|
SendTo(out, "frametop_relay", "vrhello"); // the relay answers with the mapped buttons
|
|
OverlayList overlays;
|
|
overlays.Start();
|
|
std::map<std::string, vr::VROverlayHandle_t> handles;
|
|
std::map<std::string, bool> sceneGraph; // no texture: plane test instead of ComputeOverlayIntersection
|
|
std::map<std::string, bool> visible; // refreshed every 50 ms
|
|
auto lastVisible = std::chrono::steady_clock::now();
|
|
auto lastSlow = std::chrono::steady_clock::now() - std::chrono::seconds(10);
|
|
unsigned listGeneration = 0; // the overlay list the handles were looked up from
|
|
// The plane of the last panel the cursor was on, and the last point on it (panel edges).
|
|
Vec3 edgePoint, edgeNormal, edgeLast;
|
|
std::string edgeKey;
|
|
|
|
bool active = false, recenter = false, anchored = false;
|
|
using Clock = std::chrono::steady_clock;
|
|
// Unchanged frames (see the top): visibleGen goes up when an overlay shows or hides or the
|
|
// handles change; pass is the last collision pass, posed the last plain pose sent.
|
|
unsigned visibleGen = 0;
|
|
struct Pass {
|
|
bool valid = false;
|
|
Clock::time_point at{};
|
|
unsigned gen = 0;
|
|
double yaw = 0, pitch = 0;
|
|
Vec3 anchor, eye;
|
|
double best = 1e9, distance = 0;
|
|
std::string key;
|
|
bool scene = false, onEdge = false, occluded = false;
|
|
Vec3 point;
|
|
} pass;
|
|
struct Posed {
|
|
bool valid = false;
|
|
Clock::time_point at{};
|
|
Vec3 origin;
|
|
double yaw = 0, pitch = 0;
|
|
} posed;
|
|
Clock::time_point lastMouse{}, claimAt{}, claimRelease{}, wokeAt{}, noWakeUntil{};
|
|
bool claimPending = false, claimHeld = false;
|
|
// Last used wins: since when each controller has been moving (zero: it isn't).
|
|
Clock::time_point movingSince[vr::k_unMaxTrackedDeviceCount] = {};
|
|
// Tilt mode (see top of file).
|
|
bool leftHeld = false, tilting = false, tiltStart = false, swallowedRight = false;
|
|
// The mouse's buttons in gaze mode (see the top): leftDown, rightDown as the relay last said;
|
|
// aimRight, the held-back press is the right button's; chordDrag, a drag the right button
|
|
// began during the left's held-back press (the first release drops it, the other's is nothing).
|
|
bool leftDown = false, rightDown = false, aimRight = false, chordDrag = false;
|
|
bool ignoreLeftUp = false;
|
|
// The keyboard clicks' keys as the relay last said, and a tilt gaze_right holds during a
|
|
// keyboard drag (see the top): the head turns the panel, from where it was (keyTiltYaw, keyTiltPitch).
|
|
bool keyLeftDown = false, keyRightDown = false, keyTilting = false;
|
|
double keyTiltYaw = 0, keyTiltPitch = 0;
|
|
double tiltYaw = 0, tiltPitch = 0;
|
|
double dragDistance = 0, lastDistance = 1.5; // drag lock: distance from the anchor at the press
|
|
bool onVrSettings = false; // the cursor is on the SteamVR Settings page (kept while dragging)
|
|
bool catcherHidesHit = false; // the laser-catching dot hides SteamVR's hit dot (Settings page)
|
|
Clock::time_point dropHoldUntil{}; // after a left release: keep the drag pose this long
|
|
bool debug = false;
|
|
std::string lastHit;
|
|
// The ft-screens panel the left button was pressed on, and where it was then (see the top).
|
|
std::string pressKey;
|
|
vr::HmdMatrix34_t pressPose{};
|
|
auto lastDebug = Clock::now();
|
|
vr::VROverlayHandle_t systemPointer = vr::k_ulOverlayHandleInvalid;
|
|
overlay->FindOverlay("system.pointer", &systemPointer);
|
|
Vec3 pivot, tiltOrigin, lastPoint, lastOrigin, lastAim{0, 0, -1};
|
|
Basis tiltBasis{};
|
|
bool headsetOff = false; // nobody is wearing the headset (see the main loop)
|
|
auto wake = [&](Clock::time_point t) {
|
|
if (t < noWakeUntil || headsetOff) return;
|
|
wokeAt = t;
|
|
active = true;
|
|
recenter = true;
|
|
// Overlay handles and visibility weren't looked at while it was off (see "Idle" at the top).
|
|
lastSlow = lastVisible = t - std::chrono::seconds(10);
|
|
pass.valid = posed.valid = false;
|
|
SendTo(out, "ft_pointer", "role " + role); // POINTER_ROLE, before it takes it
|
|
SendTo(out, "ft_pointer", "show");
|
|
claimPending = true; // take the laser without clicking, once SteamVR has bound the device
|
|
claimAt = t + std::chrono::milliseconds(300);
|
|
};
|
|
Vec3 anchor;
|
|
double yaw = 0, pitch = 0;
|
|
Vec3 followRef{0, 0, -1}; // head follow's reference direction (see the top)
|
|
auto followAt = std::chrono::steady_clock::now(); // its last update, for the easing
|
|
bool following = false; // past the leash: easing toward the head
|
|
double followLag = 0; // radians the reference trails the head
|
|
std::chrono::steady_clock::time_point leashOutSince{}; // head past the leash since (delay)
|
|
// Gaze mode (see the top).
|
|
struct Gaze {
|
|
double hy = 0, hp = 0, rhy = 0, rhp = 0; // corrected, and raw
|
|
Clock::time_point at{};
|
|
} gz;
|
|
bool gazeOwns = true; // the pointer follows the gaze; false: the mouse has it
|
|
bool nudging = false; // the mouse took it from the gaze: the next click may be a lesson
|
|
double nudgeRawHy = 0, nudgeRawHp = 0, nudgeMoved = 0;
|
|
vr::HmdMatrix34_t nudgeHead{}, lastHead{};
|
|
bool haveHead = false, havePoint = false;
|
|
Clock::time_point nudgeAt{}, retakeSince{};
|
|
// A held-back press (see the top): aimHeld while the button is down; then the click
|
|
// (clickPress at the end of the next frame, clickRelease 40 ms later). gazeBack: give
|
|
// the gaze the pointer again when the press or click is over.
|
|
vr::TrackedDeviceIndex_t ours = vr::k_unTrackedDeviceIndexInvalid;
|
|
bool aimHeld = false, clickPress = false, clickRelease = false, gazeBack = false;
|
|
bool aimHand = false; // the held-back press is a hold's (below): it never turns into a real press
|
|
Clock::time_point aimSince{}, clickReleaseAt{};
|
|
// Holds (see "Gaze precision" and "Hands" at the top): a pinch or grip, or a gaze
|
|
// precision or gaze drag button, held now. What steers the pointer meanwhile (a hand, or
|
|
// the mouse through its own moves), from where it pointed when the hold began (for a
|
|
// hand: seen from the eye then, in the room), and the pointer then.
|
|
HandGestures handFile;
|
|
PoseHistory poses;
|
|
enum class Src { None, Hand, Mouse, Head };
|
|
struct Hold {
|
|
Src src = Src::None;
|
|
int side = -1; // a hand's side (0 left, 1 right)
|
|
bool grip = false; // pressed at once and dragging (a grip, gaze drag), not a click on release
|
|
bool engaged = false; // past the dead zone
|
|
bool pressed = false; // a real press went out (a grip or gaze drag, or a pinch without gaze mode)
|
|
bool promote = false; // held still for POINTER_GAZE_HOLD, it becomes a real press (keyboard clicks)
|
|
bool right = false; // the right button (gaze_right)
|
|
bool keyDrag = false; // gaze_right pressed while gaze_left aimed: a left press, either key ends it
|
|
double pressYaw = 0, pressPitch = 0; // the pointer at the press (a quick tap clicks there)
|
|
Vec3 origin;
|
|
double refYaw = 0, refPitch = 0, startYaw = 0, startPitch = 0, lastYaw = 0, lastPitch = 0;
|
|
} hold;
|
|
// "precision|gazedrag <source> 1|0" from the relay, done in the frame (see Holds).
|
|
struct DevicePress {
|
|
std::string source;
|
|
bool drag, down;
|
|
};
|
|
std::vector<DevicePress> devicePresses;
|
|
// "gazekey left|right 1|0" from the relay (keyboard clicks), done in the frame.
|
|
struct KeyPress {
|
|
bool right, down;
|
|
};
|
|
std::vector<KeyPress> keyPresses;
|
|
uint32_t seenBegins[2][2] = {}, seenEnds[2][2] = {}; // [pinch, grip][side], as last read
|
|
bool handBaseline = false;
|
|
int handOpens = 0;
|
|
uint64_t handSeq = 0, handPublished = 0;
|
|
Clock::time_point handUsed{}; // a gesture began then (keeps the pointer, like gaze mode)
|
|
Clock::time_point lastTyping{}; // the relay's last "typing": a key on a keyboard
|
|
// Gaze mode outside games, and its dot (see the top): lastMove/lastHeld/pulseAt.
|
|
bool inGame = false, gazeAwake = false, gameSent = false;
|
|
Clock::time_point inGameAt{}, gazeAwakeAt{}, gameSentAt{};
|
|
Clock::time_point lastMove{}, lastHeld{}, pulseAt{};
|
|
// The left button, as sent to the driver; pressRight: the next press is the right button
|
|
// instead (gaze_right), heldButton: the one pressed.
|
|
bool pressRight = false;
|
|
std::string heldButton = "trigger";
|
|
bool confirmLesson = false; // a keyboard click's quick tap: a lesson with no correction (see the top)
|
|
// The gaze calibration panel is up until then ("calpanel 1"; see the top); calOpened: it just
|
|
// came up, so a press in progress ends without a click.
|
|
Clock::time_point calPanelUntil{};
|
|
bool calOpened = false;
|
|
auto pressLeft = [&] {
|
|
// ft-screens sends the keyboard to the panel clicked last; it sees clicks on
|
|
// its own screens, but only we know when one lands on another panel.
|
|
SendTo(out, "ft_screens", "click " + (lastHit.empty() ? std::string("-") : lastHit));
|
|
// A click after nudging the gaze-placed pointer: the nudge is a lesson, or past
|
|
// POINTER_GAZE_NUDGE_MAX, a quick check (see the top).
|
|
if (gazeOn && nudging && !gazeOwns && havePoint && Clock::now() - nudgeAt < std::chrono::seconds(10) &&
|
|
(confirmLesson || nudgeMoved >= 0.2)) {
|
|
const Vec3 d = RotateInverse(nudgeHead, Normalize(lastPoint - Position(nudgeHead)));
|
|
const double ty = std::atan2(-d.x, -d.z) * 180 / M_PI, tp = std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI;
|
|
const double off = std::hypot(std::remainder(ty - nudgeRawHy, 360.0), tp - nudgeRawHp);
|
|
char msg[160];
|
|
if (off <= gazeNudgeMax)
|
|
std::snprintf(msg, sizeof msg, "lesson %.3f %.3f %.3f %.3f", nudgeRawHy, nudgeRawHp, ty, tp);
|
|
else
|
|
std::snprintf(msg, sizeof msg, "recheck %.0f", off);
|
|
SendTo(out, "ft_gazed", msg);
|
|
if (debug) std::printf("gaze %s (nudged %.2f deg, off %.2f)\n", msg, nudgeMoved, off);
|
|
if (debug) std::fflush(stdout);
|
|
}
|
|
nudging = confirmLesson = false;
|
|
// A click on nothing: maybe on an overlay that came up since the list was read.
|
|
if (lastHit.empty()) overlays.Kick();
|
|
leftHeld = true;
|
|
dragDistance = lastDistance;
|
|
pressKey.clear();
|
|
if (FramePanel(lastHit)) {
|
|
vr::ETrackingUniverseOrigin uo;
|
|
auto it = handles.find(lastHit);
|
|
if (it != handles.end() &&
|
|
overlay->GetOverlayTransformAbsolute(it->second, &uo, &pressPose) == vr::VROverlayError_None)
|
|
pressKey = lastHit;
|
|
}
|
|
tiltYaw = tiltPitch = 0; // a new drag starts untilted
|
|
dropHoldUntil = {};
|
|
pulseAt = Clock::now();
|
|
heldButton = pressRight ? "b" : "trigger";
|
|
pressRight = false;
|
|
SendTo(out, "ft_pointer", "btn " + heldButton + " 1");
|
|
};
|
|
auto releaseLeft = [&] {
|
|
leftHeld = false;
|
|
tilting = false;
|
|
// Hold the drag pose (tilt, frozen distance) while SteamVR finishes the drop.
|
|
dropHoldUntil = Clock::now() + std::chrono::milliseconds(500);
|
|
SendTo(out, "ft_pointer", "btn " + heldButton + " 0");
|
|
// ft-screens releases a button held on its screens in KWin even when SteamVR hands
|
|
// the release to some other overlay (its catcher usually gets it; this is the backstop).
|
|
SendTo(out, "ft_screens", "up");
|
|
if (gazeBack) gazeOwns = true, gazeBack = false;
|
|
};
|
|
// The left button, from the relay's "btn trigger", with gaze mode's held-back press (see
|
|
// the top).
|
|
auto canAim = [&] {
|
|
return gazeOn && gazeMousePrecision && gazeOwns && !aimHeld && !clickPress && !clickRelease &&
|
|
hold.src == Src::None;
|
|
};
|
|
// Hold the press back: the pointer stops where the gaze put it.
|
|
auto aimStart = [&](bool right) {
|
|
gazeOwns = false;
|
|
nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
|
|
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
|
|
nudgeAt = aimSince = Clock::now(), nudgeMoved = 0;
|
|
aimHeld = true, aimRight = right;
|
|
};
|
|
// A drag the right button began (chordDrag): it lasts while either button is held.
|
|
auto chordDrop = [&] {
|
|
chordDrag = false;
|
|
if (leftHeld) releaseLeft();
|
|
if (debug) std::printf("mouse drag dropped\n");
|
|
if (debug) std::fflush(stdout);
|
|
};
|
|
auto leftButton = [&](bool down) {
|
|
leftDown = down;
|
|
if (down) {
|
|
if (aimHeld && aimRight && !aimHand) {
|
|
ignoreLeftUp = true; // the right's press is held back: the left does nothing
|
|
return;
|
|
}
|
|
if (canAim()) {
|
|
aimStart(false);
|
|
return;
|
|
}
|
|
pressLeft();
|
|
return;
|
|
}
|
|
if (ignoreLeftUp) {
|
|
ignoreLeftUp = false;
|
|
return;
|
|
}
|
|
if (chordDrag) {
|
|
if (!rightDown) chordDrop(); // otherwise the right holds it (tilting, maybe)
|
|
return;
|
|
}
|
|
if (aimHeld && !aimHand && !aimRight) {
|
|
aimHeld = false;
|
|
clickPress = true; // after this frame's pose, so it lands where the pointer was moved to
|
|
gazeBack = nudgeMoved < 0.2;
|
|
return;
|
|
}
|
|
if (leftHeld) releaseLeft();
|
|
};
|
|
// The right button (see the top); false: not taken here (a tilt, or passed on as it is).
|
|
auto rightButton = [&](bool down) {
|
|
rightDown = down;
|
|
if (down) {
|
|
if (aimHeld && !aimHand && !aimRight) {
|
|
// During the left's held-back press: press the left where the pointer is now (the
|
|
// correction is a lesson), and the mouse drags.
|
|
aimHeld = false;
|
|
chordDrag = gazeBack = true;
|
|
pressLeft();
|
|
if (debug) std::printf("mouse drag began (right during left)\n");
|
|
if (debug) std::fflush(stdout);
|
|
return true;
|
|
}
|
|
if (canAim() && !leftHeld) {
|
|
aimStart(true);
|
|
return true;
|
|
}
|
|
return false;
|
|
}
|
|
if (chordDrag) {
|
|
if (leftDown) return !swallowedRight; // the left holds it; a tilt's release ends the tilt
|
|
tilting = swallowedRight = false;
|
|
chordDrop();
|
|
return true;
|
|
}
|
|
if (aimHeld && !aimHand && aimRight) {
|
|
aimHeld = aimRight = false;
|
|
pressRight = clickPress = true; // the right click, where the pointer was moved to
|
|
gazeBack = nudgeMoved < 0.2;
|
|
return true;
|
|
}
|
|
if (leftHeld && heldButton == "b") { // its press, held still into a real one
|
|
releaseLeft();
|
|
return true;
|
|
}
|
|
return false;
|
|
};
|
|
// POINTER_GAZE_MOUSE_MOVE=held (see the top): the gaze is fresh and nothing is pressed, so a
|
|
// mouse move doesn't move the pointer.
|
|
auto mouseMoveHeld = [&] {
|
|
return gazeOn && gazeMouseHeld && !inGame && !headsetOff && Clock::now() - gz.at < std::chrono::seconds(1) &&
|
|
!aimHeld && !leftHeld && !tilting && hold.src == Src::None && !clickPress && !clickRelease;
|
|
};
|
|
|
|
// --- Panel placement (see "Placement" at the top of the file) ---
|
|
// Device pose, given in the standing universe, sent to the driver in raw space.
|
|
auto sendPose = [&](Vec3 originStanding, const Basis &b) {
|
|
vr::TrackedDevicePose_t s, r;
|
|
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &s, 1);
|
|
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseRawAndUncalibrated, 0, &r, 1);
|
|
const auto &S = s.mDeviceToAbsoluteTracking, &R = r.mDeviceToAbsoluteTracking;
|
|
auto toRaw = [&](Vec3 v) { return Rotate(R, RotateInverse(S, v)); };
|
|
const Vec3 o = Position(R) + toRaw(originStanding - Position(S));
|
|
double q[4];
|
|
BasisQuat({toRaw(b.x), toRaw(b.y), toRaw(b.z)}, q);
|
|
char msg[200];
|
|
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", o.x, o.y, o.z, q[0], q[1], q[2], q[3]);
|
|
SendTo(out, "ft_pointer", msg);
|
|
posed.valid = false;
|
|
};
|
|
auto sleepMs = [](int ms) { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); };
|
|
auto headPos = [&] {
|
|
vr::TrackedDevicePose_t s;
|
|
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &s, 1);
|
|
return std::make_pair(s.bPoseIsValid, Position(s.mDeviceToAbsoluteTracking));
|
|
};
|
|
// Borrow the device and the laser for a placement: connect, claim, laser mode on.
|
|
auto borrow = [&] {
|
|
SendTo(out, "ft_pointer", "show");
|
|
overlay->ShowOverlay(laserMode);
|
|
cursorDot.Show(false);
|
|
markerDot.Show(false);
|
|
sleepMs(active ? 50 : 400); // a fresh connect needs SteamVR to bind the device
|
|
SendTo(out, "ft_pointer", "btn a 1");
|
|
sleepMs(60);
|
|
SendTo(out, "ft_pointer", "btn a 0");
|
|
};
|
|
auto giveBack = [&] {
|
|
if (!active) {
|
|
SendTo(out, "ft_pointer", "hide");
|
|
overlay->HideOverlay(laserMode);
|
|
laserModeShown = false;
|
|
}
|
|
};
|
|
auto findPanel = [&](const char *key, Panel &p, Vec3 &eye) -> std::string {
|
|
vr::VROverlayHandle_t h;
|
|
if (overlay->FindOverlay(key, &h) != vr::VROverlayError_None) return std::string("no overlay ") + key;
|
|
bool valid;
|
|
std::tie(valid, eye) = headPos();
|
|
if (!valid) return "no head pose (headset off?)";
|
|
p = ScanPanel(h, eye, 0.5);
|
|
if (!p.found) return std::string("panel not visible: ") + key;
|
|
return "";
|
|
};
|
|
|
|
// grabprobe (maintenance): aim down from the panel's bottom edge, 1 cm a step, and log where
|
|
// SteamVR's laser hits something (the hit dot shows) and whether the window controls are up.
|
|
auto grabProbe = [&](const char *key) {
|
|
Panel p;
|
|
Vec3 eye;
|
|
const std::string err = findPanel(key, p, eye);
|
|
if (!err.empty()) {
|
|
std::printf("grabprobe: %s\n", err.c_str());
|
|
std::fflush(stdout);
|
|
return;
|
|
}
|
|
borrow();
|
|
vr::VROverlayHandle_t footer = vr::k_ulOverlayHandleInvalid;
|
|
overlay->FindOverlay("valve.steam.gamepadui.floatingfooter", &footer);
|
|
const Vec3 bottom = p.center - p.basis.y * (p.height / 2);
|
|
std::printf("grabprobe %s: center (%.3f %.3f %.3f) %.3f x %.3f m\n", key, p.center.x, p.center.y, p.center.z,
|
|
p.width, p.height);
|
|
for (int cm = -5; cm <= 45; ++cm) {
|
|
const Vec3 target = bottom - p.basis.y * (cm / 100.0);
|
|
sendPose(eye, AimBasis(target - eye));
|
|
sleepMs(90);
|
|
const bool dot = systemPointer != vr::k_ulOverlayHandleInvalid && overlay->IsOverlayVisible(systemPointer);
|
|
const bool foot = footer != vr::k_ulOverlayHandleInvalid && overlay->IsOverlayVisible(footer);
|
|
std::printf(" %+3d cm below the bottom edge: steamvr_dot=%d footer=%d", cm, dot, foot);
|
|
if (foot) {
|
|
vr::ETrackingUniverseOrigin uo;
|
|
vr::HmdMatrix34_t t{};
|
|
if (overlay->GetOverlayTransformAbsolute(footer, &uo, &t) == vr::VROverlayError_None) {
|
|
const Vec3 f = Position(t) - p.center;
|
|
std::printf(" footer at panel (%.3f %.3f %.3f)", Dot(f, p.basis.x), Dot(f, p.basis.y),
|
|
Dot(f, p.basis.z));
|
|
}
|
|
}
|
|
std::printf("\n");
|
|
}
|
|
std::fflush(stdout);
|
|
giveBack();
|
|
};
|
|
|
|
// Carry a floating panel so its centre lands on `target` with frame `bt` (see
|
|
// "Placement" at the top). Returns "ok ..." or "error ...".
|
|
auto place = [&](const char *key, Vec3 target, const Basis &bt, double grabBelow) -> std::string {
|
|
Panel p;
|
|
Vec3 eye;
|
|
std::string err = findPanel(key, p, eye);
|
|
if (!err.empty()) return "error " + err;
|
|
auto offBy = [&](const Panel &q, double &cm, double °) {
|
|
cm = Length(q.center - target) * 100;
|
|
const double c = (Dot(q.basis.x, bt.x) + Dot(q.basis.y, bt.y) + Dot(q.basis.z, bt.z) - 1) / 2;
|
|
deg = std::acos(std::clamp(c, -1.0, 1.0)) * 180 / M_PI;
|
|
};
|
|
double cm, deg;
|
|
int moves = 0;
|
|
borrow();
|
|
for (int attempt = 0; attempt < 3; ++attempt) {
|
|
offBy(p, cm, deg);
|
|
if (cm < 1.5 && deg < 1.0) break;
|
|
// The rigid motion that takes the panel to the target: rotate by R, then move.
|
|
auto turn = [&](Vec3 v) { return FromBasis(bt, ToBasis(p.basis, v)); };
|
|
double q[4];
|
|
BasisQuat({turn({1, 0, 0}), turn({0, 1, 0}), turn({0, 0, 1})}, q);
|
|
const double angle = 2 * std::acos(std::clamp(q[0], -1.0, 1.0));
|
|
const Vec3 axis = std::sin(angle / 2) > 1e-6 ? Normalize({q[1], q[2], q[3]}) : Vec3{0, 1, 0};
|
|
const Vec3 grab = p.center - p.basis.y * (p.height / 2 + grabBelow);
|
|
const Basis d0 = AimBasis(grab - eye);
|
|
const Vec3 o1 = target + turn(eye - p.center); // device origin at the end
|
|
++moves;
|
|
sendPose(eye, d0);
|
|
sleepMs(150); // hover: the window controls come up
|
|
SendTo(out, "ft_pointer", "btn trigger 1");
|
|
sleepMs(150);
|
|
// 1. Rotate about the device origin (the eye): the dashboard applies it exactly.
|
|
const int rsteps = std::max(4, int(angle * 180 / M_PI / kPlaceDegPerSec * 60));
|
|
for (int i = 1; i <= rsteps; ++i) {
|
|
const double a = angle * i / rsteps;
|
|
auto r = [&](Vec3 v) { return RotateAbout(v, axis, a); };
|
|
sendPose(eye, {r(d0.x), r(d0.y), r(d0.z)});
|
|
sleepMs(16);
|
|
}
|
|
// 2. Slide the device slowly: fast moves are accelerated by the dashboard.
|
|
const Basis d1{turn(d0.x), turn(d0.y), turn(d0.z)};
|
|
const int tsteps = std::max(4, int(Length(o1 - eye) / slideSpeed * 60));
|
|
for (int i = 1; i <= tsteps; ++i) {
|
|
sendPose(eye + (o1 - eye) * (double(i) / tsteps), d1);
|
|
sleepMs(16);
|
|
}
|
|
sleepMs(100);
|
|
SendTo(out, "ft_pointer", "btn trigger 0");
|
|
sleepMs(600); // the dashboard re-reads the pose up to 150 ms after the release
|
|
err = findPanel(key, p, eye);
|
|
if (!err.empty()) break;
|
|
}
|
|
giveBack();
|
|
if (!err.empty()) return "error after the move: " + err;
|
|
offBy(p, cm, deg);
|
|
char msg[160];
|
|
std::snprintf(msg, sizeof msg, "ok %s off by %.1f cm, %.1f deg after %d move%s", key, cm, deg, moves,
|
|
moves == 1 ? "" : "s");
|
|
std::printf("place: %s\n", msg);
|
|
std::fflush(stdout);
|
|
return msg;
|
|
};
|
|
|
|
// "move <dyaw> <dpitch>" from the relay.
|
|
auto mouseMove = [&](double a, double b) {
|
|
// A move held back (POINTER_GAZE_MOUSE_MOVE=held) only wakes the pointer: it isn't using
|
|
// the mouse, so a drifting mouse doesn't keep the gaze from taking the pointer back.
|
|
const bool heldBack = mouseMoveHeld();
|
|
if (!heldBack) lastMouse = Clock::now();
|
|
// Any mouse input wakes the pointer (after a controller took over, or a helper restart).
|
|
if (!active) wake(Clock::now());
|
|
if (heldBack) return;
|
|
if (tilting) {
|
|
tiltYaw += a;
|
|
tiltPitch = std::clamp(tiltPitch + b, -80.0, 80.0);
|
|
return;
|
|
}
|
|
lastMove = Clock::now(); // the dot shows while the mouse moves it (gaze mode)
|
|
if (gazeOn && gazeOwns) {
|
|
// The mouse takes the pointer from the gaze, from where the gaze left it.
|
|
gazeOwns = false;
|
|
nudging = haveHead && Clock::now() - gz.at < std::chrono::milliseconds(200);
|
|
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
|
|
nudgeAt = Clock::now(), nudgeMoved = 0;
|
|
}
|
|
if (nudging || aimHeld) nudgeMoved += std::hypot(a, b);
|
|
if (!anchored) recenter = true;
|
|
yaw += a;
|
|
while (yaw > 180) yaw -= 360;
|
|
while (yaw < -180) yaw += 360;
|
|
pitch = std::clamp(pitch + b, -85.0, 85.0);
|
|
};
|
|
|
|
std::printf("ft-pointer running: free distance %.2f m, dot %.2f deg\n", freeDistance, cursorDeg);
|
|
std::fflush(stdout);
|
|
|
|
while (true) {
|
|
// The headset off: SteamVR drops the HMD's activity to idle as soon as it comes off.
|
|
// An awake pointer (a connected controller, SteamVR's laser mode forced on) kept the
|
|
// displays from sleeping, so it's released at once, and the mouse can't wake it until
|
|
// the headset is back on (then the first mouse input does).
|
|
const auto level = sys->GetTrackedDeviceActivityLevel(vr::k_unTrackedDeviceIndex_Hmd);
|
|
headsetOff = level == vr::k_EDeviceActivityLevel_Idle || level == vr::k_EDeviceActivityLevel_Standby ||
|
|
level == vr::k_EDeviceActivityLevel_Idle_Timeout;
|
|
if (headsetOff && active) {
|
|
active = false;
|
|
claimPending = claimHeld = false;
|
|
cursorDot.Show(false);
|
|
markerDot.Show(false);
|
|
SendTo(out, "ft_pointer", "btn a 0");
|
|
SendTo(out, "ft_pointer", "hide");
|
|
std::printf("headset off: pointer released\n");
|
|
std::fflush(stdout);
|
|
}
|
|
|
|
// Outside games, gaze mode keeps the pointer (see the top); the relay needs to know.
|
|
{
|
|
const auto t = Clock::now();
|
|
if (t - inGameAt > std::chrono::milliseconds(500)) {
|
|
inGameAt = t;
|
|
inGame = vr::VRApplications()->GetCurrentSceneProcessId() != 0;
|
|
}
|
|
// Hand gestures in the last two minutes keep it the same way.
|
|
const bool handsRecent = handsOn && handUsed != Clock::time_point{} && t - handUsed < std::chrono::minutes(2);
|
|
const bool awake = (gazeOn || handsRecent) && !inGame && !headsetOff;
|
|
if (awake != gazeAwake || t - gazeAwakeAt > std::chrono::seconds(5)) {
|
|
if (awake != gazeAwake)
|
|
std::printf("%s keeps the pointer: %s\n", gazeOn ? "gaze" : "hand use",
|
|
awake ? "yes" : "no (off, in a game, or headset off)");
|
|
if (awake != gazeAwake) std::fflush(stdout);
|
|
gazeAwake = awake;
|
|
gazeAwakeAt = t;
|
|
SendTo(out, "frametop_relay", awake ? "gazeawake 1" : "gazeawake 0");
|
|
}
|
|
// The relay pauses Frametop for VR games (input/game_pause.py). Repeated, so a relay
|
|
// that restarts learns it, and silence (SteamVR gone) ends the game there.
|
|
if (inGame != gameSent || t - gameSentAt > std::chrono::seconds(5)) {
|
|
gameSent = inGame;
|
|
gameSentAt = t;
|
|
SendTo(out, "frametop_relay", inGame ? "vrgame 1" : "vrgame 0");
|
|
}
|
|
}
|
|
|
|
// Commands from the relay.
|
|
char buf[256];
|
|
ssize_t n;
|
|
sockaddr_un from{};
|
|
socklen_t fromLen = sizeof from;
|
|
while ((n = recvfrom(in, buf, sizeof buf - 1, 0, reinterpret_cast<sockaddr *>(&from), &fromLen)) > 0) {
|
|
buf[n] = 0;
|
|
// Reply to the sender (the layout tool binds an abstract address to get answers).
|
|
const sockaddr_un sender = from;
|
|
const socklen_t senderLen = fromLen;
|
|
fromLen = sizeof from;
|
|
auto reply = [&](const std::string &msg) {
|
|
if (senderLen > offsetof(sockaddr_un, sun_path))
|
|
sendto(out, msg.data(), msg.size(), 0, reinterpret_cast<const sockaddr *>(&sender), senderLen);
|
|
};
|
|
// Mouse moves first, the most frequent command.
|
|
double mx, my;
|
|
if (std::strncmp(buf, "move ", 5) == 0 && std::sscanf(buf + 5, "%lf %lf", &mx, &my) == 2) {
|
|
mouseMove(mx, my);
|
|
continue;
|
|
}
|
|
// The gaze, from ft-gazed: not mouse input, it never wakes the pointer.
|
|
double g[4];
|
|
if (std::sscanf(buf, "gz %lf %lf %lf %lf", &g[0], &g[1], &g[2], &g[3]) == 4) {
|
|
gz = {g[0], g[1], g[2], g[3], Clock::now()};
|
|
continue;
|
|
}
|
|
// The gaze calibration panel (see the top): presses answer it instead of clicking.
|
|
{
|
|
int on;
|
|
if (std::sscanf(buf, "calpanel %d", &on) == 1) {
|
|
const bool was = Clock::now() < calPanelUntil;
|
|
calPanelUntil = on ? Clock::now() + std::chrono::seconds(3) : Clock::time_point{};
|
|
if (on && !was) calOpened = true;
|
|
continue;
|
|
}
|
|
}
|
|
if (Clock::now() < calPanelUntil) {
|
|
const bool accept = !std::strncmp(buf, "btn trigger 1", 13) || !std::strncmp(buf, "gazekey left 1", 14);
|
|
const bool quit = !std::strncmp(buf, "btn b 1", 7) || !std::strncmp(buf, "gazekey right 1", 15);
|
|
if (accept || quit) {
|
|
SendTo(out, "ft_gazed", accept ? "calaccept" : "calquit");
|
|
continue;
|
|
}
|
|
if (!std::strncmp(buf, "btn ", 4) || !std::strncmp(buf, "gazekey ", 8) ||
|
|
!std::strncmp(buf, "precision ", 10) || !std::strncmp(buf, "gazedrag ", 9))
|
|
continue; // their releases, and the other buttons: nothing to click now
|
|
}
|
|
{
|
|
char kind[16], source[16];
|
|
int v;
|
|
if (std::sscanf(buf, "%15s %15s %d", kind, source, &v) == 3 &&
|
|
(!std::strcmp(kind, "precision") || !std::strcmp(kind, "gazedrag"))) {
|
|
lastMouse = Clock::now();
|
|
if (!active) wake(Clock::now());
|
|
devicePresses.push_back({source, !std::strcmp(kind, "gazedrag"), v != 0});
|
|
continue;
|
|
}
|
|
if (std::sscanf(buf, "gazekey %15s %d", source, &v) == 2 &&
|
|
(!std::strcmp(source, "left") || !std::strcmp(source, "right"))) {
|
|
lastMouse = Clock::now();
|
|
if (!active) wake(Clock::now());
|
|
keyPresses.push_back({!std::strcmp(source, "right"), v != 0});
|
|
continue;
|
|
}
|
|
}
|
|
if (std::strcmp(buf, "typing") == 0) { // not mouse input: it never wakes the pointer
|
|
lastTyping = Clock::now();
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "vrbind", 6) == 0) {
|
|
std::printf("controller buttons: %s\n", controllerButtons.Bind(buf + 6).c_str());
|
|
std::fflush(stdout);
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "vrglobal", 8) == 0) {
|
|
const char *arg = buf + 8;
|
|
while (*arg == ' ') ++arg;
|
|
ControllerButtons::SetGlobal(std::strncmp(arg, "off", 3) != 0);
|
|
reply(controllerButtons.Status());
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "vrstatus", 8) == 0) {
|
|
reply(controllerButtons.Status());
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "overlay ", 8) == 0 && std::strncmp(buf + 8, "frametop.", 9) == 0) {
|
|
overlays.Add(buf + 8);
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "overlays", 8) == 0) {
|
|
overlays.Request(sender, senderLen); // answered from the list's thread
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "gaze", 4) == 0) {
|
|
const char *arg = buf + 4;
|
|
while (*arg == ' ') ++arg;
|
|
if (*arg != '?') { // "gaze ?" only asks
|
|
gazeOn = std::strncmp(arg, "on", 2) == 0 ? true
|
|
: std::strncmp(arg, "off", 3) == 0 ? false
|
|
: !gazeOn;
|
|
gazeOwns = true, nudging = false;
|
|
std::printf("gaze mode %s\n", gazeOn ? "on" : "off");
|
|
std::fflush(stdout);
|
|
}
|
|
// "gaze ? headset" (the gaze service, which idles while nobody wears the headset)
|
|
// also says whether someone does.
|
|
if (std::strstr(arg, "headset"))
|
|
reply(gazeOn ? (headsetOff ? "ok on away" : "ok on worn") : (headsetOff ? "ok off away" : "ok off worn"));
|
|
else
|
|
reply(gazeOn ? "ok on" : "ok off");
|
|
continue;
|
|
}
|
|
// (Moves were taken first, above.)
|
|
const bool mouseInput = std::strncmp(buf, "btn", 3) == 0 || std::strncmp(buf, "scroll", 6) == 0;
|
|
if (mouseInput) lastMouse = Clock::now();
|
|
// Any mouse input wakes the pointer (after a controller took over, or a helper restart).
|
|
if (!active && mouseInput) wake(Clock::now());
|
|
char key[128];
|
|
double px, py, pz, pyaw, ppitch, proll = 0, pgrab = -1;
|
|
if (std::sscanf(buf, "grabprobe %127s", key) == 1) {
|
|
grabProbe(key);
|
|
continue;
|
|
}
|
|
if (std::sscanf(buf, "place %127s %lf %lf %lf %lf %lf %lf %lf", key, &px, &py, &pz, &pyaw, &ppitch, &proll,
|
|
&pgrab) >= 6) {
|
|
reply(place(key, {px, py, pz}, PanelBasis(pyaw, ppitch, proll), pgrab >= 0 ? pgrab : grabOffset));
|
|
continue;
|
|
}
|
|
if (std::sscanf(buf, "measure %127s", key) == 1) {
|
|
Panel p;
|
|
Vec3 eye;
|
|
const std::string err = findPanel(key, p, eye);
|
|
char msg[400] = "";
|
|
if (err.empty())
|
|
std::snprintf(msg, sizeof msg, "ok %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f %.4f",
|
|
p.center.x, p.center.y, p.center.z, p.width, p.height, p.basis.x.x, p.basis.x.y,
|
|
p.basis.x.z, p.basis.y.x, p.basis.y.y, p.basis.y.z, p.basis.z.x, p.basis.z.y,
|
|
p.basis.z.z);
|
|
reply(err.empty() ? msg : "error " + err);
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "head", 4) == 0) {
|
|
vr::TrackedDevicePose_t h;
|
|
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &h, 1);
|
|
const Vec3 e = Position(h.mDeviceToAbsoluteTracking);
|
|
const Vec3 f = Rotate(h.mDeviceToAbsoluteTracking, {0, 0, -1});
|
|
char msg[200];
|
|
std::snprintf(msg, sizeof msg, "ok %.4f %.4f %.4f %.2f %.2f", e.x, e.y, e.z,
|
|
std::atan2(-f.x, -f.z) * 180 / M_PI, std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI);
|
|
reply(h.bPoseIsValid ? msg : "error no head pose (headset off?)");
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "debug", 5) == 0) {
|
|
debug = !debug;
|
|
std::printf("debug %s\n", debug ? "on" : "off");
|
|
std::fflush(stdout);
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "btn trigger 1", 13) == 0) {
|
|
leftButton(true);
|
|
continue;
|
|
} else if (std::strncmp(buf, "btn trigger 0", 13) == 0) {
|
|
leftButton(false);
|
|
continue;
|
|
} else if (std::strncmp(buf, "btn b 1", 7) == 0 && rightButton(true)) {
|
|
continue;
|
|
} else if (std::strncmp(buf, "btn b 0", 7) == 0 && rightButton(false)) {
|
|
continue;
|
|
} else if (std::strncmp(buf, "btn b 1", 7) == 0 && leftHeld) {
|
|
tilting = tiltStart = swallowedRight = true; // right press while dragging: tilt, no right-click
|
|
continue;
|
|
} else if (std::strncmp(buf, "btn b 0", 7) == 0 && swallowedRight) {
|
|
tilting = swallowedRight = false;
|
|
continue;
|
|
}
|
|
if (std::strncmp(buf, "recenter", 8) == 0) {
|
|
recenter = true;
|
|
} else if (std::strncmp(buf, "reload", 6) == 0) {
|
|
loadConfig();
|
|
lastSlow = Clock::now() - std::chrono::seconds(10); // apply POINTER_IGNORE now
|
|
pass.valid = false;
|
|
std::printf("reloaded: free distance %.2f m, dot %.2f deg, origin %.2f, head follow %s, leash %.0f deg, "
|
|
"controller pickup %.1fx, %zu ignored\n",
|
|
freeDistance, cursorDeg, originFraction, follow ? "on" : "off", leashDeg, pickupScale,
|
|
ignore.size());
|
|
std::fflush(stdout);
|
|
} else if (std::strncmp(buf, "follow", 6) == 0) {
|
|
const char *arg = buf + 6;
|
|
while (*arg == ' ') ++arg;
|
|
const bool was = follow;
|
|
follow = std::strncmp(arg, "on", 2) == 0 ? true : std::strncmp(arg, "off", 3) == 0 ? false : !follow;
|
|
if (follow && !was) followReset = true;
|
|
std::printf("head follow %s (leash %.0f deg)\n", follow ? "on" : "off", leashDeg);
|
|
std::fflush(stdout);
|
|
} else if (std::strncmp(buf, "show", 4) == 0) {
|
|
if (!active) wake(Clock::now());
|
|
} else if (std::strncmp(buf, "hide", 4) == 0) {
|
|
active = false;
|
|
cursorDot.Show(false);
|
|
markerDot.Show(false);
|
|
SendTo(out, "ft_pointer", "hide");
|
|
} else {
|
|
SendTo(out, "ft_pointer", buf); // btn, scroll
|
|
}
|
|
}
|
|
|
|
vr::TrackedDevicePose_t all[vr::k_unMaxTrackedDeviceCount];
|
|
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0.011f, all, vr::k_unMaxTrackedDeviceCount);
|
|
const vr::TrackedDevicePose_t &hmd = all[0];
|
|
vr::TrackedDevicePose_t hmdRaw;
|
|
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseRawAndUncalibrated, 0.011f, &hmdRaw, 1);
|
|
const auto tnow = Clock::now();
|
|
|
|
// Claim pulse (switchlaserhand on the driver's "a" button, no click).
|
|
if (claimPending && tnow >= claimAt) {
|
|
SendTo(out, "ft_pointer", "btn a 1");
|
|
claimPending = false;
|
|
claimHeld = true;
|
|
claimRelease = tnow + std::chrono::milliseconds(60);
|
|
} else if (claimHeld && tnow >= claimRelease) {
|
|
SendTo(out, "ft_pointer", "btn a 0");
|
|
claimHeld = false;
|
|
}
|
|
|
|
// The overlay list is only needed while the pointer is awake (see OverlayList).
|
|
overlays.SetPaused(!active || headsetOff);
|
|
|
|
// Laser mode on while the pointer is awake.
|
|
if (active != laserModeShown) {
|
|
laserModeShown = active;
|
|
if (active) overlay->ShowOverlay(laserMode);
|
|
else overlay->HideOverlay(laserMode);
|
|
if (debug) std::printf("laser mode %s\n", active ? "forced on" : "released");
|
|
if (debug) std::fflush(stdout);
|
|
}
|
|
|
|
// Didn't get the hand role (a held controller keeps it): release, back off.
|
|
if (active && tnow - wokeAt > std::chrono::seconds(1) && ours != vr::k_unTrackedDeviceIndexInvalid &&
|
|
sys->GetControllerRoleForTrackedDeviceIndex(ours) == vr::TrackedControllerRole_Invalid) {
|
|
active = false;
|
|
claimPending = claimHeld = false;
|
|
cursorDot.Show(false);
|
|
markerDot.Show(false);
|
|
SendTo(out, "ft_pointer", "btn a 0");
|
|
SendTo(out, "ft_pointer", "hide");
|
|
noWakeUntil = tnow + std::chrono::seconds(2);
|
|
std::printf("no hand role (a controller is in use): pointer released\n");
|
|
std::fflush(stdout);
|
|
}
|
|
|
|
// Last used wins: a real controller being moved releases the pointer (see the top),
|
|
// in gaze mode too.
|
|
if (active && hold.src == Src::None && tnow - lastMouse > std::chrono::milliseconds(500)) {
|
|
for (vr::TrackedDeviceIndex_t i = 1; i < vr::k_unMaxTrackedDeviceCount; ++i) {
|
|
if (i == ours || !all[i].bPoseIsValid || all[i].eTrackingResult != vr::TrackingResult_Running_OK ||
|
|
sys->GetTrackedDeviceClass(i) != vr::TrackedDeviceClass_Controller) {
|
|
movingSince[i] = {};
|
|
continue;
|
|
}
|
|
const auto &v = all[i].vVelocity.v, &w = all[i].vAngularVelocity.v;
|
|
const double speed = std::sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2]);
|
|
const double spin = std::sqrt(w[0] * w[0] + w[1] * w[1] + w[2] * w[2]);
|
|
if (speed <= 0.35 * pickupScale && spin <= 2.0 * pickupScale) {
|
|
movingSince[i] = {};
|
|
continue;
|
|
}
|
|
if (movingSince[i] == Clock::time_point{}) movingSince[i] = tnow;
|
|
if (tnow - movingSince[i] >= std::chrono::milliseconds(100)) {
|
|
active = false;
|
|
claimPending = claimHeld = false;
|
|
cursorDot.Show(false);
|
|
markerDot.Show(false);
|
|
SendTo(out, "ft_pointer", "btn a 0");
|
|
SendTo(out, "ft_pointer", "hide");
|
|
std::printf("controller %u moved (%.2f m/s, %.1f rad/s): pointer released\n", i, speed, spin);
|
|
std::fflush(stdout);
|
|
break;
|
|
}
|
|
}
|
|
} else {
|
|
std::fill(std::begin(movingSince), std::end(movingSince), Clock::time_point{});
|
|
}
|
|
const auto &hm = hmd.mDeviceToAbsoluteTracking.m;
|
|
const Vec3 eye{hm[0][3], hm[1][3], hm[2][3]};
|
|
|
|
if (recenter && hmd.bPoseIsValid) {
|
|
anchor = eye;
|
|
const Vec3 f{-hm[0][2], -hm[1][2], -hm[2][2]};
|
|
yaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
|
|
pitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
|
|
anchored = true;
|
|
recenter = false;
|
|
followReset = true;
|
|
}
|
|
|
|
// Gaze mode (see the top): the gaze has the pointer, or takes it back when you look
|
|
// well away from it. Not while a press holds the pointer, and only on fresh gaze.
|
|
if (hmd.bPoseIsValid) lastHead = hmd.mDeviceToAbsoluteTracking, haveHead = true;
|
|
if (gazeOn && active && hmd.bPoseIsValid && !tilting && !leftHeld && !aimHeld && !clickPress && !clickRelease &&
|
|
tnow >= dropHoldUntil &&
|
|
tnow - gz.at < std::chrono::milliseconds(150)) {
|
|
const Vec3 g = Rotate(hmd.mDeviceToAbsoluteTracking, Direction(gz.hy, gz.hp));
|
|
if (!gazeOwns && havePoint) {
|
|
const double off = std::acos(std::clamp(Dot(g, Normalize(lastPoint - eye)), -1.0, 1.0)) * 180 / M_PI;
|
|
if (off > gazeRetake && tnow - lastMouse > std::chrono::milliseconds(300)) {
|
|
if (retakeSince == Clock::time_point{}) retakeSince = tnow;
|
|
if (tnow - retakeSince >= std::chrono::milliseconds(120)) gazeOwns = true, nudging = false;
|
|
} else {
|
|
retakeSince = {};
|
|
}
|
|
}
|
|
if (gazeOwns) {
|
|
retakeSince = {};
|
|
anchor = eye;
|
|
anchored = true;
|
|
yaw = std::atan2(-g.x, -g.z) * 180 / M_PI;
|
|
pitch = std::clamp(std::asin(std::clamp(g.y, -1.0, 1.0)) * 180 / M_PI, -85.0, 85.0);
|
|
}
|
|
}
|
|
|
|
// Hands (see the top): pinches and grips from ft-hands.
|
|
if (handsOn) {
|
|
vr::TrackedDevicePose_t h0;
|
|
sys->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, &h0, 1);
|
|
if (h0.bPoseIsValid) poses.Add(tnow, h0.mDeviceToAbsoluteTracking);
|
|
}
|
|
// Where a gesture's point is, seen from the hold's origin: yaw and pitch, degrees.
|
|
auto handAngles = [&](const float p[3], uint64_t t_ns, const Vec3 &from, double &hy, double &hp) {
|
|
vr::HmdMatrix34_t head;
|
|
if (!poses.At(t_ns, head)) return false;
|
|
const Vec3 d = Normalize(Position(head) + Rotate(head, Vec3{p[0], p[1], p[2]}) - from);
|
|
hy = std::atan2(-d.x, -d.z) * 180 / M_PI;
|
|
hp = std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI;
|
|
return true;
|
|
};
|
|
auto endHold = [&](bool lost) {
|
|
if (hold.pressed) {
|
|
if (leftHeld) releaseLeft();
|
|
} else if (aimHeld) {
|
|
aimHeld = aimHand = false;
|
|
if (lost) {
|
|
if (gazeOn) gazeOwns = true, nudging = false; // no click: the pointer goes back to the gaze
|
|
} else {
|
|
clickPress = true; // the click is on the release, where the pointer is now
|
|
pressRight = hold.right;
|
|
gazeBack = nudgeMoved < 0.2;
|
|
}
|
|
}
|
|
if (debug)
|
|
std::printf("hold %s %s%s\n", hold.src == Src::Hand ? (hold.grip ? "grip" : "pinch")
|
|
: hold.src == Src::Head ? (hold.right ? "key right" : "key left")
|
|
: hold.grip ? "gaze drag" : "gaze precision",
|
|
lost ? "lost" : "released", hold.pressed ? "" : lost ? " (no click)" : " (click)");
|
|
if (debug) std::fflush(stdout);
|
|
hold = {};
|
|
};
|
|
// The press, once a hold's source is set: a real press (dragging until the release), or
|
|
// held back like gaze mode's mouse press (see the top): the click comes on the release.
|
|
auto startHold = [&](bool press) {
|
|
hold.startYaw = hold.lastYaw = yaw, hold.startPitch = hold.lastPitch = pitch;
|
|
hold.pressed = press;
|
|
if (press) {
|
|
gazeBack = gazeOn && gazeOwns;
|
|
pressLeft();
|
|
} else {
|
|
gazeOwns = false;
|
|
nudging = gazeOn && haveHead && tnow - gz.at < std::chrono::milliseconds(200);
|
|
nudgeRawHy = gz.rhy, nudgeRawHp = gz.rhp, nudgeHead = lastHead;
|
|
nudgeAt = aimSince = tnow, nudgeMoved = 0;
|
|
aimHeld = aimHand = true;
|
|
pulseAt = tnow;
|
|
}
|
|
};
|
|
// The hold's source moved to (hy, hp): past the dead zone, the pointer follows at the gain,
|
|
// from where it was when it got past.
|
|
auto steer = [&](double hy, double hp, double deadzone, double gain) {
|
|
const double dy = std::remainder(hy - hold.refYaw, 360.0), dp = hp - hold.refPitch;
|
|
if (!hold.engaged) {
|
|
if (std::hypot(dy, dp) <= deadzone) return;
|
|
hold.engaged = true;
|
|
hold.refYaw = hy, hold.refPitch = hp;
|
|
hold.startYaw = hold.lastYaw = yaw, hold.startPitch = hold.lastPitch = pitch;
|
|
return;
|
|
}
|
|
yaw = std::remainder(hold.startYaw + gain * dy, 360.0);
|
|
pitch = std::clamp(hold.startPitch + gain * dp, -85.0, 85.0);
|
|
// How far the nudge went: for a keyboard click, from where the dot was at the press
|
|
// (a head wobbles on the way); otherwise the path, as with the mouse.
|
|
if (!hold.pressed && hold.src == Src::Head)
|
|
nudgeMoved = std::hypot(std::remainder(yaw - hold.pressYaw, 360.0), pitch - hold.pressPitch);
|
|
else if (!hold.pressed)
|
|
nudgeMoved += std::hypot(std::remainder(yaw - hold.lastYaw, 360.0), pitch - hold.lastPitch);
|
|
hold.lastYaw = yaw, hold.lastPitch = pitch;
|
|
lastMove = tnow; // the dot shows while it moves (gaze mode)
|
|
};
|
|
auto beginHold = [&](int side, bool grip, const fh_pinch_t &g) {
|
|
handUsed = tnow;
|
|
if (hold.src != Src::None) {
|
|
// A grip takes over a pinch on the way to it (closing the hand passes through
|
|
// one); otherwise one hold at a time.
|
|
if (hold.src != Src::Hand || !grip || hold.grip) return;
|
|
aimHeld = aimHand = false;
|
|
hold = {};
|
|
}
|
|
if (leftHeld || aimHeld || clickPress || clickRelease) return; // the mouse's button is busy
|
|
if (!active) {
|
|
wake(tnow); // the first gesture only wakes the pointer, like the first mouse move
|
|
return;
|
|
}
|
|
vr::HmdMatrix34_t head;
|
|
if (!poses.At(g.begin_ns, head)) return;
|
|
// No pinch just after a key (typing touches thumb to index), and a grip only with
|
|
// the hand held up (hands on a desk curl like a loose fist; looking down at them
|
|
// puts them straight ahead in the head's frame, where ft-hands can't tell).
|
|
const Vec3 at = Position(head) + Rotate(head, Vec3{g.begin_point[0], g.begin_point[1], g.begin_point[2]});
|
|
if (!grip && tnow - lastTyping < std::chrono::duration<double>(typingHold)) {
|
|
if (debug) std::printf("hand pinch ignored: typing\n");
|
|
return;
|
|
}
|
|
if (grip && Position(head).y - at.y > handBelow) {
|
|
if (debug) std::printf("hand grip ignored: %.2f m below the eyes\n", Position(head).y - at.y);
|
|
return;
|
|
}
|
|
Hold h;
|
|
h.src = Src::Hand, h.side = side, h.grip = grip, h.origin = Position(head);
|
|
if (!handAngles(g.begin_point, g.begin_ns, h.origin, h.refYaw, h.refPitch)) return;
|
|
hold = h;
|
|
// A grip, or a pinch without gaze mode: a real press where the pointer is (where
|
|
// you look, in gaze mode), dragging with the hand until it opens.
|
|
startHold(grip || !gazeOn);
|
|
if (debug) std::printf("hand %s %s began\n", side ? "right" : "left", grip ? "grip" : "pinch");
|
|
if (debug) std::fflush(stdout);
|
|
};
|
|
// Gaze precision and gaze drag buttons (see the top): the mouse steers.
|
|
for (const DevicePress &p : devicePresses) {
|
|
if (p.source == "left" || p.source == "right") continue; // no controllers in gaze mode
|
|
if (!p.down) {
|
|
if (hold.src == Src::Mouse) endHold(false);
|
|
continue;
|
|
}
|
|
if (hold.src != Src::None || leftHeld || aimHeld || clickPress || clickRelease) continue;
|
|
Hold h;
|
|
h.src = Src::Mouse, h.grip = p.drag;
|
|
hold = h;
|
|
startHold(p.drag);
|
|
if (debug) std::printf("hold gaze %s began (%s)\n", p.drag ? "drag" : "precision", p.source.c_str());
|
|
if (debug) std::fflush(stdout);
|
|
}
|
|
devicePresses.clear();
|
|
// Keyboard clicks (see the top): held back at the gaze, steered by the head.
|
|
auto headAngles = [&](double &hy, double &hp) {
|
|
if (!hmd.bPoseIsValid) return false;
|
|
const Vec3 f = Normalize({-hm[0][2], -hm[1][2], -hm[2][2]});
|
|
hy = std::atan2(-f.x, -f.z) * 180 / M_PI;
|
|
hp = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
|
|
return true;
|
|
};
|
|
for (const KeyPress &k : keyPresses) {
|
|
(k.right ? keyRightDown : keyLeftDown) = k.down;
|
|
if (!k.down) {
|
|
if (keyTilting && k.right) { // the tilt ends; the head drags again from here
|
|
keyTilting = tilting = false;
|
|
hold.engaged = false;
|
|
if (debug) std::printf("hold key left: tilt ended\n");
|
|
if (debug) std::fflush(stdout);
|
|
}
|
|
// The keys holding it: a gaze_left then gaze_right drag, either; otherwise its own.
|
|
const bool held = hold.keyDrag ? keyLeftDown || keyRightDown : hold.right ? keyRightDown : keyLeftDown;
|
|
if (hold.src == Src::Head && !held) {
|
|
if (!hold.pressed && aimHeld && tnow - aimSince < std::chrono::duration<double>(keyTap)) {
|
|
// A quick tap: a click where the dot was at the press, and the gaze was right.
|
|
yaw = hold.pressYaw, pitch = hold.pressPitch;
|
|
nudgeMoved = 0;
|
|
confirmLesson = true;
|
|
}
|
|
endHold(false);
|
|
keyTilting = false;
|
|
}
|
|
continue;
|
|
}
|
|
if (k.right && hold.src == Src::Head && !hold.right && hold.pressed && leftHeld) {
|
|
// gaze_right during a gaze_left drag: tilt while it's held (see the top).
|
|
tilting = tiltStart = keyTilting = true;
|
|
headAngles(keyTiltYaw, keyTiltPitch);
|
|
if (debug) std::printf("hold key left: tilt began\n");
|
|
if (debug) std::fflush(stdout);
|
|
continue;
|
|
}
|
|
if (k.right && hold.src == Src::Head && !hold.right && !hold.pressed && aimHeld) {
|
|
// gaze_right while gaze_left aims: press the left button where the dot is now
|
|
// (the correction is a lesson), then the head drags.
|
|
aimHeld = aimHand = false;
|
|
hold.pressed = hold.grip = hold.keyDrag = true, hold.promote = false, hold.engaged = false;
|
|
headAngles(hold.refYaw, hold.refPitch);
|
|
gazeBack = gazeOn;
|
|
pressRight = false;
|
|
pressLeft();
|
|
if (debug) std::printf("hold key left: pressed (right key)\n");
|
|
if (debug) std::fflush(stdout);
|
|
continue;
|
|
}
|
|
if (hold.src != Src::None || leftHeld || aimHeld || clickPress || clickRelease) continue;
|
|
Hold h;
|
|
h.src = Src::Head, h.promote = true, h.right = k.right, h.pressYaw = yaw, h.pressPitch = pitch;
|
|
if (!headAngles(h.refYaw, h.refPitch)) continue;
|
|
hold = h;
|
|
startHold(false);
|
|
if (debug) std::printf("hold key %s began\n", k.right ? "right" : "left");
|
|
if (debug) std::fflush(stdout);
|
|
}
|
|
keyPresses.clear();
|
|
if (calOpened) { // the calibration panel came up: a press in progress ends, no click
|
|
calOpened = false;
|
|
if (hold.src != Src::None) endHold(true);
|
|
if (leftHeld) releaseLeft();
|
|
aimHeld = aimHand = clickPress = false;
|
|
}
|
|
if (hold.src == Src::Head) {
|
|
double hy, hp;
|
|
if (headAngles(hy, hp) && keyTilting) {
|
|
// The head turns the panel, as the mouse does in a tilt; the pointer stays put.
|
|
tiltYaw += std::remainder(hy - keyTiltYaw, 360.0);
|
|
tiltPitch = std::clamp(tiltPitch + hp - keyTiltPitch, -80.0, 80.0);
|
|
keyTiltYaw = hy, keyTiltPitch = hp;
|
|
} else if (headAngles(hy, hp)) {
|
|
steer(hy, hp, hold.pressed ? 0.3 : headDeadzone, 1.0);
|
|
}
|
|
}
|
|
fh_gestures_t hg;
|
|
const bool handOk = handsOn && handFile.Read(hg);
|
|
if (handOk && (hg.seq != handSeq || handFile.opens != handOpens)) {
|
|
handSeq = hg.seq;
|
|
handPublished = hg.publish_ns;
|
|
const fh_pinch_t *slots[2] = {hg.pinch, hg.grip};
|
|
// A new file, or a tracker whose counters went back: start counting from here.
|
|
bool rebase = !handBaseline || handFile.opens != handOpens;
|
|
for (int k = 0; k < 2; ++k)
|
|
for (int s = 0; s < 2; ++s)
|
|
rebase = rebase || slots[k][s].begins < seenBegins[k][s] || slots[k][s].ends < seenEnds[k][s];
|
|
if (rebase) {
|
|
if (hold.src == Src::Hand) endHold(true);
|
|
for (int k = 0; k < 2; ++k)
|
|
for (int s = 0; s < 2; ++s) seenBegins[k][s] = slots[k][s].begins, seenEnds[k][s] = slots[k][s].ends;
|
|
handBaseline = true, handOpens = handFile.opens;
|
|
}
|
|
// Not over a VR game (unless the dashboard is up), and not with the headset off.
|
|
const bool allowed = !headsetOff && (!inGame || overlay->IsDashboardVisible());
|
|
for (int k = 1; k >= 0; --k) // grips first: one takes over a pinch
|
|
for (int s = 0; s < 2; ++s) {
|
|
const fh_pinch_t &g = slots[k][s];
|
|
const bool began = g.begins != seenBegins[k][s], ended = g.ends != seenEnds[k][s];
|
|
const bool lost = g.flags & FH_PINCH_LOST;
|
|
seenBegins[k][s] = g.begins, seenEnds[k][s] = g.ends;
|
|
auto holding = [&] { return hold.src == Src::Hand && hold.side == s && hold.grip == (k == 1); };
|
|
if (holding() && ended) endHold(lost); // the one held ended (another may have begun)
|
|
if (began && allowed) {
|
|
beginHold(s, k == 1, g);
|
|
// begun and ended since the last read (a quick tap): its release too
|
|
if (!(g.flags & FH_PINCH_DOWN) && holding()) endHold(lost);
|
|
}
|
|
}
|
|
// The held gesture's hand moves the pointer: past the dead zone (a tap's jitter,
|
|
// the pinch point shifting as the fingers close), then at the gain, from there.
|
|
if (hold.src == Src::Hand) {
|
|
const fh_pinch_t &g = hold.grip ? hg.grip[hold.side] : hg.pinch[hold.side];
|
|
double hy, hp;
|
|
if ((g.flags & FH_PINCH_TRACKED) && handAngles(g.point, hg.capture_ns, hold.origin, hy, hp))
|
|
steer(hy, hp, hold.grip ? 0.5 : pinchDeadzone, hold.grip ? gripGain : pinchGain);
|
|
}
|
|
}
|
|
// ft-hands stopped (or hung) with a gesture held: it's over, as lost.
|
|
const uint64_t nowNs =
|
|
uint64_t(std::chrono::duration_cast<std::chrono::nanoseconds>(tnow.time_since_epoch()).count());
|
|
if (hold.src == Src::Hand && (!handOk || !active || nowNs - handPublished > 1'500'000'000ull)) endHold(true);
|
|
if (hold.src != Src::None && hold.src != Src::Hand && !active) endHold(true);
|
|
|
|
if (aimHeld || leftHeld || clickPress || clickRelease) lastHeld = tnow;
|
|
|
|
// Head follow (see the top): past the leash (for the delay), ease the reference to the
|
|
// head's facing, and turn the cursor with it. Not in gaze mode: the gaze places it.
|
|
if (follow && !gazeOn && active && anchored && hmd.bPoseIsValid) {
|
|
const Vec3 head = LimitPitch(Vec3{-hm[0][2], -hm[1][2], -hm[2][2]}, 85);
|
|
if (followReset) {
|
|
followRef = head, followLag = 0, leashOutSince = {};
|
|
followReset = following = false;
|
|
}
|
|
const double dt = std::min(0.1, std::chrono::duration<double>(tnow - followAt).count());
|
|
const double leash = leashDeg * M_PI / 180;
|
|
const double lag = std::acos(std::clamp(Dot(followRef, head), -1.0, 1.0));
|
|
double keep = lag; // how far the reference stays behind the head after this frame
|
|
if (leash <= 0) {
|
|
keep = 0; // head-locked
|
|
} else if (following) {
|
|
keep = lag * (leashReturn > 0 ? std::exp(-dt / leashReturn) : 0.0);
|
|
// A fast turn drags it at the leash's end; past it already (after the delay), it
|
|
// can't fall further behind, and it closes in from there without a jump.
|
|
keep = std::min(keep, std::max(leash, followLag));
|
|
if (keep < 0.05 * M_PI / 180) keep = 0, following = false; // landed on the facing
|
|
} else if (lag > leash) {
|
|
if (leashOutSince == decltype(leashOutSince){}) leashOutSince = tnow;
|
|
if (std::chrono::duration<double>(tnow - leashOutSince).count() >= leashDelay)
|
|
following = true, leashOutSince = {};
|
|
} else {
|
|
leashOutSince = {}; // back inside before the delay: a glance
|
|
}
|
|
followLag = keep;
|
|
const Vec3 ref = LimitPitch(PullWithin(followRef, head, keep), 85);
|
|
if (!leftHeld && tnow >= dropHoldUntil) {
|
|
// The cursor keeps its offset from the reference (frames without roll).
|
|
Vec3 d = FromBasis(AimBasis(ref), ToBasis(AimBasis(followRef), Direction(yaw, pitch)));
|
|
d = PullWithin(Normalize(d), ref, followReach * M_PI / 180);
|
|
yaw = std::atan2(-d.x, -d.z) * 180 / M_PI;
|
|
pitch = std::clamp(std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI, -85.0, 85.0);
|
|
// The ray origin closes in on the eye as the reference does on the facing.
|
|
anchor = eye + (anchor - eye) * (leash <= 0 ? 0.0 : lag > 1e-6 ? keep / lag : following ? 0.0 : 1.0);
|
|
}
|
|
followRef = ref;
|
|
}
|
|
followAt = tnow;
|
|
|
|
// Slow work, once a second and when the overlay list is new: overlay handles, our device
|
|
// index. Only while the pointer is awake (see "Idle" at the top).
|
|
const auto now = std::chrono::steady_clock::now();
|
|
if (active && (now - lastSlow > std::chrono::seconds(1) || overlays.Generation() != listGeneration)) {
|
|
lastSlow = now;
|
|
listGeneration = overlays.Generation();
|
|
const auto before = handles;
|
|
handles.clear();
|
|
for (const auto &key : overlays.Keys()) {
|
|
if (Ignored(ignore, key)) continue;
|
|
vr::VROverlayHandle_t h;
|
|
if (overlay->FindOverlay(key.c_str(), &h) != vr::VROverlayError_None) continue;
|
|
handles[key] = h;
|
|
// Scene-graph overlays are placed absolutely. Other overlays can report no
|
|
// texture too (gamescope's app panels, placed as dashboard tabs, share theirs
|
|
// from another process), and ComputeOverlayIntersection handles those.
|
|
uint32_t tw = 0, th = 0;
|
|
overlay->GetOverlayTextureSize(h, &tw, &th);
|
|
vr::VROverlayTransformType tt = vr::VROverlayTransform_Invalid;
|
|
overlay->GetOverlayTransformType(h, &tt);
|
|
// ft-screens' panels (frametop.screen.N, floating windows with their popups,
|
|
// frametop.float.N..., the keyboard) are 0x0 and absolute too (a shared
|
|
// texture), but they're real panels of any size.
|
|
sceneGraph[key] = (tw == 0 || th == 0) && tt == vr::VROverlayTransform_Absolute &&
|
|
key.rfind("frametop.", 0) != 0;
|
|
}
|
|
if (handles != before) ++visibleGen;
|
|
// Our device keeps its index; it's looked for again when a device is activated or
|
|
// deactivated (the events below).
|
|
for (vr::TrackedDeviceIndex_t i = 0; ours == vr::k_unTrackedDeviceIndexInvalid && i < vr::k_unMaxTrackedDeviceCount; ++i) {
|
|
char type[64] = "";
|
|
sys->GetStringTrackedDeviceProperty(i, vr::Prop_ControllerType_String, type, sizeof type);
|
|
if (std::strcmp(type, "ft_pointer") == 0) ours = i;
|
|
}
|
|
}
|
|
|
|
if (active && (now - lastVisible > std::chrono::milliseconds(50) || visible.size() != handles.size())) {
|
|
lastVisible = now;
|
|
std::map<std::string, bool> seen;
|
|
for (const auto &[key, h] : handles) seen[key] = overlay->IsOverlayVisible(h);
|
|
if (seen != visible) visible.swap(seen), ++visibleGen;
|
|
}
|
|
|
|
if (active && anchored && hmd.bPoseIsValid && tilting) {
|
|
// Rotate the device around the grab point; the grabbed panel turns with it.
|
|
if (tiltStart) {
|
|
pivot = lastPoint;
|
|
tiltOrigin = lastOrigin;
|
|
tiltBasis = AimBasis(lastAim);
|
|
tiltStart = false; // angles carry on from any earlier tilt in this drag
|
|
cursorDot.Show(false);
|
|
markerDot.Show(false);
|
|
}
|
|
const double yr = tiltYaw * M_PI / 180, pr = tiltPitch * M_PI / 180;
|
|
const Vec3 up{0, 1, 0}, side = tiltBasis.x;
|
|
auto turn = [&](Vec3 v) { return RotateAbout(RotateAbout(v, side, pr), up, yr); };
|
|
const Vec3 originStanding = pivot + turn(tiltOrigin - pivot);
|
|
const Basis b{turn(tiltBasis.x), turn(tiltBasis.y), turn(tiltBasis.z)};
|
|
const auto &S = hmd.mDeviceToAbsoluteTracking, &R = hmdRaw.mDeviceToAbsoluteTracking;
|
|
auto toRaw = [&](Vec3 v) { return Rotate(R, RotateInverse(S, v)); }; // directions: raw <- standing
|
|
const Vec3 originRaw = Position(R) + toRaw(originStanding - eye);
|
|
double q[4];
|
|
BasisQuat({toRaw(b.x), toRaw(b.y), toRaw(b.z)}, q);
|
|
char msg[200];
|
|
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", originRaw.x, originRaw.y,
|
|
originRaw.z, q[0], q[1], q[2], q[3]);
|
|
SendTo(out, "ft_pointer", msg);
|
|
posed.valid = false;
|
|
} else if (active && anchored && hmd.bPoseIsValid) {
|
|
const bool dragging = leftHeld || tnow < dropHoldUntil;
|
|
if (!dragging) tiltYaw = tiltPitch = 0; // drop finished: back to plain pointing
|
|
const Vec3 dir = Direction(yaw, pitch);
|
|
// Nearest visible overlay along a ray (frozen while dragging).
|
|
struct Hit {
|
|
double along = 1e9;
|
|
std::string key;
|
|
bool scene = false;
|
|
Vec3 point, normal;
|
|
};
|
|
auto nearest = [&](Vec3 from, Vec3 d) {
|
|
Hit h;
|
|
for (const auto &[key, handle] : handles) {
|
|
if (!visible[key]) continue;
|
|
if (sceneGraph[key]) {
|
|
// Plane test: overlay origin and its +Z normal, within sceneRadius of the origin.
|
|
vr::ETrackingUniverseOrigin uo;
|
|
vr::HmdMatrix34_t t{};
|
|
if (overlay->GetOverlayTransformAbsolute(handle, &uo, &t) != vr::VROverlayError_None) continue;
|
|
const Vec3 center = Position(t), normal{t.m[0][2], t.m[1][2], t.m[2][2]};
|
|
const double denom = Dot(d, normal);
|
|
if (std::fabs(denom) < 1e-4) continue;
|
|
const double along = Dot(center - from, normal) / denom;
|
|
const Vec3 at = from + d * along;
|
|
if (along > 0.05 && along < h.along && std::sqrt(Dot(at - center, at - center)) <= sceneRadius)
|
|
h.along = along, h.key = key, h.scene = true, h.point = at, h.normal = normal;
|
|
continue;
|
|
}
|
|
vr::VROverlayIntersectionParams_t params{};
|
|
params.vSource = {float(from.x), float(from.y), float(from.z)};
|
|
params.vDirection = {float(d.x), float(d.y), float(d.z)};
|
|
params.eOrigin = vr::TrackingUniverseStanding;
|
|
vr::VROverlayIntersectionResults_t r{};
|
|
if (overlay->ComputeOverlayIntersection(handle, ¶ms, &r) && r.fDistance > 0.05f &&
|
|
r.fDistance < h.along) {
|
|
h.along = r.fDistance, h.key = key, h.scene = false;
|
|
h.point = {r.vPoint.v[0], r.vPoint.v[1], r.vPoint.v[2]};
|
|
h.normal = {r.vNormal.v[0], r.vNormal.v[1], r.vNormal.v[2]};
|
|
}
|
|
}
|
|
return h;
|
|
};
|
|
// Unchanged frames (see the top): the last pass still holds.
|
|
const bool reuse = !dragging && pass.valid && tnow - pass.at < std::chrono::milliseconds(100) &&
|
|
pass.gen == visibleGen && std::fabs(yaw - pass.yaw) < 1e-6 &&
|
|
std::fabs(pitch - pass.pitch) < 1e-6 && Length(anchor - pass.anchor) < 0.001 &&
|
|
Length(eye - pass.eye) < 0.005;
|
|
double best = pass.best, distance = pass.distance;
|
|
std::string bestKey = pass.key;
|
|
bool bestScene = pass.scene, onEdge = pass.onEdge, occluded = pass.occluded;
|
|
Vec3 point = pass.point;
|
|
if (!reuse) {
|
|
Hit first;
|
|
if (!dragging) first = nearest(anchor, dir);
|
|
best = first.along;
|
|
bestKey = first.key;
|
|
bestScene = first.scene;
|
|
Vec3 bestPoint = first.point, bestNormal = first.normal;
|
|
onEdge = false;
|
|
if (!dragging && best < 1e8 && !bestScene) {
|
|
edgeKey = bestKey, edgePoint = bestPoint, edgeNormal = Normalize(bestNormal), edgeLast = bestPoint;
|
|
} else if (!dragging && best >= 1e8 && !edgeKey.empty() && visible.count(edgeKey) &&
|
|
visible.at(edgeKey)) {
|
|
// Just off a panel: stay on its plane (see "Panel edges" at the top).
|
|
const double denom = Dot(dir, edgeNormal);
|
|
if (std::fabs(denom) > 1e-4) {
|
|
const double along = Dot(edgePoint - anchor, edgeNormal) / denom;
|
|
const Vec3 at = anchor + dir * along;
|
|
if (along > 0.05 && std::sqrt(Dot(at - edgeLast, at - edgeLast)) <= edgeReach) {
|
|
best = along;
|
|
bestKey = edgeKey;
|
|
onEdge = true;
|
|
}
|
|
}
|
|
}
|
|
// Held on one of ft-screens' panels that stays where it is: onto whichever of them
|
|
// the ray meets (see the top).
|
|
if (leftHeld && !pressKey.empty()) {
|
|
vr::ETrackingUniverseOrigin uo;
|
|
vr::HmdMatrix34_t now{};
|
|
auto it = handles.find(pressKey);
|
|
bool still = it != handles.end() &&
|
|
overlay->GetOverlayTransformAbsolute(it->second, &uo, &now) == vr::VROverlayError_None;
|
|
for (int i = 0; still && i < 3; ++i)
|
|
for (int j = 0; j < 4; ++j)
|
|
if (std::fabs(now.m[i][j] - pressPose.m[i][j]) > 0.001f) still = false;
|
|
if (still) {
|
|
Hit h;
|
|
for (const auto &[key, handle] : handles) {
|
|
if (!visible[key] || !FramePanel(key)) continue;
|
|
vr::VROverlayIntersectionParams_t params{};
|
|
params.vSource = {float(anchor.x), float(anchor.y), float(anchor.z)};
|
|
params.vDirection = {float(dir.x), float(dir.y), float(dir.z)};
|
|
params.eOrigin = vr::TrackingUniverseStanding;
|
|
vr::VROverlayIntersectionResults_t r{};
|
|
if (overlay->ComputeOverlayIntersection(handle, ¶ms, &r) && r.fDistance > 0.05f &&
|
|
r.fDistance < h.along)
|
|
h.along = r.fDistance, h.key = key;
|
|
}
|
|
if (h.along < 1e8) dragDistance = h.along, lastHit = h.key;
|
|
} else {
|
|
pressKey.clear(); // carried: the lock holds for the rest of this press
|
|
}
|
|
}
|
|
// While dragging: keep the press-time distance and show the non-interactive marker.
|
|
// On a scene-graph plane or a panel's edge: the laser-catching dot goes 5 cm behind it.
|
|
distance = dragging ? dragDistance : (best < 1e8 ? best : freeDistance);
|
|
point = anchor + dir * distance;
|
|
occluded = false;
|
|
if (!dragging) {
|
|
// The cursor lands on what you see under it: the ray above starts at the anchor,
|
|
// not the eye, so after leaning it can pick a panel that something nearer
|
|
// covers from where you are now (panels close together in view, at different
|
|
// depths). Anything in front of the point on the eye's line of sight wins.
|
|
const double toPoint = std::sqrt(Dot(point - eye, point - eye));
|
|
const Hit front = nearest(eye, Normalize(point - eye));
|
|
if (front.along < toPoint - 0.02) {
|
|
occluded = true;
|
|
bestKey = front.key, bestScene = front.scene;
|
|
point = front.point;
|
|
if (!front.scene)
|
|
edgeKey = front.key, edgePoint = front.point, edgeNormal = Normalize(front.normal),
|
|
edgeLast = front.point;
|
|
distance = std::sqrt(Dot(point - anchor, point - anchor));
|
|
best = distance;
|
|
onEdge = false;
|
|
}
|
|
lastDistance = distance, lastHit = bestKey;
|
|
}
|
|
}
|
|
const bool onScene = !dragging && ((bestScene && best < 1e8) || onEdge);
|
|
const bool onPanel = dragging || (best < 1e8 && !onScene);
|
|
const Vec3 sight = Normalize(point - eye);
|
|
if (!dragging && !reuse) {
|
|
// SteamVR Settings (see the top): the dashboard's main panel is hidden and its
|
|
// scene-graph panel shows the page.
|
|
onVrSettings = false;
|
|
const auto mainIt = visible.find("valve.steam.gamepadui.main");
|
|
if (overlay->IsDashboardVisible() && !(mainIt != visible.end() && mainIt->second)) {
|
|
for (const auto &[key, handle] : handles) {
|
|
if (!visible[key] || key.rfind("valve.steam.gamepadui.frame.menu.", 0) != 0) continue;
|
|
vr::ETrackingUniverseOrigin uo;
|
|
vr::HmdMatrix34_t t{};
|
|
if (overlay->GetOverlayTransformAbsolute(handle, &uo, &t) == vr::VROverlayError_None &&
|
|
OnSettingsPage(t, eye, sight))
|
|
onVrSettings = true;
|
|
}
|
|
}
|
|
}
|
|
if (!reuse) {
|
|
pass = {!dragging, tnow, visibleGen, yaw, pitch, anchor, eye,
|
|
best, distance, bestKey, bestScene, onEdge, occluded, point};
|
|
}
|
|
if (onVrSettings != catcherHidesHit) {
|
|
overlay->SetOverlayFlag(cursor, vr::VROverlayFlags_HideLaserIntersection, onVrSettings);
|
|
catcherHidesHit = onVrSettings;
|
|
}
|
|
|
|
if (onVrSettings) {
|
|
// The dot close in front of the page, which is nearer than any guess of ours;
|
|
// the laser-catching dot far behind everything, invisible, so it never covers
|
|
// the page, with SteamVR's hit dot hidden on it.
|
|
const Vec3 near = eye + sight * SETTINGS_DOT, far = eye + sight * SETTINGS_CATCHER;
|
|
double alpha = 1;
|
|
if (gazeOn && !gazeDotAlways) {
|
|
auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
|
|
alpha = std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
|
|
}
|
|
if (tnow < calPanelUntil) alpha = 0;
|
|
markerDot.Alpha(float(alpha));
|
|
markerDot.Place(near, eye, float(2 * SETTINGS_DOT * std::tan(cursorDeg * M_PI / 360)));
|
|
cursorDot.Alpha(0);
|
|
cursorDot.Place(far, eye, float(2 * SETTINGS_CATCHER * std::tan(cursorDeg * M_PI / 360)));
|
|
markerDot.Show(true);
|
|
cursorDot.Show(true);
|
|
} else {
|
|
// On a panel: the non-interactive marker, pulled 5 mm toward the eye so it
|
|
// draws on top. In free space: the interactive dot the laser lands on.
|
|
DotOverlay &show = onPanel ? markerDot : cursorDot, &hide = onPanel ? cursorDot : markerDot;
|
|
const Vec3 at = onPanel ? point + Normalize(eye - point) * 0.005 : onScene ? point + dir * 0.05 : point;
|
|
const double dist = std::sqrt(Dot(at - eye, at - eye));
|
|
// Gaze mode: a pulse for each click, and with POINTER_GAZE_DOT=moving, shown only
|
|
// while something moves it or a press holds it (see the top); transparent
|
|
// otherwise, the laser still lands on it.
|
|
double scale = 1, alpha = 1;
|
|
if (gazeOn) {
|
|
auto secs = [&](Clock::time_point t) { return std::chrono::duration<double>(tnow - t).count(); };
|
|
alpha = gazeDotAlways ? 1.0 : std::clamp(1 - std::min(secs(lastMove) - gazeShow, secs(lastHeld)) / 0.25, 0.0, 1.0);
|
|
const double pulse = secs(pulseAt);
|
|
if (pulse < 0.6) {
|
|
scale = 1 + 1.5 * std::max(0.0, 1 - pulse / 0.3);
|
|
alpha = std::max(alpha, std::clamp((0.6 - pulse) / 0.3, 0.0, 1.0));
|
|
}
|
|
}
|
|
if (tnow < calPanelUntil) alpha = 0; // the calibration panel is up (see the top)
|
|
show.Alpha(float(alpha));
|
|
show.Place(at, eye, float(2 * dist * std::tan(scale * cursorDeg * M_PI / 360)));
|
|
show.Show(true);
|
|
hide.Show(false);
|
|
}
|
|
|
|
// Controller ray: from the eye, aimed at the cursor point, converted from the
|
|
// standing universe to raw tracking space via the HMD's pose in both.
|
|
const Vec3 aimStanding = Normalize(point - eye);
|
|
const auto &S = hmd.mDeviceToAbsoluteTracking, &R = hmdRaw.mDeviceToAbsoluteTracking;
|
|
const Vec3 aim = Rotate(R, RotateInverse(S, aimStanding)); // raw <- head <- standing
|
|
// Origin partway along the line of sight to the cursor (smaller hit dot).
|
|
const double toPoint = std::sqrt(Dot(point - eye, point - eye));
|
|
double originDist = std::max(0.0, std::min(toPoint * originFraction, toPoint - originMargin));
|
|
if (onVrSettings) originDist = std::min(originDist, SETTINGS_ORIGIN); // SteamVR finds the page
|
|
const Vec3 originStanding = eye + Normalize(point - eye) * originDist;
|
|
const Vec3 eyeRaw = Position(R) + Rotate(R, RotateInverse(S, originStanding - eye));
|
|
lastPoint = point, lastOrigin = originStanding, lastAim = aimStanding; // tilt starts from here
|
|
havePoint = true;
|
|
if (debug && tnow - lastDebug > std::chrono::milliseconds(500)) {
|
|
lastDebug = tnow;
|
|
if (systemPointer == vr::k_ulOverlayHandleInvalid) overlay->FindOverlay("system.pointer", &systemPointer);
|
|
std::printf("dbg %s hit=%s dist=%.2f eye->point=%.2f origin=%.2f vrsettings=%d yaw=%.1f pitch=%.1f gaze=%s steamvr_dot=%d primary=%u\n",
|
|
dragging ? "DRAG" : occluded ? "INFRONT" : onEdge ? "EDGE" : onScene ? "SCENE" : (best < 1e8 ? "PANEL" : "FREE"), lastHit.empty() ? "-" : lastHit.c_str(),
|
|
distance, toPoint, originDist, onVrSettings, yaw, pitch,
|
|
!gazeOn ? "off" : tnow - gz.at > std::chrono::milliseconds(150) ? "stale" : gazeOwns ? "owns" : "mouse",
|
|
systemPointer != vr::k_ulOverlayHandleInvalid && overlay->IsOverlayVisible(systemPointer),
|
|
overlay->GetPrimaryDashboardDevice());
|
|
std::fflush(stdout);
|
|
}
|
|
const double ayaw = std::atan2(-aim.x, -aim.z) * 180 / M_PI;
|
|
const double apitch = std::asin(std::clamp(aim.y, -1.0, 1.0)) * 180 / M_PI;
|
|
if (dragging && (tiltYaw != 0 || tiltPitch != 0)) {
|
|
// Keep this drag's tilt applied, about the current cursor point.
|
|
const double yr = tiltYaw * M_PI / 180, pr = tiltPitch * M_PI / 180;
|
|
const Basis base = AimBasis(aimStanding);
|
|
const Vec3 up{0, 1, 0}, side = base.x;
|
|
auto turn = [&](Vec3 v) { return RotateAbout(RotateAbout(v, side, pr), up, yr); };
|
|
const Vec3 o = point + turn(originStanding - point);
|
|
const Basis b{turn(base.x), turn(base.y), turn(base.z)};
|
|
auto toRaw = [&](Vec3 v) { return Rotate(R, RotateInverse(S, v)); };
|
|
const Vec3 oRaw = Position(R) + toRaw(o - eye);
|
|
double q[4];
|
|
BasisQuat({toRaw(b.x), toRaw(b.y), toRaw(b.z)}, q);
|
|
char msg[200];
|
|
std::snprintf(msg, sizeof msg, "posq %.5f %.5f %.5f %.6f %.6f %.6f %.6f", oRaw.x, oRaw.y, oRaw.z, q[0],
|
|
q[1], q[2], q[3]);
|
|
SendTo(out, "ft_pointer", msg);
|
|
posed.valid = false;
|
|
} else if (!posed.valid || tnow - posed.at >= std::chrono::milliseconds(100) ||
|
|
Length(eyeRaw - posed.origin) >= 0.0002 || std::fabs(std::remainder(ayaw - posed.yaw, 360.0)) >= 0.04 ||
|
|
std::fabs(apitch - posed.pitch) >= 0.04) {
|
|
// The driver keeps the last pose: only a change goes out (see "Unchanged frames" at the top).
|
|
char msg[160];
|
|
std::snprintf(msg, sizeof msg, "pose %.5f %.5f %.5f %.4f %.4f", eyeRaw.x, eyeRaw.y, eyeRaw.z, ayaw, apitch);
|
|
SendTo(out, "ft_pointer", msg);
|
|
posed = {true, tnow, eyeRaw, ayaw, apitch};
|
|
}
|
|
}
|
|
|
|
// A held-back press (see the top): held still long enough, it's a real press (a drag);
|
|
// released, it's a click where the pointer is now (this frame's pose has gone out).
|
|
if (!active) aimHeld = aimRight = clickPress = aimHand = confirmLesson = false; // released meanwhile: nothing to click
|
|
if (aimHeld && !aimHand && nudgeMoved < 0.2 && tnow - aimSince >= std::chrono::duration<double>(gazeHold)) {
|
|
aimHeld = false;
|
|
gazeBack = true;
|
|
pressRight = aimRight; // the right button's: a real right press (see the top)
|
|
aimRight = false;
|
|
pressLeft();
|
|
}
|
|
// A keyboard click held still for POINTER_GAZE_HOLD: a real press, then the head drags
|
|
// (from where it is now).
|
|
if (aimHeld && aimHand && hold.promote && !hold.engaged &&
|
|
tnow - aimSince >= std::chrono::duration<double>(gazeHold)) {
|
|
aimHeld = aimHand = false;
|
|
hold.pressed = hold.grip = true;
|
|
if (hmd.bPoseIsValid) {
|
|
const Vec3 f = Normalize({-hm[0][2], -hm[1][2], -hm[2][2]});
|
|
hold.refYaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
|
|
hold.refPitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
|
|
}
|
|
gazeBack = gazeOn;
|
|
pressRight = hold.right;
|
|
pressLeft();
|
|
if (debug) std::printf("hold key %s: pressed (held still)\n", hold.right ? "right" : "left");
|
|
if (debug) std::fflush(stdout);
|
|
}
|
|
if (clickPress) {
|
|
clickPress = false;
|
|
pressLeft();
|
|
clickRelease = true;
|
|
clickReleaseAt = tnow + std::chrono::milliseconds(40);
|
|
} else if (clickRelease && tnow >= clickReleaseAt) {
|
|
clickRelease = false;
|
|
releaseLeft();
|
|
}
|
|
|
|
controllerButtons.Poll(
|
|
[&](const char *button, bool down) {
|
|
SendTo(out, "frametop_relay", std::string("vrbtn ") + button + (down ? " 1" : " 0"));
|
|
},
|
|
inGame);
|
|
|
|
vr::VREvent_t ev;
|
|
while (sys->PollNextEvent(&ev, sizeof ev)) {
|
|
// The dashboard's overlays come and go with it, and a game brings its own.
|
|
if (ev.eventType == vr::VREvent_DashboardActivated || ev.eventType == vr::VREvent_DashboardDeactivated ||
|
|
ev.eventType == vr::VREvent_DashboardOverlayCreated || ev.eventType == vr::VREvent_SceneApplicationChanged)
|
|
overlays.Kick();
|
|
if (ev.eventType == vr::VREvent_TrackedDeviceActivated || ev.eventType == vr::VREvent_TrackedDeviceDeactivated)
|
|
ours = vr::k_unTrackedDeviceIndexInvalid;
|
|
if (ev.eventType == vr::VREvent_Quit) {
|
|
sys->AcknowledgeQuit_Exiting();
|
|
|
|
overlays.Stop();
|
|
vr::VR_Shutdown();
|
|
return 0;
|
|
}
|
|
}
|
|
// Idle (see the top): wait for a command instead. Not with anything still to finish (the
|
|
// claim pulse, a click's release, a press).
|
|
const bool idle = !active && !handsOn && !claimPending && !claimHeld && !clickRelease && !leftHeld &&
|
|
hold.src == Src::None;
|
|
if (idle) {
|
|
pollfd p{in, POLLIN, 0};
|
|
poll(&p, 1, controllerButtons.Watching(inGame) ? 20 : 250);
|
|
} else {
|
|
std::this_thread::sleep_for(std::chrono::milliseconds(8));
|
|
}
|
|
}
|
|
}
|